Bug 290735 - bhyve resume hangs
Summary: bhyve resume hangs
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 16.0-CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-11-02 09:39 UTC by Roman Bogorodskiy
Modified: 2025-11-20 17:38 UTC (History)
6 users (show)

See Also:


Attachments
dmesg.boot (23.58 KB, text/plain)
2025-11-17 18:26 UTC, Roman Bogorodskiy
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Roman Bogorodskiy freebsd_committer freebsd_triage 2025-11-02 09:39:22 UTC
I'm trying to use suspend/resume in bhyve.

I use FreeBSD 16.0-CURRENT #0 main-n280778-f45608124286 as a host and Ubuntu 25.10 as a guest.

I run the guest using:

bhyve -c 2 -m 4096 -S -A -I -u -H -P -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 1:0,lpc -s 7:0,virtio-blk,/data/img/ubuntu2510.img -s 5:0,fbuf,tcp=127.0.0.1:5957 ubuntu2510

I can verify that the guest works fine, i.e. I can attach to it by VNC and use it as expected.

Then I suspend it using:

bhyvectl --suspend=/var/run/libvirt/bhyve/ubuntu2510 --vm ubuntu2510

That also works as expected. Then I try to resume it using:

bhyve -r /var/run/libvirt/bhyve/ubuntu2510 -c 2 -m 4096 -S -A -I -u -H -P -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 1:0,lpc -s 7:0,virtio-blk,/data/img/ubuntu2510.img -s 5:0,fbuf,tcp=127.0.0.1:5957 ubuntu2510

At this point VM freezes. I can connect to VNC, but it only seems to display the last frame of the previous session, it doesn't react to any inputs. I also don't see anything on stderr or stdout from bhyve(8).

When I do:

truss -p $bhyvepid

I see that it cycles through:

ioctl(7,VM_RUN,0x2a80700efda8)                   = 0 (0x0)

When in this state I destroy the VM, truss prints out the following before exiting:

ioctl(7,VM_RUN,0x643205adda8)                    = 0 (0x0)
ioctl(7,VM_RUN,0x64320a9eda8)                    ERR#6 'Device not configured'
ioctl(7,VM_RUN,0x643205adda8)                    ERR#6 'Device not configured'
write(2,"vm_run error -1, errno 6\n",25)         = 25 (0x19)
_umtx_op(0x2df9dae5f488,UMTX_OP_MUTEX_WAKE2,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x2df9dae5f488,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
write(2,"vm_run error -1, errno 6\n",25)         = 25 (0x19)
_umtx_op(0x641ced8c6a8,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0)
_umtx_op(0x641ced8c6a8,UMTX_OP_WAIT_UINT_PRIVATE,0x3,0x0,0x0) = 0 (0x0)
write(1,"fbuf frame buffer base: 0x643196"...,165) = 165 (0xa5)
<thread 805176 exited>
<thread 805175 exited>
<thread 805180 exited>
<thread 805178 exited>
<thread 805177 exited>
<thread 805179 exited>
<thread 805174 exited>
<thread 805172 exited>
<thread 805171 exited>
<thread 805173 exited>
<thread 804987 exited>
<thread 805187 exited>
_exit(0x4)
process exit, rval = 4


In bhyve output I can see after that:

Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88
vm_run error -1, errno 6
vm_run error -1, errno 6
fbuf frame buffer base: 0x64319600000 [sz 33554432]
Pausing pci devs...
Restoring vm mem...
^M
Restoring pci devs...
Restoring kernel structs...
Resuming pci devs...
Comment 1 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-11-12 19:10:54 UTC
A bit info from attaching gdb to the stuck bhyve process:

(gdb) info threads                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                        
  Id   Target Id                                       Frame                                                                                                                                                                                                            
  1    LWP 100945 of process 57077 "mevent"            _kevent () at _kevent.S:4                                                                                                                                                                                        
  2    LWP 514902 of process 57077 "blk-2:0-0"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  3    LWP 514903 of process 57077 "blk-2:0-1"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  4    LWP 514904 of process 57077 "blk-2:0-2"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  5    LWP 514905 of process 57077 "blk-2:0-3"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  6    LWP 514906 of process 57077 "blk-2:0-4"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  7    LWP 514907 of process 57077 "blk-2:0-5"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  8    LWP 514908 of process 57077 "blk-2:0-6"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  9    LWP 514909 of process 57077 "blk-2:0-7"         _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38                                                                                                                                                   
  10   LWP 514910 of process 57077 "rfb"               _read () at _read.S:4                                                                                                                                                                                            
* 11   LWP 514911 of process 57077 "vcpu 0"            vm_loop (ctx=0x22061e821000, vcpu=0x22061e80d1f0) at /usr/src/usr.sbin/bhyve/bhyverun.c:641                                                                                                                      
  12   LWP 514912 of process 57077 "vcpu 1"            ioctl () at ioctl.S:4                                                                                                                                                                                            
  13   LWP 514929 of process 57077 "checkpoint thread" _accept () at _accept.S:4                                                                                                                                                                                        
  14   LWP 526525 of process 57077 "rfbout"            fast_crc32 (crcval=3104297178, buf=<optimized out>, len=<optimized out>) at /usr/src/usr.sbin/bhyve/rfb.c:444                                                                                                    
(gdb) list                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                        
636             vmrun.cpuset = &dmask;                                                                                                                                                                                                                                  
637             vmrun.cpusetsize = sizeof(dmask);                                                                                                                                                                                                                       
638                                                                                                                                                                                                                                                                     
639             while (1) {                                                                                                                                                                                                                                             
640                     error = vm_run(vcpu, &vmrun);                                                                                                                                                                                                                   
641                     if (error != 0)                                                                                                                                                                                                                                 
642                             break;                                                                                                                                                                                                                                  
643                                                                                                                                                                                                                                                                     
644                     exitcode = vme.exitcode;                                                                                                                                                                                                                        
645                     if (exitcode >= VM_EXITCODE_MAX ||                                                                                                                                                                                                              
(gdb) p vme                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                        
$6 = {exitcode = VM_EXITCODE_PAUSE, inst_length = 2, rip = 18446744072317830082, u = {inout = {bytes = 0, in = 0, string = 1, rep = 1, port = 65232, eax = 40}, inout_str = {inout = {bytes = 0, in = 0, string = 1, rep = 1, port = 65232, eax = 40}, paging = {       
        cr3 = 0, cpl = 0, cpu_mode = CPU_MODE_REAL, paging_mode = PAGING_MODE_FLAT}, rflags = 4357496832, cr0 = 12884901888, index = 3, count = 14359727411886588043, addrsize = 1082130631, seg_name = 256215471, seg_desc = {base = 1125900041078790, limit = 0,      
        access = 0}, cs_d = 49, cs_base = 240}, paging = {gpa = 176073736432, fault_type = 0}, inst_emul = {gpa = 176073736432, gla = 0, cs_base = 0, cs_d = 0, paging = {cr3 = 4357496832, cpl = 0, cpu_mode = CPU_MODE_64BIT, paging_mode = PAGING_MODE_64}, vie = {  
        inst = "\213\200\360\000\000\000H\307\307\000\200@\257\211E", num_valid = 15 '\017', num_processed = 6 '\006', addrsize = 8 '\b', opsize = 4 '\004', rex_w = 0 '\000', rex_r = 0 '\000', rex_x = 0 '\000', rex_b = 0 '\000', rex_present = 0 '\000',            
        repz_present = 0 '\000', repnz_present = 0 '\000', opsize_override = 0 '\000', addrsize_override = 0 '\000', segment_override = 0 '\000', mod = 2 '\002', reg = 0 '\000', rm = 0 '\000', ss = 0 '\000', vex_present = 0 '\000', vex_l = 0 '\000',               
        index = 0 '\000', base = 0 '\000', disp_bytes = 4 '\004', imm_bytes = 0 '\000', scale = 0 '\000', vex_reg = 0 '\000', vex_pp = 0 '\000', _sparebits = 0 '\000', _sparebytes = "\000", base_register = 0, index_register = 49, segment_register = 49,            
        displacement = 240, immediate = 0, decoded = 1 '\001', _sparebyte = 0 '\000', op = {op_byte = 139 '\213', op_type = 1 '\001', op_flags = 0}}}, vmx = {status = -19922704, exit_reason = 40, exit_qualification = 0, inst_type = 0, inst_error = 0}, svm = {     
      exitcode = 176073736432, exitinfo1 = 0, exitinfo2 = 0}, bpt = {inst_length = -19922704}, dbg = {trace_trap = -19922704, pushf_intercept = 40, tf_shadow_val = 0, paging = {cr3 = 0, cpl = 0, cpu_mode = CPU_MODE_REAL, paging_mode = 62529536}}, msr = {          
      code = 4275044592, wval = 0}, spinup_ap = {vcpu = -19922704, rip = 0}, hlt = {rflags = 176073736432, intr_status = 0}, ioapic_eoi = {vector = -19922704}, suspended = {how = 4275044592}, ipi = {mode = 4275044592, vector = 40 '('}, task_switch = {             
      tsssel = 240, ext = 40, errcode = 0, errcode_valid = 0, reason = TSR_CALL, paging = {cr3 = 0, cpl = 62529536, cpu_mode = CPU_MODE_PROTECTED, paging_mode = PAGING_MODE_FLAT}}}}                                                                                   

(gdb) n                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                        
644                     exitcode = vme.exitcode;                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                        
(gdb) n                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                        
645                     if (exitcode >= VM_EXITCODE_MAX ||                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                        
(gdb) n                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                        
646                         vmexit_handlers[exitcode] == NULL) {                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                        
(gdb) n                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                        
645                     if (exitcode >= VM_EXITCODE_MAX ||                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                        
(gdb) n                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                        
651                     rc = (*vmexit_handlers[exitcode])(ctx, vcpu, &vmrun);                                                                                                                                                                                           
                                                                                                                                                                                                                                                                        
(gdb) n                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                        
653                     switch (rc) {                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                        
(gdb) p rc                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                        
$7 = 0                                                                                                                                                                                                                                                                  
(gdb)  

Apparently the userland part of bhyve cycling is looping within the VM_EXITCODE_PAUSE state.
Comment 2 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-11-15 09:39:08 UTC
Updated -CURRENT to 16.0-CURRENT #1 main-n281878-587490dabc64: Thu Nov 13 20:55:10 CET 2025, no effect.
Comment 3 Aleksandr Fedorov freebsd_committer freebsd_triage 2025-11-16 18:39:16 UTC
Have you tried running the VM without VNC? Or, for example, adding -l com1,stdio?

I'm seeing a similar issue with VNC, but I'd like to localise the problem.
Comment 4 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-11-17 17:50:00 UTC
(In reply to Aleksandr Fedorov from comment #3)

Yes, it does somewhat work for me if I disable VNC and enable the NMDM console.
"Somewhat" is because the virtio networking doesn't work after resume, e.g.:

[  593.333434] virtio_net virtio0 enp0s2: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 351912 ms
[  593.335253] virtio_net virtio0 enp0s2: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 351912000 usecs ago
[  598.455996] virtio_net virtio0 enp0s2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 357031 ms
[  598.457399] virtio_net virtio0 enp0s2: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 357031000 usecs ago
Comment 5 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-11-17 18:26:09 UTC
Created attachment 265476 [details]
dmesg.boot
Comment 6 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-11-20 17:38:43 UTC
Checked on 14.3-RELEASE-p5 releng/14.3-n271450-382f54740ee4, everything works, including VNC. So it's likely a regression in -CURRENT (cannot be 100% sure because it's a different box with a different hardware, but also Intel: i5-6600.