I'm trying to use suspend/resume in bhyve. I use FreeBSD 16.0-CURRENT #0 main-n280778-f45608124286 as a host and Ubuntu 25.10 as a guest. I run the guest using: bhyve -c 2 -m 4096 -S -A -I -u -H -P -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 1:0,lpc -s 7:0,virtio-blk,/data/img/ubuntu2510.img -s 5:0,fbuf,tcp=127.0.0.1:5957 ubuntu2510 I can verify that the guest works fine, i.e. I can attach to it by VNC and use it as expected. Then I suspend it using: bhyvectl --suspend=/var/run/libvirt/bhyve/ubuntu2510 --vm ubuntu2510 That also works as expected. Then I try to resume it using: bhyve -r /var/run/libvirt/bhyve/ubuntu2510 -c 2 -m 4096 -S -A -I -u -H -P -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 1:0,lpc -s 7:0,virtio-blk,/data/img/ubuntu2510.img -s 5:0,fbuf,tcp=127.0.0.1:5957 ubuntu2510 At this point VM freezes. I can connect to VNC, but it only seems to display the last frame of the previous session, it doesn't react to any inputs. I also don't see anything on stderr or stdout from bhyve(8). When I do: truss -p $bhyvepid I see that it cycles through: ioctl(7,VM_RUN,0x2a80700efda8) = 0 (0x0) When in this state I destroy the VM, truss prints out the following before exiting: ioctl(7,VM_RUN,0x643205adda8) = 0 (0x0) ioctl(7,VM_RUN,0x64320a9eda8) ERR#6 'Device not configured' ioctl(7,VM_RUN,0x643205adda8) ERR#6 'Device not configured' write(2,"vm_run error -1, errno 6\n",25) = 25 (0x19) _umtx_op(0x2df9dae5f488,UMTX_OP_MUTEX_WAKE2,0x0,0x0,0x0) = 0 (0x0) _umtx_op(0x2df9dae5f488,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0) write(2,"vm_run error -1, errno 6\n",25) = 25 (0x19) _umtx_op(0x641ced8c6a8,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0) _umtx_op(0x641ced8c6a8,UMTX_OP_WAIT_UINT_PRIVATE,0x3,0x0,0x0) = 0 (0x0) write(1,"fbuf frame buffer base: 0x643196"...,165) = 165 (0xa5) <thread 805176 exited> <thread 805175 exited> <thread 805180 exited> <thread 805178 exited> <thread 805177 exited> <thread 805179 exited> <thread 805174 exited> <thread 805172 exited> <thread 805171 exited> <thread 805173 exited> <thread 804987 exited> <thread 805187 exited> _exit(0x4) process exit, rval = 4 In bhyve output I can see after that: Unhandled ps2 mouse command 0x0a Unhandled ps2 mouse command 0x01 Unhandled ps2 mouse command 0x41 Unhandled ps2 mouse command 0x88 vm_run error -1, errno 6 vm_run error -1, errno 6 fbuf frame buffer base: 0x64319600000 [sz 33554432] Pausing pci devs... Restoring vm mem... ^M Restoring pci devs... Restoring kernel structs... Resuming pci devs...
A bit info from attaching gdb to the stuck bhyve process: (gdb) info threads Id Target Id Frame 1 LWP 100945 of process 57077 "mevent" _kevent () at _kevent.S:4 2 LWP 514902 of process 57077 "blk-2:0-0" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 3 LWP 514903 of process 57077 "blk-2:0-1" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 4 LWP 514904 of process 57077 "blk-2:0-2" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 5 LWP 514905 of process 57077 "blk-2:0-3" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 6 LWP 514906 of process 57077 "blk-2:0-4" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 7 LWP 514907 of process 57077 "blk-2:0-5" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 8 LWP 514908 of process 57077 "blk-2:0-6" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 9 LWP 514909 of process 57077 "blk-2:0-7" _umtx_op_err () at /usr/src/lib/libsys/amd64/_umtx_op_err.S:38 10 LWP 514910 of process 57077 "rfb" _read () at _read.S:4 * 11 LWP 514911 of process 57077 "vcpu 0" vm_loop (ctx=0x22061e821000, vcpu=0x22061e80d1f0) at /usr/src/usr.sbin/bhyve/bhyverun.c:641 12 LWP 514912 of process 57077 "vcpu 1" ioctl () at ioctl.S:4 13 LWP 514929 of process 57077 "checkpoint thread" _accept () at _accept.S:4 14 LWP 526525 of process 57077 "rfbout" fast_crc32 (crcval=3104297178, buf=<optimized out>, len=<optimized out>) at /usr/src/usr.sbin/bhyve/rfb.c:444 (gdb) list 636 vmrun.cpuset = &dmask; 637 vmrun.cpusetsize = sizeof(dmask); 638 639 while (1) { 640 error = vm_run(vcpu, &vmrun); 641 if (error != 0) 642 break; 643 644 exitcode = vme.exitcode; 645 if (exitcode >= VM_EXITCODE_MAX || (gdb) p vme $6 = {exitcode = VM_EXITCODE_PAUSE, inst_length = 2, rip = 18446744072317830082, u = {inout = {bytes = 0, in = 0, string = 1, rep = 1, port = 65232, eax = 40}, inout_str = {inout = {bytes = 0, in = 0, string = 1, rep = 1, port = 65232, eax = 40}, paging = { cr3 = 0, cpl = 0, cpu_mode = CPU_MODE_REAL, paging_mode = PAGING_MODE_FLAT}, rflags = 4357496832, cr0 = 12884901888, index = 3, count = 14359727411886588043, addrsize = 1082130631, seg_name = 256215471, seg_desc = {base = 1125900041078790, limit = 0, access = 0}, cs_d = 49, cs_base = 240}, paging = {gpa = 176073736432, fault_type = 0}, inst_emul = {gpa = 176073736432, gla = 0, cs_base = 0, cs_d = 0, paging = {cr3 = 4357496832, cpl = 0, cpu_mode = CPU_MODE_64BIT, paging_mode = PAGING_MODE_64}, vie = { inst = "\213\200\360\000\000\000H\307\307\000\200@\257\211E", num_valid = 15 '\017', num_processed = 6 '\006', addrsize = 8 '\b', opsize = 4 '\004', rex_w = 0 '\000', rex_r = 0 '\000', rex_x = 0 '\000', rex_b = 0 '\000', rex_present = 0 '\000', repz_present = 0 '\000', repnz_present = 0 '\000', opsize_override = 0 '\000', addrsize_override = 0 '\000', segment_override = 0 '\000', mod = 2 '\002', reg = 0 '\000', rm = 0 '\000', ss = 0 '\000', vex_present = 0 '\000', vex_l = 0 '\000', index = 0 '\000', base = 0 '\000', disp_bytes = 4 '\004', imm_bytes = 0 '\000', scale = 0 '\000', vex_reg = 0 '\000', vex_pp = 0 '\000', _sparebits = 0 '\000', _sparebytes = "\000", base_register = 0, index_register = 49, segment_register = 49, displacement = 240, immediate = 0, decoded = 1 '\001', _sparebyte = 0 '\000', op = {op_byte = 139 '\213', op_type = 1 '\001', op_flags = 0}}}, vmx = {status = -19922704, exit_reason = 40, exit_qualification = 0, inst_type = 0, inst_error = 0}, svm = { exitcode = 176073736432, exitinfo1 = 0, exitinfo2 = 0}, bpt = {inst_length = -19922704}, dbg = {trace_trap = -19922704, pushf_intercept = 40, tf_shadow_val = 0, paging = {cr3 = 0, cpl = 0, cpu_mode = CPU_MODE_REAL, paging_mode = 62529536}}, msr = { code = 4275044592, wval = 0}, spinup_ap = {vcpu = -19922704, rip = 0}, hlt = {rflags = 176073736432, intr_status = 0}, ioapic_eoi = {vector = -19922704}, suspended = {how = 4275044592}, ipi = {mode = 4275044592, vector = 40 '('}, task_switch = { tsssel = 240, ext = 40, errcode = 0, errcode_valid = 0, reason = TSR_CALL, paging = {cr3 = 0, cpl = 62529536, cpu_mode = CPU_MODE_PROTECTED, paging_mode = PAGING_MODE_FLAT}}}} (gdb) n 644 exitcode = vme.exitcode; (gdb) n 645 if (exitcode >= VM_EXITCODE_MAX || (gdb) n 646 vmexit_handlers[exitcode] == NULL) { (gdb) n 645 if (exitcode >= VM_EXITCODE_MAX || (gdb) n 651 rc = (*vmexit_handlers[exitcode])(ctx, vcpu, &vmrun); (gdb) n 653 switch (rc) { (gdb) p rc $7 = 0 (gdb) Apparently the userland part of bhyve cycling is looping within the VM_EXITCODE_PAUSE state.
Updated -CURRENT to 16.0-CURRENT #1 main-n281878-587490dabc64: Thu Nov 13 20:55:10 CET 2025, no effect.
Have you tried running the VM without VNC? Or, for example, adding -l com1,stdio? I'm seeing a similar issue with VNC, but I'd like to localise the problem.
(In reply to Aleksandr Fedorov from comment #3) Yes, it does somewhat work for me if I disable VNC and enable the NMDM console. "Somewhat" is because the virtio networking doesn't work after resume, e.g.: [ 593.333434] virtio_net virtio0 enp0s2: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 351912 ms [ 593.335253] virtio_net virtio0 enp0s2: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 351912000 usecs ago [ 598.455996] virtio_net virtio0 enp0s2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 357031 ms [ 598.457399] virtio_net virtio0 enp0s2: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 357031000 usecs ago
Created attachment 265476 [details] dmesg.boot
Checked on 14.3-RELEASE-p5 releng/14.3-n271450-382f54740ee4, everything works, including VNC. So it's likely a regression in -CURRENT (cannot be 100% sure because it's a different box with a different hardware, but also Intel: i5-6600.