The same 9front VM used to work fine on stable/14. The same issue occurs on amd64 as well as Intel platforms. This is what bhyve prints before dying: Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. fbuf frame buffer base: 0x1018ff800000 [sz 33554432] The same VM when started from a stable/14 machine works (albeit starts up much slower since the VM files are on an NFS mount). I see a number of changes to pci_emul.c (comparing git log of stable/14 and stable/15) that may have a bearing on this. Some people have reported this even happens when you try to create a new VM using 9front installation.
Can you provide a simple reproducer? That is, how do I set up a 9front VM, and which device models are you configuring?
fetch https://9front.org/iso/9front-11250.amd64.iso.gz gzcat < 9front-11250.amd64.iso.gz > 9front.iso truncate -s 8G 9front.img bhyve -c 1 \ -s 0,hostbridge \ -s 4,ahci-cd,9front.iso \ -s 5,virtio-blk,9front.img \ -s 10,virtio-net,tap1,mac=08:00:27:68:77:1E \ -s 11,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \ -s 20,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -m 2G -H -w \ 9front Normally: Once you connect vnc to its port (:5900), it should boot up in 9front from the iso itself and then you can do an install on the disk by following directions. What happens is it dies even before 9front is started.
I tried with these parameters on FreeBSD 14.3-RELEASE-p2 : bhyve-lin -S -c sockets=1,cores=1,threads=1 -m 1G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-cd,9front.iso,bootindex=1 \ -s 2,ahci-hd,9front.img \ -s 13,virtio-net,tap4 \ -s 29,fbuf,tcp=0.0.0.0:5904,w=1600,h=950,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:4 < /dev/null & sleep 5 && vncviewer 0:4 & bhyve-lin is bhyve patched with the Corvin's patches. this is what happened : https://ibb.co/MxCwByJg it did not crash but I don't know what to do next....mouse pointer works...
This is normal! If you hit return it will bring up rio. For more help see https://lunduke.locals.com/post/4184409/how-to-install-9front-a-plan-9-fork More at the 9front.org site. No patches should be necessary (I just used normal bhyve on stable/14).
I pressed ENTER and this is what happened : https://ibb.co/j9LxXLVG more messages appeared,but now it is frozen on the last one and it is asking a lot of resources to my machine.
type xga where it asks for (vesa,xga, lcd, ...) [vesa] and hit return a couple of times. Note that this bug report for bhyve crashing on stable/15 when bringing up 9front. If it gets to the first screen (like on stable/14), that means 9front kernel came up and that would be good enough.
What do u want to ask to 15-STABLE ? it is yet in development...
(In reply to mario felicioni from comment #7) > What do u want to ask to 15-STABLE ? it is yet in development... The git branch where FreeBSD 15 is in development is called stable/15.
What's the difference between 15-STABLE and stable/15 ?
(In reply to mario felicioni from comment #9) > What's the difference between 15-STABLE and stable/15 ? stable/15 is the name of the branch containing the code. 15-STABLE is the name of snapshot releases which would be built out of there, but we aren't there yet. HTH.
why stable/15 is called stable/15 and not unstable/15 ? Is it unstable or not ?
(In reply to mario felicioni from comment #11) >why stable/15 is called stable/15 and not unstable/15 ? Is it unstable or not ? It's called stable because it's not for feature development, it's for stability development. Features are developed in main, and then "stabilized" in stable.
folks, please discuss stuff irrelevant to this bug report elsewhere. Thanks!
(In reply to Alexander Ziaee from comment #12) ---> It's called stable because it's not for feature development, it's for stability development. Features are developed in main, and then "stabilized" in stable. My latest consideration. Thats right,for FUTURE development,BUT when it is developed it is NOT stable and cannot be called stable. The snapshots are NOT stable for sure. So,in my HUMBLE opinion,the focus should be on the word "unstable",not on the word "stable". Furthermore,using the word stable two times is confusing.
A NetBSD VM kills bhyve the same way (though it dies during NetBSD booting into multiuser mode).
Though this will be harder to replicate given where it dies (+ there were other issues as upgrade to NetBSD 10.1 didn't work smoothly). Documenting this to indicate the problem is not specific to 9front.
Finally getting around to this.... I set a breakpoint on the line in question and ran bhyve until the assert was triggered. I don't know PCI or bhyve code well but happy to work with anyone to track this down. Note also that a netbsd vm dies on the same assert. Running bhyve under gdb I see [Switching to LWP 252876 of process 93322] Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2a900, idx=0, registration=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) c Continuing. Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2ac00, idx=0, registration=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) Continuing. Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2a900, idx=0, registration=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) Continuing. Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2ac00, idx=0, registration=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) Continuing. Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2a900, idx=0, registration=registration@entry=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) c Continuing. Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2a900, idx=0, registration=registration@entry=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) Continuing. Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2ac00, idx=0, registration=registration@entry=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) Continuing. Thread 20 "vcpu 0" hit Breakpoint 1, modify_bar_registration (pi=0x801e2ac00, idx=0, registration=registration@entry=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 706 assert(error == 0); (gdb) Continuing. Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. Thread 20 "vcpu 0" received signal SIGABRT, Aborted. Sent by thr_kill() from pid 93322 and user 0. 0x0000000801b409ea in thr_kill () from /lib/libsys.so.7 (gdb) where #0 0x0000000801b409ea in thr_kill () from /lib/libsys.so.7 #1 0x00000008018c8804 in raise () from /lib/libc.so.7 #2 0x0000000801979969 in abort () from /lib/libc.so.7 #3 0x00000008018ab3f1 in __assert () from /lib/libc.so.7 #4 0x0000000001067d27 in modify_bar_registration (pi=0x801e2af00, idx=1, registration=registration@entry=1) at /usr/src/usr.sbin/bhyve/pci_emul.c:706 #5 0x00000000010679a9 in register_bar (pi=0x3dbcc, idx=6) at /usr/src/usr.sbin/bhyve/pci_emul.c:723 #6 0x00000000010677f6 in pci_cfgrw (in=<optimized out>, bus=<optimized out>, slot=<optimized out>, func=<optimized out>, coff=<optimized out>, bytes=<optimized out>, valp=0x7fffddbead0c) at /usr/src/usr.sbin/bhyve/pci_emul.c:2367 #7 0x0000000001068134 in pci_emul_cfgdata (ctx=<optimized out>, in=252876, port=<optimized out>, bytes=0, eax=<optimized out>, arg=<optimized out>) at /usr/src/usr.sbin/bhyve/pci_emul.c:2468 #8 0x0000000001080a89 in emulate_inout (ctx=0x801e1a000, vcpu=0x801e0d060, vmexit=vmexit@entry=0x7fffddbeaec8) at /usr/src/usr.sbin/bhyve/amd64/inout.c:222 #9 0x000000000107de50 in vmexit_inout (ctx=0x3dbcc, vcpu=0x6, vmrun=<optimized out>) at /usr/src/usr.sbin/bhyve/amd64/vmexit.c:84 #10 0x0000000001050740 in vm_loop (ctx=0x801e1a000, vcpu=0x801e0d060) at /usr/src/usr.sbin/bhyve/bhyverun.c:651 #11 0x000000000104f4c7 in fbsdrun_start_thread (param=0x801e0b040) at /usr/src/usr.sbin/bhyve/bhyverun.c:563 #12 0x00000008011d0d21 in ?? () from /lib/libthr.so.3 #13 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffddbeb000
Changed version to 15.0-RELEASE since this bug is present on all 15.0-BETAn and 15.0-RCn.
Reproduced on a recent 16.0-CURRENT, the error value is 17 (EEXIST) which is coming from mem.c:mmio_rb_add(). Adding another debug line there and turning on RB_DEBUG shows this: Adding fee00000:fee00fff kern-lapic-mmio Adding fec00000:fec00fff kern-ioapic-mmio Adding fed00000:fed003ff kern-hpet-mmio fbuf frame buffer base: 0x33a20c00000 [sz 33554432] Adding 80000000:ffffffff PCI hole Adding e0000000:efffffff PCI ECFG Adding c2004000:c2005fff ahci@pci.0.4.0 Adding c2002000:c2003fff virtio-blk@pci.0.5.0 Adding c2000000:c2001fff virtio-net@pci.0.10.0 Adding c2007000:c200707f fbuf@pci.0.11.0 Adding c0000000:c1ffffff fbuf@pci.0.11.0 Adding c2006000:c2006fff xhci@pci.0.20.0 Adding c2004000:c2005fff ahci@pci.0.4.0 Adding c2002000:c2003fff virtio-blk@pci.0.5.0 Adding c2000000:c2001fff virtio-net@pci.0.10.0 Adding c2007000:c200707f fbuf@pci.0.11.0 Adding c0000000:c1ffffff fbuf@pci.0.11.0 Adding c2006000:c2006fff xhci@pci.0.20.0 Adding ffffe000:ffffffff ahci@pci.0.4.0 Adding c2004000:c2005fff ahci@pci.0.4.0 Adding ffffe000:ffffffff virtio-blk@pci.0.5.0 Adding c2002000:c2003fff virtio-blk@pci.0.5.0 Adding ffffe000:ffffffff virtio-net@pci.0.10.0 Adding c2000000:c2001fff virtio-net@pci.0.10.0 Adding ffffff80:ffffffff fbuf@pci.0.11.0 Adding c2007000:c200707f fbuf@pci.0.11.0 Adding fe000000:ffffffff fbuf@pci.0.11.0 overlap detected: new fe000000:ffffffff, tree fed00000:fed003ff, 'fbuf@pci.0.11.0' claims region already claimed for 'kern-hpet-mmio' Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. For comparison, running on a 14.3-STABLE system does not abort and shows this: Adding fee00000:fee00fff kern-lapic-mmio Adding fec00000:fec00fff kern-ioapic-mmio Adding fed00000:fed003ff kern-hpet-mmio fbuf frame buffer base: 0x9f041a00000 [sz 16777216] Adding c0000000:c0ffffff fbuf@pci.0.11.0 Adding c1000000:c1001fff ahci@pci.0.4.0 Adding c1002000:c1003fff virtio-blk@pci.0.5.0 Adding c1004000:c1005fff virtio-net@pci.0.10.0 Adding c1006000:c1006fff xhci@pci.0.20.0 Adding c1007000:c100707f fbuf@pci.0.11.0 Adding 80000000:ffffffff PCI hole Adding e0000000:efffffff PCI ECFG Adding c1004000:c1005fff ahci@pci.0.4.0 Adding c1002000:c1003fff virtio-blk@pci.0.5.0 Adding c1000000:c1001fff virtio-net@pci.0.10.0 Adding c1007000:c100707f fbuf@pci.0.11.0 Adding c0000000:c0ffffff fbuf@pci.0.11.0 Adding c1006000:c1006fff xhci@pci.0.20.0 Adding c1004000:c1005fff ahci@pci.0.4.0 Adding c1002000:c1003fff virtio-blk@pci.0.5.0 Adding c1000000:c1001fff virtio-net@pci.0.10.0 Adding c1007000:c100707f fbuf@pci.0.11.0 Adding c0000000:c0ffffff fbuf@pci.0.11.0 Adding c1006000:c1006fff xhci@pci.0.20.0 wrmsr to register 0x401(0) on vcpu 0 Adding ffffe000:ffffffff ahci@pci.0.4.0 Adding c1004000:c1005fff ahci@pci.0.4.0 Adding ffffe000:ffffffff virtio-blk@pci.0.5.0 Adding c1002000:c1003fff virtio-blk@pci.0.5.0 Adding ffffe000:ffffffff virtio-net@pci.0.10.0 Adding c1000000:c1001fff virtio-net@pci.0.10.0 Adding ffffff80:ffffffff fbuf@pci.0.11.0 Adding c1007000:c100707f fbuf@pci.0.11.0 Adding ff000000:ffffffff fbuf@pci.0.11.0 pci_fbuf: mmap_memseg failed pci_fbuf: munmap_memseg failed Adding c0000000:c0ffffff fbuf@pci.0.11.0 Adding fffff000:ffffffff xhci@pci.0.20.0 Adding c1006000:c1006fff xhci@pci.0.20.0 Note the change in 14.3 does not overlap with kern-hpet-mmio: Adding fe000000:ffffffff fbuf@pci.0.11.0 - -CURRENT vs. Adding ff000000:ffffffff fbuf@pci.0.11.0 - 14.3-STABLE Bisecting might be challenging given other changes so commenting here in case someone has an idea of what might have changed.
Some more problem data: 1. openbsd 7.8 can not be installed from downloaded https://cdn.openbsd.org/pub/OpenBSD/7.8/amd64/install78.img on 15.0-stable. Installing it on a 14.3-stable host works and the resulting VM works on 15. The failed install appears to die prior to openbsd loading (in EFI?). I observed the same thing on 9front. install78.iso fails on both 14 and 15 in the same way. 2. netbsd 10.1 crashes every time on 15. Running the same VM on 14 works flawlessly. I did not try installing netbsd from scratch. Others have reported issues with ubuntu, debian etc. on the freebsd forum.
(In reply to Bakul Shah from comment #20) 1. openbsd 7.8 can not be installed from downloaded From what I can tell NetBSD did not support UEFI until 8.0. 2. netbsd 10.1 crashes every time on 15. Running the same VM on 14 works Interesting, I was able to boot 10.1 to the installer on FreeBSD 16.0-CURRENT #0 main-n282278-639e65144aa7. Were you seeing the same abort from this bug report? Did it occur at initial boot the same as 9front? For 9front, I was able to get past the abort on 16-current and 15.0 by reverting this commit and recompiling bhyve. commit fb51ddb20d57a43d666508e600af1bc7ac85c4e8 bhyve: increase fbuf display resolution limit https://github.com/freebsd/freebsd-src/commit/fb51ddb20d57a43d666508e600af1bc7ac85c4e8 This changes the PCI frame buffer memory configuration back to what was used in FreeBSD 14 and as such does not overlap with another device to cause the abort. This is just a workaround as I think there is fundamentally a different issue causing the overlap/abort.
> 1. openbsd 7.8 can not be installed from downloaded > From what I can tell NetBSD did not support UEFI until 8.0. *OpenBSD* not NetBSD! I have been running openbsd at least since 7.3. Note that it doesn't proceed past EFI but doesn't crash so not the same symptom as 9front. > 2. netbsd 10.1 crashes every time on 15. Running the same VM on 14 works > Interesting, I was able to boot 10.1 to the installer on > FreeBSD 16.0-CURRENT #0 main-n282278-639e65144aa7. It may die randomly.... > Were you seeing the same abort from this bug report? Did it occur at initial boot the same as 9front? No. NetBSD comes up but dies randomly. > For 9front, I was able to get past the abort on 16-current and 15.0 by > reverting this commit and recompiling bhyve. Interesting.... The change made 9front work on 15. Thanks!
(In reply to Bakul Shah from comment #22) > *OpenBSD* not NetBSD! I have been running openbsd at least since 7.3. Note that it doesn't proceed past EFI but doesn't crash so not the same symptom as 9front. Doh! I'll blame lack of coffee this morning for mixing it up. I looked again and noticed you had linked to install78.img. I downloaded install78.iso and was able to get to the *OpenBSD* installer. > Interesting.... The change made 9front work on 15. Thanks! Thank you for the confirmation.