I've been using this script to run an OpenBSD VM ever since bhyve landed: #!/bin/sh RAM=1G VM=vpn VMROOT=/usr/local/vm CPUS=2 bhyvectl --vm=${VM} --destroy > /dev/null 2>&1 while [ 1 ]; do bhyve -A -D -H -P -w -S -u -c ${CPUS} -m ${RAM} \ -s 0,amd_hostbridge \ -s 3,virtio-blk,/dev/zvol/tank/vm/${VM} \ -s 10,virtio-net,tap0,mac=1e:17:37:23:2f:cb \ -s 20,virtio-rnd \ -s 31,lpc \ -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd ${VM} bhyve_exit=$? if [ $bhyve_exit -ne 0 ]; then break fi done case $bhyve_exit in 0|1|2) bhyvectl --vm=${VM} --destroy > /dev/null 2>&1 ;; esac exit $bhyve_exit This has worked up until about a month ago. Now the VM throws "io address conflict" and "can't map i/o space" errors. Adding -W and/or -x doesn't help, and adding -Y causes the boot loader to hang. This is what happens now: probing: pc0 com0 com1 mem[640K 1000M 16M 3M] disk: hd0 >> OpenBSD/amd64 BOOTX64 3.65 switching console to com0 >> OpenBSD/amd64 BOOTX64 3.65 boot> NOTE: random seed is being reused. booting hd0a:/bsd: 17233228+4138000+368672+0+1241088 [1360125+128+1320504+1012965]=0x19730a0 entry point at 0x1001000 [ using 3694752 bytes of bsd ELF symbol table ] Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2023 OpenBSD. All rights reserved. https://www.OpenBSD.org OpenBSD 7.3-current (GENERIC.MP) #1358: Thu Aug 31 09:39:50 MDT 2023 deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 1032982528 (985MB) avail mem = 981991424 (936MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x3fbcf000 (11 entries) bios0: vendor BHYVE version "14.0" date 10/17/2021 bios0: FreeBSD BHYVE efi0 at bios0: UEFI 2.7 efi0: BHYVE rev 0x10000 acpi0 at bios0: ACPI 5.1 acpi0: sleep states S5 acpi0: tables DSDT FACP APIC HPET MCFG SPCR acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz, 2397.78 MHz, 06-3f-02 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,FMA3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 134MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz, 2397.67 MHz, 06-3f-02 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,FMA3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache cpu1: smt 0, core 0, package 1 ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 32 pins acpihpet0 at acpi0: 16777216 Hz acpimcfg0 at acpi0 acpimcfg0: addr 0xe0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PC00) acpipci0 at acpi0 PC00 com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo com0: console com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo com2 at acpi0 COM3 addr 0x3e8/0x8 irq 4: ns16550a, 16 byte fifo com3 at acpi0 COM4 addr 0x2e8/0x8 irq 3: ns16550a, 16 byte fifo acpicmos0 at acpi0 "Bhyve_V_Gen_Counter_V1" at acpi0 not configured cpu0: using VERW MDS workaround pvbus0 at mainbus0: bhyve pci0 at mainbus0 bus 0 0:3:0: io address conflict 0xc000/0x80 0:10:0: io address conflict 0xc080/0x40 0:20:0: io address conflict 0xc0c0/0x20 pchb0 at pci0 dev 0 function 0 vendor "AMD", unknown product 0x7432 rev 0x00 virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00virtio0: can't map i/o space : Cannot attach (5) virtio1 at pci0 dev 10 function 0 "Qumranet Virtio Network" rev 0x00 vio0 at virtio1: address 1e:17:37:23:2f:cb virtio1: msix per-VQ virtio2 at pci0 dev 20 function 0 "Qumranet Virtio RNG" rev 0x00 viornd0 at virtio2 virtio2: msix per-VQ pcib0 at pci0 dev 31 function 0 "Intel 82371SB ISA" rev 0x00 isa0 at pcib0 isadma0 at isa0 efifb at mainbus0 not configured vm_run error -1, errno 5
> This has worked up until about a month ago. Can you convert this into a pair of known-good and known-bad revisions? That is, what is the last revision where this worked, and what is the first revision where it doesn't? When I try booting the OpenBSD 7.3 installer, I hit an assertion failure in modify_bar_registration(): 697 assert(error == 0); (gdb) bt #0 thr_kill () at thr_kill.S:4 #1 0x00000008016f83b4 in __raise (s=s@entry=6) at /root/freebsd/lib/libc/gen/raise.c:50 #2 0x00000008017aa4c9 in abort () at /root/freebsd/lib/libc/stdlib/abort.c:65 #3 0x00000008016db741 in __assert (func=<optimized out>, file=<optimized out>, line=line@entry=703, failedexpr=<optimized out>) at /root/freebsd/lib/libc/gen/assert.c:49 #4 0x0000000001059205 in modify_bar_registration (pi=0x801e32600, idx=0, registration=registration@entry=1) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/pci_emul.c:703 #5 0x0000000001058ceb in register_bar (pi=0xb6df6, idx=6) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/pci_emul.c:720 #6 0x0000000001058af9 in pci_cfgrw (in=<optimized out>, bus=<optimized out>, slot=<optimized out>, func=<optimized out>, coff=<optimized out>, bytes=<optimized out>, valp=0x7fffdeff5d0c) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/pci_emul.c:2375 #7 0x0000000001059614 in pci_emul_cfgdata (ctx=<optimized out>, in=749046, port=<optimized out>, bytes=0, eax=<optimized out>, arg=<optimized out>) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/pci_emul.c:2451 #8 0x000000000106e1b9 in emulate_inout (ctx=0x801e19500, vcpu=0x801e1b070, vmexit=vmexit@entry=0x7fffdeff5ec8) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/amd64/inout.c:223 #9 0x000000000106b9f0 in vmexit_inout (ctx=0xb6df6, vcpu=0x6, vmrun=<optimized out>) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/amd64/vmexit.c:84 #10 0x000000000104b610 in vm_loop (ctx=0x801e19500, vcpu=0x801e1b070) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/bhyverun.c:523 #11 0x000000000104a217 in fbsdrun_start_thread (param=0x801e25030) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/bhyverun.c:434 #12 0x000000080162aac5 in thread_start (curthread=0x801e15f00) at /root/freebsd/lib/libthr/thread/thr_create.c:290 #13 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdeff6000 (gdb) frame 4 #4 0x0000000001059205 in modify_bar_registration (pi=0x801e32600, idx=0, registration=registration@entry=1) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/pci_emul.c:703 703 assert(error == 0); (gdb) p pi $2 = (struct pci_devinst *) 0x801e32600 (gdb) p pi->pi_bar[0] $3 = {type = PCIBAR_IO, size = 128, addr = 0, lobits = 1 '\001'} (gdb) frame 9 #9 0x000000000106b9f0 in vmexit_inout (ctx=0xb6fa6, vcpu=0x6, vmrun=<optimized out>) at /usr/home/markj/src/freebsd/usr.sbin/bhyve/amd64/vmexit.c:84 84 error = emulate_inout(ctx, vcpu, vme) (gdb) p vme->u.inout $3 = {bytes = 4, in = 0, string = 0, rep = 0, port = 3324, eax = 0}
NetBSD on FreeBSD 14 probably has the same problems (assert in modify_bar_registration): screenshot: https://snipboard.io/IAg1V3.jpg bt: [New LWP 109671 of process 45161] [New LWP 109672 of process 45161] Assertion failed: (error == 0), function modify_bar_registration, file /usr/jails/src/src_14.0/src/usr.sbin/bhyve/pci_emul.c, line 637. Thread 20 "vcpu 0" received signal SIGABRT, Aborted. Sent by thr_kill() from pid 45161 and user 0. [Switching to LWP 109670 of process 45161] thr_kill () at thr_kill.S:4 4 RSYSCALL(thr_kill) (gdb) un Ambiguous command "un": undisplay, unset, until. (gdb) bt #0 thr_kill () at thr_kill.S:4 #1 0x000000080174b324 in __raise (s=s@entry=6) at /usr/jails/src/src_14.0/src/lib/libc/gen/raise.c:50 #2 0x00000008017fc219 in abort () at /usr/jails/src/src_14.0/src/lib/libc/stdlib/abort.c:65 #3 0x000000080172e6b1 in __assert (func=<optimized out>, file=<optimized out>, line=line@entry=637, failedexpr=<optimized out>) at /usr/jails/src/src_14.0/src/lib/libc/gen/assert.c:49 #4 0x000000000106f399 in modify_bar_registration (pi=0x801e36f00, idx=1, registration=registration@entry=1) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/pci_emul.c:637 #5 0x000000000106ef5b in register_bar (pi=0x1ac66, idx=6) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/pci_emul.c:654 #6 0x000000000106ed69 in pci_cfgrw (in=<optimized out>, bus=<optimized out>, slot=<optimized out>, func=<optimized out>, coff=<optimized out>, bytes=<optimized out>, valp=0x7fffddbead0c) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/pci_emul.c:2294 #7 0x000000000106f7a4 in pci_emul_cfgdata (ctx=<optimized out>, in=109670, port=<optimized out>, bytes=0, eax=<optimized out>, arg=<optimized out>) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/pci_emul.c:2369 #8 0x000000000105c9e9 in emulate_inout (ctx=0x801e19680, vcpu=0x801e1b080, vmexit=vmexit@entry=0x7fffddbeaec8) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/inout.c:223 #9 0x0000000001050810 in vmexit_inout (ctx=0x1ac66, vcpu=0x6, vmrun=<optimized out>) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/bhyverun.c:588 #10 0x0000000001050744 in vm_loop (ctx=0x801e19680, vcpu=0x801e1b080) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/bhyverun.c:972 #11 0x000000000105063f in fbsdrun_start_thread (param=0x801e1a040) at /usr/jails/src/src_14.0/src/usr.sbin/bhyve/bhyverun.c:524 #12 0x000000080167db35 in thread_start (curthread=0x801ec0500) at /usr/jails/src/src_14.0/src/lib/libthr/thread/thr_create.c:290 #13 0x0000000000000000 in ?? ()
Would be good to get more details of the actual config space write the guest OS is making. I suspect the assertion failure is because we are getting a conflict (looks like from comment #1 that OpenBSD is setting the base address of an I/O BAR to 0, and probably some other BAR is already at I/O port 0, hence the error return and assertion failure). The bug though in that case is why is OpenBSD writing a 0 base address. Are we advertising an existing base address of 0 for BARs in bhyve? If so, that might be the regression.
Sorry for the delay in answers. I was trying to replace a bad DIMM in my 2nd identical machine so I could do some bisecting without taking the one machine down. I do know that 565c887 worked (so Aug 31) and e39e6be didn't (Sep 20th). So I have it down to a 20 day window, working on shrinking it more.
I should also note that I have a Windows Server 2022 VM running also, and that one started reporting not enough resources errors for COM2 & COM4. I don't recall these errors before hand, so it might be related.
(In reply to Dustin Marquess from comment #4) Hmm, there were no commits to bhyve or vmm in that window, so that's a bit surprising.
I still don't fully understand the problem yet. What's happening is: - virtio-blk0 has an I/O bar, initially at 0x2000. - something maps it at 0xc000 early during boot, I speculate it's edk2. This happens before any I/O is done. - OpenBSD's boot loader loads the kernel, kernel boots. - OpenBSD enumerates devices, tries to enable virtio-blk0. It sees that the BAR has address 0xc000, and because this apparently conflicts with an existing resource, it disables the BAR by setting its address to 0, but that conflicts with an I/O port owned by atkbd That triggers the assertion failure. I spent quite some time trying to understand what it's conflicting with, the boot logs don't make it clear. I did notice that edk2-bhyve was upgraded recently; I tried downgrading to edk2-bhyve-g202202_10 and the problem went away. Corvin, do you have any idea what's going on here?
So it's an OS bug that it's writing 0 to "disable" a BAR as that doesn't work. However, bhyve crashing isn't really ideal either. For a case like this where there is a conflict, I think we should do something that isn't a crash. Probably the guest OS will assign a new range in the future if it actually cares about the BAR in question. The simplest approach might be to let the "first" BAR that claims a region win and have other register attempts simply fail (but mark the BAR as "unmapped" so we don't try to unregister it in the future). This would not fix the edge case that if the first device to register the range moves its BAR (or disables decoding), the second device's BAR would not start working, instead it would remain disabled. However, given how odd of a case this is, I'm fine with that. Note that this is just about the assertion failure, and it doesn't help with the first error reported as the original bug here which I think we still don't understand.
To be clear, when I downgrade edk2, the I/O BAR for virtio-blk0 stays at 0x2000, it doesn't get remapped. (In reply to John Baldwin from comment #8) If I work around the assertion, OpenBSD does try to map the BAR elsewhere, but its fallback choice then conflicts with PM1A_EVT_ADDR and it gives up. I have a running OpenBSD installation now but I can't find an equivalent to "devinfo -r" which might help me figure out what it thinks is at 0xc000.
Sure enough, downgrading edk2-bhyve made my VM boot again, so I guess I should open up a separate bug for that? Should this one be closed, or do you want it open still? Thanks, Mark, for figuring out the issue!
(In reply to Dustin Marquess from comment #10) I think keeping everything in this bug is fine. Let's see if Corvin has any comment about edk2's behaviour here.
I've enabled BAR remapping in edk2 because it's required for ROM execution (e.g. PXE or VGA ROM) and VGA ROM shadowing. See https://github.com/tianocore/edk2/commit/70f3e62dc73d28962b833373246ef25c865c575e. The remapping range of OVMF is defined by: https://github.com/tianocore/edk2/blob/fb044b7fe893a4545995bfe2701fd38e593355d9/OvmfPkg/Bhyve/PlatformPei/Platform.c#L156-L157 (base = 0xC000, size = 0x4000). We may want to update those values to match our bhyve io range or we may want to adjust our bhyve io range: https://github.com/freebsd/freebsd-src/blob/82ea0132c8b17a7a6067c8a36c6434e587ede6de/usr.sbin/bhyve/pci_emul.c#L133-L134 (base = 0x2000, size = 0x10000). Nevertheless, the OVMF range is part of the bhyve io range. So, that shouldn't be an issue. Another difference: Latest OVMF installs the ACPI tables provided by bhyve, while the old one uses static tables. However, bhyve's tables should report a correct io range (not checked yet) while it looks like the static ones do report a faulty one: https://github.com/tianocore/edk2/blob/fb044b7fe893a4545995bfe2701fd38e593355d9/OvmfPkg/Bhyve/AcpiTables/Dsdt.asl#L77-L83 (base = 0x0D00, size = 0xF300)
(In reply to Corvin Köhne from comment #12) For what it's worth, updating OVMF to use [0x2000, 0x10000] as the I/O port mapping range fixes the problem for me.
Please, how could I used your solution? I'm having the same problem with OpenBSD running in VM. And I'm sure it's edk2 and not bhyve itself responsible. Because I downgraded to 13.2 to test if the problem would not go away -- and it didn't. But downgrading the edk2 port is a bit of trouble... I'll check that option as well, though. BTW, Windows VM runs fine, no problems at all.
Could someone please check if https://reviews.freebsd.org/D42627 solves the issue?
https://reviews.freebsd.org/D42627 indeed solves the problem for me! Many thanks!
Solves for me, too. Thank you.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=231c5bc3a4c964746ccd21f3d44720617dd3d40a commit 231c5bc3a4c964746ccd21f3d44720617dd3d40a Author: Corvin Köhne <corvink@FreeBSD.org> AuthorDate: 2023-11-16 05:58:01 +0000 Commit: Corvin Köhne <corvink@FreeBSD.org> CommitDate: 2023-11-24 09:30:35 +0000 sysutils/edk2: set IO port range properly for bhyve Bhyve uses an io port range of [ 0x2000, 0x10000 ] [1]. At the moment, EDKII is using a subset of this range [ 0xC000, 0x10000 ] [2]. Even though the EDKII range doesn't exceed the bhyve range, it's causing issues on some guests like OpenBSD. We don't know why it's causing issues yet. However, using the same IO port range in EDKII fixes the issue. [1] https://github.com/freebsd/freebsd-src/blob/82ea0132c8b17a7a6067c8a36c6434e587ede6de/usr.sbin/bhyve/pci_emul.c#L133-L134 [2] https://github.com/tianocore/edk2/blob/fb044b7fe893a4545995bfe2701fd38e593355d9/OvmfPkg/Bhyve/PlatformPei/Platform.c#L156-L157 PR: 274389 Reviewed by: manu, markj Approved by: manu Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D42627 sysutils/edk2/Makefile | 2 +- .../files/patch-OvmfPkg_Bhyve_PlatformPei_Platform.c (new) | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
A commit in branch 2023Q4 references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=9b7e23c09b6b4c693a89044b1dd848aedfed0361 commit 9b7e23c09b6b4c693a89044b1dd848aedfed0361 Author: Corvin Köhne <corvink@FreeBSD.org> AuthorDate: 2023-11-16 05:58:01 +0000 Commit: Corvin Köhne <corvink@FreeBSD.org> CommitDate: 2023-11-24 09:38:53 +0000 sysutils/edk2: set IO port range properly for bhyve Bhyve uses an io port range of [ 0x2000, 0x10000 ] [1]. At the moment, EDKII is using a subset of this range [ 0xC000, 0x10000 ] [2]. Even though the EDKII range doesn't exceed the bhyve range, it's causing issues on some guests like OpenBSD. We don't know why it's causing issues yet. However, using the same IO port range in EDKII fixes the issue. [1] https://github.com/freebsd/freebsd-src/blob/82ea0132c8b17a7a6067c8a36c6434e587ede6de/usr.sbin/bhyve/pci_emul.c#L133-L134 [2] https://github.com/tianocore/edk2/blob/fb044b7fe893a4545995bfe2701fd38e593355d9/OvmfPkg/Bhyve/PlatformPei/Platform.c#L156-L157 PR: 274389 Reviewed by: manu, markj Approved by: manu Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D42627 (cherry picked from commit 231c5bc3a4c964746ccd21f3d44720617dd3d40a) sysutils/edk2/Makefile | 2 +- .../files/patch-OvmfPkg_Bhyve_PlatformPei_Platform.c (new) | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
I have same problem with Windows 10 guest. I mean bug with IO ports. For each com port (COM1-4) I see the following status: "There are not enough free resources to operate this device. (Code 12)" I updated port edk2-bhyve to latest version. pkg info | grep edk2 edk2-bhyve-g202308_4 EDK2 Firmware for bhyve The error remains. Version OS: FreeBSD 14.0-RELEASE I'm using vm-bhyve to run a VM guest. To the configuration file of each UEFI guest option added: bhyve_options="-A" Debugging is enabled. There are no errors in the log. Also, the VM with Windows 7 version stops at the boot stage Rolling back to version edk2-bhyve-g202202_10 solves all problems. Where to look for a possible error?
(In reply to arkadyi from comment #20) Just in case you're still having this issue, you might like to know I've opened bug #280286 regarding COM port errors in Windows 10 with FreeBSD-14.1. Maybe you could follow-up there and post your system details, etc. to help narrow down the issue?