I've updated my system to n251155-57512573ae05 and suddenly I am getting: bhyve: Failed to map MSI-X table BAR on 2/0/0: Device busy bhyve: failed to initialize BARs for PCI 2/0/0 device emulation initialization error: Device busy when restarting my development VM. I hadn't seen this problem in the last months while working on iwlwifi (since it was made work). ppt0@pci0:2:0:0: class=0x028000 rev=0x1a hdr=0x00 vendor=0x8086 device=0x2723 subvendor=0x1a56 subdevice=0x1653 vendor = 'Intel Corporation' device = 'Wi-Fi 6 AX200' class = network bar [10] = type Memory, range 64, base rx9e200000, size 16384, disabled cap 01[c8] = powerspec 3 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[40] = PCI-Express 2 endpoint max data 128(128) FLR RO NS max read 128 link x1(x1) speed 5.0(5.0) ASPM L1(L1) ClockPM enabled cap 11[80] = MSI-X supports 16 messages Table in map 0x10[0x2000], PBA in map 0x10[0x3000] ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0018[14c] = LTR 1 ecap 001e[154] = L1 PM Substates 1 Corrected = Advisory Non-Fatal Error The driver for the guest device hasn't updated in 2-ish weeks either so it's not due to updating the guest. It might be a result of a weird state situation of the driver in the guest (firmware paniced) but that shouldn't leave us with something we cannot get back up on next guest start. There've been recent changes to bhyve in that area so it may as well be a side-effect of those? I figured I am documenting this before rebooting my host now.
I assume this has to do with 876 if (!pci_bar_enabled(pcidev, pm)) 877 return (EBUSY); /* XXXKIB enable if _ACTIVATE */ in sys/dev/pci/pci_user.c:pci_bar_mmap() So mmap fails and we won't start again. I am just not sure if that XXXKIB is the right fix or if bhyve should make sure the device is usable?
Indeed, from your pciconf output we have bar [10] = type Memory, range 64, base rx9e200000, size 16384, disabled but we should be enabling memory accesses on the host, even if no driver attaches. I thought this was handled by pci_enable_io_modes.
(In reply to Mark Johnston from comment #2) we do on the host; we should also do this on a bhyve restart; order of events as I understand them currently: host boot bhyve boot driver load driver unload (disabling bar) bhyve shutdown bhyve start -- fails as bar is not enabled. My understanding was that pci_emul_alloc_bar() called right before pci_msix_table_bar() in cfginitbar() should enable the resources again setting PCIM_CMD_MEMEN. So the mmap should not fail. I am currently seeing if anything makes this not work as I thought it would or if I am misunderstanding something.
okay, here's a fix:@@ -487,6 +487,7 @@ cfginitbar(struct vmctx *ctx, struct passthru_softc *sc) struct pci_bar_io bar; enum pcibar_type bartype; uint64_t base, size; + uint16_t cmd, orig_cmd; pi = sc->psc_pi; @@ -533,11 +534,22 @@ cfginitbar(struct vmctx *ctx, struct passthru_softc *sc) sc->psc_bar[i].addr = base; sc->psc_bar[i].lobits = 0; + orig_cmd = pci_get_cfgdata16(pi, PCIR_COMMAND); + /* Allocate the BAR in the guest I/O or MMIO space */ error = pci_emul_alloc_bar(pi, i, bartype, size); if (error) return (-1); + /* + * We may need this here and now to enable the BAR + * before pci_msix_table_bar() tries to mmap it as + * the kernel otherwise returns EBUSY from pci_bar_mmap(). + */ + cmd = pci_get_cfgdata16(pi, PCIR_COMMAND); + if (cmd != orig_cmd) + write_config(&sc->psc_sel, PCIR_COMMAND, 2, cmd); + /* Use same lobits as physical bar */ uint8_t lobits = read_config(&sc->psc_sel, PCIR_BAR(i), 0x01); if (bartype == PCIBAR_MEM32 || bartype == PCIBAR_MEM64) { This is very similar to what @jhb is doing in https://reviews.freebsd.org/D20623 for busmaster. Unfortunately the combined write at the end of cfginit() is too late these days. So we end up with several individual writes to PCIR_COMMAND and possibly yet-another-one not actually chaning anything anymore after all in cfginit().
Thanks, I see now. It might be a bit cleaner to map the MSI-X table after the device and emulated command registers are synced in cfginit().
(In reply to Mark Johnston from comment #5) I'll take my own bug report, see if I can rework the fix and will push it to phabricator for review then.
https://reviews.freebsd.org/D33628
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f1442847c9404d4bc5f5524a0c3362dd39cb14f9 commit f1442847c9404d4bc5f5524a0c3362dd39cb14f9 Author: Bjoern A. Zeeb <bz@FreeBSD.org> AuthorDate: 2021-12-23 14:59:49 +0000 Commit: Bjoern A. Zeeb <bz@FreeBSD.org> CommitDate: 2021-12-29 17:01:05 +0000 bhyve: passthru: enable BARs before possibly mmap(2)ing them The first time we start bhyve with a passthru device everything is fine as on boot we do enable BARs. If a driver (unload) inside bhyve disables the BAR(s) as some Linux drivers do, we need to make sure we re-enable them on next bhyve start. If we are trying to mmap a disabled BAR for MSI-X (PCIOCBARMMAP) the kernel will give us an EBUSY. While we were re-enabling the BAR(s) in the current code loop cfginit() was writing the changes out too late to the real hardware. Move the call to init_msix_table() after the register on the real hardware was updated. That way the kernel will be happy and the mmap will succeed and bhyve will start. Also simplify the code given the last argument to init_msix_table() is unused we do not need to do checks for each bar. [1] MFC after: 3 days PR: 260148 Pointed out by: markj [1] Sponsored by: The FreeBSD Foundation Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D33628 usr.sbin/bhyve/pci_passthru.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=338a1be836308f6d807f8bfe9b335463d537abc4 commit 338a1be836308f6d807f8bfe9b335463d537abc4 Author: Corvin Köhne <C.Koehne@beckhoff.com> AuthorDate: 2022-01-03 14:48:10 +0000 Commit: Bjoern A. Zeeb <bz@FreeBSD.org> CommitDate: 2022-01-03 14:55:10 +0000 bhyve: only init MSI-X table if passthru device supports it Some passthru devices only support MSI instead of MSI-X. For those devices the initialization of MSI-X table will fail. Re-add the check erroneously removed in f1442847c9404d4bc5f5524a0c3362dd39cb14f9. MFC after: 3 days X-MFC with: f1442847c9404d4bc5f5524a0c3362dd39cb14f9 PR: 260148 Reviewed by: manu, bz Differential Revision: https://reviews.freebsd.org/D33728 usr.sbin/bhyve/pci_passthru.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7ea16192a01ea3de7b088ef6dc3cbf29d8e21a54 commit 7ea16192a01ea3de7b088ef6dc3cbf29d8e21a54 Author: Bjoern A. Zeeb <bz@FreeBSD.org> AuthorDate: 2021-12-23 14:59:49 +0000 Commit: Bjoern A. Zeeb <bz@FreeBSD.org> CommitDate: 2022-01-09 02:38:58 +0000 bhyve: passthru: enable BARs before possibly mmap(2)ing them The first time we start bhyve with a passthru device everything is fine as on boot we do enable BARs. If a driver (unload) inside bhyve disables the BAR(s) as some Linux drivers do, we need to make sure we re-enable them on next bhyve start. If we are trying to mmap a disabled BAR for MSI-X (PCIOCBARMMAP) the kernel will give us an EBUSY. While we were re-enabling the BAR(s) in the current code loop cfginit() was writing the changes out too late to the real hardware. Move the call to init_msix_table() after the register on the real hardware was updated. That way the kernel will be happy and the mmap will succeed and bhyve will start. Also simplify the code given the last argument to init_msix_table() is unused we do not need to do checks for each bar. [1] Some passthru devices only support MSI instead of MSI-X. For those devices the initialization of MSI-X table will fail. Re-add (or in the MFC case keep) the check erroneously removed in the initial commit. [2] PR: 260148 Pointed out by: markj [1] Sponsored by: The FreeBSD Foundation Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D33628 Submitted by: (C.Koehne beckhoff.com) [2] Reviewed by: manu, bz Differential Revision: https://reviews.freebsd.org/D33728 (cherry picked from commit f1442847c9404d4bc5f5524a0c3362dd39cb14f9) (cherry picked from commit 338a1be836308f6d807f8bfe9b335463d537abc4) usr.sbin/bhyve/pci_passthru.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
I am seeing this problem on today's 13-STABLE. I'm passing through a USB 3.0 controller to a bhyve VM and it works the first time but if the VM reboots it fails with these error messages: bhyve: Failed to map MSI-X table BAR on 6/0/3: Device busy bhyve: failed to initialize BARs for PCI 6/0/3 device emulation initialization error: Device busy Should I open a new bug or what other details do we need to track this down?
(In reply to John Nielsen from comment #11) I'll re-ropen it for you and re-assign it to virtualization so people see that there is a new error on stable/13. I have no idea which of the latest updates might have broken things again but people working on bhyve might want to have a look at the restart problem with PCI passthru.
I've just upgraded my Windows 11 VM and it does not boot anymore. I see an interesting error,never seen before : root@marietto:/home/marietto/bhyve # ./win11-phy-nvd0-vm4.sh vm_open: vm4 could not be opened: No such file or directory vm_open: vm4 could not be opened: No such file or directory da5 nvd0 da2 da4 ada2 [1] 8326 bhyve: Failed to map MSI-X table BAR on 1/0/0: Device busy bhyve: failed to initialize BARs for PCI 1/0/0 device emulation initialization error: Device busy these are the bhyve parameters that I've used : bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-hd,/dev/nvd0,bootindex=1 \ -s 5,passthru,1/0/0 \ -s 6,passthru,5/0/0 \ -s 7,virtio-net,tap4 \ -s 10,hda,play=/dev/dsp,rec=/dev/dsp \ -s 29,fbuf,tcp=0.0.0.0:5904,w=1440,h=900 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm4 < /dev/null & sleep 2 && vncviewer 0:4 before to launch the windows 11 vm,I've ran this script : setxkbmap it bhyvectl --vm=vm0 --force-reset bhyvectl --vm=vm0 --destroy bhyvectl --vm=vm1 --force-reset bhyvectl --vm=vm1 --destroy bhyvectl --vm=vm2 --force-reset bhyvectl --vm=vm2 --destroy bhyvectl --vm=vm3 --force-reset bhyvectl --vm=vm3 --destroy bhyvectl --vm=vm4 --force-reset bhyvectl --vm=vm4 --destroy bhyvectl --vm=vm5 --force-reset bhyvectl --vm=vm5 --destroy bhyvectl --vm=vm6 --force-reset bhyvectl --vm=vm6 --destroy bhyvectl --vm=vm7 --force-reset bhyvectl --vm=vm7 --destroy bhyvectl --vm=vm8 --force-reset bhyvectl --vm=vm8 --destroy bhyvectl --vm=vm9 --force-reset bhyvectl --vm=vm9 --destroy bhyvectl --vm=vm10 --force-reset bhyvectl --vm=vm10 --destroy bhyvectl --vm=vm11 --force-reset bhyvectl --vm=vm11 --destroy bhyvectl --vm=vm12 --force-reset bhyvectl --vm=vm12 --destroy bhyvectl --vm=vm13 --force-reset bhyvectl --vm=vm13 --destroy bhyvectl --vm=vm14 --force-reset bhyvectl --vm=vm14 --destroy bhyvectl --vm=vm15 --force-reset bhyvectl --vm=vm15 --destroy bhyvectl --vm=vm16 --force-reset bhyvectl --vm=vm16 --destroy bhyvectl --vm=vm17 --force-reset bhyvectl --vm=vm17 --destroy bhyvectl --vm=vm18 --force-reset bhyvectl --vm=vm18 --destroy bhyvectl --vm=vm19 --force-reset bhyvectl --vm=vm19 --destroy bhyvectl --vm=vm20 --force-reset bhyvectl --vm=vm20 --destroy I have no disks attached to the 1/0/0...
I'm on : FreeBSD marietto 13.1-RELEASE FreeBSD 13.1-RELEASE n250232-e981f9a8e335 GENERIC amd64
I just updated from 13-STABLE to 14-CURRENT yesterday and PCI passthru including rebooting the VM is working as expected. I'm also not seeing the "vm_open.. could not be opened" error. (I think the latter is fixed by https://cgit.freebsd.org/src/commit/?id=7a0c23da4eaa63f00e53aa18f3ab1f2bb32f593a .) For context, I'm on an AMD machine. Partial dmesg output: CPU: AMD Ryzen 5 3600X 6-Core Processor (3792.99-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x870f10 Family=0x17 Model=0x71 Stepping=0 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x75c237ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX,ADMSKX> Structured Extended Features=0x219c91a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA> Structured Extended Features2=0x400004<UMIP,RDPID> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> AMD Extended Feature Extensions ID EBX=0x108b657<CLZERO,IRPerf,XSaveErPtr,RDPRU,WBNOINVD,IBPB,STIBP,SSBD> SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 TSC: P-state invariant, performance statistics AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0 amdiommu0 at device 0.2 on pci0 ivhd0: <AMD-Vi/IOMMU ivhd with EFR> on acpi0 ivhd0: Flag:b0<IotlbSup,Coherent> ivhd0: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0 ivhd0: Extended features[31:0]:22294a5a<PPRSup,NXSup,GTSup,IASup,PCSup> HATS = 0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd0: Extended features[62:32]:58f77ef<USSup,PprOvrflwEarlySup,PPRAutoRspSup,BlKStopMrkSup,PerfOptSup,MsiCapMmioSup,GIOSup,HASup,EPHSup,AttrFWSup> Max PASID: 0x2f DevTblSegSup = 0x3 MarcSup = 0x1 ivhd0: supported paging level:7, will use only: 4 ivhd0: device [0x8 - 0xfffe] config:0 ivhd0: device [0xff00 - 0xffff] config:0 ivhd0: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR,CapExt>
Good for you. I'm not lucky as you. I can't use bhyve on FreeBSD 14-CURRENT,because it does not have some bug fixes that I need that are only in the 13.1-RELEASE modified by Corvin. I can only wait and hope that he understand what's wrong in his code and that have been already patched on the 14-CURRENT.
John Nielsen confirms it is working on 14 for him. Other reports are for out-of-tree modifications (not yet) pushed to FreeBSD so not a FreeBSD bug.