I'm running into an issue when using PCI passthrough with bhyve. I am running into this on SmartOS but when googling the problem I found a few FreeBSD users also running into the same problem. After chatting with Michael Dexter, I as suggested to also fine a bug bere. List of other people hitting the problem on FreeBSD: - https://forums.freebsd.org/threads/vm-bhyve-windows-2012-r2-and-passthru.60832/ - https://www.ixsystems.com/community/threads/bhyve-pci-passthrough-errors.57284/ - https://forums.freebsd.org/threads/bhyve-passthru-issue-with-lsi-logic-sas2008-- pci-express-fusion-mpt-sas-2.65269/ - bug #211062, comment #8 SmartOS bug report: - https://github.com/joyent/smartos-live/issues/901 Bhyve is dumping core when the following assert is tripped: Assertion failed: error == 0, file pci_emul.c, line 517, function modify_bar_registration This is only tripped when the guest OS is windows (10). I do not have issues with a freebsd, illumos, or linux guest. So something windows is doing triggeres it. I also noticed I am only getting this on PCIe devices that have multiple BAR entries. I captured the output in a FreeBSD guest that got the PCIe devices via passthrough, you can see they have 2 BARs. ( Found of no easy way to capture this info on SmartOS, but this is probably easier for here anyway ) ``` ixl0@pci0:0:8:0: class=0x020000 card=0x37d215d9 chip=0x37d28086 rev=0x09 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection X722 for 10GBASE-T' class = network subclass = ethernet bar [10] = type Prefetchable Memory, range 64, base rxc1000000, size 16777216, enabled bar [1c] = type Prefetchable Memory, range 64, base rxc2000000, size 32768, enabled cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 8 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x1000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) cap 03[e0] = VPD VPD ident = 'Example VPD' ixl1@pci0:0:8:1: class=0x020000 card=0x37d215d9 chip=0x37d28086 rev=0x09 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection X722 for 10GBASE-T' class = network subclass = ethernet bar [10] = type Prefetchable Memory, range 64, base rxc3000000, size 16777216, enabled bar [1c] = type Prefetchable Memory, range 64, base rxc4000000, size 32768, enabled cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 8 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x1000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) cap 03[e0] = VPD VPD ident = 'Example VPD' ``` Michael Zeller who is also on the bhyve weekly called help me debug this a bit (on SmartOS) We traced it to `Also the call to unregister_mem() ends up seeing an ENOENT from mmio_rb_lookup(&mmio_rb_root, memp->base, &entry);` mmio_rb_lookup is macro that we (illumos) copied from FreeBSD and it wasn't changed. So it would make sense that there is indeed a bug in there both FreeBSd and SmartOS bhyve users would hit the same error.
This was fixed in FreeBSD with https://svnweb.freebsd.org/base?view=revision&revision=348779
Interesting, as we do have these changes in the SmartOS bhyve tree too... I wonder if that somehow got mismerged. I have poked Micheal Zeller on the SmartOS side again.
Thanks Peter, I fixed it on SmartOS by applying most of the diff to pci_passthru.c, it's working now!