Attempting to use PCI passthrough with bhyve, but it appears to be passing interrupts or the PCI bus configuration incorrectly. My OpenBSD 5.8 guest is reporting this error on boot: fxp0 at pci0 dev 2 function 0 "Intel 8255x" rev 0x08pci_intr_map: bad interrupt line 20 : couldn't map interrupt And the network card is unusable. When I boot OpenBSD 5.8 on the same hardware bare metal (without bhyve), it correctly detects and configures the PCI device: fxp0 at pci4 dev 0 function 0 "Intel 8255x" rev 0x08, i82559: apic 2 int 20, address 00:20:e0:90:e5 inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4 Configuration: following the instructions at https://wiki.freebsd.org/bhyve/pci_passthru and https://forums.freebsd.org/threads/howto-bhyve-using-openbsd-as-main-firewall-in-freebsd.50470/ /boot/loader.conf pptdevs="4/0/0" # passthrough Intel EtherExpress 'fxp0' as ppt0 sudo bhyve -A -H -P -s 0:0:0,hostbridge \ -s 0:1:0,lpc \ -s 0:2:0,passthru,4/0/0 \ -s 0:3:0,virtio-blk,./vm.img \ -l com1,stdio \ -c 1 -m 256M \ -S \ -W \ -x \ -Y \ openbsd I've tried various flags to bhyve (with and without -W to force MSI instead of MSI-X, with and without -x to configure the guest's local APIC in x2APIC mode, hostbridge and amdbridge, other virtual PCI slots besides 0:2:0 including placing the NIC on a different PCI bus and the same as the physical slot, etc.), no difference. FreeBSD host: CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3300.09-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306a9 Family=0x6 Model=0x3a Stepping=9 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS> XSAVE Features=0x1<XSAVEOPT> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics With pptdevs="4/0/0": ppt0 port 0xc000-0xc03f mem 0xdfa04000-0xdfa04fff,0xdf900000-0xdf9fffff irq 20 at device 0.0 on pci4 Without pptdevs="4/0/0" (for comparison - the card is functional): fxp0: <Intel 82559 Pro/100 Ethernet> port 0xc100-0xc13f mem 0xdfa25000-0xdfa25fff,0xdf900000-0xdf9fffff irq 20 at device 0.0 on pci4 miibus0: <MII bus> on fxp0 fxp0: Ethernet address: 00:20:e0:90:e5:ea Tested on FreeBSD 10.2-STABLE and 11-CURRENT r292163 (same error in both cases) What I find unusual is that the same interrupt line (20) is used in both cases, but under bhyve OpenBSD fails to map the interrupt, and on bare metal OpenBSD can map it without error. Is this a case of bhyve not supporting certain APIC functionality? (dmesg on the virtualized guest is missing ", i82559: apic 2")
Is bhyve using some kind of "legacy" interrupts? The OpenBSD 5.8 source code printing the error is: arch/amd64/pci/pci_machdep.c if (line >= NUM_LEGACY_IRQS) { printf("pci_intr_map: bad interrupt line %d\n", line); (where NUM_LEGACY_IRQS is 16)
Correct assignment.
PCI passthru with bhyve is restricted to devices that support MSI or MSI-X, which is generally PCI-e devices. Unfortunately the fxp adapter is parallel PCI and only supports legacy interrupts. I'll modify the bhyve wiki page to clarify this restriction.
I've got the same Issue under OpenBSD 5.9 and -current. The Interfaces: Aug 12 03:27:07 gefjon kernel: [1] em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xf080-0xf09f mem 0xf7f00000-0xf7f1ffff,0xf7f38000-0xf7f38fff irq 20 at device 25.0 on pci0 Aug 12 03:27:07 gefjon kernel: [1] em0: Using an MSI interrupt Aug 12 03:27:07 gefjon kernel: [1] em0: Ethernet address: 40:8d:5c:b7:ca:41 Aug 12 03:27:07 gefjon kernel: [1] em0: netmap queues/slots: TX 1/1024, RX 1/1024 Aug 12 03:27:07 gefjon kernel: [1] ahciem0: <AHCI enclosure management bridge> on ahci0 Aug 12 03:27:07 gefjon kernel: [1] re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xf7d00000-0xf7d00fff,0xf0000000-0xf0003fff irq 19 at device 0.0 on pci3 Aug 12 03:27:07 gefjon kernel: [1] re0: Using 1 MSI-X message Aug 12 03:27:07 gefjon kernel: [1] re0: ASPM disabled Aug 12 03:27:07 gefjon kernel: [1] re0: Chip rev. 0x4c000000 Aug 12 03:27:07 gefjon kernel: [1] re0: MAC rev. 0x00000000 Aug 12 03:27:07 gefjon kernel: [1] miibus0: <MII bus> on re0 For both interfaces under PPTDEVS and vmm loaded with bhyve: Aug 12 02:31:20 aphrodite /bsd: em0 at pci0 dev 5 function 0 "Intel I217-V" rev 0x00pci_intr_map: bad interrupt line 20 Aug 12 03:44:15 aphrodite /bsd: re0 at pci0 dev 5 function 0 "Realtek 8168" rev 0x0cpci_intr_map: bad interrupt line 19 The CPU: CPU: Intel(R) Core(TM) i5-4570S CPU @ 2.90GHz (2893.36-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306c3 Family=0x6 Model=0x3c Stepping=3 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG> XSAVE Features=0x1<XSAVEOPT> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics I also tried it on an actual iwm card: iwm0 at pci0 dev 6 function 0 "Intel Dual Band Wireless AC 7260" rev 0xbbpci_intr_map: bad interrupt line 16 iwm0: can't map interrupt The OS is 11.0-beta4 (but the same on 10.3).
Apologies - I should have been a bit clearer. >PCI passthru with bhyve is restricted to devices that support >MSI or MSI-X, which is generally PCI-e devices. This also requires guest support of MSI/MSI-x, where there are no legacy interrupts available. The OpenBSD em/iwm drivers may either not use MSI, or require legacy interrupts to be valid before using MSI.
Maybe that's an issue. I addressed it with sendbug command under OpenBSD Guest. It's working under all other OSes I tried. So no MSI support could be the answer. PS: Should not be part of a bugreport. But bless bhyve. Together with bhyve-vm it's so much better and easier than a lot of other stuff I tried :)
This appears to have a workaround or is a known limitation. Closing, and please continue to report issues like this.