Bug 205445 - [bhyve] PCI passthru pci_intr_map: bad interrupt line 20, OpenBSD guest couldn't map interrupt
Summary: [bhyve] PCI passthru pci_intr_map: bad interrupt line 20, OpenBSD guest could...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-12-20 01:09 UTC by m8r-592a0p
Modified: 2016-08-12 03:15 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description m8r-592a0p 2015-12-20 01:09:14 UTC
Attempting to use PCI passthrough with bhyve, but it appears to be passing interrupts or the PCI bus configuration incorrectly. My OpenBSD 5.8 guest is reporting this error on boot:

fxp0 at pci0 dev 2 function 0 "Intel 8255x" rev 0x08pci_intr_map: bad interrupt line 20
: couldn't map interrupt

And the network card is unusable. When I boot OpenBSD 5.8 on the same hardware bare metal (without bhyve), it correctly detects and configures the PCI device:

fxp0 at pci4 dev 0 function 0 "Intel 8255x" rev 0x08, i82559: apic 2 int 20, address 00:20:e0:90:e5
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4



Configuration: following the instructions at https://wiki.freebsd.org/bhyve/pci_passthru and 
https://forums.freebsd.org/threads/howto-bhyve-using-openbsd-as-main-firewall-in-freebsd.50470/

/boot/loader.conf
pptdevs="4/0/0"		# passthrough Intel EtherExpress 'fxp0' as ppt0

sudo bhyve -A -H -P -s 0:0:0,hostbridge \
        -s 0:1:0,lpc \
        -s 0:2:0,passthru,4/0/0 \
        -s 0:3:0,virtio-blk,./vm.img \
        -l com1,stdio \
        -c 1 -m 256M \
        -S \
        -W \
        -x \
        -Y \
        openbsd

I've tried various flags to bhyve (with and without -W to force MSI instead of MSI-X, with and without -x to configure the guest's local APIC in x2APIC mode, hostbridge and amdbridge, other virtual PCI slots besides 0:2:0 including placing the NIC on a different PCI bus and the same as the physical slot, etc.), no difference.

FreeBSD host:
CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3300.09-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306a9  Family=0x6  Model=0x3a  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics

With pptdevs="4/0/0":

ppt0 port 0xc000-0xc03f mem 0xdfa04000-0xdfa04fff,0xdf900000-0xdf9fffff irq 20 at device 0.0 on pci4

Without pptdevs="4/0/0" (for comparison - the card is functional):

fxp0: <Intel 82559 Pro/100 Ethernet> port 0xc100-0xc13f mem 0xdfa25000-0xdfa25fff,0xdf900000-0xdf9fffff irq 20 at device 0.0 on pci4
miibus0: <MII bus> on fxp0
fxp0: Ethernet address: 00:20:e0:90:e5:ea



Tested on FreeBSD 10.2-STABLE and 11-CURRENT r292163 (same error in both cases)



What I find unusual is that the same interrupt line (20) is used in both cases, but under bhyve OpenBSD fails to map the interrupt, and on bare metal OpenBSD can map it without error. Is this a case of bhyve not supporting certain APIC functionality? (dmesg on the virtualized guest is missing ", i82559: apic 2")
Comment 1 m8r-592a0p 2015-12-20 01:43:53 UTC
Is bhyve using some kind of "legacy" interrupts? The OpenBSD 5.8 source code printing the error is:

arch/amd64/pci/pci_machdep.c
    if (line >= NUM_LEGACY_IRQS) {
        printf("pci_intr_map: bad interrupt line %d\n", line);

(where NUM_LEGACY_IRQS is 16)
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2015-12-25 23:57:43 UTC
Correct assignment.
Comment 3 Peter Grehan freebsd_committer 2015-12-26 01:44:18 UTC
PCI passthru with bhyve is restricted to devices that support MSI or MSI-X, which is generally PCI-e devices.

Unfortunately the fxp adapter is parallel PCI and only supports legacy interrupts.

I'll modify the bhyve wiki page to clarify this restriction.
Comment 4 borisboris@gmx.net 2016-08-12 02:11:42 UTC
I've got the same Issue under OpenBSD 5.9 and -current. 

The Interfaces:

Aug 12 03:27:07 gefjon kernel: [1] em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xf080-0xf09f mem 0xf7f00000-0xf7f1ffff,0xf7f38000-0xf7f38fff irq 20 at device 25.0 on pci0
Aug 12 03:27:07 gefjon kernel: [1] em0: Using an MSI interrupt
Aug 12 03:27:07 gefjon kernel: [1] em0: Ethernet address: 40:8d:5c:b7:ca:41
Aug 12 03:27:07 gefjon kernel: [1] em0: netmap queues/slots: TX 1/1024, RX 1/1024
Aug 12 03:27:07 gefjon kernel: [1] ahciem0: <AHCI enclosure management bridge> on ahci0

Aug 12 03:27:07 gefjon kernel: [1] re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xf7d00000-0xf7d00fff,0xf0000000-0xf0003fff irq 19 at device 0.0 on pci3
Aug 12 03:27:07 gefjon kernel: [1] re0: Using 1 MSI-X message
Aug 12 03:27:07 gefjon kernel: [1] re0: ASPM disabled
Aug 12 03:27:07 gefjon kernel: [1] re0: Chip rev. 0x4c000000
Aug 12 03:27:07 gefjon kernel: [1] re0: MAC rev. 0x00000000
Aug 12 03:27:07 gefjon kernel: [1] miibus0: <MII bus> on re0

For both interfaces under PPTDEVS and vmm loaded with bhyve:

Aug 12 02:31:20 aphrodite /bsd: em0 at pci0 dev 5 function 0 "Intel I217-V" rev 0x00pci_intr_map: bad interrupt line 20

Aug 12 03:44:15 aphrodite /bsd: re0 at pci0 dev 5 function 0 "Realtek 8168" rev 0x0cpci_intr_map: bad interrupt line 19

The CPU:

CPU: Intel(R) Core(TM) i5-4570S CPU @ 2.90GHz (2893.36-MHz K8-class CPU)
Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>
XSAVE Features=0x1<XSAVEOPT>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics

I also tried it on an actual iwm card:

iwm0 at pci0 dev 6 function 0 "Intel Dual Band Wireless AC 7260" rev 0xbbpci_intr_map: bad interrupt line 16
iwm0: can't map interrupt

The OS is 11.0-beta4 (but the same on 10.3).
Comment 5 Peter Grehan freebsd_committer 2016-08-12 03:01:25 UTC
Apologies - I should have been a bit clearer.

>PCI passthru with bhyve is restricted to devices that support
>MSI or MSI-X, which is generally PCI-e devices.

 This also requires guest support of MSI/MSI-x, where there are no legacy interrupts available.

 The OpenBSD em/iwm drivers may either not use MSI, or require legacy interrupts to be valid before using MSI.
Comment 6 borisboris@gmx.net 2016-08-12 03:15:07 UTC
Maybe that's an issue. I addressed it with sendbug command under OpenBSD Guest. It's working under all other OSes I tried. So no MSI support could be the answer. PS: Should not be part of a bugreport. But bless bhyve. Together with bhyve-vm it's so much better and easier than a lot of other stuff I tried :)