Bug 178782 - [ixgbe] 82599EB SFP does not work with passthrough under KVM.
Summary: [ixgbe] 82599EB SFP does not work with passthrough under KVM.
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-RELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking
Depends on:
Blocks:
 
Reported: 2013-05-20 07:30 UTC by legumen
Modified: 2016-03-18 16:04 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description legumen 2013-05-20 07:30:00 UTC
The 10G Intel 82599EB SFP NIC could not work with passthrough on KVM.
I did some tests with different Hypervisors and Guest OS, which make me think it's a FreeBSD problem.

1. OpenXen + FreeBSD9.1 guest os.             -- [OK]
2. OpenXen + Linux guest os (e.g. Fedora 12). -- [OK]
3. KVM + Linux guest os (e.g. Fedora 12).     -- [OK]
4. KVM + FreeBSD9.1 guest os.                 -- [Failed]

How-To-Repeat: 1.Make the pci device of 82599SFP assignale. From lspci on OpenSUSE:
0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: QLogic, Corp. Device 00e4
        Flags: fast devsel, IRQ 25
        Memory at f8fe0000 (64-bit, prefetchable) [size=128K]
        I/O ports at cc00 [size=32]
        Memory at f8fdc000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable- Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: pci-stub

2. Passthrough it to the guest FreeBSD os. Here is my vm configuration file
<domain type='kvm'>
  <name>FreeBSD9.1</name>
  <uuid>4d56613e-400c-f37d-2a9a-e0332d81ca63</uuid>
  <memory>2097152</memory>
  <currentMemory>2097152</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <pae/>
    <apic/>
    <acpi/>
  </features>
    <clock offset='utc'>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
      <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='default'/>
      <source file='/home/vm/FreeBSD9.1/FreeBSD9.1.raw'/>
      <target dev='hdc'/>
    </disk>
      <interface type='bridge'>
        <source bridge='br0'/>
        <mac address='52:54:00:0d:92:6f'/>
        <model type='rtl8139'/>
      </interface>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
  
  </devices>
</domain>

3.The device could be probed and attached by FreeBSD. From dmesg on FreeBSD:
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.8> port 0xc020-0xc03f mem 0xfe000000-0xfe01ffff,0xfe020000-0xfe023fff irq 11 at device 4.0 on pci0
ix0: Using MSIX interrupts with 2 vectors
ix0: Ethernet address: 00:0c:bd:05:7d:4a
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8

But no interrupt could be found. From "vmstat -i":
interrupt                          total       rate
irq1: atkbd0                         143          0
irq4: uart0                          177          1
irq15: ata1                         1593         10
cpu0:timer                          7140         46
Total                               9053         58

And ix0 keeps to be "no carrier" (even if I config an IP address on it and ping). From "ifconfig":
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:0c:bd:05:7d:4a
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2013-05-20 07:35:14 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-net

Over to maintainer(s).
Comment 2 Sean Bruno freebsd_committer 2015-08-03 17:37:22 UTC
Since the adapter works just fine from FreeBSD, I'm not sure how to proceed with a test of the pass through capabilities for Linux.

Does this still occur now?
Comment 3 legumen 2015-08-13 06:50:03 UTC
(In reply to Sean Bruno from comment #2)

Sorry Sean. I'm not sure if it still occur now or not.
I changed my job for quite a long time!
And I believe my former colleagues may solve(or have workaround for) this, because I noticed their product was announced last year.  ;>
Comment 4 Sean Bruno freebsd_committer 2015-08-13 16:29:31 UTC
(In reply to legumen from comment #3)
ok, I'm hunting around for some linux folks that can test this and validate.
Comment 5 Jeff Pieper 2015-08-13 17:35:15 UTC
I can test this. I'll be using 10.1 VM on a RHEL 7.1 host.
Comment 6 Jeff Pieper 2015-08-13 21:38:29 UTC
10.1 (ixgbe-2.5.15) still fails (no link and no interrupts), however I also tested a VM using an 11.0-CURRENT-amd64-20150804-r286285 snapshot, which has ix-3.1.0 and it gets link and passes traffic. 

Since the driver in HEAD is working, I think this can be closed.
Comment 7 Sean Bruno freebsd_committer 2015-08-13 21:42:16 UTC
(In reply to Jeff Pieper from comment #6)
Thanks for testing this, it really saved me quite a bit of time in setup.

Do you have time to test 10.2R in the next week or so?  I think its about to be pushed out by release engineering.
Comment 8 Sean Bruno freebsd_committer 2015-08-13 21:42:57 UTC
(In reply to Sean Bruno from comment #7)
ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.2/
Comment 9 Jeff Pieper 2015-08-13 22:39:21 UTC
(In reply to Sean Bruno from comment #8)

10.2-RELEASE works too (ix-2.8.3).
Comment 10 jarodwl 2016-03-18 10:46:43 UTC
(In reply to Jeff Pieper from comment #9)
I still can reproduce the issue using 10.2-RELEASE. My basic environment is below:
Hypervisor: KVM
Host: Centos 7.0

Could you help share your basic environment like hypervisor and host version?

PS, libvirt XML is below:
<domain type='kvm'>
  <name>test</name>
  <uuid>7f68743b-d225-4c7d-dc7e-9aa6712584b3</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cpu mode="host-passthrough"/>
  <os>
    <type arch='x86_64'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/opt/FreeBSD-10.2-RELEASE-amd64.qcow2'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <interface type='bridge'>
      <mac address='52:54:00:72:26:88'/>
      <source bridge='br0'/>
      <model type='virtio'/>
    </interface>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x06" slot="0x00" function="0x01"/>
      </source>
    </hostdev>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <serial type="pty">
      <target port="0"/>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
  </devices>
</domain>