Bug 232472

Summary: ixgbe(4): SR-IOV passthru not working on Hyper-V 13-CURRENT guest with Intel X540 (0x152E)
Product: Base System Reporter: Michael <michael.adm>
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: Open ---    
Severity: Affects Many People CC: dmitry_kuleshov, erj, freebsd, kbowling, krzysztof.galazka, lwhsu, net, weh, whu
Priority: --- Keywords: feature, needs-qa
Version: 13.0-RELEASEFlags: koobs: maintainer-feedback? (whu)
koobs: maintainer-feedback? (freebsd)
koobs: maintainer-feedback? (kbowling)
koobs: maintainer-feedback? (erj)
koobs: maintainer-feedback? (krzysztof.galazka)
koobs: mfc-stable13?
koobs: mfc-stable12?
Hardware: amd64   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239849
Attachments:
Description Flags
Make FreeBSD 11.3 to support SRIOV on Intel NIC 82599 VF (0x152E) on HyperV koobs: maintainer-approval? (freebsd)

Description Michael 2018-10-20 14:25:08 UTC
Hyper-V 2016 and Hyper-V 2019 (windows servers) HOSTS,
Network adapters Intel X540-T2, X540-T1, X520-T1, X540-AT2 and 10G 2P X540-t.

FreeBSD Guest system ( /var/log/messages ):

f12rw syslogd: kernel boot file is /boot/kernel/kernel
f12rw kernel: ---<<BOOT>>---
f12rw kernel: Copyright (c) 1992-2018 The FreeBSD Project.
f12rw kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
f12rw kernel:   The Regents of the University of California. All rights reserved.
f12rw kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
f12rw kernel: FreeBSD 13.0-CURRENT #0 e1b49380456(master)-dirty: Sat Oct 20 15:26:15 EEST 2018
f12rw kernel:     root@f12rw:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-DBGless amd64
f12rw kernel: FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
f12rw kernel: SRAT: Ignoring memory at addr 0x3f0000000
f12rw kernel: SRAT: Ignoring memory at addr 0x1000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x10000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x20000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x40000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x80000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x9e8000000
f12rw kernel: SRAT: Ignoring memory at addr 0x8800000000
f12rw kernel: SRAT: Ignoring memory at addr 0x18000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x30000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x60000000000
f12rw kernel: SRAT: Ignoring memory at addr 0xc0000000000
f12rw kernel: VT(efifb): resolution 1024x768
f12rw kernel: Hyper-V Version: 10.0.17763 [SP0]
f12rw kernel:   Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
f12rw kernel:   PM Features=0x0 [C2]
f12rw kernel:   Features3=0xbed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
f12rw kernel: Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
f12rw kernel: module_register: cannot register pci/ixv from kernel; already loaded from if_ixv.ko
f12rw kernel: Module pci/ixv failed to register: 17
f12rw kernel: CPU: Genuine Intel(R) CPU 0000 @ 2.90GHz (2900.00-MHz K8-class CPU)
f12rw kernel:   Origin="GenuineIntel"  Id=0x406f0  Family=0x6  Model=0x4f  Stepping=0
f12rw kernel:   Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
f12rw kernel:   Features2=0xfeda3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
f12rw kernel:   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
f12rw kernel:   AMD Features2=0x121<LAHF,ABM,Prefetch>
f12rw kernel:   Structured Extended Features=0x1c2fb9<FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,RDSEED,ADX,SMAP>
f12rw kernel:   Structured Extended Features3=0x20000000<ARCH_CAP>
f12rw kernel:   XSAVE Features=0x1<XSAVEOPT>
f12rw kernel:   IA32_ARCH_CAPS=0x4
f12rw kernel: Hypervisor: Origin = "Microsoft Hv"
f12rw kernel: real memory  = 16777216000 (16000 MB)
f12rw kernel: avail memory = 16249954304 (15497 MB)
f12rw kernel: Event timer "LAPIC" quality 100
f12rw kernel: ACPI APIC Table: <VRTUAL MICROSFT>
f12rw kernel: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
f12rw kernel: FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 hardware threads
f12rw kernel: random: unblocking device.
f12rw kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
f12rw kernel: Launching APs: 1 3 2 6 7 4 5
f12rw kernel: Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
f12rw kernel: random: entropy device external interface
f12rw kernel: [ath_hal] loaded
f12rw kernel: kbd0 at kbdmux0
f12rw kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff811cdc50, 0) error 19
f12rw kernel: random: registering fast source Intel Secure Key RNG
f12rw kernel: random: fast provider: "Intel Secure Key RNG"
f12rw kernel: netmap: loaded module
f12rw kernel: nexus0
f12rw kernel: efirtc0: <EFI Realtime Clock> on motherboard
f12rw kernel: efirtc0: registered as a time-of-day clock, resolution 1.000000s
f12rw kernel: cryptosoft0: <software crypto> on motherboard
f12rw kernel: acpi0: <VRTUAL MICROSFT> on motherboard
f12rw kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
f12rw kernel: atrtc0: registered as a time-of-day clock, resolution 1.000000s
f12rw kernel: Event timer "RTC" frequency 32768 Hz quality 0
f12rw kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
f12rw kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
f12rw kernel: cpu0: <ACPI CPU> on acpi0
f12rw kernel: acpi_syscontainer0: <System Container> iomem 0xf8000000-0xffffffff,0xfe0000000-0xfffffffff on acpi0
f12rw kernel: vmbus0: <Hyper-V Vmbus> on acpi_syscontainer0
f12rw kernel: vmbus_res0: <Hyper-V Vmbus Resource> irq 5 on acpi0
f12rw kernel: Timecounters tick every 10.000 msec
f12rw kernel: ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to deny, logging disabled
f12rw kernel: DUMMYNET 0xfffff801fc02b0c0 with IPv6 initialized (100409)
f12rw kernel: load_dn_sched dn_sched RR loaded
f12rw kernel: load_dn_sched dn_sched WF2Q+ loaded
f12rw kernel: load_dn_sched dn_sched FIFO loaded
f12rw kernel: load_dn_sched dn_sched FQ_CODEL loaded
f12rw kernel: load_dn_sched dn_sched FQ_PIE loaded
f12rw kernel: load_dn_sched dn_sched PRIO loaded
f12rw kernel: load_dn_sched dn_sched QFQ loaded
f12rw kernel: load_dn_aqm dn_aqm CODEL loaded
f12rw kernel: load_dn_aqm dn_aqm PIE loaded
f12rw kernel: usb_needs_explore_all: no devclass
f12rw kernel: vmbus0: version 3.0
f12rw kernel: hvet0: <Hyper-V event timer> on vmbus0
f12rw kernel: Event timer "Hyper-V" frequency 10000000 Hz quality 1000
f12rw kernel: hvkbd0: <Hyper-V KBD> on vmbus0
f12rw kernel: hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
f12rw kernel: hvkvp0: <Hyper-V KVP> on vmbus0
f12rw kernel: hvshutdown0: <Hyper-V Shutdown> on vmbus0
f12rw kernel: hvtimesync0: <Hyper-V Timesync> on vmbus0
f12rw kernel: hvtimesync0: RTT
f12rw kernel: hvvss0: <Hyper-V VSS> on vmbus0
f12rw kernel: hn0: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn0: Ethernet address: 00:15:5d:00:8b:13
f12rw kernel: hn0: link state changed to UP
f12rw kernel: storvsc0: <Hyper-V SCSI> on vmbus0
f12rw kernel: hn1: <Hyper-V Network Interface> on vmbus0
f12rw kernel: da0 at storvsc0 bus 0 scbus0 target 0 lun 0
f12rw kernel: da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
f12rw kernel: da0: 300.000MB/s transfers
f12rw kernel: da0: Command Queueing enabled
f12rw kernel: da0: 61440MB (125829120 512 byte sectors)
f12rw kernel: da1 at storvsc0 bus 0 scbus0 target 0 lun 1
f12rw kernel: da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
f12rw kernel: da1: 300.000MB/s transfers
f12rw kernel: da1: Command Queueing enabled
f12rw kernel: da1: 409600MB (838860800 512 byte sectors)
f12rw kernel: hn1: Ethernet address: 00:15:5d:00:8b:14
f12rw kernel: hn1: link state changed to UP
f12rw kernel: hn2: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn2: Ethernet address: 00:15:5d:00:8b:1b
f12rw kernel: hn2: link state changed to UP
f12rw kernel: hn3: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn3: Ethernet address: 00:15:5d:00:8b:1c
f12rw kernel: hn3: link state changed to UP
f12rw kernel: hn4: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn4: Ethernet address: 00:15:5d:00:8b:29
f12rw kernel: hn4: link state changed to UP
f12rw kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
f12rw kernel: lo0: link state changed to UP
f12rw kernel: .
f12rw login[1288]: ROOT LOGIN (root) ON ttyv0

When we try to enable the SR-IOV function of Network-Adapter in the properties of the virtual machine, we get 

f12rw kernel: hn0: got notify, nvs type 128
f12rw kernel: pcib0: <Hyper-V PCI Express Pass Through> on vmbus0
f12rw kernel: pcib0: failed to get resource for cfg window
f12rw kernel: device_attach: pcib0 attach returned 6

In Windows Guest systems 10G SR-IOV function works perfect on this Hyper-V hosts.
Comment 1 Dmitry 2018-12-25 14:16:43 UTC
Look like same problem:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222996#c12
Comment 2 Dmitry 2018-12-25 14:17:56 UTC
Try to use Generation 1 VM, i thnik on 99% it will works.
Comment 3 Michael 2019-08-14 11:54:56 UTC
(In reply to Dmitry from comment #2)

Not. Does not work on Gen 1 version of VM.

Further more.
The host has the latest Intel drivers for the x710
network adapter. Second Generation Virtual Machine with GERERIC
Recompiled FreeBSD-Current Core for today. Host is Windows Server 2019
with Intel drivers (i40ea68.sys 1.10.128.0).

hn0 without SR-IOV function
hn1 with SR-IOV

Log messages from virtual machine:

Aug 14 10:56:18 r03 kernel: ---<<BOOT>>---
Aug 14 10:56:18 r03 kernel: Copyright (c) 1992-2019 The FreeBSD Project.
Aug 14 10:56:18 r03 kernel: Copyright (c) 1979, 1980, 1983, 1986,
1988, 1989, 1991, 1992, 1993, 1994
Aug 14 10:56:18 r03 kernel:     The Regents of the University of
California. All rights reserved.
Aug 14 10:56:18 r03 kernel: FreeBSD is a registered trademark of The
FreeBSD Foundation.
Aug 14 10:56:18 r03 kernel: FreeBSD 13.0-CURRENT
a2166b0cec5-c261904(master) R03 amd64
Aug 14 10:56:18 r03 kernel: FreeBSD clang version 8.0.1
(tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
Aug 14 10:56:18 r03 kernel: SRAT: Ignoring memory at addr 0x1fc000000
...
Aug 14 10:56:18 r03 kernel: SRAT: Ignoring memory at addr 0xc0000000000
Aug 14 10:56:18 r03 kernel: VT(efifb): resolution 1024x768
Aug 14 10:56:18 r03 kernel: Hyper-V Version: 10.0.17763 [SP0]
Aug 14 10:56:18 r03 kernel:
Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
Aug 14 10:56:18 r03 kernel:   PM Features=0x0 [C2]
Aug 14 10:56:18 r03 kernel:
Features3=0xbed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Aug 14 10:56:18 r03 kernel: Timecounter "Hyper-V" frequency 10000000
Hz quality 2000
Aug 14 10:56:18 r03 kernel: module iavf already present!
Aug 14 10:56:18 r03 kernel: CPU: Intel(R) Xeon(R) CPU E5-2623 v3 @
3.00GHz (2996.53-MHz K8-class CPU)
Aug 14 10:56:18 r03 kernel:   Origin="GenuineIntel"  Id=0x306f2
Family=0x6  Model=0x3f  Stepping=2
...
Aug 14 10:56:18 r03 kernel: vmbus0: <Hyper-V Vmbus> on acpi_syscontainer0
Aug 14 10:56:18 r03 kernel: vmbus_res0: <Hyper-V Vmbus Resource> irq 5 on acpi0
...
Aug 14 10:56:18 r03 kernel: vmbus0: version 4.0
Aug 14 10:56:18 r03 kernel: hvet0: <Hyper-V event timer> on vmbus0
Aug 14 10:56:18 r03 kernel: Event timer "Hyper-V" frequency 10000000
Hz quality 1000
Aug 14 10:56:18 r03 kernel: hvkbd0: <Hyper-V KBD> on vmbus0
Aug 14 10:56:18 r03 kernel: hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
Aug 14 10:56:18 r03 kernel: hvkvp0: <Hyper-V KVP> on vmbus0
Aug 14 10:56:18 r03 kernel: hvshutdown0: <Hyper-V Shutdown> on vmbus0
Aug 14 10:56:18 r03 kernel: hvtimesync0: <Hyper-V Timesync> on vmbus0
Aug 14 10:56:18 r03 kernel: hvtimesync0: RTT
Aug 14 10:56:18 r03 kernel: hvvss0: <Hyper-V VSS> on vmbus0
Aug 14 10:56:18 r03 kernel: storvsc0: <Hyper-V SCSI> on vmbus0
Aug 14 10:56:18 r03 kernel: hn0: <Hyper-V Network Interface> on vmbus0
Aug 14 10:56:18 r03 kernel: hn0: Ethernet address: 00:15:5d:00:88:29
Aug 14 10:56:18 r03 kernel: hn0: link state changed to UP
Aug 14 10:56:18 r03 kernel: hn1: <Hyper-V Network Interface> on vmbus0
Aug 14 10:56:18 r03 kernel: hn1: got notify, nvs type 128
Aug 14 10:56:18 r03 kernel: hn1: Ethernet address: 00:15:5d:00:88:2a
Aug 14 10:56:18 r03 kernel: hn1: link state changed to UP
Aug 14 10:56:18 r03 kernel: pcib0: <Hyper-V PCI Express Pass Through> on vmbus0
Aug 14 10:56:18 r03 kernel: pci0: <PCI bus> on pcib0
Aug 14 10:56:18 r03 kernel: pci0: <network, ethernet> at device 2.0
(no driver attached)
Aug 14 10:56:18 r03 kernel: lo0: link state changed to UP
...

root@r03:/# pciconf -lvb
none0@pci2:0:2:0:       class=0x020000 card=0x00018086 chip=0x15718086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Virtual Function 700 Series'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base r, size 65536, disabled
    bar   [1c] = type Prefetchable Memory, range 64, base r, size 16384, disabled
Comment 4 Wei Hu 2020-10-29 10:13:43 UTC
Created attachment 219194 [details]
Make FreeBSD 11.3 to support SRIOV on Intel NIC 82599 VF (0x152E) on HyperV

This is code changes to support SRIOV for Intel NIC 82599 VF (Device ID 0x152E) on HyperV. I have done some test on FreeBSD 11.3. The similar code changes will need to make other Intel VFs work on HyperV.
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2022-05-30 00:02:08 UTC
*** Bug 239849 has been marked as a duplicate of this bug. ***
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2022-05-30 22:38:27 UTC
(In reply to Wei Hu from comment #4)

Is this a backport of changes committed to later (> 11.x) or a direct patch for 11.x? If the latter, are we able to produce a patch for CURRENT for user testing?

@Intel Could you review attachment 219194 [details] and provide advice on getting these changes into base and upstream versions of the driver
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2022-05-30 22:44:22 UTC
^Triage: Request feedback from folks that play in the sys/dev/ixgbe area