Bug 232472 - Hyper-V Guest FreeBSD-current and Intel X540 SR-IOV pass-thru not working
Summary: Hyper-V Guest FreeBSD-current and Intel X540 SR-IOV pass-thru not working
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-virtualization mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-10-20 14:25 UTC by Michael
Modified: 2019-08-15 18:05 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael 2018-10-20 14:25:08 UTC
Hyper-V 2016 and Hyper-V 2019 (windows servers) HOSTS,
Network adapters Intel X540-T2, X540-T1, X520-T1, X540-AT2 and 10G 2P X540-t.

FreeBSD Guest system ( /var/log/messages ):

f12rw syslogd: kernel boot file is /boot/kernel/kernel
f12rw kernel: ---<<BOOT>>---
f12rw kernel: Copyright (c) 1992-2018 The FreeBSD Project.
f12rw kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
f12rw kernel:   The Regents of the University of California. All rights reserved.
f12rw kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
f12rw kernel: FreeBSD 13.0-CURRENT #0 e1b49380456(master)-dirty: Sat Oct 20 15:26:15 EEST 2018
f12rw kernel:     root@f12rw:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-DBGless amd64
f12rw kernel: FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
f12rw kernel: SRAT: Ignoring memory at addr 0x3f0000000
f12rw kernel: SRAT: Ignoring memory at addr 0x1000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x10000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x20000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x40000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x80000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x9e8000000
f12rw kernel: SRAT: Ignoring memory at addr 0x8800000000
f12rw kernel: SRAT: Ignoring memory at addr 0x18000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x30000000000
f12rw kernel: SRAT: Ignoring memory at addr 0x60000000000
f12rw kernel: SRAT: Ignoring memory at addr 0xc0000000000
f12rw kernel: VT(efifb): resolution 1024x768
f12rw kernel: Hyper-V Version: 10.0.17763 [SP0]
f12rw kernel:   Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
f12rw kernel:   PM Features=0x0 [C2]
f12rw kernel:   Features3=0xbed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
f12rw kernel: Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
f12rw kernel: module_register: cannot register pci/ixv from kernel; already loaded from if_ixv.ko
f12rw kernel: Module pci/ixv failed to register: 17
f12rw kernel: CPU: Genuine Intel(R) CPU 0000 @ 2.90GHz (2900.00-MHz K8-class CPU)
f12rw kernel:   Origin="GenuineIntel"  Id=0x406f0  Family=0x6  Model=0x4f  Stepping=0
f12rw kernel:   Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
f12rw kernel:   Features2=0xfeda3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
f12rw kernel:   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
f12rw kernel:   AMD Features2=0x121<LAHF,ABM,Prefetch>
f12rw kernel:   Structured Extended Features=0x1c2fb9<FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,RDSEED,ADX,SMAP>
f12rw kernel:   Structured Extended Features3=0x20000000<ARCH_CAP>
f12rw kernel:   XSAVE Features=0x1<XSAVEOPT>
f12rw kernel:   IA32_ARCH_CAPS=0x4
f12rw kernel: Hypervisor: Origin = "Microsoft Hv"
f12rw kernel: real memory  = 16777216000 (16000 MB)
f12rw kernel: avail memory = 16249954304 (15497 MB)
f12rw kernel: Event timer "LAPIC" quality 100
f12rw kernel: ACPI APIC Table: <VRTUAL MICROSFT>
f12rw kernel: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
f12rw kernel: FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 hardware threads
f12rw kernel: random: unblocking device.
f12rw kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
f12rw kernel: Launching APs: 1 3 2 6 7 4 5
f12rw kernel: Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
f12rw kernel: random: entropy device external interface
f12rw kernel: [ath_hal] loaded
f12rw kernel: kbd0 at kbdmux0
f12rw kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff811cdc50, 0) error 19
f12rw kernel: random: registering fast source Intel Secure Key RNG
f12rw kernel: random: fast provider: "Intel Secure Key RNG"
f12rw kernel: netmap: loaded module
f12rw kernel: nexus0
f12rw kernel: efirtc0: <EFI Realtime Clock> on motherboard
f12rw kernel: efirtc0: registered as a time-of-day clock, resolution 1.000000s
f12rw kernel: cryptosoft0: <software crypto> on motherboard
f12rw kernel: acpi0: <VRTUAL MICROSFT> on motherboard
f12rw kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
f12rw kernel: atrtc0: registered as a time-of-day clock, resolution 1.000000s
f12rw kernel: Event timer "RTC" frequency 32768 Hz quality 0
f12rw kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
f12rw kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
f12rw kernel: cpu0: <ACPI CPU> on acpi0
f12rw kernel: acpi_syscontainer0: <System Container> iomem 0xf8000000-0xffffffff,0xfe0000000-0xfffffffff on acpi0
f12rw kernel: vmbus0: <Hyper-V Vmbus> on acpi_syscontainer0
f12rw kernel: vmbus_res0: <Hyper-V Vmbus Resource> irq 5 on acpi0
f12rw kernel: Timecounters tick every 10.000 msec
f12rw kernel: ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to deny, logging disabled
f12rw kernel: DUMMYNET 0xfffff801fc02b0c0 with IPv6 initialized (100409)
f12rw kernel: load_dn_sched dn_sched RR loaded
f12rw kernel: load_dn_sched dn_sched WF2Q+ loaded
f12rw kernel: load_dn_sched dn_sched FIFO loaded
f12rw kernel: load_dn_sched dn_sched FQ_CODEL loaded
f12rw kernel: load_dn_sched dn_sched FQ_PIE loaded
f12rw kernel: load_dn_sched dn_sched PRIO loaded
f12rw kernel: load_dn_sched dn_sched QFQ loaded
f12rw kernel: load_dn_aqm dn_aqm CODEL loaded
f12rw kernel: load_dn_aqm dn_aqm PIE loaded
f12rw kernel: usb_needs_explore_all: no devclass
f12rw kernel: vmbus0: version 3.0
f12rw kernel: hvet0: <Hyper-V event timer> on vmbus0
f12rw kernel: Event timer "Hyper-V" frequency 10000000 Hz quality 1000
f12rw kernel: hvkbd0: <Hyper-V KBD> on vmbus0
f12rw kernel: hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
f12rw kernel: hvkvp0: <Hyper-V KVP> on vmbus0
f12rw kernel: hvshutdown0: <Hyper-V Shutdown> on vmbus0
f12rw kernel: hvtimesync0: <Hyper-V Timesync> on vmbus0
f12rw kernel: hvtimesync0: RTT
f12rw kernel: hvvss0: <Hyper-V VSS> on vmbus0
f12rw kernel: hn0: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn0: Ethernet address: 00:15:5d:00:8b:13
f12rw kernel: hn0: link state changed to UP
f12rw kernel: storvsc0: <Hyper-V SCSI> on vmbus0
f12rw kernel: hn1: <Hyper-V Network Interface> on vmbus0
f12rw kernel: da0 at storvsc0 bus 0 scbus0 target 0 lun 0
f12rw kernel: da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
f12rw kernel: da0: 300.000MB/s transfers
f12rw kernel: da0: Command Queueing enabled
f12rw kernel: da0: 61440MB (125829120 512 byte sectors)
f12rw kernel: da1 at storvsc0 bus 0 scbus0 target 0 lun 1
f12rw kernel: da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
f12rw kernel: da1: 300.000MB/s transfers
f12rw kernel: da1: Command Queueing enabled
f12rw kernel: da1: 409600MB (838860800 512 byte sectors)
f12rw kernel: hn1: Ethernet address: 00:15:5d:00:8b:14
f12rw kernel: hn1: link state changed to UP
f12rw kernel: hn2: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn2: Ethernet address: 00:15:5d:00:8b:1b
f12rw kernel: hn2: link state changed to UP
f12rw kernel: hn3: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn3: Ethernet address: 00:15:5d:00:8b:1c
f12rw kernel: hn3: link state changed to UP
f12rw kernel: hn4: <Hyper-V Network Interface> on vmbus0
f12rw kernel: hn4: Ethernet address: 00:15:5d:00:8b:29
f12rw kernel: hn4: link state changed to UP
f12rw kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
f12rw kernel: lo0: link state changed to UP
f12rw kernel: .
f12rw login[1288]: ROOT LOGIN (root) ON ttyv0

When we try to enable the SR-IOV function of Network-Adapter in the properties of the virtual machine, we get 

f12rw kernel: hn0: got notify, nvs type 128
f12rw kernel: pcib0: <Hyper-V PCI Express Pass Through> on vmbus0
f12rw kernel: pcib0: failed to get resource for cfg window
f12rw kernel: device_attach: pcib0 attach returned 6

In Windows Guest systems 10G SR-IOV function works perfect on this Hyper-V hosts.
Comment 1 Dmitry 2018-12-25 14:16:43 UTC
Look like same problem:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222996#c12
Comment 2 Dmitry 2018-12-25 14:17:56 UTC
Try to use Generation 1 VM, i thnik on 99% it will works.
Comment 3 Michael 2019-08-14 11:54:56 UTC
(In reply to Dmitry from comment #2)

Not. Does not work on Gen 1 version of VM.

Further more.
The host has the latest Intel drivers for the x710
network adapter. Second Generation Virtual Machine with GERERIC
Recompiled FreeBSD-Current Core for today. Host is Windows Server 2019
with Intel drivers (i40ea68.sys 1.10.128.0).

hn0 without SR-IOV function
hn1 with SR-IOV

Log messages from virtual machine:

Aug 14 10:56:18 r03 kernel: ---<<BOOT>>---
Aug 14 10:56:18 r03 kernel: Copyright (c) 1992-2019 The FreeBSD Project.
Aug 14 10:56:18 r03 kernel: Copyright (c) 1979, 1980, 1983, 1986,
1988, 1989, 1991, 1992, 1993, 1994
Aug 14 10:56:18 r03 kernel:     The Regents of the University of
California. All rights reserved.
Aug 14 10:56:18 r03 kernel: FreeBSD is a registered trademark of The
FreeBSD Foundation.
Aug 14 10:56:18 r03 kernel: FreeBSD 13.0-CURRENT
a2166b0cec5-c261904(master) R03 amd64
Aug 14 10:56:18 r03 kernel: FreeBSD clang version 8.0.1
(tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
Aug 14 10:56:18 r03 kernel: SRAT: Ignoring memory at addr 0x1fc000000
...
Aug 14 10:56:18 r03 kernel: SRAT: Ignoring memory at addr 0xc0000000000
Aug 14 10:56:18 r03 kernel: VT(efifb): resolution 1024x768
Aug 14 10:56:18 r03 kernel: Hyper-V Version: 10.0.17763 [SP0]
Aug 14 10:56:18 r03 kernel:
Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
Aug 14 10:56:18 r03 kernel:   PM Features=0x0 [C2]
Aug 14 10:56:18 r03 kernel:
Features3=0xbed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Aug 14 10:56:18 r03 kernel: Timecounter "Hyper-V" frequency 10000000
Hz quality 2000
Aug 14 10:56:18 r03 kernel: module iavf already present!
Aug 14 10:56:18 r03 kernel: CPU: Intel(R) Xeon(R) CPU E5-2623 v3 @
3.00GHz (2996.53-MHz K8-class CPU)
Aug 14 10:56:18 r03 kernel:   Origin="GenuineIntel"  Id=0x306f2
Family=0x6  Model=0x3f  Stepping=2
...
Aug 14 10:56:18 r03 kernel: vmbus0: <Hyper-V Vmbus> on acpi_syscontainer0
Aug 14 10:56:18 r03 kernel: vmbus_res0: <Hyper-V Vmbus Resource> irq 5 on acpi0
...
Aug 14 10:56:18 r03 kernel: vmbus0: version 4.0
Aug 14 10:56:18 r03 kernel: hvet0: <Hyper-V event timer> on vmbus0
Aug 14 10:56:18 r03 kernel: Event timer "Hyper-V" frequency 10000000
Hz quality 1000
Aug 14 10:56:18 r03 kernel: hvkbd0: <Hyper-V KBD> on vmbus0
Aug 14 10:56:18 r03 kernel: hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
Aug 14 10:56:18 r03 kernel: hvkvp0: <Hyper-V KVP> on vmbus0
Aug 14 10:56:18 r03 kernel: hvshutdown0: <Hyper-V Shutdown> on vmbus0
Aug 14 10:56:18 r03 kernel: hvtimesync0: <Hyper-V Timesync> on vmbus0
Aug 14 10:56:18 r03 kernel: hvtimesync0: RTT
Aug 14 10:56:18 r03 kernel: hvvss0: <Hyper-V VSS> on vmbus0
Aug 14 10:56:18 r03 kernel: storvsc0: <Hyper-V SCSI> on vmbus0
Aug 14 10:56:18 r03 kernel: hn0: <Hyper-V Network Interface> on vmbus0
Aug 14 10:56:18 r03 kernel: hn0: Ethernet address: 00:15:5d:00:88:29
Aug 14 10:56:18 r03 kernel: hn0: link state changed to UP
Aug 14 10:56:18 r03 kernel: hn1: <Hyper-V Network Interface> on vmbus0
Aug 14 10:56:18 r03 kernel: hn1: got notify, nvs type 128
Aug 14 10:56:18 r03 kernel: hn1: Ethernet address: 00:15:5d:00:88:2a
Aug 14 10:56:18 r03 kernel: hn1: link state changed to UP
Aug 14 10:56:18 r03 kernel: pcib0: <Hyper-V PCI Express Pass Through> on vmbus0
Aug 14 10:56:18 r03 kernel: pci0: <PCI bus> on pcib0
Aug 14 10:56:18 r03 kernel: pci0: <network, ethernet> at device 2.0
(no driver attached)
Aug 14 10:56:18 r03 kernel: lo0: link state changed to UP
...

root@r03:/# pciconf -lvb
none0@pci2:0:2:0:       class=0x020000 card=0x00018086 chip=0x15718086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Virtual Function 700 Series'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base r, size 65536, disabled
    bar   [1c] = type Prefetchable Memory, range 64, base r, size 16384, disabled