On multiple servers running 12.0-RELEASE, the ixgbe(4) driver enables only one of eight queues on Intel 10Gb NICs. When booting the same hardware to 11.3-RELEASE, the driver configures all eight queues as expected. dmidecode: --- System Information Manufacturer: HPE Product Name: ProLiant DL380 Gen10 Version: Not Specified Serial Number: XXXXXXXXXX UUID: 37383638-3330-4d32-3239-313130303757 Wake-up Type: Power Switch SKU Number: 868703-B21 Family: ProLiant Handle 0x00BF, DMI type 203, 34 bytes OEM-specific Type Header and Data: CB 22 BF 00 AF 00 FE FF 86 80 FB 10 3C 10 D3 17 02 00 FE FF 00 00 05 0A 01 01 FF FF 01 02 03 04 FE FF Strings: PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0) NIC.Slot.1.1 HPE Ethernet 10Gb 2-port 560SFP+ Adapter - NIC Slot 1 --- dmesg on 12.0-RELEASE: --- ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0 numa-domain 0 on pci4 ix0: using 2048 tx descriptors and 2048 rx descriptors ix0: msix_init qsets capped at 16 ix0: Unable to map MSIX table ix0: Using an MSI interrupt ix0: allocated for 1 queues ix0: allocated for 1 rx queues ix0: Ethernet address: 48:df:37:62:be:38 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 1/2048, RX 1/2048 --- dmesg on 11.3-RELEASE: --- ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0 numa-domain 0 on pci4 ix0: Using MSI-X interrupts with 9 vectors ix0: Ethernet address: 48:df:37:62:be:38 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 8/2048, RX 8/2048 --- pciconf -lvct on 12.0-RELEASE: --- ix0@pci0:17:0:0: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82599ES 10-Gigabit SFI/SFP+ Network Connection' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message cap 11[70] = MSI-X supports 64 messages Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS link x8(x8) speed 5.0(5.0) ASPM disabled(L0s) cap 03[e0] = VPD ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 48df37ffff62be38 ecap 000e[150] = ARI 1 ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI enabled 0 VFs configured out of 64 supported First VF RID Offset 0x0080, VF RID Stride 0x0002 VF Device ID 0x10ed Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error --- pciconf -lvct on 11.3-RELEASE: --- ix0@pci0:17:0:0: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82599ES 10-Gigabit SFI/SFP+ Network Connection' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 64 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS link x8(x8) speed 5.0(5.0) ASPM disabled(L0s) cap 03[e0] = VPD ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 48df37ffff62be38 ecap 000e[150] = ARI 1 ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI enabled 0 VFs configured out of 64 supported First VF RID Offset 0x0080, VF RID Stride 0x0002 VF Device ID 0x10ed Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error --- sysctl hw.ix on 12.0-RELEASE: --- hw.ix.enable_rss: 1 hw.ix.enable_fdir: 0 hw.ix.unsupported_sfp: 0 hw.ix.enable_msix: 1 hw.ix.advertise_speed: 0 hw.ix.flow_control: 3 hw.ix.max_interrupt_rate: 31250 --- sysctl hw.ix on 11.3-RELEASE: --- hw.ix.enable_rss: 1 hw.ix.enable_legacy_tx: 0 hw.ix.enable_fdir: 0 hw.ix.unsupported_sfp: 0 hw.ix.rxd: 2048 hw.ix.txd: 2048 hw.ix.num_queues: 8 hw.ix.enable_msix: 1 hw.ix.advertise_speed: 0 hw.ix.flow_control: 0 hw.ix.tx_process_limit: -1 hw.ix.rx_process_limit: -1 hw.ix.max_interrupt_rate: 31250 hw.ix.enable_aim: 1 --- Also see <https://lists.freebsd.org/pipermail/freebsd-stable/2019-August/091415.html>. I will be happy to provide more information or assist in debugging.
I didn't check what version of the ixgbe driver was in 12.0-RELEASE, but it may be worth testing with the ix-kmod driver from ports/packages, which is 3.3.6 at the moment: https://www.freshports.org/net/intel-ix-kmod/
(In reply to Kubilay Kocak from comment #1) Thanks for the suggestion; I didn't know about the ports version of the driver. That version works! So perhaps all that's needed is to commit the ports driver to base. $ fgrep ix /boot/loader.conf if_ix_updated_load="YES" $ grep ^ix0 /var/run/dmesg.boot ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.6> port 0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0 numa-domain 0 on pci4 ix0: Using MSI-X interrupts with 9 vectors ix0: Ethernet address: 48:df:37:62:be:38 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: link state changed to UP $ pciconf -lvce ix0 ix0@pci0:17:0:0: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82599ES 10-Gigabit SFI/SFP+ Network Connection' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 64 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS link x8(x8) speed 5.0(5.0) ASPM disabled(L0s) cap 03[e0] = VPD ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 48df37ffff62be38 ecap 000e[150] = ARI 1 ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI enabled 0 VFs configured out of 64 supported First VF RID Offset 0x0080, VF RID Stride 0x0002 VF Device ID 0x10ed Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error $ sysctl hw.ix hw.ix.enable_rss: 1 hw.ix.enable_fdir: 0 hw.ix.unsupported_sfp: 0 hw.ix.enable_msix: 1 hw.ix.advertise_speed: 0 hw.ix.flow_control: 3 hw.ix.max_interrupt_rate: 31250 $ vmstat -i | fgrep ix0 irq269: ix0:q0 1807 5 irq270: ix0:q1 3 0 irq271: ix0:q2 6 0 irq272: ix0:q3 3 0 irq273: ix0:q4 27 0 irq274: ix0:q5 517 1 irq275: ix0:q6 15 0 irq276: ix0:q7 28 0 irq277: ix0:link 1 0 Interesting that the 12.0 driver has fewer sysctl knobs to turn. Also interesting that the current base version of the driver does not print its version number in the probe message.
The driver that is in 12.0 uses the iflib framework, which makes it impossible to port 3.3.6 into BASE. The driver version is printed via sysctl: # sysctl dev.ix|grep version dev.ix.0.iflib.driver_version: 4.0.1-k dev.ix.1.iflib.driver_version: 4.0.1-k The vendor-specific (non-iflib) sysctls for ix are: dev.ix.x.advertise_speed: 0 dev.ix.x.fc: 0 You can read more about iflib (available sysctls etc) here: https://www.freebsd.org/cgi/man.cgi?query=iflib&sektion=4&apropos=0&manpath=FreeBSD+12.0-RELEASE+and+Ports One of our developers will work on investigating this, but it may be specific to your hardware, as we do not see this on OEM-generic adapters. There also may be relevant iflib changes in 12-STABLE or CURRENT as well.
(In reply to Greg Rivers from comment #2) It's good news that at least something works, but I wouldn't have expected the iflib version to differ when it comes to allocating interrupts. We'll look into it. To add to what Jeff said, the driver version is not printed out in the probe message because the intent of that printout is to print out the name of the device that the driver is attaching to; though in our drivers for newer products we end up printing out both the device name and driver version because the driver version is expected to be there anyway.
Another data point: I get the same result on a ProLiant DL380 Gen9 with the same NIC. Please let me know if there's anything I can do to facilitate.
It appears that HPE branded 82599ES NIC uses different PCI bar for MSI-X table and in-kernel driver was not expecting that. Patch fixing this issue is uploaded on Phabricator and waiting for review ( https://reviews.freebsd.org/D21547 ).
A commit references this bug: Author: erj Date: Tue Sep 24 17:06:33 UTC 2019 New revision: 352656 URL: https://svnweb.freebsd.org/changeset/base/352656 Log: ix, ixv: Read msix_bar from device configuration Instead of predicting the MSI-X bar index based on the device's MAC type, read it from the device's PCI configuration instead. PR: 239704 Submitted by: Piotr Pietruszewski <piotr.pietruszewski@intel.com> Reviewed by: erj@ MFC after: 3 days Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D21547 Changes: head/sys/dev/ixgbe/if_ix.c head/sys/dev/ixgbe/if_ixv.c head/sys/dev/ixgbe/ixgbe.h
^Triage: Assign to committer resolving. @Eric Could you please arrange for a approval from re@ to merge this to releng/12.1 so it makes it to 12.1-RELEASE please
(In reply to Piotr Pietruszewski from comment #6) Many thanks to you and all of the folks at Intel for your work.
(In reply to Kubilay Kocak from comment #8) Ok; I hope to get this merged to stable/12 and sent to re@ to approval today.
The fix has been merged into stable/12 (r352911) and releng/12.1 (r352912).
Leave issue in blocks (bug 240700) so re@ can see which blocking issues have been resolved, and for relnotes purposes