Bug 239704 - ixgbe(4): Only one queue (of eight) enabled on 12.0-RELEASE (ProLiant DL380 Gen10)
Summary: ixgbe(4): Only one queue (of eight) enabled on 12.0-RELEASE (ProLiant DL380 G...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking, needs-qa, regression
Depends on:
Blocks:
 
Reported: 2019-08-07 21:45 UTC by Greg Rivers
Modified: 2019-08-08 18:27 UTC (History)
4 users (show)

See Also:
koobs: mfc-stable12?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Greg Rivers 2019-08-07 21:45:00 UTC
On multiple servers running 12.0-RELEASE, the ixgbe(4) driver enables only one of eight queues on Intel 10Gb NICs. When booting the same hardware to 11.3-RELEASE, the driver configures all eight queues as expected.

dmidecode:
---
System Information
	Manufacturer: HPE
	Product Name: ProLiant DL380 Gen10
	Version: Not Specified
	Serial Number: XXXXXXXXXX
	UUID: 37383638-3330-4d32-3239-313130303757
	Wake-up Type: Power Switch
	SKU Number: 868703-B21
	Family: ProLiant
Handle 0x00BF, DMI type 203, 34 bytes
OEM-specific Type
	Header and Data:
		CB 22 BF 00 AF 00 FE FF 86 80 FB 10 3C 10 D3 17
		02 00 FE FF 00 00 05 0A 01 01 FF FF 01 02 03 04
		FE FF
	Strings:
		PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)
		NIC.Slot.1.1
		HPE Ethernet 10Gb 2-port 560SFP+ Adapter - NIC
		Slot 1
---

dmesg on 12.0-RELEASE:
---
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0 numa-domain 0 on pci4
ix0: using 2048 tx descriptors and 2048 rx descriptors
ix0: msix_init qsets capped at 16
ix0: Unable to map MSIX table 
ix0: Using an MSI interrupt
ix0: allocated for 1 queues
ix0: allocated for 1 rx queues
ix0: Ethernet address: 48:df:37:62:be:38
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 1/2048, RX 1/2048
---

dmesg on 11.3-RELEASE:
---
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0 numa-domain 0 on pci4
ix0: Using MSI-X interrupts with 9 vectors
ix0: Ethernet address: 48:df:37:62:be:38
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 8/2048, RX 8/2048
---

pciconf -lvct on 12.0-RELEASE:
---
ix0@pci0:17:0:0:	class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message
    cap 11[70] = MSI-X supports 64 messages
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 48df37ffff62be38
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI enabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0080, VF RID Stride 0x0002
                     VF Device ID 0x10ed
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
---

pciconf -lvct on 11.3-RELEASE:
---
ix0@pci0:17:0:0:	class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 48df37ffff62be38
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI enabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0080, VF RID Stride 0x0002
                     VF Device ID 0x10ed
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
---

sysctl hw.ix on 12.0-RELEASE:
---
hw.ix.enable_rss: 1
hw.ix.enable_fdir: 0
hw.ix.unsupported_sfp: 0
hw.ix.enable_msix: 1
hw.ix.advertise_speed: 0
hw.ix.flow_control: 3
hw.ix.max_interrupt_rate: 31250
---

sysctl hw.ix on 11.3-RELEASE:
---
hw.ix.enable_rss: 1
hw.ix.enable_legacy_tx: 0
hw.ix.enable_fdir: 0
hw.ix.unsupported_sfp: 0
hw.ix.rxd: 2048
hw.ix.txd: 2048
hw.ix.num_queues: 8
hw.ix.enable_msix: 1
hw.ix.advertise_speed: 0
hw.ix.flow_control: 0
hw.ix.tx_process_limit: -1
hw.ix.rx_process_limit: -1
hw.ix.max_interrupt_rate: 31250
hw.ix.enable_aim: 1
---


Also see <https://lists.freebsd.org/pipermail/freebsd-stable/2019-August/091415.html>. I will be happy to provide more information or assist in debugging.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-08-08 03:40:56 UTC
I didn't check what version of the ixgbe driver was in 12.0-RELEASE, but it may be worth testing with the ix-kmod driver from ports/packages, which is 3.3.6 at the moment:

https://www.freshports.org/net/intel-ix-kmod/
Comment 2 Greg Rivers 2019-08-08 04:23:48 UTC
(In reply to Kubilay Kocak from comment #1)
Thanks for the suggestion; I didn't know about the ports version of the driver. That version works! So perhaps all that's needed is to commit the ports driver to base.

$ fgrep ix /boot/loader.conf
if_ix_updated_load="YES"

$ grep ^ix0 /var/run/dmesg.boot
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.6> port 0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0 numa-domain 0 on pci4
ix0: Using MSI-X interrupts with 9 vectors
ix0: Ethernet address: 48:df:37:62:be:38
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: link state changed to UP

$ pciconf -lvce ix0            
ix0@pci0:17:0:0:        class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks 
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 48df37ffff62be38
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI enabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0080, VF RID Stride 0x0002
                     VF Device ID 0x10ed
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error

$ sysctl hw.ix                 
hw.ix.enable_rss: 1
hw.ix.enable_fdir: 0
hw.ix.unsupported_sfp: 0
hw.ix.enable_msix: 1
hw.ix.advertise_speed: 0
hw.ix.flow_control: 3
hw.ix.max_interrupt_rate: 31250

$ vmstat -i | fgrep ix0
irq269: ix0:q0                      1807          5
irq270: ix0:q1                         3          0
irq271: ix0:q2                         6          0
irq272: ix0:q3                         3          0
irq273: ix0:q4                        27          0
irq274: ix0:q5                       517          1
irq275: ix0:q6                        15          0
irq276: ix0:q7                        28          0
irq277: ix0:link                       1          0

Interesting that the 12.0 driver has fewer sysctl knobs to turn. Also interesting that the current base version of the driver does not print its version number in the probe message.
Comment 3 Jeff Pieper 2019-08-08 09:53:58 UTC
The driver that is in 12.0 uses the iflib framework, which makes it impossible to port 3.3.6 into BASE. The driver version is printed via sysctl:

# sysctl dev.ix|grep version
dev.ix.0.iflib.driver_version: 4.0.1-k
dev.ix.1.iflib.driver_version: 4.0.1-k

The vendor-specific (non-iflib) sysctls for ix are:

dev.ix.x.advertise_speed: 0
dev.ix.x.fc: 0

You can read more about iflib (available sysctls etc) here:

https://www.freebsd.org/cgi/man.cgi?query=iflib&sektion=4&apropos=0&manpath=FreeBSD+12.0-RELEASE+and+Ports

One of our developers will work on investigating this, but it may be specific to your hardware, as we do not see this on OEM-generic adapters. There also may be relevant iflib changes in 12-STABLE or CURRENT as well.
Comment 4 Eric Joyner freebsd_committer 2019-08-08 18:27:36 UTC
(In reply to Greg Rivers from comment #2)

It's good news that at least something works, but I wouldn't have expected the iflib version to differ when it comes to allocating interrupts. We'll look into it.

To add to what Jeff said, the driver version is not printed out in the probe message because the intent of that printout is to print out the name of the device that the driver is attaching to; though in our drivers for newer products we end up printing out both the device name and driver version because the driver version is expected to be there anyway.