|Summary:||ixgbe(4) driver for Intel 82598 chip fails to attach if used 10G-SR SFP+ with message "ix0 attach returned 5"|
|Product:||Base System||Reporter:||Andrey V. Elsukov <ae>|
|Component:||kern||Assignee:||Sean Bruno <sbruno>|
|Severity:||Affects Only Me||CC:||cramerj, erj, jeffrey.e.pieper, khj|
Description Andrey V. Elsukov 2017-08-31 22:30:58 UTC
Created attachment 185963 [details] some debuginfo from ipmi console We have several INTEL-E10G42AFDA cards that worked well with Twinax cable, but driver fails to attach if 10G-SR SFP plugged. The output from patched driver: % pciconf -l | grep 0x0200 ix0@pci0:4:0:0: class=0x020000 card=0xa21f8086 chip=0x10f18086 rev=0x01 hdr=0x00 ix1@pci0:4:0:1: class=0x020000 card=0xa21f8086 chip=0x10f18086 rev=0x01 hdr=0x00 ix2@pci0:7:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00 ix3@pci0:7:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00 % grep ^ix /var/run/dmesg.boot ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x7020-0x703f mem 0xdf720000-0xdf73ffff,0xdf6c0000-0xdf6fffff,0xdf744000-0xdf747fff irq 32 at device 0.0 on pci4 ix0: Using MSIX interrupts with 8 vectors ix0: Ethernet address: 00:1b:21:6c:e1:8f ix0: PCI Express Bus: Speed 2.5GT/s Width x8 ix0: netmap queues/slots: TX 7/2048, RX 7/2048 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x7000-0x701f mem 0xdf700000-0xdf71ffff,0xdf640000-0xdf67ffff,0xdf740000-0xdf743fff irq 36 at device 0.1 on pci4 ix1: Using MSIX interrupts with 8 vectors ix1: Ethernet address: 00:1b:21:6c:e1:8e ix1: PCI Express Bus: Speed 2.5GT/s Width x8 ix1: netmap queues/slots: TX 7/2048, RX 7/2048 ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x6020-0x603f mem 0xde680000-0xde6fffff,0xde704000-0xde707fff irq 42 at device 0.0 on pci7 ix2: Using MSIX interrupts with 8 vectors ix2: Ethernet address: 90:e2:ba:2b:52:ac ix2: PCI Express Bus: Speed 5.0GT/s Width x8 ix2: netmap queues/slots: TX 7/2048, RX 7/2048 ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x6000-0x601f mem 0xde600000-0xde67ffff,0xde700000-0xde703fff irq 45 at device 0.1 on pci7 ix3: Using MSIX interrupts with 8 vectors ix3: Ethernet address: 90:e2:ba:2b:52:ad ix3: PCI Express Bus: Speed 5.0GT/s Width x8 ix3: netmap queues/slots: TX 7/2048, RX 7/2048 ix0: link state changed to UP ix2: link state changed to UP ix3: link state changed to UP ix1: link state changed to UP All cards use the same SFP modules plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: Upnet PN: SFP+-10G-SR SN: SB3A680013 DATE: 2011-03-14 but 82598 card fails to attach it with message: ix0: Unsupported SFP+ Module device_attach: ix0 attach returned 5 I did some debugging to determine where the driver fails. The fail is in ixgbe_get_sfp_init_sequence_offsets() function. As I understand, there should be some values correspoinding to sfp_type, but in the table only one value, that doesn't match. I modifed the driver that it thinks that SFP module has ixgbe_sfp_type_da_cu type, and after that it successful attached and works, but of course an information about SFP is unavailable: % ifconfig -v ix0 ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=8100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER> ether 00:1b:21:6c:e1:8f inet6 fe80::21b:21ff:fe6c:e18f%ix0 prefixlen 64 scopeid 0x3 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active Also there is another version of patch, that seems solves the same issue, but I didn't test it. http://www.grosbein.net/freebsd/patches/patch-if_ix.c
Comment 2 Andrey V. Elsukov 2017-08-31 22:34:50 UTC
Forgot to mention, in the loader.conf we have these tunables set, and this doesn't help: hw.ixgbe.unsupported_sfp="1" hw.ix.unsupported_sfp="1"
Comment 3 Sean Bruno 2017-09-06 17:20:14 UTC
Eric: Does this patch look right to you?
Comment 4 Eric Joyner 2017-09-06 17:46:18 UTC
(In reply to Sean Bruno from comment #3) I think the patch linked in the original comment might be a better choice than the attached one if it works, but ultimately, the decision on our side should be Jeb's -- I'll cc him.
Comment 5 Eitan Adler 2018-05-20 23:59:44 UTC
For bugs matching the following conditions: - Status == In Progress - Assignee == "bugs@FreeBSD.org" - Last Modified Year <= 2017 Do - Set Status to "Open"
Comment 6 Jeff Pieper 2018-05-21 14:28:51 UTC
E10G42AFDA does not support SFP+ optics: https://www.intel.com/content/dam/doc/product-brief/10-gigabit-af-da-dual-port-server-adapter-brief.pdf Under specifications: Cabling (passive) SFP+ direct attached cable (twinaxial) only SFP+ optics and active cables cannot be used with this adapter
Comment 7 Kristian 2019-04-16 22:41:55 UTC
Seems like a general issue with the intel driver. It would be good to get this implemented: http://www.grosbein.net/freebsd/patches/patch-if_ix.c Even better would be to have it permanently enabled without having to use the allow_unsupported_sfp setting. I mean in what situation would you want your interfaces disappearing during boot because of this.
Comment 8 Jeff Pieper 2019-04-16 22:52:24 UTC
(In reply to Kristian from comment #7) If by "issue" you mean that this adapter doesn't support SFP+ optics, I agree...I linked the specs in a previous post and was ignored.
Comment 9 Kristian 2019-04-17 08:36:56 UTC
(In reply to Jeff Pieper from comment #8) Yes I saw the reference to that specific NIC. By this issue I meant that this issue is also present on intel NICs that support SPF optics when the optic is not programmed with the intel eeprom flags. Here is talk of a similar fix: https://sourceforge.net/p/e1000/mailman/message/28694855/