Bug 221967 - ixgbe(4) driver for Intel 82598 chip fails to attach if used 10G-SR SFP+ with message "ix0 attach returned 5"
Summary: ixgbe(4) driver for Intel 82598 chip fails to attach if used 10G-SR SFP+ with...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Sean Bruno
URL:
Keywords: IntelNetworking, patch
Depends on:
Blocks:
 
Reported: 2017-08-31 22:30 UTC by Andrey V. Elsukov
Modified: 2019-04-17 08:36 UTC (History)
4 users (show)

See Also:


Attachments
some debuginfo from ipmi console (113.66 KB, image/png)
2017-08-31 22:30 UTC, Andrey V. Elsukov
no flags Details
used patch (832 bytes, patch)
2017-08-31 22:31 UTC, Andrey V. Elsukov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey V. Elsukov freebsd_committer 2017-08-31 22:30:58 UTC
Created attachment 185963 [details]
some debuginfo from ipmi console

We have several INTEL-E10G42AFDA cards that worked well with Twinax cable, but driver fails to attach if 10G-SR SFP plugged.

The output from patched driver:

% pciconf -l | grep 0x0200
ix0@pci0:4:0:0:	class=0x020000 card=0xa21f8086 chip=0x10f18086 rev=0x01 hdr=0x00
ix1@pci0:4:0:1:	class=0x020000 card=0xa21f8086 chip=0x10f18086 rev=0x01 hdr=0x00
ix2@pci0:7:0:0:	class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
ix3@pci0:7:0:1:	class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00

% grep ^ix /var/run/dmesg.boot
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x7020-0x703f mem 0xdf720000-0xdf73ffff,0xdf6c0000-0xdf6fffff,0xdf744000-0xdf747fff irq 32 at device 0.0 on pci4
ix0: Using MSIX interrupts with 8 vectors
ix0: Ethernet address: 00:1b:21:6c:e1:8f
ix0: PCI Express Bus: Speed 2.5GT/s Width x8
ix0: netmap queues/slots: TX 7/2048, RX 7/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x7000-0x701f mem 0xdf700000-0xdf71ffff,0xdf640000-0xdf67ffff,0xdf740000-0xdf743fff irq 36 at device 0.1 on pci4
ix1: Using MSIX interrupts with 8 vectors
ix1: Ethernet address: 00:1b:21:6c:e1:8e
ix1: PCI Express Bus: Speed 2.5GT/s Width x8
ix1: netmap queues/slots: TX 7/2048, RX 7/2048
ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x6020-0x603f mem 0xde680000-0xde6fffff,0xde704000-0xde707fff irq 42 at device 0.0 on pci7
ix2: Using MSIX interrupts with 8 vectors
ix2: Ethernet address: 90:e2:ba:2b:52:ac
ix2: PCI Express Bus: Speed 5.0GT/s Width x8
ix2: netmap queues/slots: TX 7/2048, RX 7/2048
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x6000-0x601f mem 0xde600000-0xde67ffff,0xde700000-0xde703fff irq 45 at device 0.1 on pci7
ix3: Using MSIX interrupts with 8 vectors
ix3: Ethernet address: 90:e2:ba:2b:52:ad
ix3: PCI Express Bus: Speed 5.0GT/s Width x8
ix3: netmap queues/slots: TX 7/2048, RX 7/2048
ix0: link state changed to UP
ix2: link state changed to UP
ix3: link state changed to UP
ix1: link state changed to UP

All cards use the same SFP modules
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: Upnet PN: SFP+-10G-SR SN: SB3A680013 DATE: 2011-03-14

but 82598 card fails to attach it with message:
ix0: Unsupported SFP+ Module
device_attach: ix0 attach returned 5

I did some debugging to determine where the driver fails. The fail is in ixgbe_get_sfp_init_sequence_offsets() function. As I understand, there should be some values correspoinding to sfp_type, but in the table only one value, that doesn't match.
I modifed the driver that it thinks that SFP module has ixgbe_sfp_type_da_cu type, and after that it successful attached and works, but of course an information about SFP is unavailable:

% ifconfig -v ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=8100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:21:6c:e1:8f
	inet6 fe80::21b:21ff:fe6c:e18f%ix0 prefixlen 64 scopeid 0x3 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active

Also there is another version of patch, that seems solves the same issue, but I didn't test it.
http://www.grosbein.net/freebsd/patches/patch-if_ix.c
Comment 1 Andrey V. Elsukov freebsd_committer 2017-08-31 22:31:47 UTC
Created attachment 185964 [details]
used patch
Comment 2 Andrey V. Elsukov freebsd_committer 2017-08-31 22:34:50 UTC
Forgot to mention, in the loader.conf we have these tunables set, and this doesn't help: 
hw.ixgbe.unsupported_sfp="1"
hw.ix.unsupported_sfp="1"
Comment 3 Sean Bruno freebsd_committer 2017-09-06 17:20:14 UTC
Eric:

Does this patch look right to you?
Comment 4 Eric Joyner freebsd_committer 2017-09-06 17:46:18 UTC
(In reply to Sean Bruno from comment #3)

I think the patch linked in the original comment might be a better choice than the attached one if it works, but ultimately, the decision on our side should be Jeb's -- I'll cc him.
Comment 5 Eitan Adler freebsd_committer freebsd_triage 2018-05-20 23:59:44 UTC
For bugs matching the following conditions:
- Status == In Progress
- Assignee == "bugs@FreeBSD.org"
- Last Modified Year <= 2017

Do
- Set Status to "Open"
Comment 6 Jeff Pieper 2018-05-21 14:28:51 UTC
E10G42AFDA does not support SFP+ optics:
 
https://www.intel.com/content/dam/doc/product-brief/10-gigabit-af-da-dual-port-server-adapter-brief.pdf
 
Under specifications:
 
Cabling (passive)
 
SFP+ direct attached cable (twinaxial) only
SFP+ optics and active cables cannot be used with this adapter
Comment 7 Kristian 2019-04-16 22:41:55 UTC
Seems like a general issue with the intel driver.

It would be good to get this implemented:
http://www.grosbein.net/freebsd/patches/patch-if_ix.c

Even better would be to have it permanently enabled without having to use the allow_unsupported_sfp setting.
I mean in what situation would you want your interfaces disappearing during boot because of this.
Comment 8 Jeff Pieper 2019-04-16 22:52:24 UTC
(In reply to Kristian from comment #7)

If by "issue" you mean that this adapter doesn't support SFP+ optics, I agree...I linked the specs in a previous post and was ignored.
Comment 9 Kristian 2019-04-17 08:36:56 UTC
(In reply to Jeff Pieper from comment #8)

Yes I saw the reference to that specific NIC.
By this issue I meant that this issue is also present on intel NICs that support SPF optics when the optic is not programmed with the intel eeprom flags.

Here is talk of a similar fix:
https://sourceforge.net/p/e1000/mailman/message/28694855/