Bug 221967

Summary: ixgbe(4) driver for Intel 82598 chip fails to attach if used 10G-SR SFP+ with message "ix0 attach returned 5"
Product: Base System Reporter: Andrey V. Elsukov <ae>
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: cramerj, erj, jeffrey.e.pieper, kbowling, khj
Priority: --- Keywords: IntelNetworking, patch
Version: CURRENT   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
some debuginfo from ipmi console
none
used patch none

Description Andrey V. Elsukov freebsd_committer freebsd_triage 2017-08-31 22:30:58 UTC 
Created attachment 185963 [details]
some debuginfo from ipmi console

We have several INTEL-E10G42AFDA cards that worked well with Twinax cable, but driver fails to attach if 10G-SR SFP plugged.

The output from patched driver:

% pciconf -l | grep 0x0200
ix0@pci0:4:0:0:	class=0x020000 card=0xa21f8086 chip=0x10f18086 rev=0x01 hdr=0x00
ix1@pci0:4:0:1:	class=0x020000 card=0xa21f8086 chip=0x10f18086 rev=0x01 hdr=0x00
ix2@pci0:7:0:0:	class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
ix3@pci0:7:0:1:	class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00

% grep ^ix /var/run/dmesg.boot
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x7020-0x703f mem 0xdf720000-0xdf73ffff,0xdf6c0000-0xdf6fffff,0xdf744000-0xdf747fff irq 32 at device 0.0 on pci4
ix0: Using MSIX interrupts with 8 vectors
ix0: Ethernet address: 00:1b:21:6c:e1:8f
ix0: PCI Express Bus: Speed 2.5GT/s Width x8
ix0: netmap queues/slots: TX 7/2048, RX 7/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x7000-0x701f mem 0xdf700000-0xdf71ffff,0xdf640000-0xdf67ffff,0xdf740000-0xdf743fff irq 36 at device 0.1 on pci4
ix1: Using MSIX interrupts with 8 vectors
ix1: Ethernet address: 00:1b:21:6c:e1:8e
ix1: PCI Express Bus: Speed 2.5GT/s Width x8
ix1: netmap queues/slots: TX 7/2048, RX 7/2048
ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x6020-0x603f mem 0xde680000-0xde6fffff,0xde704000-0xde707fff irq 42 at device 0.0 on pci7
ix2: Using MSIX interrupts with 8 vectors
ix2: Ethernet address: 90:e2:ba:2b:52:ac
ix2: PCI Express Bus: Speed 5.0GT/s Width x8
ix2: netmap queues/slots: TX 7/2048, RX 7/2048
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.0> port 0x6000-0x601f mem 0xde600000-0xde67ffff,0xde700000-0xde703fff irq 45 at device 0.1 on pci7
ix3: Using MSIX interrupts with 8 vectors
ix3: Ethernet address: 90:e2:ba:2b:52:ad
ix3: PCI Express Bus: Speed 5.0GT/s Width x8
ix3: netmap queues/slots: TX 7/2048, RX 7/2048
ix0: link state changed to UP
ix2: link state changed to UP
ix3: link state changed to UP
ix1: link state changed to UP

All cards use the same SFP modules
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: Upnet PN: SFP+-10G-SR SN: SB3A680013 DATE: 2011-03-14

but 82598 card fails to attach it with message:
ix0: Unsupported SFP+ Module
device_attach: ix0 attach returned 5

I did some debugging to determine where the driver fails. The fail is in ixgbe_get_sfp_init_sequence_offsets() function. As I understand, there should be some values correspoinding to sfp_type, but in the table only one value, that doesn't match.
I modifed the driver that it thinks that SFP module has ixgbe_sfp_type_da_cu type, and after that it successful attached and works, but of course an information about SFP is unavailable:

% ifconfig -v ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=8100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:21:6c:e1:8f
	inet6 fe80::21b:21ff:fe6c:e18f%ix0 prefixlen 64 scopeid 0x3 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active

Also there is another version of patch, that seems solves the same issue, but I didn't test it.
http://www.grosbein.net/freebsd/patches/patch-if_ix.c
Comment 1 Andrey V. Elsukov freebsd_committer freebsd_triage 2017-08-31 22:31:47 UTC 
Created attachment 185964 [details]
used patch
Comment 2 Andrey V. Elsukov freebsd_committer freebsd_triage 2017-08-31 22:34:50 UTC 
Forgot to mention, in the loader.conf we have these tunables set, and this doesn't help: 
hw.ixgbe.unsupported_sfp="1"
hw.ix.unsupported_sfp="1"
Comment 3 Sean Bruno freebsd_committer freebsd_triage 2017-09-06 17:20:14 UTC 
Eric:

Does this patch look right to you?
Comment 4 Eric Joyner freebsd_committer freebsd_triage 2017-09-06 17:46:18 UTC 
(In reply to Sean Bruno from comment #3)

I think the patch linked in the original comment might be a better choice than the attached one if it works, but ultimately, the decision on our side should be Jeb's -- I'll cc him.
Comment 5 Eitan Adler freebsd_committer freebsd_triage 2018-05-20 23:59:44 UTC 
For bugs matching the following conditions:
- Status == In Progress
- Assignee == "bugs@FreeBSD.org"
- Last Modified Year <= 2017

Do
- Set Status to "Open"
Comment 6 Jeff Pieper 2018-05-21 14:28:51 UTC 
E10G42AFDA does not support SFP+ optics:
 
https://www.intel.com/content/dam/doc/product-brief/10-gigabit-af-da-dual-port-server-adapter-brief.pdf
 
Under specifications:
 
Cabling (passive)
 
SFP+ direct attached cable (twinaxial) only
SFP+ optics and active cables cannot be used with this adapter
Comment 7 Kristian 2019-04-16 22:41:55 UTC 
Seems like a general issue with the intel driver.

It would be good to get this implemented:
http://www.grosbein.net/freebsd/patches/patch-if_ix.c

Even better would be to have it permanently enabled without having to use the allow_unsupported_sfp setting.
I mean in what situation would you want your interfaces disappearing during boot because of this.
Comment 8 Jeff Pieper 2019-04-16 22:52:24 UTC 
(In reply to Kristian from comment #7)

If by "issue" you mean that this adapter doesn't support SFP+ optics, I agree...I linked the specs in a previous post and was ignored.
Comment 9 Kristian 2019-04-17 08:36:56 UTC 
(In reply to Jeff Pieper from comment #8)

Yes I saw the reference to that specific NIC.
By this issue I meant that this issue is also present on intel NICs that support SPF optics when the optic is not programmed with the intel eeprom flags.

Here is talk of a similar fix:
https://sourceforge.net/p/e1000/mailman/message/28694855/
Comment 10 Mark Linimon freebsd_committer freebsd_triage 2021-05-14 22:30:03 UTC 
^Triage: reassign to pool by assignee request.

To submitter: is this PR still valid?
Comment 11 Kevin Bowling freebsd_committer freebsd_triage 2021-10-04 17:50:56 UTC 
(In reply to Andrey V. Elsukov from comment #2)
Can you test the patch in comment #7 or is this two distinct problems?
Comment 12 Andrey V. Elsukov freebsd_committer freebsd_triage 2021-10-06 11:45:23 UTC 
(In reply to Kevin Bowling from comment #11)
Sorry, but I'm unable to test this. We have migrated to Mellanox cards.
Comment 13 Kevin Bowling freebsd_committer freebsd_triage 2021-10-06 22:53:41 UTC 
Intel doesn't support optics in the card.  Without an 82598 AF to test it is unclear if there is a clean way of using the sanctioned 'unsupported SFP' sysctl, there are many places that need to be inspected.  I'm not interested enough in this card to buy one on eBay with my own money so closing as OBE.