Bug 232142 - Chelsio T520-SO cxgbe driver no longer recognized as 10Gb
Summary: Chelsio T520-SO cxgbe driver no longer recognized as 10Gb
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 11.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Navdeep Parhar
Keywords: regression
Depends on:
Reported: 2018-10-10 01:40 UTC by cyba
Modified: 2020-07-24 19:09 UTC (History)
5 users (show)

See Also:

ifconfig of 10GB Nic (36.37 KB, image/png)
2018-10-13 10:50 UTC, cyba
no flags Details
sysctl dev.t5nex.0.firmware_version (12.42 KB, image/png)
2018-10-15 00:26 UTC, cyba
no flags Details
loader.conf (10.30 KB, image/png)
2018-10-16 03:52 UTC, cyba
no flags Details
ifconfig of cxl card (31.42 KB, image/png)
2018-10-16 03:53 UTC, cyba
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description cyba 2018-10-10 01:40:44 UTC
I have a chelsio T520-SO on a pfsense box that is running freeBSD 11.2 Release on version 2.4.4 of pfsense. On teh previous release, the network cards were recognized as a 10GB card, now they are only seen as a 1GB card. Not sure what is going on here, I have tried everything I could to try and get it working, but it appears to be capped at 1GB only. Need help
Comment 1 Navdeep Parhar freebsd_committer 2018-10-12 22:58:37 UTC
What exactly do you mean when you say the NIC is recognized as a 1GB card?
There's not even an ifconfig output in this bug and it's hard to tell what's
being reported here.
Comment 2 cyba 2018-10-13 10:50:52 UTC
Created attachment 198097 [details]
ifconfig of 10GB Nic

This is an ifconfig printout of our pfsense firewall running pfsense version 2.4.4 that is running on freeBSD 11.2p3
Comment 3 cyba 2018-10-13 10:52:52 UTC
I uploaded a screenshot, I apologize I forgot to do that. These are 10GB cards, the Chelsio T-520-SO cards, but they are not recognized as a 10GB NIC, just 1GB. Not sure what is causing this to happen, however, this was not an issue until the system was upgraded to the latest version of pfsense based on FreeBSD 11.2
Comment 4 Jason Wolfe 2018-10-14 00:52:10 UTC
This looks like it could an interoperability issue with a new firmware and an old driver, we ran into this at Limelight. They moved over from outputting speeds in mb to kb I believe it was, so the old driver gets confused if running a newer firmware.

What's the sysctll output of dev.t5nex.0.firmware_version ? If you are hitting this bug you have 2 options:

- Upgrade to a newer driver. It looks like you are running the newer version of pfsense w FreeBSD 11.2, so it'd take you pulling the new driver into the 11.2 tree, and likely reverting a bunch of the tie in the driver has with the rest of the kernel, like the epoch and TLS changes. This could get rough, but we did test this and it's doable.

- Downgrade the firmware to one that's compatible with the driver. The binary firmware blob is actually included in the driver, but by default it will only upgrade the firmware on the card if it's newer. If you set hw.cxgbe.fw_install="1" in loader.conf, that tells the driver it's allowed to downgrade the firmware also. Give this a try if you'd like.
Comment 5 cyba 2018-10-15 00:26:56 UTC
Created attachment 198159 [details]
sysctl dev.t5nex.0.firmware_version

Attached is the printout from sysctl dev.t5nex.0.firmware_version
Comment 6 cyba 2018-10-15 00:28:21 UTC
I appreciate you getting back to me on the issue. I attached the screenshot for what you were asking. 

I have also added the line in loader.conf and it did not produce anything different. I am not sure what else to do.

Thanks for your time so far
Comment 7 Jason Wolfe 2018-10-15 15:30:14 UTC
My mistake, 1 is the default. Try setting hw.cxgbe.fw_install="2" in loader.conf to allow the driver to downgrade the firmware.
Comment 8 cyba 2018-10-16 03:52:37 UTC
Created attachment 198195 [details]

I ended up doing what you suggested, but I still have the same issue of only 1GB versus 10GB. I appreciate your help thus far. I am uploading the screenshots. Thanks
Comment 9 cyba 2018-10-16 03:53:42 UTC
Created attachment 198196 [details]
ifconfig of cxl card

Attached 2nd Screen Shot
Comment 10 Navdeep Parhar freebsd_committer 2019-01-10 22:39:59 UTC
Do you still see the problem with a recent 11-STABLE or 12.0?  There have been
some ifmedia related fixes in the driver since 11.2 and they have likely fixed
this.  Please close this bug as resolved if that's the case.
Comment 11 cyba 2019-01-10 23:16:45 UTC
(In reply to Navdeep Parhar from comment #10)

I have not been able to update yet. At the mercy of pfsense and if they updated to bsd 12. We are going to update the firewalls and provide feedback. Issues still open
Comment 12 Navdeep Parhar freebsd_committer 2019-01-10 23:26:54 UTC
(In reply to cyba from comment #11)
I see.  Are you able to temporarily boot this system from a recent 11.2-STABLE
or 12.0-RELEASE boot media, load the driver, and verify that it displays
correct media/link information?

If this bug is fixed in FreeBSD we should close it here.
Comment 13 cyba 2019-01-10 23:40:29 UTC
I am going to be out of town until the 25th. We will update the firewall and see if it works. If it doesn’t I will let you know then I can try what you suggested to see if that works as well. I apologize. Thanks for the patience
Comment 14 cyba 2019-01-11 04:22:16 UTC
Updated pfsense, they are still using FreeBSD 11.2 so I have no way to tell other than booting from a live usb. Thanks
Comment 15 Navdeep Parhar freebsd_committer 2020-07-24 19:09:11 UTC
A number of link/ifmedia bugs have been fixed since 11.2 and the problem that
was reported here should not occur with any recent FreeBSD release.  I'm closing
this.  Please reopen if the problem still occurs on a recent 11-STABLE,