Hello, I started diagnosing a FreeBSD-12.3 system with a X550-T2 10GbE controller that was having trouble making outgoing TCP connections (ssh for example). I used tcpdump and could see a response returning to the host but the client software didn't seem to receive the packet and move forward. The port was configured for 1 queue per direction (dev.ix.0.iflib.override_nrxqs=1 and dev.ix.0.iflib.override_ntxqs=1) and all offloads were disabled. I discovered that enabling rxcsum restored stable networking. I could complete ssh connections as well as generic TCP/UDP connections using nc. As soon as I removed the rxcsum offload, the initial behavior returned. I also discovered setting the number of queues > 1 was stable while queues=1 hung connections. I looked at ixgbe_initialize_receive_units() in if_ix.c where the RXCSUM register is set. Here's the relevant section: rxcsum = IXGBE_READ_REG(hw, IXGBE_RXCSUM); ixgbe_initialize_rss_mapping(sc); if (sc->num_rx_queues > 1) { /* RSS and RX IPP Checksum are mutually exclusive */ rxcsum |= IXGBE_RXCSUM_PCSD; } if (ifp->if_capenable & IFCAP_RXCSUM) rxcsum |= IXGBE_RXCSUM_PCSD; /* This is useful for calculating UDP/IP fragment checksums */ if (!(rxcsum & IXGBE_RXCSUM_PCSD)) rxcsum |= IXGBE_RXCSUM_IPPCSE; IXGBE_WRITE_REG(hw, IXGBE_RXCSUM, rxcsum); Not hard to see that if either queues>1 OR rxcsum is enabled the PCSD bit is set and IPPCSE is not set. In the case where queues==1 AND !rxcsum, PCSD is not set and IPPCSE is set (and my TCP connections hang). I believe there is a problem here but I'm not sure which part. A few notes: * The linux driver unconditionally sets PCSD and doesn't touch IPPCSE in ixgbe_set_mrqc(). * The e1000 FreeBSD driver sets PCSD if rx_queues > 1. It only calls xxx_initialize_rss_mapping() if rx_queues > 1 is true. ixgbe calls xxx_initialize_rss_mapping outside of the conditional so it is always run. Does this mean PCSD must always be set? * I couldn't find anywhere in the driver where the UDP/IP fragment checksum was used. Is there any benefit to setting IPPCSE? I modified the above code to always set PCSD and never set IPPCSE. In my testing on the X550-T2 and an optical X520 82599ES, this change eliminated the hangs even with queues=1 and -rxcsum. I'm hoping this is enough detail for someone to identity the true root cause and best solution. Please let me know if you need additional information.
Did you check it on 13.1-STABLE as well?
(In reply to Przemysław Lewandowski from comment #1) No I have not tested 13.x. I have seen this issue on both 12.3 and 12.4. I might be able to back port any relevant commits to my build but I don't expect to run 13.x anytime soon.
I cannot reproduce this issue. I set dev.ix.0.iflib.override_nrxqs=1 and dev.ix.0.iflib.override_ntxqs=1 in /boot/loader.conf and disabled all offloads: ifconfig ix0 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso -lro -vlanmtu -vlanhwcsum -vlanhwtag -vlanhwfilter -vlanhwtso -vxlanhwcsum -vxlanhwtso. SSH and nc traffic works properly. Could you provide exact steps to reproduce and observe issue? Maybe I missed something.
(In reply to Przemysław Lewandowski from comment #3) I apologize for wasting your time. I went back to my system which uses a custom kernel config and began testing. My configuration file includes RSS which is not part of the GENERIC config: options PCBGROUP # Distributed Protocol Control Block Groups options RSS # Receive Side Scaling When I commented out RSS and PCBGROUP (required for RSS) and rebuilt the kernel, I had no trouble with ssh/nc TCP connections. However I use RSS across multiple NICs and with it included in the kernel I see instability with TCP (ssh inbound failing, wget outbound failing).
This issue exists on 13.1 as well. I cannot reproduce on 14.0 but here PCBGROUP is not support (I cannot compile source with this flag), I used only RSS flag. https://reviews.freebsd.org/D33019 - regarding PCBGROUP flag. I remove PCBGROUP flag from 13.1 and the issue does not occur. So I think a fix is needed in 13 and 12 versions. I am preparing patch. Regarding your notes: I looked at ixgbe driver for ESXi. There is PCSD set only when RSS is enabling. Document for 10G cards suggests the same.
Maybe the best solution is merged this patch to 12 and 13: https://reviews.freebsd.org/D33019
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=156424fce98a0240c702da0a918b027d1979bfdb commit 156424fce98a0240c702da0a918b027d1979bfdb Author: Przemyslaw Lewandowski <przemyslawx.lewandowski@intel.com> AuthorDate: 2023-05-24 23:24:54 +0000 Commit: Eric Joyner <erj@FreeBSD.org> CommitDate: 2023-05-24 23:29:40 +0000 ixgbe: Change if condition for RSS and rxcsum This patch fixes TCP connection hangs for 1 rxq and 1 txq without rxcsum enabled. Documentation for 10G cards and other drivers suggest enabling rxcsum for RSS and disabling otherwise. When PCSD bit is not set then fragment checksum and IP identification are reported in the rx descriptor. When PCSD bit is set then RSS hash value is reported in the rx descriptor. RSS and RX IPP checksum are mutually exclusive. Signed-off-by: Eric Joyner <erj@FreeBSD.org> PR: 268910 Reviewed by: erj@ Tested by: jeffrey.e.pieper@intel.com MFC after: 1 week Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D38621 sys/dev/ixgbe/if_ix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Thank you for the commit. I just applied it to FreeBSD 13.2 and observed the correct behavior.
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=95aced446a8c6ada94a5a9dead6e2c68ce48ecb3 commit 95aced446a8c6ada94a5a9dead6e2c68ce48ecb3 Author: Przemyslaw Lewandowski <przemyslawx.lewandowski@intel.com> AuthorDate: 2023-05-24 23:24:54 +0000 Commit: Eric Joyner <erj@FreeBSD.org> CommitDate: 2023-06-02 19:27:45 +0000 ixgbe: Change if condition for RSS and rxcsum This patch fixes TCP connection hangs for 1 rxq and 1 txq without rxcsum enabled. Documentation for 10G cards and other drivers suggest enabling rxcsum for RSS and disabling otherwise. When PCSD bit is not set then fragment checksum and IP identification are reported in the rx descriptor. When PCSD bit is set then RSS hash value is reported in the rx descriptor. RSS and RX IPP checksum are mutually exclusive. Signed-off-by: Eric Joyner <erj@FreeBSD.org> PR: 268910 Reviewed by: erj@ Tested by: jeffrey.e.pieper@intel.com Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D38621 (cherry picked from commit 156424fce98a0240c702da0a918b027d1979bfdb) sys/dev/ixgbe/if_ix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=197106cf41dff164a16d09cf84fe10321ca1ebdb commit 197106cf41dff164a16d09cf84fe10321ca1ebdb Author: Przemyslaw Lewandowski <przemyslawx.lewandowski@intel.com> AuthorDate: 2023-05-24 23:24:54 +0000 Commit: Eric Joyner <erj@FreeBSD.org> CommitDate: 2023-06-02 19:35:05 +0000 ixgbe: Change if condition for RSS and rxcsum This patch fixes TCP connection hangs for 1 rxq and 1 txq without rxcsum enabled. Documentation for 10G cards and other drivers suggest enabling rxcsum for RSS and disabling otherwise. When PCSD bit is not set then fragment checksum and IP identification are reported in the rx descriptor. When PCSD bit is set then RSS hash value is reported in the rx descriptor. RSS and RX IPP checksum are mutually exclusive. Signed-off-by: Eric Joyner <erj@FreeBSD.org> PR: 268910 Reviewed by: erj@ Tested by: jeffrey.e.pieper@intel.com Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D38621 (cherry picked from commit 156424fce98a0240c702da0a918b027d1979bfdb) sys/dev/ixgbe/if_ix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)