Bug 218894 - Network dropouts on em(4) due to jumbo cluster failures
Summary: Network dropouts on em(4) due to jumbo cluster failures
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking
Depends on:
Blocks:
 
Reported: 2017-04-26 16:50 UTC by Mike Andrews
Modified: 2018-11-06 03:18 UTC (History)
3 users (show)

See Also:


Attachments
a patch for jumbo frame and others (7.35 KB, patch)
2017-05-31 05:29 UTC, Kaho Toshikazu
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Andrews 2017-04-26 16:50:04 UTC
This is a hard one to reproduce on demand, unfortunately.

After about two weeks of uptime, several of our systems of varying vintage will lose most/all network connectivity for several minutes, then go back to working as normal without anything having been done.

When these drops happen, "netstat -i" shows a jump in Ierrs, and "netstat -m" shows a jump in "requests for jumbo clusters denied" for 9K.  So something is causing jumbo allocations to fail, which in turn causes Ierrs, which in turn causes temporary loss of connectivity.  When it starts happening, it usually happens once or twice a day, and gets worse over time until a reboot clears it up for a while.

Some Googling on this indicates that this might be a memory fragmentation issue, and that after some uptime, it might be hard for the kernel to find a contiguous block of memory larger than 4K (1 page), and that there might be a defragmentation process that isn't happening.  I'm now unable to find the specific page that led me to that wild theory though; that was a few weeks ago.  Oopsie.

We only run jumbo frames on one VLAN, and we use an MTU of 5000 instead of 9000 because we have some Supermicro PDSMi+-based systems we can't yet get rid of (grumble) that use 82573L NICs that have hardware bugs and once choked on anything bigger (and 82573E NICs that don't do jumbos at all).  Whether that's contributing to the problem, in trying to allocate 4K+1K vs 4K+4K+1K, I'm not sure.

I'm also running with "-tso" because leaving TSO on causes problems with NFS stalls for us -- similar problems, but probably unrelated to this issue.

The affected systems have 82574L NICs and 82579LM NICs -- Supermicro X9SCM-F, X8STi-F, X8SIE-F, X8DT6-F.

The one igb-based (I350) system we have, a Supermicro X9DRD-7LN4F, doesn't seem to be affected by this issue at all.

This is 11.0-RELEASE, which has em driver 7.6.1 in it.  I have tried the net/intel-em-kmod-7.6.2 port and it doesn't help.

Short of buying a crapload of igb or ixgbe cards and/or turning off jumbo frames, any ideas on how to troubleshoot and fix this before 11.1-RELEASE?  Anything I can pull out of netstat, sysctl, etc?
Comment 1 Kaho Toshikazu 2017-05-31 05:29:40 UTC
Created attachment 183084 [details]
a patch for jumbo frame and others

I don't have a trouble denied some packets temporarily but a trouble dropped
many packets with jumbo frames. I use 12-current with a similar patch attached
this comment, and the patch does not make everything correct but makes
better condition.

I have three devices, 82574L, I217-V, I218-V. I don't have any trouble for I21[78] regardless with or without the patch when mtu is 1500.
With 9k jumbo, without the patch, scp speed is very slow and dev.em.0.rx_overruns increases.

On 82574L, dev.em.0.rx_overruns increases without the patch and
dev.em.0.mac_stats.recv_no_buff increases with the patch.
Above 6k jumbo, scp speed is almost 10kB/s without the patch
and the speed is about 10MB/s with the patch.
Comment 2 Mike Andrews 2018-11-05 16:34:44 UTC
Dusting this old one off because it's still an issue.

This recent commit looks like, from the description anyway (haven't tested), that it might fix the issue:

https://reviews.freebsd.org/D16534
Comment 3 Kaho Toshikazu 2018-11-06 03:18:06 UTC
(In reply to Mike Andrews from comment #2)

I don't know the patch resolves the problems originally reported in this PR.
82574L is very slow, but "netstat -m" shows "0/0/0 requests for jumbo clusters denied" with/without the patch. 

At least, PBA and high/low-water setting for I21[789] is broken because of 
receive buffer size is too small to save JUMBO frames. PBA should be calculated from MTU, and high/low-water should be calculated from PBA instead of loading
immediate values, I think.