Bug 218894

Summary: Network dropouts on em(4) due to jumbo cluster failures
Product: Base System Reporter: Mike Andrews <mandrews>
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: New ---    
Severity: Affects Many People CC: kaho, kbowling, sbruno, shurd
Priority: --- Keywords: IntelNetworking
Version: 11.0-RELEASE   
Hardware: amd64   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246003
Bug Depends on:    
Bug Blocks: 255070    
Attachments:
Description Flags
a patch for jumbo frame and others none

Description Mike Andrews 2017-04-26 16:50:04 UTC 
This is a hard one to reproduce on demand, unfortunately.

After about two weeks of uptime, several of our systems of varying vintage will lose most/all network connectivity for several minutes, then go back to working as normal without anything having been done.

When these drops happen, "netstat -i" shows a jump in Ierrs, and "netstat -m" shows a jump in "requests for jumbo clusters denied" for 9K.  So something is causing jumbo allocations to fail, which in turn causes Ierrs, which in turn causes temporary loss of connectivity.  When it starts happening, it usually happens once or twice a day, and gets worse over time until a reboot clears it up for a while.

Some Googling on this indicates that this might be a memory fragmentation issue, and that after some uptime, it might be hard for the kernel to find a contiguous block of memory larger than 4K (1 page), and that there might be a defragmentation process that isn't happening.  I'm now unable to find the specific page that led me to that wild theory though; that was a few weeks ago.  Oopsie.

We only run jumbo frames on one VLAN, and we use an MTU of 5000 instead of 9000 because we have some Supermicro PDSMi+-based systems we can't yet get rid of (grumble) that use 82573L NICs that have hardware bugs and once choked on anything bigger (and 82573E NICs that don't do jumbos at all).  Whether that's contributing to the problem, in trying to allocate 4K+1K vs 4K+4K+1K, I'm not sure.

I'm also running with "-tso" because leaving TSO on causes problems with NFS stalls for us -- similar problems, but probably unrelated to this issue.

The affected systems have 82574L NICs and 82579LM NICs -- Supermicro X9SCM-F, X8STi-F, X8SIE-F, X8DT6-F.

The one igb-based (I350) system we have, a Supermicro X9DRD-7LN4F, doesn't seem to be affected by this issue at all.

This is 11.0-RELEASE, which has em driver 7.6.1 in it.  I have tried the net/intel-em-kmod-7.6.2 port and it doesn't help.

Short of buying a crapload of igb or ixgbe cards and/or turning off jumbo frames, any ideas on how to troubleshoot and fix this before 11.1-RELEASE?  Anything I can pull out of netstat, sysctl, etc?
Comment 1 Kaho Toshikazu 2017-05-31 05:29:40 UTC 
Created attachment 183084 [details]
a patch for jumbo frame and others

I don't have a trouble denied some packets temporarily but a trouble dropped
many packets with jumbo frames. I use 12-current with a similar patch attached
this comment, and the patch does not make everything correct but makes
better condition.

I have three devices, 82574L, I217-V, I218-V. I don't have any trouble for I21[78] regardless with or without the patch when mtu is 1500.
With 9k jumbo, without the patch, scp speed is very slow and dev.em.0.rx_overruns increases.

On 82574L, dev.em.0.rx_overruns increases without the patch and
dev.em.0.mac_stats.recv_no_buff increases with the patch.
Above 6k jumbo, scp speed is almost 10kB/s without the patch
and the speed is about 10MB/s with the patch.
Comment 2 Mike Andrews 2018-11-05 16:34:44 UTC 
Dusting this old one off because it's still an issue.

This recent commit looks like, from the description anyway (haven't tested), that it might fix the issue:

https://reviews.freebsd.org/D16534
Comment 3 Kaho Toshikazu 2018-11-06 03:18:06 UTC 
(In reply to Mike Andrews from comment #2)

I don't know the patch resolves the problems originally reported in this PR.
82574L is very slow, but "netstat -m" shows "0/0/0 requests for jumbo clusters denied" with/without the patch. 

At least, PBA and high/low-water setting for I21[789] is broken because of 
receive buffer size is too small to save JUMBO frames. PBA should be calculated from MTU, and high/low-water should be calculated from PBA instead of loading
immediate values, I think.