Bug 237720 - [tcp] tcpip network stack seized for six hours after large high-throughput file transfer
Summary: [tcp] tcpip network stack seized for six hours after large high-throughput fi...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-net (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-05-03 05:35 UTC by Naveen Nathan
Modified: 2020-10-23 18:06 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Naveen Nathan 2019-05-03 05:35:16 UTC
I have a server running FreeBSD 12.0-RELEASE-p3 which has been upgraded over the years, originally starting its life as FreeBSD 8. The firewall (pf/ipfw) has never been enabled.

I recently sftp'd a 5GB file from another system to my server at 40-70MB/s and the transfer completed. Shortly after (about 1-5 minutes) it appears the entire tcpip network stack seized and several hours later unseized. I lost connectivity and was unable to SSH to the server. After several hours, the server came back online. Looking at the logs, I came across several lines like this:

May  3 09:16:08 redactedhostname sendmail[20886]: x42NFHn5020886: SYSERR(redacteduser): timeout writing message to [127.0.0.1]

These were of course cron jobs that were failing and so mail was trying to be sent. Going by the logs, the networking stack seized for 6 hours.

I have experienced this problem before on the same server: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=218653 . However it is notable that since that bug has been filed, the server was upgraded with new hardware. I am now using the igb driver for "Intel 82576 Gigabit Network Connection" PCI card. However given that connectivity to the loopback went down, I don't think this issue is related to hardware or a specific NIC driver.
Comment 1 Naveen Nathan 2019-05-04 00:27:38 UTC
There has been some suggestions that this issue might have happened due to mbuf exhaustion. Here are a few commands which I think might help with a post-mortem diagnosis:

$ uptime
10:25AM  up 19 days,  9:45, 7 users, load averages: 0.02, 0.03, 0.00

$ uname -a
FreeBSD myhostname 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  amd64

$ netstat -m
8394/15306/23700 mbufs in use (current/cache/total)
8208/6802/15010/131072 mbuf clusters in use (current/cache/total/max)
17/6149 mbuf+clusters out of packet secondary zone in use (current/cache)
4/42/46/1020649 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/302414 9k jumbo clusters in use (current/cache/total/max)
0/0/0/170108 16k jumbo clusters in use (current/cache/total/max)
18530K/17598K/36129K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
11 sendfile syscalls
6 sendfile syscalls completed without I/O request
3 requests for I/O initiated by sendfile
14 pages read by sendfile as part of a request
54 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
47 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

$ vmstat -z                             
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP  
                                                                            
UMA Kegs:               296,      0,     261,       3,     261,   0,   0  
UMA Zones:             3488,      0,     278,       0,     278,   0,   0    
UMA Slabs:               80,      0, 6257315,  191035,13619139,   0,   0  
UMA Hash:               256,      0,      51,      39,     109,   0,   0    
4 Bucket:                32,      0,    2978,   22397, 6557670,   0,   0  
6 Bucket:                48,      0,     532,    9179, 1453684,   0,   0    
8 Bucket:                64,      0,     203,    8353, 1091498,  21,   0  
12 Bucket:               96,      0,     376,    4380, 1240085,   0,   0 
16 Bucket:              128,      0,     462,    4622, 1140684,   1,   0  
32 Bucket:              256,      0,     476,    2749, 1654959, 411,   0  
64 Bucket:              512,      0,     765,    1891, 2208533,6308,   0    
128 Bucket:            1024,      0,     592,     620,  870033,6781,   0  
256 Bucket:            2048,      0,     814,     294,  772380,6883,   0    
vmem:                  1792,      0,       9,       1,       9,   0,   0  
vmem btag:               56,      0,   69378,   37193,  491288, 751,   0    
VM OBJECT:              256,      0,  462620, 1109560,34906108,   0,   0  
RADIX NODE:             144,      0,   63068,   31243,69681498,   0,   0    
MAP:                    232,      0,       3,      65,       3,   0,   0 
KMAP ENTRY:             120,      0,      10,     221,      12,   0,   0  
MAP ENTRY:              120,      0,    5566,    7931,111879850,   0,   0
VMSPACE:               2512,      0,      41,     250, 1933974,   0,   0       
fakepg:                 104,      0,       1,    1253,      21,   0,   0 
64 pcpu:                  8,      0,    3208,    2424,    3640,   0,   0  
mt_stats_zone:           64,      0,     437,     139,     437,   0,   0 
mt_zone:                 24,      0,     437,     565,     437,   0,   0  
16:                      16,      0,  621313,   18988,114067699,   0,   0
32:                      32,      0,  149317,   86308,290305615,   0,   0 
64:                      64,      0, 1266541,  457431,315561388,   0,   0 
128:                    128,      0,   80300,   52535,336400015,   0,   0 
256:                    256,      0,  496496,  104164,119539158,   0,   0 
512:                    512,      0,  154487,   15945,93550244,   0,   0  
1024:                  1024,      0,    1661,     883,33271013,   0,   0       
2048:                  2048,      0,     168,     238,435995430,   0,   0 
4096:                  4096,      0,   28898,      50,21765487,   0,   0  
8192:                  8192,      0,     234,      46, 1000755,   0,   0  
16384:                16384,      0,      42,      44,   42734,   0,   0  
32768:                32768,      0,     125,      15,    1281,   0,   0  
65536:                65536,      0,      16,      24,   16746,   0,   0  
SLEEPQUEUE:              80,      0,    1765,     653,    1765,   0,   0  
Files:                   80,      0,     239,    3911,40338051,   0,   0  
filedesc0:             1104,      0,      87,     288, 1934019,   0,   0  
TURNSTILE:              136,      0,    1765,     815,    1765,   0,   0  
rl_entry:                40,      0,     838,    4262,     838,   0,   0    
umtx pi:                 96,      0,       0,       0,       0,   0,   0  
umtx_shm:                88,      0,       0,       0,       0,   0,   0    
MAC labels:              40,      0,       0,       0,       0,   0,   0  
PROC:                  1320,      0,      86,     418, 1934018,   0,   0    
THREAD:                1400,      0,    1119,     645,   39613,   0,   0  
cpuset:                 104,      0,      51,    2150,     472,   0,   0    
domainset:               40,      0,       0,       0,       0,   0,   0  
audit_record:          1280,      0,       0,       0,       0,   0,   0 
mbuf_packet:            256, 13064310,       5,    7173, 2882769,   0,   0
mbuf:                   256, 13064310,    8372,    8930,91081495,   0,   0
mbuf_cluster:          2048, 131072,   15369,     655,19066913,   0,   0    
mbuf_jumbo_page:       4096, 1020649,       4,      83, 9188654,   0,   0 
mbuf_jumbo_9k:         9216, 302414,       0,       0,       0,   0,   0    
mbuf_jumbo_16k:       16384, 170108,       0,       0,       0,   0,   0  
epoch_record pcpu:      256,      0,       4,      12,       4,   0,   0    
FPU_save_area:          512,      0,       0,       0,       0,   0,   0  
g_bio:                  376,      0,       0,    1410,117520574,   0,   0   
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0 
ttyinq:                 160,      0,     195,    1430,29127360,   0,   0  
ttyoutq:                256,      0,     103,    1157,15211063,   0,   0 
nvme_request:           128,      0,       0,       0,       0,   0,   0       
cryptop:                128,      0,       0,       0,       0,   0,   0 
cryptodesc:             120,      0,       0,       0,       0,   0,   0  
crypto_session:          24,      0,       0,       0,       0,   0,   0 
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0  
taskq_zone:              48,      0,       0,    8134,23449413,   0,   0 
VNODE:                  480,      0,  621967,     721, 2749903,   0,   0  
VNODEPOLL:              120,      0,       0,     132,       8,   0,   0  
BUF TRIE:               144,      0,       0,  105948,       0,   0,   0  
S VFS Cache:            108,      0,  622856,   12289, 4991101,   0,   0  
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0  
L VFS Cache:            328,      0,   32040,   12780,  406199,   0,   0       
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0  
NAMEI:                 1024,      0,       0,     388,163185077,   0,   0 
rentr:                   24,      0,       0,     835,   12563,   0,   0  
DIRHASH:               1024,      0,       0,       0,       0,   0,   0  
NCLNODE:                592,      0,       0,       0,       0,   0,   0  
Mountpoints:           2744,      0,      22,     251,     529,   0,   0  
procdesc:               136,      0,       0,     638,      18,   0,   0  
AIO:                    208,      0,       0,       0,       0,   0,   0  
AIOP:                    32,      0,       0,       0,       0,   0,   0  
AIOCB:                  752,      0,       0,       0,       0,   0,   0  
AIOLIO:                 280,      0,       0,       0,       0,   0,   0    
range_seg_cache:         72,      0,   60784,   67201,22386000,   0,   0  
metaslab_alloc_trace_cache:     72,      0,       0,    3630,20430002,   0,   0
zio_cache:             1048,      0,      28,    2012,369175190,   0,   0 
zio_link_cache:          48,      0,       0,    9960,295409260,   0,   0   
zio_buf_512:            512,      0,  680913,   13823,36704101,   0,   0  
zio_data_buf_512:       512,      0,      85,    3003, 2619222,   0,   0    
zio_buf_1024:          1024,      0,      43,     369, 2608237,   0,   0  
zio_data_buf_1024:     1024,      0,      74,     506, 1166617,   0,   0 
zio_buf_1536:          1536,      0,      17,     287,  857875,   0,   0  
zio_data_buf_1536:     1536,      0,      50,     246, 1031438,   0,   0  
zio_buf_2048:          2048,      0,      49,     143, 2926535,   0,   0    
zio_data_buf_2048:     2048,      0,      38,     174,  743270,   0,   0  
zio_buf_2560:          2560,      0,       3,      88,  796858,   0,   0    
zio_data_buf_2560:     2560,      0,      32,      68,  444729,   0,   0  
zio_buf_3072:          3072,      0,       5,      87,  855179,   0,   0    
zio_data_buf_3072:     3072,      0,      36,      67,  418626,   0,   0  
zio_buf_3584:          3584,      0,       3,      35,  909208,   0,   0    
zio_data_buf_3584:     3584,      0,      25,      17,  344368,   0,   0 
zio_buf_4096:          4096,      0,     139,      83, 4390435,   0,   0  
zio_data_buf_4096:     4096,      0,      27,      20,  374270,   0,   0 
zio_buf_5120:          5120,      0,       6,      45, 1008401,   0,   0       
zio_data_buf_5120:     5120,      0,      32,      45,  338943,   0,   0 
zio_buf_6144:          6144,      0,    1366,     102, 1179423,   0,   0  
zio_data_buf_6144:     6144,      0,      33,      22,  285231,   0,   0 
zio_buf_7168:          7168,      0,      35,      36,  542980,   0,   0  
zio_data_buf_7168:     7168,      0,      28,      25,  204349,   0,   0 
zio_buf_8192:          8192,      0,       5,      33, 1068758,   0,   0  
zio_data_buf_8192:     8192,      0,      20,      29,  256157,   0,   0  
zio_buf_10240:        10240,      0,       2,      42, 1071550,   0,   0  
zio_data_buf_10240:   10240,      0,      43,      36,  348392,   0,   0  
zio_buf_12288:        12288,      0,       7,      35,  839958,   0,   0  
zio_data_buf_12288:   12288,      0,      27,      29,  264215,   0,   0       
zio_buf_14336:        14336,      0,       3,      33,  611067,   0,   0  
zio_data_buf_14336:   14336,      0,      29,      38,  174953,   0,   0  
zio_buf_16384:        16384,      0,   29106,     233, 9934508,   0,   0  
zio_data_buf_16384:   16384,      0,      15,      27,  184931,   0,   0  
zio_buf_20480:        20480,      0,       2,      33,  786252,   0,   0  
zio_data_buf_20480:   20480,      0,      29,      30,  238139,   0,   0  
zio_buf_24576:        24576,      0,       1,      29,  517586,   0,   0  
zio_data_buf_24576:   24576,      0,      30,      36,  186779,   0,   0  
zio_buf_28672:        28672,      0,       1,      46,  238846,   0,   0  
zio_data_buf_28672:   28672,      0,      10,      37,  146537,   0,   0  
zio_buf_32768:        32768,      0,       1,      32,  328727,   0,   0    
zio_data_buf_32768:   32768,      0,      12,      36,  138941,   0,   0  
zio_buf_40960:        40960,      0,       6,      30,  232479,   0,   0       
zio_data_buf_40960:   40960,      0,      15,      41,  216165,   0,   0  
zio_buf_49152:        49152,      0,       1,      53,  378777,   0,   0    
zio_data_buf_49152:   49152,      0,      12,      43,  183832,   0,   0  
zio_buf_57344:        57344,      0,       2,      27,  125230,   0,   0    
zio_data_buf_57344:   57344,      0,       8,      44,  130295,   0,   0  
zio_buf_65536:        65536,      0,       0,      33,  143598,   0,   0 
zio_data_buf_65536:   65536,      0,       7,      46,  154120,   0,   0  
zio_buf_81920:        81920,      0,       2,      56,  242951,   0,   0  
zio_data_buf_81920:   81920,      0,       7,      44,  526745,   0,   0    
zio_buf_98304:        98304,      0,       0,      34,  262299,   0,   0  
zio_data_buf_98304:   98304,      0,       4,      42,  713406,   0,   0    
zio_buf_114688:      114688,      0,       1,      33,  168828,   0,   0  
zio_data_buf_114688: 114688,      0,       2,      44,  112192,   0,   0    
zio_buf_131072:      131072,      0,     130,     120, 1339480,   0,   0  
zio_data_buf_131072: 131072,      0,    6818,     110, 6759995,   0,   0    
zio_buf_163840:      163840,      0,       0,      27,  134271,   0,   0 
zio_data_buf_163840: 163840,      0,       0,       0,       0,   0,   0  
zio_buf_196608:      196608,      0,       0,      25,   42169,   0,   0 
zio_data_buf_196608: 196608,      0,       0,       0,       0,   0,   0       
zio_buf_229376:      229376,      0,       0,      31,   25436,   0,   0 
zio_data_buf_229376: 229376,      0,       0,       0,       0,   0,   0  
zio_buf_262144:      262144,      0,       0,      26,   56652,   0,   0 
zio_data_buf_262144: 262144,      0,       0,       0,       0,   0,   0  
zio_buf_327680:      327680,      0,       0,      28,   43238,   0,   0 
zio_data_buf_327680: 327680,      0,       0,       0,       0,   0,   0  
zio_buf_393216:      393216,      0,       0,      24,   41616,   0,   0  
zio_data_buf_393216: 393216,      0,       0,       0,       0,   0,   0  
zio_buf_458752:      458752,      0,       0,      24,    9353,   0,   0  
zio_data_buf_458752: 458752,      0,       0,       0,       0,   0,   0  
zio_buf_524288:      524288,      0,       0,      25,   30041,   0,   0       
zio_data_buf_524288: 524288,      0,       0,       0,       0,   0,   0  
zio_buf_655360:      655360,      0,       0,      28,   34362,   0,   0  
zio_data_buf_655360: 655360,      0,       0,       0,       0,   0,   0  
zio_buf_786432:      786432,      0,       0,      20,   54420,   0,   0  
zio_data_buf_786432: 786432,      0,       0,       0,       0,   0,   0  
zio_buf_917504:      917504,      0,       0,      18,   33793,   0,   0  
zio_data_buf_917504: 917504,      0,       0,       0,       0,   0,   0  
zio_buf_1048576:     1048576,      0,       0,      34,  299267,   0,   0 
zio_data_buf_1048576: 1048576,      0,       0,       0,       0,   0,   0
zio_buf_1310720:     1310720,      0,       0,       0,       0,   0,   0 
zio_data_buf_1310720: 1310720,      0,       0,       0,       0,   0,   0  
zio_buf_1572864:     1572864,      0,       0,       0,       0,   0,   0 
zio_data_buf_1572864: 1572864,      0,       0,       0,       0,   0,   0     
zio_buf_1835008:     1835008,      0,       0,       0,       0,   0,   0 
zio_data_buf_1835008: 1835008,      0,       0,       0,       0,   0,   0  
zio_buf_2097152:     2097152,      0,       0,       0,       0,   0,   0 
zio_data_buf_2097152: 2097152,      0,       0,       0,       0,   0,   0  
zio_buf_2621440:     2621440,      0,       0,       0,       0,   0,   0 
zio_data_buf_2621440: 2621440,      0,       0,       0,       0,   0,   0
zio_buf_3145728:     3145728,      0,       0,       0,       0,   0,   0 
zio_data_buf_3145728: 3145728,      0,       0,       0,       0,   0,   0
zio_buf_3670016:     3670016,      0,       0,       0,       0,   0,   0   
zio_data_buf_3670016: 3670016,      0,       0,       0,       0,   0,   0
zio_buf_4194304:     4194304,      0,       0,       0,       0,   0,   0   
zio_data_buf_4194304: 4194304,      0,       0,       0,       0,   0,   0
zio_buf_5242880:     5242880,      0,       0,       0,       0,   0,   0   
zio_data_buf_5242880: 5242880,      0,       0,       0,       0,   0,   0
zio_buf_6291456:     6291456,      0,       0,       0,       0,   0,   0   
zio_data_buf_6291456: 6291456,      0,       0,       0,       0,   0,   0
zio_buf_7340032:     7340032,      0,       0,       0,       0,   0,   0 
zio_data_buf_7340032: 7340032,      0,       0,       0,       0,   0,   0
zio_buf_8388608:     8388608,      0,       0,       0,       0,   0,   0      
zio_data_buf_8388608: 8388608,      0,       0,       0,       0,   0,   0
zio_buf_10485760:    10485760,      0,       0,       0,       0,   0,   0
zio_data_buf_10485760: 10485760,      0,       0,       0,       0,   0,   0
zio_buf_12582912:    12582912,      0,       0,       0,       0,   0,   0
zio_data_buf_12582912: 12582912,      0,       0,       0,       0,   0,   0
zio_buf_14680064:    14680064,      0,       0,       0,       0,   0,   0
zio_data_buf_14680064: 14680064,      0,       0,       0,       0,   0,   0
zio_buf_16777216:    16777216,      0,       0,       0,       0,   0,   0
zio_data_buf_16777216: 16777216,      0,       0,       0,       0,   0,   0
lz4_ctx:              16384,      0,       0,      47,13646201,   0,   0  
abd_chunk:             4096,      0, 6062533,    7608,129128463,   0,   0      
sa_cache:               144,      0,  621778,    1274, 2425352,   0,   0  
dnode_t:                728,      0,  779201,    3374, 3728924,   0,   0  
arc_buf_hdr_t_full:     256,      0,  922135,   69095,14402646,   0,   0  
arc_buf_hdr_t_l2only:     96,      0,       0,       0,       0,   0,   0 
arc_buf_t:               64,      0,   38564,   81096,21561760,   0,   0  
dmu_buf_impl_t:         240,      0,  719348,   61628,21105559,   0,   0  
zil_lwb_cache:          320,      0,      10,    1010,  116043,   0,   0  
zil_zcw_cache:           80,      0,       0,    2650,   90988,   0,   0  
sio_cache:              128,      0,       0,       0,       0,   0,   0  
zfs_znode_cache:        272,      0,  621778,     718, 2425012,   0,   0  
pipe:                   760,      0,      24,     391,  899076,   0,   0    
ksiginfo:               112,      0,     466,    5134, 2051786,   0,   0  
itimer:                 352,      0,       0,      33,       2,   0,   0       
KNOTE:                  160,      0,      39,    1961,32692147,   0,   0  
socket:                 872,  65536,      82,     446,  811406,   0,   0    
unpcb:                  256,  65550,      57,    1833,  265881,   0,   0  
ipq:                     56,   4118,       0,       0,       0,   0,   0    
udp_inpcb:              488,  65536,      10,     902,  468540,   0,   0  
udpcb:                   32,  65625,      10,    5615,  468540,   0,   0  
tcp_inpcb:              488,  65536,      14,     618,   76937,   0,   0  
tcpcb:                  976,  65536,      14,     330,   76937,   0,   0  
tcptw:                   88,  13140,       0,    3780,   26469,   0,   0    
syncache:               168,  15364,       0,    1472,   42801,   0,   0  
hostcache:               96,  15375,      12,    1259,    5095,   0,   0    
sackhole:                32,      0,       0,    2125,   25188,   0,   0  
tfo:                      4,      0,       0,       0,       0,   0,   0    
tfo_ccache_entries:      80,      0,       0,       0,       0,   0,   0  
tcpreass:                48,   8217,       0,    2075,   38014,   0,   0    
tcp_log:                400, 1000000,       0,       0,       0,   0,   0 
tcp_log_bucket:         144,      0,       0,       0,       0,   0,   0  
tcp_log_node:           120,      0,       0,       0,       0,   0,   0  
sctp_ep:               1280,  65538,       0,       0,       0,   0,   0       
sctp_asoc:             2408,  40000,       0,       0,       0,   0,   0  
sctp_laddr:              48,  80012,       0,       0,       3,   0,   0  
sctp_raddr:             736,  80000,       0,       0,       0,   0,   0    
sctp_chunk:             152, 400010,       0,       0,       0,   0,   0  
sctp_readq:             152, 400010,       0,       0,       0,   0,   0    
sctp_stream_msg_out:    112, 400015,       0,       0,       0,   0,   0  
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0    
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0  
udplite_inpcb:          488,  65536,       0,       0,       0,   0,   0    
ripcb:                  488,  65536,       0,     320,      39,   0,   0  
IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0       
rtentry:                208,      0,      11,     122,      13,   0,   0  
selfd:                   64,      0,     865,    6575,151996806,   0,   0 
swpctrie:               144, 4082643,       0,       0,       0,   0,   0 
swblk:                  136, 4082620,       0,       0,       0,   0,   0 
pf mtags:                48,      0,       0,       0,       0,   0,   0  
pf states:              296, 100009,       0,       0,       0,   0,   0  
pf state keys:           88,      0,       0,       0,       0,   0,   0  
pf source nodes:        136,  10005,       0,       0,       0,   0,   0  
pf table entries:       160,      0,       0,       0,       0,   0,   0  
pf table counters:       64,      0,       0,       0,       0,   0,   0  
pf frags:               112,      0,       0,       0,       0,   0,   0    
pf frag entries:         40,   5000,       0,       0,       0,   0,   0  
pf state scrubs:         40,      0,       0,       0,       0,   0,   0       
FFS inode:              160,      0,       0,       0,       0,   0,   0  
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0    
FFS2 dinode:            256,      0,       0,       0,       0,   0,   0  

$ sysctl dev.igb.0
dev.igb.0.interrupts.rx_overrun: 0
dev.igb.0.interrupts.rx_desc_min_thresh: 0
dev.igb.0.interrupts.tx_queue_min_thresh: 0
dev.igb.0.interrupts.tx_queue_empty: 0
dev.igb.0.interrupts.tx_abs_timer: 0
dev.igb.0.interrupts.tx_pkt_timer: 0
dev.igb.0.interrupts.rx_abs_timer: 0
dev.igb.0.interrupts.rx_pkt_timer: 0
dev.igb.0.interrupts.asserts: 0
dev.igb.0.mac_stats.tso_ctx_fail: 0
dev.igb.0.mac_stats.tso_txd: 0
dev.igb.0.mac_stats.tx_frames_1024_1522: 0
dev.igb.0.mac_stats.tx_frames_512_1023: 0
dev.igb.0.mac_stats.tx_frames_256_511: 0
dev.igb.0.mac_stats.tx_frames_128_255: 0
dev.igb.0.mac_stats.tx_frames_65_127: 0
dev.igb.0.mac_stats.tx_frames_64: 0
dev.igb.0.mac_stats.mcast_pkts_txd: 0
dev.igb.0.mac_stats.bcast_pkts_txd: 0
dev.igb.0.mac_stats.good_pkts_txd: 0
dev.igb.0.mac_stats.total_pkts_txd: 0
dev.igb.0.mac_stats.good_octets_txd: 0
dev.igb.0.mac_stats.good_octets_recvd: 0
dev.igb.0.mac_stats.rx_frames_1024_1522: 0
dev.igb.0.mac_stats.rx_frames_512_1023: 0
dev.igb.0.mac_stats.rx_frames_256_511: 0
dev.igb.0.mac_stats.rx_frames_128_255: 0
dev.igb.0.mac_stats.rx_frames_65_127: 0
dev.igb.0.mac_stats.rx_frames_64: 0
dev.igb.0.mac_stats.mcast_pkts_recvd: 0
dev.igb.0.mac_stats.bcast_pkts_recvd: 0
dev.igb.0.mac_stats.good_pkts_recvd: 0
dev.igb.0.mac_stats.total_pkts_recvd: 0
dev.igb.0.mac_stats.xoff_txd: 0
dev.igb.0.mac_stats.xoff_recvd: 0
dev.igb.0.mac_stats.xon_txd: 0
dev.igb.0.mac_stats.xon_recvd: 0
dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.recv_errs: 0
dev.igb.0.mac_stats.recv_jabber: 0
dev.igb.0.mac_stats.recv_oversize: 0
dev.igb.0.mac_stats.recv_fragmented: 0
dev.igb.0.mac_stats.recv_undersize: 0
dev.igb.0.mac_stats.recv_no_buff: 0
dev.igb.0.mac_stats.missed_packets: 0
dev.igb.0.mac_stats.defer_count: 0
dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.collision_count: 0
dev.igb.0.mac_stats.late_coll: 0
dev.igb.0.mac_stats.multiple_coll: 0
dev.igb.0.mac_stats.single_coll: 0
dev.igb.0.mac_stats.excess_coll: 0
dev.igb.0.queue_rx_7.rx_irq: 0
dev.igb.0.queue_rx_7.rxd_tail: 0
dev.igb.0.queue_rx_7.rxd_head: 0
dev.igb.0.queue_rx_6.rx_irq: 0
dev.igb.0.queue_rx_6.rxd_tail: 0
dev.igb.0.queue_rx_6.rxd_head: 0
dev.igb.0.queue_rx_5.rx_irq: 0
dev.igb.0.queue_rx_5.rxd_tail: 0
dev.igb.0.queue_rx_5.rxd_head: 0
dev.igb.0.queue_rx_4.rx_irq: 0
dev.igb.0.queue_rx_4.rxd_tail: 0
dev.igb.0.queue_rx_4.rxd_head: 0
dev.igb.0.queue_rx_3.rx_irq: 0
dev.igb.0.queue_rx_3.rxd_tail: 0
dev.igb.0.queue_rx_3.rxd_head: 0
dev.igb.0.queue_rx_2.rx_irq: 0
dev.igb.0.queue_rx_2.rxd_tail: 0
dev.igb.0.queue_rx_2.rxd_head: 0
dev.igb.0.queue_rx_1.rx_irq: 0
dev.igb.0.queue_rx_1.rxd_tail: 0
dev.igb.0.queue_rx_1.rxd_head: 0
dev.igb.0.queue_rx_0.rx_irq: 0
dev.igb.0.queue_rx_0.rxd_tail: 0
dev.igb.0.queue_rx_0.rxd_head: 0
dev.igb.0.queue_tx_7.tx_irq: 0
dev.igb.0.queue_tx_7.txd_tail: 0
dev.igb.0.queue_tx_7.txd_head: 0
dev.igb.0.queue_tx_6.tx_irq: 0
dev.igb.0.queue_tx_6.txd_tail: 0
dev.igb.0.queue_tx_6.txd_head: 0
dev.igb.0.queue_tx_5.tx_irq: 0
dev.igb.0.queue_tx_5.txd_tail: 0
dev.igb.0.queue_tx_5.txd_head: 0
dev.igb.0.queue_tx_4.tx_irq: 0
dev.igb.0.queue_tx_4.txd_tail: 0
dev.igb.0.queue_tx_4.txd_head: 0
dev.igb.0.queue_tx_3.tx_irq: 0
dev.igb.0.queue_tx_3.txd_tail: 0
dev.igb.0.queue_tx_3.txd_head: 0
dev.igb.0.queue_tx_2.tx_irq: 0
dev.igb.0.queue_tx_2.txd_tail: 0
dev.igb.0.queue_tx_2.txd_head: 0
dev.igb.0.queue_tx_1.tx_irq: 0
dev.igb.0.queue_tx_1.txd_tail: 0
dev.igb.0.queue_tx_1.txd_head: 0
dev.igb.0.queue_tx_0.tx_irq: 0
dev.igb.0.queue_tx_0.txd_tail: 0
dev.igb.0.queue_tx_0.txd_head: 0
dev.igb.0.fc_low_water: 4294965240
dev.igb.0.fc_high_water: 4294965248
dev.igb.0.rx_control: 0
dev.igb.0.device_control: 12583489
dev.igb.0.watchdog_timeouts: 0
dev.igb.0.rx_overruns: 0
dev.igb.0.tx_dma_fail: 0
dev.igb.0.mbuf_defrag_fail: 0
dev.igb.0.link_irq: 0
dev.igb.0.dropped: 0
dev.igb.0.eee_control: 1
dev.igb.0.itr: 488
dev.igb.0.tx_abs_int_delay: 66
dev.igb.0.rx_abs_int_delay: 66
dev.igb.0.tx_int_delay: 66
dev.igb.0.rx_int_delay: 0
dev.igb.0.rs_dump: 0
dev.igb.0.reg_dump: General Registers
	CTRL	 00c00241
	STATUS	 00080780
	CTRL_EXIT	 10180c00

Interrupt Registers
	ICR	 00000000

RX Registers
	RCTL	 00000000
	RDLEN	 00000000
	RDH	 00000000
	RDT	 00000000
	RXDCTL	 00010000
	RDBAL	 00000000
	RDBAH	 00000000

TX Registers
	TCTL	 a40400f8
	TDBAL	 00000000
	TDBAH	 00000000
	TDLEN	 00000000
	TDH	 00000000
	TDT	 00000000
	TXDCTL	 00000000
	TDFH	 40020002
	TDFT	 00000000
	TDFHS	 00000000
	TDFPC	 00000000


dev.igb.0.fc: 3
dev.igb.0.debug: -1
dev.igb.0.nvm: -1
dev.igb.0.iflib.rxq7.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq7.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq7.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq6.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq6.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq6.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq5.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq5.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq5.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq4.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq4.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq4.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq3.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq3.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq3.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq2.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq2.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq2.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq1.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq1.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq1.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq0.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq0.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq0.rxq_fl0.pidx: 0
dev.igb.0.iflib.txq7.r_abdications: 0
dev.igb.0.iflib.txq7.r_restarts: 0
dev.igb.0.iflib.txq7.r_stalls: 0
dev.igb.0.iflib.txq7.r_starts: 0
dev.igb.0.iflib.txq7.r_drops: 0
dev.igb.0.iflib.txq7.r_enqueues: 0
dev.igb.0.iflib.txq7.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq7.txq_cleaned: 0
dev.igb.0.iflib.txq7.txq_processed: 0
dev.igb.0.iflib.txq7.txq_in_use: 0
dev.igb.0.iflib.txq7.txq_cidx_processed: 0
dev.igb.0.iflib.txq7.txq_cidx: 0
dev.igb.0.iflib.txq7.txq_pidx: 0
dev.igb.0.iflib.txq7.no_tx_dma_setup: 0
dev.igb.0.iflib.txq7.txd_encap_efbig: 0
dev.igb.0.iflib.txq7.tx_map_failed: 0
dev.igb.0.iflib.txq7.no_desc_avail: 0
dev.igb.0.iflib.txq7.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq7.m_pullups: 0
dev.igb.0.iflib.txq7.mbuf_defrag: 0
dev.igb.0.iflib.txq6.r_abdications: 0
dev.igb.0.iflib.txq6.r_restarts: 0
dev.igb.0.iflib.txq6.r_stalls: 0
dev.igb.0.iflib.txq6.r_starts: 0
dev.igb.0.iflib.txq6.r_drops: 0
dev.igb.0.iflib.txq6.r_enqueues: 0
dev.igb.0.iflib.txq6.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq6.txq_cleaned: 0
dev.igb.0.iflib.txq6.txq_processed: 0
dev.igb.0.iflib.txq6.txq_in_use: 0
dev.igb.0.iflib.txq6.txq_cidx_processed: 0
dev.igb.0.iflib.txq6.txq_cidx: 0
dev.igb.0.iflib.txq6.txq_pidx: 0
dev.igb.0.iflib.txq6.no_tx_dma_setup: 0
dev.igb.0.iflib.txq6.txd_encap_efbig: 0
dev.igb.0.iflib.txq6.tx_map_failed: 0
dev.igb.0.iflib.txq6.no_desc_avail: 0
dev.igb.0.iflib.txq6.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq6.m_pullups: 0
dev.igb.0.iflib.txq6.mbuf_defrag: 0
dev.igb.0.iflib.txq5.r_abdications: 0
dev.igb.0.iflib.txq5.r_restarts: 0
dev.igb.0.iflib.txq5.r_stalls: 0
dev.igb.0.iflib.txq5.r_starts: 0
dev.igb.0.iflib.txq5.r_drops: 0
dev.igb.0.iflib.txq5.r_enqueues: 0
dev.igb.0.iflib.txq5.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq5.txq_cleaned: 0
dev.igb.0.iflib.txq5.txq_processed: 0
dev.igb.0.iflib.txq5.txq_in_use: 0
dev.igb.0.iflib.txq5.txq_cidx_processed: 0
dev.igb.0.iflib.txq5.txq_cidx: 0
dev.igb.0.iflib.txq5.txq_pidx: 0
dev.igb.0.iflib.txq5.no_tx_dma_setup: 0
dev.igb.0.iflib.txq5.txd_encap_efbig: 0
dev.igb.0.iflib.txq5.tx_map_failed: 0
dev.igb.0.iflib.txq5.no_desc_avail: 0
dev.igb.0.iflib.txq5.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq5.m_pullups: 0
dev.igb.0.iflib.txq5.mbuf_defrag: 0
dev.igb.0.iflib.txq4.r_abdications: 0
dev.igb.0.iflib.txq4.r_restarts: 0
dev.igb.0.iflib.txq4.r_stalls: 0
dev.igb.0.iflib.txq4.r_starts: 0
dev.igb.0.iflib.txq4.r_drops: 0
dev.igb.0.iflib.txq4.r_enqueues: 0
dev.igb.0.iflib.txq4.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq4.txq_cleaned: 0
dev.igb.0.iflib.txq4.txq_processed: 0
dev.igb.0.iflib.txq4.txq_in_use: 0
dev.igb.0.iflib.txq4.txq_cidx_processed: 0
dev.igb.0.iflib.txq4.txq_cidx: 0
dev.igb.0.iflib.txq4.txq_pidx: 0
dev.igb.0.iflib.txq4.no_tx_dma_setup: 0
dev.igb.0.iflib.txq4.txd_encap_efbig: 0
dev.igb.0.iflib.txq4.tx_map_failed: 0
dev.igb.0.iflib.txq4.no_desc_avail: 0
dev.igb.0.iflib.txq4.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq4.m_pullups: 0
dev.igb.0.iflib.txq4.mbuf_defrag: 0
dev.igb.0.iflib.txq3.r_abdications: 0
dev.igb.0.iflib.txq3.r_restarts: 0
dev.igb.0.iflib.txq3.r_stalls: 0
dev.igb.0.iflib.txq3.r_starts: 0
dev.igb.0.iflib.txq3.r_drops: 0
dev.igb.0.iflib.txq3.r_enqueues: 0
dev.igb.0.iflib.txq3.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq3.txq_cleaned: 0
dev.igb.0.iflib.txq3.txq_processed: 0
dev.igb.0.iflib.txq3.txq_in_use: 0
dev.igb.0.iflib.txq3.txq_cidx_processed: 0
dev.igb.0.iflib.txq3.txq_cidx: 0
dev.igb.0.iflib.txq3.txq_pidx: 0
dev.igb.0.iflib.txq3.no_tx_dma_setup: 0
dev.igb.0.iflib.txq3.txd_encap_efbig: 0
dev.igb.0.iflib.txq3.tx_map_failed: 0
dev.igb.0.iflib.txq3.no_desc_avail: 0
dev.igb.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq3.m_pullups: 0
dev.igb.0.iflib.txq3.mbuf_defrag: 0
dev.igb.0.iflib.txq2.r_abdications: 0
dev.igb.0.iflib.txq2.r_restarts: 0
dev.igb.0.iflib.txq2.r_stalls: 0
dev.igb.0.iflib.txq2.r_starts: 0
dev.igb.0.iflib.txq2.r_drops: 0
dev.igb.0.iflib.txq2.r_enqueues: 0
dev.igb.0.iflib.txq2.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq2.txq_cleaned: 0
dev.igb.0.iflib.txq2.txq_processed: 0
dev.igb.0.iflib.txq2.txq_in_use: 0
dev.igb.0.iflib.txq2.txq_cidx_processed: 0
dev.igb.0.iflib.txq2.txq_cidx: 0
dev.igb.0.iflib.txq2.txq_pidx: 0
dev.igb.0.iflib.txq2.no_tx_dma_setup: 0
dev.igb.0.iflib.txq2.txd_encap_efbig: 0
dev.igb.0.iflib.txq2.tx_map_failed: 0
dev.igb.0.iflib.txq2.no_desc_avail: 0
dev.igb.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq2.m_pullups: 0
dev.igb.0.iflib.txq2.mbuf_defrag: 0
dev.igb.0.iflib.txq1.r_abdications: 0
dev.igb.0.iflib.txq1.r_restarts: 0
dev.igb.0.iflib.txq1.r_stalls: 0
dev.igb.0.iflib.txq1.r_starts: 0
dev.igb.0.iflib.txq1.r_drops: 0
dev.igb.0.iflib.txq1.r_enqueues: 0
dev.igb.0.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq1.txq_cleaned: 0
dev.igb.0.iflib.txq1.txq_processed: 0
dev.igb.0.iflib.txq1.txq_in_use: 0
dev.igb.0.iflib.txq1.txq_cidx_processed: 0
dev.igb.0.iflib.txq1.txq_cidx: 0
dev.igb.0.iflib.txq1.txq_pidx: 0
dev.igb.0.iflib.txq1.no_tx_dma_setup: 0
dev.igb.0.iflib.txq1.txd_encap_efbig: 0
dev.igb.0.iflib.txq1.tx_map_failed: 0
dev.igb.0.iflib.txq1.no_desc_avail: 0
dev.igb.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq1.m_pullups: 0
dev.igb.0.iflib.txq1.mbuf_defrag: 0
dev.igb.0.iflib.txq0.r_abdications: 0
dev.igb.0.iflib.txq0.r_restarts: 0
dev.igb.0.iflib.txq0.r_stalls: 0
dev.igb.0.iflib.txq0.r_starts: 0
dev.igb.0.iflib.txq0.r_drops: 0
dev.igb.0.iflib.txq0.r_enqueues: 0
dev.igb.0.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq0.txq_cleaned: 0
dev.igb.0.iflib.txq0.txq_processed: 0
dev.igb.0.iflib.txq0.txq_in_use: 0
dev.igb.0.iflib.txq0.txq_cidx_processed: 0
dev.igb.0.iflib.txq0.txq_cidx: 0
dev.igb.0.iflib.txq0.txq_pidx: 0
dev.igb.0.iflib.txq0.no_tx_dma_setup: 0
dev.igb.0.iflib.txq0.txd_encap_efbig: 0
dev.igb.0.iflib.txq0.tx_map_failed: 0
dev.igb.0.iflib.txq0.no_desc_avail: 0
dev.igb.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq0.m_pullups: 0
dev.igb.0.iflib.txq0.mbuf_defrag: 0
dev.igb.0.iflib.override_nrxds: 0
dev.igb.0.iflib.override_ntxds: 0
dev.igb.0.iflib.tx_abdicate: 0
dev.igb.0.iflib.rx_budget: 0
dev.igb.0.iflib.disable_msix: 0
dev.igb.0.iflib.override_qs_enable: 0
dev.igb.0.iflib.override_nrxqs: 0
dev.igb.0.iflib.override_ntxqs: 0
dev.igb.0.iflib.driver_version: 7.6.1-k
dev.igb.0.%parent: pci3
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1526 subvendor=0x8086 subdevice=0xa06c class=0x020000
dev.igb.0.%location: slot=0 function=0 dbsf=pci0:6:0:0
dev.igb.0.%driver: igb
dev.igb.0.%desc: Intel(R) PRO/1000 PCI-Express Network Driver
Comment 2 Naveen Nathan 2019-05-04 10:23:13 UTC
If needed I can reproduce the problem and perhaps run a monitoring program while the issue is occurring to collect useful information to help understand the problem.
Comment 3 Michael Tuexen freebsd_committer 2019-05-04 10:54:11 UTC
(In reply to Naveen Nathan from comment #2)
First of all it would be great to know that you can reproduce it and if that is the case how you do it.
Comment 4 Rick Macklem freebsd_committer 2019-05-04 16:17:41 UTC
Since no one else has mentioned this yet...
The stats suggest to me that you've fragmented the mbuf cluster memory pool.
9K mbuf clusters are known to be a serious problem, see this recent post:
http://docs.FreeBSD.org/cgi/mid.cgi?23756.39015.553779.526064

Some net interfaces have a setting that tells them to not use 9K mbuf clusters
even if the interface is using 9K jumbo packets.
If that exists for this net driver, I'd suggest you try it.

For some reason, the stats show large numbers of both 9K and 4K mbuf clusters.
(The 4K mbuf clusters aren't nearly as bad w.r.t. fragmentation, but mixing
 them with the 9K ones seems likely to cause fragmentation.)

Alternately, I'd suggest you turn of jumbo packets and try it with ordinary
1500 byte ethernet packets.

Hopefully others more conversant with this net driver and the mbuf stats will
comment.
Comment 5 Michael Tuexen freebsd_committer 2019-05-04 17:22:53 UTC
(In reply to Rick Macklem from comment #4)
From which stat value do you conclude that 9k cluster elements where used? I read the line

0/0/0/302414 9k jumbo clusters in use (current/cache/total/max)

as none are used.
Comment 6 Rick Macklem freebsd_committer 2019-05-04 21:20:01 UTC
Oops, yea. I read the 4K line as the 9K line...
Comment 7 Rick Macklem freebsd_committer 2019-05-04 21:22:41 UTC
and misread the "max" line as the in use...
Comment 8 Rick Macklem freebsd_committer 2019-05-04 21:32:04 UTC
Ok, this time I think I have read the stats correctly and the VNODE
numbers look like they may be high?

You could check to see what the value of kern.maxvnodes is?
# sysctl -a | fgrep maxvnodes
Comment 9 Naveen Nathan 2019-05-07 22:11:11 UTC
(In reply to Rick Macklem from comment #8)

$ sysctl -a | fgrep maxvnodes
kern.maxvnodes: 622168
Comment 10 Naveen Nathan 2019-05-07 22:13:19 UTC
(In reply to Michael Tuexen from comment #3)

To reproduce the issue I simply do a large 5+ GB transfer over sftp to the server. After transfer is complete, a few moments later the network stack will seize, and hours later unseize.
Comment 11 Rick Macklem freebsd_committer 2019-05-07 23:15:28 UTC
Well, the stats you posted show that you are near your vnode limit, but
I have no idea how that might affect your sftp transfer.

You could try increasing kern.maxvnodes and see if it has any effect?