Bug 237720
| Summary: | [tcp] tcpip network stack seized for six hours after large high-throughput file transfer | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Naveen Nathan <freebsd> |
| Component: | kern | Assignee: | freebsd-net (Nobody) <net> |
| Status: | Closed Overcome By Events | ||
| Severity: | Affects Only Me | CC: | kbowling, rmacklem, tuexen |
| Priority: | --- | ||
| Version: | 12.0-RELEASE | ||
| Hardware: | amd64 | ||
| OS: | Any | ||
|
Description
Naveen Nathan
2019-05-03 05:35:16 UTC
There has been some suggestions that this issue might have happened due to mbuf exhaustion. Here are a few commands which I think might help with a post-mortem diagnosis:
$ uptime
10:25AM up 19 days, 9:45, 7 users, load averages: 0.02, 0.03, 0.00
$ uname -a
FreeBSD myhostname 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC amd64
$ netstat -m
8394/15306/23700 mbufs in use (current/cache/total)
8208/6802/15010/131072 mbuf clusters in use (current/cache/total/max)
17/6149 mbuf+clusters out of packet secondary zone in use (current/cache)
4/42/46/1020649 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/302414 9k jumbo clusters in use (current/cache/total/max)
0/0/0/170108 16k jumbo clusters in use (current/cache/total/max)
18530K/17598K/36129K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
11 sendfile syscalls
6 sendfile syscalls completed without I/O request
3 requests for I/O initiated by sendfile
14 pages read by sendfile as part of a request
54 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
47 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed
$ vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 296, 0, 261, 3, 261, 0, 0
UMA Zones: 3488, 0, 278, 0, 278, 0, 0
UMA Slabs: 80, 0, 6257315, 191035,13619139, 0, 0
UMA Hash: 256, 0, 51, 39, 109, 0, 0
4 Bucket: 32, 0, 2978, 22397, 6557670, 0, 0
6 Bucket: 48, 0, 532, 9179, 1453684, 0, 0
8 Bucket: 64, 0, 203, 8353, 1091498, 21, 0
12 Bucket: 96, 0, 376, 4380, 1240085, 0, 0
16 Bucket: 128, 0, 462, 4622, 1140684, 1, 0
32 Bucket: 256, 0, 476, 2749, 1654959, 411, 0
64 Bucket: 512, 0, 765, 1891, 2208533,6308, 0
128 Bucket: 1024, 0, 592, 620, 870033,6781, 0
256 Bucket: 2048, 0, 814, 294, 772380,6883, 0
vmem: 1792, 0, 9, 1, 9, 0, 0
vmem btag: 56, 0, 69378, 37193, 491288, 751, 0
VM OBJECT: 256, 0, 462620, 1109560,34906108, 0, 0
RADIX NODE: 144, 0, 63068, 31243,69681498, 0, 0
MAP: 232, 0, 3, 65, 3, 0, 0
KMAP ENTRY: 120, 0, 10, 221, 12, 0, 0
MAP ENTRY: 120, 0, 5566, 7931,111879850, 0, 0
VMSPACE: 2512, 0, 41, 250, 1933974, 0, 0
fakepg: 104, 0, 1, 1253, 21, 0, 0
64 pcpu: 8, 0, 3208, 2424, 3640, 0, 0
mt_stats_zone: 64, 0, 437, 139, 437, 0, 0
mt_zone: 24, 0, 437, 565, 437, 0, 0
16: 16, 0, 621313, 18988,114067699, 0, 0
32: 32, 0, 149317, 86308,290305615, 0, 0
64: 64, 0, 1266541, 457431,315561388, 0, 0
128: 128, 0, 80300, 52535,336400015, 0, 0
256: 256, 0, 496496, 104164,119539158, 0, 0
512: 512, 0, 154487, 15945,93550244, 0, 0
1024: 1024, 0, 1661, 883,33271013, 0, 0
2048: 2048, 0, 168, 238,435995430, 0, 0
4096: 4096, 0, 28898, 50,21765487, 0, 0
8192: 8192, 0, 234, 46, 1000755, 0, 0
16384: 16384, 0, 42, 44, 42734, 0, 0
32768: 32768, 0, 125, 15, 1281, 0, 0
65536: 65536, 0, 16, 24, 16746, 0, 0
SLEEPQUEUE: 80, 0, 1765, 653, 1765, 0, 0
Files: 80, 0, 239, 3911,40338051, 0, 0
filedesc0: 1104, 0, 87, 288, 1934019, 0, 0
TURNSTILE: 136, 0, 1765, 815, 1765, 0, 0
rl_entry: 40, 0, 838, 4262, 838, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
umtx_shm: 88, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1320, 0, 86, 418, 1934018, 0, 0
THREAD: 1400, 0, 1119, 645, 39613, 0, 0
cpuset: 104, 0, 51, 2150, 472, 0, 0
domainset: 40, 0, 0, 0, 0, 0, 0
audit_record: 1280, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 13064310, 5, 7173, 2882769, 0, 0
mbuf: 256, 13064310, 8372, 8930,91081495, 0, 0
mbuf_cluster: 2048, 131072, 15369, 655,19066913, 0, 0
mbuf_jumbo_page: 4096, 1020649, 4, 83, 9188654, 0, 0
mbuf_jumbo_9k: 9216, 302414, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 170108, 0, 0, 0, 0, 0
epoch_record pcpu: 256, 0, 4, 12, 4, 0, 0
FPU_save_area: 512, 0, 0, 0, 0, 0, 0
g_bio: 376, 0, 0, 1410,117520574, 0, 0
DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0
ttyinq: 160, 0, 195, 1430,29127360, 0, 0
ttyoutq: 256, 0, 103, 1157,15211063, 0, 0
nvme_request: 128, 0, 0, 0, 0, 0, 0
cryptop: 128, 0, 0, 0, 0, 0, 0
cryptodesc: 120, 0, 0, 0, 0, 0, 0
crypto_session: 24, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0
taskq_zone: 48, 0, 0, 8134,23449413, 0, 0
VNODE: 480, 0, 621967, 721, 2749903, 0, 0
VNODEPOLL: 120, 0, 0, 132, 8, 0, 0
BUF TRIE: 144, 0, 0, 105948, 0, 0, 0
S VFS Cache: 108, 0, 622856, 12289, 4991101, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 32040, 12780, 406199, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 388,163185077, 0, 0
rentr: 24, 0, 0, 835, 12563, 0, 0
DIRHASH: 1024, 0, 0, 0, 0, 0, 0
NCLNODE: 592, 0, 0, 0, 0, 0, 0
Mountpoints: 2744, 0, 22, 251, 529, 0, 0
procdesc: 136, 0, 0, 638, 18, 0, 0
AIO: 208, 0, 0, 0, 0, 0, 0
AIOP: 32, 0, 0, 0, 0, 0, 0
AIOCB: 752, 0, 0, 0, 0, 0, 0
AIOLIO: 280, 0, 0, 0, 0, 0, 0
range_seg_cache: 72, 0, 60784, 67201,22386000, 0, 0
metaslab_alloc_trace_cache: 72, 0, 0, 3630,20430002, 0, 0
zio_cache: 1048, 0, 28, 2012,369175190, 0, 0
zio_link_cache: 48, 0, 0, 9960,295409260, 0, 0
zio_buf_512: 512, 0, 680913, 13823,36704101, 0, 0
zio_data_buf_512: 512, 0, 85, 3003, 2619222, 0, 0
zio_buf_1024: 1024, 0, 43, 369, 2608237, 0, 0
zio_data_buf_1024: 1024, 0, 74, 506, 1166617, 0, 0
zio_buf_1536: 1536, 0, 17, 287, 857875, 0, 0
zio_data_buf_1536: 1536, 0, 50, 246, 1031438, 0, 0
zio_buf_2048: 2048, 0, 49, 143, 2926535, 0, 0
zio_data_buf_2048: 2048, 0, 38, 174, 743270, 0, 0
zio_buf_2560: 2560, 0, 3, 88, 796858, 0, 0
zio_data_buf_2560: 2560, 0, 32, 68, 444729, 0, 0
zio_buf_3072: 3072, 0, 5, 87, 855179, 0, 0
zio_data_buf_3072: 3072, 0, 36, 67, 418626, 0, 0
zio_buf_3584: 3584, 0, 3, 35, 909208, 0, 0
zio_data_buf_3584: 3584, 0, 25, 17, 344368, 0, 0
zio_buf_4096: 4096, 0, 139, 83, 4390435, 0, 0
zio_data_buf_4096: 4096, 0, 27, 20, 374270, 0, 0
zio_buf_5120: 5120, 0, 6, 45, 1008401, 0, 0
zio_data_buf_5120: 5120, 0, 32, 45, 338943, 0, 0
zio_buf_6144: 6144, 0, 1366, 102, 1179423, 0, 0
zio_data_buf_6144: 6144, 0, 33, 22, 285231, 0, 0
zio_buf_7168: 7168, 0, 35, 36, 542980, 0, 0
zio_data_buf_7168: 7168, 0, 28, 25, 204349, 0, 0
zio_buf_8192: 8192, 0, 5, 33, 1068758, 0, 0
zio_data_buf_8192: 8192, 0, 20, 29, 256157, 0, 0
zio_buf_10240: 10240, 0, 2, 42, 1071550, 0, 0
zio_data_buf_10240: 10240, 0, 43, 36, 348392, 0, 0
zio_buf_12288: 12288, 0, 7, 35, 839958, 0, 0
zio_data_buf_12288: 12288, 0, 27, 29, 264215, 0, 0
zio_buf_14336: 14336, 0, 3, 33, 611067, 0, 0
zio_data_buf_14336: 14336, 0, 29, 38, 174953, 0, 0
zio_buf_16384: 16384, 0, 29106, 233, 9934508, 0, 0
zio_data_buf_16384: 16384, 0, 15, 27, 184931, 0, 0
zio_buf_20480: 20480, 0, 2, 33, 786252, 0, 0
zio_data_buf_20480: 20480, 0, 29, 30, 238139, 0, 0
zio_buf_24576: 24576, 0, 1, 29, 517586, 0, 0
zio_data_buf_24576: 24576, 0, 30, 36, 186779, 0, 0
zio_buf_28672: 28672, 0, 1, 46, 238846, 0, 0
zio_data_buf_28672: 28672, 0, 10, 37, 146537, 0, 0
zio_buf_32768: 32768, 0, 1, 32, 328727, 0, 0
zio_data_buf_32768: 32768, 0, 12, 36, 138941, 0, 0
zio_buf_40960: 40960, 0, 6, 30, 232479, 0, 0
zio_data_buf_40960: 40960, 0, 15, 41, 216165, 0, 0
zio_buf_49152: 49152, 0, 1, 53, 378777, 0, 0
zio_data_buf_49152: 49152, 0, 12, 43, 183832, 0, 0
zio_buf_57344: 57344, 0, 2, 27, 125230, 0, 0
zio_data_buf_57344: 57344, 0, 8, 44, 130295, 0, 0
zio_buf_65536: 65536, 0, 0, 33, 143598, 0, 0
zio_data_buf_65536: 65536, 0, 7, 46, 154120, 0, 0
zio_buf_81920: 81920, 0, 2, 56, 242951, 0, 0
zio_data_buf_81920: 81920, 0, 7, 44, 526745, 0, 0
zio_buf_98304: 98304, 0, 0, 34, 262299, 0, 0
zio_data_buf_98304: 98304, 0, 4, 42, 713406, 0, 0
zio_buf_114688: 114688, 0, 1, 33, 168828, 0, 0
zio_data_buf_114688: 114688, 0, 2, 44, 112192, 0, 0
zio_buf_131072: 131072, 0, 130, 120, 1339480, 0, 0
zio_data_buf_131072: 131072, 0, 6818, 110, 6759995, 0, 0
zio_buf_163840: 163840, 0, 0, 27, 134271, 0, 0
zio_data_buf_163840: 163840, 0, 0, 0, 0, 0, 0
zio_buf_196608: 196608, 0, 0, 25, 42169, 0, 0
zio_data_buf_196608: 196608, 0, 0, 0, 0, 0, 0
zio_buf_229376: 229376, 0, 0, 31, 25436, 0, 0
zio_data_buf_229376: 229376, 0, 0, 0, 0, 0, 0
zio_buf_262144: 262144, 0, 0, 26, 56652, 0, 0
zio_data_buf_262144: 262144, 0, 0, 0, 0, 0, 0
zio_buf_327680: 327680, 0, 0, 28, 43238, 0, 0
zio_data_buf_327680: 327680, 0, 0, 0, 0, 0, 0
zio_buf_393216: 393216, 0, 0, 24, 41616, 0, 0
zio_data_buf_393216: 393216, 0, 0, 0, 0, 0, 0
zio_buf_458752: 458752, 0, 0, 24, 9353, 0, 0
zio_data_buf_458752: 458752, 0, 0, 0, 0, 0, 0
zio_buf_524288: 524288, 0, 0, 25, 30041, 0, 0
zio_data_buf_524288: 524288, 0, 0, 0, 0, 0, 0
zio_buf_655360: 655360, 0, 0, 28, 34362, 0, 0
zio_data_buf_655360: 655360, 0, 0, 0, 0, 0, 0
zio_buf_786432: 786432, 0, 0, 20, 54420, 0, 0
zio_data_buf_786432: 786432, 0, 0, 0, 0, 0, 0
zio_buf_917504: 917504, 0, 0, 18, 33793, 0, 0
zio_data_buf_917504: 917504, 0, 0, 0, 0, 0, 0
zio_buf_1048576: 1048576, 0, 0, 34, 299267, 0, 0
zio_data_buf_1048576: 1048576, 0, 0, 0, 0, 0, 0
zio_buf_1310720: 1310720, 0, 0, 0, 0, 0, 0
zio_data_buf_1310720: 1310720, 0, 0, 0, 0, 0, 0
zio_buf_1572864: 1572864, 0, 0, 0, 0, 0, 0
zio_data_buf_1572864: 1572864, 0, 0, 0, 0, 0, 0
zio_buf_1835008: 1835008, 0, 0, 0, 0, 0, 0
zio_data_buf_1835008: 1835008, 0, 0, 0, 0, 0, 0
zio_buf_2097152: 2097152, 0, 0, 0, 0, 0, 0
zio_data_buf_2097152: 2097152, 0, 0, 0, 0, 0, 0
zio_buf_2621440: 2621440, 0, 0, 0, 0, 0, 0
zio_data_buf_2621440: 2621440, 0, 0, 0, 0, 0, 0
zio_buf_3145728: 3145728, 0, 0, 0, 0, 0, 0
zio_data_buf_3145728: 3145728, 0, 0, 0, 0, 0, 0
zio_buf_3670016: 3670016, 0, 0, 0, 0, 0, 0
zio_data_buf_3670016: 3670016, 0, 0, 0, 0, 0, 0
zio_buf_4194304: 4194304, 0, 0, 0, 0, 0, 0
zio_data_buf_4194304: 4194304, 0, 0, 0, 0, 0, 0
zio_buf_5242880: 5242880, 0, 0, 0, 0, 0, 0
zio_data_buf_5242880: 5242880, 0, 0, 0, 0, 0, 0
zio_buf_6291456: 6291456, 0, 0, 0, 0, 0, 0
zio_data_buf_6291456: 6291456, 0, 0, 0, 0, 0, 0
zio_buf_7340032: 7340032, 0, 0, 0, 0, 0, 0
zio_data_buf_7340032: 7340032, 0, 0, 0, 0, 0, 0
zio_buf_8388608: 8388608, 0, 0, 0, 0, 0, 0
zio_data_buf_8388608: 8388608, 0, 0, 0, 0, 0, 0
zio_buf_10485760: 10485760, 0, 0, 0, 0, 0, 0
zio_data_buf_10485760: 10485760, 0, 0, 0, 0, 0, 0
zio_buf_12582912: 12582912, 0, 0, 0, 0, 0, 0
zio_data_buf_12582912: 12582912, 0, 0, 0, 0, 0, 0
zio_buf_14680064: 14680064, 0, 0, 0, 0, 0, 0
zio_data_buf_14680064: 14680064, 0, 0, 0, 0, 0, 0
zio_buf_16777216: 16777216, 0, 0, 0, 0, 0, 0
zio_data_buf_16777216: 16777216, 0, 0, 0, 0, 0, 0
lz4_ctx: 16384, 0, 0, 47,13646201, 0, 0
abd_chunk: 4096, 0, 6062533, 7608,129128463, 0, 0
sa_cache: 144, 0, 621778, 1274, 2425352, 0, 0
dnode_t: 728, 0, 779201, 3374, 3728924, 0, 0
arc_buf_hdr_t_full: 256, 0, 922135, 69095,14402646, 0, 0
arc_buf_hdr_t_l2only: 96, 0, 0, 0, 0, 0, 0
arc_buf_t: 64, 0, 38564, 81096,21561760, 0, 0
dmu_buf_impl_t: 240, 0, 719348, 61628,21105559, 0, 0
zil_lwb_cache: 320, 0, 10, 1010, 116043, 0, 0
zil_zcw_cache: 80, 0, 0, 2650, 90988, 0, 0
sio_cache: 128, 0, 0, 0, 0, 0, 0
zfs_znode_cache: 272, 0, 621778, 718, 2425012, 0, 0
pipe: 760, 0, 24, 391, 899076, 0, 0
ksiginfo: 112, 0, 466, 5134, 2051786, 0, 0
itimer: 352, 0, 0, 33, 2, 0, 0
KNOTE: 160, 0, 39, 1961,32692147, 0, 0
socket: 872, 65536, 82, 446, 811406, 0, 0
unpcb: 256, 65550, 57, 1833, 265881, 0, 0
ipq: 56, 4118, 0, 0, 0, 0, 0
udp_inpcb: 488, 65536, 10, 902, 468540, 0, 0
udpcb: 32, 65625, 10, 5615, 468540, 0, 0
tcp_inpcb: 488, 65536, 14, 618, 76937, 0, 0
tcpcb: 976, 65536, 14, 330, 76937, 0, 0
tcptw: 88, 13140, 0, 3780, 26469, 0, 0
syncache: 168, 15364, 0, 1472, 42801, 0, 0
hostcache: 96, 15375, 12, 1259, 5095, 0, 0
sackhole: 32, 0, 0, 2125, 25188, 0, 0
tfo: 4, 0, 0, 0, 0, 0, 0
tfo_ccache_entries: 80, 0, 0, 0, 0, 0, 0
tcpreass: 48, 8217, 0, 2075, 38014, 0, 0
tcp_log: 400, 1000000, 0, 0, 0, 0, 0
tcp_log_bucket: 144, 0, 0, 0, 0, 0, 0
tcp_log_node: 120, 0, 0, 0, 0, 0, 0
sctp_ep: 1280, 65538, 0, 0, 0, 0, 0
sctp_asoc: 2408, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80012, 0, 0, 3, 0, 0
sctp_raddr: 736, 80000, 0, 0, 0, 0, 0
sctp_chunk: 152, 400010, 0, 0, 0, 0, 0
sctp_readq: 152, 400010, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400015, 0, 0, 0, 0, 0
sctp_asconf: 40, 400000, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0
udplite_inpcb: 488, 65536, 0, 0, 0, 0, 0
ripcb: 488, 65536, 0, 320, 39, 0, 0
IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0
rtentry: 208, 0, 11, 122, 13, 0, 0
selfd: 64, 0, 865, 6575,151996806, 0, 0
swpctrie: 144, 4082643, 0, 0, 0, 0, 0
swblk: 136, 4082620, 0, 0, 0, 0, 0
pf mtags: 48, 0, 0, 0, 0, 0, 0
pf states: 296, 100009, 0, 0, 0, 0, 0
pf state keys: 88, 0, 0, 0, 0, 0, 0
pf source nodes: 136, 10005, 0, 0, 0, 0, 0
pf table entries: 160, 0, 0, 0, 0, 0, 0
pf table counters: 64, 0, 0, 0, 0, 0, 0
pf frags: 112, 0, 0, 0, 0, 0, 0
pf frag entries: 40, 5000, 0, 0, 0, 0, 0
pf state scrubs: 40, 0, 0, 0, 0, 0, 0
FFS inode: 160, 0, 0, 0, 0, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 0, 0, 0, 0, 0
$ sysctl dev.igb.0
dev.igb.0.interrupts.rx_overrun: 0
dev.igb.0.interrupts.rx_desc_min_thresh: 0
dev.igb.0.interrupts.tx_queue_min_thresh: 0
dev.igb.0.interrupts.tx_queue_empty: 0
dev.igb.0.interrupts.tx_abs_timer: 0
dev.igb.0.interrupts.tx_pkt_timer: 0
dev.igb.0.interrupts.rx_abs_timer: 0
dev.igb.0.interrupts.rx_pkt_timer: 0
dev.igb.0.interrupts.asserts: 0
dev.igb.0.mac_stats.tso_ctx_fail: 0
dev.igb.0.mac_stats.tso_txd: 0
dev.igb.0.mac_stats.tx_frames_1024_1522: 0
dev.igb.0.mac_stats.tx_frames_512_1023: 0
dev.igb.0.mac_stats.tx_frames_256_511: 0
dev.igb.0.mac_stats.tx_frames_128_255: 0
dev.igb.0.mac_stats.tx_frames_65_127: 0
dev.igb.0.mac_stats.tx_frames_64: 0
dev.igb.0.mac_stats.mcast_pkts_txd: 0
dev.igb.0.mac_stats.bcast_pkts_txd: 0
dev.igb.0.mac_stats.good_pkts_txd: 0
dev.igb.0.mac_stats.total_pkts_txd: 0
dev.igb.0.mac_stats.good_octets_txd: 0
dev.igb.0.mac_stats.good_octets_recvd: 0
dev.igb.0.mac_stats.rx_frames_1024_1522: 0
dev.igb.0.mac_stats.rx_frames_512_1023: 0
dev.igb.0.mac_stats.rx_frames_256_511: 0
dev.igb.0.mac_stats.rx_frames_128_255: 0
dev.igb.0.mac_stats.rx_frames_65_127: 0
dev.igb.0.mac_stats.rx_frames_64: 0
dev.igb.0.mac_stats.mcast_pkts_recvd: 0
dev.igb.0.mac_stats.bcast_pkts_recvd: 0
dev.igb.0.mac_stats.good_pkts_recvd: 0
dev.igb.0.mac_stats.total_pkts_recvd: 0
dev.igb.0.mac_stats.xoff_txd: 0
dev.igb.0.mac_stats.xoff_recvd: 0
dev.igb.0.mac_stats.xon_txd: 0
dev.igb.0.mac_stats.xon_recvd: 0
dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.recv_errs: 0
dev.igb.0.mac_stats.recv_jabber: 0
dev.igb.0.mac_stats.recv_oversize: 0
dev.igb.0.mac_stats.recv_fragmented: 0
dev.igb.0.mac_stats.recv_undersize: 0
dev.igb.0.mac_stats.recv_no_buff: 0
dev.igb.0.mac_stats.missed_packets: 0
dev.igb.0.mac_stats.defer_count: 0
dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.collision_count: 0
dev.igb.0.mac_stats.late_coll: 0
dev.igb.0.mac_stats.multiple_coll: 0
dev.igb.0.mac_stats.single_coll: 0
dev.igb.0.mac_stats.excess_coll: 0
dev.igb.0.queue_rx_7.rx_irq: 0
dev.igb.0.queue_rx_7.rxd_tail: 0
dev.igb.0.queue_rx_7.rxd_head: 0
dev.igb.0.queue_rx_6.rx_irq: 0
dev.igb.0.queue_rx_6.rxd_tail: 0
dev.igb.0.queue_rx_6.rxd_head: 0
dev.igb.0.queue_rx_5.rx_irq: 0
dev.igb.0.queue_rx_5.rxd_tail: 0
dev.igb.0.queue_rx_5.rxd_head: 0
dev.igb.0.queue_rx_4.rx_irq: 0
dev.igb.0.queue_rx_4.rxd_tail: 0
dev.igb.0.queue_rx_4.rxd_head: 0
dev.igb.0.queue_rx_3.rx_irq: 0
dev.igb.0.queue_rx_3.rxd_tail: 0
dev.igb.0.queue_rx_3.rxd_head: 0
dev.igb.0.queue_rx_2.rx_irq: 0
dev.igb.0.queue_rx_2.rxd_tail: 0
dev.igb.0.queue_rx_2.rxd_head: 0
dev.igb.0.queue_rx_1.rx_irq: 0
dev.igb.0.queue_rx_1.rxd_tail: 0
dev.igb.0.queue_rx_1.rxd_head: 0
dev.igb.0.queue_rx_0.rx_irq: 0
dev.igb.0.queue_rx_0.rxd_tail: 0
dev.igb.0.queue_rx_0.rxd_head: 0
dev.igb.0.queue_tx_7.tx_irq: 0
dev.igb.0.queue_tx_7.txd_tail: 0
dev.igb.0.queue_tx_7.txd_head: 0
dev.igb.0.queue_tx_6.tx_irq: 0
dev.igb.0.queue_tx_6.txd_tail: 0
dev.igb.0.queue_tx_6.txd_head: 0
dev.igb.0.queue_tx_5.tx_irq: 0
dev.igb.0.queue_tx_5.txd_tail: 0
dev.igb.0.queue_tx_5.txd_head: 0
dev.igb.0.queue_tx_4.tx_irq: 0
dev.igb.0.queue_tx_4.txd_tail: 0
dev.igb.0.queue_tx_4.txd_head: 0
dev.igb.0.queue_tx_3.tx_irq: 0
dev.igb.0.queue_tx_3.txd_tail: 0
dev.igb.0.queue_tx_3.txd_head: 0
dev.igb.0.queue_tx_2.tx_irq: 0
dev.igb.0.queue_tx_2.txd_tail: 0
dev.igb.0.queue_tx_2.txd_head: 0
dev.igb.0.queue_tx_1.tx_irq: 0
dev.igb.0.queue_tx_1.txd_tail: 0
dev.igb.0.queue_tx_1.txd_head: 0
dev.igb.0.queue_tx_0.tx_irq: 0
dev.igb.0.queue_tx_0.txd_tail: 0
dev.igb.0.queue_tx_0.txd_head: 0
dev.igb.0.fc_low_water: 4294965240
dev.igb.0.fc_high_water: 4294965248
dev.igb.0.rx_control: 0
dev.igb.0.device_control: 12583489
dev.igb.0.watchdog_timeouts: 0
dev.igb.0.rx_overruns: 0
dev.igb.0.tx_dma_fail: 0
dev.igb.0.mbuf_defrag_fail: 0
dev.igb.0.link_irq: 0
dev.igb.0.dropped: 0
dev.igb.0.eee_control: 1
dev.igb.0.itr: 488
dev.igb.0.tx_abs_int_delay: 66
dev.igb.0.rx_abs_int_delay: 66
dev.igb.0.tx_int_delay: 66
dev.igb.0.rx_int_delay: 0
dev.igb.0.rs_dump: 0
dev.igb.0.reg_dump: General Registers
CTRL 00c00241
STATUS 00080780
CTRL_EXIT 10180c00
Interrupt Registers
ICR 00000000
RX Registers
RCTL 00000000
RDLEN 00000000
RDH 00000000
RDT 00000000
RXDCTL 00010000
RDBAL 00000000
RDBAH 00000000
TX Registers
TCTL a40400f8
TDBAL 00000000
TDBAH 00000000
TDLEN 00000000
TDH 00000000
TDT 00000000
TXDCTL 00000000
TDFH 40020002
TDFT 00000000
TDFHS 00000000
TDFPC 00000000
dev.igb.0.fc: 3
dev.igb.0.debug: -1
dev.igb.0.nvm: -1
dev.igb.0.iflib.rxq7.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq7.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq7.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq6.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq6.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq6.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq5.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq5.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq5.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq4.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq4.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq4.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq3.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq3.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq3.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq2.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq2.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq2.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq1.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq1.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq1.rxq_fl0.pidx: 0
dev.igb.0.iflib.rxq0.rxq_fl0.credits: 0
dev.igb.0.iflib.rxq0.rxq_fl0.cidx: 0
dev.igb.0.iflib.rxq0.rxq_fl0.pidx: 0
dev.igb.0.iflib.txq7.r_abdications: 0
dev.igb.0.iflib.txq7.r_restarts: 0
dev.igb.0.iflib.txq7.r_stalls: 0
dev.igb.0.iflib.txq7.r_starts: 0
dev.igb.0.iflib.txq7.r_drops: 0
dev.igb.0.iflib.txq7.r_enqueues: 0
dev.igb.0.iflib.txq7.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq7.txq_cleaned: 0
dev.igb.0.iflib.txq7.txq_processed: 0
dev.igb.0.iflib.txq7.txq_in_use: 0
dev.igb.0.iflib.txq7.txq_cidx_processed: 0
dev.igb.0.iflib.txq7.txq_cidx: 0
dev.igb.0.iflib.txq7.txq_pidx: 0
dev.igb.0.iflib.txq7.no_tx_dma_setup: 0
dev.igb.0.iflib.txq7.txd_encap_efbig: 0
dev.igb.0.iflib.txq7.tx_map_failed: 0
dev.igb.0.iflib.txq7.no_desc_avail: 0
dev.igb.0.iflib.txq7.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq7.m_pullups: 0
dev.igb.0.iflib.txq7.mbuf_defrag: 0
dev.igb.0.iflib.txq6.r_abdications: 0
dev.igb.0.iflib.txq6.r_restarts: 0
dev.igb.0.iflib.txq6.r_stalls: 0
dev.igb.0.iflib.txq6.r_starts: 0
dev.igb.0.iflib.txq6.r_drops: 0
dev.igb.0.iflib.txq6.r_enqueues: 0
dev.igb.0.iflib.txq6.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq6.txq_cleaned: 0
dev.igb.0.iflib.txq6.txq_processed: 0
dev.igb.0.iflib.txq6.txq_in_use: 0
dev.igb.0.iflib.txq6.txq_cidx_processed: 0
dev.igb.0.iflib.txq6.txq_cidx: 0
dev.igb.0.iflib.txq6.txq_pidx: 0
dev.igb.0.iflib.txq6.no_tx_dma_setup: 0
dev.igb.0.iflib.txq6.txd_encap_efbig: 0
dev.igb.0.iflib.txq6.tx_map_failed: 0
dev.igb.0.iflib.txq6.no_desc_avail: 0
dev.igb.0.iflib.txq6.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq6.m_pullups: 0
dev.igb.0.iflib.txq6.mbuf_defrag: 0
dev.igb.0.iflib.txq5.r_abdications: 0
dev.igb.0.iflib.txq5.r_restarts: 0
dev.igb.0.iflib.txq5.r_stalls: 0
dev.igb.0.iflib.txq5.r_starts: 0
dev.igb.0.iflib.txq5.r_drops: 0
dev.igb.0.iflib.txq5.r_enqueues: 0
dev.igb.0.iflib.txq5.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq5.txq_cleaned: 0
dev.igb.0.iflib.txq5.txq_processed: 0
dev.igb.0.iflib.txq5.txq_in_use: 0
dev.igb.0.iflib.txq5.txq_cidx_processed: 0
dev.igb.0.iflib.txq5.txq_cidx: 0
dev.igb.0.iflib.txq5.txq_pidx: 0
dev.igb.0.iflib.txq5.no_tx_dma_setup: 0
dev.igb.0.iflib.txq5.txd_encap_efbig: 0
dev.igb.0.iflib.txq5.tx_map_failed: 0
dev.igb.0.iflib.txq5.no_desc_avail: 0
dev.igb.0.iflib.txq5.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq5.m_pullups: 0
dev.igb.0.iflib.txq5.mbuf_defrag: 0
dev.igb.0.iflib.txq4.r_abdications: 0
dev.igb.0.iflib.txq4.r_restarts: 0
dev.igb.0.iflib.txq4.r_stalls: 0
dev.igb.0.iflib.txq4.r_starts: 0
dev.igb.0.iflib.txq4.r_drops: 0
dev.igb.0.iflib.txq4.r_enqueues: 0
dev.igb.0.iflib.txq4.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq4.txq_cleaned: 0
dev.igb.0.iflib.txq4.txq_processed: 0
dev.igb.0.iflib.txq4.txq_in_use: 0
dev.igb.0.iflib.txq4.txq_cidx_processed: 0
dev.igb.0.iflib.txq4.txq_cidx: 0
dev.igb.0.iflib.txq4.txq_pidx: 0
dev.igb.0.iflib.txq4.no_tx_dma_setup: 0
dev.igb.0.iflib.txq4.txd_encap_efbig: 0
dev.igb.0.iflib.txq4.tx_map_failed: 0
dev.igb.0.iflib.txq4.no_desc_avail: 0
dev.igb.0.iflib.txq4.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq4.m_pullups: 0
dev.igb.0.iflib.txq4.mbuf_defrag: 0
dev.igb.0.iflib.txq3.r_abdications: 0
dev.igb.0.iflib.txq3.r_restarts: 0
dev.igb.0.iflib.txq3.r_stalls: 0
dev.igb.0.iflib.txq3.r_starts: 0
dev.igb.0.iflib.txq3.r_drops: 0
dev.igb.0.iflib.txq3.r_enqueues: 0
dev.igb.0.iflib.txq3.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq3.txq_cleaned: 0
dev.igb.0.iflib.txq3.txq_processed: 0
dev.igb.0.iflib.txq3.txq_in_use: 0
dev.igb.0.iflib.txq3.txq_cidx_processed: 0
dev.igb.0.iflib.txq3.txq_cidx: 0
dev.igb.0.iflib.txq3.txq_pidx: 0
dev.igb.0.iflib.txq3.no_tx_dma_setup: 0
dev.igb.0.iflib.txq3.txd_encap_efbig: 0
dev.igb.0.iflib.txq3.tx_map_failed: 0
dev.igb.0.iflib.txq3.no_desc_avail: 0
dev.igb.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq3.m_pullups: 0
dev.igb.0.iflib.txq3.mbuf_defrag: 0
dev.igb.0.iflib.txq2.r_abdications: 0
dev.igb.0.iflib.txq2.r_restarts: 0
dev.igb.0.iflib.txq2.r_stalls: 0
dev.igb.0.iflib.txq2.r_starts: 0
dev.igb.0.iflib.txq2.r_drops: 0
dev.igb.0.iflib.txq2.r_enqueues: 0
dev.igb.0.iflib.txq2.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq2.txq_cleaned: 0
dev.igb.0.iflib.txq2.txq_processed: 0
dev.igb.0.iflib.txq2.txq_in_use: 0
dev.igb.0.iflib.txq2.txq_cidx_processed: 0
dev.igb.0.iflib.txq2.txq_cidx: 0
dev.igb.0.iflib.txq2.txq_pidx: 0
dev.igb.0.iflib.txq2.no_tx_dma_setup: 0
dev.igb.0.iflib.txq2.txd_encap_efbig: 0
dev.igb.0.iflib.txq2.tx_map_failed: 0
dev.igb.0.iflib.txq2.no_desc_avail: 0
dev.igb.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq2.m_pullups: 0
dev.igb.0.iflib.txq2.mbuf_defrag: 0
dev.igb.0.iflib.txq1.r_abdications: 0
dev.igb.0.iflib.txq1.r_restarts: 0
dev.igb.0.iflib.txq1.r_stalls: 0
dev.igb.0.iflib.txq1.r_starts: 0
dev.igb.0.iflib.txq1.r_drops: 0
dev.igb.0.iflib.txq1.r_enqueues: 0
dev.igb.0.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq1.txq_cleaned: 0
dev.igb.0.iflib.txq1.txq_processed: 0
dev.igb.0.iflib.txq1.txq_in_use: 0
dev.igb.0.iflib.txq1.txq_cidx_processed: 0
dev.igb.0.iflib.txq1.txq_cidx: 0
dev.igb.0.iflib.txq1.txq_pidx: 0
dev.igb.0.iflib.txq1.no_tx_dma_setup: 0
dev.igb.0.iflib.txq1.txd_encap_efbig: 0
dev.igb.0.iflib.txq1.tx_map_failed: 0
dev.igb.0.iflib.txq1.no_desc_avail: 0
dev.igb.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq1.m_pullups: 0
dev.igb.0.iflib.txq1.mbuf_defrag: 0
dev.igb.0.iflib.txq0.r_abdications: 0
dev.igb.0.iflib.txq0.r_restarts: 0
dev.igb.0.iflib.txq0.r_stalls: 0
dev.igb.0.iflib.txq0.r_starts: 0
dev.igb.0.iflib.txq0.r_drops: 0
dev.igb.0.iflib.txq0.r_enqueues: 0
dev.igb.0.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igb.0.iflib.txq0.txq_cleaned: 0
dev.igb.0.iflib.txq0.txq_processed: 0
dev.igb.0.iflib.txq0.txq_in_use: 0
dev.igb.0.iflib.txq0.txq_cidx_processed: 0
dev.igb.0.iflib.txq0.txq_cidx: 0
dev.igb.0.iflib.txq0.txq_pidx: 0
dev.igb.0.iflib.txq0.no_tx_dma_setup: 0
dev.igb.0.iflib.txq0.txd_encap_efbig: 0
dev.igb.0.iflib.txq0.tx_map_failed: 0
dev.igb.0.iflib.txq0.no_desc_avail: 0
dev.igb.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq0.m_pullups: 0
dev.igb.0.iflib.txq0.mbuf_defrag: 0
dev.igb.0.iflib.override_nrxds: 0
dev.igb.0.iflib.override_ntxds: 0
dev.igb.0.iflib.tx_abdicate: 0
dev.igb.0.iflib.rx_budget: 0
dev.igb.0.iflib.disable_msix: 0
dev.igb.0.iflib.override_qs_enable: 0
dev.igb.0.iflib.override_nrxqs: 0
dev.igb.0.iflib.override_ntxqs: 0
dev.igb.0.iflib.driver_version: 7.6.1-k
dev.igb.0.%parent: pci3
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1526 subvendor=0x8086 subdevice=0xa06c class=0x020000
dev.igb.0.%location: slot=0 function=0 dbsf=pci0:6:0:0
dev.igb.0.%driver: igb
dev.igb.0.%desc: Intel(R) PRO/1000 PCI-Express Network Driver
If needed I can reproduce the problem and perhaps run a monitoring program while the issue is occurring to collect useful information to help understand the problem. (In reply to Naveen Nathan from comment #2) First of all it would be great to know that you can reproduce it and if that is the case how you do it. Since no one else has mentioned this yet... The stats suggest to me that you've fragmented the mbuf cluster memory pool. 9K mbuf clusters are known to be a serious problem, see this recent post: http://docs.FreeBSD.org/cgi/mid.cgi?23756.39015.553779.526064 Some net interfaces have a setting that tells them to not use 9K mbuf clusters even if the interface is using 9K jumbo packets. If that exists for this net driver, I'd suggest you try it. For some reason, the stats show large numbers of both 9K and 4K mbuf clusters. (The 4K mbuf clusters aren't nearly as bad w.r.t. fragmentation, but mixing them with the 9K ones seems likely to cause fragmentation.) Alternately, I'd suggest you turn of jumbo packets and try it with ordinary 1500 byte ethernet packets. Hopefully others more conversant with this net driver and the mbuf stats will comment. (In reply to Rick Macklem from comment #4) From which stat value do you conclude that 9k cluster elements where used? I read the line 0/0/0/302414 9k jumbo clusters in use (current/cache/total/max) as none are used. Oops, yea. I read the 4K line as the 9K line... and misread the "max" line as the in use... Ok, this time I think I have read the stats correctly and the VNODE numbers look like they may be high? You could check to see what the value of kern.maxvnodes is? # sysctl -a | fgrep maxvnodes (In reply to Rick Macklem from comment #8) $ sysctl -a | fgrep maxvnodes kern.maxvnodes: 622168 (In reply to Michael Tuexen from comment #3) To reproduce the issue I simply do a large 5+ GB transfer over sftp to the server. After transfer is complete, a few moments later the network stack will seize, and hours later unseize. Well, the stats you posted show that you are near your vnode limit, but I have no idea how that might affect your sftp transfer. You could try increasing kern.maxvnodes and see if it has any effect? I cannot reproduce this, tried multiple times. Also 12.0 is not supported anymore. So I'm closing this, but reopen if this problem persists. |
