RTL8811AU and RTL8821AU usb nic tested on recent main amd64/aarch64, mode 11ac: `git pull --rebase` takes too long and sometimes fails iperf3 -c local.pc -R Connecting to host hp, port 5201 Reverse mode, remote host hp is sending [ 5] local 192.168.1.82 port 43550 connected to 192.168.1.100 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 0.00 Bytes 0.00 bits/sec [ 5] 1.00-2.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 2.06-3.00 sec 128 KBytes 1.11 Mbits/sec [ 5] 3.00-4.03 sec 128 KBytes 1.02 Mbits/sec [ 5] 4.03-5.01 sec 0.00 Bytes 0.00 bits/sec [ 5] 5.01-6.01 sec 256 KBytes 2.10 Mbits/sec [ 5] 6.01-7.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 7.06-8.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 8.06-9.01 sec 0.00 Bytes 0.00 bits/sec [ 5] 9.01-10.01 sec 128 KBytes 1.04 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 768 KBytes 628 Kbits/sec 156 sender [ 5] 0.00-10.01 sec 640 KBytes 524 Kbits/sec receiver iperf Done. relevant /var/log/messages: ... rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf ...
So, I got nuno to add some extra debugging to the usb RX path to see how big that RX buffer was, and it was trying to allocate > 4k buffers and failing. Unfortunately it failed with no counters in the OS, making it difficult to diagnose. I asked them to change m_get2() to m_get3(), and it fixed it. So it's purely an allocation size problem. We're going to need to figure out which mbuf allocation function/path to use here for RX aggregation reassembly/handling, as m_get2() isn't going to cut it.
Created attachment 259889 [details] Debug: show mbuf values % iperf3 -c hp -R Connecting to host hp, port 5201 Reverse mode, remote host hp is sending [ 5] local 192.168.1.82 port 35544 connected to 192.168.1.100 port 5201 rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) [ ID] Interval Transfer Bitrate [ 5] 0.00-1.06 sec 0.00 Bytes 0.00 bits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (4672 bytes) [ 5] 1.06-2.00 sec 0.00 Bytes 0.00 bits/sec [ 5] 2.00-3.03 sec 0.00 Bytes 0.00 bits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (4672 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) [ 5] 3.03-4.01 sec 0.00 Bytes 0.00 bits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (4672 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) [ 5] 4.01-5.05 sec 0.00 Bytes 0.00 bits/sec [ 5] 5.05-6.06 sec 0.00 Bytes 0.00 bits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (4672 bytes) [ 5] 6.06-7.06 sec 128 KBytes 1.05 Mbits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (4672 bytes) [ 5] 7.06-8.06 sec 256 KBytes 2.10 Mbits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf (6196 bytes) rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf Previous tests: 1. netstat -mb && vmstat -z 2. `iperf3 -R` trigger on a fresh boot 3. netstat -mb && vmstat -z ==> FRESH BOOT 1/5594/5595 mbufs in use (current/cache/total) 0/3556/3556/1009905 mbuf clusters in use (current/cache/total/max) 0/2794 mbuf+clusters out of packet secondary zone in use (current/cache) 0/254/254/504952 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/149615 9k jumbo clusters in use (current/cache/total/max) 0/0/0/84158 16k jumbo clusters in use (current/cache/total/max) 0K/9526K/9526K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP XDOM kstack_cache: 16384, 0, 481, 17, 549, 0, 0, 0 buffer arena-40: 4096, 0, 0, 0, 0, 0, 0, 0 buffer arena-81: 8192, 0, 0, 0, 0, 0, 0, 0 buffer arena-12: 12288, 0, 0, 0, 0, 0, 0, 0 buffer arena-16: 16384, 0, 10, 0, 10, 0, 0, 0 buffer arena-20: 20480, 0, 0, 0, 0, 0, 0, 0 buffer arena-24: 24576, 0, 0, 0, 0, 0, 0, 0 buffer arena-28: 28672, 0, 0, 0, 0, 0, 0, 0 buffer arena-32: 32768, 0, 1, 0, 1, 0, 0, 0 buffer arena-36: 36864, 0, 0, 0, 0, 0, 0, 0 buffer arena-40: 40960, 0, 0, 0, 0, 0, 0, 0 buffer arena-45: 45056, 0, 0, 0, 0, 0, 0, 0 buffer arena-49: 49152, 0, 0, 0, 0, 0, 0, 0 buffer arena-53: 53248, 0, 0, 0, 0, 0, 0, 0 buffer arena-57: 57344, 0, 0, 0, 0, 0, 0, 0 buffer arena-61: 61440, 0, 0, 0, 0, 0, 0, 0 buffer arena-65: 65536, 0, 0, 0, 0, 0, 0, 0 buf free cache: 432, 0, 11, 199, 11, 0, 0, 0 vm pgcache: 4096, 0, 0, 0, 0, 0, 0, 0 vm pgcache: 4096, 0, 10162, 893, 133125, 16, 0, 0 vm pgcache: 4096, 0, 45571, 1067, 197085, 0, 0, 0 iwlwifi:bc: 4294967295, 0, 0, 0, 0, 0, 0, 0 UMA Kegs: 384, 0, 166, 7, 166, 0, 0, 0 UMA Zones: 2048, 0, 194, 5, 194, 0, 0, 0 UMA Slabs 0: 80, 0, 37546, 48, 37546, 0, 0, 0 UMA Slabs 1: 112, 0, 11, 24, 11, 0, 0, 0 UMA Hash: 256, 0, 0, 0, 0, 0, 0, 0 2 Bucket: 32, 0, 397, 4769, 50194, 0, 0, 0 4 Bucket: 48, 0, 39, 3069, 168, 0, 0, 0 8 Bucket: 80, 0, 200, 1350, 2212, 0, 0, 0 16 Bucket: 144, 0, 157, 739, 1679, 3, 0, 0 32 Bucket: 256, 0, 329, 556, 1274, 3, 0, 0 64 Bucket: 512, 0, 217, 167, 3246, 110, 0, 0 128 Bucket: 1024, 0, 244, 191, 2849,1615, 0, 0 256 Bucket: 2048, 0, 315, 47, 894, 0, 0, 0 SMR SHARED: 24, 0, 8, 503, 8, 0, 0, 0 SMR CPU: 32, 0, 8, 503, 8, 0, 0, 0 vmem: 1856, 0, 2, 7, 2, 0, 0, 0 vmem btag: 56, 0, 13300, 1448, 13547, 102, 0, 0 VM OBJECT: 264, 0, 1337, 373, 64085, 0, 0, 0 RADIX NODE: 152, 0, 7621, 4180, 177415, 0, 0, 0 KMAP ENTRY: 96, 0, 199, 8, 300, 0, 0, 0 MAP ENTRY: 96, 0, 1164, 1860, 221218, 0, 0, 0 VMSPACE: 584, 0, 29, 279, 4346, 0, 0, 0 fakepg: 104, 0, 435, 345, 721, 0, 0, 0 pcpu-4: 4, 0, 0, 0, 0, 0, 0, 0 pcpu-8: 8, 0, 4436, 1708, 4438, 0, 0, 0 pcpu-16: 16, 0, 48, 2512, 48, 0, 0, 0 pcpu-32: 32, 0, 0, 0, 0, 0, 0, 0 pcpu-64: 64, 0, 509, 1539, 509, 0, 0, 0 malloc-16: 16, 0, 18921, 2499, 381981, 0, 0, 0 malloc-32: 32, 0, 12553, 2441, 81424, 0, 0, 0 malloc-64: 64, 0, 23490, 2970, 152215, 0, 0, 0 malloc-128: 128, 0, 31727, 1567, 434736, 0, 0, 0 malloc-256: 256, 0, 3227, 688, 47753, 0, 0, 0 malloc-384: 384, 0, 3102, 258, 32259, 0, 0, 0 malloc-512: 512, 0, 355, 309, 16813, 0, 0, 0 malloc-1024: 1024, 0, 2843, 149, 31468, 0, 0, 0 malloc-2048: 2048, 0, 894, 122, 12662, 0, 0, 0 malloc-4096: 4096, 0, 1248, 24, 28359, 0, 0, 0 malloc-8192: 8192, 0, 162, 14, 1232, 0, 0, 0 malloc-16384: 16384, 0, 117, 17, 2259, 0, 0, 0 malloc-32768: 32768, 0, 102, 7, 1136, 0, 0, 0 malloc-65536: 65536, 0, 46, 5, 990, 0, 0, 0 SLEEPQUEUE: 80, 0, 545, 1119, 545, 0, 0, 0 kenv: 258, 0, 23, 337, 14955, 0, 0, 0 Files: 80, 0, 105, 1695, 35143, 0, 0, 0 filedesc0: 1072, 0, 48, 141, 4363, 0, 0, 0 PWD: 40, 0, 12, 3119, 200, 0, 0, 0 rl_entry: 40, 0, 0, 0, 0, 0, 0, 0 rangeset pctrie nodes: 152, 0, 0, 0, 0, 0, 0, 0 TURNSTILE: 136, 0, 545, 463, 545, 0, 0, 0 lkpicurr: 168, 0, 6, 3186, 9, 0, 0, 0 lkpimm: 56, 0, 2, 3238, 5, 0, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0, 0 umtx_shm: 88, 0, 0, 0, 0, 0, 0, 0 ertt: 72, 0, 0, 280, 3, 0, 0, 0 ertt_txseginfo: 40, 0, 0, 0, 0, 0, 0, 0 swrbuf: 2360, 192, 0, 0, 0, 0, 0, 0 swwbuf: 2360, 96, 0, 0, 0, 0, 0, 0 nfspbuf: 2360, 192, 0, 0, 0, 0, 0, 0 vnpbuf: 2360, 768, 0, 0, 0, 0, 0, 0 clpbuf: 2360, 192, 0, 0, 0, 0, 0, 0 pbuf: 2360, 16, 0, 1460, 0, 0, 0, 0 FPU_save_area: 2696, 0, 483, 35, 612, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0, 0 PROC: 1376, 0, 47, 96, 4362, 0, 0, 0 PGRP: 120, 0, 26, 1261, 44, 0, 0, 0 THREAD: 1824, 0, 481, 63, 549, 0, 0, 0 cpuset: 200, 0, 33, 655, 33, 0, 0, 0 domainset: 40, 0, 0, 0, 0, 0, 0, 0 pkru ranges: 24, 0, 0, 0, 0, 0, 0, 0 audit_record: 1280, 0, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 0, 2794, 724, 0, 0, 0 mbuf: 256,6463393, 1, 2800, 705, 0, 0, 0 mbuf_cluster: 2048,1009905, 2794, 762, 2803, 0, 0, 0 mbuf_jumbo_page: 4096, 504952, 0, 254, 2, 0, 0, 0 mbuf_jumbo_9k: 9216, 149615, 0, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 84158, 0, 0, 0, 0, 0, 0 epoch_record pcpu: 256, 0, 4, 60, 4, 0, 0, 0 ttyinq: 160, 0, 120, 505, 420, 0, 0, 0 ttyoutq: 256, 0, 64, 566, 224, 0, 0, 0 skbuff: 1808, 0, 0, 0, 0, 0, 0, 0 IOMMU_MAP_ENTRY: 112, 0, 0, 0, 0, 0, 0, 0 DEVCTL: 1024, 1000, 0, 800, 752, 0, 0, 0 g_bio: 376, 0, 0, 480, 14032, 0, 0, 0 linux_dma_pctrie: 152, 0, 0, 0, 0, 0, 0, 0 linux_dma_object: 32, 0, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 168, 0, 0, 0, 0 cryptop: 280, 0, 0, 0, 0, 0, 0, 0 taskq_zone: 160, 0, 0, 945, 1510, 0, 0, 0 VNODE: 440, 0, 933, 363, 1409, 0, 0, 0 BUF TRIE: 152, 0, 5, 104593, 5, 0, 0, 0 S VFS Cache: 104, 0, 1359, 669, 1446, 0, 0, 0 STS VFS Cache: 144, 0, 0, 0, 0, 0, 0, 0 L VFS Cache: 320, 0, 0, 36, 1, 0, 0, 0 LTS VFS Cache: 360, 0, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 192, 46690, 0, 0, 0 NCLNODE: 608, 0, 0, 0, 0, 0, 0, 0 TMPFS node: 240, 0, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 0, 0, 0, 0, 0, 0 pipe: 736, 0, 2, 174, 2969, 0, 0, 0 AIO: 208, 0, 0, 0, 0, 0, 0, 0 AIOCB: 552, 0, 0, 0, 0, 0, 0, 0 AIOLIO: 272, 0, 0, 0, 0, 0, 0, 0 Mountpoints: 2816, 0, 29, 19, 29, 0, 0, 0 zfs_btree_leaf_cache: 4096, 0, 1648, 56, 2122, 0, 0, 0 metaslab_alloc_trace_cache:72, 0, 0, 0, 0, 0, 0, 0 brt_entry_cache: 168, 0, 0, 0, 0, 0, 0, 0 ddt_cache: 29240, 0, 5, 1, 5, 0, 0, 0 ddt_entry_flat_cache: 184, 0, 0, 0, 0, 0, 0, 0 ddt_entry_trad_cache: 368, 0, 0, 0, 0, 0, 0, 0 ddt_log_entry_flat_cache:144, 0, 0, 0, 0, 0, 0, 0 ddt_log_entry_trad_cache:328, 0, 0, 0, 0, 0, 0, 0 zio_cache: 1216, 0, 16, 304, 102017, 0, 0, 0 zio_link_cache: 48, 0, 0, 4116, 40478, 0, 0, 0 zio_buf_comb_512: 512, 0, 134, 618, 56623, 0, 0, 0 zio_buf_comb_1024: 1024, 0, 105, 135, 202, 0, 0, 0 zio_buf_comb_1536: 1536, 0, 60, 45, 73, 0, 0, 0 zio_buf_comb_2048: 2048, 0, 42, 62, 44, 0, 0, 0 zio_buf_comb_3072: 3072, 0, 37, 31, 43, 0, 0, 0 zio_buf_comb_4096: 4096, 0, 1281, 21, 2763, 0, 0, 0 zio_buf_comb_6144: 6144, 0, 25, 17, 88, 0, 0, 0 zio_buf_comb_8192: 8192, 0, 15, 14, 169, 0, 0, 0 zio_buf_comb_12288: 12288, 0, 16, 21, 134, 0, 0, 0 zio_buf_comb_16384: 16384, 0, 578, 27, 1222, 0, 0, 0 zio_buf_comb_24576: 24576, 0, 14, 24, 95, 0, 0, 0 zio_buf_comb_32768: 32768, 0, 16, 10, 74, 0, 0, 0 zio_buf_comb_49152: 49152, 0, 15, 18, 247, 0, 0, 0 zio_buf_comb_65536: 65536, 0, 7, 17, 119, 0, 0, 0 zio_buf_comb_98304: 98304, 0, 5, 20, 299, 0, 0, 0 zio_buf_comb_131072: 131072, 0, 291, 36, 1662, 0, 0, 0 zio_buf_comb_196608: 196608, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_262144: 262144, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_393216: 393216, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_524288: 524288, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_786432: 786432, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_1048576:1048576, 0, 0, 1, 15, 0, 0, 0 zio_buf_comb_1572864:1572864, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_2097152:2097152, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_3145728:3145728, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_4194304:4194304, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_6291456:6291456, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_8388608:8388608, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_12582912:12582912, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_16777216:16777216, 0, 0, 0, 0, 0, 0, 0 abd_chunk: 4096, 0, 26560, 20, 88729, 0, 0, 0 sa_cache: 264, 0, 844, 206, 883, 0, 0, 0 dnode_t: 776, 0, 1510, 110, 3659, 0, 0, 0 arc_buf_hdr_t_full: 232, 0, 2997, 420, 31112, 0, 0, 0 arc_buf_hdr_t_l2only: 96, 0, 0, 0, 0, 0, 0, 0 arc_buf_t: 32, 0, 1314, 2340, 31894, 0, 0, 0 dmu_buf_impl_t: 280, 0, 2419, 283, 32336, 0, 0, 0 dbuf_dirty_record_t: 368, 0, 0, 484, 1027, 0, 0, 0 zil_lwb_cache: 376, 0, 2, 118, 8, 0, 0, 0 zil_zcw_cache: 80, 0, 0, 650, 7, 0, 0, 0 sio_cache_0: 136, 0, 0, 0, 0, 0, 0, 0 sio_cache_1: 152, 0, 0, 0, 0, 0, 0, 0 sio_cache_2: 168, 0, 0, 0, 0, 0, 0, 0 zap_name: 328, 0, 0, 360, 135610, 0, 0, 0 zap_attr_cache: 288, 0, 0, 364, 988, 0, 0, 0 zap_name_long: 1096, 0, 0, 0, 0, 0, 0, 0 zap_attr_long_cache: 1056, 0, 0, 98, 4263, 0, 0, 0 zfs_znode_cache: 464, 0, 844, 236, 883, 0, 0, 0 ksiginfo: 112, 0, 87, 1425, 181, 0, 0, 0 itimer: 352, 0, 0, 0, 0, 0, 0, 0 KNOTE: 160, 0, 14, 111, 66, 0, 0, 0 socket: 1024, 519607, 34, 3014, 6635, 0, 0, 0 IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0, 0 unpcb: 320, 519607, 20, 3028, 6442, 0, 0, 0 ripcb: 376, 0, 1, 29, 1, 0, 0, 0 tcp_inpcb: 1304, 0, 0, 18, 3, 0, 0, 0 udp_inpcb: 408, 0, 9, 324, 106, 0, 0, 0 udplite_inpcb: 408, 0, 0, 0, 0, 0, 0, 0 routing nhops: 256, 0, 8, 307, 13, 0, 0, 0 tcpreass: 48, 63119, 0, 0, 0, 0, 0, 0 tcp_log: 408,5000000, 0, 0, 0, 0, 0, 0 tcp_log_id_bucket: 176, 0, 0, 0, 0, 0, 0, 0 tcp_log_id_node: 120, 0, 0, 0, 0, 0, 0, 0 rtentry: 168, 0, 11, 301, 13, 0, 0, 0 ipq: 56, 31559, 0, 0, 0, 0, 0, 0 syncache: 168, 15360, 0, 0, 0, 0, 0, 0 hostcache: 64, 15360, 0, 0, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0, 0 tfo: 4, 0, 0, 0, 0, 0, 0, 0 tfo_ccache_entries: 80, 0, 0, 0, 0, 0, 0, 0 ada_ccb: 272, 0, 0, 0, 0, 0, 0, 0 da_ccb: 536, 0, 0, 70, 225, 0, 0, 0 swpctrie: 152,2021292, 0, 0, 0, 0, 0, 0 swblk: 136,2021271, 0, 0, 0, 0, 0, 0 drm_buddy_block: 72, 0, 0, 0, 0, 0, 0, 0 drm_sched_fence: 184, 0, 0, 0, 0, 0, 0, 0 mqnode: 416, 0, 3, 33, 3, 0, 0, 0 mqueue: 248, 0, 0, 0, 0, 0, 0, 0 mvdata: 64, 0, 0, 0, 0, 0, 0, 0 mqnotifier: 216, 0, 0, 0, 0, 0, 0, 0 nvidia_stack_t: 12288, 0, 5, 5, 773, 0, 0, 0 fuse_ticket: 184, 0, 0, 0, 0, 0, 0, 0 iwl_cmd_pool:iwlwifi0: 60, 0, 0, 0, 0, 0, 0, 0 NetGraph items: 72, 4096, 0, 704, 60, 0, 0, 0 NetGraph data items: 72, 4096, 0, 864, 54, 0, 0, 0 bridge_rtnode: 88, 0, 0, 0, 0, 0, 0, 0 ==> IPERF % iperf3 -c hp -R Connecting to host hp, port 5201 Reverse mode, remote host hp is sending [ 5] local 192.168.1.82 port 12202 connected to 192.168.1.100 port 5201 rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf [ ID] Interval Transfer Bitrate [ 5] 0.00-1.06 sec 0.00 Bytes 0.00 bits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf [ 5] 1.06-2.06 sec 0.00 Bytes 0.00 bits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf [ 5] 2.06-3.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 3.06-4.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 4.06-5.01 sec 0.00 Bytes 0.00 bits/sec rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf rtwn0: rtwn_rx_copy_to_mbuf: could not allocate RX mbuf [ 5] 5.01-6.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 6.06-7.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 7.06-8.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 8.06-9.06 sec 0.00 Bytes 0.00 bits/sec [ 5] 9.06-10.06 sec 0.00 Bytes 0.00 bits/sec - - - - - - - - - - - - ==> AFTER 1/6359/6360 mbufs in use (current/cache/total) 0/3810/3810/1009905 mbuf clusters in use (current/cache/total/max) 0/3048 mbuf+clusters out of packet secondary zone in use (current/cache) 0/762/762/504952 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/149615 9k jumbo clusters in use (current/cache/total/max) 0/0/0/84158 16k jumbo clusters in use (current/cache/total/max) 0K/12257K/12258K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP XDOM kstack_cache: 16384, 0, 480, 18, 550, 0, 0, 0 buffer arena-40: 4096, 0, 0, 0, 0, 0, 0, 0 buffer arena-81: 8192, 0, 0, 0, 0, 0, 0, 0 buffer arena-12: 12288, 0, 0, 0, 0, 0, 0, 0 buffer arena-16: 16384, 0, 10, 0, 10, 0, 0, 0 buffer arena-20: 20480, 0, 0, 0, 0, 0, 0, 0 buffer arena-24: 24576, 0, 0, 0, 0, 0, 0, 0 buffer arena-28: 28672, 0, 0, 0, 0, 0, 0, 0 buffer arena-32: 32768, 0, 1, 0, 1, 0, 0, 0 buffer arena-36: 36864, 0, 0, 0, 0, 0, 0, 0 buffer arena-40: 40960, 0, 0, 0, 0, 0, 0, 0 buffer arena-45: 45056, 0, 0, 0, 0, 0, 0, 0 buffer arena-49: 49152, 0, 0, 0, 0, 0, 0, 0 buffer arena-53: 53248, 0, 0, 0, 0, 0, 0, 0 buffer arena-57: 57344, 0, 0, 0, 0, 0, 0, 0 buffer arena-61: 61440, 0, 0, 0, 0, 0, 0, 0 buffer arena-65: 65536, 0, 0, 0, 0, 0, 0, 0 buf free cache: 432, 0, 11, 199, 11, 0, 0, 0 vm pgcache: 4096, 0, 0, 0, 0, 0, 0, 0 vm pgcache: 4096, 0, 10211, 1264, 134553, 16, 0, 0 vm pgcache: 4096, 0, 46478, 1252, 199558, 0, 0, 0 iwlwifi:bc: 4294967295, 0, 0, 0, 0, 0, 0, 0 UMA Kegs: 384, 0, 166, 7, 166, 0, 0, 0 UMA Zones: 2048, 0, 194, 5, 194, 0, 0, 0 UMA Slabs 0: 80, 0, 38429, 15, 38429, 0, 0, 0 UMA Slabs 1: 112, 0, 11, 24, 11, 0, 0, 0 UMA Hash: 256, 0, 0, 0, 0, 0, 0, 0 2 Bucket: 32, 0, 414, 4752, 51039, 0, 0, 0 4 Bucket: 48, 0, 39, 3069, 168, 0, 0, 0 8 Bucket: 80, 0, 207, 1343, 2471, 0, 0, 0 16 Bucket: 144, 0, 157, 739, 1682, 3, 0, 0 32 Bucket: 256, 0, 343, 662, 1370, 3, 0, 0 64 Bucket: 512, 0, 254, 162, 3375, 118, 0, 0 128 Bucket: 1024, 0, 258, 177, 2891,1615, 0, 0 256 Bucket: 2048, 0, 331, 47, 924, 0, 0, 0 SMR SHARED: 24, 0, 8, 503, 8, 0, 0, 0 SMR CPU: 32, 0, 8, 503, 8, 0, 0, 0 vmem: 1856, 0, 2, 7, 2, 0, 0, 0 vmem btag: 56, 0, 13333, 1415, 13580, 102, 0, 0 VM OBJECT: 264, 0, 1279, 431, 64575, 0, 0, 0 RADIX NODE: 152, 0, 7556, 4245, 182023, 0, 0, 0 KMAP ENTRY: 96, 0, 199, 8, 300, 0, 0, 0 MAP ENTRY: 96, 0, 1031, 1993, 222995, 0, 0, 0 VMSPACE: 584, 0, 25, 283, 4362, 0, 0, 0 fakepg: 104, 0, 435, 345, 721, 0, 0, 0 pcpu-4: 4, 0, 0, 0, 0, 0, 0, 0 pcpu-8: 8, 0, 4436, 1708, 4438, 0, 0, 0 pcpu-16: 16, 0, 48, 2512, 48, 0, 0, 0 pcpu-32: 32, 0, 0, 0, 0, 0, 0, 0 pcpu-64: 64, 0, 509, 1539, 509, 0, 0, 0 malloc-16: 16, 0, 18918, 2502, 387787, 0, 0, 0 malloc-32: 32, 0, 12555, 2943, 83887, 0, 0, 0 malloc-64: 64, 0, 23475, 3237, 163211, 0, 0, 0 malloc-128: 128, 0, 31801, 1617, 445543, 0, 0, 0 malloc-256: 256, 0, 3252, 783, 52442, 0, 0, 0 malloc-384: 384, 0, 3103, 287, 32746, 0, 0, 0 malloc-512: 512, 0, 323, 341, 17128, 0, 0, 0 malloc-1024: 1024, 0, 2843, 165, 31696, 0, 0, 0 malloc-2048: 2048, 0, 893, 123, 15222, 0, 0, 0 malloc-4096: 4096, 0, 1249, 29, 28782, 0, 0, 0 malloc-8192: 8192, 0, 162, 14, 1232, 0, 0, 0 malloc-16384: 16384, 0, 117, 22, 3078, 0, 0, 0 malloc-32768: 32768, 0, 102, 7, 1136, 0, 0, 0 malloc-65536: 65536, 0, 47, 4, 991, 0, 0, 0 SLEEPQUEUE: 80, 0, 545, 1119, 545, 0, 0, 0 kenv: 258, 0, 23, 337, 14955, 0, 0, 0 Files: 80, 0, 95, 1705, 35503, 0, 0, 0 filedesc0: 1072, 0, 44, 145, 4379, 0, 0, 0 PWD: 40, 0, 12, 3119, 202, 0, 0, 0 rl_entry: 40, 0, 0, 0, 0, 0, 0, 0 rangeset pctrie nodes: 152, 0, 0, 0, 0, 0, 0, 0 TURNSTILE: 136, 0, 545, 463, 545, 0, 0, 0 lkpicurr: 168, 0, 6, 3186, 9, 0, 0, 0 lkpimm: 56, 0, 2, 3238, 5, 0, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0, 0 umtx_shm: 88, 0, 0, 0, 0, 0, 0, 0 ertt: 72, 0, 0, 280, 5, 0, 0, 0 ertt_txseginfo: 40, 0, 0, 606, 8, 0, 0, 0 swrbuf: 2360, 192, 0, 0, 0, 0, 0, 0 swwbuf: 2360, 96, 0, 0, 0, 0, 0, 0 nfspbuf: 2360, 192, 0, 0, 0, 0, 0, 0 vnpbuf: 2360, 768, 0, 0, 0, 0, 0, 0 clpbuf: 2360, 192, 0, 0, 0, 0, 0, 0 pbuf: 2360, 16, 0, 1460, 0, 0, 0, 0 FPU_save_area: 2696, 0, 482, 36, 802, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0, 0 PROC: 1376, 0, 43, 100, 4378, 0, 0, 0 PGRP: 120, 0, 24, 1362, 61, 0, 0, 0 THREAD: 1824, 0, 480, 64, 550, 0, 0, 0 cpuset: 200, 0, 33, 655, 33, 0, 0, 0 domainset: 40, 0, 0, 0, 0, 0, 0, 0 pkru ranges: 24, 0, 0, 0, 0, 0, 0, 0 audit_record: 1280, 0, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 0, 3048, 3558, 0, 0, 0 mbuf: 256,6463393, 1, 3311, 2004, 0, 0, 0 mbuf_cluster: 2048,1009905, 3048, 762, 3057, 0, 0, 0 mbuf_jumbo_page: 4096, 504952, 0, 762, 15, 0, 0, 0 mbuf_jumbo_9k: 9216, 149615, 0, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 84158, 0, 0, 0, 0, 0, 0 epoch_record pcpu: 256, 0, 4, 60, 4, 0, 0, 0 ttyinq: 160, 0, 120, 505, 420, 0, 0, 0 ttyoutq: 256, 0, 64, 566, 224, 0, 0, 0 skbuff: 1808, 0, 0, 0, 0, 0, 0, 0 IOMMU_MAP_ENTRY: 112, 0, 0, 0, 0, 0, 0, 0 DEVCTL: 1024, 1000, 0, 800, 1012, 0, 0, 0 g_bio: 376, 0, 0, 480, 17503, 0, 0, 0 linux_dma_pctrie: 152, 0, 0, 0, 0, 0, 0, 0 linux_dma_object: 32, 0, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 168, 0, 0, 0, 0 cryptop: 280, 0, 0, 0, 0, 0, 0, 0 taskq_zone: 160, 0, 0, 1071, 3055, 0, 0, 0 VNODE: 440, 0, 945, 405, 1507, 0, 0, 0 BUF TRIE: 152, 0, 5, 104593, 5, 0, 0, 0 S VFS Cache: 104, 0, 1391, 637, 1488, 0, 0, 0 STS VFS Cache: 144, 0, 0, 0, 0, 0, 0, 0 L VFS Cache: 320, 0, 0, 36, 1, 0, 0, 0 LTS VFS Cache: 360, 0, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 192, 47261, 0, 0, 0 NCLNODE: 608, 0, 0, 0, 0, 0, 0, 0 TMPFS node: 240, 0, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 0, 0, 0, 0, 0, 0 pipe: 736, 0, 1, 175, 2972, 0, 0, 0 AIO: 208, 0, 0, 0, 0, 0, 0, 0 AIOCB: 552, 0, 0, 0, 0, 0, 0, 0 AIOLIO: 272, 0, 0, 0, 0, 0, 0, 0 Mountpoints: 2816, 0, 29, 19, 29, 0, 0, 0 zfs_btree_leaf_cache: 4096, 0, 1644, 60, 2638, 0, 0, 0 metaslab_alloc_trace_cache:72, 0, 0, 0, 0, 0, 0, 0 brt_entry_cache: 168, 0, 0, 0, 0, 0, 0, 0 ddt_cache: 29240, 0, 5, 1, 5, 0, 0, 0 ddt_entry_flat_cache: 184, 0, 0, 0, 0, 0, 0, 0 ddt_entry_trad_cache: 368, 0, 0, 0, 0, 0, 0, 0 ddt_log_entry_flat_cache:144, 0, 0, 0, 0, 0, 0, 0 ddt_log_entry_trad_cache:328, 0, 0, 0, 0, 0, 0, 0 zio_cache: 1216, 0, 16, 304, 110653, 0, 0, 0 zio_link_cache: 48, 0, 0, 4116, 45797, 0, 0, 0 zio_buf_comb_512: 512, 0, 136, 616, 57515, 0, 0, 0 zio_buf_comb_1024: 1024, 0, 105, 135, 236, 0, 0, 0 zio_buf_comb_1536: 1536, 0, 59, 61, 94, 0, 0, 0 zio_buf_comb_2048: 2048, 0, 42, 70, 50, 0, 0, 0 zio_buf_comb_3072: 3072, 0, 37, 31, 43, 0, 0, 0 zio_buf_comb_4096: 4096, 0, 1308, 18, 3609, 0, 0, 0 zio_buf_comb_6144: 6144, 0, 19, 23, 95, 0, 0, 0 zio_buf_comb_8192: 8192, 0, 12, 24, 203, 0, 0, 0 zio_buf_comb_12288: 12288, 0, 15, 22, 144, 0, 0, 0 zio_buf_comb_16384: 16384, 0, 588, 30, 1580, 0, 0, 0 zio_buf_comb_24576: 24576, 0, 9, 29, 123, 0, 0, 0 zio_buf_comb_32768: 32768, 0, 11, 17, 86, 0, 0, 0 zio_buf_comb_49152: 49152, 0, 9, 24, 265, 0, 0, 0 zio_buf_comb_65536: 65536, 0, 3, 21, 122, 0, 0, 0 zio_buf_comb_98304: 98304, 0, 4, 21, 308, 0, 0, 0 zio_buf_comb_131072: 131072, 0, 298, 31, 2030, 0, 0, 0 zio_buf_comb_196608: 196608, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_262144: 262144, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_393216: 393216, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_524288: 524288, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_786432: 786432, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_1048576:1048576, 0, 0, 1, 15, 0, 0, 0 zio_buf_comb_1572864:1572864, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_2097152:2097152, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_3145728:3145728, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_4194304:4194304, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_6291456:6291456, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_8388608:8388608, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_12582912:12582912, 0, 0, 0, 0, 0, 0, 0 zio_buf_comb_16777216:16777216, 0, 0, 0, 0, 0, 0, 0 abd_chunk: 4096, 0, 26721, 27, 89340, 0, 0, 0 sa_cache: 264, 0, 856, 194, 913, 0, 0, 0 dnode_t: 776, 0, 1540, 110, 3705, 0, 0, 0 arc_buf_hdr_t_full: 232, 0, 3040, 445, 31607, 0, 0, 0 arc_buf_hdr_t_l2only: 96, 0, 0, 0, 0, 0, 0, 0 arc_buf_t: 32, 0, 1215, 2439, 32432, 0, 0, 0 dmu_buf_impl_t: 280, 0, 2333, 369, 32856, 0, 0, 0 dbuf_dirty_record_t: 368, 0, 0, 517, 2241, 0, 0, 0 zil_lwb_cache: 376, 0, 4, 206, 27, 0, 0, 0 zil_zcw_cache: 80, 0, 0, 1050, 20, 0, 0, 0 sio_cache_0: 136, 0, 0, 0, 0, 0, 0, 0 sio_cache_1: 152, 0, 0, 0, 0, 0, 0, 0 sio_cache_2: 168, 0, 0, 0, 0, 0, 0, 0 zap_name: 328, 0, 0, 360, 136781, 0, 0, 0 zap_attr_cache: 288, 0, 0, 364, 988, 0, 0, 0 zap_name_long: 1096, 0, 0, 0, 0, 0, 0, 0 zap_attr_long_cache: 1056, 0, 0, 98, 4394, 0, 0, 0 zfs_znode_cache: 464, 0, 856, 224, 913, 0, 0, 0 ksiginfo: 112, 0, 87, 1425, 361, 0, 0, 0 itimer: 352, 0, 0, 0, 0, 0, 0, 0 KNOTE: 160, 0, 14, 111, 1118, 0, 0, 0 socket: 1024, 519607, 30, 3018, 6638, 0, 0, 0 IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0, 0 unpcb: 320, 519607, 16, 3032, 6443, 0, 0, 0 ripcb: 376, 0, 1, 29, 1, 0, 0, 0 tcp_inpcb: 1304, 0, 0, 18, 5, 0, 0, 0 udp_inpcb: 408, 0, 9, 324, 106, 0, 0, 0 udplite_inpcb: 408, 0, 0, 0, 0, 0, 0, 0 routing nhops: 256, 0, 8, 307, 13, 0, 0, 0 tcpreass: 48, 63119, 0, 840, 4, 0, 0, 0 tcp_log: 408,5000000, 0, 0, 0, 0, 0, 0 tcp_log_id_bucket: 176, 0, 0, 0, 0, 0, 0, 0 tcp_log_id_node: 120, 0, 0, 0, 0, 0, 0, 0 rtentry: 168, 0, 11, 301, 13, 0, 0, 0 ipq: 56, 31559, 0, 0, 0, 0, 0, 0 syncache: 168, 15360, 0, 0, 0, 0, 0, 0 hostcache: 64, 15360, 1, 314, 1, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0, 0 tfo: 4, 0, 0, 0, 0, 0, 0, 0 tfo_ccache_entries: 80, 0, 0, 0, 0, 0, 0, 0 ada_ccb: 272, 0, 0, 0, 0, 0, 0, 0 da_ccb: 536, 0, 0, 70, 225, 0, 0, 0 swpctrie: 152,2021292, 0, 0, 0, 0, 0, 0 swblk: 136,2021271, 0, 0, 0, 0, 0, 0 drm_buddy_block: 72, 0, 0, 0, 0, 0, 0, 0 drm_sched_fence: 184, 0, 0, 0, 0, 0, 0, 0 mqnode: 416, 0, 3, 33, 3, 0, 0, 0 mqueue: 248, 0, 0, 0, 0, 0, 0, 0 mvdata: 64, 0, 0, 0, 0, 0, 0, 0 mqnotifier: 216, 0, 0, 0, 0, 0, 0, 0 nvidia_stack_t: 12288, 0, 5, 6, 1133, 0, 0, 0 fuse_ticket: 184, 0, 0, 0, 0, 0, 0, 0 iwl_cmd_pool:iwlwifi0: 60, 0, 0, 0, 0, 0, 0, 0 NetGraph items: 72, 4096, 0, 704, 60, 0, 0, 0 NetGraph data items: 72, 4096, 0, 864, 54, 0, 0, 0 bridge_rtnode: 88, 0, 0, 0, 0, 0, 0, 0
Created attachment 259890 [details] Fix problem currently being tested on amd64/aarch64 OK iperf3 -R is happy, and git pull --rebase completed OK and fast: Connecting to host hp, port 5201 Reverse mode, remote host hp is sending [ 5] local 192.168.1.82 port 39601 connected to 192.168.1.100 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.06 sec 6.00 MBytes 47.4 Mbits/sec [ 5] 1.06-2.06 sec 15.9 MBytes 134 Mbits/sec [ 5] 2.06-3.06 sec 16.1 MBytes 135 Mbits/sec [ 5] 3.06-4.06 sec 16.4 MBytes 137 Mbits/sec [ 5] 4.06-5.00 sec 15.0 MBytes 134 Mbits/sec [ 5] 5.00-6.06 sec 16.4 MBytes 129 Mbits/sec [ 5] 6.06-7.06 sec 15.2 MBytes 128 Mbits/sec [ 5] 7.06-8.06 sec 14.8 MBytes 124 Mbits/sec [ 5] 8.06-9.06 sec 15.8 MBytes 132 Mbits/sec [ 5] 9.06-10.06 sec 16.1 MBytes 135 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.08 sec 148 MBytes 123 Mbits/sec 0 sender [ 5] 0.00-10.06 sec 148 MBytes 123 Mbits/sec receiver iperf Done.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=28e89934a9bec3eca08f4d291a5699485c29d203 commit 28e89934a9bec3eca08f4d291a5699485c29d203 Author: Adrian Chadd <adrian@FreeBSD.org> AuthorDate: 2025-04-27 23:50:51 +0000 Commit: Adrian Chadd <adrian@FreeBSD.org> CommitDate: 2025-05-01 03:01:05 +0000 rtwn: fix mbuf allocation errors in USB RX path for > 4 KiB frames We can and do receive > 4 KiB frames in the RX path (A-MSDU frames can be up to 11KiB.) At least one user has reported seeing this and having it break their traffic flows. Use m_get3() to try and grab an mbuf jumbo cluster. This may not be the best permanent solution, but it at least will fail for frame sizes we expect to see up and including the largest A-MPDU frame (11Kib) and keep a counter if it can't allocate, versus just returning NULL because it's too large (and not keeping counters.) Differential Revision: https://reviews.freebsd.org/D50049 PR: kern/286366 Reviewed by: bz sys/dev/rtwn/usb/rtwn_usb_rx.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)