Created attachment 223464 [details] netstat -m Server the same is in bug #254419, running on 10gbit ix with nginx sendfile = off. # sysctl net.inet.tcp.functions_available net.inet.tcp.functions_available: Stack D Alias PCB count freebsd freebsd 61 rack rack 0 bbr * bbr 122508 After some hours network is hang. Works only local console (ikvm). In /var/log/messages: Mar 21 07:57:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached Mar 21 08:02:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached Mar 21 08:07:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached Mar 21 08:12:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached Mar 21 08:17:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached Mar 21 08:22:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached Mar 21 08:27:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached netstat -m in attached screenshot Server with the same load and configuration but on FreeBSD-12.2-p4 and net.inet.tcp.functions_default=rack works fine.
Created attachment 223465 [details] top
Created attachment 223466 [details] system tty
Bytes allocated to network in netstat -m is constantly growing. After approx 1 day uptime network is hang with: dmesg: [zone: mbuf] kern.ipc.nmbufs limit reached [zone: mbuf] kern.ipc.nmbufs limit reached [zone: mbuf] kern.ipc.nmbufs limit reached [zone: mbuf] kern.ipc.nmbufs limit reached [zone: mbuf] kern.ipc.nmbufs limit reached [zone: mbuf] kern.ipc.nmbufs limit reached netstat -m: 31994144/57556/32051700 mbufs in use (current/cache/total) 32898/14358/47256/5000000 mbuf clusters in use (current/cache/total/max) 8/10270 mbuf+clusters out of packet secondary zone in use (current/cache) 34079/298/34377/2000000 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/900000 9k jumbo clusters in use (current/cache/total/max) 0/0/0/600000 16k jumbo clusters in use (current/cache/total/max) 8200648K/44297K/8244945K bytes allocated to network (current/cache/total) 383111218/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 27833/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 1681210 sendfile syscalls 1678853 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 12525327 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 2703 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed net.inet.tcp.functions_available: Stack D Alias PCB count freebsd freebsd 10 bbr * bbr 44987 rack rack 0 vmstat -m: Type InUse MemUse Requests Size(s) filedesc 64 17654K 473 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 pwddesc 111 7K 23949 64 sigio 31 2K 33 64 filecaps 0 0K 2314 16,32,64 kdtrace 1048 241K 24983 64,256 kenv 117 13K 117 16,32,64,128,8192 kqueue 204 2260K 24965 64,256,512,2048,4096,8192,16384,32768,65536 proc-args 311 12K 48278 16,32,64,128,256 Fail Points 0 0K 20 1024 hhook 13 4K 13 256 ithread 448 74K 448 32,128,256 prison 8 1K 8 16,32 KTRACE 100 13K 100 128 linker 213 2001K 257 16,32,64,128,256,384,512,1024,2048,4096,8192,16384 lockf 23 3K 2515 64,128 loginclass 3 1K 7 64 devbuf 18525 117840K 20947 16,32,64,128,256,384,512,1024,2048,4096,8192,65536 temp 195 20K 2142283 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536 acpiintr 1 1K 1 64 module 234 30K 234 128 mtx_pool 2 72K 2 8192,65536 osd 134 3K 145 16,32,64,128,256 pmchooks 1 1K 1 128 pmc 2 1K 2 64 session 19 3K 7515 128 proc 3 132K 3 4096,65536 subproc 614 637K 24451 384,4096 cred 53 14K 74359 256 acpica 32007 3114K 4383696 16,32,64,128,256,384,512,1024,2048 acpitask 1 64K 1 65536 acpisem 33 5K 33 128 plimit 15 4K 28343 256 uidinfo 5 17K 6847 128,16384 sendfile 0 0K 4179384 64,128,256,384,512,1024,2048,4096,8192,16384 dumper 2 1K 2 128,512 acpidev 232 15K 232 64 sysctl 1 1K 401 32,64 sysctloid 50151 2825K 50666 16,32,64,128,256,384 sysctltmp 0 0K 34983 16,32,64,256,1024 CAM SIM 14 4K 14 256 CAM XPT 43 5K 97 16,32,128,256,512,1024,2048 tidhash 3 260K 3 8192 callout 32 8192K 32 umtx 2124 266K 2124 128 p1003.1b 1 1K 1 16 SWAP 1 4164K 1 bus 1555 207K 264553 16,32,64,128,256,384,4096 bus-sc 225 2460K 20627 16,32,64,128,256,384,512,1024,2048,4096,8192,16384,32768,65536 CAM DEV 16 32K 38 2048 devstat 12 25K 12 32,4096 eventhandler 142 12K 142 64,128 gtaskqueue 198 44K 198 16,32,256,8192 kobj 132 528K 484 4096 Per-cpu 1 1K 1 32 CAM CCB 0 0K 63419 2048 rman 378 43K 769 16,32,128 sbuf 1 1K 7285 16,32,64,128,256,512,1024,2048,4096,8192,16384 toponodes 84 11K 84 128 CAM path 17 1K 110 32 CAM periph 6 2K 39 16,32,64,128,256 acpi_perf 32 12K 32 384 stack 0 0K 201 256 taskqueue 54 7K 54 16,32,256 terminal 11 3K 11 256 Unitno 22 2K 4932 32,64 vmem 4 448K 37 2048,4096,8192,16384,32768,65536 ioctlops 0 0K 11948 256,384,512,1024,2048,4096 select 12 2K 12853 128 iov 0 0K 1542747788 16,64,128,256,384,512,1024,2048 selfd 41 3K 2383115 64 procdesc 4 1K 21 128 msg 4 30K 4 2048,4096,8192,16384 sem 4 106K 4 2048,4096 shm 1 32K 1 32768 tty 14 14K 18 1024 pts 0 0K 4 256 ktls 1 4K 1 4096 mbuf_tag 0 0K 1013618 32,64 shmfd 1 8K 1 8192 soname 4 1K 256798386 16,32,64,128 pcb 2309 3224K 48094843 16,32,64,1024,2048 ix 22 74K 22 1024,2048,4096 kbdmux 7 22K 7 16,384,1024,2048,16384 LED 40 3K 40 16,128 mfibuf 133 51K 13638 16,32,128,256,384,1024,2048,4096,8192,32768 sr_iov 2 1K 2 384 entropy 1 1K 29218 32,64,4096 UART 6 5K 6 16,1024 CAM queue 30 9K 98 16,32,512 USB 61 97K 69 16,128,256,384,1024,2048,4096,8192,16384 vfscache 3 8384K 3 65536 USBdev 47 10K 47 32,64,128,256,384,512,4096 vtbuf 24 1968K 46 4096 vt 13 41K 13 384,4096,32768 cl_savebuf 0 0K 168291 64,128,256,384 vfs_hash 1 4096K 1 vnodes 1 1K 2 32,256 mount 76 4K 153 16,32,64,128,256,1024 statfs 0 0K 319 4096 vnodemarker 3 2K 60944 512 fadvise 1 1K 5 32 chacha20random 1 8K 1 8192 BPF 12 1026K 14 32,128,384,512 ifnet 9 17K 9 128,2048 ifaddr 90 19K 90 16,32,64,128,256,384,2048,4096 ether_multi 64 6K 104 16,32,64,128 clone 7 1K 7 128 lltable 47 14K 1280 256,384 tun 3 1K 3 32 vlan 13 2K 33 64,128 iflib 402 3951K 418 64,128,256,512,1024,8192,16384,32768 pfil 7 1K 15 64,128 nhops 6 1K 6 128 routetbl 19 4K 19 32,64,128,384,1024 vnet 1 1K 1 64 vnet_data 1 180K 1 vnet_data_free 1 1K 1 32 igmp 8 1K 8 128 in_multi 3 1K 3 256 ip_moptions 2 1K 2 64 tfo_ccache 1 128K 1 hostcache 1 32K 1 32768 LRO 64 1280K 64 8192,32768 tcp_hpts 65 51209K 65 256 DEVFS3 168 42K 205 256 tcpfunc 3 1K 3 64 syncache 1 136K 1 in6_multi 31 4K 31 32,256 ip6_moptions 3 1K 3 32 ip6opt 5 2K 11 32,256 mld 7 1K 7 128 ip6ndp 10 2K 12 64,256 inpcbpolicy 45039 1408K 253441302 32 secasvar 1 1K 1 1024 sahead 1 1K 1 1024 ipsecpolicy 1 1K 1 1024 ipsec-saq 2 2K 2 1024 pf_temp 0 0K 3381 32 pf_hash 5 11524K 5 2048 pf_ifnet 13 5K 57 256,2048 pf_rule 38 24K 38 128,1024 pf_osfp 1191 123K 1191 64,128 crypto 3 5K 3 64,256,4096 DEVFS1 139 139K 166 1024 audit_evclass 236 8K 294 32 pagedep 4 2048K 293568 256 inodedep 4 16384K 1319452 384 bmsafemap 4 32K 469650 256,8192 newblk 4 32768K 4749893 256 indirdep 0 0K 24729 128,32768 freefrag 0 0K 496137 128 freeblks 0 0K 529155 256 freefile 0 0K 534502 64 diradd 0 0K 795038 128 mkdir 0 0K 18 128 dirrem 0 0K 932077 128 newdirblk 0 0K 297 64 freework 4 1K 934622 64,128 jaddref 0 0K 6 128 jremref 0 0K 6 128 jnewblk 0 0K 30 128 jfreefrag 0 0K 3 128 jseg 0 0K 6 128 jsegdep 0 0K 45 64 sbdep 0 0K 6177 64 savedino 0 0K 393583 256 jblocks 2 1K 2 128,256 softdep 4 2K 4 512 ufs_dirhash 1768 738K 380492 16,32,64,128,256,384,512,1024,2048 ufs_quota 1 4096K 1 ufs_mount 25 631K 42 32,64,512,4096,8192,16384,32768 vm_pgdata 1 1K 1 128 DEVFS_RULE 59 21K 59 64,384 UMA 240 5K 243 16,32 DEVFS 19 1K 20 16,128 DEVFSP 2 1K 4 64 pfs_nodes 20 8K 20 384 pfs_vncache 1 64K 1 65536 GEOM 126 22K 1445 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768 raid_data 0 0K 222 32,128,256 pci_link 16 2K 16 64,128 apmdev 1 1K 1 128 intr 4 216K 4 32768,65536 MCA 65 17K 65 128,256,384 cpus 2 1K 2 128 nexusdev 6 1K 6 16 memdesc 1 4K 1 4096 ioat 16 7168K 16 isadev 6 1K 6 128 scsi_cd 0 0K 12 16 CAM dev queue 14 1K 14 64 nvlist 94 9K 94 16,64,128 madt_table 0 0K 2 256,4096 cdev 1 1K 1 256 io_apic 3 6K 3 2048 local_apic 1 32K 1 32768 msi 65 9K 65 128 acpiwmi 2 1K 4 32,64 kstat_data 17 17K 17 1024 solaris 131 65839K 248 16,32,64,128,256,1024,2048,4096,8192,65536 taskq 2 3K 2 1024,2048
Ethernet Controller 10-Gigabit X540-AT2 pf enabled ufs (no zfs) nginx with sendfile on and proxy_pass + proxy_cache.
If functions_default=rack: net.inet.tcp.functions_available: Stack D Alias PCB count freebsd freebsd 43 bbr bbr 18 rack * rack 110632 mbufs behavior is the same (growing) 7738694K/48547K/7787241K bytes allocated to network (current/cache/total)
Created attachment 223696 [details] netstat -m
Created attachment 223697 [details] vmstat -m
Created attachment 223698 [details] vnstat -z
Also affected in 13.0-RELEASE. Last hour. Server is hang. netstat -an has only listen tcp sockets. mbufs denied growing continuously. mbufs in use and bytes allocated to network near the limit. This server works with nginx, 10gbit network. ix0 + ix1 -> lagg interfaces on http/https load - 10000-20000 qps. Another servers on freebsd-13 with databases and redis - little rate long-live connections - works fine!
Created attachment 224020 [details] netstat -an (11 apr)
Created attachment 224021 [details] netstat -m (11 apr)
Created attachment 224022 [details] vmstat -z (11 apr)
Created attachment 224023 [details] dmesg (11 apr)