Bug 254453 - [zone: mbuf] kern.ipc.mbufs limit reached - mbufs leaks?
Summary: [zone: mbuf] kern.ipc.mbufs limit reached - mbufs leaks?
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-21 06:53 UTC by Igor A. Valkov
Modified: 2021-04-11 12:21 UTC (History)
0 users

See Also:


Attachments
netstat -m (209.24 KB, image/jpeg)
2021-03-21 06:53 UTC, Igor A. Valkov
no flags Details
top (208.16 KB, image/jpeg)
2021-03-21 06:54 UTC, Igor A. Valkov
no flags Details
system tty (212.61 KB, image/jpeg)
2021-03-21 06:55 UTC, Igor A. Valkov
no flags Details
netstat -m (1.20 KB, text/plain)
2021-03-29 22:44 UTC, Igor A. Valkov
no flags Details
vmstat -m (9.19 KB, text/plain)
2021-03-29 22:44 UTC, Igor A. Valkov
no flags Details
vnstat -z (12.75 KB, text/plain)
2021-03-29 22:48 UTC, Igor A. Valkov
no flags Details
netstat -an (11 apr) (14.07 KB, text/plain)
2021-04-11 12:19 UTC, Igor A. Valkov
no flags Details
netstat -m (11 apr) (1.20 KB, text/plain)
2021-04-11 12:19 UTC, Igor A. Valkov
no flags Details
vmstat -z (11 apr) (12.73 KB, text/plain)
2021-04-11 12:20 UTC, Igor A. Valkov
no flags Details
dmesg (11 apr) (4.62 KB, text/plain)
2021-04-11 12:21 UTC, Igor A. Valkov
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Igor A. Valkov 2021-03-21 06:53:34 UTC
Created attachment 223464 [details]
netstat -m

Server the same is in bug #254419, running on 10gbit ix with nginx sendfile = off.

# sysctl net.inet.tcp.functions_available
net.inet.tcp.functions_available: 
Stack                           D Alias                            PCB count
freebsd                           freebsd                          61
rack                              rack                             0
bbr                             * bbr                              122508

After some hours network is hang. Works only local console (ikvm).

In /var/log/messages:
Mar 21 07:57:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached
Mar 21 08:02:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached
Mar 21 08:07:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached
Mar 21 08:12:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached
Mar 21 08:17:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached
Mar 21 08:22:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached
Mar 21 08:27:45 serv11 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached

netstat -m in attached screenshot

Server with the same load and configuration but on FreeBSD-12.2-p4 and net.inet.tcp.functions_default=rack works fine.
Comment 1 Igor A. Valkov 2021-03-21 06:54:47 UTC
Created attachment 223465 [details]
top
Comment 2 Igor A. Valkov 2021-03-21 06:55:13 UTC
Created attachment 223466 [details]
system tty
Comment 3 Igor A. Valkov 2021-03-28 07:07:18 UTC
Bytes allocated to network in netstat -m is constantly growing.
After approx 1 day uptime network is hang with:

dmesg:
[zone: mbuf] kern.ipc.nmbufs limit reached
[zone: mbuf] kern.ipc.nmbufs limit reached
[zone: mbuf] kern.ipc.nmbufs limit reached
[zone: mbuf] kern.ipc.nmbufs limit reached
[zone: mbuf] kern.ipc.nmbufs limit reached
[zone: mbuf] kern.ipc.nmbufs limit reached

netstat -m:
31994144/57556/32051700 mbufs in use (current/cache/total)
32898/14358/47256/5000000 mbuf clusters in use (current/cache/total/max)
8/10270 mbuf+clusters out of packet secondary zone in use (current/cache)
34079/298/34377/2000000 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/900000 9k jumbo clusters in use (current/cache/total/max)
0/0/0/600000 16k jumbo clusters in use (current/cache/total/max)
8200648K/44297K/8244945K bytes allocated to network (current/cache/total)
383111218/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
27833/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
1681210 sendfile syscalls
1678853 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
12525327 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
2703 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

net.inet.tcp.functions_available:
Stack                           D Alias                            PCB count
freebsd                           freebsd                          10
bbr                             * bbr                              44987
rack                              rack                             0

vmstat -m:
         Type InUse MemUse Requests  Size(s)
     filedesc    64 17654K      473  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
      pwddesc   111     7K    23949  64
        sigio    31     2K       33  64
     filecaps     0     0K     2314  16,32,64
      kdtrace  1048   241K    24983  64,256
         kenv   117    13K      117  16,32,64,128,8192
       kqueue   204  2260K    24965  64,256,512,2048,4096,8192,16384,32768,65536
    proc-args   311    12K    48278  16,32,64,128,256
  Fail Points     0     0K       20  1024
        hhook    13     4K       13  256
      ithread   448    74K      448  32,128,256
       prison     8     1K        8  16,32
       KTRACE   100    13K      100  128
       linker   213  2001K      257  16,32,64,128,256,384,512,1024,2048,4096,8192,16384
        lockf    23     3K     2515  64,128
   loginclass     3     1K        7  64
       devbuf 18525 117840K    20947  16,32,64,128,256,384,512,1024,2048,4096,8192,65536
         temp   195    20K  2142283  16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
     acpiintr     1     1K        1  64
       module   234    30K      234  128
     mtx_pool     2    72K        2  8192,65536
          osd   134     3K      145  16,32,64,128,256
     pmchooks     1     1K        1  128
          pmc     2     1K        2  64
      session    19     3K     7515  128
         proc     3   132K        3  4096,65536
      subproc   614   637K    24451  384,4096
         cred    53    14K    74359  256
       acpica 32007  3114K  4383696  16,32,64,128,256,384,512,1024,2048
     acpitask     1    64K        1  65536
      acpisem    33     5K       33  128
       plimit    15     4K    28343  256
      uidinfo     5    17K     6847  128,16384
     sendfile     0     0K  4179384  64,128,256,384,512,1024,2048,4096,8192,16384
       dumper     2     1K        2  128,512
      acpidev   232    15K      232  64
       sysctl     1     1K      401  32,64
    sysctloid 50151  2825K    50666  16,32,64,128,256,384
    sysctltmp     0     0K    34983  16,32,64,256,1024
      CAM SIM    14     4K       14  256
      CAM XPT    43     5K       97  16,32,128,256,512,1024,2048
      tidhash     3   260K        3  8192
      callout    32  8192K       32
         umtx  2124   266K     2124  128
     p1003.1b     1     1K        1  16
         SWAP     1  4164K        1
          bus  1555   207K   264553  16,32,64,128,256,384,4096
       bus-sc   225  2460K    20627  16,32,64,128,256,384,512,1024,2048,4096,8192,16384,32768,65536
      CAM DEV    16    32K       38  2048
      devstat    12    25K       12  32,4096
 eventhandler   142    12K      142  64,128
   gtaskqueue   198    44K      198  16,32,256,8192
         kobj   132   528K      484  4096
      Per-cpu     1     1K        1  32
      CAM CCB     0     0K    63419  2048
         rman   378    43K      769  16,32,128
         sbuf     1     1K     7285  16,32,64,128,256,512,1024,2048,4096,8192,16384
    toponodes    84    11K       84  128
     CAM path    17     1K      110  32
   CAM periph     6     2K       39  16,32,64,128,256
    acpi_perf    32    12K       32  384
        stack     0     0K      201  256
    taskqueue    54     7K       54  16,32,256
     terminal    11     3K       11  256
       Unitno    22     2K     4932  32,64
         vmem     4   448K       37  2048,4096,8192,16384,32768,65536
     ioctlops     0     0K    11948  256,384,512,1024,2048,4096
       select    12     2K    12853  128
          iov     0     0K 1542747788  16,64,128,256,384,512,1024,2048
        selfd    41     3K  2383115  64
     procdesc     4     1K       21  128
          msg     4    30K        4  2048,4096,8192,16384
          sem     4   106K        4  2048,4096
          shm     1    32K        1  32768
          tty    14    14K       18  1024
          pts     0     0K        4  256
         ktls     1     4K        1  4096
     mbuf_tag     0     0K  1013618  32,64
        shmfd     1     8K        1  8192
       soname     4     1K 256798386  16,32,64,128
          pcb  2309  3224K 48094843  16,32,64,1024,2048
           ix    22    74K       22  1024,2048,4096
       kbdmux     7    22K        7  16,384,1024,2048,16384
          LED    40     3K       40  16,128
       mfibuf   133    51K    13638  16,32,128,256,384,1024,2048,4096,8192,32768
       sr_iov     2     1K        2  384
      entropy     1     1K    29218  32,64,4096
         UART     6     5K        6  16,1024
    CAM queue    30     9K       98  16,32,512
          USB    61    97K       69  16,128,256,384,1024,2048,4096,8192,16384
     vfscache     3  8384K        3  65536
       USBdev    47    10K       47  32,64,128,256,384,512,4096
        vtbuf    24  1968K       46  4096
           vt    13    41K       13  384,4096,32768
   cl_savebuf     0     0K   168291  64,128,256,384
     vfs_hash     1  4096K        1
       vnodes     1     1K        2  32,256
        mount    76     4K      153  16,32,64,128,256,1024
       statfs     0     0K      319  4096
  vnodemarker     3     2K    60944  512
      fadvise     1     1K        5  32
chacha20random     1     8K        1  8192
          BPF    12  1026K       14  32,128,384,512
        ifnet     9    17K        9  128,2048
       ifaddr    90    19K       90  16,32,64,128,256,384,2048,4096
  ether_multi    64     6K      104  16,32,64,128
        clone     7     1K        7  128
      lltable    47    14K     1280  256,384
          tun     3     1K        3  32
         vlan    13     2K       33  64,128
        iflib   402  3951K      418  64,128,256,512,1024,8192,16384,32768
         pfil     7     1K       15  64,128
        nhops     6     1K        6  128
     routetbl    19     4K       19  32,64,128,384,1024
         vnet     1     1K        1  64
    vnet_data     1   180K        1
vnet_data_free     1     1K        1  32
         igmp     8     1K        8  128
     in_multi     3     1K        3  256
  ip_moptions     2     1K        2  64
   tfo_ccache     1   128K        1
    hostcache     1    32K        1  32768
          LRO    64  1280K       64  8192,32768
     tcp_hpts    65 51209K       65  256
       DEVFS3   168    42K      205  256
      tcpfunc     3     1K        3  64
     syncache     1   136K        1
    in6_multi    31     4K       31  32,256
 ip6_moptions     3     1K        3  32
       ip6opt     5     2K       11  32,256
          mld     7     1K        7  128
       ip6ndp    10     2K       12  64,256
  inpcbpolicy 45039  1408K 253441302  32
     secasvar     1     1K        1  1024
       sahead     1     1K        1  1024
  ipsecpolicy     1     1K        1  1024
    ipsec-saq     2     2K        2  1024
      pf_temp     0     0K     3381  32
      pf_hash     5 11524K        5  2048
     pf_ifnet    13     5K       57  256,2048
      pf_rule    38    24K       38  128,1024
      pf_osfp  1191   123K     1191  64,128
       crypto     3     5K        3  64,256,4096
       DEVFS1   139   139K      166  1024
audit_evclass   236     8K      294  32
      pagedep     4  2048K   293568  256
     inodedep     4 16384K  1319452  384
    bmsafemap     4    32K   469650  256,8192
       newblk     4 32768K  4749893  256
     indirdep     0     0K    24729  128,32768
     freefrag     0     0K   496137  128
     freeblks     0     0K   529155  256
     freefile     0     0K   534502  64
       diradd     0     0K   795038  128
        mkdir     0     0K       18  128
       dirrem     0     0K   932077  128
    newdirblk     0     0K      297  64
     freework     4     1K   934622  64,128
      jaddref     0     0K        6  128
      jremref     0     0K        6  128
      jnewblk     0     0K       30  128
    jfreefrag     0     0K        3  128
         jseg     0     0K        6  128
      jsegdep     0     0K       45  64
        sbdep     0     0K     6177  64
     savedino     0     0K   393583  256
      jblocks     2     1K        2  128,256
      softdep     4     2K        4  512
  ufs_dirhash  1768   738K   380492  16,32,64,128,256,384,512,1024,2048
    ufs_quota     1  4096K        1
    ufs_mount    25   631K       42  32,64,512,4096,8192,16384,32768
    vm_pgdata     1     1K        1  128
   DEVFS_RULE    59    21K       59  64,384
          UMA   240     5K      243  16,32
        DEVFS    19     1K       20  16,128
       DEVFSP     2     1K        4  64
    pfs_nodes    20     8K       20  384
  pfs_vncache     1    64K        1  65536
         GEOM   126    22K     1445  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
    raid_data     0     0K      222  32,128,256
     pci_link    16     2K       16  64,128
       apmdev     1     1K        1  128
         intr     4   216K        4  32768,65536
          MCA    65    17K       65  128,256,384
         cpus     2     1K        2  128
     nexusdev     6     1K        6  16
      memdesc     1     4K        1  4096
         ioat    16  7168K       16
       isadev     6     1K        6  128
      scsi_cd     0     0K       12  16
CAM dev queue    14     1K       14  64
       nvlist    94     9K       94  16,64,128
   madt_table     0     0K        2  256,4096
         cdev     1     1K        1  256
      io_apic     3     6K        3  2048
   local_apic     1    32K        1  32768
          msi    65     9K       65  128
      acpiwmi     2     1K        4  32,64
   kstat_data    17    17K       17  1024
      solaris   131 65839K      248  16,32,64,128,256,1024,2048,4096,8192,65536
        taskq     2     3K        2  1024,2048
Comment 4 Igor A. Valkov 2021-03-28 07:09:26 UTC
Ethernet Controller 10-Gigabit X540-AT2
pf enabled
ufs (no zfs)
nginx with sendfile on and proxy_pass + proxy_cache.
Comment 5 Igor A. Valkov 2021-03-29 20:34:09 UTC
If functions_default=rack:

net.inet.tcp.functions_available: 
Stack                           D Alias                            PCB count
freebsd                           freebsd                          43
bbr                               bbr                              18
rack                            * rack                             110632

mbufs behavior is the same (growing)


7738694K/48547K/7787241K bytes allocated to network (current/cache/total)
Comment 6 Igor A. Valkov 2021-03-29 22:44:30 UTC
Created attachment 223696 [details]
netstat -m
Comment 7 Igor A. Valkov 2021-03-29 22:44:57 UTC
Created attachment 223697 [details]
vmstat -m
Comment 8 Igor A. Valkov 2021-03-29 22:48:21 UTC
Created attachment 223698 [details]
vnstat -z
Comment 9 Igor A. Valkov 2021-04-11 12:17:16 UTC
Also affected in 13.0-RELEASE.

Last hour. Server is hang.
netstat -an has only listen tcp sockets.
mbufs denied growing continuously.
mbufs in use and bytes allocated to network near the limit.

This server works with nginx, 10gbit network. ix0 + ix1 -> lagg interfaces on http/https load - 10000-20000 qps.

Another servers on freebsd-13 with databases and redis - little rate long-live connections - works fine!
Comment 10 Igor A. Valkov 2021-04-11 12:19:25 UTC
Created attachment 224020 [details]
netstat -an (11 apr)
Comment 11 Igor A. Valkov 2021-04-11 12:19:51 UTC
Created attachment 224021 [details]
netstat -m (11 apr)
Comment 12 Igor A. Valkov 2021-04-11 12:20:24 UTC
Created attachment 224022 [details]
vmstat -z (11 apr)
Comment 13 Igor A. Valkov 2021-04-11 12:21:19 UTC
Created attachment 224023 [details]
dmesg (11 apr)