Bug 243646 - panic: bad pte va 966000 pte 0
Summary: panic: bad pte va 966000 pte 0
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2020-01-27 07:26 UTC by Dirk Meyer
Modified: 2022-10-12 00:49 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Meyer freebsd_committer freebsd_triage 2020-01-27 07:26:22 UTC
FreeBSD netcom2.dinoex.sub.de 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 r357113 GENERIC  amd64

panic: bad pte va 966000 pte 0

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: bad pte va 966000 pte 0
cpuid = 0
time = 1580083866
KDB: stack backtrace:
#0 0xffffffff80c1d297 at kdb_backtrace+0x67
#1 0xffffffff80bd05cd at vpanic+0x19d
#2 0xffffffff80bd0423 at panic+0x43
#3 0xffffffff81099fb7 at pmap_remove_pages+0x8b7
#4 0xffffffff80f1171c at vmspace_exit+0x9c
#5 0xffffffff80b8a509 at exit1+0x5d9
#6 0xffffffff80b89f2d at sys_sys_exit+0xd
#7 0xffffffff810a8984 at amd64_syscall+0x364
#8 0xffffffff81082330 at fast_syscall_common+0x101
Uptime: 14h50m29s
Dumping 937 out of 16235 MB:..2%..11%..21%..31%..41%..52%..62%..72%..81%..91%

Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/linprocfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux_common.ko
Reading symbols from /boot/kernel/linsysfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/linsysfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linsysfs.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/radeonkms.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkms.ko.debug...done.
done.
Loaded symbols for /boot/kernel/radeonkms.ko
Reading symbols from /boot/kernel/drm2.ko...Reading symbols from /usr/lib/debug//boot/kernel/drm2.ko.debug...done.
done.
Loaded symbols for /boot/kernel/drm2.ko
Reading symbols from /boot/kernel/iicbus.ko...Reading symbols from /usr/lib/debug//boot/kernel/iicbus.ko.debug...done.
done.
Loaded symbols for /boot/kernel/iicbus.ko
Reading symbols from /boot/kernel/iic.ko...Reading symbols from /usr/lib/debug//boot/kernel/iic.ko.debug...done.
done.
Loaded symbols for /boot/kernel/iic.ko
Reading symbols from /boot/kernel/iicbb.ko...Reading symbols from /usr/lib/debug//boot/kernel/iicbb.ko.debug...done.
done.
Loaded symbols for /boot/kernel/iicbb.ko
Reading symbols from /boot/kernel/radeonkmsfw_BARTS_pfp.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_BARTS_pfp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/radeonkmsfw_BARTS_pfp.ko
Reading symbols from /boot/kernel/radeonkmsfw_BARTS_me.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_BARTS_me.ko.debug...done.
done.
Loaded symbols for /boot/kernel/radeonkmsfw_BARTS_me.ko
Reading symbols from /boot/kernel/radeonkmsfw_BTC_rlc.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_BTC_rlc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/radeonkmsfw_BTC_rlc.ko
Reading symbols from /boot/kernel/radeonkmsfw_BARTS_mc.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_BARTS_mc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/radeonkmsfw_BARTS_mc.ko
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
done.
Loaded symbols for /boot/kernel/uhid.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux64.ko
Reading symbols from /boot/kernel/speaker.ko...Reading symbols from /usr/lib/debug//boot/kernel/speaker.ko.debug...done.
done.
Loaded symbols for /boot/kernel/speaker.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/sem.ko...Reading symbols from /usr/lib/debug//boot/kernel/sem.ko.debug...done.
done.
Loaded symbols for /boot/kernel/sem.ko
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/tmpfs.ko
Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
#0  doadump () at src/sys/amd64/include/pcpu.h:234
234             __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) #0  doadump () at src/sys/amd64/include/pcpu.h:234
#1  0xffffffff80bd01c8 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff80bd0629 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:877
#3  0xffffffff80bd0423 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:804
#4  0xffffffff81099fb7 in pmap_remove_pages (pmap=<value optimized out>)
    at /usr/src/sys/amd64/amd64/pmap.c:6932
#5  0xffffffff80f1171c in vmspace_exit (td=0xfffff8019974e5e0)
    at /usr/src/sys/vm/vm_map.c:409
#6  0xffffffff80b8a509 in exit1 (td=0xfffff8019974e5e0, 
    rval=<value optimized out>, signo=0) at /usr/src/sys/kern/kern_exit.c:396
#7  0xffffffff80b89f2d in sys_sys_exit (td=<value optimized out>, 
    uap=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:175
#8  0xffffffff810a8984 in amd64_syscall (td=0xfffff8019974e5e0, traced=0)
    at src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#9  0xffffffff81082330 in fast_syscall_common ()
    at /usr/src/sys/amd64/amd64/exception.S:581
#10 0x0000000003e6794a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------
ps -axlww

  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN   STAT TT        TIME COMMAND
    0     0     0   0 -16  0     0     0 swapin   DLs   -     0:00.04 [kernel]
    0     1     0   0  25  0  9960   896 wait     DLs   -     0:00.00 [init]
    0     2     0   0 -16  0     0     0 crypto_w DL    -     0:00.00 [crypto]
    0     3     0   0 -16  0     0     0 crypto_r DL    -     0:00.00 [crypto returns 0]
    0     4     0   0 -16  0     0     0 crypto_r DL    -     0:00.00 [crypto returns 1]
    0     5     0   0 -16  0     0     0 crypto_r DL    -     0:00.00 [crypto returns 2]
    0     6     0   0 -16  0     0     0 crypto_r DL    -     0:00.00 [crypto returns 3]
    0     7     0   0 -16  0     0     0 -        RL    -     0:00.43 [cam]
    0     8     0   0 -16  0     0     0 -        DL    -     0:00.00 [soaiod1]
    0     9     0   0 -16  0     0     0 -        DL    -     0:00.00 [soaiod2]
    0    10     0   0 -16  0     0     0 audit_wo DL    -     0:00.00 [audit]
    0    11     0   0 155  0     0     0 -        RL    -  3539:05.53 [idle]
    0    12     0   0 -56  0     0     0 -        WL    -     0:29.93 [intr]
    0    13     0   0  -8  0     0     0 -        DL    -     0:00.00 [geom]
    0    14     0   0 -16  0     0     0 seqstate DL    -     0:00.00 [sequencer 00]
    0    15     0   0 -68  0     0     0 -        DL    -     0:02.98 [usb]
    0    16     0   0 -16  0     0     0 -        DL    -     0:00.00 [soaiod3]
    0    17     0   0 -16  0     0     0 -        DL    -     0:00.00 [soaiod4]
    0    18     0   0 -16  0     0     0 waiting_ DL    -     0:00.00 [sctp_iterator]
    0    19     0   0 -16  0     0     0 tzpoll   DL    -     0:00.17 [acpi_thermal]
    0    20     0   0 -16  0     0     0 cooling  DL    -     0:00.03 [acpi_cooling1]
    0    21     0   0 -16  0     0     0 -        DL    -     0:02.89 [rand_harvestq]
    0    22     0   0 -16  0     0     0 psleep   DL    -     0:00.87 [pagedaemon]
    0    23     0   0 -16  0     0     0 psleep   DL    -     0:00.00 [vmdaemon]
    0    24     0   0  20  0     0     0 qsleep   DL    -     0:01.14 [bufdaemon]
    0    25     0   0 -16  0     0     0 vlruwt   DL    -     0:00.04 [vnlru]
    0    26     0   0  16  0     0     0 syncer   DL    -     0:00.33 [syncer]
    0   550     1   0  20  0 11048  2388 select   Ds    -     0:00.00 [moused]
    0   579     1   0  20  0 11388  2620 select   Ds    -     0:00.00 [dhclient]
    0   582     1   0  43  0 11612  2760 select   Ds    -     0:00.00 [dhclient]
   65   628     1   0  20  0 11752  2876 select   DCs   -     0:00.00 [dhclient]
    0   629     1   0  20  0 10456  1476 select   Ds    -     0:02.54 [devd]
    0   637     1   0  20  0 16272  5500 select   Ds    -     0:00.32 [tincd]
    0   711     1   0  20  0 10908  2216 select   Ds    -     0:00.01 [rtsold]
    0   732     1   0  20  0 11400  2668 select   Ds    -     0:01.88 [syslogd]
    0   946     1   0  52  0 11084  2452 select   Ds    -     0:00.00 [lpd]
   25   982     1   0  20  0 16800  6368 pause    Ds    -     0:00.01 [sendmail]
    0   995     1   0  20  0 17156  6912 select   Ds    -     0:00.24 [openvpn]
    0  1006     1   0  20  0 17104  6956 select   Ds    -     0:00.35 [openvpn]
    0  1048     1   0  20  0 19652  8372 select   Ds    -     0:00.00 [sshd]
    0  1051     1   0  44  0 11464  2676 nanslp   Ds    -     0:00.07 [cron]
    0  1127     1   0  52  0 11444  2600 select   Ds    -     0:00.00 [inetd]
    0  1152     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  1153     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  1154     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  1155     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  1156     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  1157     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  1158     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  1159     1   0  52  0 10880  2288 ttyin    Ds+   -     0:00.00 [getty]
    0  4668  1048   0  20  0 21052  9584 select   Ds    -     0:00.02 [sshd]
    0  4670  4668   0  20  0 14408  4656 pause    Ds    -     0:00.02 [csh]
    0  4682  4668   0  20  0 13604  4332 ttyin    Ds+   -     0:00.01 [csh]
    0  4688  4670   0  21  0 13204  4100 select   D+    -     0:00.00 [tmux]
    0  4690     1   0  20  0 14096  5056 select   Ds    -     0:00.01 [tmux]
    0  4691  4690   0  20  0 13608  4348 pause    Ds    -     0:00.01 [csh]
    0 25060  4691   0  52  0 12932  3852 select   D+    -     0:00.82 [sh]
    0 25368 25060   0  52  0 12932  3720 nanslp   D+    -     0:00.16 [sh]
    0 45759 25060   0  27  0 12932  3844 piperd   D+    -     0:00.00 [sh]
    0 64142 25060   0  52  0 12932  3860 select   D     -     0:00.00 [sh]
    0 64610 64142   0  52  0 12932  3856 wait     D     -     0:00.00 [sh]
65534 64611 64610   0  52  0 10304  1744 wait     D     -     0:00.00 [make]
65534 64652 64611   0  52  0 11644  3144 wait     D     -     0:00.00 [sh]
65534 64950 64652   0  52  0 12616  3668 wait     D     -     0:00.00 [sh]
65534 67594 64950   0  72  0 72556 51644 -        RE    -     0:00.00 [cpp]
65534 67595 64950   0  72  0 11388   532 -        RE    -     0:00.00 [egrep]

------------------------------------------------------------------------
vmstat -s

 15831555 cpu context switches
  2131396 device interrupts
   768226 software interrupts
  8267188 traps
 14996492 system calls
       26 kernel threads created
    44730  fork() calls
    22838 vfork() calls
        2 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     2615 vnode pager pageins
    10932 vnode pager pages paged in
      167 vnode pager pageouts
     1812 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 clean page reclamation shortfalls
        0 pages reactivated by the page daemon
  2572938 copy-on-write faults
     4867 copy-on-write optimized faults
  5287614 zero fill pages zeroed
        0 zero fill pages prezeroed
       14 intransit blocking page faults
  8296951 total VM faults taken
     2407 page faults requiring I/O
        0 pages affected by kernel thread creation
  1745990 pages affected by  fork()
  1648727 pages affected by vfork()
      114 pages affected by rfork()
 10486245 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
        0 pages active
        0 pages inactive
        0 pages in the laundry queue
        0 pages wired down
        0 pages free
        0 bytes per page
        0 total name lookups
          cache hits (0% pos + 0% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
      CAM DEV    10    20K       -       12  2048
      entropy     1     1K       -     1942  32,4096
      CAM CCB     0     0K       -   438443  2048
     CAM path    17     1K       -       65  32
   CAM periph    16     4K       -    87966  16,32,64,128,256
    acpi_perf     4     2K       -        4  512
      acpipwr    11     1K       -       11  64
         hdaa    14    57K       -       14  256,512,1024,2048,16384,32768
         hdac     3     2K       -        3  128,256,1024
        hdacc     3     1K       -        3  32
       feeder    21     2K       -       27  32,128
        mixer     4    16K       -        4  4096
CAM I/O Scheduler     6     1K       -        6  128
    CAM queue    14     6K       -       26  16,32,64,512
      scsi_cd     0     0K       -        6  16
          USB    60    92K       -       67  16,32,128,256,512,1024,2048,4096,8192,16384
       USBdev    44    10K       -       44  32,64,128,256,512,4096
CAM dev queue     4     1K       -        4  64
      scsi_da     0     0K       -       60  16,256
        vtbuf    24  5712K       -       70  4096
           vt    11     6K       -       11  512
       DEVFS2   141     5K       -      282  16,32
       DEVFS3   515   129K       -      709  256
       DEVFS1   141    71K       -      170  512
   DEVFS_RULE    65    31K       -       65  64,512
        DEVFS    53     2K       -       71  16,128
       DEVFSP     3     1K       -        3  64
NFSD V4client     1     1K       -        1  256
 NFSD lckfile     1     1K       -        1  256
 NFSD session     1     1K       -        1  1024
    pfs_nodes   388   194K       -      388  512
  pfs_vncache     9     1K       -       13  64
         GEOM   139    27K       -     1293  16,32,64,128,256,512,1024,2048,8192,16384,32768
    raid_data     0     0K       -      174  32,128,256
       isadev     8     1K       -        8  128
         cdev     5     2K       -        5  256
     filedesc     1     4K       -        5  32,64,4096,16384,32768
        sigio     0     0K       -        1  64
     filecaps     1     1K       -     2506  16,32,64
      kdtrace   206    40K       -   136098  64,256
         kenv    95    12K       -       98  16,32,64,128,8192
       kqueue    66     5K       -    81287  64,256,512
    proc-args    68     4K       -    34506  16,32,64,128,256
        hhook    13     4K       -       13  256
      ithread    96    17K       -       96  32,128,256
       prison    30    22K       -       32  16,32,256,4096
       KTRACE   100    13K       -      100  128
       linker   545 10577K       -      682  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        lockf    40     5K       -     4348  64,128
   loginclass     3     1K       -        3  64
        cache     1     1K       -        1  32
       devbuf 18069 37898K       -    20274  16,32,64,128,256,512,1024,2048,4096,8192,65536
         temp    54    18K       -   216655  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        evdev     5     5K       -        5  1024
       module   532    67K       -      533  128
     mtx_pool     2    72K       -        2  8192,65536
          osd     8     1K       -       34  16,32,64,128,256
     pmchooks     1     1K       -        1  128
          pmc     1     1K       -        1  64
         pgrp    32     4K       -      747  128
      session    29     4K       -      675  128
         proc     2    64K       -        2  32768
      subproc   184   323K       -    67893  512,4096
         cred    62    16K       -    31160  256
     acpiintr     1     1K       -        1  64
       plimit    24     6K       -    56427  256
      uidinfo     5     9K       -      198  128,8192
       dumper     1     1K       -        1  512
       acpica  6964   688K       -   282579  16,32,64,128,256,512,1024,2048,4096
       sysctl     0     0K       -       27  64
    sysctloid  6234   318K       -     6392  16,32,64,128
    sysctltmp     0     0K       -   357537  16,64,256,1024
     acpitask     1    64K       -        1  65536
      tidhash     1    64K       -        1  65536
      callout     5  2184K       -        5  
         umtx   468    59K       -      468  128
     p1003.1b     1     1K       -        1  16
         SWAP     1   232K       -        1  
          bus  1461   130K       -   189623  16,32,64,128,256,1024,4096
       bus-sc   111  1914K       -     3733  16,32,128,256,512,1024,2048,4096,8192,16384,32768,65536
      acpisem   114    15K       -      114  128
      devstat    10    21K       -       10  32,4096
        epoch     4     1K       -        4  128
 eventhandler   127    11K       -      127  64,128
   gtaskqueue    34    35K       -       34  16,32,256,8192
         kobj   357  1428K       -      657  4096
      Per-cpu     1     1K       -        1  32
       kbdmux     7    22K       -        7  16,512,1024,2048,16384
         rman   243    28K       -      670  32,128
         sbuf     0     0K       -     1066  16,32,64,128,256,4096,8192,16384
    toponodes    18     3K       -       18  128
    taskqueue    39     5K       -       39  16,32,256
     terminal    11     3K       -       11  256
       Unitno    31     2K       -      113  32,64
         vmem     3   260K       -       13  2048,4096,8192,16384,32768,65536
     ioctlops     0     0K       -     1413  256,512,1024,2048
       select   101    13K       -      101  128
          iov     0     0K       -   759172  16,32,64,128,256,512,4096
          msg     4    30K       -        4  2048,4096,8192,16384
          sem     4   106K       -        4  2048,4096
          shm     1    32K       -        1  32768
          tty    15    15K       -       15  1024
          pts     3     1K       -        3  256
     mbuf_tag     0     0K       -      457  32
        shmfd     1     8K       -        1  8192
       soname     6     1K       -   396173  16,32,128
          pcb    32  2201K       -     2854  16,32,64,128,1024,2048,8192
          acl     0     0K       -      298  4096
     vfscache     4  4145K       -        4  256,16384,32768
   cl_savebuf     0     0K       -      533  64
     vfs_hash     1  2048K       -        1  
       vnodes     4     1K       -      149  32,256
        mount   476    25K       -    20142  16,32,64,128,256,1024
       statfs     0     0K       -    95211  4096
  vnodemarker     0     0K       -     6732  512
chacha20random     1     1K       -        1  1024
          BPF    13    18K       -       13  128,512,4096
        ifnet     6    11K       -        6  128,2048
       ifaddr    72    21K       -       72  16,32,64,128,256,512,2048,4096
  ether_multi    76     6K       -       76  16,32,64,128
        clone     7     1K       -        7  128
        ipsec     3     1K       -        3  256
      lltable    28     9K       -       28  256,512
          tun     3     1K       -        3  256
     routetbl   100     9K       -      109  32,64,128,512
         vnet     1     1K       -        1  64
    vnet_data     1   240K       -        1  
vnet_data_free     1     1K       -        1  32
         igmp     5     1K       -        5  128
     in_multi     5     2K       -        5  256
encap_export_host    12     1K       -       12  32,64
    sctp_a_it     0     0K       -        9  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa    14     2K       -       14  128
     sctp_ifn     5     1K       -        5  128
    sctp_iter     0     0K       -        9  256
   tfo_ccache     1   128K       -        1  
    hostcache     1    32K       -        1  32768
      tcpfunc     1     1K       -        1  64
     syncache     1    68K       -        1  
    in6_multi    60     8K       -       60  32,256
       ip6opt     1     1K       -        2  32,256
          mld     5     1K       -        5  128
       ip6ndp    13     2K       -       18  32,64,256
  inpcbpolicy    20     1K       -     3622  32
     secasvar     1     1K       -        1  1024
       sahead     1     1K       -        1  1024
  ipsecpolicy     2     2K       -        2  256,1024
    ipsec-saq     2     2K       -        2  1024
       crypto     2     2K       -        2  512,1024
          rpc     2     8K       -        2  4096
audit_evclass   230     8K       -      285  32
      pagedep   517   641K       -     7050  256
     inodedep  5417  4756K       -    79685  512
    bmsafemap    10    11K       -    69888  256,8192
       newblk  1988  4593K       -   200778  256
     indirdep    23     3K       -      850  128,32768
     freefrag     3     1K       -     2314  128
     freeblks   200    50K       -    58299  256
     freefile    14     1K       -    59177  64
       diradd  5369   672K       -   102743  128
        mkdir   665    84K       -    11638  128
       dirrem    18     3K       -    96801  128
    newdirblk   491    31K       -     6051  64
     freework   307    39K       -    80981  64,128
      freedep     0     0K       -      795  64
      jaddref     1     1K       -   114381  128
      jremref     2     1K       -   107327  128
       jmvref     0     0K       -      741  128
      jnewblk     1     1K       -   200777  128
    jfreefrag     0     0K       -     2314  128
         jseg 26729  3342K       -    39109  128
      jsegdep 10920   683K       -   424799  64
        sbdep     0     0K       -      820  64
     savedino     0     0K       -     4295  256
      jblocks     2     1K       -        2  128,256
      softdep     1     1K       -        1  512
  ufs_dirhash   384    80K       -     1449  16,32,64,128,256,512,1024
    ufs_quota     1  2048K       -        1  
    ufs_mount     3    41K       -        5  512,8192,32768
    vm_pgdata     1     1K       -        1  128
      UMAHash     5    22K       -       13  512,1024,2048,4096,8192
      memdesc     1     4K       -        1  4096
     pci_link    16     2K       -       16  64,128
     atkbddev     2     1K       -        2  64
      acpidev    61     4K       -       61  64
      CAM SIM     4     1K       -        4  256
      CAM XPT    46     4K       -      123  16,32,64,128,256,512,1024,2048,65536
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        2  64,4096
         intr     4   400K       -        4  65536
      io_apic     1     2K       -        1  2048
   local_apic     1     8K       -        1  8192
          MCA    14     3K       -       14  32,128,256
         cpus     2     1K       -        2  32
          msi    15     2K       -       15  128
     nexusdev     5     1K       -        5  16
        linux    32     2K       -       32  64
     gem_name    61    10K       -       73  32,4096
   drm_global     2     1K       -        2  128,256
      drm_dma     1     1K       -        1  32
    drm_sarea     1     1K       -        1  16
   drm_driver    70  2264K       -      150  16,32,64,128,256,512,1024,2048,4096,8192,32768
    drm_minor     2     1K       -        2  128
drm_ctxbitmap     1     4K       -        1  4096
     drm_sman    18     3K       -       18  128
  drm_hashtab     2  4128K       -        2  32768
      drm_kms    92    25K       -      123  16,32,64,128,256,512,2048,4096,8192
   drm_vblank     7     1K       -        7  32,256
       ttm_pd    10     9K       -       10  16,128,2048
     ttm_rman     2     1K       -        2  256
     ttm_zone     2     1K       -        2  64
  ttm_poolmgr     1     1K       -        1  512
       nvidia   208  1183K       -      209  16,32,64,128,256,512,2048,4096,8192,16384,65536
         ksem     1     8K       -        1  8192
  nullfs_hash     1  2048K       -        1  
  nullfs_node  3171   199K       -     6018  64
 nullfs_mount    17     1K       -       27  32
  tmpfs mount     3     1K       -        5  256
   tmpfs name  7868   273K       -   108030  16,32,64,128
  fdesc_mount     2     1K       -        3  16

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               224,      0,     131,       5,     135,   0,   0
UMA Zones:              936,      0,     134,       2,     138,   0,   0
UMA Slabs:               80,      0,    6032,      18,    6251,   0,   0
UMA Hash:               256,      0,       8,       7,      13,   0,   0
4 Bucket:                32,      0,      93,    1032,   11520,   0,   0
6 Bucket:                48,      0,      18,    1061,    4845,   0,   0
8 Bucket:                64,      0,      21,    1777,   56587,  21,   0
12 Bucket:               96,      0,      28,     505,    2110,   0,   0
16 Bucket:              128,      0,      77,    2217,   11169,   1,   0
32 Bucket:              256,      0,    2933,    1192,   57010,   9,   0
64 Bucket:              512,      0,    1531,     429,   17988,22077,   0
128 Bucket:            1024,      0,     578,      74,    9757,   2,   0
256 Bucket:            2048,      0,     816,     212,   24319,2571,   0
vmem:                  1856,      0,       3,       1,       3,   0,   0
vmem btag:               56,      0,   18018,    3921,   49688, 155,   0
VM OBJECT:              256,      0,   32467,   29153, 1175586,   0,   0
RADIX NODE:             144,      0,   84577,    4388, 2039774,   0,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             120,      0,      27,     501,      27,   0,   0
MAP ENTRY:              120,      0,    1505,    1003, 2871613,   0,   0
VMSPACE:               2560,      0,      40,      29,   67571,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
64 pcpu:                  8,      0,    2897,     687,    2897,   0,   0
mt_stats_zone:           64,      0,     470,     554,     470,   0,   0
mt_zone:                 24,      0,     470,     699,     470,   0,   0
16:                      16,      0,    5740,    8065,  577599,   0,   0
32:                      32,      0,    6998,   17377,  165207,   0,   0
64:                      64,      0,   29768,   66146, 1457836,   0,   0
128:                    128,      0,   41768,   31485,  930682,   0,   0
256:                    256,      0,    4255,   52655, 1061390,   0,   0
512:                    512,      0,    6471,   16649,  136736,   0,   0
1024:                  1024,      0,     122,     206,   60157,   0,   0
2048:                  2048,      0,     196,      60,  438829,   0,   0
4096:                  4096,      0,     582,      24,  265581,   0,   0
8192:                  8192,      0,      43,       2,     408,   0,   0
16384:                16384,      0,      16,      16,     491,   0,   0
32768:                32768,      0,      14,       7,     620,   0,   0
65536:                65536,      0,      26,       5,    1137,   0,   0
SLEEPQUEUE:              80,      0,     235,     292,     235,   0,   0
Files:                   80,      0,     131,     619, 1005206,   0,   0
filedesc0:             1104,      0,      64,      62,   67597,   0,   0
rangeset pctrie nodes:    144,      0,       0,       0,       0,   0,   0
rl_entry:                40,      0,      93,    1007,      93,   0,   0
TURNSTILE:              136,      0,     235,     205,     235,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
umtx_shm:                88,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1328,      0,      65,      52,   67596,   0,   0
THREAD:                1488,      0,     191,      43,    1021,   0,   0
cpuset:                 104,      0,      16,     573,    5475,   0,   0
domainset:               40,      0,       0,    1054,    5458,   0,   0
audit_record:          1280,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 6475185,     256,    1526,  380257,   0,   0
mbuf:                   256, 6475185,       1,    1528,  667250,   0,   0
mbuf_cluster:          2048, 1011746,    1771,       9,    1771,   0,   0
mbuf_jumbo_page:       4096, 505873,       0,      26,      31,   0,   0
mbuf_jumbo_9k:         9216, 449664,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 337248,       0,       0,       0,   0,   0
epoch_record pcpu:      256,      0,       4,      60,       4,   0,   0
ttyinq:                 160,      0,     255,     120,     555,   0,   0
ttyoutq:                256,      0,     134,     121,     294,   0,   0
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
g_bio:                  376,      0,       0,   11750,  789956,   0,   0
nvme_request:           128,      0,       0,       0,       0,   0,   0
cryptop:                128,      0,       0,       0,       0,   0,   0
cryptodesc:             120,      0,       0,       0,       0,   0,   0
crypto_session:          24,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
VNODE:                  480,      0,   38597,   32747,  200711,   0,   0
VNODEPOLL:              120,      0,       0,       0,       0,   0,   0
BUF TRIE:               144,      0,    5679,   98919,  157900,   0,   0
NAMEI:                 1024,      0,       0,      80, 2611442,   0,   0
rentr:                   24,      0,       0,    1169,     300,   0,   0
S VFS Cache:            108,      0,   31023,   10732,  369981,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,    1564,   12044,   38479,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NCLNODE:                592,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,     857,     107,    2131,   0,   0
procdesc:               136,      0,       1,     144,       2,   0,   0
AIO:                    208,      0,       0,       0,       0,   0,   0
AIOP:                    32,      0,       0,       0,       0,   0,   0
AIOCB:                  752,      0,       0,       0,       0,   0,   0
AIOLIO:                 280,      0,       0,       0,       0,   0,   0
pipe:                   760,      0,       9,      86,   22164,   0,   0
Mountpoints:           2744,      0,      33,       6,      49,   0,   0
ksiginfo:               112,      0,     100,     950,    2518,   0,   0
itimer:                 352,      0,       0,     121,     178,   0,   0
KNOTE:                  160,      0,       0,     250,    7575,   0,   0
socket:                 872, 519552,      39,      53,    8271,   0,   0
unpcb:                  256, 519555,      17,     238,    4596,   0,   0
IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0
ipq:                     56,  31666,       0,     284,       1,   0,   0
udp_inpcb:              488, 519552,       5,     115,    3054,   0,   0
udpcb:                   32, 519625,       5,    1120,    3054,   0,   0
tcp_inpcb:              488, 519552,      12,     140,     565,   0,   0
tcpcb:                  976, 519552,      12,      56,     565,   0,   0
tcptw:                   88,  27810,       0,     270,      58,   0,   0
syncache:               168,  15364,       0,      69,       1,   0,   0
hostcache:               96,  15375,       4,     160,       6,   0,   0
sackhole:                32,      0,       0,     375,       3,   0,   0
tfo:                      4,      0,       0,       0,       0,   0,   0
tfo_ccache_entries:      80,      0,       0,       0,       0,   0,   0
tcpreass:                48,  63246,       0,     332,       3,   0,   0
tcp_log:                400, 1000000,       0,       0,       0,   0,   0
tcp_log_bucket:         144,      0,       0,       0,       0,   0,   0
tcp_log_node:           120,      0,       0,       0,       0,   0,   0
sctp_ep:               1280, 519552,       0,       0,       0,   0,   0
sctp_asoc:             2408,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     581,      13,   0,   0
sctp_raddr:             736,  80000,       0,       0,       0,   0,   0
sctp_chunk:             152, 400010,       0,       0,       0,   0,   0
sctp_readq:             152, 400010,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400015,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
udplite_inpcb:          488, 519552,       0,       0,       0,   0,   0
ripcb:                  488, 519552,       3,      61,       3,   0,   0
rtentry:                208,      0,      63,     184,      66,   0,   0
selfd:                   64,      0,     154,     900, 1717806,   0,   0
swpctrie:               144, 2023515,       0,       0,       0,   0,   0
swblk:                  136, 2023504,       0,       0,       0,   0,   0
FFS inode:              160,      0,   27373,    7052,   86551,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,   27373,    7037,   86550,   0,   0
nvidia_stack_t:       12288,      0,       1,       1,       3,   0,   0
TMPFS dirent:            64,      0,    7867,   41485,   54027,   0,   0
TMPFS node:             232,      0,    7923,   41122,   54028,   0,   0
TMPFS dirent:            64,      0,       0,       0,       0,   0,   0
TMPFS node:             232,      0,       1,      67,       1,   0,   0
TMPFS dirent:            64,      0,       1,    1053,      79,   0,   0
TMPFS node:             232,      0,      41,     214,      80,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq16: ehci0                       80299       1444
irq23: ehci1                      144788       2604
cpu0:timer                       1152857      20732
cpu1:timer                        801388      14412
cpu2:timer                        715379      12865
cpu3:timer                        718580      12922
irq264: hdac0                          3          0
irq265: hdac1                          6          0
irq266: xhci0                    1407762      25316
irq267: hdac2                         56          1
irq268: re0                       235923       4243
irq269: ahci0                     261951       4711
irq270: vgapci0                      608         11
Total                            5519600      99261

------------------------------------------------------------------------
pstat -T

131/519551 files
0M/0M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity

------------------------------------------------------------------------
iostat

       tty            ada0              da0              da1             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0     6 38.92   5  0.19   0.00   0  0.00   0.00   0  0.00   0  0  0  0 100

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
        msgmax:        16384    (max characters in a message)
        msgmni:           40    (# of message queues)
        msgmnb:         2048    (max characters in a message queue)
        msgtql:           40    (max # of messages in system)
        msgssz:            8    (size of a message segment)
        msgseg:         2048    (# of message segments in system)

shminfo:
        shmmax:    536870912    (max shared memory segment size)
        shmmin:            1    (min shared memory segment size)
        shmmni:          192    (max number of shared memory identifiers)
        shmseg:          128    (max shared memory segments per process)
        shmall:       131072    (max amount of shared memory in pages)

seminfo:
        semmni:           50    (# of semaphore identifiers)
        semmns:          340    (# of semaphores in system)
        semmnu:          150    (# of undo structures in system)
        semmsl:          340    (max # of semaphores per id)
        semopm:          100    (max # of operations per semop call)
        semume:           50    (max # of undo entries per process)
        semusz:          632    (size in bytes of undo structure)
        semvmx:        32767    (semaphore maximum value)
        semaem:        16384    (adjust on exit max value)


------------------------------------------------------------------------
netstat -s

tcp:
        96240 packets sent
                13557 data packets (2131686 bytes)
                8 data packets (8962 bytes) retransmitted
                0 data packets unnecessarily retransmitted
                0 resends initiated by MTU discovery
                81258 ack-only packets (3366 delayed)
                0 URG only packets
                0 window probe packets
                317 window update packets
                1100 control packets
        177922 packets received
                13722 acks (for 2131715 bytes)
                12 duplicate acks
                0 acks for unsent data
                164550 packets (217025714 bytes) received in-sequence
                0 completely duplicate packets (0 bytes)
                0 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                11 out-of-order packets (8895 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                6 window update packets
                433 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
                0 discarded due to memory problems
        551 connection requests
        1 connection accept
        0 bad connection attempts
        0 listen queue overflows
        0 ignored RSTs in the windows
        552 connections established (including accepts)
                538 times used RTT from hostcache
                538 times used RTT variance from hostcache
                0 times used slow-start threshold from hostcache
        553 connections closed (including 2 drops)
                544 connections updated cached RTT on close
                544 connections updated cached RTT variance on close
                0 connections updated cached ssthresh on close
        0 embryonic connections dropped
        13722 segments updated rtt (of 13900 attempts)
        2 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 Connections (fin_wait_2) dropped because of timeout
        0 keepalive timeouts
                0 keepalive probes sent
                0 connections dropped by keepalive
        9307 correct ACK header predictions
        163045 correct data packet header predictions
        1 syncache entry added
                0 retransmitted
                0 dupsyn
                0 dropped
                1 completed
                0 bucket overflow
                0 cache overflow
                0 reset
                0 stale
                0 aborted
                0 badack
                0 unreach
                0 zone failures
        1 cookie sent
        0 cookies received
        6 hostcache entries added
                0 bucket overflow
        1 SACK recovery episode
        6 segment rexmits in SACK recovery episodes
        8244 byte rexmits in SACK recovery episodes
        11 SACK options (SACK blocks) received
        11 SACK options (SACK blocks) sent
        0 SACK scoreboard overflow
        0 packets with ECN CE bit set
        0 packets with ECN ECT(0) bit set
        0 packets with ECN ECT(1) bit set
        0 successful ECN handshakes
        0 times ECN reduced the congestion window
        0 packets with matching signature received
        0 packets with bad signature received
        0 times failed to make signature due to no SA
        0 times unexpected signature received
        0 times no signature provided by segment
        0 Path MTU discovery black hole detection activations
        0 Path MTU discovery black hole detection min MSS activations
        0 Path MTU discovery black hole detection failures
TCP connection count by state:
        0 connections in CLOSED state
        9 connections in LISTEN state
        0 connections in SYN_SENT state
        0 connections in SYN_RCVD state
        3 connections in ESTABLISHED state
        0 connections in CLOSE_WAIT state
        0 connections in FIN_WAIT_1 state
        0 connections in CLOSING state
        0 connections in LAST_ACK state
        0 connections in FIN_WAIT_2 state
        0 connections in TIME_WAIT state
udp:
        6713 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        3 with no checksum
        17 dropped due to no socket
        119 broadcast/multicast datagrams undelivered
        0 dropped due to full socket buffers
        0 not for hashed pcb
        6577 delivered
        9861 datagrams output
        0 times multicast source filter matched
ip:
        17123 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        2 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        1 packet reassembled ok
        17120 packets for this host
        0 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        2 packets not forwardable
        0 packets received for unknown multicast group
        0 redirects sent
        18885 packets sent from this host
        7 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        823 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        17 calls to icmp_error
        0 errors not generated in response to an icmp message
        Output histogram:
                destination unreachable: 17
        0 messages with bad code fields
        0 messages less than the minimum length
        0 messages with bad checksum
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        0 message responses generated
        0 invalid return addresses
        0 no return routes
ipsec:
        0 inbound packets violated process security policy
        0 inbound packets failed due to insufficient memory
        0 invalid inbound packets
        0 outbound packets violated process security policy
        0 outbound packets with no SA available
        0 outbound packets failed due to insufficient memory
        0 outbound packets with no route available
        0 invalid outbound packets
        0 outbound packets with bundled SAs
        0 spd cache hits
        0 spd cache misses
        0 clusters copied during clone
        0 mbufs inserted during makespace
ah:
        0 packets shorter than header shows
        0 packets dropped; protocol family not supported
        0 packets dropped; no TDB
        0 packets dropped; bad KCR
        0 packets dropped; queue full
        0 packets dropped; no transform
        0 replay counter wraps
        0 packets dropped; bad authentication detected
        0 packets dropped; bad authentication length
        0 possible replay packets detected
        0 packets in
        0 packets out
        0 packets dropped; invalid TDB
        0 bytes in
        0 bytes out
        0 packets dropped; larger than IP_MAXPACKET
        0 packets blocked due to policy
        0 crypto processing failures
        0 tunnel sanity check failures
esp:
        0 packets shorter than header shows
        0 packets dropped; protocol family not supported
        0 packets dropped; no TDB
        0 packets dropped; bad KCR
        0 packets dropped; queue full
        0 packets dropped; no transform
        0 packets dropped; bad ilen
        0 replay counter wraps
        0 packets dropped; bad encryption detected
        0 packets dropped; bad authentication detected
        0 possible replay packets detected
        0 packets in
        0 packets out
        0 packets dropped; invalid TDB
        0 bytes in
        0 bytes out
        0 packets dropped; larger than IP_MAXPACKET
        0 packets blocked due to policy
        0 crypto processing failures
        0 tunnel sanity check failures
ipcomp:
        0 packets shorter than header shows
        0 packets dropped; protocol family not supported
        0 packets dropped; no TDB
        0 packets dropped; bad KCR
        0 packets dropped; queue full
        0 packets dropped; no transform
        0 replay counter wraps
        0 packets in
        0 packets out
        0 packets dropped; invalid TDB
        0 bytes in
        0 bytes out
        0 packets dropped; larger than IP_MAXPACKET
        0 packets blocked due to policy
        0 crypto processing failures
        0 packets sent uncompressed; size < compr. algo. threshold
        0 packets sent uncompressed; compression was useless
arp:
        4 ARP requests sent
        1412 ARP replies sent
        12674 ARP requests received
        1 ARP reply received
        12675 ARP packets received
        0 total packets dropped due to no ARP entry
        0 ARP entrys timed out
        0 Duplicate IPs seen
ip6:
        168638 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        168089 packets for this host
        0 packets forwarded
        0 packets not forwardable
        0 redirects sent
        88573 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        12 multicast packets which we don't join
        Input histogram:
                hop by hop: 11
                TCP: 167939
                ICMP6: 688
        Mbuf statistics:
                12 one mbuf
                168626 one ext mbuf
                0 two or more ext mbuf
        0 packets whose headers are not contiguous
        0 tunneling packets that can't find gif
        0 packets discarded because of too many headers
        0 failures of source address selection
        source addresses on an outgoing I/F
                182 link-locals
                1465 globals
        source addresses of same scope
                181 link-locals
                1465 globals
        source addresses of a different scope
                1 link-local
        Source addresses selection rule applied:
                1647 first candidate
                1465 appropriate scope
icmp6:
        0 calls to icmp6_error
        0 errors not generated in response to an icmp6 message
        0 errors not generated because of rate limitation
        Output histogram:
                router solicitation: 1
                neighbor solicitation: 367
                neighbor advertisement: 187
                MLDv2 listener report: 13
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Input histogram:
                router advertisement: 138
                neighbor solicitation: 187
                neighbor advertisement: 362
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 bad redirect messages
        0 path MTU changes
ipsec6:
        0 inbound packets violated process security policy
        0 inbound packets failed due to insufficient memory
        0 invalid inbound packets
        0 outbound packets violated process security policy
        0 outbound packets with no SA available
        0 outbound packets failed due to insufficient memory
        0 outbound packets with no route available
        0 invalid outbound packets
        0 outbound packets with bundled SAs
        0 spd cache hits
        0 spd cache misses
        0 clusters copied during clone
        0 mbufs inserted during makespace
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output
pfkey:
        0 requests sent from userland
        0 bytes sent from userland
        0 messages with invalid length field
        0 messages with invalid version field
        0 messages with invalid message type field
        0 messages too short
        0 messages with memory allocation failure
        0 messages with duplicate extension
------------------------------------------------------------------------
netstat -m

257/3054/3311 mbufs in use (current/cache/total)
245/1535/1780/1011746 mbuf clusters in use (current/cache/total/max)
256/1526 mbuf+clusters out of packet secondary zone in use (current/cache)
0/26/26/505873 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/449664 9k jumbo clusters in use (current/cache/total/max)
0/0/0/337248 16k jumbo clusters in use (current/cache/total/max)
554K/3937K/4491K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

------------------------------------------------------------------------
Comment 1 Dirk Meyer freebsd_committer freebsd_triage 2020-01-27 07:28:58 UTC
some Kernel modules removed:

FreeBSD netcom2.dinoex.sub.de 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 r357113 GENERIC  amd64

panic: bad pte va 1180000 pte fffff80081787bf8

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: bad pte va 1180000 pte fffff80081787bf8
cpuid = 1
time = 1580092419
KDB: stack backtrace:
#0 0xffffffff80c1d297 at kdb_backtrace+0x67
#1 0xffffffff80bd05cd at vpanic+0x19d
#2 0xffffffff80bd0423 at panic+0x43
#3 0xffffffff81099fb7 at pmap_remove_pages+0x8b7
#4 0xffffffff80f1171c at vmspace_exit+0x9c
#5 0xffffffff80b8a509 at exit1+0x5d9
#6 0xffffffff80b89f2d at sys_sys_exit+0xd
#7 0xffffffff810a8984 at amd64_syscall+0x364
#8 0xffffffff81082330 at fast_syscall_common+0x101
Uptime: 2h21m57s
Dumping 1413 out of 16235 MB:..2%..11%..21%..31%..41%..51%..62%..71%..81%..91%

Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/linprocfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux_common.ko
Reading symbols from /boot/kernel/linsysfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/linsysfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linsysfs.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
done.
Loaded symbols for /boot/kernel/uhid.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux64.ko
Reading symbols from /boot/kernel/speaker.ko...Reading symbols from /usr/lib/debug//boot/kernel/speaker.ko.debug...done.
done.
Loaded symbols for /boot/kernel/speaker.ko
Reading symbols from /boot/kernel/sem.ko...Reading symbols from /usr/lib/debug//boot/kernel/sem.ko.debug...done.
done.
Loaded symbols for /boot/kernel/sem.ko
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/tmpfs.ko
Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
#0  doadump () at src/sys/amd64/include/pcpu.h:234
234             __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) #0  doadump () at src/sys/amd64/include/pcpu.h:234
#1  0xffffffff80bd01c8 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff80bd0629 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:877
#3  0xffffffff80bd0423 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:804
#4  0xffffffff81099fb7 in pmap_remove_pages (pmap=<value optimized out>)
    at /usr/src/sys/amd64/amd64/pmap.c:6932
#5  0xffffffff80f1171c in vmspace_exit (td=0xfffff80315f12000)
    at /usr/src/sys/vm/vm_map.c:409
#6  0xffffffff80b8a509 in exit1 (td=0xfffff80315f12000, 
    rval=<value optimized out>, signo=0) at /usr/src/sys/kern/kern_exit.c:396
#7  0xffffffff80b89f2d in sys_sys_exit (td=<value optimized out>, 
    uap=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:175
#8  0xffffffff810a8984 in amd64_syscall (td=0xfffff80315f12000, traced=0)
    at src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#9  0xffffffff81082330 in fast_syscall_common ()
    at /usr/src/sys/amd64/amd64/exception.S:581
#10 0x00000008029656aa in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------
ps -axlww

  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN   STAT TT     TIME COMMAND
    0     0     0   0 -16  0     0     0 swapin   DLs   -  0:00.02 [kernel]
    0     1     0   0  25  0  9960   896 wait     DLs   -  0:00.00 [init]
    0     2     0   0 -16  0     0     0 crypto_w DL    -  0:00.00 [crypto]
    0     3     0   0 -16  0     0     0 crypto_r DL    -  0:00.00 [crypto returns 0]
    0     4     0   0 -16  0     0     0 crypto_r DL    -  0:00.00 [crypto returns 1]
    0     5     0   0 -16  0     0     0 crypto_r DL    -  0:00.00 [crypto returns 2]
    0     6     0   0 -16  0     0     0 crypto_r DL    -  0:00.00 [crypto returns 3]
    0     7     0   0 -16  0     0     0 -        RL    -  0:00.00 [cam]
    0     8     0   0 -16  0     0     0 -        DL    -  0:00.00 [soaiod1]
    0     9     0   0 -16  0     0     0 -        DL    -  0:00.00 [soaiod2]
    0    10     0   0 -16  0     0     0 audit_wo DL    -  0:00.00 [audit]
    0    11     0   0 155  0     0     0 -        RL    -  3:43.29 [idle]
    0    12     0   0 -56  0     0     0 -        WL    -  0:00.72 [intr]
    0    13     0   0  -8  0     0     0 -        DL    -  0:00.00 [geom]
    0    14     0   0 -16  0     0     0 seqstate DL    -  0:00.00 [sequencer 00]
    0    15     0   0 -68  0     0     0 -        DL    -  0:00.02 [usb]
    0    16     0   0 -16  0     0     0 -        DL    -  0:00.00 [soaiod3]
    0    17     0   0 -16  0     0     0 -        DL    -  0:00.00 [soaiod4]
    0    18     0   0 -16  0     0     0 waiting_ DL    -  0:00.00 [sctp_iterator]
    0    19     0   0 -16  0     0     0 tzpoll   DL    -  0:00.00 [acpi_thermal]
    0    20     0   0 -16  0     0     0 cooling  DL    -  0:00.00 [acpi_cooling1]
    0    21     0   0 -16  0     0     0 -        DL    -  0:00.01 [rand_harvestq]
    0    22     0   0 -16  0     0     0 psleep   DL    -  0:00.00 [pagedaemon]
    0    23     0   0 -16  0     0     0 psleep   DL    -  0:00.00 [vmdaemon]
    0    24     0   0  20  0     0     0 qsleep   DL    -  0:00.02 [bufdaemon]
    0    25     0   0  20  0     0     0 vlruwt   DL    -  0:00.00 [vnlru]
    0    26     0   0  16  0     0     0 syncer   DL    -  0:00.00 [syncer]
    0   550     1   0  52  0 11048  2328 select   Ds    -  0:00.00 [moused]
    0   579     1   0  20  0 11388  2620 select   Ds    -  0:00.00 [dhclient]
    0   582     1   0  39  0 11612  2760 select   Ds    -  0:00.00 [dhclient]
   65   628     1   0  20  0 11752  2876 select   DCs   -  0:00.00 [dhclient]
    0   629     1   0  20  0 10456  1476 select   Ds    -  0:00.00 [devd]
    0   637     1   0  20  0 16092  5500 select   Ds    -  0:00.01 [tincd]
    0   711     1   0  20  0 10908  2216 select   Ds    -  0:00.00 [rtsold]
    0   732     1   0  20  0 11400  2668 select   Ds    -  0:00.00 [syslogd]
    0   946     1   0  52  0 11084  2452 select   Ds    -  0:00.00 [lpd]
   25   982     1   0  20  0 16800  6368 pause    Ds    -  0:00.00 [sendmail]
    0   995     1   0  20  0 17156  6896 select   Ds    -  0:00.01 [openvpn]
    0  1006     1   0  20  0 17076  6984 select   Ds    -  0:00.01 [openvpn]
    0  1048     1   0  20  0 19652  8372 select   Ds    -  0:00.00 [sshd]
    0  1051     1   0  43  0 11464  2676 nanslp   Ds    -  0:00.00 [cron]
    0  1127     1   0  52  0 11444  2600 select   Ds    -  0:00.00 [inetd]
    0  1152     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1153     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1154     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1155     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1156     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1157     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1158     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1159     1   0  52  0 10880  2288 ttyin    Ds+   -  0:00.00 [getty]
    0  1199  1048   0  20  0 20984  9536 select   Ds    -  0:00.00 [sshd]
    0  1201  1199   0  20  0 14408  4652 pause    Ds    -  0:00.00 [csh]
    0  1207  1199   0  20  0 14448  4720 ttyin    Ds+   -  0:00.00 [csh]
    0  1225  1201   0  21  0 13204  3868 select   D+    -  0:00.00 [tmux]
    0  1227     1   0  20  0 14048  5020 select   Ds    -  0:00.00 [tmux]
    0  1228  1227   0  20  0 13608  4344 pause    Ds    -  0:00.01 [csh]
    0  1356  1228   0  20  0 12604  3908 select   D+    -  0:01.01 [sh]
    0  1668  1356   0  52  0 13156  4180 nanslp   D+    -  0:08.80 [sh]
    0  4977 96509   0  52  0 11796  2988 piperd   D     -  0:00.00 [sh]
    0  4980  4977   0  52  0 11796  2984 wait     D     -  0:00.00 [sh]
    0  4981  4980   0  52  0 19864  9060 wait     D     -  0:00.00 [pkg]
    0  4982  4980   0  52  0 10912  2228 piperd   D     -  0:00.00 [sed]
    0  4983  4980   0  52  0 10764  2148 piperd   DC    -  0:00.00 [tee]
    0  4984  4980   0  52  0 11052  2400 piperd   D     -  0:00.00 [wc]
    0  4985  4981   0  33  0 28512 18072 -        R     -  0:00.00 [pkg]
    0 22405  1356   0  26  0 12604  3832 piperd   D+    -  0:00.23 [sh]
    0 30973  1356   0  42  0 12604  3912 select   D     -  0:00.00 [sh]
    0 32926 30973   0  43  0 12604  3908 wait     D     -  0:00.00 [sh]
65534 32927 32926   0  52  0 10480  1824 wait     D     -  0:00.00 [make]
65534 33025 32927   0  52  0 11648  3148 wait     D     -  0:00.00 [sh]
65534 33028 33025   0  52  0 11976  3220 wait     D     -  0:00.00 [sh]
65534 33135 33028   0  52  0 21020 11944 piperd   D     -  0:00.00 [qmake]
65534 33412 33135   0  52  0 10024  1320 wait     D     -  0:00.00 [make]
65534 33450 33412   0  52  0 72556 63576 wait     D     -  0:00.00 [clang++]
65534 33451 33450   0  52  0 66924 29948 -        RE    -  0:00.00 [ld.lld]
    0 94762  1051   0  20  0 11464  2684 piperd   D     -  0:00.00 [cron]
    0 94764 94762   0  22  0 11700  2968 wait     Ds    -  0:00.00 [sh]
    0 94766 94764   0  22  0 10680  2128 wait     D     -  0:00.00 [lockf]
    0 94767 94766   0  52  0 11700  2968 wait     D     -  0:00.00 [sh]
    0 94775 94767   0  52  0 11752  2972 wait     D     -  0:00.00 [sh]
    0 94776 94767   0  27  0 10892  2328 piperd   D     -  0:00.00 [mail]
    0 96487 94775   0  52  0 11712  2972 wait     D     -  0:00.00 [sh]
    0 96491 96487   0  52  0 11704  2968 wait     D     -  0:00.00 [sh]
    0 96496 96491   0  52  0 10680  2128 wait     D     -  0:00.00 [lockf]
    0 96498 96496   0  52  0 11704  2968 wait     D     -  0:00.00 [sh]
    0 96509 96498   0  52  0 11752  2972 wait     D     -  0:00.00 [sh]
    0 96510 96498   0  20  0 10892  2328 piperd   D     -  0:00.00 [mail]

------------------------------------------------------------------------
vmstat -s

 23471056 cpu context switches
  4280481 device interrupts
   148337 software interrupts
158386910 traps
172407235 system calls
       26 kernel threads created
   706241  fork() calls
   225962 vfork() calls
        2 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     3724 vnode pager pageins
    15594 vnode pager pages paged in
     8582 vnode pager pageouts
   167467 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 clean page reclamation shortfalls
        0 pages reactivated by the page daemon
 23915075 copy-on-write faults
    24024 copy-on-write optimized faults
126595742 zero fill pages zeroed
        0 zero fill pages prezeroed
      119 intransit blocking page faults
158210567 total VM faults taken
     3238 page faults requiring I/O
        0 pages affected by kernel thread creation
 26828679 pages affected by  fork()
 46839162 pages affected by vfork()
      114 pages affected by rfork()
193659777 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
        0 pages active
        0 pages inactive
        0 pages in the laundry queue
        0 pages wired down
        0 pages free
        0 bytes per page
        0 total name lookups
          cache hits (0% pos + 0% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
      CAM DEV    10    20K       -       12  2048
      entropy     1     1K       -    41937  32,4096
      CAM CCB     0     0K       -  3841585  2048
     CAM path    17     1K       -       66  32
   CAM periph    16     4K       -    14171  16,32,64,128,256
    acpi_perf     4     2K       -        4  512
      acpipwr    11     1K       -       11  64
         hdaa    14    57K       -       14  256,512,1024,2048,16384,32768
         hdac     3     2K       -        3  128,256,1024
        hdacc     3     1K       -        3  32
       feeder    21     2K       -       27  32,128
        mixer     4    16K       -        4  4096
CAM I/O Scheduler     6     1K       -        6  128
    CAM queue    14     6K       -       26  16,32,64,512
      scsi_cd     0     0K       -        6  16
          USB    60    92K       -       67  16,32,128,256,512,1024,2048,4096,8192,16384
       USBdev    44    10K       -       44  32,64,128,256,512,4096
CAM dev queue     4     1K       -        4  64
      scsi_da     0     0K       -       60  16,256
        vtbuf    24  4752K       -       94  4096
           vt    11     6K       -       11  512
       DEVFS2   131     5K       -      262  16,32
       DEVFS3   476   119K       -      512  256
       DEVFS1   131    66K       -      170  512
   DEVFS_RULE    65    31K       -       65  64,512
        DEVFS    53     2K       -       54  16,128
       DEVFSP     3     1K       -        3  64
NFSD V4client     1     1K       -        1  256
 NFSD lckfile     1     1K       -        1  256
 NFSD session     1     1K       -        1  1024
    pfs_nodes   388   194K       -      388  512
  pfs_vncache     8     1K       -       63  64
         GEOM   139    27K       -     1293  16,32,64,128,256,512,1024,2048,8192,16384,32768
    raid_data     0     0K       -      174  32,128,256
       isadev     8     1K       -        8  128
         cdev     5     2K       -        5  256
     filedesc     1     4K       -       68  32,64,4096,16384,32768
        sigio     0     0K       -        1  64
     filecaps     4     1K       -    52378  16,32,64
      kdtrace   250    47K       -  1882747  64,256
         kenv    95    12K       -       98  16,32,64,128,8192
       kqueue    87     6K       -   941658  64,256,512,2048
    proc-args    82     5K       -   535938  16,32,64,128,256
        hhook    13     4K       -       13  256
      ithread    95    16K       -       96  32,128,256
       prison    26    18K       -       26  16,32,256,4096
       KTRACE   100    13K       -      100  128
       linker   384  1172K       -      684  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        lockf    42     5K       -   105108  64,128
   loginclass     3     1K       -        3  64
        cache     1     1K       -        1  32
       devbuf 18066 37897K       -    19638  16,32,64,128,256,512,1024,2048,4096,8192,65536
         temp    50    18K       - 14416436  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        evdev     5     5K       -        5  1024
       module   516    65K       -      533  128
     mtx_pool     2    72K       -        2  8192,65536
          osd     7     1K       -       26  16,32,64,128,256
     pmchooks     1     1K       -        1  128
          pmc     1     1K       -        1  64
         pgrp    33     5K       -    37111  128
      session    30     4K       -      138  128
         proc     2    64K       -        2  32768
      subproc   223   416K       -   932397  512,4096
         cred    62    16K       -    73042  256
     acpiintr     1     1K       -        1  64
       plimit    25     7K       -    94735  256
      uidinfo     6     9K       -       64  128,8192
       dumper     1     1K       -        1  512
       acpica  6964   688K       -   195267  16,32,64,128,256,512,1024,2048,4096
       sysctl     0     0K       -       18  64
    sysctloid  5190   265K       -     6392  16,32,64,128
    sysctltmp     0     0K       -    72973  16,64,256,1024
     acpitask     1    64K       -        1  65536
      tidhash     1    64K       -        1  65536
      callout     5  2184K       -        5  
         umtx   524    66K       -      524  128
     p1003.1b     1     1K       -        1  16
         SWAP     1   232K       -        1  
          bus  1379   120K       -    42352  16,32,64,128,256,1024,4096
       bus-sc    84  1910K       -     3733  16,32,128,256,512,1024,2048,4096,8192,16384,32768,65536
      acpisem   114    15K       -      114  128
      devstat    10    21K       -       10  32,4096
        epoch     4     1K       -        4  128
 eventhandler   126    11K       -      127  64,128
   gtaskqueue    34    35K       -       34  16,32,256,8192
         kobj   348  1392K       -      657  4096
      Per-cpu     1     1K       -        1  32
       kbdmux     7    22K       -        7  16,512,1024,2048,16384
         rman   243    28K       -      670  32,128
         sbuf     0     0K       -     1094  16,32,64,128,256,4096,8192,16384,32768
    toponodes    18     3K       -       18  128
    taskqueue    33     4K       -       39  16,32,256
     terminal    11     3K       -       11  256
       Unitno    29     2K       -    34599  32,64
         vmem     3   260K       -       21  2048,4096,8192,16384,32768,65536
     ioctlops     0     0K       -      420  256,512,1024,2048
       select   107    14K       -      107  128
          iov     1     1K       - 10212321  16,32,64,128,256,512,4096
          msg     4    30K       -        4  2048,4096,8192,16384
          sem     4   106K       -        4  2048,4096
          shm     1    32K       -        1  32768
          tty    15    15K       -       15  1024
          pts     3     1K       -        3  256
     mbuf_tag     0     0K       -      200  32
        shmfd     1     8K       -        1  8192
       soname     6     1K       -   111707  16,32,128
          pcb    32  2201K       -     1077  16,32,64,128,1024,2048,8192
          acl     0     0K       -   102961  4096
     vfscache     4  4145K       -        4  256,16384,32768
   cl_savebuf     0     0K       -    13864  64
     vfs_hash     1  2048K       -        1  
       vnodes     4     1K       -     1360  32,256
        mount   476    25K       -    43455  16,32,64,128,256,1024
       statfs     0     0K       -  1610811  4096
  vnodemarker     0     0K       -     5178  512
chacha20random     1     1K       -        1  1024
          BPF    13    18K       -       13  128,512,4096
        ifnet     6    11K       -        6  128,2048
       ifaddr    72    21K       -       72  16,32,64,128,256,512,2048,4096
  ether_multi    76     6K       -       76  16,32,64,128
        clone     7     1K       -        7  128
        ipsec     3     1K       -        3  256
      lltable    28     9K       -       28  256,512
          tun     3     1K       -        3  256
     routetbl   100     9K       -      109  32,64,128,512
         vnet     1     1K       -        1  64
    vnet_data     1   240K       -        1  
vnet_data_free     1     1K       -        1  32
         igmp     5     1K       -        5  128
     in_multi     5     2K       -        5  256
encap_export_host    12     1K       -       12  32,64
    sctp_a_it     0     0K       -        8  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa    14     2K       -       14  128
     sctp_ifn     5     1K       -        5  128
    sctp_iter     0     0K       -        8  256
   tfo_ccache     1   128K       -        1  
    hostcache     1    32K       -        1  32768
      tcpfunc     1     1K       -        1  64
     syncache     1    68K       -        1  
    in6_multi    60     8K       -       60  32,256
       ip6opt     1     1K       -        2  32,256
          mld     5     1K       -        5  128
       ip6ndp    13     2K       -       18  32,64,256
  inpcbpolicy    20     1K       -     1364  32
     secasvar     1     1K       -        1  1024
       sahead     1     1K       -        1  1024
  ipsecpolicy     2     2K       -        2  256,1024
    ipsec-saq     2     2K       -        2  1024
       crypto     2     2K       -        2  512,1024
          rpc     2     8K       -        2  4096
audit_evclass   230     8K       -      285  32
      pagedep  1316   841K       -   170334  256
     inodedep 18603 11349K       -  1458476  512
    bmsafemap    12    11K       -  1112239  256,8192
       newblk  6192  5644K       -  1900939  256
     indirdep    60     8K       -     6079  128,32768
     freefrag     2     1K       -    61705  128
     freeblks   323    81K       -  1048986  256
     freefile    16     1K       -  1029900  64
       diradd 18521  2316K       -  1605777  128
        mkdir   995   125K       -   272436  128
       dirrem    20     3K       -  1577774  128
    newdirblk   988    62K       -   140455  64
     freework   419    53K       -  1268677  64,128
      freedep     0     0K       -     4730  64
      jaddref     1     1K       -  1878213  128
      jremref     1     1K       -  1845374  128
       jmvref     0     0K       -    23401  128
      jnewblk     1     1K       -  1900938  128
    jfreefrag     0     0K       -    61705  128
         jseg 20524  2566K       -   538964  128
      jsegdep 25931  1621K       -  5686230  64
        sbdep     0     0K       -      261  64
     savedino     0     0K       -   108216  256
      jblocks     2     1K       -        2  128,256
      softdep     1     1K       -        1  512
  ufs_dirhash  8550  2295K       -   602673  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
    ufs_quota     1  2048K       -        1  
    ufs_mount     3    41K       -        5  512,8192,32768
    vm_pgdata     1     1K       -        1  128
      UMAHash     5    96K       -       23  512,1024,2048,4096,8192,16384,32768,65536
      memdesc     1     4K       -        1  4096
     pci_link    16     2K       -       16  64,128
     atkbddev     2     1K       -        2  64
      acpidev    61     4K       -       61  64
      CAM SIM     4     1K       -        4  256
      CAM XPT    46     4K       -      123  16,32,64,128,256,512,1024,2048,65536
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        2  64,4096
         intr     4   400K       -        4  65536
      io_apic     1     2K       -        1  2048
   local_apic     1     8K       -        1  8192
          MCA    14     3K       -       14  32,128,256
         cpus     2     1K       -        2  32
          msi    15     2K       -       15  128
     nexusdev     5     1K       -        5  16
        linux    31     2K       -       32  64
         ksem     1     8K       -        1  8192
  nullfs_hash     1  2048K       -        1  
  nullfs_node   718    45K       -     5154  64
 nullfs_mount    17     1K       -      150  32
  tmpfs mount     3     1K       -        3  256
   tmpfs name  5252   212K       -    58195  16,32,64,128
  fdesc_mount     2     1K       -        2  16

------------------------------------------------------------------------
Comment 2 Mark Johnston freebsd_committer freebsd_triage 2020-01-27 15:22:18 UTC
This looks a bit like a page table page is being freed while it is still in use.  Is there something specific that triggers this?  Which process is exiting during the crash?
Comment 3 Dirk Meyer freebsd_committer freebsd_triage 2020-01-27 20:17:18 UTC
running poudriere bulk -J1
Comment 4 Mark Johnston freebsd_committer freebsd_triage 2020-01-27 20:24:27 UTC
(In reply to Dirk Meyer from comment #3)
Are you able to try running with INVARIANTS enabled?
Comment 5 Dirk Meyer freebsd_committer freebsd_triage 2020-02-03 15:46:02 UTC
running gerneric Kernel only

I could reproduce the problems until I disabled journaling on the root fs (ufs, softupdates)