Bug 188162

Summary: Fatal trap 12: page fault while in kernel mode (in function pmap_remove_pages)
Product: Base System Reporter: António Trindade <Antonio.Trindade>
Component: i386Assignee: Graham Perrin <grahamperrin>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: chris, grahamperrin, i386
Priority: Normal Keywords: crash, needs-qa
Version: 10.0-RELEASE   
Hardware: Any   
OS: Any   

Description António Trindade 2014-04-01 10:20:00 UTC 
I've had several kernel panix in the last few hours (from 4:24 AM till 8:27 AM, I got 5).

I will gladly share the kernel core dump if requested.

Next is /var/crash/core.txt.4:
gatekeeper.darklair.homeunix.net dumped core - see /var/crash/vmcore.4

Tue Apr  1 08:27:13 WEST 2014

FreeBSD gatekeeper.darklair.homeunix.net 10.0-RELEASE FreeBSD 10.0-RELEASE #1 r263747: Thu Mar 27 00:17:42 WET 2014     root@gatekeeper.darklair.homeunix.net:/usr/obj/usr/src/sys/GATEKEEPER  i386

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xbfc00464
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc09df266
stack pointer           = 0x28:0xf83ba8a8
frame pointer           = 0x28:0xf83ba908
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 3096 (sh)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xc075d3b2 at kdb_backtrace+0x52
#1 0xc07232c1 at panic+0x121
#2 0xc09e3e39 at trap_fatal+0x339
#3 0xc09e41f5 at trap_pfault+0x3a5
#4 0xc09e3780 at trap+0x530
#5 0xc09cdb0c at calltrap+0x6
#6 0xc06eb2ea at exec_new_vmspace+0x1da
#7 0xc06cc9d3 at exec_elf32_imgact+0x553
#8 0xc06e97cb at kern_execve+0x73b
#9 0xc06e8e73 at sys_execve+0x53
#10 0xc09e4849 at syscall+0x479
#11 0xc09cdba1 at Xint0x80_syscall+0x21
Uptime: 12m38s
Physical memory: 3059 MB
Dumping 379 MB: 364 348 332 316 300 284 268 252 236 220 204 188 172 156 140 124 108 92 76 60 44 28 12

Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
#0  doadump (textdump=-949394560) at pcpu.h:233
233     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=-949394560) at pcpu.h:233
#1  0xc0722f39 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xc07232ff in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xc09e3e39 in trap_fatal (frame=<value optimized out>, eva=3217032292)
    at /usr/src/sys/i386/i386/trap.c:1047
#4  0xc09e41f5 in trap_pfault (frame=0x0, usermode=0, eva=0)
    at /usr/src/sys/i386/i386/trap.c:859
#5  0xc09e3780 in trap (frame=0xf83ba868) at /usr/src/sys/i386/i386/trap.c:556
#6  0xc09cdb0c in calltrap () at /usr/src/sys/i386/i386/exception.s:170
#7  0xc09df266 in pmap_remove_pages (pmap=<value optimized out>)
    at /usr/src/sys/i386/i386/pmap.c:4457
#8  0xc06eb2ea in exec_new_vmspace (imgp=<value optimized out>, sv=0xc0b0d150)
    at /usr/src/sys/kern/kern_exec.c:1049
#9  0xc06cc9d3 in exec_elf32_imgact (imgp=0xf83baad0)
    at /usr/src/sys/kern/imgact_elf.c:819
#10 0xc06e97cb in kern_execve (td=0x28802000, args=0x28807000)
    at /usr/src/sys/kern/kern_exec.c:501
#11 0xc06e8e73 in sys_execve (td=<value optimized out>, uap=0x288062b4)
    at /usr/src/sys/kern/kern_exec.c:213
#12 0xc09e4849 in syscall (frame=<value optimized out>) at subr_syscall.c:134
#13 0xc09cdba1 in Xint0x80_syscall ()
    at /usr/src/sys/i386/i386/exception.s:270
#14 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)

------------------------------------------------------------------------
ps -axl

 UID  PID PPID CPU PRI NI    VSZ     RSS MWCHAN   STAT TT      TIME COMMAND
   0    0    0   0  -8  0      0       0 -        DLs   -   0:04.45 [kernel]
   0    1    0   0  52  0   9332  913536 wait     DLs   -   0:00.01 [init]
   0    2    0   0 -16  0      0       0 -        DL    -   0:01.58 [fdc0]
   0    3    0   0  -8  0      0       0 tx->tx_s DL    -   0:00.49 [zfskern]
   0    4    0   0 -16  0      0       0 waiting_ DL    -   0:00.00 [sctp_itera
   0    5    0   0 -16  0      0       0 ccb_scan DL    -   0:00.00 [xpt_thrd]
   0    6    0   0 -16  0      0       0 psleep   DL    -   0:00.09 [pagedaemon
   0    7    0   0 -16  0      0       0 psleep   DL    -   0:00.00 [vmdaemon]
   0    8    0   0 155  0      0       0 pgzero   DL    -   0:00.00 [pagezero]
   0    9    0   0  20  0      0       0 -        RL    -   0:00.02 [bufdaemon]
   0   10    0   0 -16  0      0       0 audit_wo DL    -   0:00.00 [audit]
   0   11    0   0 155  0      0       0 -        RL    -  12:39.71 [idle]
   0   12    0   0 -84  0      0       0 -        WL    -   0:03.45 [intr]
   0   13    0   0  -8  0      0       0 -        DL    -   0:01.60 [geom]
   0   14    0   0 -16  0      0       0 -        RL    -   0:00.33 [rand_harve
   0   15    0   0 -68  0      0       0 -        DL    -   0:00.01 [usb]
   0   16    0   0 -16  0      0       0 tzpoll   DL    -   0:00.04 [acpi_therm
   0   17    0   0 -16  0      0       0 cooling  DL    -   0:00.00 [acpi_cooli
   0   18    0   0 -16  0      0       0 -        RL    -   0:00.01 [vnlru]
   0   19    0   0  16  0      0       0 syncer   DL    -   0:00.07 [syncer]
   0   20    0   0  20  0      0       0 sdflush  DL    -   0:00.02 [softdepflu
   0  897    1   0  20  0   9380  900032 select   Ds    -   0:00.00 [devd]
   0 1026    1   0  20  0  10092 3918592 select   Ds    -   0:00.05 [syslogd]
   0 1166    1   0  20  0  12352 3914688 select   Ds    -   0:00.06 [ntpd]
   0 1169    1   0  20  0  10064  899072 -        Rs    -   0:00.40 [powerd]
   0 1193    1   0  20  0  16844  922240 select   Ds    -   0:00.03 [nmbd]
   0 1196    1   0  23  0  24632 3910848 select   Ds    -   0:00.02 [smbd]
   0 1213 1196   0  20  0  25156 3919552 select   D     -   0:00.00 [smbd]
 100 1217    1   0  41  0  19480 3911808 wait     Ds    -   0:00.00 [squid]
 100 1219 1217   0  20  0  72728 3909888 kqread   D     -   0:02.36 [squid]
 100 1221 1219   0  20  0  11096 3908928 sbwait   D     -   0:00.02 [log_file_d
 100 1222 1219   0  42  0  11092 3917632 piperd   D     -   0:00.02 [unlinkd]
1003 1224 1196   0  20  0  24976 3907008 -        R     -   0:00.54 [smbd]
   0 1226    1   0  20  0  20744 3906048 select   Ds    -   0:00.02 [snmptrapd]
   0 1230    1   0  20  0  16460 3912768 select   D     -   0:00.48 [snmpd]
   0 1237    1   0  20  0  19320 3916672 sbwait   Ds    -   0:00.03 [nfacctd]
   0 1239 1237   0  20  0  27512 3907968 select   D     -   0:00.05 [nfacctd]
  53 1242    1   0  20  0  26832 3901248 kqread   Ds    -   0:00.24 [named]
  88 1249    1   0  52  0  10472 2907584 wait     Ds    -   0:00.05 [sh]
   0 1547    1   0  20  0  10792  904896 kqread   Ds    -   0:00.05 [dovecot]
  88 1593 1249   0  20  0 779800  910656 select   D     -   0:04.61 [mysqld]
 143 1594 1547   0  20  0  10744 3913728 kqread   D     -   0:00.01 [anvil]
   0 1595 1547   0  20  0  10744 3904128 kqread   D     -   0:00.01 [log]
   0 1599    1   0  20  0  14816 3921472 select   Ds    -   0:00.00 [sshd]
   0 1600 1026   0  20  0  10512 3888704 nanslp   Ds    -   0:00.01 [sshguard]
   0 1615    1   0  20  0 102404 3894464 -        Rs    -   0:00.60 [httpd]
   0 1618 1615   0  25  0  10124 3893504 piperd   D     -   0:00.01 [cronolog]
   0 1619 1615   0  34  0  10124 3898304 piperd   D     -   0:00.00 [cronolog]
   0 1620 1615   0  20  0  10124 3897344 piperd   D     -   0:00.00 [cronolog]
   0 1621 1615   0  35  0  10124 3895424 piperd   D     -   0:00.00 [cronolog]
   0 1622 1615   0  35  0  10124 3892544 piperd   D     -   0:00.00 [cronolog]
   0 1623 1615   0  35  0  10124 3891584 piperd   D     -   0:00.00 [cronolog]
   0 1624 1615   0  20  0  10124 3890624 piperd   D     -   0:00.00 [cronolog]
   0 1625 1615   0  20  0  10124 2894080 piperd   D     -   0:00.00 [cronolog]
   0 1626 1615   0  37  0  10124 3905088 piperd   D     -   0:00.00 [cronolog]
   0 1628 1615   0  20  0  10124 3903168 piperd   D     -   0:00.01 [cronolog]
  80 1636 1615   0  42  0 123016 3900288 lockf    D     -   0:02.99 [httpd]
  80 1637 1615   0  20  0 102404  907776 lockf    D     -   0:00.09 [httpd]
  80 1638 1615   0  30  0 119304 3887744 lockf    D     -   0:08.80 [httpd]
  80 1639 1615   0  20  0 131336 3886784 lockf    D     -   0:10.06 [httpd]
  80 1640 1615   0  20  0 123144  911616 kqread   D     -   0:04.02 [httpd]
  80 1641 1615   0  20  0 123144 3884864 lockf    D     -   0:02.54 [httpd]
  80 1643 1615   0  20  0 114824 3883904 lockf    D     -   0:03.34 [httpd]
   0 1692 1599   0  26  0  17576 3880960 select   Ds    -   0:00.06 [sshd]
1001 1694 1692   0  20  0  17576 3902208 select   D     -   0:01.38 [sshd]
1001 1695 1694   0  20  0  10904 3881920 wait     Ds    -   0:00.04 [bash]
   0 1707 1695   0  22  0  13440 3896384 select   D     -   0:00.02 [sudo]
   0 1708 1707   0  20  0  10904 3885824 wait     D     -   0:00.03 [bash]
   0 1715 1708   0  52  0   9016  905856 wait     D+    -   0:00.07 [make]
   0 1738 1715   0  52  0   9016 3880000 wait     D+    -   0:00.06 [make]
   0 1754 1738   0  52  0   9016 2891200 wait     D+    -   0:00.06 [make]
   0 1771 1754   0  52  0   9016 2887360 wait     D+    -   0:00.07 [make]
   0 1891 1771   0  52  0  10472  872128 wait     D+    -   0:00.08 [sh]
   0 2133    1   0  52  0  10128  869248 nanslp   Ds    -   0:00.01 [cron]
   0 2139 2133   0  36  0  10128  874048 wait     D     -   0:00.00 [cron]
 107 2141 2139   0  32  0  66032 2919168 -        Rs    -   0:00.00 [php]
   0 2216    1   0  52  0  10472 3875200 wait     D     -   0:00.00 [sh]
   0 2217    1   0  52  0   9976  877888 piperd   D     -   0:00.00 [logger]
   0 2218 2216   0  52  0   5876 2896000 nanslp   D     -   0:00.00 [sleep]
   0 2250    1   0  44  0  10080 3874240 ttyin    Ds+   -   0:00.00 [getty]
   0 2251    1   0  44  0  10080 3878080 ttyin    Ds+   -   0:00.00 [getty]
   0 2252    1   0  45  0  10080 3873280 ttyin    Ds+   -   0:00.00 [getty]
   0 2253    1   0  44  0  10080 3872320 ttyin    Ds+   -   0:00.00 [getty]
   0 2254    1   0  44  0  10080 3871360 ttyin    Ds+   -   0:00.00 [getty]
   0 2255    1   0  45  0  10080 3870400 ttyin    Ds+   -   0:00.00 [getty]
   0 2256    1   0  46  0  10080 3869440 ttyin    Ds+   -   0:00.00 [getty]
   0 2257    1   0  45  0  10080  930880 ttyin    Ds+   -   0:00.00 [getty]
 107 3030    1   0  42  0  66032  931840 piperd   D     -   0:00.00 [php]
 107 3040    1   0  52  0  66032  921280 piperd   D     -   0:00.00 [php]
 107 3045 2141   0  23  0  47004  919360 vnread   DL    -   0:00.00 [rrdtool]
 107 3082 3030   0  34  0  11948 3889664 vnread   DL    -   0:00.00 [perl5.16.3
 107 3096 3040   0  48  0  10472  928960 -        R     -   0:00.00 [sh]
   0 3098 1891   0  52  0  10472  875008 wait     D+    -   0:00.00 [sh]
   0 3099 3098   0  52  0  10472  914496 wait     D+    -   0:00.00 [sh]
   0 3100 3099   0  52  0  22268 1526784 vnread   DL+   -   0:00.00 [cmp]

------------------------------------------------------------------------
vmstat -s

   703915 cpu context switches
    70762 device interrupts
    91567 software interrupts
 32455961 traps
 38039052 system calls
       20 kernel threads created
     2417  fork() calls
      663 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     3810 vnode pager pageins
    27212 vnode pager pages paged in
       31 vnode pager pageouts
       32 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
      203 pages reactivated
  4619063 copy-on-write faults
      500 copy-on-write optimized faults
 30539528 zero fill pages zeroed
  1689455 zero fill pages prezeroed
       21 intransit blocking page faults
 35234976 total VM faults taken
     3862 page faults requiring I/O
        0 pages affected by kernel thread creation
    85129 pages affected by  fork()
    24782 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
 39165227 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
   156469 pages active
   156155 pages inactive
     1680 pages in VM cache
    81855 pages wired down
   373664 pages free
     4096 bytes per page
 22660958 total name lookups
          cache hits (94% pos + 5% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
        sigio     1     1K       -        1  32
     filecaps     0     0K       -       24  64
      kdtrace   423    88K       -     3582  64,256
         kenv    90     7K       -      113  16,32,64,128,4096
       kqueue    24    17K       -      203  128,1024,4096
    proc-args    61     5K       -   569966  16,32,64,128,256
       DEVFS1   127    32K       -      177  256
        hhook     2     1K       -        2  128
      ithread    74     6K       -       74  16,64,128
       KTRACE   100    13K       -      100  128
        DEVFS    28     1K       -       33  16,64
       linker   131   513K       -      187  16,32,256,1024,2048,4096
       DEVFSP     1     1K       -        1  32
        lockf   121     6K       -   163497  32,64
   loginclass     2     1K       -       15  64
       devbuf  3355  3427K       -     3462  16,32,64,128,256,512,1024,2048,4096
         temp    22   233K       -  1188840  16,32,64,128,256,512,1024,2048
       ip6opt     0     0K       -        1  32
       ip6ndp     8     1K       -       10  16,64,128
       module   291    19K       -      291  64,128
     mtx_pool     2     8K       -        2  4096
          osd    15     1K       -       22  16,32,64
     pmchooks     1     1K       -        1  64
         pgrp    33     3K       -       64  64
      session    30     4K       -       31  128
         proc     2     8K       -        2  4096
      subproc   217   411K       -     3223  256,4096
         cred   114    11K       -  3437550  64,128
       plimit    19     5K       -      336  256
      uidinfo    10     2K       -      257  64,1024
       sysctl     0     0K       -     1024  16,32,64
    sysctloid  3597   109K       -     3674  16,32,64,128
    sysctltmp     0     0K       -      365  16,32,64,256
      tidhash     1     8K       -        1
      callout     3  1052K       -        3
         umtx   680    64K       -      680  64,128
     p1003.1b     1     1K       -        1  16
         SWAP     2   549K       -        2  64
          bus   760    36K       -     2996  16,32,64,128,256,1024
       bus-sc    66   120K       -     1220  16,64,128,256,512,1024,2048,4096
      devstat    22    45K       -       22  16,4096
 eventhandler    84     4K       -       84  32,64,128
         kobj   191   382K       -      789  2048
      Per-cpu     1     1K       -        1  16
         rman   234    15K       -      557  16,32,64
         sbuf     0     0K       -     2864  16,32,64,128,256,512,1024,2048,4096
    taskqueue    49     5K       -       79  16,32,128,512
       Unitno    20     2K       -     1294  16,64
         vmem     3    80K       -       13  128,256,1024,2048,4096
     ioctlops     0     0K       -     6089  16,32,64,128,256,512,1024,2048
       select    38     3K       -       38  64
          iov     0     0K       -  1815537  16,64,128,256
          msg     4    25K       -        4  1024,4096
          sem     4   101K       -        4  1024,4096
          shm     1    12K       -        1
          tty    19    10K       -       19  512
          pts     1     1K       -        1  128
     mbuf_tag     0     0K       -      116  32,64
        shmfd     1     4K       -        1  4096
       soname    37     5K       -    18918  16,32,128
          pcb    43   331K       -      488  16,32,64,512,1024,4096
          acl     0     0K       -       21  4096
     vfscache     1   512K       -        1
   cl_savebuf     0     0K       -       19  32
     vfs_hash     1   256K       -        1
       vnodes     3     1K       -        3  16,128
        mount    86     3K       -      202  16,32,64,128,256
  vnodemarker     0     0K       -      278  512
          BPF     3     1K       -        3  64
        ifnet     4     4K       -        4  64,1024
       ifaddr    56    11K       -       56  16,32,64,128,256,512,1024,2048
  ether_multi    40     2K       -       46  16,32,64
        clone     6     1K       -        6  128
       arpcom     2     1K       -        2  16
      lltable    17     5K       -       17  256
    pfs_nodes    73    10K       -       73  128
  pfs_vncache     1     1K       -        1  32
         GEOM   300    61K       -     3141  16,32,64,128,256,512,1024,2048
     acpiintr     1     1K       -        1  32
      ata_pci     2     1K       -        2  32
       acpica  1993   103K       -    50851  16,32,64,128,256,512,1024
     routetbl    42     5K       -      872  16,32,64,128,256
         igmp     3     1K       -        3  128
     in_multi     2     1K       -        2  128
      CAM CCB     8    16K       -       41  2048
        mixer     1     4K       -        1  4096
       kbdmux     6    18K       -        6  16,256,1024,2048
    sctp_a_it     0     0K       -        4  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa     6     1K       -        6  128
     sctp_ifn     2     1K       -        2  128
    sctp_iter     0     0K       -        4  256
    hostcache     1    16K       -        1
     syncache     1    44K       -        1
    in6_multi    30     3K       -       30  16,256
          mld     3     1K       -        3  128
      NFS FHA     1     1K       -        1  1024
          rpc     2     1K       -        2  128
audit_evclass   187     3K       -      228  16
      pagedep    10    66K       -       49  128
     inodedep   113   284K       -      564  256
    bmsafemap    10     6K       -      435  128,4096
       newblk   114   527K       -    19814  128
     indirdep     2     1K       -       32  64
     freefrag     0     0K       -       58  64
     freeblks    10     2K       -      336  128
     freefile     3     1K       -      279  32
       diradd    77     5K       -      404  64
        mkdir     0     0K       -       14  64
       dirrem     7     1K       -      314  64
    newdirblk     1     1K       -        8  32
     freework    12     2K       -     3240  16,128
      freedep     0     0K       -       11  32
      jaddref     0     0K       -      418  64
      jremref     0     0K       -      328  64
       jmvref     0     0K       -        2  64
      jnewblk     5     1K       -    19813  64
    jfreefrag     0     0K       -       58  64
         jseg    17     3K       -      129  128
      jsegdep   206     7K       -    20617  32
        sbdep     0     0K       -       36  32
     savedino     0     0K       -       57  256
      jblocks     8     1K       -        8  128
  ufs_dirhash   105    22K       -      105  16,32,64,128,256,512
    ufs_quota     1   256K       -        1
    ufs_mount    12    49K       -       13  256,4096
    vm_pgdata     2    65K       -        2  64
      UMAHash     2     1K       -        2  256
     acpitask     1     2K       -        1  2048
    raid_data     0     0K       -      492  16,128
         UART     3     2K       -        3  16,512,1024
      acpisem    17     2K       -       17  64,128
     CAM path    19     1K       -       62  16
      acpidev    36     2K       -       36  32
   CAM periph    14     2K       -       34  16,32,64,128
    CAM queue    27     4K       -       76  16,256
     atkbddev     2     1K       -        2  32
CAM dev queue     7     1K       -        7  32
          USB    24    10K       -       24  16,64,256,2048
       USBdev    20     2K       -       20  32,64,128
md_nvidia_data     0     0K       -       82  512
      entropy  1026    65K       -    15727  16,64,4096
  md_sii_data     0     0K       -       82  512
      CAM SIM     7     1K       -        7  128
      CAM XPT    45     2K       -      309  16,32,64,128,256,1024
      memdesc     1     4K       -        1  4096
         ac97     2     1K       -        2  16,512
     pci_link    16     2K       -       16  64,128
       apmdev     2     1K       -        2  64
   madt_table     0     0K       -        1  2048
      acpipwr     1     1K       -        1  32
       isadev     7     1K       -        7  64
      io_apic     1     1K       -        1  1024
          agp     2   129K       -        2  16
          MCA     4     1K       -        4  128
      CAM DEV    13    13K       -       18  1024
       feeder    12     1K       -       14  16,64
     nexusdev     4     1K       -        4  16
         cdev     8     1K       -        8  128
       DEVFS3   150    19K       -      204  128,256
     filedesc   162   300K       -     3313  16,32,128,1024,2048,4096
      solaris 23597 213023K       -   891430  16,32,64,128,256,512,1024,2048,4096
   kstat_data     6     1K       -        6  32
        linux    15     1K       -       15  32,64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               256,      0,     105,       0,     105,   0,   0
UMA Zones:              432,      0,     105,       3,     105,   0,   0
UMA Slabs:               56,      0,    4424,     112,   20744,   0,   0
UMA RCntSlabs:           64,      0,     345,      33,     345,   0,   0
UMA Hash:               128,      0,       2,      29,       4,   0,   0
4 Bucket:                16,      0,      29,     475,    1285,   0,   0
8 Bucket:                32,      0,       8,     244,     580,   0,   0
16 Bucket:               64,      0,     310,     320,    7735,   0,   0
32 Bucket:              128,      0,      25,     254,     633,  50,   0
64 Bucket:              256,      0,     848,     142,    3156,   3,   0
128 Bucket:             512,      0,     284,     108,    3262,  75,   0
vmem btag:               28,      0,   16816,     176,   18322, 247,   0
VM OBJECT:              156,      0,    5780,      95, 5156256,   0,   0
RADIX NODE:              44, 761852,   59484,     212, 6998757,   3,   0
MAP:                    144,      0,       3,      81,       3,   0,   0
KMAP ENTRY:              76,      0,       5,     154,       5,   0,   0
MAP ENTRY:               76,      0,    4792,     243,15440320,   0,   0
VMSPACE:                240,      0,      75,     117,    3082,   0,   0
fakepg:                  68,      0,       0,       0,       0,   0,   0
mt_zone:               2060,      0,     303,       0,     303,   0,   0
16:                      16,      0,    4293,     747,  195422,   0,   0
32:                      32,      0,   12790,    3086,  841263,   0,   0
64:                      64,      0,    6756,     930, 4113352,   0,   0
128:                    128,      0,    7837,   11290, 2324838,   0,   0
256:                    256,      0,    1160,   47770,   93586,   0,   0
512:                    512,      0,    2518,      34,  607128,   0,   0
1024:                  1024,      0,     287,    1405,   43250,   0,   0
2048:                  2048,      0,     666,      12,    4099,   0,   0
4096:                  4096,      0,     322,       6,   21384,   0,   0
uint64 pcpu:              8,      0,    1339,      69,    1339,   0,   0
SLEEPQUEUE:              44,      0,     341,     163,     341,   0,   0
Files:                   56,      0,     480,     312, 2880980,   0,   0
rl_entry:                28,      0,      98,     190,      98,   0,   0
TURNSTILE:               72,      0,     341,     186,     341,   0,   0
umtx pi:                 52,      0,       0,       0,       0,   0,   0
MAC labels:              20,      0,       0,       0,       0,   0,   0
PROC:                   748,      0,      94,      26,    3100,   0,   0
THREAD:                 772,      0,     327,      13,     480,   0,   0
cpuset:                  40,      0,     207,      96,     346,   0,   0
audit_record:          1112,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 209715,     320,     321,   29050, 261,   0
mbuf:                   256, 209715,       2,     412,   85145,  34,   0
mbuf_cluster:          2048,  32768,     636,       4,     636,   0,   0
mbuf_jumbo_page:       4096,  16384,       0,      25,    1122,   0,   0
mbuf_jumbo_9k:         9216,  14562,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  10920,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
ttyinq:                 152,      0,     180,      28,     315,   0,   0
ttyoutq:                256,      0,      95,     100,     167,   0,   0
g_bio:                  152,      0,       8,     668,  201764,   0,   0
ata_request:            220,      0,       1,     197,   59047,   0,   0
vtnet_tx_hdr:            16,      0,       0,       0,       0,   0,   0
taskq_zone:              24,      0,       0,     336,     125,   0,   0
VNODE:                  284,      0,   13638,      54,   13936,   0,   0
VNODEPOLL:               60,      0,       0,       0,       0,   0,   0
BUF TRIE:                44,      0,     473,    5351,   12712,   0,   0
NAMEI:                 1024,      0,       1,      39, 9220886,   0,   0
S VFS Cache:             72,      0,   19889,     159,   23315,   0,   0
STS VFS Cache:           92,      0,       0,       0,       0,   0,   0
L VFS Cache:            292,      0,     153,      29,     200,   0,   0
LTS VFS Cache:          312,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,     271,      17,     271,   0,   0
NCLNODE:                356,      0,       0,       0,       0,   0,   0
pipe:                   408,      0,      87,      30,    1557,   0,   0
procdesc:                72,      0,       0,       0,       0,   0,   0
space_seg_cache:         48,      0,    5665,    1895,   22999,   0,   0
zio_cache:              744,      0,       1,    2264,  319266,   0,   0
zio_link_cache:          24,      0,       0,    3192,  201550,   0,   0
sa_cache:                44,      0,    1866,     136,    1881,   0,   0
dnode_t:                540,      0,    2226,      14,    2277,   0,   0
dmu_buf_impl_t:         136,      0,    5570,      56,    5771,   0,   0
arc_buf_hdr_t:          148,      0,    3534,      84,    4181,   0,   0
arc_buf_t:               40,      0,    3433,     203,    4220,   0,   0
zil_lwb_cache:          168,      0,      83,     109,     502,   0,   0
zfs_znode_cache:        252,      0,    1866,      38,    1881,   0,   0
Mountpoints:            672,      0,       7,      23,       7,   0,   0
ksiginfo:                80,      0,     106,     944,     177,   0,   0
itimer:                 232,      0,       1,      67,       1,   0,   0
KNOTE:                   72,      0,     143,     249,   18916,   0,   0
socket:                 424,  97902,     130,      77,    4578,   0,   0
unpcb:                  172,  97911,      58,     126,    3057,   0,   0
ipq:                     32,   1134,       0,     126,       1,   0,   0
udp_inpcb:              252,  97904,      25,     103,    1273,   0,   0
udpcb:                    8,  98028,      25,     227,    1273,   0,   0
tcp_inpcb:              252,  97904,      46,      82,     239,   0,   0
tcpcb:                  752,  97905,      46,      34,     239,   0,   0
tcptw:                   60,  19631,       0,     268,      37,   0,   0
syncache:               124,  15360,       0,     256,     103,   0,   0
hostcache:               76,  15370,      30,     235,      30,   0,   0
tcpreass:                20,   2222,       0,     404,       6,   0,   0
sackhole:                20,      0,       0,     404,       3,   0,   0
sctp_ep:                976,  97904,       0,       0,       0,   0,   0
sctp_asoc:             1592,  40000,       0,       0,       0,   0,   0
sctp_laddr:              24,  80136,       0,     336,       5,   0,   0
sctp_raddr:             532,  80003,       0,       0,       0,   0,   0
sctp_chunk:              96, 400008,       0,       0,       0,   0,   0
sctp_readq:              76, 400044,       0,       0,       0,   0,   0
sctp_stream_msg_out:     68, 400020,       0,       0,       0,   0,   0
sctp_asconf:             24, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         24, 400008,       0,       0,       0,   0,   0
ripcb:                  252,  97904,       0,      64,       2,   0,   0
rtentry:                108,      0,      20,     239,      20,   0,   0
selfd:                   28,      0,     102,     330,   98240,   0,   0
SWAPMETA:               276, 131082,       0,       0,       0,   0,   0
FFS inode:              116,      0,   11727,     173,   12008,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,   11727,      48,   12008,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq9: acpi0                            1          0
irq14: ata0                        16143        264
irq17: pcm0 re0+                   29985        491
irq18: uhci2++                     24634        403
cpu0:timer                        469716       7700
cpu1:timer                        616212      10101
Total                            1156691      18962

------------------------------------------------------------------------
pstat -T

480/97901 files
0M/4095M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/ada2p3        8388352        0  8388352     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
            ada0             ada1             ada2             cpu
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
 23.11 152  3.43  23.32 152  3.45  76.36 252 18.83  17  0 32  0 51

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME


------------------------------------------------------------------------
ipcs -T

msginfo:
        msgmax:        16384    (max characters in a message)
        msgmni:           40    (# of message queues)
        msgmnb:         2048    (max characters in a message queue)
        msgtql:           40    (max # of messages in system)
        msgssz:            8    (size of a message segment)
        msgseg:         2048    (# of message segments in system)

shminfo:
        shmmax:    536870912    (max shared memory segment size)
        shmmin:            1    (min shared memory segment size)
        shmmni:          192    (max number of shared memory identifiers)
        shmseg:          128    (max shared memory segments per process)
        shmall:       131072    (max amount of shared memory in pages)

seminfo:
        semmni:           50    (# of semaphore identifiers)
        semmns:          340    (# of semaphores in system)
        semmnu:          150    (# of undo structures in system)
        semmsl:          340    (max # of semaphores per id)
        semopm:          100    (max # of operations per semop call)
        semume:           50    (max # of undo entries per process)
        semusz:          616    (size in bytes of undo structure)
        semvmx:        32767    (semaphore maximum value)
        semaem:        16384    (adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
        0 packets sent
                4 data packets (0 bytes)
                6 data packets (0 bytes) retransmitted
                25 data packets unnecessarily retransmitted
                13973213274039250800 resends initiated by MTU discovery
                89 ack-only packets (8320719 delayed)
                4 URG only packets
                18096 window probe packets
                12467 window update packets
                0 control packets
        107 packets received
                0 acks (for 0 bytes)
                0 duplicate acks
                0 acks for unsent data
                90 packets (42 bytes) received in-sequence
                103 completely duplicate packets (0 bytes)
                0 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                0 out-of-order packets (0 bytes)
                0 packets (0 bytes) of data after window
                3 window probes
                0 window update packets
                0 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
                103 discarded due to memory problems
        104 connection requests
        159 connection accepts
        0 bad connection attempts
        0 listen queue overflows
        15768657978964039129 ignored RSTs in the windows
        48 connections established (including accepts)
        0 connections closed (including 193 drops)
                0 connections updated cached RTT on close
                0 connections updated cached RTT variance on close
                10400501526695997124 connections updated cached ssthresh on close
        17989 embryonic connections dropped
        39967 segments updated rtt (of 0 attempts)
        2612 retransmit timeouts
                63662 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 Connections (fin_wait_2) dropped because of timeout
        249 keepalive timeouts
                7419 keepalive probes sent
                0 connections dropped by keepalive
        0 correct ACK header predictions
        0 correct data packet header predictions
        0 syncache entries added
                0 retransmitted
                0 dupsyn
                0 dropped
                0 completed
                0 bucket overflow
                13886076930284952932 cache overflow
                13884448639463327847 reset
                14369119888824951142 stale
                0 aborted
                0 badack
                0 unreach
                0 zone failures
        0 cookies sent
        0 cookies received
        0 hostcache entries added
                11 bucket overflow
        0 SACK recovery episodes
        0 segment rexmits in SACK recovery episodes
        0 byte rexmits in SACK recovery episodes
        15334073515277538020 SACK options (SACK blocks) received
        0 SACK options (SACK blocks) sent
        0 SACK scoreboard overflow
        0 packets with ECN CE bit set
        0 packets with ECN ECT(0) bit set
        0 packets with ECN ECT(1) bit set
        0 successful ECN handshakes
        0 times ECN reduced the congestion window
udp:
        684 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        42 with no checksum
        475 dropped due to no socket
        0 broadcast/multicast datagrams undelivered
        10337168320257321432 dropped due to full socket buffers
        0 not for hashed pcb
        8109575753452230393 delivered
        9826303899083371608 datagrams output
        90916420709766232 times multicast source filter matched
ip:
        24325 total packets received
        0 bad header checksums
        2 with size smaller than minimum
        0 with data size < data length
        13887355951133609936 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        24300 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets reassembled ok
        0 packets for this host
        0 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        1 packet not forwardable
        0 packets received for unknown multicast group
        0 redirects sent
        13887359418233782272 packets sent from this host
        0 packets sent with fabricated ip header
        13887359452593520640 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        11 calls to icmp_error
        0 errors not generated in response to an icmp message
        Output histogram:
                #23: 24
        0 messages with bad code fields
        0 messages less than the minimum length
        0 messages with bad checksum
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                destination unreachable: 3358664368
                source quench: 3358664384
        0 message responses generated
        0 invalid return addresses
        0 no return routes
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with wrong TTL
        0 messages received with bad checksum
        0 V1/V2 membership queries received
        0 V3 membership queries received
        0 membership queries received with invalid field(s)
        0 general queries received
        0 group queries received
        0 group-source queries received
        0 group-source queries dropped
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 V3 reports received without Router Alert
        0 membership reports sent
arp:
        3 ARP requests sent
        139 ARP replies sent
        141 ARP requests received
        0 ARP replies received
        0 ARP packets received
        15481077771237653545 total packets dropped due to no ARP entry
        0 ARP entrys timed out
        0 Duplicate IPs seen
ip6:
        73 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        41 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        0 packets for this host
        0 packets forwarded
        0 packets not forwardable
        0 redirects sent
        1 packet sent from this host
        37 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        0 multicast packets which we don't join
        Input histogram:
                #133: 70
                #137: 65
                #143: 18
                #153: 64
                #159: 18
                #161: 1
                #176: 83
                #177: 18
                #187: 219269014384766456
                #194: 13887359418233782272
                #195: 2660997166872578770
                #198: 14446079808121077760
                #233: 14355189146500852380
                #236: 9027100100361080280
                #239: 10383612547033435136
                #240: 9894905896530074024
                #248: 15261738849122041527
        Mbuf statistics:
                0 one mbuf
                0 one ext mbuf
                0 two or more ext mbuf
        0 packets whose headers are not contiguous
        0 tunneling packets that can't find gif
        0 packets discarded because of too many headers
        0 failures of source address selection
        Source addresses selection rule applied:
icmp6:
        0 calls to icmp6_error
        0 errors not generated in response to an icmp6 message
        0 errors not generated because of rate limitation
        Output histogram:
                #65: 1
                #66: 16
                #70: 2
                #196: 3
                #197: 17
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Input histogram:
                #11: 75153822013882590
                #15: 13884491396925743745
                #16: 13884472038445903576
                #17: 5250621203072
                #19: 73
                #24: 41
                #30: 1
                #39: 37
                #175: 70
                #179: 65
                #185: 18
                #195: 64
                #201: 18
                #203: 1
                #218: 83
                #219: 18
                #229: 219269014384766456
                #236: 13887359418233782272
                #237: 2660997166872578770
                #240: 14446079808121077760
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        14355189146500852380 bad router advertisement messages
        0 bad redirect messages
        0 path MTU changes
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output

------------------------------------------------------------------------
netstat -m

322/733/1055 mbufs in use (current/cache/total)
315/325/640/32768 mbuf clusters in use (current/cache/total/max)
320/321 mbuf+clusters out of packet secondary zone in use (current/cache)
0/25/25/16384 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/14562 9k jumbo clusters in use (current/cache/total/max)
0/0/0/10920 16k jumbo clusters in use (current/cache/total/max)
710K/933K/1643K bytes allocated to network (current/cache/total)
34/0/261 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

------------------------------------------------------------------------
netstat -idW

Name      Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
re0      1500 <Link#1>      00:01:29:89:10:cd    22956     0     0    39043     0     0    0
re0      1500 10.0.0.0      gatekeeper           24120     -     -    38963     -     -    -
re0      1500 fe80::201:29f fe80::201:29ff:fe       63     -     -       66     -     -    -
re0      1500 2001:b18:2000 2001:b18:2000:81c        0     -     -        0     -     -    -
fxp0*    1500 <Link#2>      00:02:a5:4b:89:a0        0     0     0        0     0     0    0
lo0     16384 <Link#3>                            1585     0     0     1585     0     0    0
lo0     16384 localhost     ::1                      2     -     -        2     -     -    -
lo0     16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    -
lo0     16384 your-net      localhost              183     -     -     1583     -     -    -

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            10.0.0.254         UGS         0     3233    re0
10.0.0.0/24        link#1             U           0    35730    re0
10.0.0.1           link#1             UHS         0     1400    lo0
127.0.0.1          link#3             UH          0      183    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0 =>
default                           fe80::beae:c5ff:fec4:d1f1%re0 UG          re0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2001:b18:2000:81c2::/64           link#1                        U           re0
2001:b18:2000:81c2:201:29ff:fe89:10cd link#1                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%re0/64                     link#1                        U           re0
fe80::201:29ff:fe89:10cd%re0      link#1                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff01::%re0/32                     fe80::201:29ff:fe89:10cd%re0  U           re0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%re0/32                     fe80::201:29ff:fe89:10cd%re0  U           re0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb    Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
d23ca5e0 tcp4       0      0 10.0.0.1.13082     212.113.185.59.443 ESTABLISHED
d239a2f0 tcp4       0      0 127.0.0.1.53       127.0.0.1.25059    ESTABLISHED
cd626000 tcp4       0      0 127.0.0.1.25059    127.0.0.1.53       ESTABLISHED
d23c95e0 tcp4       0      0 10.0.0.1.3128      10.0.0.30.51111    ESTABLISHED
cd627000 tcp4       0      0 10.0.0.1.65506     195.10.8.40.443    ESTABLISHED
d23ca2f0 tcp4       0      0 10.0.0.1.3128      10.0.0.30.51106    ESTABLISHED
cd6278d0 tcp4       0      0 10.0.0.1.47849     31.13.80.33.443    ESTABLISHED
cd6265e0 tcp4       0      0 10.0.0.1.3128      10.0.0.30.51105    ESTABLISHED
d23dc000 tcp4       0      0 10.0.0.1.58973     69.171.248.16.443  ESTABLISHED
d23c92f0 tcp4       0      0 10.0.0.1.3128      10.0.0.30.51104    ESTABLISHED
d221d000 tcp4       0      0 10.0.0.1.23863     212.113.185.34.443 ESTABLISHED
d23ca000 tcp4       0      0 10.0.0.1.3128      10.0.0.30.51103    ESTABLISHED
d23dcbc0 tcp4       0      0 10.0.0.1.22        10.0.0.30.57828    ESTABLISHED
c8a268d0 tcp4       0      0 10.0.0.1.32253     212.113.185.34.443 ESTABLISHED
d23dd2f0 tcp4       0      0 10.0.0.1.3128      10.0.0.30.51100    ESTABLISHED
d23c98d0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58946    FIN_WAIT_1
d23dc5e0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58944    FIN_WAIT_1
c8a265e0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58943    FIN_WAIT_1
d23dc8d0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58942    FIN_WAIT_1
d239a8d0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58938    FIN_WAIT_1
d23dd5e0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58937    FIN_WAIT_1
d23dd8d0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58935    FIN_WAIT_1
d23ddbc0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58934    FIN_WAIT_1
c89fa8d0 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58930    FIN_WAIT_1
d221e000 tcp4       0      0 10.0.0.1.3128      10.0.0.10.58898    FIN_WAIT_1
cd6275e0 tcp4       0      0 10.0.0.1.3306      *.*                LISTEN
c89f8000 tcp4       0      0 *.443              *.*                LISTEN
c89f82f0 tcp6       0      0 *.443              *.*                LISTEN
c89f85e0 tcp4       0      0 *.80               *.*                LISTEN
c89f88d0 tcp6       0      0 *.80               *.*                LISTEN
c89f8bc0 tcp4       0      0 *.22               *.*                LISTEN
c89f9000 tcp6       0      0 *.22               *.*                LISTEN
c89f92f0 tcp6       0      0 *.993              *.*                LISTEN
c89f95e0 tcp4       0      0 *.993              *.*                LISTEN
c89f98d0 tcp6       0      0 *.143              *.*                LISTEN
c89f9bc0 tcp4       0      0 *.143              *.*                LISTEN
c8a24bc0 tcp4       0      0 127.0.0.1.953      *.*                LISTEN
c8a25000 tcp4       0      0 127.0.0.1.53       *.*                LISTEN
c8a252f0 tcp4       0      0 10.0.0.1.53        *.*                LISTEN
c89fa000 tcp4       0      0 *.199              *.*                LISTEN
c89fa2f0 tcp4       0      0 10.0.0.1.445       10.0.0.254.1705    ESTABLISHED
c89fa5e0 tcp4       0      0 *.3128             *.*                LISTEN
c8a255e0 tcp4       0      0 *.139              *.*                LISTEN
c8a258d0 tcp4       0      0 *.445              *.*                LISTEN
c8a25bc0 tcp6       0      0 *.139              *.*                LISTEN
c8a26000 tcp6       0      0 *.445              *.*                LISTEN
c82df1f8 udp4       0      0 127.0.0.1.53       *.*
c82df2f4 udp4       0      0 10.0.0.1.53        *.*
c82e9bd0 udp4       0      0 *.2055             *.*
c82e9ccc udp4       0      0 *.161              *.*
c82e9dc8 udp4       0      0 *.*                *.*
c82e9ec4 udp4       0      0 *.54879            *.*
c82df3f0 udp4       0      0 *.162              *.*
c82ea5e8 udp4       0      0 *.3401             *.*
c82df4ec udp4       0      0 *.14877            *.*
c82ea000 udp4       0      0 10.0.0.255.138     *.*
c82ea0fc udp4       0      0 10.0.0.1.138       *.*
c82ea1f8 udp4       0      0 10.0.0.255.137     *.*
c82ea2f4 udp4       0      0 10.0.0.1.137       *.*
c82ea3f0 udp4       0      0 *.138              *.*
c82ea4ec udp4       0      0 *.137              *.*
c82df5e8 udp4       0      0 127.0.0.1.123      *.*
c82df6e4 udp6       0      0 fe80::1%lo0.123    *.*
c82df7e0 udp6       0      0 ::1.123            *.*
c82df8dc udp6       0      0 2001:b18:2000:81.1 *.*
c82df9d8 udp6       0      0 fe80::201:29ff:f.1 *.*
c82dfad4 udp4       0      0 10.0.0.1.123       *.*
c82ea6e4 udp6       0      0 *.123              *.*
c82dfbd0 udp4       0      0 *.123              *.*
c82ea7e0 udp4       0      0 *.514              *.*
c82ea8dc udp6       0      0 *.514              *.*
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
c86c0c18 stream      0      0        0 c86c1cc4        0        0 /tmp/mysql.sock
c86c1cc4 stream      0      0        0 c86c0c18        0        0
c86c0a14 stream      0      0        0 d2405764        0        0 /tmp/mysql.sock
d2405764 stream      0      0        0 c86c0a14        0        0
c8676408 stream     90      0        0 c86c0ec8        0        0 /tmp/mysql.sock
c86c0ec8 stream      0      0        0 c8676408        0        0
c86c1810 stream      0      0        0        0        0        0
d24056b8 stream      0      0        0 d240560c        0        0
d240560c stream      0      0        0 d24056b8        0        0
c86c235c stream      0      0 cb0396a8        0        0        0 /tmp/mysql.sock
c8676ac0 stream      0      0 c8cfe11c        0        0        0 /var/run/dovecot/anvil-auth-penalty
c8676b6c stream      0      0 c8cfe238        0        0        0 /var/run/dovecot/anvil
c8676c18 stream      0      0 c8cfe354        0        0        0 /var/run/dovecot/auth-worker
c8676cc4 stream      0      0 c8cfe470        0        0        0 /var/run/dovecot/auth-master
c8676d70 stream      0      0 c8cfe58c        0        0        0 /var/run/dovecot/auth-userdb
c8676e1c stream      0      0 c8cfe6a8        0        0        0 /var/run/dovecot/auth-client
c8676ec8 stream      0      0 c8b70470        0        0        0 /var/run/dovecot/auth-login
c8677000 stream      0      0 c8b7058c        0        0        0 /var/run/dovecot/token-login/tokenlogin
c86770ac stream      0      0 c8b706a8        0        0        0 /var/run/dovecot/login/login
c8677158 stream      0      0 c8b707c4        0        0        0 /var/run/dovecot/config
c8677204 stream      0      0 c8b708e0        0        0        0 /var/run/dovecot/dict
c86772b0 stream      0      0 c8b709fc        0        0        0 /var/run/dovecot/director-userdb
c867735c stream      0      0 c8b70b18        0        0        0 /var/run/dovecot/director-admin
c8677408 stream      0      0 c8b70c34        0        0        0 /var/run/dovecot/dns-client
c86774b4 stream      0      0 c8b70d50        0        0        0 /var/run/dovecot/doveadm-server
c8677560 stream      0      0 c8b70e6c        0        0        0 /var/run/dovecot/imap-urlauth
c867760c stream      0      0 c8c49000        0        0        0 /var/run/dovecot/token-login/imap-urlauth
c86776b8 stream      0      0 c8c4911c        0        0        0 /var/run/dovecot/imap-urlauth-worker
c8677764 stream      0      0 c8c49238        0        0        0 /var/run/dovecot/login/imap
c8677810 stream      0      0 c8c49354        0        0        0 /var/run/dovecot/indexer
c86778bc stream      0      0 c8c49470        0        0        0 /var/run/dovecot/indexer-worker
c8677968 stream      0      0 c8c4958c        0        0        0 /var/run/dovecot/login/ipc-proxy
c8677a14 stream      0      0 c8c496a8        0        0        0 /var/run/dovecot/ipc
c8677ac0 stream      0      0 c8c497c4        0        0        0 /var/run/dovecot/log-errors
c8677b6c stream      0      0 c8c499fc        0        0        0 /var/run/dovecot/replication-notify
c8677c18 stream      0      0 c8c49b18        0        0        0 /var/run/dovecot/replicator
c8677cc4 stream      0      0 c8c49c34        0        0        0 /var/run/dovecot/login/ssl-params
c8677d70 stream      0      0 c8c49d50        0        0        0 /var/run/dovecot/ssl-params
c86c24b4 stream      0      0 c8c4d58c        0        0        0 /var/run/dovecot/stats
c86c2560 stream      0      0        0 c86c260c        0        0
c86c260c stream      0      0        0 c86c2560        0        0
c8678158 stream      0      0        0 c8678204        0        0
c8678204 stream      0      0        0 c8678158        0        0
c86782b0 stream      0      0 c8a07470        0        0        0 /var/run/samba/nmbd/unexpected
c86c28bc stream      0      0 c82c0238        0        0        0 /var/run/devd.pipe
c86c0d70 dgram       0      0        0 c8678408        0 c86c22b0
c86c22b0 dgram       0      0        0 c8678408        0 c8676a14
c8676a14 dgram       0      0        0 c8678408        0 c86c26b8
c86c26b8 dgram       0      0        0 c8678408        0 c8677e1c
c8677e1c dgram       0      0        0 c8678408        0 c86780ac
c8677ec8 dgram       0      0        0 c8678000 c8678000        0
c8678000 dgram       0      0        0 c8677ec8 c8677ec8        0
c86780ac dgram       0      0        0 c8678408        0 c86c2764
c86c2764 dgram       0      0        0 c8678408        0 c867835c
c86c2810 dgram       0      0        0 c86c2968        0        0
c867835c dgram       0      0        0 c8678408        0        0
c8678408 dgram       0      0 c8674d50        0 c86c0d70        0 /var/run/logpriv
c86c2968 dgram       0      0 c8674e6c        0 c86c2810        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/80         gatekeeper.mysql
tcp4  0/0/128        *.https
tcp6  0/0/128        *.https
tcp4  0/0/128        *.http
tcp6  0/0/128        *.http
tcp4  0/0/128        *.ssh
tcp6  0/0/128        *.ssh
tcp6  0/0/100        *.imaps
tcp4  0/0/100        *.imaps
tcp6  0/0/100        *.imap
tcp4  0/0/100        *.imap
tcp4  0/0/128        localhost.rndc
tcp4  0/0/10         localhost.domain
tcp4  0/0/10         gatekeeper.domain
tcp4  0/0/128        *.smux
tcp4  0/0/128        *.3128
tcp4  0/0/50         *.netbios-ssn
tcp4  0/0/50         *.microsoft-ds
tcp6  0/0/50         *.netbios-ssn
tcp6  0/0/50         *.microsoft-ds
unix  0/0/80         /tmp/mysql.sock
unix  0/0/128        /var/run/dovecot/anvil-auth-penalty
unix  0/0/128        /var/run/dovecot/anvil
unix  0/0/100        /var/run/dovecot/auth-worker
unix  0/0/128        /var/run/dovecot/auth-master
unix  0/0/128        /var/run/dovecot/auth-userdb
unix  0/0/128        /var/run/dovecot/auth-client
unix  0/0/128        /var/run/dovecot/auth-login
unix  0/0/128        /var/run/dovecot/token-login/tokenlogin
unix  0/0/128        /var/run/dovecot/login/login
unix  0/0/128        /var/run/dovecot/config
unix  0/0/100        /var/run/dovecot/dict
unix  0/0/128        /var/run/dovecot/director-userdb
unix  0/0/128        /var/run/dovecot/director-admin
unix  0/0/100        /var/run/dovecot/dns-client
unix  0/0/100        /var/run/dovecot/doveadm-server
unix  0/0/100        /var/run/dovecot/imap-urlauth
unix  0/0/128        /var/run/dovecot/token-login/imap-urlauth
unix  0/0/128        /var/run/dovecot/imap-urlauth-worker
unix  0/0/128        /var/run/dovecot/login/imap
unix  0/0/128        /var/run/dovecot/indexer
unix  0/0/10         /var/run/dovecot/indexer-worker
unix  0/0/128        /var/run/dovecot/login/ipc-proxy
unix  0/0/128        /var/run/dovecot/ipc
unix  0/0/128        /var/run/dovecot/log-errors
unix  0/0/128        /var/run/dovecot/replication-notify
unix  0/0/128        /var/run/dovecot/replicator
unix  0/0/128        /var/run/dovecot/login/ssl-params
unix  0/0/128        /var/run/dovecot/ssl-params
unix  0/0/128        /var/run/dovecot/stats
unix  0/0/5          /var/run/samba/nmbd/unexpected
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 6 at 0xffff
fstat: can't read file 7 at 0x78
fstat: can't read file 10 at 0xffffffff
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 6 at 0xffff
fstat: can't read file 7 at 0x78
fstat: can't read file 10 at 0xffffffff
fstat: can't read file 11 at 0x200007f
fstat: can't read file 12 at 0x1fffff
fstat: can't read file 13 at 0x4000000
fstat: can't read file 15 at 0xffff
fstat: can't read file 16 at 0x78
fstat: can't read file 19 at 0xffffffff
fstat: can't read file 20 at 0x200007f
fstat: can't read file 21 at 0x1fffff
fstat: can't read file 22 at 0x4000000
fstat: can't read file 24 at 0xffff
fstat: can't read file 25 at 0x78
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read fil
Comment 1 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 08:00:18 UTC 
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped
Comment 2 Graham Perrin freebsd_committer freebsd_triage 2022-10-17 12:17:12 UTC 
Keyword: 

    crash

– in lieu of summary line prefix: 

    [panic]

* bulk change for the keyword
* summary lines may be edited manually (not in bulk). 

Keyword descriptions and search interface: 

    <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>
Comment 3 Graham Perrin freebsd_committer freebsd_triage 2022-12-23 11:03:40 UTC 
We might reasonably assume that the type of crash that occurred in 2014 is no longer reproducible in 2022. 

I'm sorry that there was no response around the time of the original report.