Bug 141463 - [nfs] [panic] Frequent kernel panics after upgrade from 7.2-STABLE to 8-STABLE [regression]
Summary: [nfs] [panic] Frequent kernel panics after upgrade from 7.2-STABLE to 8-STABL...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 8.0-STABLE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2009-12-14 17:30 UTC by Geoffrey Mainland
Modified: 2022-10-17 12:19 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Geoffrey Mainland 2009-12-14 17:30:01 UTC
Since upgrading from 7.2-STABLE, I've been seeing kernel panics multiple
times per day. core.txt from the latest panic is below.

zeno.apeiron.net dumped core - see /var/crash/vmcore.11

Mon Dec 14 12:15:48 EST 2009

FreeBSD zeno.apeiron.net 8.0-STABLE FreeBSD 8.0-STABLE #2: Sun Dec 13 13:52:22 EST 2009     toor@zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO  i386

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x20
fault code		= supervisor read, page not present
instruction pointer	= 0x20:0xc067db8e
stack pointer	        = 0x28:0xe65ab738
frame pointer	        = 0x28:0xe65ab768
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1673 (nfsd: service)
trap number		= 12
panic: page fault
cpuid = 0
Uptime: 7h52m13s
Physical memory: 1015 MB
Dumping 172 MB: 157 141 125 109 93 77 61 45 29 13

Reading symbols from /boot/kernel/if_em.ko...Reading symbols from /boot/kernel/if_em.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_em.ko
Reading symbols from /boot/kernel/if_fxp.ko...Reading symbols from /boot/kernel/if_fxp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_fxp.ko
Reading symbols from /boot/kernel/miibus.ko...Reading symbols from /boot/kernel/miibus.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/miibus.ko
Reading symbols from /boot/kernel/if_vr.ko...Reading symbols from /boot/kernel/if_vr.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_vr.ko
Reading symbols from /boot/kernel/ulpt.ko...Reading symbols from /boot/kernel/ulpt.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ulpt.ko
Reading symbols from /boot/kernel/ucom.ko...Reading symbols from /boot/kernel/ucom.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ucom.ko
Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boot/kernel/accf_data.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
#0  doadump () at pcpu.h:246
246	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump () at pcpu.h:246
#1  0xc0605f0c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2  0xc06061e5 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:579
#3  0xc08c7a13 in trap_fatal (frame=0xe65ab6f8, eva=32)
    at /usr/src/sys/i386/i386/trap.c:938
#4  0xc08c7be2 in trap_pfault (frame=0xe65ab6f8, usermode=0, eva=32)
    at /usr/src/sys/i386/i386/trap.c:851
#5  0xc08c8563 in trap (frame=0xe65ab6f8) at /usr/src/sys/i386/i386/trap.c:533
#6  0xc08abbdb in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#7  0xc067db8e in cache_lookup (dvp=0xc5fa6b84, vpp=0xe65aba7c, 
    cnp=0xe65aba90) at /usr/src/sys/kern/vfs_cache.c:442
#8  0xc067e1c9 in vfs_cache_lookup (ap=0xe65ab81c)
    at /usr/src/sys/kern/vfs_cache.c:882
#9  0xc08daed5 in VOP_LOOKUP_APV (vop=0xc0999cc0, a=0xe65ab81c)
    at vnode_if.c:123
#10 0xc0685441 in lookup (ndp=0xe65aba64) at vnode_if.h:54
#11 0xc07cbd43 in nfs_namei (ndp=0xe65aba64, nfsd=0xe65abbb0, fhp=0xe65abae4, 
    len=Variable "len" is not available.
) at /usr/src/sys/nfsserver/nfs_srvsubs.c:760
#12 0xc07bcb21 in nfsrv_lookup (nfsd=0xe65abbb0, slp=0x0, mrq=0xe65abba8)
    at /usr/src/sys/nfsserver/nfs_serv.c:505
#13 0xc07ca85e in nfssvc_program (rqst=0xc4172000, xprt=0xc46a9600)
    at /usr/src/sys/nfsserver/nfs_srvkrpc.c:410
#14 0xc07eb182 in svc_run_internal (pool=0xc4015500, ismaster=0)
    at /usr/src/sys/rpc/svc.c:893
#15 0xc07eb460 in svc_thread_start (arg=0xc4015500)
    at /usr/src/sys/rpc/svc.c:1198
#16 0xc05ddf28 in fork_exit (callout=0xc07eb450 <svc_thread_start>, 
    arg=0xc4015500, frame=0xe65abd38) at /usr/src/sys/kern/kern_fork.c:843
#17 0xc08abc50 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:270
(kgdb) 

------------------------------------------------------------------------
ps -axl

  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0     0     0   0 -68  0     0     0 -      RLs   ??  -33756765:-54.55 [kernel]
    0     1     0   0  44  0  2912     0 wait   DLs   ??  223915:01.00 [init]
    0     2     0   0  -8  0     0     0 -      DL    ??  4906490:24.00 [g_event]
    0     3     0   0  -8  0     0     0 -      DL    ??  -7790340:-6.55 [g_up]
    0     4     0   0  -8  0     0     0 -      DL    ??  -27268965:-2.55 [g_down]
    0     5     0   0 -16  0     0     0 crypto DL    ??    0:00.00 [crypto]
    0     6     0   0 -16  0     0     0 crypto DL    ??    0:00.00 [crypto ret
    0     7     0   0 -16  0     0     0 ccb_sc DL    ??    0:00.00 [xpt_thrd]
    0     8     0   0 -16  0     0     0 -      DL    ??  380928:31.00 [fdc0]
    0     9     0   0  44  0     0     0 pftm   DL    ??  20637975:44.00 [pfpurge]
    0    10     0   0 -16  0     0     0 audit_ DL    ??    0:00.00 [audit]
    0    11     0   0 171  0     0     0 -      RL    ??  23787828:12.00 [idle]
    0    12     0   0 -48  0     0     0 -      WL    ??  -12917208:-17.55 [intr]
    0    13     0   0 -16  0     0     0 sleep  DL    ??    0:00.00 [ng_queue]
    0    14     0   0 -16  0     0     0 -      DL    ??  12636660:42.00 [yarrow]
    0    15     0   0 -64  0     0     0 -      DL    ??  1700841:51.00 [usb]
    0    16     0   0  44  0     0     0 psleep DL    ??  31968105:14.00 [pagedaemon
    0    17     0   0  73  0     0     0 psleep DL    ??   75:32.00 [vmdaemon]
    0    18     0   0  76  0     0     0 pgzero DL    ??  1229:00.00 [pagezero]
    0    19     0   0  44  0     0     0 psleep DL    ??  901273:27.00 [bufdaemon]
    0    20     0   0  44  0     0     0 syncer DL    ??  19108101:36.00 [syncer]
    0    21     0   0 -16  0     0     0 vlruwt DL    ??  390332:01.00 [vnlru]
    0    22     0   0  44  0     0     0 sdflus DL    ??  -10994237:-5.55 [softdepflu
    0    23     0   0 -16  0     0     0 flowcl DL    ??  208731:27.00 [flowcleane
    0  1144     1   0  44  0  1888     0 select Ds    ??  4619:56.00 [devd]
    0  1407     1   0  44  0  3344     0 select Ds    ??  8921029:27.00 [syslogd]
   53  1538     1   0  44  0 38944     0 kqread Ds    ??  12504791:30.00 [named]
    0  1555     1   0  44  0  3504     0 select Ds    ??  1318894:20.00 [rpcbind]
    0  1570     1   0  44  0  3316     0 select Ds    ??  -34229393:-37.55 [ypserv]
    0  1577     1   0  44  0  3284     0 select Ds    ??  610008:21.00 [ypbind]
    0  1591     1   0  44  0  3316     0 select Ds    ??  34121956:53.55 [amd]
    0  1662     1   0  44  0  3344     0 select Ds    ??  156672:18.00 [mountd]
    0  1671     1   0  44  0  3284     0 select Ds    ??  1078931:03.00 [nfsd]
    0  1673  1671   0  44  0  3284     0 rpcsvc D     ??  5459434:29.00 [nfsd]
    0  1679     1   0  44  0 265464     0 select Ds    ??  116987:10.00 [rpc.statd]
    0  1686     1   0  44  0  3380     0 rpcsvc Ds    ??  378831:43.00 [rpc.lockd]
    0  1700     1   0  76  0  5212     0 sbwait D     ??  5746:32.00 [stunnel]
    0  1701     1   0  76  0  5212     0 sbwait D     ??  2802:30.00 [stunnel]
    0  1702     1   0  76  0  5212     0 sbwait D     ??  2594:02.00 [stunnel]
    0  1703     1   0  76  0  5212     0 sbwait D     ??  2596:43.00 [stunnel]
    0  1704     1   0  76  0  5212     0 sbwait D     ??  3698:01.00 [stunnel]
    0  1705     1   0  76  0  5212     0 select Ds    ??  26260:17.00 [stunnel]
    0  1721     1   0  44  0  4504     0 nanslp D     ??  215274:27.00 [smartd]
    1  1727     1   0  44  0  4680     0 select Ds    ??  321304:41.00 [slpd]
    0  1735     1   0  44  0  7424     0 kqread Ds    ??  7417015:59.00 [cupsd]
    0  1753     1   0  44  0  9360     0 select Ds    ??  5635062:08.00 [nmbd]
    0  1755  1753   0  44  0  9236     0 piperd D     ??  6599:33.00 [nmbd]
    0  1759     1   0  44  0 16264     0 select Ds    ??  661205:34.00 [smbd]
    0  1826  1759   0  76  0 16264     0 select D     ??  26160:36.00 [smbd]
    0  1827     1   0  44  0  4860     0 select Ds    ??  6922772:40.00 [ntpd]
    0  1876     1   0  44  0  3480     0 select Ds    ??  204490:01.00 [rpc.yppass
10002  1895     1   0  44  0  4308     0 select Ds    ??  73179:20.00 [dhcpd]
    0  1901     1   0  44  0  6208     0 select Ds    ??  787857:20.00 [proftpd]
  225  1919     1   0  44  0 12652     0 select Ds    ??  2778459:11.00 [perl5.8.9]
   70  1932     1   0  44  0 40348     0 select Ds    ??  6266910:18.00 [postgres]
   70  1934  1932   0  44  0 40348     0 select Ds    ??  14373173:35.00 [postgres]
   70  1935  1932   0  44  0 40348     0 select Ds    ??  4557941:07.00 [postgres]
   70  1936  1932   0  44  0 40348     0 select Ds    ??  2117024:28.00 [postgres]
   70  1937  1932   0  44  0 11668     0 select Ds    ??  1773671:41.00 [postgres]
  106  1960     1   0  44  0 92004     0 select Ds    ??  15325217:54.00 [clamd]
  106  1966     1   0  45  0  5328     0 pause  Ds    ??  123348:44.00 [freshclam]
  525  1991     1   0  44  0 14124     0 select Ds    ??  11654240:02.00 [perl5.8.9]
  525  2030  1991   0  44  0 15148     0 lockf  D     ??  2188744:12.00 [perl5.8.9]
  525  2031  1991   0  44  0 15148     0 lockf  D     ??  1591237:35.00 [perl5.8.9]
  525  2032  1991   0  44  0 14124     0 accept D     ??  1564262:27.00 [perl5.8.9]
  525  2033  1991   0  44  0 14124     0 lockf  D     ??  1607096:57.00 [perl5.8.9]
  525  2034  1991   0  44  0 14124     0 lockf  D     ??  1583604:57.00 [perl5.8.9]
    0  2057     1   0  44  0  3304     0 kqread Ds    ??  2965941:47.00 [dovecot]
    0  2060  2057   0  44  0  3432     0 kqread D     ??  1118817:49.00 [dovecot-au
    0  2065  2057   0  44  0  3468     0 kqread D     ??  513340:18.00 [dovecot-au
    0  2132     1   0  44  0  6500     0 kqread Ds    ??  3443138:43.00 [master]
  125  2137  2132   0  44  0  7524     0 kqread D     ??  1402138:53.00 [qmgr]
    0  2156     1   0  44  0  4272     0 select D     ??  1091792:27.00 [atalkd]
  143  2157  2057   0  68  0  4980     0 kqread D     ??  334858:59.00 [pop3-login
  143  2158  2057   0  67  0  4980     0 kqread D     ??  328540:01.00 [pop3-login
  143  2159  2057   0  69  0  4980     0 kqread D     ??  316152:32.00 [pop3-login
  143  2160  2057   0  69  0  4980     0 kqread D     ??  326030:43.00 [imap-login
  143  2161  2057   0  69  0  4980     0 kqread D     ??  322926:18.00 [imap-login
  143  2162  2057   0  44  0  4980     0 kqread D     ??  2163645:10.00 [imap-login
    0  2166     1   0  44  0  4272     0 select D     ??  71607:13.00 [papd]
    0  2168     1   0  44  0  4272     0 select D     ??  260114:01.00 [cnid_metad
    0  2170     1   0  44  0  6724     0 select D     ??  267703:28.00 [afpd]
10007  2178     1   0  44  0 43312     0 ucond  Ds    ??  -27838512:-51.55 [mediatomb]
   91  2182     1   0  64  0 12420     0 wait   Ds    ??  352877:08.00 [python2.6]
   91  2189  2182   0  44  0 11392     0 select D     ??  17705890:45.00 [python2.6]
   91  2190  2182   0  44  0 11392     0 select D     ??  18695687:33.00 [python2.6]
   91  2191  2182   0  44  0 11392     0 select D     ??  17631947:36.00 [python2.6]
   91  2192  2182   0  44  0 11392     0 select D     ??  28809461:06.00 [python2.6]
   91  2193  2182   0  44  0 11392     0 select D     ??  16992298:05.00 [python2.6]
   91  2194  2182   0  44  0 11392     0 select D     ??  21944252:02.00 [python2.6]
   91  2195  2182   0  44  0 11392     0 select D     ??  19557856:00.00 [python2.6]
   91  2196  2182   0  44  0 11392     0 select D     ??  7413230:11.00 [python2.6]
    1  2204     1   0  44  0  3316     0 select Ds    ??  2011890:20.00 [healthd]
10000  2271     1   0  76  0  3092     0 select Ds    ??  30011:08.00 [cvsupd]
    0  2301     1   0  44  0 15924     0 select Ds    ??  14008773:10.00 [httpd]
    0  2313  2301   0  55  0  1680     0 piperd D     ??  119320:12.00 [cronolog]
    0  2314  2301   0  44  0  1680     0 piperd D     ??  109748:52.00 [cronolog]
    0  2315  2301   0  76  0  1680     0 piperd D     ??  105791:45.00 [cronolog]
    0  2316  2301   0  44  0  1680     0 piperd D     ??  127066:48.00 [cronolog]
    0  2317  2301   0  44  0  1680     0 piperd D     ??  136628:04.00 [cronolog]
    0  2331     1   0  45  0  6676     0 select Ds    ??  162543:03.00 [sshd]
    0  2337     1   0  44  0  3372     0 nanslp Ds    ??  443673:16.00 [cron]
    0  2377     1   0  76  0  3404     0 select Ds    ??  220042:04.00 [inetd]
    0  2410     1   0  76  0  3344     0 ttyin  Ds+   ??  82535:06.00 [getty]
    0  2411     1   0  76  0  3344     0 ttyin  Ds+   ??  67145:53.00 [getty]
    0  2412     1   0  76  0  3344     0 ttyin  Ds+   ??  70385:00.00 [getty]
    0  2413     1   0  76  0  3344     0 ttyin  Ds+   ??  63195:41.00 [getty]
    0  2414     1   0  76  0  3344     0 ttyin  Ds+   ??  62258:16.00 [getty]
    0  2415     1   0  76  0  3344     0 ttyin  Ds+   ??  61955:42.00 [getty]
    0  2416     1   0  76  0  3344     0 ttyin  Ds+   ??  60616:07.00 [getty]
    0  2417     1   0  76  0  3344     0 ttyin  Ds+   ??  61471:33.00 [getty]
   80  2418  2301   0  76  0 19044     0 umtxn  D     ??  2971196:43.00 [httpd]
   80  2419  2301   0  44  0 16948     0 lockf  D     ??  5150863:28.00 [httpd]
   80  2420  2301   0  44  0 16948     0 lockf  D     ??  4974529:14.00 [httpd]
   80  2421  2301   0  44  0 16948     0 lockf  D     ??  692727:30.00 [httpd]
   80  2422  2301   0  44  0 16948     0 lockf  D     ??  1360146:54.00 [httpd]
   80  2423  2301   0  44  0 16948     0 lockf  D     ??  700525:11.00 [httpd]
  125  2438  2132   0  44  0  7524     0 kqread D     ??  798988:16.00 [tlsmgr]
  143  2475  2057   0  44  0  4980     0 kqread D     ??  1905057:40.00 [imap-login
 4001  2476  2057   0  44  0  5360     0 kqread D     ??  3013165:20.00 [imap]
   80  2558  2301   0  44  0 16948     0 kqread D     ??  703880:19.00 [httpd]
 4001  2566  2057   0  44  0  5360     0 kqread D     ??  2270297:37.00 [imap]
  143  2567  2057   0  44  0  4980     0 kqread D     ??  334086:03.00 [imap-login
   80  2692  2301   0  44  0 16948     0 lockf  D     ??  645967:24.00 [httpd]
    0  2938     1   0  44  0 47132     0 select Ds    ??  -28205897:-34.55 [perl5.8.9]
  110  2968     1   0  44  0 74804     0 select Ds    ??    0:00.00 [perl]
    0  2971  2938   0  44  0 47132     0 select D     ??    0:00.00 [perl5.8.9]
    0  2972  2938   0  44  0 47132     0 select D     ??    0:00.00 [perl5.8.9]
  110  2973  2968   0  44  0 77484     0 select D     ??    0:00.00 [perl]
  110  2974  2968   0  51  0 78508     0 lockf  D     ??    0:00.00 [perl]
   80  3294  2301   0  44  0 16948     0 lockf  D     ??    0:00.00 [httpd]
    0  5481  2331   0  48  0  9400     0 sbwait Ds    ??    0:00.00 [sshd]
 4001  5483  5481   0  44  0  9400     0 select D     ??    0:00.00 [sshd]
 4001  5484  5483   0  44  0  4572     0 ttyin  Ds+   ??    0:00.00 [zsh]
  125  6384  2132   0  44  0  7524     0 kqread D     ??    0:00.00 [pickup]
  125  6882  2132   0  44  0  7524     0 kqread D     ??    0:00.00 [anvil]
  125  6983  2132   0  44  0  7524     0 kqread D     ??    0:00.00 [smtpd]
    0  6997  1759   0  44  0 16452     0 select D     ??    0:00.00 [smbd]

------------------------------------------------------------------------
vmstat -s

 53356942 cpu context switches
 17642994 device interrupts
  5258295 software interrupts
  2429008 traps
 11206442 system calls
       23 kernel threads created
     6612  fork() calls
      367 vfork() calls
        0 rfork() calls
       88 swap pager pageins
       94 swap pager pages paged in
       92 swap pager pageouts
       99 swap pager pages paged out
     2522 vnode pager pageins
    17824 vnode pager pages paged in
      675 vnode pager pageouts
     1117 vnode pager pages paged out
      191 page daemon wakeups
  1308250 pages examined by the page daemon
     6310 pages reactivated
   355331 copy-on-write faults
      203 copy-on-write optimized faults
   835416 zero fill pages zeroed
    13431 zero fill pages prezeroed
      195 intransit blocking page faults
  1510513 total VM faults taken
        0 pages affected by kernel thread creation
  1274059 pages affected by  fork()
    76270 pages affected by vfork()
        0 pages affected by rfork()
  1279355 pages cached
  2810551 pages freed
        0 pages freed by daemon
   630287 pages freed by exiting processes
   101514 pages active
    43383 pages inactive
       70 pages in VM cache
    37190 pages wired down
    72833 pages free
     4096 bytes per page
  5063969 total name lookups
          cache hits (92% pos + 6% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
       kbdmux     6    10K       -        6  16,256,1024,2048,4096
         cdev    11     2K       -       11  128
CAM dev queue     4     1K       -        4  64
        sigio     1     1K       -        1  32
     filedesc   208    90K       -    10219  16,32,64,128,256,512,1024,2048,4096
         kenv    81     7K       -       87  16,32,64,128,4096
       kqueue    64    39K       -    67109  128,1024,4096
    proc-args    93     8K       -     5703  32,64,128,256
      acpidev    51     2K       -       51  32
      ithread    72     6K       -       72  16,64,128
    CAM queue    14     1K       -       88  16
       KTRACE   100    13K       -      100  128
      scsi_cd     0     0K       -        3  16
       linker   132    89K       -      281  16,32,64,128,256,1024,2048,4096
        lockf   196     9K       -   274510  32,64,128,256
      CAM SIM     4     1K       -        4  128
       ip6ndp    13     2K       -       18  64,128
         temp   141   238K       -    25129  16,32,64,128,256,512,1024,2048,4096
       devbuf  3581  4215K       -    47981  16,32,64,128,256,512,1024,2048,4096
     ppbusdev     2     1K       -        2  128
       module   273    18K       -      273  64,128
      entropy  1024    64K       -     1024  64
     mtx_pool     2     8K       -        2  4096
         UART     3     2K       -        3  16,256,1024
          osd     2     1K       -        2  16,32
     pci_link    16     2K       -       16  32,128
      subproc   308   595K       -     7174  256,4096
         proc     2     8K       -        2  4096
      session    54     4K       -     1113  64
         pgrp    58     4K       -     1125  64
         cred   386    37K       -  3720382  64,128
      uidinfo    17     2K       -      579  64,1024
       plimit    37    10K       -     5962  256
       acpica  1891    99K       -    44787  16,32,64,128,256,512,1024
    sysctltmp     0     0K       -    16923  16,32,64,128
    sysctloid  3244    99K       -     3363  16,32,64,128
       sysctl     0     0K       -     6115  16,32,64
         umtx   273    18K       -      273  64
     p1003.1b     1     1K       -        1  16
         SWAP     2  1097K       -        2  64
       bus-sc    71   138K       -     1656  16,32,64,128,256,512,1024,2048,4096
          bus   740    35K       -     3914  16,32,64,128,512,1024
      devstat    10    21K       -       10  16,4096
 eventhandler    85     5K       -       85  32,64,128
     acpitask     1     1K       -        1  1024
         kobj   168   336K       -      194  2048
      Per-cpu     1     1K       -        1  16
          agp     2    33K       -        2  16
         rman   170    11K       -      565  16,32,64
         sbuf     0     0K       -      414  16,32,64,128,256,512,1024,2048,4096
       USBdev    20     6K       -       20  32,128,1024
          USB    35     6K       -       35  16,32,64,1024
      CAM XPT    30     6K       -      138  16,32,64,256,1024,2048
        stack     0     0K       -        2  128
    taskqueue    13     1K       -       13  16,64
       Unitno    14     1K       -       56  16,64
          iov     5     1K       -   486349  16,32,64,128,256
       select   116     8K       -      116  64
     ioctlops     0     0K       -    20629  16,32,64,128,256,512,1024,2048,4096
          msg     4    25K       -        4  1024,4096
          sem     4    11K       -        4  1024,2048,4096
          shm    15    26K       -      489  1024
          tty    19    10K       -       21  512,1024
          pts     1     1K       -        1  128
         accf    11     1K       -       13  16,32
     mbuf_tag     5     1K       - 24482778  32,64
         ksem     1     4K       -        1  4096
        shmfd     1     4K       -        1  4096
          pcb   133    82K       -    16101  16,64,128,512,1024,2048,4096
       soname    55     7K       -   596511  16,32,128
       biobuf     8     8K       -      116  1024,2048
     vfscache     1   512K       -        1  
   cl_savebuf     0     0K       -      440  32,64
  export_host    27     4K       -       27  128
     vfs_hash     1   256K       -        1  
       vnodes     4     1K       -        4  32,128
      acpisem    15     2K       -       15  64,128
  vnodemarker     0     0K       -    15731  512
        mount   170     7K       -     1119  16,32,64,128,256
          BPF    16    18K       -       16  64,128,256,4096
  ether_multi   117     5K       -      136  16,32,64
       ifaddr   111    24K       -      111  16,32,64,128,256,512,2048
        ifnet     9     9K       -        9  64,1024
        clone    11    41K       -       11  16,4096
       arpcom     4     1K       -        4  16
          gif     1     1K       -        1  128
      lltable    51    10K       -       64  128,256
   CAM periph     4     1K       -       12  16,32,64,128
  ata_generic     3     3K       -      149  16,512,1024
       DEVFS1    97    25K       -      109  256
       DEVFS3   222    28K       -      235  128
       DEVFS2    96     2K       -       96  16
   DEVFS_RULE    42    10K       -       80  32,256
        DEVFS    28     1K       -       29  16,64
     routetbl   128   267K       -     4688  16,32,64,128,256,512
       DEVFSP     2     1K       -        2  32
         igmp     8     1K       -        8  128
  ntfs_nthash     1   256K       -        1  
  ip_moptions     4     1K       -     2844  32,128
     in_multi     5     1K       -        5  128
   in_mfilter     2     1K       -     1422  512
encap_export_host     3     3K       -        3  1024
    sctp_iter     0     0K       -        8  128
     sctp_ifn     5     1K       -        5  128
     sctp_ifa    12     2K       -       12  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -        8  16
    hostcache     1    16K       -        1  
     syncache     1    72K       -        1  
 ip6_moptions     2     1K       -        2  32,128
    in6_multi    54     7K       -       54  16,256
  in6_mfilter     1     1K       -        1  512
          mld     8     1K       -        8  128
  inpcbpolicy   180     3K       -    17904  16
  ipsecpolicy   360    45K       -    37460  128
      NFS FHA     2     2K       -  1696744  32,1024
          NLM     1     2K       -        2  16,2048
       crypto     1     1K       -        1  512
        xform     0     0K       -     1965  16,32
          rpc  1160   287K       -  5258789  16,32,64,128,256,512,1024,2048,4096
audit_evclass   172     3K       -      211  16
     savedino     0     0K       -     1973  256
       dirrem     1     1K       -     6170  32
        mkdir     0     0K       -       58  32
       diradd     4     1K       -     6186  64
     freefile     4     1K       -     6055  32
     freeblks     4     1K       -     4131  256
     freefrag     0     0K       -     1846  32
   allocindir     0     0K       -    66941  64
     indirdep     0     0K       -    15751  32
  allocdirect     1     1K       -     5225  128
    bmsafemap     1     1K       -     5825  64
       newblk     1     1K       -    72167  64,256
     inodedep    11   258K       -     7689  128
      pagedep     5    33K       -     1540  64
  ufs_dirhash   253    54K       -      391  16,32,64,128,256,512
    ufs_mount    19   194K       -       34  64,256,512,2048,4096
      UMAHash     1     1K       -        2  256,512
    vm_pgdata     2    65K       -        2  64
     atkbddev     2     1K       -        2  32
    pfs_nodes   173    22K       -      173  128
  pfs_vncache     3     1K       -        5  32
       cpuctl     1     1K       -        1  16
       apmdev     1     1K       -        1  64
    ad_driver     2     1K       -        2  32
         GEOM   121    29K       -      619  16,32,64,128,512,1024,2048
    ar_driver     0     0K       -       12  512,2048
      io_apic     1     1K       -        1  1024
      memdesc     1     4K       -        1  4096
     nexusdev     5     1K       -        5  16
       isadev     5     1K       -        5  64
        linux    12     1K       -       12  32,64
   acd_driver     1     2K       -        1  2048
      ata_cam     3     3K       -       20  64,256,1024

------------------------------------------------------------------------
vmstat -z

ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES

UMA Kegs:                 128,        0,      112,        8,      112,        0
UMA Zones:                888,        0,      112,        0,      112,        0
UMA Slabs:                284,        0,      616,       14,    53570,        0
UMA RCntSlabs:            544,        0,      886,        3,      893,        0
UMA Hash:                 128,        0,        4,       26,        5,        0
16 Bucket:                 76,        0,       29,       21,       68,        0
32 Bucket:                140,        0,       25,       31,       77,        0
64 Bucket:                268,        0,       34,        8,      129,       13
128 Bucket:               524,        0,      138,        2,     1759,      264
VM OBJECT:                136,        0,    19237,       77,   168304,        0
MAP:                      140,        0,        7,       21,        7,        0
KMAP ENTRY:                72,    56604,       40,      119,   118702,        0
MAP ENTRY:                 72,        0,     5756,      498,   358332,        0
DP fakepg:                 72,        0,        0,        0,        0,        0
SG fakepg:                 72,        0,        0,        0,        0,        0
mt_zone:                 2056,        0,      268,        0,      268,        0
16:                        16,        0,     3451,      203,  1130766,        0
32:                        32,        0,     2645,     1084,  1771740,        0
64:                        64,        0,     5103,     7051, 26781831,        0
128:                      128,        0,     3350,      370,  1993307,        0
256:                      256,        0,     2000,      265,  1727190,        0
512:                      512,        0,      198,       18,   191011,        0
1024:                    1024,        0,      122,       66,    37786,        0
2048:                    2048,        0,      201,       23,  3393661,        0
4096:                    4096,        0,      205,       24,    13312,        0
Files:                     56,        0,      817,      188,   429250,        0
TURNSTILE:                 72,        0,      274,       26,      274,        0
umtx pi:                   52,        0,        0,        0,        0,        0
MAC labels:                20,        0,        0,        0,        0,        0
PROC:                     680,        0,      137,       31,     7002,        0
THREAD:                   572,        0,      248,       25,      258,        0
SLEEPQUEUE:                40,        0,      274,       80,      274,        0
VMSPACE:                  232,        0,      115,       38,     6981,        0
cpuset:                    40,        0,        2,      182,        2,        0
audit_record:             816,        0,        0,        0,        0,        0
mbuf_packet:              256,        0,      455,      189, 11819032,        0
mbuf:                     256,        0,     1025,      900, 27410975,        0
mbuf_cluster:            2048,    25600,      842,      860,  3035812,        0
mbuf_jumbo_page:         4096,    12800,        0,       35,     9544,        0
mbuf_jumbo_9k:           9216,    19200,        0,        0,        0,        0
mbuf_jumbo_16k:         16384,    12800,        0,        0,        0,        0
mbuf_ext_refcnt:            4,        0,        0,      203,      558,        0
NetGraph items:            36,     4130,        0,        0,        0,        0
NetGraph data items:       36,      531,        0,        0,        0,        0
g_bio:                    152,        0,        0,     1222,  3323195,        0
ttyinq:                   152,        0,      135,       21,      495,        0
ttyoutq:                  256,        0,       72,       18,      264,        0
ata_request:              204,        0,        1,      540,   833273,        0
ata_composite:            180,        0,        0,        0,        0,        0
cryptop:                   60,        0,        0,        0,        0,        0
cryptodesc:                56,        0,        0,        0,        0,        0
VNODE:                    268,        0,    19941,      107,    26010,        0
VNODEPOLL:                 60,        0,        2,      124,        7,        0
S VFS Cache:               72,        0,    19516,       41,    62727,        0
L VFS Cache:              292,        0,     1036,        4,     1131,        0
NAMEI:                   1024,        0,        1,       11,  1251704,        0
NFSMOUNT:                 520,        0,        1,       13,        1,        0
NFSNODE:                  464,        0,       75,       13,       75,        0
DIRHASH:                 1024,        0,      348,       12,      781,        0
UDF translation buffer, zone:      510,        0,        0,        0,        0,        0
UDF Node zone:             20,        0,        0,        0,        0,        0
UDF Dirstream zone:        48,        0,        0,        0,        0,        0
pipe:                     392,        0,       87,       13,     2724,        0
ksiginfo:                  80,        0,      149,       43,      149,        0
itimer:                   220,        0,        1,       35,        2,        0
KNOTE:                     72,        0,      221,       97,   276469,        0
socket:                   412,    25605,      326,       70,    24685,        0
ipq:                       32,      904,        0,        0,        0,        0
udp_inpcb:                220,    25614,       93,       51,    16733,        0
udpcb:                      8,    25781,       93,      110,    16733,        0
tcp_inpcb:                220,    25614,       86,       22,     1170,        0
tcpcb:                    632,    25602,       86,       16,     1170,        0
tcptw:                     52,     5184,        0,      144,       92,        0
syncache:                 112,    15365,        0,       70,      852,        0
hostcache:                 76,    15400,       46,       54,      235,        0
tcpreass:                  20,     1690,        0,      169,       20,        0
sackhole:                  20,        0,        0,      169,     1115,        0
sctp_ep:                  860,    25600,        0,        0,        0,        0
sctp_asoc:               1492,    40000,        0,        0,        0,        0
sctp_laddr:                24,    80040,        0,      145,       11,        0
sctp_raddr:               432,    80001,        0,        0,        0,        0
sctp_chunk:                96,   400000,        0,        0,        0,        0
sctp_readq:                76,   400000,        0,        0,        0,        0
sctp_stream_msg_out:       64,   400020,        0,        0,        0,        0
sctp_asconf:               24,   400055,        0,        0,        0,        0
sctp_asconf_ack:           24,   400055,        0,        0,        0,        0
ripcb:                    220,    25614,        1,       35,        1,        0
unpcb:                    172,    25622,      134,       73,     6743,        0
rtentry:                  108,        0,       51,       57,       51,        0
pfsrctrpl:                124,    10013,        0,        0,        0,        0
pfrulepl:                 828,        0,      109,        3,      109,        0
pfstatepl:                284,    10010,      697,     6009,   916103,        0
pfaltqpl:                 224,        0,        0,        0,        0,        0
pfpooladdrpl:              68,        0,        9,      103,        9,        0
pfrktable:               1240,     1002,        2,        4,        3,        0
pfrkentry:                156,   200000,        5,       45,        5,        0
pfrkentry2:               156,        0,        0,        0,        0,        0
pffrent:                   16,     5075,        0,      203,     1876,        0
pffrag:                    48,        0,        0,      156,      707,        0
pffrcache:                 48,    10062,        0,        0,        0,        0
pffrcent:                  12,    50141,        0,        0,        0,        0
pfstatescrub:              28,        0,        0,        0,        0,        0
pfiaddrpl:                100,        0,       27,       51,       27,        0
pfospfen:                 108,        0,      696,       24,      696,        0
pfosfp:                    28,        0,      407,      228,      407,        0
selfd:                     28,        0,      274,      234,  1499195,        0
ip4flow:                   40,     4140,        9,      267,     6896,        0
ip6flow:                   64,     4118,        0,        0,        0,        0
SWAPMETA:                 276,   121576,       54,       16,       60,        0
Mountpoints:              644,        0,       10,        8,       10,        0
FFS inode:                116,        0,    19804,       29,    25870,        0
FFS1 dinode:              128,        0,        0,        0,        0,        0
FFS2 dinode:              256,        0,    19804,       56,    25870,        0
bridge_rtnode:             36,        0,        5,      197,       13,        0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq6: fdc0                             5          0
irq15: ata1                           66          0
irq16: em0                      11610276      12457
irq17: fxp0                      5191368       5570
irq20: atapci0                    829950        890
irq23: vr0                         11329         12
cpu0: timer                     56662583      60796
Total                           74305577      79727

------------------------------------------------------------------------
pstat -T

817/12328 files
0M/8191M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/ad4s1b       16776960      696 16776264     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
iostat: kvm_getcptime: invalid address (0x0)
iostat: disabling CPU time statistics
             ad4              ad6              cd0 
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s 
 16.53 638 10.30  16.44 252  4.04   0.00   0  0.00 

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   
m        65536      5432001 --rw------- pgsql    pgsql    pgsql    pgsql               4     29368320         1932         1932  4:07:57 11:58:56  4:07:57
m        65537     22837368 --rw------- root     wheel    root     wheel               9        77852         2301         2301  4:08:54  8:29:30  4:08:54

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   
s        65536      5432001 --rw------- pgsql    pgsql    pgsql    pgsql              17  4:07:57  4:07:57
s        65537      5432002 --rw------- pgsql    pgsql    pgsql    pgsql              17  4:07:57  4:07:57
s        65538      5432003 --rw------- pgsql    pgsql    pgsql    pgsql              17  4:07:57  4:07:57
s        65539      5432004 --rw------- pgsql    pgsql    pgsql    pgsql              17  4:07:57  4:07:57
s        65540      5432005 --rw------- pgsql    pgsql    pgsql    pgsql              17  4:07:57  4:07:57
s        65541      5432006 --rw------- pgsql    pgsql    pgsql    pgsql              17  4:07:57  4:07:57
s        65542      5432007 --rw------- pgsql    pgsql    pgsql    pgsql              17  4:07:57  4:07:57


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:     33554432	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:         8192	(max amount of shared memory in pages)

seminfo:
	semmap:           30	(# of entries in semaphore map)
	semmni:           40	(# of semaphore identifiers)
	semmns:          300	(# of semaphores in system)
	semmnu:           30	(# of undo structures in system)
	semmsl:           60	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           10	(max # of undo entries per process)
	semusz:          136	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
   168083         0        74        87         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         4         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0    168248
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0    168083     55953        74         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses
    55931        87         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
  1616051        79     14155         1      3804       280        79         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
       79         0         0         0         0       553         0     54332
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0      6317        11         0       280
Server Ret-Failed
              171
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
      280       280         0

------------------------------------------------------------------------
netstat -s

tcp:
	1833843 packets sent
		1815876 data packets (360947517 bytes)
		4733 data packets (5682536 bytes) retransmitted
		52 data packets unnecessarily retransmitted
		18 resends initiated by MTU discovery
		11339 ack-only packets (968 delayed)
		0 URG only packets
		0 window probe packets
		837 window update packets
		1172 control packets
	1796616 packets received
		1762309 acks (for 360746924 bytes)
		2838 duplicate acks
		0 acks for unsent data
		1720555 packets (232994084 bytes) received in-sequence
		301 completely duplicate packets (16894 bytes)
		0 old duplicate packets
		1 packet with some dup. data (20 bytes duped)
		20 out-of-order packets (1623 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		773 window update packets
		126 packets received after close
		198 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	270 connection requests
	823 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	102 ignored RSTs in the windows
	1091 connections established (including accepts)
	1089 connections closed (including 203 drops)
		840 connections updated cached RTT on close
		849 connections updated cached RTT variance on close
		153 connections updated cached ssthresh on close
	0 embryonic connections dropped
	1759745 segments updated rtt (of 1712673 attempts)
	2711 retransmit timeouts
		5 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	5 keepalive timeouts
		5 keepalive probes sent
		0 connections dropped by keepalive
	7432 correct ACK header predictions
	28742 correct data packet header predictions
	851 syncache entries added
		154 retransmitted
		122 dupsyn
		1 dropped
		823 completed
		0 bucket overflow
		0 cache overflow
		8 reset
		20 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	852 cookies sent
	0 cookies received
	276 SACK recovery episodes
	546 segment rexmits in SACK recovery episodes
	785992 byte rexmits in SACK recovery episodes
	2652 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	456353 datagrams received
	0 with incomplete header
	0 with bad data length field
	11 with bad checksum
	0 with no checksum
	550 dropped due to no socket
	28121 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	427671 delivered
	429287 datagrams output
	0 times multicast source filter matched
ip:
	12215925 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	2251135 packets for this host
	31 packets for unknown/unsupported protocol
	9953371 packets forwarded (0 packets fast forwarded)
	356 packets not forwardable
	269 packets received for unknown multicast group
	0 redirects sent
	2269230 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	23 output packets discarded due to no route
	639 output datagrams fragmented
	1334 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	3478 calls to icmp_error
	0 errors not generated in response to an icmp message
	Output histogram:
		echo reply: 8
		destination unreachable: 3459
	0 messages with bad code fields
	0 messages less than the minimum length
	47 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	Input histogram:
		destination unreachable: 189
		echo: 8
	8 message responses generated
	0 invalid return addresses
	0 no return routes
igmp:
	0 messages received
	0 messages received with too few bytes
	0 messages received with wrong TTL
	0 messages received with bad checksum
	0 V1/V2 membership queries received
	0 V3 membership queries received
	0 membership queries received with invalid field(s)
	0 general queries received
	0 group queries received
	0 group-source queries received
	0 group-source queries dropped
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 V3 reports received without Router Alert
	0 membership reports sent
ipsec:
	0 inbound packets processed successfully
	0 inbound packets violated process security policy
	0 inbound packets with no SA available
	0 invalid inbound packets
	0 inbound packets failed due to insufficient memory
	0 inbound packets failed getting SPI
	0 inbound packets failed on AH replay check
	0 inbound packets failed on ESP replay check
	0 inbound packets considered authentic
	0 inbound packets failed on authentication
	0 outbound packets processed successfully
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 invalid outbound packets
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route
	0 SPD cache lookups
	0 SPD cache misses
	0 inbound packets violated process security policy
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	0 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
ah:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets dropped; bad authentication detected
	0 packets dropped; bad authentication length
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
esp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 packets dropped; bad ilen
	0 replay counter wraps
	0 packets dropped; bad encryption detected
	0 packets dropped; bad authentication detected
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
ipcomp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 packets sent uncompressed; size < compr. algo. threshold
	0 packets sent uncompressed; compression was useless
pfsync:
	0 packets received (IPv4)
	0 packets received (IPv6)
		0 packets discarded for bad interface
		0 packets discarded for bad ttl
		0 packets shorter than header
		0 packets discarded for bad version
		0 packets discarded for bad HMAC
		0 packets discarded for bad action
		0 packets discarded for short packet
		0 states discarded for bad values
		0 stale states
		0 failed state lookup/inserts
	0 packets sent (IPv4)
	0 packets sent (IPv6)
		0 send failed due to mbuf memory error
		0 send error
ip6:
	2076 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	2052 packets for this host
	0 packets forwarded
	0 packets not forwardable
	0 redirects sent
	2027 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	52 multicast packets which we don't join
	Input histogram:
		hop by hop: 40
		UDP: 2024
		ICMP6: 12
	Mbuf statistics:
		585 one mbuf
		two or more mbuf:
			lo0= 2
			bridge0= 38
		1451 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not continuous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	0 failures of source address selection
	Source addresses selection rule applied:
		12 first candidate
		9 same address
		4 appropriate scope
		4 matching label
icmp6:
	38 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	Output histogram:
		neighbor solicitation: 5
		MLDv2 listener report: 21
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		38 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	0 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
ipsec6:
	0 inbound packets processed successfully
	0 inbound packets violated process security policy
	0 inbound packets with no SA available
	0 invalid inbound packets
	0 inbound packets failed due to insufficient memory
	0 inbound packets failed getting SPI
	0 inbound packets failed on AH replay check
	0 inbound packets failed on ESP replay check
	0 inbound packets considered authentic
	0 inbound packets failed on authentication
	0 outbound packets processed successfully
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 invalid outbound packets
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route
	0 SPD cache lookups
	0 SPD cache misses
	0 inbound packets violated process security policy
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	0 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output
pfkey:
	0 requests sent from userland
	0 bytes sent from userland
	0 messages with invalid length field
	0 messages with invalid version field
	0 messages with invalid message type field
	0 messages too short
	0 messages with memory allocation failure
	0 messages with duplicate extension
	0 messages with invalid extension type
	0 messages with invalid sa type
	0 messages with invalid address extension
	0 requests sent to userland
	0 bytes sent to userland
	0 messages toward single socket
	0 messages toward all sockets
	0 messages toward registered sockets
	0 messages with memory allocation failure
ddp:
	0 packets with short headers 
	2861 packets with long headers 
	0 packets with no checksum 
	0 packets too short 
	0 packets with bad checksum 
	0 packets with not enough data 
	0 packets forwarded 
	0 packets encapsulated 
	0 packets rcvd for unreachable dest 
	0 packets dropped due to no socket space 

------------------------------------------------------------------------
netstat -m

1480/1089/2569 mbufs in use (current/cache/total)
653/1049/1702/25600 mbuf clusters in use (current/cache/total/max)
455/189 mbuf+clusters out of packet secondary zone in use (current/cache)
0/35/35/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
1676K/2510K/4186K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
19 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs    Opkts Oerrs  Coll Drop
em0    1500 <Link#1>      00:0e:0c:5f:c1:f8  6583446     0  6999903     0     0    0 
em0    1500 fe80:1::20e:c fe80:1::20e:cff:f        0     -        3     -     -    - 
fxp0   1500 <Link#2>      00:90:27:62:87:4d  5204566     0  4815643     0     0    0 
fxp0   1500 68.164.219.96 68.164.219.98        20954     -    34469     -     -    - 
fxp0   1500 fe80:2::290:2 fe80:2::290:27ff:        0     -        5     -     -    - 
fxp0   1500 68.164.219.99 68.164.219.99          917     -        0     -     -    - 
fxp0   1500 68.164.219.10 68.164.219.100         917     -        0     -     -    - 
fxp0   1500 68.164.219.10 68.164.219.101        1050     -        0     -     -    - 
vr0    1500 <Link#3>      00:15:f2:43:48:7b      793     0    21681     0     0    0 
vr0    1500 fe80:3::215:f fe80:3::215:f2ff:        0     -        3     -     -    - 
pflog 33200 <Link#4>                               0     0     7191     0     0    0 
pfsyn  1460 <Link#5>                               0     0        0     0     0    0 
lo0   16384 <Link#6>                          412960     0   412955     0     0    0 
lo0   16384 your-net      localhost           341598     -   410946     -     -    - 
lo0   16384 localhost     ::1                      0     -     1988     -     -    - 
lo0   16384 fe80:6::1     fe80:6::1                0     -        0     -     -    - 
lo0   16384 atalk:0            0.0                 -     -        -     -     -    - 
bridg  1500 <Link#7>      e2:bf:e8:32:f6:79  6590490     0  7021583     0     0    0 
bridg  1500 192.168.0.0   192.168.0.1        1835800     -  1825001     -     -    - 
bridg  1500 192.168.0.254 192.168.0.254        19186     -        0     -     -    - 
bridg  1500 atalk:1-128        1.1                 -     -        -     -     -    - 
gif0   1280 <Link#8>                               4     0       12     2     0    0 
gif0   1280 fe80:8::20e:c fe80:8::20e:cff:f        0     -        8     -     -    - 
gif0   1280 2001:4978:f:3 2001:4978:f:309::        0     -        0     -     -    - 

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            68.164.219.97      UGS         4  4814425   fxp0
68.164.219.96/29   link#2             U           1      915   fxp0
68.164.219.98      link#2             UHS         0       18    lo0
68.164.219.99      link#2             UHS         0      917    lo0 =>
68.164.219.99/32   link#2             U           0        0   fxp0
68.164.219.100     link#2             UHS         0      917    lo0 =>
68.164.219.100/32  link#2             U           0        0   fxp0
68.164.219.101     link#2             UHS         0      917    lo0 =>
68.164.219.101/32  link#2             U           0        0   fxp0
127.0.0.1          link#6             UH          0   341695    lo0
192.168.0.0/24     link#7             U           4  6994012 bridge
192.168.0.1        link#7             UHS         0    53605    lo0
192.168.0.254      link#7             UHS         0    12974    lo0 =>
192.168.0.254/32   link#7             U           0        0 bridge

AppleTalk:
Destination        Gateway            Flags    Netif Expire
0                  0.0                U         lo0
1                  1.1                U      bridge
1.1                0.0                UH        lo0
2-3                1.1                U      bridge
4-7                1.1                U      bridge
8-15               1.1                U      bridge
16-31              1.1                U      bridge
32-63              1.1                U      bridge
64-127             1.1                U      bridge
128                1.1                U      bridge

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0 =>
default                           2001:4978:f:309::1            UGS        gif0
::1                               ::1                           UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2001:4978:f:309::1                2001:4978:f:309::2            UH         gif0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em0/64                     link#1                        U           em0
fe80::20e:cff:fe5f:c1f8%em0       link#1                        UHS         lo0
fe80::%fxp0/64                    link#2                        U          fxp0
fe80::290:27ff:fe62:874d%fxp0     link#2                        UHS         lo0
fe80::%vr0/64                     link#3                        U           vr0
fe80::215:f2ff:fe43:487b%vr0      link#3                        UHS         lo0
fe80::%lo0/64                     link#6                        U           lo0
fe80::1%lo0                       link#6                        UHS         lo0
fe80::%8/64                       link#8                        U          gif0
fe80::20e:cff:fe5f:c1f8%8         link#8                        UHS         lo0
ff01:1::/32                       fe80::20e:cff:fe5f:c1f8%em0   U           em0
ff01:2::/32                       fe80::290:27ff:fe62:874d%fxp0 U          fxp0
ff01:3::/32                       fe80::215:f2ff:fe43:487b%vr0  U           vr0
ff01:6::/32                       ::1                           U           lo0
ff01:8::/32                       fe80::20e:cff:fe5f:c1f8%8     U          gif0
ff02::/16                         ::1                           UGRS        lo0
ff02::%em0/32                     fe80::20e:cff:fe5f:c1f8%em0   U           em0
ff02::%fxp0/32                    fe80::290:27ff:fe62:874d%fxp0 U          fxp0
ff02::%vr0/32                     fe80::215:f2ff:fe43:487b%vr0  U           vr0
ff02::%lo0/32                     ::1                           U           lo0
ff02::%8/32                       fe80::20e:cff:fe5f:c1f8%8     U          gif0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb    Proto Recv-Q Send-Q  Local Address      Foreign Address   (state)
c501c768 tcp4       0      0 192.168.0.1.445    192.168.0.17.49685 ESTABLISHED
c47ce000 tcp4       0      0 127.0.0.1.10023    127.0.0.1.51301    ESTABLISHED
c4c839e0 tcp4       0      0 127.0.0.1.51301    127.0.0.1.10023    ESTABLISHED
c4fa59e0 tcp4       0      0 127.0.0.1.10028    127.0.0.1.35839    FIN_WAIT_2
c49a2000 tcp4       0      0 127.0.0.1.35839    127.0.0.1.10028    CLOSE_WAIT
c501d278 tcp4       0      0 127.0.0.1.10027    127.0.0.1.46557    CLOSED
c501d4f0 tcp4       0      0 127.0.0.1.10028    127.0.0.1.31284    FIN_WAIT_2
c4fa6000 tcp4       0      0 127.0.0.1.31284    127.0.0.1.10028    CLOSE_WAIT
c501d9e0 tcp4       0      0 127.0.0.1.10027    127.0.0.1.48438    CLOSED
c4c824f0 tcp4       0      0 127.0.0.1.10028    127.0.0.1.54423    FIN_WAIT_2
c501d768 tcp4       0      0 127.0.0.1.54423    127.0.0.1.10028    CLOSE_WAIT
c4fa69e0 tcp4       0      0 127.0.0.1.10027    127.0.0.1.10072    CLOSED
c5020000 tcp4       0      0 127.0.0.1.63999    127.0.0.1.10028    CLOSE_WAIT
c501cc58 tcp4       0      0 127.0.0.1.10027    127.0.0.1.61360    CLOSED
c4fa64f0 tcp4       0      0 127.0.0.1.16582    127.0.0.1.10028    CLOSE_WAIT
c5020278 tcp4       0      0 127.0.0.1.10027    127.0.0.1.34477    CLOSED
c50204f0 tcp4       0      0 192.168.0.1.2049   192.168.0.16.991   ESTABLISHED
c501c278 tcp4       0      0 192.168.0.1.22     192.168.0.16.57414 ESTABLISHED
c4fa6c58 tcp4       0      0 127.0.0.1.10024    *.*                LISTEN
c4fa6278 tcp4       0      0 127.0.0.1.783      *.*                LISTEN
c4fa5278 tcp4       0      0 192.168.0.1.993    192.168.0.16.53799 ESTABLISHED
c4fa6768 tcp4       0      0 68.164.219.98.993  140.247.60.67.4019 ESTABLISHED
c4c82768 tcp4       0      0 *.8021             *.*                LISTEN
c4c829e0 tcp4       0      0 *.119              *.*                LISTEN
c4c82c58 tcp4       0      0 *.22               *.*                LISTEN
c49a3768 tcp6       0      0 *.22               *.*                LISTEN
c49a39e0 tcp4       0      0 *.*                *.*                CLOSED
c49a3c58 tcp46      0      0 *.443              *.*                LISTEN
c4c82000 tcp4       0      0 *.*                *.*                CLOSED
c4c82278 tcp46      0      0 *.80               *.*                LISTEN
c4c83000 tcp4       0      0 *.5999             *.*                LISTEN
c4c83278 tcp4       0      0 *.1281             *.*                LISTEN
c4c834f0 tcp6       0      0 *.1281             *.*                LISTEN
c49a3000 tcp4       0      0 192.168.0.1.548    *.*                LISTEN
c47cf000 tcp4       0      0 *.49152            *.*                LISTEN
c49a34f0 tcp4       0      0 127.0.0.1.4700     *.*                LISTEN
c49a3278 tcp4       0      0 127.0.0.1.10028    *.*                LISTEN
c49a29e0 tcp4       0      0 127.0.0.1.10025    *.*                LISTEN
c47cf768 tcp4       0      0 *.465              *.*                LISTEN
c47cf9e0 tcp4       0      0 *.587              *.*                LISTEN
c47cfc58 tcp4       0      0 *.25               *.*                LISTEN
c49a2278 tcp4       0      0 *.995              *.*                LISTEN
c49a24f0 tcp4       0      0 *.993              *.*                LISTEN
c49a2768 tcp4       0      0 127.0.0.1.10027    *.*                LISTEN
c47cec58 tcp4       0      0 192.168.0.1.631    192.168.0.16.16329 ESTABLISHED
c47ce4f0 tcp4       0      0 *.5432             *.*                LISTEN
c474b9e0 tcp6       0      0 *.5432             *.*                LISTEN
c474bc58 tcp4       0      0 127.0.0.1.10023    *.*                LISTEN
c47ce278 tcp4       0      0 *.21               *.*                LISTEN
c47ce768 tcp4       0      0 *.988              *.*                LISTEN
c47ce9e0 tcp6       0      0 *.905              *.*                LISTEN
c47cf278 tcp4       0      0 *.139              *.*                LISTEN
c47cf4f0 tcp4       0      0 *.445              *.*                LISTEN
c474a000 tcp6       0      0 *.139              *.*                LISTEN
c474a278 tcp6       0      0 *.445              *.*                LISTEN
c474a4f0 tcp4       0      0 *.631              *.*                LISTEN
c474a768 tcp6       0      0 *.631              *.*                LISTEN
c474a9e0 tcp4       0      0 192.168.0.1.427    *.*                LISTEN
c474ac58 tcp
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2009-12-14 23:26:42 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-fs

possibly nfs-related?
Comment 2 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 08:00:14 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped
Comment 3 Graham Perrin freebsd_committer freebsd_triage 2022-10-17 12:19:16 UTC
Keyword: 

    crash

– in lieu of summary line prefix: 

    [panic]

* bulk change for the keyword
* summary lines may be edited manually (not in bulk). 

Keyword descriptions and search interface: 

    <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>