Attachment 190824 Details for Bug 226064 – core text file

core text file

core.txt.9 (text/plain), 136.69 KB, created by Manish Kumar on 2018-02-20 12:08:18 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Manish Kumar

Created: 2018-02-20 12:08:18 UTC

Size: 136.69 KB

patch

obsolete

>munnar dumped core - see /var/crash/vmcore.9
>
>Tue Feb 20 09:54:14 IST 2018
>
>FreeBSD munnar 12.0-CURRENT FreeBSD 12.0-CURRENT #8: Fri Feb 16 14:58:28 IST 2018     root@munnar:/usr/obj/usr/src/amd64.amd64/sys/MYKERNEL  amd64
>
>panic: Memory modified after free 0xfffff80003ba8b80(32) val=0 @ 0xfffff80003ba8b80
>
>GNU gdb 6.1.1 [FreeBSD]
>Copyright 2004 Free Software Foundation, Inc.
>GDB is free software, covered by the GNU General Public License, and you are
>welcome to change it and/or distribute copies of it under certain conditions.
>Type "show copying" to see the conditions.
>There is absolutely no warranty for GDB.  Type "show warranty" for details.
>This GDB was configured as "amd64-marcel-freebsd"...
>
>Unread portion of the kernel message buffer:
>panic: Memory modified after free 0xfffff80003ba8b80(32) val=0 @ 0xfffff80003ba8b80
>
>cpuid = 5
>time = 1519077207
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe006210ee80
>kdb_backtrace() at kdb_backtrace+0x53/frame 0xfffffe006210ef50
>vpanic() at vpanic+0x268/frame 0xfffffe006210f020
>kproc_shutdown() at kproc_shutdown/frame 0xfffffe006210f080
>trash_ctor() at trash_ctor+0x6b/frame 0xfffffe006210f0c0
>uma_zalloc_arg() at uma_zalloc_arg+0x284/frame 0xfffffe006210f200
>bucket_alloc() at bucket_alloc+0x102/frame 0xfffffe006210f240
>uma_zfree_arg() at uma_zfree_arg+0x647/frame 0xfffffe006210f330
>uma_zfree() at uma_zfree+0x29/frame 0xfffffe006210f360
>ctl_free_beio() at ctl_free_beio+0x80/frame 0xfffffe006210f390
>ctl_complete_beio() at ctl_complete_beio+0x48/frame 0xfffffe006210f3b0
>ctl_be_block_move_done() at ctl_be_block_move_done+0x266/frame 0xfffffe006210f430
>cfiscsi_datamove_in() at cfiscsi_datamove_in+0x92d/frame 0xfffffe006210f510
>cfiscsi_datamove() at cfiscsi_datamove+0x28/frame 0xfffffe006210f530
>ctl_datamove() at ctl_datamove+0x498/frame 0xfffffe006210f7d0
>ctl_be_block_dispatch_zvol() at ctl_be_block_dispatch_zvol+0x6c2/frame 0xfffffe006210f8e0
>ctl_be_block_dispatch() at ctl_be_block_dispatch+0x626/frame 0xfffffe006210f9b0
>ctl_be_block_worker() at ctl_be_block_worker+0x68d/frame 0xfffffe006210fa40
>taskqueue_run_locked() at taskqueue_run_locked+0x246/frame 0xfffffe006210fab0
>taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe006210fb30
>fork_exit() at fork_exit+0x145/frame 0xfffffe006210fbb0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe006210fbb0
>--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>KDB: enter: panic
>
>Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/fdescfs.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/fdescfs.ko
>Reading symbols from /boot/kernel/uhid.ko...Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/uhid.ko
>Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/zfs.ko
>Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/opensolaris.ko
>Reading symbols from /boot/kernel/ctl.ko...Reading symbols from /usr/lib/debug//boot/kernel/ctl.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/ctl.ko
>Reading symbols from /boot/kernel/cfiscsi.ko...Reading symbols from /usr/lib/debug//boot/kernel/cfiscsi.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/cfiscsi.ko
>Reading symbols from /boot/kernel/iscsi.ko...Reading symbols from /usr/lib/debug//boot/kernel/iscsi.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/iscsi.ko
>Reading symbols from /boot/kernel/t4_tom.ko...Reading symbols from /usr/lib/debug//boot/kernel/t4_tom.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/t4_tom.ko
>Reading symbols from /boot/kernel/toecore.ko...Reading symbols from /usr/lib/debug//boot/kernel/toecore.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/toecore.ko
>Reading symbols from /boot/kernel/if_cxgbe.ko...Reading symbols from /usr/lib/debug//boot/kernel/if_cxgbe.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/if_cxgbe.ko
>Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/dtraceall.ko
>Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtrace.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/dtrace.ko
>Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/dtmalloc.ko
>Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtnfscl.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/dtnfscl.ko
>Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/lib/debug//boot/kernel/fbt.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/fbt.ko
>Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /usr/lib/debug//boot/kernel/fasttrap.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/fasttrap.ko
>Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /usr/lib/debug//boot/kernel/sdt.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/sdt.ko
>Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/systrace.ko
>Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/systrace_freebsd32.ko
>Reading symbols from /boot/kernel/profile.ko...Reading symbols from /usr/lib/debug//boot/kernel/profile.ko.debug...done.
>done.
>Loaded symbols for /boot/kernel/profile.ko
>#0  doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:346
>346		dumptid = curthread->td_tid;
>(kgdb) #0  doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:346
>#1  0xffffffff8048cd72 in db_dump (dummy=-2129994987, dummy2=false, dummy3=-1, 
>    dummy4=0xfffffe006210e930 "") at /usr/src/sys/ddb/db_command.c:574
>#2  0xffffffff8048cb5f in db_command (last_cmdp=0xffffffff82144080, 
>    cmd_table=0x0, dopager=1) at /usr/src/sys/ddb/db_command.c:481
>#3  0xffffffff8048c669 in db_command_loop ()
>    at /usr/src/sys/ddb/db_command.c:534
>#4  0xffffffff804915aa in db_trap (type=3, code=-61456)
>    at /usr/src/sys/ddb/db_main.c:250
>#5  0xffffffff810ae3e1 in kdb_trap (type=3, code=-61456, tf=0xfffffe006210ee60)
>    at /usr/src/sys/kern/subr_kdb.c:697
>#6  0xffffffff81728494 in trap (frame=0xfffffe006210ee60)
>    at /usr/src/sys/amd64/amd64/trap.c:547
>#7  0xffffffff817296ca in trap_check (frame=0xfffffe006210ee60)
>    at /usr/src/sys/amd64/amd64/trap.c:608
>#8  0xffffffff816f9c7c in calltrap ()
>    at /usr/src/sys/amd64/amd64/exception.S:232
>#9  0xffffffff810adb15 in breakpoint () at cpufunc.h:65
>#10 0xffffffff810ad71a in kdb_enter (why=0xffffffff81a9fc95 "panic", 
>    msg=0xffffffff81a9fc95 "panic") at /usr/src/sys/kern/subr_kdb.c:478
>#11 0xffffffff81041aba in vpanic (
>    fmt=0xffffffff81acda54 "Memory modified after free %p(%d) val=%x @ %p\n", 
>    ap=0xfffffe006210f060) at /usr/src/sys/kern/kern_shutdown.c:800
>#12 0xffffffff81041ba0 in panic (
>    fmt=0xffffffff81acda54 "Memory modified after free %p(%d) val=%x @ %p\n")
>    at /usr/src/sys/kern/kern_shutdown.c:738
>#13 0xffffffff8156515b in trash_ctor (mem=0xfffff80003ba8b80, size=32, 
>    arg=0x108, flags=1) at /usr/src/sys/vm/uma_dbg.c:82
>#14 0xffffffff8155c914 in uma_zalloc_arg (zone=0xfffff8022ffe4740, 
>    udata=0x108, flags=1) at /usr/src/sys/vm/uma_core.c:2240
>#15 0xffffffff8155e192 in bucket_alloc (zone=0xfffff800c2139ae0, udata=0x108, 
>    flags=1) at /usr/src/sys/vm/uma_core.c:409
>#16 0xffffffff8155dd87 in uma_zfree_arg (zone=0xfffff800c2139ae0, 
>    item=0xfffffe00ab636000, udata=0x0) at /usr/src/sys/vm/uma_core.c:2966
>#17 0xffffffff82f8e549 in uma_zfree (zone=0xfffff800c2139ae0, 
>    item=0xfffffe00ab636000) at uma.h:395
>#18 0xffffffff82f8e420 in ctl_free_beio (beio=0xfffff8018ae37cf0)
>    at /usr/src/sys/cam/ctl/ctl_backend_block.c:322
>#19 0xffffffff82f8e238 in ctl_complete_beio (beio=0xfffff8018ae37cf0)
>    at /usr/src/sys/cam/ctl/ctl_backend_block.c:349
>#20 0xffffffff82f8d616 in ctl_be_block_move_done (io=0xfffff80187c512a0)
>    at /usr/src/sys/cam/ctl/ctl_backend_block.c:452
>#21 0xffffffff82fc846d in cfiscsi_datamove_in (io=0xfffff80187c512a0)
>    at /usr/src/sys/cam/ctl/ctl_frontend_iscsi.c:2614
>#22 0xffffffff82fc7818 in cfiscsi_datamove (io=0xfffff80187c512a0)
>    at /usr/src/sys/cam/ctl/ctl_frontend_iscsi.c:2775
>#23 0xffffffff82f6bad8 in ctl_datamove (io=0xfffff80187c512a0)
>    at /usr/src/sys/cam/ctl/ctl.c:12503
>#24 0xffffffff82f90e12 in ctl_be_block_dispatch_zvol (
>    be_lun=0xfffff80005c9c400, beio=0xfffff8018ae37cf0)
>    at /usr/src/sys/cam/ctl/ctl_backend_block.c:954
>#25 0xffffffff82f95f56 in ctl_be_block_dispatch (be_lun=0xfffff80005c9c400, 
>    io=0xfffff80187c512a0) at /usr/src/sys/cam/ctl/ctl_backend_block.c:1645
>#26 0xffffffff82f9562d in ctl_be_block_worker (context=0xfffff80005c9c400, 
>    pending=1) at /usr/src/sys/cam/ctl/ctl_backend_block.c:1725
>#27 0xffffffff810ccd36 in taskqueue_run_locked (queue=0xfffff80018148a00)
>    at /usr/src/sys/kern/subr_taskqueue.c:465
>#28 0xffffffff810ce3e2 in taskqueue_thread_loop (arg=0xfffff80005c9c5d0)
>    at /usr/src/sys/kern/subr_taskqueue.c:757
>#29 0xffffffff80fd94c5 in fork_exit (
>    callout=0xffffffff810ce320 <taskqueue_thread_loop>, 
>    arg=0xfffff80005c9c5d0, frame=0xfffffe006210fbc0)
>    at /usr/src/sys/kern/kern_fork.c:1039
>#30 0xffffffff816fa99e in fork_trampoline ()
>    at /usr/src/sys/amd64/amd64/exception.S:843
>#31 0x0000000000000000 in ?? ()
>Current language:  auto; currently minimal
>(kgdb) 
>
>------------------------------------------------------------------------
>ps -axlww
>
>UID  PID PPID CPU PRI NI   VSZ  RSS MWCHAN   STAT TT       TIME COMMAND
>  0    0    0   0 -16  0     0    0 swapin   DLs   -    0:11.06 [kernel]
>  0    1    0   0  20  0 10032  700 wait     DLs   -    0:00.14 [init]
>  0    2    0   0 -16  0     0    0 crypto_w DL    -    0:00.00 [crypto]
>  0    3    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 0]
>  0    4    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 1]
>  0    5    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 2]
>  0    6    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 3]
>  0    7    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 4]
>  0    8    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 5]
>  0    9    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 6]
>  0   10    0   0 -16  0     0    0 audit_wo DL    -    0:00.00 [audit]
>  0   11    0   0 155  0     0    0 -        RL    -  496:10.34 [idle]
>  0   12    0   0 -52  0     0    0 -        WL    -    0:03.95 [intr]
>  0   13    0   0  -8  0     0    0 -        DL    -    0:00.01 [geom]
>  0   14    0   0 -16  0     0    0 crypto_r DL    -    0:00.00 [crypto returns 7]
>  0   15    0   0 -16  0     0    0 -        DL    -    0:00.14 [cam]
>  0   16    0   0 -68  0     0    0 -        DL    -    0:01.11 [usb]
>  0   17    0   0 -16  0     0    0 -        DL    -    0:00.00 [fdc0]
>  0   18    0   0 -16  0     0    0 -        DL    -    0:00.00 [soaiod1]
>  0   19    0   0 -16  0     0    0 -        DL    -    0:00.00 [soaiod2]
>  0   20    0   0 -16  0     0    0 -        DL    -    0:00.00 [soaiod3]
>  0   21    0   0 -16  0     0    0 -        DL    -    0:00.00 [soaiod4]
>  0   22    0   0 -16  0     0    0 waiting_ DL    -    0:00.00 [sctp_iterator]
>  0   23    0   0 -16  0     0    0 -        DL    -    0:00.24 [rand_harvestq]
>  0   24    0   0 -16  0     0    0 psleep   DL    -    0:00.10 [pagedaemon]
>  0   25    0   0 -16  0     0    0 psleep   DL    -    0:00.00 [vmdaemon]
>  0   26    0   0 -16  0     0    0 -        DL    -    0:00.02 [bufspacedaemon]
>  0   27    0   0 -16  0     0    0 psleep   DL    -    0:00.09 [bufdaemon]
>  0   28    0   0 -16  0     0    0 vlruwt   DL    -    0:00.02 [vnlru]
>  0   29    0   0  16  0     0    0 syncer   DL    -    0:00.18 [syncer]
>  0  125    1   0  52  0 10880    0 pause    DWs   -    0:00.00 [adjkerntz]
>  0  425    1   0  20  0 13920  812 select   Ds    -    0:00.06 [devd]
>  0  571    1   0  20  0 11476 2320 select   Ds    -    0:00.05 [syslogd]
>  0  696    1   0  20  0 18200 5804 select   Ds    -    0:00.01 [sshd]
>  0  699    1   0  20  0 15912 5048 select   Ds    -    0:00.05 [sendmail]
> 25  702    1   0  20  0 15760    0 pause    DWs   -    0:00.00 [sendmail]
>  0  706    1   0  20  0 11468  844 nanslp   Ds    -    0:00.03 [cron]
>  0  753    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  754    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  755    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  756    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  757    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  758    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  759    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  760    1   0  52  0 10928 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  761    1   0  52  0 10916 1936 ttyin    Ds+   -    0:00.01 [getty]
>  0  859    0   0  -8  0     0    0 arc_recl DL    -    0:00.04 [zfskern]
>  0  872    0   0 -16  0     0    0 -        RL    -    0:00.00 [ctl]
>  0 1137    0   0 -16  0     0    0 ftcl     DL    -    0:00.00 [ftcleanup]
>  0 1253    1   0  52  0 13200 2532 select   Ds    -    0:00.00 [ctld]
>  0 1979  706   0  20  0 11468 1680 piperd   D     -    0:00.00 [cron]
>  0 1981 1979   0  22  0 11756    0 wait     DWs   -    0:00.00 [sh]
>  0 1983 1981   0  22  0 10732    0 wait     DW    -    0:00.00 [lockf]
>  0 1984 1983   0  52  0 11756    0 wait     DW    -    0:00.00 [sh]
>  0 1993 1984   0  52  0 11804    0 wait     DW    -    0:00.00 [sh]
>  0 1994 1984   0  22  0 10936 2344 piperd   D     -    0:00.00 [mail]
>  0 2080 1993   0  52  0 11764    0 wait     DW    -    0:00.00 [sh]
>  0 2081 2080   0  52  0 11756    0 wait     DW    -    0:00.00 [sh]
>  0 2083 2081   0  52  0 10732    0 wait     DW    -    0:00.00 [lockf]
>  0 2084 2083   0  52  0 11756    0 wait     DW    -    0:00.00 [sh]
>  0 2092 2084   0  52  0 11804    0 wait     DW    -    0:00.00 [sh]
>  0 2093 2084   0  52  0 10936 2344 piperd   D     -    0:00.00 [mail]
>  0 2094 2092   0  52  0 11824    0 wait     DW    -    0:00.00 [sh]
>  0 2098 2094   0  22  0 14908 6336 -        R     -    0:00.00 [find]
>  0 2099 2094   0  52  0 11824    0 wait     DW    -    0:00.00 [sh]
>  0 2101 2099   0  52  0 11768 3196 piperd   D     -    0:00.00 [cat]
>
>------------------------------------------------------------------------
>vmstat -s
>
>432795458 cpu context switches
> 53755621 device interrupts
>  1098616 software interrupts
>   650512 traps
> 45880077 system calls
>       33 kernel threads created
>     1224  fork() calls
>      876 vfork() calls
>        0 rfork() calls
>      482 swap pager pageins
>     1496 swap pager pages paged in
>     1652 swap pager pageouts
>     6894 swap pager pages paged out
>     1160 vnode pager pageins
>     8415 vnode pager pages paged in
>        0 vnode pager pageouts
>        0 vnode pager pages paged out
>        0 page daemon wakeups
>        0 pages examined by the page daemon
>        0 clean page reclamation shortfalls
>        0 pages reactivated by the page daemon
>    82890 copy-on-write faults
>       81 copy-on-write optimized faults
>   197614 zero fill pages zeroed
>       24 zero fill pages prezeroed
>      577 intransit blocking page faults
>   299311 total VM faults taken
>     1486 page faults requiring I/O
>        0 pages affected by kernel thread creation
>    61525 pages affected by  fork()
>    31373 pages affected by vfork()
>        0 pages affected by rfork()
> 29782891 pages freed
>        0 pages freed by daemon
>        0 pages freed by exiting processes
>        0 pages active
>        0 pages inactive
>        0 pages in the laundry queue
>        0 pages wired down
>        0 pages free
>        0 bytes per page
>        0 total name lookups
>          cache hits (0% pos + 0% neg) system 0% per-directory
>          deletions 0%, falsehits 0%, toolong 0%
>
>------------------------------------------------------------------------
>vmstat -m
>
>         Type InUse MemUse HighUse Requests  Size(s)
>       feeder     7     1K       -        7  32
>CAM I/O Scheduler     1     1K       -        1  128
>    CAM queue    11     4K       -       51  16,32,512
>         UART     6     5K       -        6  16,1024
>          USB    41   153K       -       42  16,128,256,512,1024,4096,32768
>       USBdev    35     5K       -       35  32,64,128,256,512
>CAM dev queue     5     1K       -        5  64
>      ata_pci     2     1K       -        2  64
>        vtbuf    24  1968K       -       46  4096
>           vt    11     6K       -       11  512
>       DEVFS3   242    61K       -      265  256
>       DEVFS1   213   107K       -      223  512
>   DEVFS_RULE    55    26K       -       55  64,512
>        DEVFS    17     1K       -       18  16,128
>       DEVFSP     0     0K       -       13  64
>NFSD V4client     1     1K       -        1  256
> NFSD lckfile     1     1K       -        1  256
> NFSD session     1     1K       -        1  1024
>    pfs_nodes    20     5K       -       20  256
>         GEOM    94   337K       -      530  16,32,64,128,256,512,1024,2048,8192
>    raid_data     0     0K       -       78  32,128,256
>       isadev     4     1K       -        4  128
>       nvlist   112    11K       -      112  16,64,128
>  ddb_capture     1    64K       -        1  65536
>         cdev     3     1K       -        3  256
>     filedesc     1     4K       -      121  32,4096,16384
>        sigio     0     0K       -        1  64
>     filecaps     0     0K       -      367  16,32
>      kdtrace  2266   555K       -     6734  64,256
>         kenv   120    12K       -      124  16,32,64,128,8192
>       kqueue    66     5K       -     2140  64,256,512,2048
>    proc-args    95     5K       -     7600  16,32,64,128,256
>        hhook    13     4K       -       13  256
>      ithread   264    42K       -      264  32,128,256
>          KTR     1 14228K       -        1  
>       KTRACE   100    13K       -      100  128
>       linker   517  3021K       -      718  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
>        lockf    26     3K       -      462  64,128
>   loginclass     3     1K       -        3  64
>        cache     1     1K       -        1  32
>       devbuf  5755  8347K       -     6113  16,32,64,128,256,512,1024,2048,4096,8192,65536
>         temp    42     2K       -     4114  16,32,64,128,256,512,1024,2048,4096,65536
>       module   542    68K       -      546  128
>     mtx_pool     2    16K       -        2  8192
>          osd   384     7K       -      411  16,32,64,128,256
>     pmchooks     1     1K       -        1  128
>         pgrp    19     3K       -      307  128
>      session    19     3K       -      187  128
>         proc     2    32K       -        2  16384
>      subproc   202   332K       -     2270  512,4096
>         cred    39    10K       -     1844  256
>     acpiintr     1     1K       -        1  64
>       acpica  1648   162K       -    66398  16,32,64,128,256,512,1024
>       plimit    19     5K       -     2786  256
>      uidinfo     5     5K       -      101  128,4096
>       dumper     1     1K       -        1  512
>     acpitask     1    64K       -        1  65536
>       sysctl     0     0K       -        1  64
>    sysctloid 11816   605K       -    11958  16,32,64,128
>    sysctltmp     0     0K       -      235  16,64,256,1024
>       kbdmux     7    22K       -        7  16,512,1024,2048,16384
>      tidhash     1    32K       -        1  32768
>      callout     9  3208K       -        9  
>         umtx  4592   574K       -     4592  128
>     p1003.1b     1     1K       -        1  16
>         SWAP     1   276K       -        1  
>          bus  1454   124K       -    18685  16,32,64,128,256,1024
>       bus-sc    90   436K       -    15636  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
>      devstat    18    37K       -       18  32,4096
> eventhandler   140    12K       -      140  64,128
>   gtaskqueue    54    30K       -       54  16,32,256,8192
>         kobj   356  1424K       -      501  4096
>      Per-cpu     1     1K       -        1  32
>         rman   409    47K       -      778  16,32,128
>         sbuf     0     0K       -      816  16,32,64,128,256,512,1024,2048,4096,8192,32768
>       sglist     8     8K       -        8  1024
>    toponodes    30     4K       -       30  128
>    taskqueue   453    59K       -      573  16,32,64,128,256
>     terminal    11     3K       -       11  256
>       Unitno    27     2K       -     1734  32,64
>         vmem     3  1184K       -       21  1024,2048,4096,8192,16384,32768,65536
>      Witness  1539  3282K       -     1539  2048,16384
>     ioctlops     0     0K       -      583  256,512,1024
>       select    29     4K       -       29  128
>          iov     0     0K       - 44914533  64,128,256,512
>          msg     4    30K       -        4  2048,4096,8192,16384
>          sem     4   106K       -        4  2048,4096
>          shm     1    32K       -        1  32768
>          tty    14    14K       -       17  1024
>          pts     0     0K       -        3  256
>     mbuf_tag     0     0K       -        7  32
>        shmfd     1     8K       -        1  8192
>       soname     4     1K       -    12820  16,32,128
>          pcb    18  1175K       -      132  16,32,128,1024,2048,8192
>          acl     0     0K       -        3  4096
>     vfscache     4  2145K       -        4  256,32768,65536
>   cl_savebuf     0     0K       -       63  64
>     vfs_hash     1  1024K       -        1  
>       vnodes     1     1K       -        1  256
>        mount    31     2K       -      101  16,32,64,128,256
>       statfs     0     0K       -     7209  4096
>  vnodemarker     0     0K       -     2733  512
>chacha20random     1     2K       -        1  2048
>          BPF     7     1K       -        7  128
>        ifnet     8    15K       -        8  128,2048
>       ifaddr    83    19K       -       98  16,32,64,128,256,512,2048,4096
>  ether_multi    45     3K       -       45  16,32,64
>        clone     8     1K       -        8  128
>        ipsec     1     1K       -        1  256
>      lltable    33    10K       -       63  256,512
>        iflib    88   548K       -       96  32,64,128,1024,4096,8192,16384,32768
>     routetbl    31     4K       -       35  32,64,128,512
>         igmp     7     1K       -        7  128
>     in_multi     3     1K       -        3  256
>encap_export_host     2     2K       -        2  1024
>    sctp_a_it     0     0K       -        4  16
>     sctp_vrf     1     1K       -        1  64
>     sctp_ifa     6     1K       -        6  128
>     sctp_ifn     3     1K       -        3  128
>    sctp_iter     0     0K       -        4  256
>    hostcache     1    32K       -        1  32768
>          LRO    32   337K       -       32  16,2048,8192,32768
>      tcpfunc     1     1K       -        1  64
>     syncache     1    68K       -        1  
>    in6_multi    25     3K       -       25  32,256
>          mld     7     1K       -        7  128
>       ip6ndp     9     1K       -       10  64,256
>  inpcbpolicy   106     4K       -      369  32
>     secasvar     1     1K       -        1  1024
>       sahead     1     1K       -        1  1024
>  ipsecpolicy     2     2K       -        2  256,1024
>    ipsec-saq     2     2K       -        2  1024
>       crypto     2     2K       -        2  512,1024
>          rpc     2     8K       -        2  4096
>audit_evclass   230     8K       -      286  32
>      pagedep     1   256K       -       93  256
>     inodedep    11  1029K       -      762  512
>    bmsafemap     1     8K       -      330  256,8192
>       newblk     1  2048K       -    20707  256
>     indirdep     0     0K       -       68  128,32768
>     freefrag     0     0K       -      188  128
>     freeblks     0     0K       -      207  256
>     freefile     1     1K       -      113  64
>       diradd     0     0K       -      482  128
>        mkdir     0     0K       -       14  128
>       dirrem     0     0K       -      455  128
>    newdirblk     0     0K       -        7  64
>     freework     1     1K       -      224  32,128
>      freedep     0     0K       -        1  64
>      jaddref     0     0K       -      496  128
>      jremref     0     0K       -      467  128
>      jnewblk     0     0K       -    20706  128
>    jfreefrag     0     0K       -      188  128
>         jseg     1     1K       -     4594  128
>      jsegdep     8     1K       -    21857  64
>        sbdep     0     0K       -      302  64
>     savedino     0     0K       -      111  256
>      jblocks     2     1K       -        2  128,256
>      softdep     1     1K       -        1  512
>  ufs_dirhash   726   180K       -     1042  16,32,64,128,256,512,1024
>    ufs_quota     1  1024K       -        1  
>    ufs_mount     3    17K       -        4  512,4096,8192
>    vm_pgdata     1     1K       -        1  128
>      UMAHash   122 19188K       -      461  512,1024,2048,4096,8192,16384,32768,65536
>      memdesc     1     4K       -        1  4096
>     pci_link    16     2K       -       16  32,64,128
>     atkbddev     2     1K       -        2  64
>      acpisem    41     6K       -       41  128
>      acpidev    36     3K       -       36  64
>      CAM SIM     5     2K       -        5  256
>       sr_iov     4     2K       -        4  512
>       apmdev     1     1K       -        1  128
>   madt_table     0     0K       -        2  64,4096
>      CAM XPT    25     2K       -      107  16,32,64,128,256,512,1024,2048,65536
>      entropy     0     0K       -       52  4096
>      CAM DEV     6    12K       -       19  2048
>      CAM CCB     0     0K       -  4324034  2048
>     CAM path     7     1K       -       70  32
>      io_apic     2     4K       -        2  2048
>   local_apic     1     8K       -        1  8192
>   CAM periph     4     1K       -       26  16,32,64,128,256
>          MCA     8     1K       -        8  128
>         cpus     2     1K       -        2  32
>          msi    54     7K       -       54  128
>     nexusdev     5     1K       -        5  16
>  fdesc_mount     1     1K       -        1  16
>      solaris 2045562 241633K       - 267431431  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
>   kstat_data     8     1K       -        8  64
>       ctlmem  3609  8302K       -    13486  16,32,64,128,256,512,4096,8192,65536
>       ctlblk   200   104K       -      208  32,1024
>        iSCSI     1     1K       -        1  128
>          icl     3     1K       -        5  16,64
>     icl_soft   100    25K       -      100  256
>      cfiscsi   200   200K       -      208  1024
>        cxgbe    66  1755K       -    17760  16,64,128,256,512,1024,2048,4096,8192,16384
>          fbt 119538 15199K       -   119693  128
>          SDT    42     2K       -       42  16,64
>
>------------------------------------------------------------------------
>vmstat -z
>
>ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
>
>UMA Kegs:               280,      0,     443,      12,     448,   0,   0
>UMA Zones:              904,      0,     460,       0,     465,   0,   0
>UMA Slabs:              112,      0, 1207250,   82850,14201824,   0,   0
>UMA Hash:               256,      0,      85,      20,     211,   0,   0
>4 Bucket:                32,      0,    2041,    4655, 2313833,   0,   0
>6 Bucket:                48,      0,     352,    3383,  322237,   0,   0
>8 Bucket:                64,      0,     252,    2910,  190961,  11,   0
>12 Bucket:               96,      0,      60,    1498,   76652,   0,   0
>16 Bucket:              128,      0,     212,    1493,   25922,   0,   0
>32 Bucket:              256,      0,     508,     482,  234762, 381,   0
>64 Bucket:              512,      0,     421,     657,  253762,28255,   0
>128 Bucket:            1024,      0,     331,     257,  193938,   0,   0
>256 Bucket:            2048,      0,    2859,     355,  983628,98374,   0
>vmem:                  1792,      0,       2,      10,       3,   0,   0
>vmem btag:               56,      0,  159302,   22742, 1176802,1314,   0
>VM OBJECT:              256,      0,   10078,     182,   39099,   0,   0
>RADIX NODE:             144,      0,   48411,   11610,  849059,   0,   0
>MAP:                    232,      0,       3,      65,       3,   0,   0
>KMAP ENTRY:             120,      0,      25,     734,      28,   0,   0
>MAP ENTRY:              120,      0,     940,    1898,   95760,   0,   0
>VMSPACE:               2512,      0,      34,      32,    2102,   0,   0
>fakepg:                 104,      0,       0,       0,       0,   0,   0
>mt_zone:              16400,      0,     428,       0,     428,   0,   0
>16:                      16,      0,      33,    1710,   27949,   0,   0
>16:                      16,      0,       7,    1238,      15,   0,   0
>16:                      16,      0, 1033513,  137534,50410986,   0,   0
>16:                      16,      0,     316,    1925,   12805,   0,   0
>16:                      16,      0,    5362,    1112,    7681,   0,   0
>16:                      16,      0,     269,    1972,     490,   0,   0
>16:                      16,      0,     132,    2109,    1525,   0,   0
>16:                      16,      0,      39,    2202,     112,   0,   0
>32:                      32,      0,      57,    1803,    7681,   0,   0
>32:                      32,      0,     437,    1671,    3008,   0,   0
>32:                      32,      0,  213045,   56779,48833930,   0,   0
>32:                      32,      0,     160,    1948,    5128,   0,   0
>32:                      32,      0,    6208,    1232,    8674,   0,   0
>32:                      32,      0,     381,    1727,     494,   0,   0
>32:                      32,      0,     170,    1938,     573,   0,   0
>32:                      32,      0,       0,     868,      35,   0,   0
>64:                      64,      0,     665,    1629,    8104,   0,   0
>64:                      64,      0,     125,    1921,    8969,   0,   0
>64:                      64,      0,   84826,    5074,43902104,   0,   0
>64:                      64,      0,     705,    1837,    1147,   0,   0
>64:                      64,      0,    1086,    3254,   26677,   0,   0
>64:                      64,      0,     158,    1888,    2342,   0,   0
>64:                      64,      0,    2038,    2054,44914727,   0,   0
>64:                      64,      0,      25,    2021,     169,   0,   0
>128:                    128,      0,     956,     811,   36988,   0,   0
>128:                    128,      0,     104,     919,     477,   0,   0
>128:                    128,      0,  124913,    4853,30046394,   0,   0
>128:                    128,      0,     369,     902,    1210,   0,   0
>128:                    128,      0,  123378,     622,  125887,   0,   0
>128:                    128,      0,    4823,     571,    4863,   0,   0
>128:                    128,      0,    1467,     951,   14202,   0,   0
>128:                    128,      0,      63,     960,     449,   0,   0
>256:                    256,      0,      19,     476,    1782,   0,   0
>256:                    256,      0,    2338,     287,    7686,   0,   0
>256:                    256,      0,  503847,  177408,63382647,   0,   0
>256:                    256,      0,     197,     478,    2896,   0,   0
>256:                    256,      0,      12,     423,    1155,   0,   0
>256:                    256,      0,     298,     437,     672,   0,   0
>256:                    256,      0,     430,     305,    2574,   0,   0
>256:                    256,      0,      40,    1715,   20858,   0,   0
>512:                    512,      0,      15,     132,      22,   0,   0
>512:                    512,      0,      30,     264,    1085,   0,   0
>512:                    512,      0,   84083,   10130,17829301,   0,   0
>512:                    512,      0,       4,     115,    4109,   0,   0
>512:                    512,      0,       2,     201,     335,   0,   0
>512:                    512,      0,     329,     112,     344,   0,   0
>512:                    512,      0,     327,     170,    3076,   0,   0
>512:                    512,      0,       5,      86,     154,   0,   0
>1024:                  1024,      0,       7,      85,    2026,   0,   0
>1024:                  1024,      0,      19,     125,     358,   0,   0
>1024:                  1024,      0,    1829,     139,12893546,   0,   0
>1024:                  1024,      0,     219,      57,    1740,   0,   0
>1024:                  1024,      0,     109,      87,    1942,   0,   0
>1024:                  1024,      0,      10,      42,      12,   0,   0
>1024:                  1024,      0,      58,      62,      63,   0,   0
>1024:                  1024,      0,       4,      48,      23,   0,   0
>2048:                  2048,      0,       9,      11,      13,   0,   0
>2048:                  2048,      0,      48,      22,     115,   0,   0
>2048:                  2048,      0,     205,      51,    1905,   0,   0
>2048:                  2048,      0,       3,      67, 4324561,   0,   0
>2048:                  2048,      0,       8,      22,      43,   0,   0
>2048:                  2048,      0,    1538,      12,    1539,   0,   0
>2048:                  2048,      0,     359,      21,     377,   0,   0
>2048:                  2048,      0,       9,      21,      22,   0,   0
>4096:                  4096,      0,       0,       1,       2,   0,   0
>4096:                  4096,      0,      59,      12,    7277,   0,   0
>4096:                  4096,      0,     200,       9,    3999,   0,   0
>4096:                  4096,      0,     360,       4,    1619,   0,   0
>4096:                  4096,      0,     221,       0,     415,   0,   0
>4096:                  4096,      0,       3,       0,       3,   0,   0
>4096:                  4096,      0,      28,       1,      30,   0,   0
>4096:                  4096,      0,       1,      10,      54,   0,   0
>8192:                  8192,      0,       5,       0,       5,   0,   0
>8192:                  8192,      0,       2,       2,       9,   0,   0
>8192:                  8192,      0,       5,      17,  157463,   0,   0
>8192:                  8192,      0,       1,       3,    1359,   0,   0
>8192:                  8192,      0,     100,       0,     102,   0,   0
>8192:                  8192,      0,       6,       0,       6,   0,   0
>8192:                  8192,      0,      39,       4,      43,   0,   0
>8192:                  8192,      0,      52,       1,      63,   0,   0
>16384:                16384,      0,      30,       0,      30,   0,   0
>16384:                16384,      0,       0,       2,       7,   0,   0
>16384:                16384,      0,       2,       4,       9,   0,   0
>16384:                16384,      0,       1,       3,    2085,   0,   0
>16384:                16384,      0,       1,       0,       1,   0,   0
>16384:                16384,      0,       1,       0,       1,   0,   0
>16384:                16384,      0,      10,      13,      71,   0,   0
>16384:                16384,      0,       1,       0,       1,   0,   0
>32768:                32768,      0,       0,       0,       0,   0,   0
>32768:                32768,      0,       5,       6,      66,   0,   0
>32768:                32768,      0,      14,       8,      50,   0,   0
>32768:                32768,      0,       0,       5,    1563,   0,   0
>32768:                32768,      0,       0,       0,       0,   0,   0
>32768:                32768,      0,       1,       0,       1,   0,   0
>32768:                32768,      0,      17,       3,      20,   0,   0
>32768:                32768,      0,       8,       0,       8,   0,   0
>65536:                65536,      0,       0,       1,       8,   0,   0
>65536:                65536,      0,       2,       1,       5,   0,   0
>65536:                65536,      0,       0,       1,       2,   0,   0
>65536:                65536,      0,       6,       3,    1068,   0,   0
>65536:                65536,      0,     104,       7,     224,   0,   0
>65536:                65536,      0,       0,       0,       0,   0,   0
>65536:                65536,      0,      11,       1,      14,   0,   0
>65536:                65536,      0,       0,       0,       0,   0,   0
>SLEEPQUEUE:              88,      0,    2297,     586,    2297,   0,   0
>64 pcpu:                  8,      0,    3257,    1095,    3263,   0,   0
>Files:                   80,      0,      85,     944,   35224,   0,   0
>filedesc0:             1104,      0,      66,      96,    2134,   0,   0
>rl_entry:                40,      0,      88,    1991,      88,   0,   0
>TURNSTILE:              136,      0,    2297,     343,    2297,   0,   0
>umtx pi:                 96,      0,       0,       0,       0,   0,   0
>umtx_shm:                88,      0,       0,       0,       0,   0,   0
>MAC labels:              40,      0,       0,       0,       0,   0,   0
>PROC:                  1336,      0,      65,      70,    2133,   0,   0
>THREAD:                1400,      0,    2257,      39,    2589,   0,   0
>cpuset:                 104,      0,    2167,     607,    2516,   0,   0
>domainset:               32,      0,       0,       0,       0,   0,   0
>audit_record:          1280,      0,       0,       0,       0,   0,   0
>mbuf_packet:            256, 3246930,       1,    4313,     540,   0,   0
>mbuf:                   256, 3246930,    4590,    5011,176880457,   0,   0
>mbuf_cluster:          2048, 507330,    6118,      74,   70330,   0,   0
>mbuf_jumbo_page:       4096, 253665,    8608,    2499,73852940,   0,   0
>mbuf_jumbo_9k:         9216, 225480,       0,       0,       0,   0,   0
>mbuf_jumbo_16k:       16384, 169108,    2032,     152,    4829,   0,   0
>g_bio:                  384,      0,      38,     762,17886150,   0,   0
>ttyinq:                 160,      0,     135,     537,     465,   0,   0
>ttyoutq:                256,      0,      72,     483,     245,   0,   0
>DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0
>nvme_request:           128,      0,       0,       0,       0,   0,   0
>cryptop:                128,      0,       0,       0,       0,   0,   0
>cryptodesc:              72,      0,       0,       0,       0,   0,   0
>vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
>FPU_save_area:          576,      0,       0,       0,       0,   0,   0
>VNODE:                  472,      0,  148395,      85,  148515,   0,   0
>VNODEPOLL:              120,      0,       1,     131,       1,   0,   0
>BUF TRIE:               144,      0,    2109,   50730,   17971,   0,   0
>NAMEI:                 1024,      0,       1,     103,  211299,   0,   0
>rentr:                   24,      0,       0,       0,       0,   0,   0
>S VFS Cache:            108,      0,  144126,     389,  148469,   0,   0
>STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
>L VFS Cache:            328,      0,    4586,      94,    4586,   0,   0
>LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
>DIRHASH:               1024,      0,    1192,      56,    1910,   0,   0
>NCLNODE:                592,      0,       0,       0,       0,   0,   0
>AIO:                    208,      0,       0,       0,       0,   0,   0
>AIOP:                    32,      0,       0,       0,       0,   0,   0
>AIOCB:                  752,      0,       0,       0,       0,   0,   0
>AIOLIO:                 280,      0,       0,       0,       0,   0,   0
>pipe:                   760,      0,       6,      99,     892,   0,   0
>Mountpoints:           2744,      0,       3,       3,       3,   0,   0
>procdesc:               136,      0,       0,       0,       0,   0,   0
>ksiginfo:               112,      0,     102,    1403,    1150,   0,   0
>itimer:                 352,      0,       0,       0,       0,   0,   0
>KNOTE:                  160,      0,       0,     432,      44,   0,   0
>socket:                 856, 260796,     114,      82,    2523,   0,   0
>IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0
>unpcb:                  232, 260797,       8,     485,    2147,   0,   0
>ipq:                     56,  15904,       0,       0,       0,   0,   0
>udp_inpcb:              424, 260802,       2,     232,     247,   0,   0
>udpcb:                   32, 260896,       2,    2106,     247,   0,   0
>tcp_inpcb:              424, 260802,     104,     130,     122,   0,   0
>tcpcb:                  872, 260796,     104,      68,     122,   0,   0
>tcptw:                   88,  27810,       0,     405,       4,   0,   0
>syncache:               168,  15364,       0,     506,     107,   0,   0
>hostcache:               96,  15375,       0,     533,       5,   0,   0
>sackhole:                32,      0,       0,     620,       3,   0,   0
>tcpreass:                40,  31779,       0,     297,       2,   0,   0
>sctp_ep:               1216, 260796,       0,       0,       0,   0,   0
>sctp_asoc:             2408,  40000,       0,       0,       0,   0,   0
>sctp_laddr:              48,  80012,       0,     830,       5,   0,   0
>sctp_raddr:             736,  80000,       0,       0,       0,   0,   0
>sctp_chunk:             152, 400010,       0,       0,       0,   0,   0
>sctp_readq:             152, 400010,       0,       0,       0,   0,   0
>sctp_stream_msg_out:    112, 400015,       0,       0,       0,   0,   0
>sctp_asconf:             40, 400059,       0,       0,       0,   0,   0
>sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
>udplite_inpcb:          424, 260802,       0,       0,       0,   0,   0
>ripcb:                  424, 260802,       0,       0,       0,   0,   0
>rtentry:                208,      0,      15,     422,      17,   0,   0
>selfd:                   64,      0,      42,    1756,   36817,   0,   0
>swpctrie:               144, 1014687,     179,     334,     247,   0,   0
>swblk:                  136, 1014681,     895,     352,    1147,   0,   0
>FFS inode:              160,      0,  148252,     212,  148365,   0,   0
>FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
>FFS2 dinode:            256,      0,  148251,     204,  148364,   0,   0
>taskq_zone:              48,      0,       0,    2656,  296782,   0,   0
>reference_cache:         40,      0,     206,    2665,22861594,   0,   0
>reference_history_cache:      8,      0,       6,    3480,22861394,   0,   0
>range_seg_cache:         64,      0,   23743,  113959, 6807087,   0,   0
>metaslab_alloc_trace_cache:     64,      0,     100,    2814,12786214,   0,   0
>zio_cache:             1040,      0,   78782,    4981,88683224,   0,   0
>zio_link_cache:          48,      0,   78901,    5261,70740133,   0,   0
>zio_buf_512:            512,      0,     523,     149,  146958,   0,   0
>zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
>zio_buf_1024:          1024,      0,       0,      52,       7,   0,   0
>zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
>zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
>zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
>zio_buf_2048:          2048,      0,     101,      47,   33252,   0,   0
>zio_data_buf_2048:     2048,      0,       0,      74,     112,   0,   0
>zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
>zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
>zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
>zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
>zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
>zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
>zio_buf_4096:          4096,      0,       0,      65,   59646,   0,   0
>zio_data_buf_4096:     4096,      0,       0,      15,   67606,   0,   0
>zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
>zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
>zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
>zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
>zio_buf_7168:          7168,      0,       0,       2,       2,   0,   0
>zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
>zio_buf_8192:          8192,      0,       4,      14,  188935,   0,   0
>zio_data_buf_8192:     8192,      0,  101645,    3153,13577591,   0,   0
>zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
>zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
>zio_buf_12288:        12288,      0,       0,      11,   11091,   0,   0
>zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
>zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
>zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
>zio_buf_16384:        16384,      0,     125,      26,  133834,   0,   0
>zio_data_buf_16384:   16384,      0,       0,       2,       3,   0,   0
>zio_buf_20480:        20480,      0,       0,      11,    6941,   0,   0
>zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
>zio_buf_24576:        24576,      0,       0,      12,   69704,   0,   0
>zio_data_buf_24576:   24576,      0,       0,       3,       3,   0,   0
>zio_buf_28672:        28672,      0,       0,      13,    4626,   0,   0
>zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
>zio_buf_32768:        32768,      0,       0,      12,   56861,   0,   0
>zio_data_buf_32768:   32768,      0,       0,       3,       3,   0,   0
>zio_buf_40960:        40960,      0,       0,      12,   53025,   0,   0
>zio_data_buf_40960:   40960,      0,       0,       4,       6,   0,   0
>zio_buf_49152:        49152,      0,       1,      12,   49967,   0,   0
>zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
>zio_buf_57344:        57344,      0,       1,       9,   44222,   0,   0
>zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
>zio_buf_65536:        65536,      0,       0,      12,   43623,   0,   0
>zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
>zio_buf_81920:        81920,      0,       1,      11,   78182,   0,   0
>zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
>zio_buf_98304:        98304,      0,       3,      10,   71898,   0,   0
>zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
>zio_buf_114688:      114688,      0,       1,      12,   68527,   0,   0
>zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
>zio_buf_131072:      131072,      0,    1185,      62, 2063870,   0,   0
>zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
>zio_buf_163840:      163840,      0,       0,       0,       0,   0,   0
>zio_data_buf_163840: 163840,      0,       0,       0,       0,   0,   0
>zio_buf_196608:      196608,      0,       0,       0,       0,   0,   0
>zio_data_buf_196608: 196608,      0,       0,       0,       0,   0,   0
>zio_buf_229376:      229376,      0,       0,       0,       0,   0,   0
>zio_data_buf_229376: 229376,      0,       0,       0,       0,   0,   0
>zio_buf_262144:      262144,      0,       0,       0,       0,   0,   0
>zio_data_buf_262144: 262144,      0,       0,       0,       0,   0,   0
>zio_buf_327680:      327680,      0,       0,       0,       0,   0,   0
>zio_data_buf_327680: 327680,      0,       0,       0,       0,   0,   0
>zio_buf_393216:      393216,      0,       0,       0,       0,   0,   0
>zio_data_buf_393216: 393216,      0,       0,       0,       0,   0,   0
>zio_buf_458752:      458752,      0,       0,       0,       0,   0,   0
>zio_data_buf_458752: 458752,      0,       0,       0,       0,   0,   0
>zio_buf_524288:      524288,      0,       0,       0,       0,   0,   0
>zio_data_buf_524288: 524288,      0,       0,       0,       0,   0,   0
>zio_buf_655360:      655360,      0,       0,       0,       0,   0,   0
>zio_data_buf_655360: 655360,      0,       0,       0,       0,   0,   0
>zio_buf_786432:      786432,      0,       0,       0,       0,   0,   0
>zio_data_buf_786432: 786432,      0,       0,       0,       0,   0,   0
>zio_buf_917504:      917504,      0,       0,       0,       0,   0,   0
>zio_data_buf_917504: 917504,      0,       0,       0,       0,   0,   0
>zio_buf_1048576:     1048576,      0,       0,       0,       0,   0,   0
>zio_data_buf_1048576: 1048576,      0,       0,       0,       0,   0,   0
>zio_buf_1310720:     1310720,      0,       0,       0,       0,   0,   0
>zio_data_buf_1310720: 1310720,      0,       0,       0,       0,   0,   0
>zio_buf_1572864:     1572864,      0,       0,       0,       0,   0,   0
>zio_data_buf_1572864: 1572864,      0,       0,       0,       0,   0,   0
>zio_buf_1835008:     1835008,      0,       0,       0,       0,   0,   0
>zio_data_buf_1835008: 1835008,      0,       0,       0,       0,   0,   0
>zio_buf_2097152:     2097152,      0,       0,       0,       0,   0,   0
>zio_data_buf_2097152: 2097152,      0,       0,       0,       0,   0,   0
>zio_buf_2621440:     2621440,      0,       0,       0,       0,   0,   0
>zio_data_buf_2621440: 2621440,      0,       0,       0,       0,   0,   0
>zio_buf_3145728:     3145728,      0,       0,       0,       0,   0,   0
>zio_data_buf_3145728: 3145728,      0,       0,       0,       0,   0,   0
>zio_buf_3670016:     3670016,      0,       0,       0,       0,   0,   0
>zio_data_buf_3670016: 3670016,      0,       0,       0,       0,   0,   0
>zio_buf_4194304:     4194304,      0,       0,       0,       0,   0,   0
>zio_data_buf_4194304: 4194304,      0,       0,       0,       0,   0,   0
>zio_buf_5242880:     5242880,      0,       0,       0,       0,   0,   0
>zio_data_buf_5242880: 5242880,      0,       0,       0,       0,   0,   0
>zio_buf_6291456:     6291456,      0,       0,       0,       0,   0,   0
>zio_data_buf_6291456: 6291456,      0,       0,       0,       0,   0,   0
>zio_buf_7340032:     7340032,      0,       0,       0,       0,   0,   0
>zio_data_buf_7340032: 7340032,      0,       0,       0,       0,   0,   0
>zio_buf_8388608:     8388608,      0,       0,       0,       0,   0,   0
>zio_data_buf_8388608: 8388608,      0,       0,       0,       0,   0,   0
>zio_buf_10485760:    10485760,      0,       0,       0,       0,   0,   0
>zio_data_buf_10485760: 10485760,      0,       0,       0,       0,   0,   0
>zio_buf_12582912:    12582912,      0,       0,       0,       0,   0,   0
>zio_data_buf_12582912: 12582912,      0,       0,       0,       0,   0,   0
>zio_buf_14680064:    14680064,      0,       0,       0,       0,   0,   0
>zio_data_buf_14680064: 14680064,      0,       0,       0,       0,   0,   0
>zio_buf_16777216:    16777216,      0,       0,       0,       0,   0,   0
>zio_data_buf_16777216: 16777216,      0,       0,       0,       0,   0,   0
>lz4_ctx:              16384,      0,       0,      14,  349389,   0,   0
>abd_chunk:             4096,      0,  805322,  271464,42895383,   0,   0
>sa_cache:               152,      0,       0,       0,       0,   0,   0
>dnode_t:                952,      0,    1250,      98,    2821,   0,   0
>arc_buf_hdr_t_full:     344,      0,  727061,  117057,10272586,   0,   0
>arc_buf_hdr_t_l2only:     88,      0,       0,       0,       0,   0,   0
>arc_buf_t:               64,      0,  102849,   53267,13654973,   0,   0
>dmu_buf_impl_t:         344,      0,   95433,   14314, 9647491,   0,   0
>zil_lwb_cache:          320,      0,     641,    8863, 1349166,   0,   0
>zil_zcw_cache:           80,      0,      46,    1326,  747269,   0,   0
>zfs_znode_cache:        264,      0,       0,       0,       0,   0,   0
>CTL IO:                 672,      0,     116,     784,15894509,   0,   0
>beio:                   368,      0,     111,     509,15885991,   0,   0
>icl_pdu:                 80,      0,     116,    2481,34735588,   0,   0
>iscsi_outstanding:       48,      0,       0,       0,       0,   0,   0
>cfiscsi_data_wait:       72,      0,       0,    1265, 6621541,   0,   0
>dtrace_state_cache:   16384,      0,       0,       3,       7,   0,   0
>taskq_zone:              48,      0,       0,       0,       0,   0,   0
>cblk0:               131072,      0,       0,      49,  157637,   0,   0
>cblk1:               131072,      0,       0,      12,  159298,   0,   0
>cblk2:               131072,      0,       2,      11,  159265,   0,   0
>cblk3:               131072,      0,       2,       9,  159279,   0,   0
>cblk4:               131072,      0,       1,      10,  158983,   0,   0
>cblk5:               131072,      0,       0,      27,  157604,   0,   0
>cblk6:               131072,      0,       0,      39,  157539,   0,   0
>cblk7:               131072,      0,       1,      11,  166138,   0,   0
>cblk8:               131072,      0,       2,       8,  159463,   0,   0
>cblk9:               131072,      0,       2,       9,  159316,   0,   0
>cblk10:              131072,      0,       1,      11,  159661,   0,   0
>cblk11:              131072,      0,       2,      10,  159458,   0,   0
>cblk12:              131072,      0,       0,      46,  157175,   0,   0
>cblk13:              131072,      0,       1,      10,  159309,   0,   0
>cblk14:              131072,      0,       2,      11,  159145,   0,   0
>cblk15:              131072,      0,       0,      32,  167576,   0,   0
>cblk16:              131072,      0,       2,      14,  165405,   0,   0
>cblk17:              131072,      0,       2,      10,  158839,   0,   0
>cblk18:              131072,      0,       2,      14,  162199,   0,   0
>cblk19:              131072,      0,       2,       9,  158496,   0,   0
>cblk20:              131072,      0,       0,      34,  167728,   0,   0
>cblk21:              131072,      0,       0,      26,  157214,   0,   0
>cblk22:              131072,      0,       1,      14,  160536,   0,   0
>cblk23:              131072,      0,       2,      13,  162388,   0,   0
>cblk24:              131072,      0,       1,      11,  162512,   0,   0
>cblk25:              131072,      0,       1,      11,  159348,   0,   0
>cblk26:              131072,      0,       2,      10,  158595,   0,   0
>cblk27:              131072,      0,       1,      10,  159268,   0,   0
>cblk28:              131072,      0,       0,      36,  157205,   0,   0
>cblk29:              131072,      0,       0,      16,  159585,   0,   0
>cblk30:              131072,      0,       1,      11,  158915,   0,   0
>cblk31:              131072,      0,       2,       9,  158397,   0,   0
>cblk32:              131072,      0,       0,      21,  157126,   0,   0
>cblk33:              131072,      0,       0,      16,  164435,   0,   0
>cblk34:              131072,      0,       1,      10,  159213,   0,   0
>cblk35:              131072,      0,       2,       8,  158768,   0,   0
>cblk36:              131072,      0,       1,      10,  159256,   0,   0
>cblk37:              131072,      0,       1,      11,  165761,   0,   0
>cblk38:              131072,      0,       0,      26,  167712,   0,   0
>cblk39:              131072,      0,       2,      10,  159270,   0,   0
>cblk40:              131072,      0,       2,       9,  158913,   0,   0
>cblk41:              131072,      0,       0,      31,  167693,   0,   0
>cblk42:              131072,      0,       1,      15,  162378,   0,   0
>cblk43:              131072,      0,       2,      12,  159233,   0,   0
>cblk44:              131072,      0,       0,      32,  167757,   0,   0
>cblk45:              131072,      0,       2,       8,  159066,   0,   0
>cblk46:              131072,      0,       1,      11,  160342,   0,   0
>cblk47:              131072,      0,       2,      13,  161711,   0,   0
>cblk48:              131072,      0,       0,      50,  157143,   0,   0
>cblk49:              131072,      0,       2,      11,  162756,   0,   0
>cblk50:              131072,      0,       2,      25,  162274,   0,   0
>cblk51:              131072,      0,       1,      15,  162381,   0,   0
>cblk52:              131072,      0,       0,      15,  160311,   0,   0
>cblk53:              131072,      0,       2,      10,  158434,   0,   0
>cblk54:              131072,      0,       2,      10,  159230,   0,   0
>cblk55:              131072,      0,       2,      10,  159312,   0,   0
>cblk56:              131072,      0,       2,      10,  158906,   0,   0
>cblk57:              131072,      0,       2,      10,  159196,   0,   0
>cblk58:              131072,      0,       0,      24,  157183,   0,   0
>cblk59:              131072,      0,       2,       9,  159171,   0,   0
>cblk60:              131072,      0,       0,       9,  157181,   0,   0
>cblk61:              131072,      0,       1,      10,  158836,   0,   0
>cblk62:              131072,      0,       0,      29,  157182,   0,   0
>cblk63:              131072,      0,       1,      15,  162359,   0,   0
>cblk64:              131072,      0,       2,      11,  159276,   0,   0
>cblk65:              131072,      0,       0,      19,  167554,   0,   0
>cblk66:              131072,      0,       1,      11,  158584,   0,   0
>cblk67:              131072,      0,       1,      13,  159437,   0,   0
>cblk68:              131072,      0,       0,      13,  162409,   0,   0
>cblk69:              131072,      0,       1,      11,  159242,   0,   0
>cblk70:              131072,      0,       1,       9,  158524,   0,   0
>cblk71:              131072,      0,       0,      15,  162211,   0,   0
>cblk72:              131072,      0,       0,      59,  167707,   0,   0
>cblk73:              131072,      0,       2,      14,  162327,   0,   0
>cblk74:              131072,      0,       1,      10,  158734,   0,   0
>cblk75:              131072,      0,       1,      11,  158548,   0,   0
>cblk76:              131072,      0,       0,      13,  157136,   0,   0
>cblk77:              131072,      0,       2,       9,  162563,   0,   0
>cblk78:              131072,      0,       0,      30,  157154,   0,   0
>cblk79:              131072,      0,       1,      11,  166064,   0,   0
>cblk80:              131072,      0,       0,      11,  159294,   0,   0
>cblk81:              131072,      0,       0,      32,  167720,   0,   0
>cblk82:              131072,      0,       1,      11,  162413,   0,   0
>cblk83:              131072,      0,       0,      23,  167596,   0,   0
>cblk84:              131072,      0,       2,      10,  158783,   0,   0
>cblk85:              131072,      0,       0,      39,  157167,   0,   0
>cblk86:              131072,      0,       1,      10,  159136,   0,   0
>cblk87:              131072,      0,       0,      27,  167636,   0,   0
>cblk88:              131072,      0,       2,       8,  158747,   0,   0
>cblk89:              131072,      0,       1,      10,  158839,   0,   0
>cblk90:              131072,      0,       0,      21,  157152,   0,   0
>cblk91:              131072,      0,       0,      10,  167677,   0,   0
>cblk92:              131072,      0,       0,      29,  157185,   0,   0
>cblk93:              131072,      0,       1,      11,  158906,   0,   0
>cblk94:              131072,      0,       0,      27,  157208,   0,   0
>cblk95:              131072,      0,       2,      10,  158907,   0,   0
>cblk96:              131072,      0,       1,      11,  162523,   0,   0
>cblk97:              131072,      0,       2,       8,  158365,   0,   0
>cblk98:              131072,      0,       0,      28,  157114,   0,   0
>cblk99:              131072,      0,       1,      11,  166187,   0,   0
>
>
>------------------------------------------------------------------------
>vmstat -i
>
>interrupt                          total       rate
>irq4: uart0                            5          0
>irq6: fdc0                             1          0
>irq17: atapci1                   4323947      40178
>irq20: hpet0 uhci0              14963078     139038
>irq23: uhci3 ehci0                     2          0
>irq256: igb0:rxq0                 305820       2842
>irq257: igb0:rxq1                  28875        268
>irq258: igb0:rxq2                  16373        152
>irq259: igb0:rxq3                  21360        198
>irq260: igb0:aq                        3          0
>irq289: t6nex0:evt                     2          0
>irq290: t6nex0:0a0               4431856      41181
>irq291: t6nex0:0a1               4110493      38195
>irq292: t6nex0:0a2               4060181      37727
>irq293: t6nex0:0a3               4195117      38981
>irq294: t6nex0:0a4               4697388      43648
>irq295: t6nex0:0a5               4018159      37337
>irq296: t6nex0:0a6               4126775      38346
>irq297: t6nex0:0a7               4413716      41012
>irq298: t6nex0:0A0                 40071        372
>irq299: t6nex0:0A1                  2399         22
>Total                           53755621     499499
>
>------------------------------------------------------------------------
>pstat -T
>
> 85/260795 files
>0M/0M swap space
>
>------------------------------------------------------------------------
>pstat -s
>
>Device          512-blocks     Used    Avail Capacity
>
>------------------------------------------------------------------------
>iostat
>
>       tty            ada0            pass0             cbb0             cpu
> tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
>   0     3 53.99 151  7.94   0.00   0  0.00  30.42   6  0.16   0  0  8  1 91
>
>------------------------------------------------------------------------
>ipcs -a
>
>Message Queues:
>T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   
>
>Shared Memory:
>T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   
>
>Semaphores:
>T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   
>
>
>------------------------------------------------------------------------
>ipcs -T
>
>msginfo:
>	msgmax:        16384	(max characters in a message)
>	msgmni:           40	(# of message queues)
>	msgmnb:         2048	(max characters in a message queue)
>	msgtql:           40	(max # of messages in system)
>	msgssz:            8	(size of a message segment)
>	msgseg:         2048	(# of message segments in system)
>
>shminfo:
>	shmmax:    536870912	(max shared memory segment size)
>	shmmin:            1	(min shared memory segment size)
>	shmmni:          192	(max number of shared memory identifiers)
>	shmseg:          128	(max shared memory segments per process)
>	shmall:       131072	(max amount of shared memory in pages)
>
>seminfo:
>	semmni:           50	(# of semaphore identifiers)
>	semmns:          340	(# of semaphores in system)
>	semmnu:          150	(# of undo structures in system)
>	semmsl:          340	(max # of semaphores per id)
>	semopm:          100	(max # of operations per semop call)
>	semume:           50	(max # of undo entries per process)
>	semusz:          632	(size in bytes of undo structure)
>	semvmx:        32767	(semaphore maximum value)
>	semaem:        16384	(adjust on exit max value)
>
>
>------------------------------------------------------------------------
>nfsstat
>
>Rpc Counts:
>      Getattr      Setattr       Lookup     Readlink         Read        Write       Create       Remove
>            0            0            0            0            0            0            0            0
>       Rename         Link      Symlink        Mkdir        Rmdir      Readdir     RdirPlus       Access
>            0            0            0            0            0            0            0            0
>        Mknod       Fsstat       Fsinfo     PathConf       Commit
>            0            0            0            0            0
>Rpc Info:
>     TimedOut      Invalid    X Replies      Retries     Requests
>            0            0            0            0            0
>Cache Info:
>    Attr Hits  Attr Misses    Lkup Hits  Lkup Misses    BioR Hits  BioR Misses    BioW Hits  BioW Misses
>            0            0            0            0            0            0            0            0
>   BioRL Hits BioRL Misses    BioD Hits  BioD Misses    DirE Hits  DirE Misses    Accs Hits  Accs Misses
>            0            0            0            0            0            0            0            0
>Server Info:
>      Getattr      Setattr       Lookup     Readlink         Read        Write       Create       Remove
>            0            0            0            0            0            0            0            0
>       Rename         Link      Symlink        Mkdir        Rmdir      Readdir     RdirPlus       Access
>            0            0            0            0            0            0            0            0
>        Mknod       Fsstat       Fsinfo     PathConf       Commit
>            0            0            0            0            0
>Server Re-Failed:
>  140737488346832
>Server Faults:
>140737488346832
>Server Write 
>     WriteOps     WriteRPC      Opsaved
>            0            0            0
>Server Cache 
>       Inprog         Idem     Non-Idem       Misses
>            0            0            0            0
>
>------------------------------------------------------------------------
>netstat -s
>
>tcp:
>	29339027 packets sent
>		19255278 data packets (141016252366 bytes)
>		77 data packets (8512 bytes) retransmitted
>		6 data packets unnecessarily retransmitted
>		0 resends initiated by MTU discovery
>		10019797 ack-only packets (994363 delayed)
>		0 URG only packets
>		0 window probe packets
>		63867 window update packets
>		8 control packets
>	34513481 packets received
>		49138369 acks (for 141016252216 bytes)
>		226 duplicate acks
>		0 acks for unsent data
>		92121353 packets (143631167516 bytes) received in-sequence
>		302 completely duplicate packets (23584 bytes)
>		0 old duplicate packets
>		0 packets with some dup. data (0 bytes duped)
>		2 out-of-order packets (192 bytes)
>		0 packets (0 bytes) of data after window
>		0 window probes
>		0 window update packets
>		0 packets received after close
>		0 discarded for bad checksums
>		0 discarded for bad header offset fields
>		0 discarded because packet too short
>		0 discarded due to memory problems
>	3 connection requests
>	107 connection accepts
>	0 bad connection attempts
>	0 listen queue overflows
>	0 ignored RSTs in the windows
>	109 connections established (including accepts)
>		0 times used RTT from hostcache
>		0 times used RTT variance from hostcache
>		0 times used slow-start threshold from hostcache
>	18 connections closed (including 4 drops)
>		5 connections updated cached RTT on close
>		5 connections updated cached RTT variance on close
>		1 connection updated cached ssthresh on close
>	1 embryonic connection dropped
>	21856269 segments updated rtt (of 18448029 attempts)
>	75 retransmit timeouts
>		1 connection dropped by rexmit timeout
>	0 persist timeouts
>		0 connections dropped by persist timeout
>	0 Connections (fin_wait_2) dropped because of timeout
>	18 keepalive timeouts
>		16 keepalive probes sent
>		2 connections dropped by keepalive
>	13374155 correct ACK header predictions
>	12656689 correct data packet header predictions
>	107 syncache entries added
>		0 retransmitted
>		0 dupsyn
>		0 dropped
>		107 completed
>		0 bucket overflow
>		0 cache overflow
>		0 reset
>		0 stale
>		0 aborted
>		0 badack
>		0 unreach
>		0 zone failures
>	107 cookies sent
>	107 cookies received
>	5 hostcache entries added
>		0 bucket overflow
>	1 SACK recovery episode
>	1 segment rexmit in SACK recovery episodes
>	64 byte rexmits in SACK recovery episodes
>	16 SACK options (SACK blocks) received
>	1 SACK option (SACK blocks) sent
>	0 SACK scoreboard overflow
>	0 packets with ECN CE bit set
>	0 packets with ECN ECT(0) bit set
>	0 packets with ECN ECT(1) bit set
>	0 successful ECN handshakes
>	0 times ECN reduced the congestion window
>	0 packets with matching signature received
>	0 packets with bad signature received
>	0 times failed to make signature due to no SA
>	0 times unexpected signature received
>	0 times no signature provided by segment
>	0 Path MTU discovery black hole detection activations
>	0 Path MTU discovery black hole detection min MSS activations
>	0 Path MTU discovery black hole detection failures
>TCP connection count by state:
>	0 connections in CLOSED state
>	4 connections in LISTEN state
>	0 connections in SYN_SENT state
>	3 connections in SYN_RCVD state
>	97 connections in ESTABLISHED state
>	0 connections in CLOSE_WAIT state
>	2 connections in FIN_WAIT_1 state
>	18446744073709551614 connections in CLOSING state
>	0 connections in LAST_ACK state
>	0 connections in FIN_WAIT_2 state
>	0 connections in TIME_WAIT state
>udp:
>	52038 datagrams received
>	0 with incomplete header
>	0 with bad data length field
>	0 with bad checksum
>	8 with no checksum
>	0 dropped due to no socket
>	51945 broadcast/multicast datagrams undelivered
>	0 dropped due to full socket buffers
>	0 not for hashed pcb
>	93 delivered
>	93 datagrams output
>	0 times multicast source filter matched
>ip:
>	34568050 total packets received
>	0 bad header checksums
>	0 with size smaller than minimum
>	0 with data size < data length
>	0 with ip length > max ip packet size
>	0 with header length < data size
>	0 with data length < header length
>	0 with bad options
>	0 with incorrect version number
>	0 fragments received
>	0 fragments dropped (dup or out of space)
>	0 fragments dropped after timeout
>	0 packets reassembled ok
>	34568050 packets for this host
>	0 packets for unknown/unsupported protocol
>	0 packets forwarded (0 packets fast forwarded)
>	0 packets not forwardable
>	0 packets received for unknown multicast group
>	0 redirects sent
>	29339134 packets sent from this host
>	0 packets sent with fabricated ip header
>	0 output packets dropped due to no bufs, etc.
>	0 output packets discarded due to no route
>	0 output datagrams fragmented
>	0 fragments created
>	0 datagrams that can't be fragmented
>	0 tunneling packets that can't find gif
>	0 datagrams with bad address in header
>icmp:
>	0 calls to icmp_error
>	0 errors not generated in response to an icmp message
>	0 messages with bad code fields
>	0 messages less than the minimum length
>	0 messages with bad checksum
>	0 messages with bad length
>	0 multicast echo requests ignored
>	0 multicast timestamp requests ignored
>	Input histogram:
>		destination unreachable: 33
>	0 message responses generated
>	0 invalid return addresses
>	0 no return routes
>ipsec:
>	0 inbound packets violated process security policy
>	0 inbound packets failed due to insufficient memory
>	0 invalid inbound packets
>	0 outbound packets violated process security policy
>	0 outbound packets with no SA available
>	0 outbound packets failed due to insufficient memory
>	0 outbound packets with no route available
>	0 invalid outbound packets
>	0 outbound packets with bundled SAs
>	0 mbufs coalesced during clone
>	0 clusters coalesced during clone
>	0 clusters copied during clone
>	0 mbufs inserted during makespace
>ah:
>	0 packets shorter than header shows
>	0 packets dropped; protocol family not supported
>	0 packets dropped; no TDB
>	0 packets dropped; bad KCR
>	0 packets dropped; queue full
>	0 packets dropped; no transform
>	0 replay counter wraps
>	0 packets dropped; bad authentication detected
>	0 packets dropped; bad authentication length
>	0 possible replay packets detected
>	0 packets in
>	0 packets out
>	0 packets dropped; invalid TDB
>	0 bytes in
>	0 bytes out
>	0 packets dropped; larger than IP_MAXPACKET
>	0 packets blocked due to policy
>	0 crypto processing failures
>	0 tunnel sanity check failures
>esp:
>	0 packets shorter than header shows
>	0 packets dropped; protocol family not supported
>	0 packets dropped; no TDB
>	0 packets dropped; bad KCR
>	0 packets dropped; queue full
>	0 packets dropped; no transform
>	0 packets dropped; bad ilen
>	0 replay counter wraps
>	0 packets dropped; bad encryption detected
>	0 packets dropped; bad authentication detected
>	0 possible replay packets detected
>	0 packets in
>	0 packets out
>	0 packets dropped; invalid TDB
>	0 bytes in
>	0 bytes out
>	0 packets dropped; larger than IP_MAXPACKET
>	0 packets blocked due to policy
>	0 crypto processing failures
>	0 tunnel sanity check failures
>ipcomp:
>	0 packets shorter than header shows
>	0 packets dropped; protocol family not supported
>	0 packets dropped; no TDB
>	0 packets dropped; bad KCR
>	0 packets dropped; queue full
>	0 packets dropped; no transform
>	0 replay counter wraps
>	0 packets in
>	0 packets out
>	0 packets dropped; invalid TDB
>	0 bytes in
>	0 bytes out
>	0 packets dropped; larger than IP_MAXPACKET
>	0 packets blocked due to policy
>	0 crypto processing failures
>	0 packets sent uncompressed; size < compr. algo. threshold
>	0 packets sent uncompressed; compression was useless
>arp:
>	27 ARP requests sent
>	10 ARP replies sent
>	255669 ARP requests received
>	1318 ARP replies received
>	256987 ARP packets received
>	61 total packets dropped due to no ARP entry
>	23 ARP entrys timed out
>	0 Duplicate IPs seen
>ip6:
>	0 total packets received
>	0 with size smaller than minimum
>	0 with data size < data length
>	0 with bad options
>	0 with incorrect version number
>	0 fragments received
>	0 fragments dropped (dup or out of space)
>	0 fragments dropped after timeout
>	0 fragments that exceeded limit
>	0 packets reassembled ok
>	0 packets for this host
>	0 packets forwarded
>	0 packets not forwardable
>	0 redirects sent
>	5 packets sent from this host
>	0 packets sent with fabricated ip header
>	0 output packets dropped due to no bufs, etc.
>	0 output packets discarded due to no route
>	0 output datagrams fragmented
>	0 fragments created
>	0 datagrams that can't be fragmented
>	0 packets that violated scope rules
>	0 multicast packets which we don't join
>	Mbuf statistics:
>		0 one mbuf
>		0 one ext mbuf
>		0 two or more ext mbuf
>	0 packets whose headers are not contiguous
>	0 tunneling packets that can't find gif
>	0 packets discarded because of too many headers
>	0 failures of source address selection
>icmp6:
>	0 calls to icmp6_error
>	0 errors not generated in response to an icmp6 message
>	0 errors not generated because of rate limitation
>	Output histogram:
>		neighbor solicitation: 1
>		MLDv2 listener report: 1
>	0 messages with bad code fields
>	0 messages < minimum length
>	0 bad checksums
>	0 messages with bad length
>	Histogram of error messages to be generated:
>		0 no route
>		0 administratively prohibited
>		0 beyond scope
>		0 address unreachable
>		0 port unreachable
>		0 packet too big
>		0 time exceed transit
>		0 time exceed reassembly
>		0 erroneous header field
>		0 unrecognized next header
>		0 unrecognized option
>		0 redirect
>		0 unknown
>	0 message responses generated
>	0 messages with too many ND options
>	0 messages with bad ND options
>	0 bad neighbor solicitation messages
>	0 bad neighbor advertisement messages
>	0 bad router solicitation messages
>	0 bad router advertisement messages
>	0 bad redirect messages
>	0 path MTU changes
>ipsec6:
>	0 inbound packets violated process security policy
>	0 inbound packets failed due to insufficient memory
>	0 invalid inbound packets
>	0 outbound packets violated process security policy
>	0 outbound packets with no SA available
>	0 outbound packets failed due to insufficient memory
>	0 outbound packets with no route available
>	0 invalid outbound packets
>	0 outbound packets with bundled SAs
>	0 mbufs coalesced during clone
>	0 clusters coalesced during clone
>	0 clusters copied during clone
>	0 mbufs inserted during makespace
>rip6:
>	0 messages received
>	0 checksum calculations on inbound
>	0 messages with bad checksum
>	0 messages dropped due to no socket
>	0 multicast messages dropped due to no socket
>	0 messages dropped due to full socket buffers
>	0 delivered
>	0 datagrams output
>pfkey:
>	0 requests sent from userland
>	0 bytes sent from userland
>	0 messages with invalid length field
>	0 messages with invalid version field
>	0 messages with invalid message type field
>	0 messages too short
>	0 messages with memory allocation failure
>	0 messages with duplicate extension
>	0 messages with invalid extension type
>	0 messages with invalid sa type
>	0 messages with invalid address extension
>	0 requests sent to userland
>	0 bytes sent to userland
>	0 messages toward single socket
>	0 messages toward all sockets
>	0 messages toward registered sockets
>	0 messages with memory allocation failure
>
>------------------------------------------------------------------------
>netstat -m
>
>4591/9324/13915 mbufs in use (current/cache/total)
>1805/4387/6192/507330 mbuf clusters in use (current/cache/total/max)
>1/4313 mbuf+clusters out of packet secondary zone in use (current/cache)
>8608/2499/11107/253665 4k (page size) jumbo clusters in use (current/cache/total/max)
>0/0/0/225480 9k jumbo clusters in use (current/cache/total/max)
>2032/152/2184/169108 16k jumbo clusters in use (current/cache/total/max)
>71701K/23533K/95234K bytes allocated to network (current/cache/total)
>0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
>0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
>0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
>0/0/0 requests for jumbo clusters denied (4k/9k/16k)
>0 sendfile syscalls
>0 sendfile syscalls completed without I/O request
>0 requests for I/O initiated by sendfile
>0 pages read by sendfile as part of a request
>0 pages were valid at time of a sendfile request
>0 pages were valid and substituted to bogus page
>0 pages were requested for read ahead by applications
>0 pages were read ahead by sendfile
>0 times sendfile encountered an already busy page
>0 requests for sfbufs denied
>0 requests for sfbufs delayed
>
>------------------------------------------------------------------------
>netstat -anA
>
>Active Internet connections (including servers)
>Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
>fffff8001f01c350 udp4       0      0 *.514              *.*                
>fffff8001f01c4f8 udp6       0      0 *.514              *.*                
>
>------------------------------------------------------------------------
>netstat -aL
>
>Current listen queue sizes (qlen/incqlen/maxqlen)
>Proto Listen                           Local Address         
>
>------------------------------------------------------------------------
>fstat
>
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0xc00000000
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0xc00000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 10 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0xa00000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 10 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0xa00000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 10 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0xa00000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 10 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0x800000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 10 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0x800000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 10 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0x400000000
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0xc00000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0x600000000
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0x400000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 10 at 0x780000ffff
>fstat: can't read file 11 at 0x400000000
>fstat: can't read file 13 at 0x20007ffffffffff
>fstat: can't read file 14 at 0x4000000001fffff
>fstat: can't read file 16 at 0x780000ffff
>fstat: can't read file 17 at 0x400000000
>fstat: can't read file 19 at 0x20007ffffffffff
>fstat: can't read file 20 at 0x4000000001fffff
>fstat: can't read file 22 at 0x780000ffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>fstat: can't read file 4 at 0x780000ffff
>fstat: can't read file 5 at 0x800000000
>fstat: can't read file 7 at 0x20007ffffffffff
>fstat: can't read file 8 at 0x4000000001fffff
>fstat: can't read file 1 at 0x20007ffffffffff
>fstat: can't read file 2 at 0x4000000001fffff
>USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
>root     cat         2101 root /             2 drwxr-xr-x    1024  r
>root     cat         2101   wd /         80256 drwxr-xr-x     512  r
>root     cat         2101 text /        5619725 -r-xr-xr-x   23096  r
>root     cat         2101    0* pipe fffff80005ba95f0 <-> fffff80005ba9758      0 rw
>root     sh          2099 root /             2 drwxr-xr-x    1024  r
>root     sh          2099   wd /         80256 drwxr-xr-x     512  r
>root     sh          2099 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          2099    0* pipe fffff80005ba95f0 <-> fffff80005ba9758      0 rw
>root     find        2098 root /             2 drwxr-xr-x    1024  r
>root     find        2098   wd /        9572578 drwxr-xr-x  109056  r
>root     find        2098 text /        5620015 -r-xr-xr-x   64056  r
>root     find        2098    0 /dev         14 crw-rw-rw-    null  r
>root     sh          2094 root /             2 drwxr-xr-x    1024  r
>root     sh          2094   wd /         80256 drwxr-xr-x     512  r
>root     sh          2094 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          2094    0 /dev         14 crw-rw-rw-    null  r
>root     sh          2094    6 /        5618063 -rw-------      36  w
>root     mail        2093 root /             2 drwxr-xr-x    1024  r
>root     mail        2093   wd /         80256 drwxr-xr-x     512  r
>root     mail        2093 text /        5620210 -r-xr-xr-x  109112  r
>root     mail        2093    0* pipe fffff80005bb8be0 <-> fffff80005bb8d48      0 rw
>root     sh          2092 root /             2 drwxr-xr-x    1024  r
>root     sh          2092   wd /         80256 drwxr-xr-x     512  r
>root     sh          2092 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          2092    0 /dev         14 crw-rw-rw-    null  r
>root     sh          2084 root /             2 drwxr-xr-x    1024  r
>root     sh          2084   wd /         80256 drwxr-xr-x     512  r
>root     sh          2084 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          2084    0 /dev         14 crw-rw-rw-    null  r
>root     sh          2084    6 /        5618042 -rw-------      48  w
>root     lockf       2083 root /             2 drwxr-xr-x    1024  r
>root     lockf       2083   wd /         80256 drwxr-xr-x     512  r
>root     lockf       2083 text /        5620053 -r-xr-xr-x   14824  r
>root     lockf       2083    0 /dev         14 crw-rw-rw-    null  r
>root     sh          2081 root /             2 drwxr-xr-x    1024  r
>root     sh          2081   wd /         80256 drwxr-xr-x     512  r
>root     sh          2081 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          2081    0 /dev         14 crw-rw-rw-    null  r
>root     sh          2081    6 /        5618042 -rw-------      48  w
>root     sh          2080 root /             2 drwxr-xr-x    1024  r
>root     sh          2080   wd /         80256 drwxr-xr-x     512  r
>root     sh          2080 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          2080    0 /dev         14 crw-rw-rw-    null  r
>root     sh          2080    6 /        5618042 -rw-------      48  w
>root     mail        1994 root /             2 drwxr-xr-x    1024  r
>root     mail        1994   wd /         80256 drwxr-xr-x     512  r
>root     mail        1994 text /        5620210 -r-xr-xr-x  109112  r
>root     mail        1994    0* pipe fffff80005dcd000 <-> fffff80005dcd168      0 rw
>root     sh          1993 root /             2 drwxr-xr-x    1024  r
>root     sh          1993   wd /         80256 drwxr-xr-x     512  r
>root     sh          1993 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          1993    0* pipe fffff80005c738e8 <-> fffff80005c73a50      0 rw
>root     sh          1984 root /             2 drwxr-xr-x    1024  r
>root     sh          1984   wd /         80256 drwxr-xr-x     512  r
>root     sh          1984 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          1984    0* pipe fffff80005c738e8 <-> fffff80005c73a50      0 rw
>root     sh          1984    6* pipe fffff80005d7e168 <-> fffff80005d7e000      0 rw
>root     lockf       1983 root /             2 drwxr-xr-x    1024  r
>root     lockf       1983   wd /         80256 drwxr-xr-x     512  r
>root     lockf       1983 text /        5620053 -r-xr-xr-x   14824  r
>root     lockf       1983    0* pipe fffff80005c738e8 <-> fffff80005c73a50      0 rw
>root     sh          1981 root /             2 drwxr-xr-x    1024  r
>root     sh          1981   wd /         80256 drwxr-xr-x     512  r
>root     sh          1981 text /        5619754 -r-xr-xr-x  166544  r
>root     sh          1981    0* pipe fffff80005c738e8 <-> fffff80005c73a50      0 rw
>root     sh          1981    6* pipe fffff80005d7e168 <-> fffff80005d7e000      0 rw
>root     cron        1979 root /             2 drwxr-xr-x    1024  r
>root     cron        1979   wd /        1685387 drwxr-x---     512  r
>root     cron        1979 text /        5620273 -r-xr-xr-x   51800  r
>root     cron        1979    0 /dev         14 crw-rw-rw-    null rw
>root     ctld        1253 root /             2 drwxr-xr-x    1024  r
>root     ctld        1253   wd /             2 drwxr-xr-x    1024  r
>root     ctld        1253 text /        5620276 -r-xr-xr-x  129688  r
>root     ctld        1253    0 /dev         14 crw-rw-rw-    null rw
>root     ctld        1253    6 /dev         14 crw-rw-rw-    null rw
>root     ftcleanup   1137 root /             2 drwxr-xr-x    1024  r
>root     ftcleanup   1137   wd /             2 drwxr-xr-x    1024  r
>root     ctl          872 root /             2 drwxr-xr-x    1024  r
>root     ctl          872   wd /             2 drwxr-xr-x    1024  r
>root     zfskern      859 root /             2 drwxr-xr-x    1024  r
>root     zfskern      859   wd /             2 drwxr-xr-x    1024  r
>root     getty        761 root /             2 drwxr-xr-x    1024  r
>root     getty        761   wd /             2 drwxr-xr-x    1024  r
>root     getty        761 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        761 ctty /dev         55 crw-------   ttyu0 rw
>root     getty        761    0 /dev         55 crw-------   ttyu0 rw
>root     getty        760 root /             2 drwxr-xr-x    1024  r
>root     getty        760   wd /             2 drwxr-xr-x    1024  r
>root     getty        760 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        760 ctty /dev         86 crw-------   ttyv7 rw
>root     getty        760    0 /dev         86 crw-------   ttyv7 rw
>root     getty        759 root /             2 drwxr-xr-x    1024  r
>root     getty        759   wd /             2 drwxr-xr-x    1024  r
>root     getty        759 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        759 ctty /dev         85 crw-------   ttyv6 rw
>root     getty        759    0 /dev         85 crw-------   ttyv6 rw
>root     getty        758 root /             2 drwxr-xr-x    1024  r
>root     getty        758   wd /             2 drwxr-xr-x    1024  r
>root     getty        758 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        758 ctty /dev         84 crw-------   ttyv5 rw
>root     getty        758    0 /dev         84 crw-------   ttyv5 rw
>root     getty        757 root /             2 drwxr-xr-x    1024  r
>root     getty        757   wd /             2 drwxr-xr-x    1024  r
>root     getty        757 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        757 ctty /dev         83 crw-------   ttyv4 rw
>root     getty        757    0 /dev         83 crw-------   ttyv4 rw
>root     getty        756 root /             2 drwxr-xr-x    1024  r
>root     getty        756   wd /             2 drwxr-xr-x    1024  r
>root     getty        756 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        756 ctty /dev         82 crw-------   ttyv3 rw
>root     getty        756    0 /dev         82 crw-------   ttyv3 rw
>root     getty        755 root /             2 drwxr-xr-x    1024  r
>root     getty        755   wd /             2 drwxr-xr-x    1024  r
>root     getty        755 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        755 ctty /dev         81 crw-------   ttyv2 rw
>root     getty        755    0 /dev         81 crw-------   ttyv2 rw
>root     getty        754 root /             2 drwxr-xr-x    1024  r
>root     getty        754   wd /             2 drwxr-xr-x    1024  r
>root     getty        754 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        754 ctty /dev         80 crw-------   ttyv1 rw
>root     getty        754    0 /dev         80 crw-------   ttyv1 rw
>root     getty        753 root /             2 drwxr-xr-x    1024  r
>root     getty        753   wd /             2 drwxr-xr-x    1024  r
>root     getty        753 text /        5619696 -r-xr-xr-x   35400  r
>root     getty        753 ctty /dev         79 crw-------   ttyv0 rw
>root     getty        753    0 /dev         79 crw-------   ttyv0 rw
>root     cron         706 root /             2 drwxr-xr-x    1024  r
>root     cron         706   wd /        1685387 drwxr-x---     512  r
>root     cron         706 text /        5620273 -r-xr-xr-x   51800  r
>root     cron         706    0 /dev         14 crw-rw-rw-    null rw
>smmsp    sendmail     702 root /             2 drwxr-xr-x    1024  r
>smmsp    sendmail     702   wd /        1685399 drwxrwx---     512  r
>smmsp    sendmail     702 text /        5620514 -r-xr-sr-x  735944  r
>smmsp    sendmail     702    0 /dev         14 crw-rw-rw-    null  r
>root     sendmail     699 root /             2 drwxr-xr-x    1024  r
>root     sendmail     699   wd /        1685402 drwxr-xr-x     512  r
>root     sendmail     699 text /        5620514 -r-xr-sr-x  735944  r
>root     sendmail     699    0 /dev         14 crw-rw-rw-    null  r
>root     sshd         696 root /             2 drwxr-xr-x    1024  r
>root     sshd         696   wd /             2 drwxr-xr-x    1024  r
>root     sshd         696 text /        5619974 -r-xr-xr-x  285472  r
>root     sshd         696    0 /dev         14 crw-rw-rw-    null rw
>root     syslogd      571 root /             2 drwxr-xr-x    1024  r
>root     syslogd      571   wd /             2 drwxr-xr-x    1024  r
>root     syslogd      571 text /        5620343 -r-xr-xr-x   51784  r
>root     syslogd      571    0 /dev         14 crw-rw-rw-    null rw
>root     syslogd      571    6 /dev         14 crw-rw-rw-    null rw
>root     syslogd      571   12 /dev         14 crw-rw-rw-    null rw
>root     syslogd      571   18* pipe fffff80005d42be0 <-> fffff80005d42d48      0 rw
>root     devd         425 root /             2 drwxr-xr-x    1024  r
>root     devd         425   wd /             2 drwxr-xr-x    1024  r
>root     devd         425 text /        5619928 -r-xr-xr-x  1036400  r
>root     devd         425    0 /dev         14 crw-rw-rw-    null rw
>root     devd         425    6 /dev         14 crw-rw-rw-    null rw
>root     adjkerntz    125 root /             2 drwxr-xr-x    1024  r
>root     adjkerntz    125   wd /             2 drwxr-xr-x    1024  r
>root     adjkerntz    125 text /        5619843 -r-xr-xr-x   14832  r
>root     adjkerntz    125    0 /dev         14 crw-rw-rw-    null rw
>root     init           1 root /             2 drwxr-xr-x    1024  r
>root     init           1   wd /             2 drwxr-xr-x    1024  r
>root     init           1 text /        5619887 -r-xr-xr-x  1277680  r
>root     kernel         0 root /             2 drwxr-xr-x    1024  r
>root     kernel         0   wd /             2 drwxr-xr-x    1024  r
>
>------------------------------------------------------------------------
>dmesg
>
>Copyright (c) 1992-2018 The FreeBSD Project.
>Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>	The Regents of the University of California. All rights reserved.
>FreeBSD is a registered trademark of The FreeBSD Foundation.
>FreeBSD 12.0-CURRENT #8: Fri Feb 16 14:58:28 IST 2018
>    root@munnar:/usr/obj/usr/src/amd64.amd64/sys/MYKERNEL amd64
>FreeBSD clang version 6.0.0 (branches/release_60 324090) (based on LLVM 6.0.0)
>WARNING: WITNESS option enabled, expect reduced performance.
>VT(vga): resolution 640x480
>CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.12-MHz K8-class CPU)
>  Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10
>  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>  Features2=0xc0ce33d<SSE3,DTES64,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,OSXSAVE>
>  AMD Features=0x20100800<SYSCALL,NX,LM>
>  AMD Features2=0x1<LAHF>
>  VT-x: (disabled in BIOS) HLT,PAUSE
>  TSC: P-state invariant, performance statistics
>real memory  = 8589934592 (8192 MB)
>avail memory = 8238014464 (7856 MB)
>Event timer "LAPIC" quality 100
>ACPI APIC Table: <PTLTD  	 APIC  >
>FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
>FreeBSD/SMP: 2 package(s) x 4 core(s)
>random: unblocking device.
>ioapic0 <Version 2.0> irqs 0-23 on motherboard
>ioapic1 <Version 2.0> irqs 24-47 on motherboard
>SMP: AP CPU #2 Launched!
>SMP: AP CPU #1 Launched!
>SMP: AP CPU #3 Launched!
>SMP: AP CPU #4 Launched!
>SMP: AP CPU #5 Launched!
>SMP: AP CPU #7 Launched!
>SMP: AP CPU #6 Launched!
>Timecounter "TSC" frequency 2000118732 Hz quality 1000
>random: entropy device external interface
>netmap: loaded module
>[ath_hal] loaded
>module_register_init: MOD_LOAD (vesa, 0xffffffff81803590, 0) error 19
>kbd1 at kbdmux0
>nexus0
>vtvga0: <VT VGA driver> on motherboard
>cryptosoft0: <software crypto> on motherboard
>acpi0: <PTLTD 	 XSDT> on motherboard
>acpi0: Power Button (fixed)
>cpu0: <ACPI CPU> on acpi0
>cpu1: <ACPI CPU> on acpi0
>cpu2: <ACPI CPU> on acpi0
>cpu3: <ACPI CPU> on acpi0
>cpu4: <ACPI CPU> on acpi0
>cpu5: <ACPI CPU> on acpi0
>cpu6: <ACPI CPU> on acpi0
>cpu7: <ACPI CPU> on acpi0
>hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
>Timecounter "HPET" frequency 14318180 Hz quality 950
>Event timer "HPET" frequency 14318180 Hz quality 350
>Event timer "HPET1" frequency 14318180 Hz quality 340
>Event timer "HPET2" frequency 14318180 Hz quality 340
>atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
>atrtc0: registered as a time-of-day clock, resolution 1.000000s
>Event timer "RTC" frequency 32768 Hz quality 0
>attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
>Timecounter "i8254" frequency 1193182 Hz quality 0
>Event timer "i8254" frequency 1193182 Hz quality 100
>Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
>pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>pci0: <ACPI PCI bus> on pcib0
>pcib1: <ACPI PCI-PCI bridge> irq 24 at device 1.0 on pci0
>pci1: <ACPI PCI bus> on pcib1
>pcib2: <ACPI PCI-PCI bridge> irq 26 at device 3.0 on pci0
>pci2: <ACPI PCI bus> on pcib2
>pcib3: <ACPI PCI-PCI bridge> irq 26 at device 0.0 on pci2
>pci3: <ACPI PCI bus> on pcib3
>pcib4: <ACPI PCI-PCI bridge> irq 26 at device 0.0 on pci3
>pci4: <ACPI PCI bus> on pcib4
>pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci2
>pci5: <ACPI PCI bus> on pcib5
>pcib6: <ACPI PCI-PCI bridge> irq 28 at device 5.0 on pci0
>pci6: <ACPI PCI bus> on pcib6
>pci6: <network, ethernet> at device 0.0 (no driver attached)
>pci6: <network, ethernet> at device 0.1 (no driver attached)
>pci6: <network, ethernet> at device 0.2 (no driver attached)
>pci6: <network, ethernet> at device 0.3 (no driver attached)
>pci6: <network, ethernet> at device 0.4 (no driver attached)
>pci6: <mass storage, SCSI> at device 0.5 (no driver attached)
>pci6: <serial bus, Fibre Channel> at device 0.6 (no driver attached)
>pcib7: <ACPI PCI-PCI bridge> irq 30 at device 7.0 on pci0
>pci7: <ACPI PCI bus> on pcib7
>pci7: <network, ethernet> at device 0.0 (no driver attached)
>pci7: <network, ethernet> at device 0.1 (no driver attached)
>pci7: <network, ethernet> at device 0.2 (no driver attached)
>pci7: <network, ethernet> at device 0.3 (no driver attached)
>pci7: <network, ethernet> at device 0.4 (no driver attached)
>pci7: <mass storage, SCSI> at device 0.5 (no driver attached)
>pci7: <serial bus, Fibre Channel> at device 0.6 (no driver attached)
>pcib8: <ACPI PCI-PCI bridge> irq 32 at device 9.0 on pci0
>pci8: <ACPI PCI bus> on pcib8
>igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x2000-0x201f mem 0xdb020000-0xdb03ffff,0xdb000000-0xdb01ffff,0xdb080000-0xdb083fff irq 32 at device 0.0 on pci8
>igb0: attach_pre capping queues at 4
>igb0: using 1024 tx descriptors and 1024 rx descriptors
>igb0: msix_init qsets capped at 4
>igb0: pxm cpus: 8 queue msgs: 9 admincnt: 1
>igb0: using 4 rx queues 4 tx queues 
>igb0: Using MSIX interrupts with 5 vectors
>igb0: allocated for 4 tx_queues
>igb0: allocated for 4 rx_queues
>igb0: Ethernet address: 00:30:48:b9:93:b2
>igb0: netmap queues/slots: TX 4/1024, RX 4/1024
>igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x2020-0x203f mem 0xdb060000-0xdb07ffff,0xdb040000-0xdb05ffff,0xdb084000-0xdb087fff irq 46 at device 0.1 on pci8
>igb1: attach_pre capping queues at 4
>igb1: using 1024 tx descriptors and 1024 rx descriptors
>igb1: msix_init qsets capped at 4
>igb1: pxm cpus: 8 queue msgs: 9 admincnt: 1
>igb1: using 4 rx queues 4 tx queues 
>igb1: Using MSIX interrupts with 5 vectors
>igb1: allocated for 4 tx_queues
>igb1: allocated for 4 rx_queues
>igb1: Ethernet address: 00:30:48:b9:93:b3
>igb1: netmap queues/slots: TX 4/1024, RX 4/1024
>uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 20 at device 29.0 on pci0
>uhci0: LegSup = 0x3000
>usbus0 on uhci0
>usbus0: 12Mbps Full Speed USB v1.0
>uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 21 at device 29.1 on pci0
>usbus1 on uhci1
>usbus1: 12Mbps Full Speed USB v1.0
>uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 22 at device 29.2 on pci0
>usbus2 on uhci2
>usbus2: 12Mbps Full Speed USB v1.0
>uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port 0x1860-0x187f irq 23 at device 29.3 on pci0
>usbus3 on uhci3
>usbus3: 12Mbps Full Speed USB v1.0
>ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xdb404000-0xdb4043ff irq 23 at device 29.7 on pci0
>usbus4: EHCI version 1.0
>usbus4 on ehci0
>usbus4: 480Mbps High Speed USB v2.0
>pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
>pci9: <ACPI PCI bus> on pcib9
>vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xd0000000-0xd7ffffff,0xdb100000-0xdb10ffff irq 18 at device 1.0 on pci9
>vgapci0: Boot video device
>isab0: <PCI-ISA bridge> at device 31.0 on pci0
>isa0: <ISA bus> on isab0
>atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1880-0x188f at device 31.1 on pci0
>ata0: <ATA channel> at channel 0 on atapci0
>atapci1: <Intel 63XXESB2 SATA300 controller> port 0x18c0-0x18c7,0x18b8-0x18bb,0x18b0-0x18b7,0x1894-0x1897,0x18a0-0x18af mem 0xdb404400-0xdb4047ff irq 17 at device 31.2 on pci0
>ata2: <ATA channel> at channel 0 on atapci1
>ata3: <ATA channel> at channel 1 on atapci1
>acpi_button0: <Power Button> on acpi0
>acpi_syscontainer0: <System Container> on acpi0
>atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
>atkbd0: <AT Keyboard> irq 1 on atkbdc0
>kbd0 at atkbd0
>atkbd0: [GIANT-LOCKED]
>uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
>uart0: console (9600,n,8,1)
>uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
>fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
>orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcefff pnpid ORM0000 on isa0
>Timecounters tick every 1.000 msec
>ugen1.1: <Intel UHCI root HUB> at usbus1
>ugen4.1: <Intel EHCI root HUB> at usbus4
>ugen3.1: <Intel UHCI root HUB> at usbus3
>ugen0.1: <Intel UHCI root HUB> at usbus0
>ugen2.1: <Intel UHCI root HUB> at usbus2
>uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
>uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
>uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
>uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
>uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
>uhub0: 2 ports with 2 removable, self powered
>uhub2: 2 ports with 2 removable, self powered
>uhub3: 2 ports with 2 removable, self powered
>uhub1: 2 ports with 2 removable, self powered
>ada0 at ata2 bus 0 scbus1 target 0 lun 0
>ada0: <ST250DM001 HD253GJ 1AJ10001> ATA8-ACS SATA 2.x device
>ada0: Serial Number S24JJ9FC200111
>ada0: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
>ada0: 238475MB (488397168 512 byte sectors)
>ada0: quirks=0x1<4K>
>WARNING: WITNESS option enabled, expect reduced performance.
>Trying to mount root from ufs:/dev/ada0s1a [rw]...
>Setting hostuuid: 53d19f64-d663-a017-8922-003048b993b2.
>Setting hostid: 0x2f4c1e73.
>uhub4: 8 ports with 8 removable, self powered
>Starting file system checks:
>/dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
>/dev/ada0s1a: clean, 7340363 free (27267 frags, 914137 blocks, 0.1% fragmentation)
>Mounting local filesystems:.
>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/perl5/5.24/mach/CORE
>32-bit compatibility ldconfig path: /usr/lib32
>Setting hostname: munnar.
>Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
>Feeding entropy: .
>ugen0.2: <Logitech USB Keyboard> at usbus0
>ukbd0 on uhub1
>ukbd0: <USB Keyboard> on usbus0
>kbd2 at ukbd0
>Link state changed to up
>igb0: link state changed to UP
>Starting Network: lo0 igb0 igb1.
>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
>	inet6 ::1 prefixlen 128 
>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
>	inet 127.0.0.1 netmask 0xff000000 
>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>	groups: lo 
>igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>	options=e525bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
>	ether 00:30:48:b9:93:b2
>	inet 10.193.184.94 netmask 0xfffffc00 broadcast 10.193.187.255 
>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>	media: Ethernet autoselect (100baseTX <full-duplex>)
>	status: active
>igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>	options=e525bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
>	ether 00:30:48:b9:93:b3
>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>	media: Ethernet autoselect
>	status: no carrier
>Starting devd.
>Starting Network: igb1.
>igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>	options=e525bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
>	ether 00:30:48:b9:93:b3
>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>	media: Ethernet autoselect
>	status: no carrier
>uhid0 on uhub1
>uhid0: <USB Keyboard> on usbus0
>add host 127.0.0.1: gateway lo0 fib 0: route already in table
>add net default: gateway 10.193.184.5
>add host ::1: gateway lo0 fib 0: route already in table
>add net fe80::: gateway ::1
>add net ff02::: gateway ::1
>add net ::ffff:0.0.0.0: gateway ::1
>add net ::0.0.0.0: gateway ::1
>Creating and/or trimming log files.
>Starting syslogd.
>No core dumps found.
>Clearing /tmp (X related).
>Recovering vi editor sessions:.
>Updating motd:.
>Mounting late filesystems:.
>Configuring vt: blanktime.
>Performing sanity check on sshd configuration.
>Starting sshd.
>Starting sendmail_submit.
>Starting sendmail_msp_queue.
>Starting cron.
>Starting background file system checks in 60 seconds.
>
>Mon Feb 19 19:24:56 IST 2018
>lock order reversal:
> 1st 0xfffffe0001c384c0 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3566
> 2nd 0xfffff80005b03a00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:289
>stack backtrace:
>#0 0xffffffff810e19d1 at witness_debugger+0xf1
>#1 0xffffffff810e12a6 at witness_checkorder+0xf16
>#2 0xffffffff81050bfc at _sx_xlock+0x19c
>#3 0xffffffff8153b50c at ufsdirhash_acquire+0x4c
>#4 0xffffffff8153b2ec at ufsdirhash_add+0x1c
>#5 0xffffffff8153fcbc at ufs_direnter+0xbac
>#6 0xffffffff8154e921 at ufs_makeinode+0x8c1
>#7 0xffffffff81548865 at ufs_create+0x55
>#8 0xffffffff8199c788 at VOP_CREATE_APV+0x228
>#9 0xffffffff811a2639 at VOP_CREATE+0x59
>#10 0xffffffff811a21bc at vn_open_cred+0x32c
>#11 0xffffffff811a1e7c at vn_open+0x4c
>#12 0xffffffff81194dd9 at kern_openat+0x2e9
>#13 0xffffffff81195239 at sys_openat+0x69
>#14 0xffffffff8172a48e at syscallenter+0x4ee
>#15 0xffffffff81729d3b at amd64_syscall+0x1b
>#16 0xffffffff816fa4cd at fast_syscall_common+0x101
>Feb 19 20:22:11 munnar sshd[828]: error: PAM: Authentication error for root from rgupta-pc.asicdesigners.com
>ZFS filesystem version: 5
>ZFS storage pool version: features support (5000)
>Feb 19 20:26:23 munnar ctld[873]: failed to apply configuration; exiting
>t5nex0: <Chelsio T520-CR> mem 0xd9400000-0xd947ffff,0xd8000000-0xd8ffffff,0xd9688000-0xd9689fff irq 28 at device 0.4 on pci6
>t5nex0: PCIe gen2 x8, 2 ports, 22 MSI-X interrupts, 55 eq, 21 iq
>t6nex0: <Chelsio T6225-CR> mem 0xd9c00000-0xd9c7ffff,0xda000000-0xdaffffff,0xd9e88000-0xd9e89fff irq 30 at device 0.4 on pci7
>t6nex0: PCIe gen2 x8, 2 ports, 22 MSI-X interrupts, 55 eq, 21 iq
>cxl0: <port 0> on t5nex0
>cxl0: Ethernet address: 00:07:43:28:cd:a0
>cxl0: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
>cxl1: <port 1> on t5nex0
>cxl1: Ethernet address: 00:07:43:28:cd:a8
>cxl1: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
>cc0: <port 0> on t6nex0
>cc0: Ethernet address: 00:07:43:3e:b5:c0
>cc0: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
>cc1: <port 1> on t6nex0
>cc1: Ethernet address: 00:07:43:3e:b5:c8
>cc1: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
>cc0: link state changed to UP
>Feb 19 20:27:12 munnar ctld[1133]: 102.1.1.2 (iqn.1994-05.com.redhat:ad152041b45a): read: Connection reset by peer
>lock order reversal:
> 1st 0xfffff80005b8b7c8 ufs (ufs) @ /usr/src/sys/kern/vfs_lookup.c:671
> 2nd 0xfffffe0001cf0bc0 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:282
> 3rd 0xfffff8005b5ecb78 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2607
>stack backtrace:
>#0 0xffffffff810e19d1 at witness_debugger+0xf1
>#1 0xffffffff810e12a6 at witness_checkorder+0xf16
>#2 0xffffffff80ffa8b9 at __lockmgr_args+0xe49
>#3 0xffffffff8153467f at _lockmgr_args+0x8f
>#4 0xffffffff815329e2 at ffs_lock+0xd2
>#5 0xffffffff819a1ffb at VOP_LOCK1_APV+0x23b
>#6 0xffffffff811a4f43 at VOP_LOCK1+0x53
>#7 0xffffffff811a2e51 at _vn_lock+0xd1
>#8 0xffffffff81184497 at vget+0x137
>#9 0xffffffff8116de11 at vfs_hash_get+0xf1
>#10 0xffffffff8152bae9 at ffs_vgetf+0x69
>#11 0xffffffff81515f8e at flush_pagedep_deps+0x65e
>#12 0xffffffff81515827 at softdep_sync_buf+0x437
>#13 0xffffffff81533f8a at ffs_syncvnode+0x39a
>#14 0xffffffff814f0ef5 at ffs_truncate+0xda5
>#15 0xffffffff8153ff45 at ufs_direnter+0xe35
>#16 0xffffffff8154e921 at ufs_makeinode+0x8c1
>#17 0xffffffff81548865 at ufs_create+0x55
>Feb 19 21:09:03 munnar ctld[1254]: 102.1.1.2 (iqn.1994-05.com.redhat:ad152041b45a): read: Connection reset by peer
>witness_lock_list_get: witness exhausted
>ctl_datamove: tag 0x0051 on (0:88:0) aborted
>(0:4:0/1): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:4:0/1): Tag: 0x0011, type 1
>(0:4:0/1): ctl_process_done: 98 seconds
>(0:102:0/99): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:102:0/99): Tag: 0x002b, type 1
>(0:102:0/99): ctl_process_done: 99 seconds
>(0:81:0/78): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:42:0/39): WRITE(10). CDB: 2a 08 00 10 b9 c8 00 00 08 00 
>(0:81:0/78): Tag: 0x0041, type 1
>(0:42:0/39): Tag: 0x0073, type 1
>(0:42:0/39): ctl_process_done: 99 seconds
>(0:81:0/78): ctl_process_done: 99 seconds
>(0:33:0/30): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:56:0/53): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:33:0/30): Tag: 0x0073, type 1
>(0:33:0/30): ctl_process_done: 108 seconds
>(0:56:0/53): Tag: 0x0069, type 1
>(0:56:0/53): ctl_process_done: 108 seconds
>(0:25:0/22): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:25:0/22): Tag: 0x0078, type 1
>(0:25:0/22): ctl_process_done: 108 seconds
>(0:97:0/94): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:97:0/94): Tag: 0x005f, type 1
>(0:97:0/94): ctl_process_done: 128 seconds
>(0:6:0/3): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:6:0/3): Tag: 0x004c, type 1
>(0:6:0/3): ctl_process_done: 129 seconds
>(0:80:0/77): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:80:0/77): Tag: 0x0067, type 1
>(0:80:0/77): ctl_process_done: 130 seconds
>(0:24:0/21): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:24:0/21): Tag: 0x0075, type 1
>(0:24:0/21): ctl_process_done: 131 seconds
>(0:15:0/12): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:18:0/15): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:15:0/12): Tag: 0x001b, type 1
>(0:18:0/15): Tag: 0x0011, type 1
>(0:18:0/15): ctl_process_done: 131 seconds
>(0:15:0/12): ctl_process_done: 132 seconds
>(0:29:0/26): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:29:0/26): Tag: 0x0030, type 1
>(0:29:0/26): ctl_process_done: 133 seconds
>(0:40:0/37): WRITE(10). CDB: 2a 08 00 10 ba 50 00 00 08 00 
>(0:40:0/37): Tag: 0x0033, type 1
>(0:71:0/68): WRITE(10). CDB: 2a 08 00 10 ad 00 00 00 08 00 
>(0:40:0/37): ctl_process_done: 140 seconds
>(0:28:0/25): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:71:0/68): Tag: 0x0007, type 1
>(0:71:0/68): ctl_process_done: 140 seconds
>(0:28:0/25): Tag: 0x0002, type 1
>(0:28:0/25): ctl_process_done: 138 seconds
>(0:67:0/64): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:67:0/64): Tag: 0x001d, type 1
>(0:67:0/64): ctl_process_done: 141 seconds
>(0:3:0/0): WRITE(10). CDB: 2a 08 00 10 b2 e8 00 00 08 00 
>(0:38:0/35): WRITE(10). CDB: 2a 08 00 10 b2 b0 00 00 08 00 
>(0:3:0/0): Tag: 0x002c, type 1
>(0:3:0/0): ctl_process_done: 131 seconds
>(0:45:0/42): WRITE(10). CDB: 2a 08 00 10 ac c8 00 00 08 00 
>(0:38:0/35): Tag: 0x0029, type 1
>(0:45:0/42): Tag: 0x0019, type 1
>(0:38:0/35): ctl_process_done: 113 seconds
>(0:45:0/42): ctl_process_done: 119 seconds
>(0:43:0/40): WRITE(10). CDB: 2a 08 00 10 b3 98 00 00 08 00 
>(0:43:0/40): Tag: 0x004b, type 1
>(0:43:0/40): ctl_process_done: 118 seconds
>(0:91:0/88): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:91:0/88): Tag: 0x006f, type 1
>(0:91:0/88): ctl_process_done: 114 seconds
>(0:35:0/32): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:35:0/32): Tag: 0x005e, type 1
>(0:35:0/32): ctl_process_done: 114 seconds
>(0:17:0/14): WRITE(10). CDB: 2a 08 00 10 b5 50 00 00 08 00 
>(0:17:0/14): Tag: 0x0060, type 1
>(0:17:0/14): ctl_process_done: 114 seconds
>(0:55:0/52): WRITE(10). CDB: 2a 08 00 10 b5 38 00 00 08 00 
>(0:55:0/52): Tag: 0x002d, type 1
>(0:55:0/52): ctl_process_done: 104 seconds
>(0:101:0/98): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:22:0/19): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:101:0/98): Tag: 0x0002, type 1
>(0:101:0/98): ctl_process_done: 113 seconds
>(0:22:0/19): Tag: 0x006c, type 1
>(0:22:0/19): ctl_process_done: 113 seconds
>(0:36:0/33): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:53:0/50): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:36:0/33): Tag: 0x0050, type 1
>(0:53:0/50): Tag: 0x0014, type 1
>(0:36:0/33): ctl_process_done: 113 seconds
>(0:53:0/50): ctl_process_done: 113 seconds
>(0:84:0/81): WRITE(10). CDB: 2a 08 00 10 b7 b0 00 00 08 00 
>(0:84:0/81): Tag: 0x002f, type 1
>(0:84:0/81): ctl_process_done: 108 seconds
>(0:58:0/55): WRITE(10). CDB: 2a 08 00 10 b5 30 00 00 08 00 
>(0:58:0/55): Tag: 0x0041, type 1
>(0:58:0/55): ctl_process_done: 108 seconds
>(0:64:0/61): WRITE(10). CDB: 2a 08 00 10 b2 20 00 00 08 00 
>(0:64:0/61): Tag: 0x0009, type 1
>(0:64:0/61): ctl_process_done: 109 seconds
>(0:9:0/6): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:9:0/6): Tag: 0x007c, type 1
>(0:9:0/6): ctl_process_done: 109 seconds
>(0:49:0/46): WRITE(10). CDB: 2a 08 00 10 b7 88 00 00 08 00 
>(0:54:0/51): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:49:0/46): Tag: 0x0032, type 1
>(0:54:0/51): Tag: 0x0024, type 1
>(0:49:0/46): ctl_process_done: 109 seconds
>(0:54:0/51): ctl_process_done: 106 seconds
>(0:23:0/20): WRITE(10). CDB: 2a 08 00 10 b8 d8 00 00 08 00 
>(0:23:0/20): Tag: 0x002f, type 1
>(0:23:0/20): ctl_process_done: 104 seconds
>(0:76:0/73): WRITE(10). CDB: 2a 08 00 10 ad d0 00 00 08 00 
>(0:93:0/90): WRITE(10). CDB: 2a 08 00 10 b7 20 00 00 08 00 
>(0:76:0/73): Tag: 0x001e, type 1
>(0:76:0/73): ctl_process_done: 107 seconds
>(0:88:0/85): WRITE(10). CDB: 2a 08 00 10 b5 58 00 00 08 00 
>(0:93:0/90): Tag: 0x0031, type 1
>(0:88:0/85): Tag: 0x004a, type 1
>(0:88:0/85): ctl_process_done: 95 seconds
>(0:72:0/69): WRITE(10). CDB: 2a 08 00 10 b5 b8 00 00 08 00 
>(0:93:0/90): ctl_process_done: 107 seconds
>(0:72:0/69): Tag: 0x0055, type 1
>(0:27:0/24): WRITE(10). CDB: 2a 08 00 10 b2 e8 00 00 08 00 
>(0:72:0/69): ctl_process_done: 116 seconds
>(0:27:0/24): Tag: 0x001b, type 1
>(0:74:0/71): WRITE(10). CDB: 2a 08 00 10 a9 80 00 00 08 00 
>(0:27:0/24): ctl_process_done: 116 seconds
>(0:74:0/71): Tag: 0x0041, type 1
>(0:74:0/71): ctl_process_done: 96 seconds
>(0:7:0/4): WRITE(10). CDB: 2a 08 00 10 a1 a0 00 00 08 00 
>(0:7:0/4): Tag: 0x0059, type 1
>(0:7:0/4): ctl_process_done: 96 seconds
>(0:31:0/28): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:31:0/28): Tag: 0x0040, type 1
>(0:31:0/28): ctl_process_done: 95 seconds
>(0:65:0/62): WRITE(10). CDB: 2a 08 00 10 b7 38 00 00 08 00 
>(0:65:0/62): Tag: 0x0070, type 1
>(0:65:0/62): ctl_process_done: 94 seconds
>(0:42:0/39): WRITE(10). CDB: 2a 08 00 10 b9 c8 00 00 08 00 
>(0:42:0/39): Tag: 0x0027, type 1
>(0:42:0/39): ctl_process_done: 118 seconds
>(0:42:0/39): WRITE(10). CDB: 2a 08 00 10 b9 f8 00 00 08 00 
>(0:39:0/36): WRITE(10). CDB: 2a 08 00 10 b8 88 00 00 08 00 
>(0:42:0/39): Tag: 0x0002, type 1
>(0:39:0/36): Tag: 0x007f, type 1
>(0:42:0/39): ctl_process_done: 94 seconds
>(0:39:0/36): ctl_process_done: 94 seconds
>(0:27:0/24): WRITE(10). CDB: 2a 08 00 10 b3 18 00 00 08 00 
>(0:27:0/24): Tag: 0x0046, type 1
>(0:27:0/24): ctl_process_done: 94 seconds
>(0:83:0/80): WRITE(10). CDB: 2a 08 00 10 b5 98 00 00 08 00 
>(0:54:0/51): WRITE(10). CDB: 2a 08 00 10 b9 18 00 00 08 00 
>(0:83:0/80): Tag: 0x002c, type 1
>(0:54:0/51): Tag: 0x0053, type 1
>(0:83:0/80): ctl_process_done: 104 seconds
>(0:54:0/51): ctl_process_done: 103 seconds
>(0:96:0/93): WRITE(10). CDB: 2a 08 00 10 a7 78 00 00 08 00 
>(0:96:0/93): Tag: 0x0047, type 1
>(0:96:0/93): ctl_process_done: 103 seconds
>(0:68:0/65): WRITE(10). CDB: 2a 08 00 10 ae 50 00 00 08 00 
>(0:68:0/65): Tag: 0x003a, type 1
>(0:68:0/65): ctl_process_done: 106 seconds
>ctl_datamove: tag 0x001a on (0:21:0) aborted
>(0:98:0/95): WRITE(10). CDB: 2a 08 00 10 b5 20 00 00 08 00 
>(0:98:0/95): Tag: 0x0060, type 1
>(0:98:0/95): ctl_process_done: 114 seconds
>(0:73:0/70): WRITE(10). CDB: 2a 00 00 00 90 00 00 08 00 00 
>(0:73:0/70): Tag: 0x0031, type 1
>(0:73:0/70): ctl_process_done: 104 seconds
>(0:48:0/45): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:48:0/45): Tag: 0x0034, type 1
>(0:48:0/45): ctl_process_done: 94 seconds
>(0:62:0/59): WRITE(10). CDB: 2a 00 00 00 98 00 00 08 00 00 
>(0:62:0/59): Tag: 0x002f, type 1
>(0:62:0/59): ctl_process_done: 102 seconds
>(0:91:0/88): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:91:0/88): Tag: 0x0025, type 1
>(0:91:0/88): ctl_process_done: 107 seconds
>(0:62:0/59): WRITE(10). CDB: 2a 00 00 00 70 00 00 08 00 00 
>(0:62:0/59): Tag: 0x0076, type 1
>(0:62:0/59): ctl_process_done: 106 seconds
>(0:23:0/20): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:23:0/20): Tag: 0x0039, type 1
>(0:23:0/20): ctl_process_done: 117 seconds
>(0:20:0/17): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:57:0/54): WRITE(10). CDB: 2a 00 00 00 78 00 00 08 00 00 
>(0:20:0/17): Tag: 0x0071, type 1
>(0:57:0/54): Tag: 0x0015, type 1
>(0:20:0/17): ctl_process_done: 91 seconds
>(0:57:0/54): ctl_process_done: 142 seconds
>(0:78:0/75): WRITE(10). CDB: 2a 00 00 00 c0 00 00 08 00 00 
>(0:78:0/75): Tag: 0x0043, type 1
>(0:78:0/75): ctl_process_done: 136 seconds
>(0:97:0/94): WRITE(10). CDB: 2a 08 00 10 d6 48 00 00 08 00 
>(0:97:0/94): Tag: 0x001f, type 1
>(0:97:0/94): ctl_process_done: 91 seconds
>(0:18:0/15): WRITE(10). CDB: 2a 00 00 00 f8 00 00 08 00 00 
>(0:18:0/15): Tag: 0x0035, type 1
>(0:18:0/15): ctl_process_done: 92 seconds
>(0:73:0/70): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:73:0/70): Tag: 0x007d, type 1
>(0:73:0/70): ctl_process_done: 139 seconds
>(0:16:0/13): WRITE(10). CDB: 2a 00 00 00 d0 00 00 08 00 00 
>(0:16:0/13): Tag: 0x0061, type 1
>(0:16:0/13): ctl_process_done: 97 seconds
>(0:57:0/54): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:57:0/54): Tag: 0x000a, type 1
>(0:57:0/54): ctl_process_done: 134 seconds
>(0:16:0/13): WRITE(10). CDB: 2a 00 00 00 d8 00 00 08 00 00 
>(0:16:0/13): Tag: 0x005a, type 1
>(0:16:0/13): ctl_process_done: 107 seconds
>(0:18:0/15): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:18:0/15): Tag: 0x0039, type 1
>(0:18:0/15): ctl_process_done: 95 seconds
>(0:77:0/74): WRITE(10). CDB: 2a 00 00 00 78 00 00 08 00 00 
>(0:77:0/74): Tag: 0x0018, type 1
>(0:77:0/74): ctl_process_done: 91 seconds
>(0:58:0/55): WRITE(10). CDB: 2a 08 00 10 d2 58 00 00 08 00 
>(0:58:0/55): Tag: 0x0020, type 1
>(0:58:0/55): ctl_process_done: 122 seconds
>(0:51:0/48): WRITE(10). CDB: 2a 08 00 10 ce 90 00 00 08 00 
>(0:51:0/48): Tag: 0x0042, type 1
>(0:51:0/48): ctl_process_done: 104 seconds
>(0:95:0/92): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:95:0/92): Tag: 0x0034, type 1
>(0:95:0/92): ctl_process_done: 98 seconds
>(0:65:0/62): WRITE(10). CDB: 2a 08 00 10 d4 f0 00 00 08 00 
>(0:65:0/62): Tag: 0x0014, type 1
>(0:65:0/62): ctl_process_done: 172 seconds
>(0:65:0/62): WRITE(10). CDB: 2a 08 00 10 d4 f0 00 00 08 00 
>(0:65:0/62): Tag: 0x002f, type 1
>(0:65:0/62): ctl_process_done: 104 seconds
>(0:58:0/55): WRITE(10). CDB: 2a 08 00 10 d2 58 00 00 08 00 
>(0:58:0/55): Tag: 0x006a, type 1
>(0:58:0/55): ctl_process_done: 91 seconds
>(0:43:0/40): WRITE(10). CDB: 2a 00 00 00 78 00 00 08 00 00 
>(0:43:0/40): Tag: 0x001c, type 1
>(0:43:0/40): ctl_process_done: 107 seconds
>(0:56:0/53): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:56:0/53): Tag: 0x0028, type 1
>(0:56:0/53): ctl_process_done: 91 seconds
>ctl_datamove: tag 0x0036 on (0:46:0) aborted
>(0:56:0/53): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:56:0/53): Tag: 0x0052, type 1
>(0:56:0/53): ctl_process_done: 91 seconds
>(0:76:0/73): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:76:0/73): Tag: 0x0016, type 1
>(0:76:0/73): ctl_process_done: 91 seconds
>(0:52:0/49): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:52:0/49): Tag: 0x0011, type 1
>(0:52:0/49): ctl_process_done: 94 seconds
>(0:64:0/61): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:64:0/61): Tag: 0x001a, type 1
>(0:64:0/61): ctl_process_done: 94 seconds
>(0:10:0/7): WRITE(10). CDB: 2a 08 00 10 d6 c8 00 00 08 00 
>(0:10:0/7): Tag: 0x0052, type 1
>(0:10:0/7): ctl_process_done: 120 seconds
>(0:87:0/84): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:87:0/84): Tag: 0x0070, type 1
>(0:60:0/57): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:87:0/84): ctl_process_done: 96 seconds
>(0:60:0/57): Tag: 0x0016, type 1
>(0:60:0/57): ctl_process_done: 97 seconds
>(0:5:0/2): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:5:0/2): Tag: 0x0010, type 1
>(0:5:0/2): ctl_process_done: 98 seconds
>(0:85:0/82): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:85:0/82): Tag: 0x0032, type 1
>(0:85:0/82): ctl_process_done: 103 seconds
>(0:38:0/35): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:7:0/4): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:7:0/4): WRITE(10). CDB: 2a 00 00 00 29 90 00 00 20 00 
>(0:38:0/35): Tag: 0x006d, type 1
>(0:7:0/4): Tag: 0x0066, type 1
>(0:7:0/4): Tag: 0x0010, type 1
>(0:38:0/35): ctl_process_done: 101 seconds
>(0:6:0/3): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:7:0/4): ctl_datamove: 101 seconds
>(0:6:0/3): Tag: 0x0041, type 1
>(0:7:0/4): ctl_process_done: 101 seconds
>(0:43:0/40): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>ctl_datamove: tag 0x0066 on (0:7:0) aborted
>(0:43:0/40): Tag: 0x002c, type 1
>(0:6:0/3): ctl_process_done: 96 seconds
>(0:12:0/9): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:12:0/9): Tag: 0x0047, type 1
>(0:43:0/40): ctl_process_done: 103 seconds
>(0:39:0/36): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:12:0/9): ctl_process_done: 103 seconds
>(0:92:0/89): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:39:0/36): Tag: 0x000a, type 1
>(0:39:0/36): ctl_process_done: 104 seconds
>(0:92:0/89): Tag: 0x0019, type 1
>(0:41:0/38): WRITE(10). CDB: 2a 08 00 10 cf 78 00 00 08 00 
>(0:92:0/89): ctl_process_done: 97 seconds
>(0:41:0/38): Tag: 0x001a, type 1
>(0:41:0/38): ctl_process_done: 136 seconds
>(0:13:0/10): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:13:0/10): Tag: 0x0047, type 1
>(0:13:0/10): ctl_process_done: 104 seconds
>(0:7:0/4): WRITE(10). CDB: 2a 00 00 00 29 90 00 00 20 00 
>(0:7:0/4): Tag: 0x0066, type 1
>(0:7:0/4): ctl_process_done: 102 seconds
>(0:47:0/44): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:47:0/44): Tag: 0x0005, type 1
>(0:47:0/44): ctl_process_done: 97 seconds
>(0:68:0/65): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:68:0/65): Tag: 0x003f, type 1
>(0:68:0/65): ctl_process_done: 98 seconds
>(0:77:0/74): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:77:0/74): WRITE(10). CDB: 2a 00 00 00 9a 00 00 00 20 00 
>(0:77:0/74): Tag: 0x0050, type 1
>(0:77:0/74): ctl_process_done: 112 seconds
>(0:77:0/74): Tag: 0x0075, type 1
>(0:77:0/74): ctl_datamove: 112 seconds
>ctl_datamove: tag 0x0075 on (0:77:0) aborted
>(0:77:0/74): WRITE(10). CDB: 2a 00 00 00 9a 00 00 00 20 00 
>(0:77:0/74): Tag: 0x0075, type 1
>(0:77:0/74): ctl_process_done: 112 seconds
>(0:40:0/37): WRITE(10). CDB: 2a 08 00 10 d5 a8 00 00 08 00 
>(0:40:0/37): Tag: 0x0008, type 1
>(0:40:0/37): ctl_process_done: 145 seconds
>(0:90:0/87): WRITE(10). CDB: 2a 08 00 10 c9 a8 00 00 08 00 
>(0:90:0/87): Tag: 0x003a, type 1
>(0:90:0/87): ctl_process_done: 145 seconds
>(0:75:0/72): WRITE(10). CDB: 2a 08 00 10 d2 90 00 00 08 00 
>(0:75:0/72): Tag: 0x004c, type 1
>(0:75:0/72): ctl_process_done: 146 seconds
>(0:22:0/19): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:22:0/19): WRITE(10). CDB: 2a 00 00 00 4a d8 00 00 20 00 
>(0:22:0/19): Tag: 0x0072, type 1
>(0:22:0/19): ctl_process_done: 105 seconds
>(0:22:0/19): Tag: 0x007c, type 1
>(0:22:0/19): ctl_datamove: 105 seconds
>ctl_datamove: tag 0x007c on (0:22:0) aborted
>(0:22:0/19): WRITE(10). CDB: 2a 00 00 00 4a d8 00 00 20 00 
>(0:22:0/19): Tag: 0x007c, type 1
>(0:22:0/19): ctl_process_done: 105 seconds
>(0:4:0/1): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:4:0/1): WRITE(10). CDB: 2a 00 00 00 91 60 00 00 20 00 
>(0:4:0/1): Tag: 0x0072, type 1
>(0:4:0/1): Tag: 0x002d, type 1
>(0:4:0/1): ctl_process_done: 114 seconds
>(0:4:0/1): ctl_datamove: 114 seconds
>ctl_datamove: tag 0x002d on (0:4:0) aborted
>(0:4:0/1): WRITE(10). CDB: 2a 00 00 00 91 60 00 00 20 00 
>(0:4:0/1): Tag: 0x002d, type 1
>(0:4:0/1): ctl_process_done: 114 seconds
>(0:32:0/29): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:32:0/29): Tag: 0x0002, type 1
>(0:32:0/29): ctl_process_done: 106 seconds
>(0:21:0/18): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:21:0/18): Tag: 0x0062, type 1
>(0:21:0/18): ctl_process_done: 112 seconds
>(0:62:0/59): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:83:0/80): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:62:0/59): Tag: 0x007a, type 1
>(0:83:0/80): Tag: 0x0016, type 1
>(0:83:0/80): ctl_process_done: 120 seconds
>(0:78:0/75): WRITE(10). CDB: 2a 00 00 00 26 70 00 00 20 00 
>(0:45:0/42): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:34:0/31): WRITE(10). CDB: 2a 00 00 00 26 28 00 00 20 00 
>(0:100:0/97): WRITE(10). CDB: 2a 00 00 00 25 18 00 00 20 00 
>(0:62:0/59): ctl_process_done: 116 seconds
>(0:100:0/97): Tag: 0x003b, type 1
>(0:73:0/70): WRITE(10). CDB: 2a 00 00 00 57 30 00 00 20 00 
>(0:78:0/75): Tag: 0x0036, type 1
>(0:72:0/69): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:45:0/42): Tag: 0x0074, type 1
>(0:34:0/31): Tag: 0x0064, type 1
>(0:72:0/69): Tag: 0x0067, type 1
>(0:100:0/97): ctl_datamove: 112 seconds
>(0:78:0/75): ctl_datamove: 112 seconds
>(0:45:0/42): ctl_process_done: 117 seconds
>(0:34:0/31): ctl_datamove: 112 seconds
>(0:72:0/69): ctl_process_done: 121 seconds
>(0:73:0/70): Tag: 0x004a, type 1
>(0:33:0/30): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>ctl_datamove: tag 0x0036 on (0:78:0) aborted
>ctl_datamove: tag 0x003b on (0:100:0) aborted
>(0:54:0/51): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>ctl_datamove: tag 0x0064 on (0:34:0) aborted
>(0:33:0/30): Tag: 0x0079, type 1
>(0:73:0/70): ctl_datamove: 111 seconds
>(0:54:0/51): Tag: 0x0023, type 1
>ctl_datamove: tag 0x004a on (0:73:0) aborted
>(0:33:0/30): ctl_process_done: 115 seconds
>(0:73:0/70): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:54:0/51): ctl_process_done: 113 seconds
>(0:73:0/70): Tag: 0x000c, type 1
>(0:73:0/70): ctl_process_done: 112 seconds
>(0:42:0/39): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:23:0/20): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:42:0/39): Tag: 0x0066, type 1
>(0:42:0/39): ctl_process_done: 123 seconds
>ctl_datamove: tag 0x0052 on (0:14:0) aborted
>ctl_datamove: tag 0x000e on (0:91:0) aborted
>(0:23:0/20): Tag: 0x0042, type 1
>(0:23:0/20): ctl_process_done: 110 seconds
>(0:31:0/28): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:78:0/75): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:78:0/75): Tag: 0x0011, type 1
>(0:31:0/28): Tag: 0x006f, type 1
>(0:31:0/28): ctl_process_done: 105 seconds
>(0:78:0/75): ctl_process_done: 114 seconds
>(0:34:0/31): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:55:0/52): WRITE(10). CDB: 2a 08 00 10 d2 d0 00 00 08 00 
>(0:34:0/31): Tag: 0x001e, type 1
>(0:34:0/31): ctl_process_done: 114 seconds
>(0:100:0/97): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:55:0/52): Tag: 0x0013, type 1
>(0:55:0/52): ctl_process_done: 105 seconds
>(0:100:0/97): Tag: 0x0050, type 1
>(0:100:0/97): ctl_process_done: 114 seconds
>(0:82:0/79): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:37:0/34): WRITE(10). CDB: 2a 08 00 10 d7 08 00 00 08 00 
>(0:82:0/79): Tag: 0x0023, type 1
>(0:37:0/34): Tag: 0x005f, type 1
>(0:82:0/79): ctl_process_done: 107 seconds
>(0:80:0/77): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:37:0/34): ctl_process_done: 103 seconds
>(0:49:0/46): WRITE(10). CDB: 2a 08 00 10 d6 78 00 00 08 00 
>(0:80:0/77): Tag: 0x0011, type 1
>(0:80:0/77): ctl_process_done: 107 seconds
>(0:74:0/71): WRITE(10). CDB: 2a 08 00 10 c6 48 00 00 08 00 
>(0:49:0/46): Tag: 0x003c, type 1
>(0:74:0/71): Tag: 0x0051, type 1
>(0:49:0/46): ctl_process_done: 99 seconds
>(0:74:0/71): ctl_process_done: 105 seconds
>(0:71:0/68): WRITE(10). CDB: 2a 08 00 10 ca 18 00 00 08 00 
>(0:48:0/45): WRITE(10). CDB: 2a 08 00 10 c9 18 00 00 08 00 
>(0:71:0/68): Tag: 0x0005, type 1
>(0:48:0/45): Tag: 0x007f, type 1
>(0:48:0/45): ctl_process_done: 105 seconds
>(0:28:0/25): WRITE(10). CDB: 2a 08 00 10 cf 90 00 00 08 00 
>(0:71:0/68): ctl_process_done: 92 seconds
>(0:28:0/25): Tag: 0x0052, type 1
>(0:73:0/70): WRITE(10). CDB: 2a 00 00 00 57 30 00 00 20 00 
>(0:28:0/25): ctl_process_done: 105 seconds
>(0:70:0/67): WRITE(10). CDB: 2a 08 00 10 cf 20 00 00 08 00 
>(0:73:0/70): Tag: 0x004a, type 1
>(0:73:0/70): ctl_process_done: 115 seconds
>(0:53:0/50): WRITE(10). CDB: 2a 08 00 10 c4 08 00 00 08 00 
>(0:70:0/67): Tag: 0x0001, type 1
>ctl_datamove: tag 0x007a on (0:102:0) aborted
>(0:53:0/50): Tag: 0x001e, type 1
>(0:70:0/67): ctl_process_done: 102 seconds
>(0:53:0/50): ctl_process_done: 93 seconds
>(0:29:0/26): WRITE(10). CDB: 2a 08 00 10 10 00 00 00 08 00 
>(0:50:0/47): WRITE(10). CDB: 2a 08 00 10 ce 30 00 00 08 00 
>(0:29:0/26): Tag: 0x004b, type 1
>(0:29:0/26): ctl_process_done: 97 seconds
>(0:50:0/47): Tag: 0x0010, type 1
>(0:50:0/47): ctl_process_done: 102 seconds
>(0:100:0/97): WRITE(10). CDB: 2a 00 00 00 25 18 00 00 20 00 
>(0:100:0/97): Tag: 0x003b, type 1
>(0:100:0/97): ctl_process_done: 116 seconds
>(0:78:0/75): WRITE(10). CDB: 2a 00 00 00 26 70 00 00 20 00 
>(0:78:0/75): Tag: 0x0036, type 1
>(0:78:0/75): ctl_process_done: 117 seconds
>(0:34:0/31): WRITE(10). CDB: 2a 00 00 00 26 28 00 00 20 00 
>(0:34:0/31): Tag: 0x0064, type 1
>(0:34:0/31): ctl_process_done: 117 seconds
>(0:20:0/17): WRITE(10). CDB: 2a 08 00 10 ce f8 00 00 08 00 
>(0:20:0/17): Tag: 0x0010, type 1
>(0:20:0/17): ctl_process_done: 94 seconds
>(0:14:0/11): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:14:0/11): Tag: 0x0040, type 1
>(0:14:0/11): ctl_process_done: 91 seconds
>(0:14:0/11): WRITE(10). CDB: 2a 00 00 00 9b 60 00 00 20 00 
>(0:14:0/11): Tag: 0x0052, type 1
>(0:14:0/11): ctl_process_done: 91 seconds
>(0:102:0/99): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 
>(0:102:0/99): Tag: 0x0053, type 1
>(0:102:0/99): ctl_process_done: 91 seconds
>(0:102:0/99): WRITE(10). CDB: 2a 00 00 00 7c 40 00 00 20 00 
>(0:102:0/99): Tag: 0x007a, type 1
>(0:102:0/99): ctl_process_done: 91 seconds
>(0:10:0/7): WRITE(10). CDB: 2a 08 00 10 d6 c8 00 00 08 00 
>(0:10:0/7): Tag: 0x002b, type 1
>(0:10:0/7): ctl_process_done: 97 seconds
>panic: Memory modified after free 0xfffff80003ba8b80(32) val=0 @ 0xfffff80003ba8b80
>
>cpuid = 5
>time = 1519077207
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe006210ee80
>kdb_backtrace() at kdb_backtrace+0x53/frame 0xfffffe006210ef50
>vpanic() at vpanic+0x268/frame 0xfffffe006210f020
>kproc_shutdown() at kproc_shutdown/frame 0xfffffe006210f080
>trash_ctor() at trash_ctor+0x6b/frame 0xfffffe006210f0c0
>uma_zalloc_arg() at uma_zalloc_arg+0x284/frame 0xfffffe006210f200
>bucket_alloc() at bucket_alloc+0x102/frame 0xfffffe006210f240
>uma_zfree_arg() at uma_zfree_arg+0x647/frame 0xfffffe006210f330
>uma_zfree() at uma_zfree+0x29/frame 0xfffffe006210f360
>ctl_free_beio() at ctl_free_beio+0x80/frame 0xfffffe006210f390
>ctl_complete_beio() at ctl_complete_beio+0x48/frame 0xfffffe006210f3b0
>ctl_be_block_move_done() at ctl_be_block_move_done+0x266/frame 0xfffffe006210f430
>cfiscsi_datamove_in() at cfiscsi_datamove_in+0x92d/frame 0xfffffe006210f510
>cfiscsi_datamove() at cfiscsi_datamove+0x28/frame 0xfffffe006210f530
>ctl_datamove() at ctl_datamove+0x498/frame 0xfffffe006210f7d0
>ctl_be_block_dispatch_zvol() at ctl_be_block_dispatch_zvol+0x6c2/frame 0xfffffe006210f8e0
>ctl_be_block_dispatch() at ctl_be_block_dispatch+0x626/frame 0xfffffe006210f9b0
>ctl_be_block_worker() at ctl_be_block_worker+0x68d/frame 0xfffffe006210fa40
>taskqueue_run_locked() at taskqueue_run_locked+0x246/frame 0xfffffe006210fab0
>taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe006210fb30
>fork_exit() at fork_exit+0x145/frame 0xfffffe006210fbb0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe006210fbb0
>--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>KDB: enter: panic
>
>------------------------------------------------------------------------
>kernel config
>
>options	CONFIG_AUTOGENERATED
>ident	MYKERNEL
>machine	amd64
>cpu	HAMMER
>makeoptions	WITH_CTF=1
>makeoptions	DEBUG=-g -O0
>options	KSTACK_PAGES=16
>options	KTR_ENTRIES=165536
>options	KTR_MASK=KTR_SPARE3
>options	KTR_COMPILE=KTR_SPARE3
>options	KTR
>options	BPF_JITTER
>options	XENHVM
>options	USB_DEBUG
>options	ATH_ENABLE_11N
>options	AH_AR5416_INTERRUPT_MITIGATION
>options	AH_SUPPORT_AR5416
>options	IEEE80211_SUPPORT_MESH
>options	IEEE80211_AMPDU_AGE
>options	IEEE80211_DEBUG
>options	IXL_IW
>options	SC_PIXEL_MODE
>options	VESA
>options	PCI_IOV
>options	PCI_HP
>options	ACPI_DMAR
>options	EARLY_AP_STARTUP
>options	SMP
>options	MALLOC_DEBUG_MAXZONES=8
>options	WITNESS_SKIPSPIN
>options	WITNESS
>options	INVARIANT_SUPPORT
>options	INVARIANTS
>options	DEADLKRES
>options	GDB
>options	FULL_BUF_TRACKING
>options	DDB
>options	BUF_TRACKING
>options	KDB_TRACE
>options	KDB
>options	RCTL
>options	RACCT_DEFAULT_TO_DISABLED
>options	RACCT
>options	INCLUDE_CONFIG_FILE
>options	DDB_CTF
>options	KDTRACE_HOOKS
>options	KDTRACE_FRAME
>options	MAC
>options	CAPABILITIES
>options	CAPABILITY_MODE
>options	AUDIT
>options	HWPMC_HOOKS
>options	KBD_INSTALL_CDEV
>options	PRINTF_BUFR_SIZE=128
>options	_KPOSIX_PRIORITY_SCHEDULING
>options	SYSVSEM
>options	SYSVMSG
>options	SYSVSHM
>options	STACK
>options	KTRACE
>options	SCSI_DELAY=5000
>options	COMPAT_FREEBSD11
>options	COMPAT_FREEBSD10
>options	COMPAT_FREEBSD9
>options	COMPAT_FREEBSD7
>options	COMPAT_FREEBSD6
>options	COMPAT_FREEBSD5
>options	COMPAT_FREEBSD4
>options	COMPAT_FREEBSD32
>options	GEOM_LABEL
>options	GEOM_RAID
>options	GEOM_PART_GPT
>options	PSEUDOFS
>options	PROCFS
>options	CD9660
>options	MSDOSFS
>options	NFS_ROOT
>options	NFSLOCKD
>options	NFSD
>options	NFSCL
>options	MD_ROOT
>options	QUOTA
>options	UFS_GJOURNAL
>options	UFS_DIRHASH
>options	UFS_ACL
>options	SOFTUPDATES
>options	FFS
>options	SCTP
>options	TCP_HHOOK
>options	TCP_OFFLOAD
>options	IPSEC_SUPPORT
>options	IPSEC
>options	INET6
>options	INET
>options	PREEMPTION
>options	SCHED_ULE
>options	NEW_PCIB
>options	GEOM_PART_MBR
>options	GEOM_PART_EBR_COMPAT
>options	GEOM_PART_EBR
>options	GEOM_PART_BSD
>device	isa
>device	mem
>device	io
>device	uart_ns8250
>device	cpufreq
>device	acpi
>device	pci
>device	fdc
>device	ahci
>device	ata
>device	mvs
>device	siis
>device	ahc
>device	ahd
>device	esp
>device	hptiop
>device	isp
>device	mpt
>device	mps
>device	mpr
>device	sym
>device	trm
>device	adv
>device	adw
>device	aic
>device	bt
>device	isci
>device	scbus
>device	ch
>device	da
>device	sa
>device	cd
>device	pass
>device	ses
>device	amr
>device	arcmsr
>device	ciss
>device	dpt
>device	hptmv
>device	hptnr
>device	hptrr
>device	hpt27xx
>device	iir
>device	ips
>device	mly
>device	twa
>device	tws
>device	aac
>device	aacp
>device	aacraid
>device	ida
>device	mfi
>device	mlx
>device	mrsas
>device	pmspcv
>device	twe
>device	nvme
>device	nvd
>device	atkbdc
>device	atkbd
>device	psm
>device	kbdmux
>device	vga
>device	splash
>device	sc
>device	vt
>device	vt_vga
>device	vt_efifb
>device	agp
>device	cbb
>device	pccard
>device	cardbus
>device	uart
>device	ppc
>device	ppbus
>device	lpt
>device	ppi
>device	puc
>device	bxe
>device	de
>device	em
>device	ix
>device	ixv
>device	ixl
>device	ixlv
>device	le
>device	ti
>device	txp
>device	vx
>device	miibus
>device	ae
>device	age
>device	alc
>device	ale
>device	bce
>device	bfe
>device	bge
>device	cas
>device	dc
>device	et
>device	fxp
>device	gem
>device	hme
>device	jme
>device	lge
>device	msk
>device	nfe
>device	nge
>device	pcn
>device	re
>device	rl
>device	sf
>device	sge
>device	sis
>device	sk
>device	ste
>device	stge
>device	tl
>device	tx
>device	vge
>device	vr
>device	wb
>device	xl
>device	wlan
>device	wlan_wep
>device	wlan_ccmp
>device	wlan_tkip
>device	wlan_amrr
>device	an
>device	ath
>device	ath_pci
>device	ath_hal
>device	ath_rate_sample
>device	ipw
>device	iwi
>device	iwn
>device	malo
>device	mwl
>device	ral
>device	wi
>device	wpi
>device	loop
>device	random
>device	padlock_rng
>device	rdrand_rng
>device	ether
>device	vlan
>device	tun
>device	md
>device	gif
>device	firmware
>device	bpf
>device	uhci
>device	ohci
>device	ehci
>device	xhci
>device	usb
>device	ukbd
>device	umass
>device	sound
>device	snd_cmi
>device	snd_csa
>device	snd_emu10kx
>device	snd_es137x
>device	snd_hda
>device	snd_ich
>device	snd_via8233
>device	mmc
>device	mmcsd
>device	sdhci
>device	virtio
>device	virtio_pci
>device	vtnet
>device	virtio_blk
>device	virtio_scsi
>device	virtio_balloon
>device	hyperv
>device	xenpci
>device	vmx
>device	netmap
>device	crypto
>
>------------------------------------------------------------------------
>ddb capture buffer
>
>

Actions: View

Attachments on bug 226064: 190824 | 191927