Bug 180001 - [zfs] [panic] Solaris(panic): zfs: allocating allocated segment
Summary: [zfs] [panic] Solaris(panic): zfs: allocating allocated segment
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: Unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-zfs-devel mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-06-26 12:10 UTC by skeletor
Modified: 2018-03-07 18:27 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description skeletor 2013-06-26 12:10:00 UTC
I have a problem pool, with which a have a loop kernel panic at boot process

panic: Solaris(panic): zfs: allocating allocated segment(offset=34630775808 size=4096)

If i disable zfs (in /etc/rc.conf, /boot/loader.conf) then boot continue without any errors. If after boot i try to run one of this command

#zpool list
#zpool status

i received a kernel panic.

Below a full chash report

Unread portion of the kernel message buffer:
panic: Solaris(panic): zfs: allocating allocated segment(offset=34630775808 size=4096)

cpuid = 0
KDB: stack backtrace:
#0 0xffffffff809916b6 at kdb_backtrace+0x66
#1 0xffffffff80958a9e at panic+0x1ce
#2 0xffffffff816d615b at vcmn_err+0x7b
#3 0xffffffff81c886aa at zfs_panic_recover+0x7a
#4 0xffffffff81c8aec4 at space_map_add+0xe4
#5 0xffffffff81c8b3b4 at space_map_load+0x1a4
#6 0xffffffff81c7567c at metaslab_activate+0xdc
#7 0xffffffff81c76432 at metaslab_alloc+0x7b2
#8 0xffffffff81cb036a at zio_dva_allocate+0x9a
#9 0xffffffff81cad063 at zio_execute+0xc3
#10 0xffffffff8099e145 at taskqueue_run_locked+0x85
#11 0xffffffff8099f0c6 at taskqueue_thread_loop+0x46
#12 0xffffffff8092733f at fork_exit+0x11f
#13 0xffffffff80c40e9e at fork_trampoline+0xe
Uptime: 50s
Dumping 508 out of 7909 MB:..4%..13%..23%..32%..41%..51%..63%..73%..82%..92%

Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtraceall.ko
Reading symbols from /boot/kernel/cyclic.ko...Reading symbols from /boot/kernel/cyclic.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/cyclic.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtrace.ko
Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtmalloc.ko
Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtnfscl.ko
Reading symbols from /boot/kernel/dtnfsclient.ko...Reading symbols from /boot/kernel/dtnfsclient.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtnfsclient.ko
Reading symbols from /boot/kernel/nfsclient.ko...Reading symbols from /boot/kernel/nfsclient.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nfsclient.ko
Reading symbols from /boot/kernel/nfs_common.ko...Reading symbols from /boot/kernel/nfs_common.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nfs_common.ko
Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /boot/kernel/fbt.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fbt.ko
Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fasttrap.ko
Reading symbols from /boot/kernel/lockstat.ko...Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/lockstat.ko
Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /boot/kernel/sdt.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/sdt.ko
Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /boot/kernel/systrace.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/systrace.ko
Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko
Reading symbols from /boot/kernel/profile.ko...Reading symbols from /boot/kernel/profile.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/profile.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_mppc.ko
Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/rc4.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff80958581 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff80958a77 in panic (fmt=0x1 <Address 0x1 out of bounds>)
    at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff816d615b in vcmn_err (ce=0, 
    fmt=0xffffffff81d325a0 "zfs: allocating allocated segment(offset=%llu size=%llu)\n", adx=0xffffff80974cf6a0)
    at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:54
#4  0xffffffff81c886aa in zfs_panic_recover (fmt=Variable "fmt" is not available.
)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c:1220
#5  0xffffffff81c8aec4 in space_map_add (sm=0xfffffe0009974d00, 
    start=34630775808, size=4096)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c:102
#6  0xffffffff81c8b3b4 in space_map_load (sm=0xfffffe0009974d00, 
    ops=0xffffffff81d3de20, maptype=Variable "maptype" is not available.
)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c:326
#7  0xffffffff81c7567c in metaslab_activate (msp=0xfffffe0009974800, 
    activation_weight=9223372036854775808)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:886
#8  0xffffffff81c76432 in metaslab_alloc (spa=0xfffffe0009ea6000, 
    mc=0xfffffe01f90791c0, psize=512, bp=0xfffffe01f93c3080, ndvas=2, 
    txg=3759540, hintbp=0x0, flags=0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1259
#9  0xffffffff81cb036a in zio_dva_allocate (zio=0xfffffe01f929ba50)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2170
#10 0xffffffff81cad063 in zio_execute (zio=0xfffffe01f929ba50)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1196
#11 0xffffffff8099e145 in taskqueue_run_locked (queue=0xfffffe0009d5fd80)
    at /usr/src/sys/kern/subr_taskqueue.c:308
#12 0xffffffff8099f0c6 in taskqueue_thread_loop (arg=Variable "arg" is not available.
)
    at /usr/src/sys/kern/subr_taskqueue.c:497
#13 0xffffffff8092733f in fork_exit (
    callout=0xffffffff8099f080 <taskqueue_thread_loop>, 
    arg=0xfffffe0009a28840, frame=0xffffff80974cfc40)
    at /usr/src/sys/kern/kern_fork.c:992
#14 0xffffffff80c40e9e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:602
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0xfffffe01f90b78b0 in ?? ()
#41 0xfffffe01f90b7470 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0xffffff80974cf600 in ?? ()
#44 0xffffff80974cf5a8 in ?? ()
#45 0xfffffe00024d4470 in ?? ()
#46 0xffffffff8098425d in sched_switch (td=0xffffffff812ee700, 
    newtd=0xfffffe0009a28840, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1921
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

------------------------------------------------------------------------
ps -axl

UID  PID PPID CPU PRI NI   VSZ RSS MWCHAN           STAT TT     TIME COMMAND
  0    0    0   0  -8  0     0   0 fffffe0009d61900 DLs  ??  0:00.02 [kernel]
  0    1    0   0  52  0  6276   0 wait             DLs  ??  0:00.00 [init]
  0    2    0   0 -16  0     0   0 ctl_work         DL   ??  0:00.00 [ctl_thrd]
  0    3    0   0 -16  0     0   0 pftm             DL   ??  0:00.00 [pfpurge]
  0    4    0   0 -16  0     0   0 waiting_         DL   ??  0:00.00 [sctp_iter
  0    5    0   0 -16  0     0   0 ccb_scan         DL   ??  0:00.00 [xpt_thrd]
  0    6    0   0 -16  0     0   0 psleep           DL   ??  0:00.00 [pagedaemo
  0    7    0   0 -16  0     0   0 psleep           DL   ??  0:00.00 [vmdaemon]
  0    8    0   0 155  0     0   0 pgzero           DL   ??  0:00.00 [pagezero]
  0    9    0   0 -16  0     0   0 psleep           DL   ??  0:00.00 [bufdaemon
  0   10    0   0 -16  0     0   0 audit_wo         DL   ??  0:00.00 [audit]
  0   11    0   0 155  0     0   0 -                RL   ??  0:14.44 [idle]
  0   12    0   0 -72  0     0   0 -                WL   ??  0:00.04 [intr]
  0   13    0   0  -8  0     0   0 -                DL   ??  0:00.04 [geom]
  0   14    0   0 -16  0     0   0 -                DL   ??  0:00.00 [yarrow]
  0   15    0   0 -68  0     0   0 -                DL   ??  0:00.00 [usb]
  0   16    0   0  16  0     0   0 syncer           DL   ??  0:00.00 [syncer]
  0   17    0   0 -16  0     0   0 vlruwt           DL   ??  0:00.00 [vnlru]
  0   18    0   0 -16  0     0   0 sdflush          DL   ??  0:00.00 [softdepfl
  0  129    1   0  52  0  9920   0 pause            Ds   ??  0:00.00 [adjkerntz
  0 1643    1   0  20  0 10376   0 select           Ds   ??  0:00.00 [devd]
  0 1800    1   0  20  0 12052   0 select           Ds   ??  0:00.00 [syslogd]
 53 1897    1   0  20  0 51224   0 kqread           Ds   ??  0:00.00 [named]
  0 1936    1   0  52  0 41564   0 select           Ds   ??  0:00.03 [mpd5]
  0 1972    0   0 -16  0     0   0 sleep            DL   ??  0:00.00 [ng_queue]
 26 1999    1   0  52  0 23124   0 select           Ds   ??  0:00.00 [exim-4.80
  0 2029    1   0  34  0 46744   0 select           Ds   ??  0:00.00 [sshd]
  0 2033    1   0  52  0 14128   0 nanslp           Ds   ??  0:00.00 [cron]
  0 2080    1   0  52  0 14504   0 wait             D    ??  0:00.00 [sh]
  0 2081    1   0  52  0  9912   0 piperd           D    ??  0:00.00 [logger]
  0 2082 2080   0  52  0  3776   0 nanslp           D    ??  0:00.00 [sleep]
  0 2084    1   0  20  0 41164   0 wait             Ds   ??  0:00.00 [login]
  0 2085    1   0  52  0 12052   0 ttyin            Ds+  ??  0:00.00 [getty]
  0 2086    1   0  52  0 12052   0 ttyin            Ds+  ??  0:00.00 [getty]
  0 2087    1   0  52  0 12052   0 ttyin            Ds+  ??  0:00.00 [getty]
  0 2088    1   0  52  0 12052   0 ttyin            Ds+  ??  0:00.00 [getty]
  0 2089    1   0  52  0 12052   0 ttyin            Ds+  ??  0:00.00 [getty]
  0 2090    1   0  52  0 12052   0 ttyin            Ds+  ??  0:00.00 [getty]
  0 2091    1   0  52  0 12052   0 ttyin            Ds+  ??  0:00.00 [getty]
  0 2092 2084   0  20  0 17532   0 pause            D    ??  0:00.00 [csh]
  0 2094 2092   0  43  0 35476   0 tx->tx_s         D+   ??  0:00.00 [zpool]
  0 2095    0   0  -8  0     0   0 zio->io_         DL   ??  0:00.00 [zfskern]

------------------------------------------------------------------------
vmstat -s

    89990 cpu context switches
    22775 device interrupts
     4274 software interrupts
   140816 traps
   190784 system calls
       20 kernel threads created
     2066  fork() calls
        9 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
      626 vnode pager pageins
     4855 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 pages reactivated
    60165 copy-on-write faults
      237 copy-on-write optimized faults
    48790 zero fill pages zeroed
        0 zero fill pages prezeroed
        1 intransit blocking page faults
   157145 total VM faults taken
        0 pages affected by kernel thread creation
  1050577 pages affected by  fork()
     4776 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
   176632 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
     6824 pages active
     8252 pages inactive
        0 pages in VM cache
    59163 pages wired down
  1885116 pages free
     4096 bytes per page
    25089 total name lookups
          cache hits (86% pos + 7% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
md_nvidia_data     0     0K       -       31  512
        mixer     3    12K       -        3  4096
  md_sii_data     0     0K       -       31  512
    acpi_perf     2     1K       -        2  128
CAM dev queue     8     1K       -        8  128
      CAM XPT   122   138K       -      311  16,32,64,128,1024,2048
       isadev     7     1K       -        7  128
         UART     3     2K       -        3  16,512,1024
         cdev     7     2K       -        7  256
        sigio     1     1K       -        1  64
     filedesc    49    50K       -     2106  16,32,64,128,512,1024,2048,4096
      kdtrace   298    67K       -     2491  64,256
         kenv    84    12K       -      100  16,32,64,128
       kqueue     2     9K       -       19  256,2048
    proc-args    22     2K       -      590  32,64,128,256
        hhook     2     1K       -        2  128
       pfsync     1     1K       -        1  512
      ithread    90    14K       -       90  32,128,256
       KTRACE   100    13K       -      100  128
     acpiintr     1     1K       -        1  64
       linker   354   908K       -      452  16,32,64,128,256,512,1024,2048,4096
        lockf    14     2K       -       90  64,128
   loginclass     2     1K       -        2  64
       ip6ndp    15     2K       -       16  64,128
       ip6opt     0     0K       -      198  256
         temp    47    30K       -     6066  16,32,64,128,256,512,1024,2048,4096
       devbuf 36192 94194K       -    36326  16,32,64,128,256,512,1024,2048,4096
       module   508    64K       -      508  128
     mtx_pool     2    16K       -        2  
          osd     3     1K       -        4  16,64
       USBdev    37     5K       -       37  64,128,256,512
     pmchooks     1     1K       -        1  128
          USB    59    39K       -       60  16,32,64,128,256,2048,4096
      subproc    94   194K       -     2148  512,4096
         proc     2    32K       -        2  
      session    18     3K       -       18  128
         pgrp    20     3K       -       20  128
         cred    36     6K       -     5188  64,256
      uidinfo     4     5K       -        6  128,4096
       plimit    12     3K       -      152  256
       acpica  5964   604K       -    64893  16,32,64,128,256,512,1024,2048,4096
     acpitask     1     2K       -        1  2048
    sysctltmp     0     0K       -      365  16,32,64,128,256,4096
    sysctloid  4457   219K       -     4562  16,32,64,128
       sysctl     0     0K       -      510  16,32,64
      tidhash     1    32K       -        1  
      callout     1   512K       -        1  
         umtx   516    65K       -      516  128
     p1003.1b     1     1K       -        1  16
         SWAP     2  2189K       -        2  64
       bus-sc    88   267K       -     3611  16,32,64,128,256,512,1024,2048,4096
          bus  1272   104K       -     6402  16,32,64,128,256,512,1024
      devstat    12    25K       -       12  32,4096
 eventhandler    97     8K       -       97  64,128
         kobj   335  1340K       -      666  4096
      Per-cpu     1     1K       -        1  32
    CAM queue    30     2K       -      152  16,32,256
         rman   246    30K       -      661  16,32,128
         sbuf     0     0K       -     1266  16,32,64,128,256,512,1024,2048,4096
      acpisem    24     3K       -       24  128
     ppbusdev     3     1K       -        3  256
        stack     0     0K       -        2  256
    taskqueue    47     5K       -       77  16,32,64,128,1024
       Unitno    15     1K       -      383  32,64
          iov     0     0K       -     2073  16,64,128,256,512
       select     8     1K       -        8  128
     ioctlops     1     8K       -     1893  16,32,64,128,256,512,1024,2048,4096
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1  
          tty    20    20K       -       22  1024,2048
         accf     2     1K       -        2  64
     mbuf_tag     0     0K       -      720  32,64
        shmfd     1     8K       -        1  
      entropy  1024    64K       -     1024  64
          pcb    24   157K       -      714  16,32,64,128,1024,2048,4096
       soname     5     1K       -     2318  16,32,128
     vfscache     1  2048K       -        1  
     vfs_hash     1  1024K       -        1  
      CAM SIM     8     2K       -        8  256
       vnodes     2     1K       -        2  256
       DEVFS1   131    66K       -      148  512
        mount    74     4K       -      177  16,32,64,128,256
  vnodemarker     0     0K       -       32  512
          BPF    13     2K       -       13  128
  ether_multi    33     2K       -       38  16,32,64
       ifaddr   136    33K       -      136  32,64,128,256,512,2048,4096
        ifnet    14    27K       -       14  128,2048
       DEVFS3   305    77K       -      326  256
        clone     8    32K       -        8  4096
       arpcom     2     1K       -        2  16
      lltable    39    17K       -       39  256,512
       DEVFS2   130     3K       -      130  16
   DEVFS_RULE    55    26K       -       55  64,512
        DEVFS    54     2K       -       55  16,128
     routetbl    35     5K       -      188  32,64,128,256,512
         igmp    13     4K       -       13  256
         ipid     2    24K       -        2  
     in_multi     2     1K       -        2  256
    sctp_iter     0     0K       -        3  256
     sctp_ifn     2     1K       -        2  128
     sctp_ifa     5     1K       -        5  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -        3  16
    hostcache     1    28K       -        1  
     syncache     1    96K       -        1  
    in6_multi    20     3K       -       20  32,256
          mld    13     2K       -       13  128
          rpc     2     1K       -        2  256
audit_evclass   179     6K       -      218  32
      jblocks     8     2K       -        8  128,256
     savedino     0     0K       -       24  256
        sbdep     0     0K       -        7  64
      jsegdep  2065   130K       -     2154  64
         jseg    16     2K       -       16  128
    jfreefrag     0     0K       -        3  128
      jnewblk     0     0K       -     2060  128
      jremref     0     0K       -       45  128
      jaddref     0     0K       -       46  128
     freework     7     1K       -       22  128
    newdirblk     4     1K       -        6  64
       dirrem     6     1K       -       33  128
        mkdir     4     1K       -       12  128
       diradd    13     2K       -       34  128
     freefile    17     2K       -       24  64
     freeblks     6     1K       -       21  128
     freefrag     0     0K       -        3  128
     indirdep     3     1K       -        6  128
       newblk  2030   764K       -     2061  256
    bmsafemap     7    10K       -       48  256
     inodedep    41  1044K       -       72  512
      pagedep    10   259K       -       17  256
  ufs_dirhash    39     8K       -       39  16,32,64,128,512
    ufs_mount    12    50K       -       13  512,4096
    vm_pgdata     2   129K       -        2  128
      memdesc     1     4K       -        1  4096
     atkbddev     2     1K       -        2  64
    pfs_nodes    21     6K       -       21  256
         GEOM    87    17K       -     1140  16,32,64,128,256,512,1024,2048
   CAM periph     8     2K       -       32  16,32,64,128,256
       ctlmem  5062 10113K       -     5062  128,2048
         hdaa     5    41K       -        5  1024,2048
         hdac     1     1K       -        1  1024
        hdacc     1     1K       -        1  32
       ctlblk   200  1600K       -      200  
      ramdisk     1  4096K       -        1  
      acpidev    36     3K       -       36  64
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
       kbdmux     7    18K       -        7  16,512,1024,2048
       feeder    19     2K       -       24  32,128
      ctlpool   532   142K       -      532  32,512
    raid_data     0     0K       -      186  32,128,256
      io_apic     1     2K       -        1  2048
          MCA     6     1K       -        6  128
          msi     4     1K       -        4  128
     nexusdev     3     1K       -        3  16
     pci_link    16     2K       -       16  16,32,128
       cyclic     8     1K       -        8  16,64,128
      solaris  1519 25596K       -     9962  16,32,64,128,256,512,1024,2048,4096
          fbt     1   256K       -        1  
 netgraph_msg     0     0K       -        5  64,128,256
netgraph_node     5     1K       -        7  128,256
netgraph_hook     0     0K       -        2  128
netgraph_sock     6     1K       -        9  32,128
netgraph_path     0     0K       -        4  16
netgraph_mppc     0     0K       -        1  1024
   kstat_data     4     1K       -        4  64
------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     209,      12,     209,   0,   0
UMA Zones:              640,      0,     209,       1,     209,   0,   0
UMA Slabs:              568,      0,    4365,     164,    6054,   0,   0
UMA RCntSlabs:          568,      0,     262,       4,     262,   0,   0
UMA Hash:               256,      0,      82,       8,      82,   0,   0
16 Bucket:              152,      0,      69,       6,      69,   0,   0
32 Bucket:              280,      0,      67,       3,      67,   0,   0
64 Bucket:              536,      0,      48,       1,      48,  57,   0
128 Bucket:            1048,      0,      91,       2,      91, 638,   0
VM OBJECT:              232,      0,    1204,      28,   25078,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 277791,     100,     179,   22165,   0,   0
MAP ENTRY:              120,      0,     559,     123,   57139,   0,   0
fakepg:                 120,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     348,      42,     348,   0,   0
16:                      16,      0,    3026,     334,   37397,   0,   0
32:                      32,      0,    2528,     199,   12778,   0,   0
64:                      64,      0,   23347,      61,   55907,   0,   0
128:                    128,      0,    8835,     648,   14855,   0,   0
256:                    256,      0,    3061,      89,    8664,   0,   0
512:                    512,      0,     876,      55,    7155,   0,   0
1024:                  1024,      0,      79,    1093,    6083,   0,   0
2048:                  2048,      0,    5368,      94,    5933,   0,   0
4096:                  4096,      0,     473,      48,    7428,   0,   0
Files:                   80,      0,      77,     103,    7930,   0,   0
TURNSTILE:              136,      0,     259,      41,     259,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1184,      0,      41,      10,    2095,   0,   0
THREAD:                1128,      0,     255,       3,     394,   0,   0
SLEEPQUEUE:              80,      0,     259,      31,     259,   0,   0
VMSPACE:                392,      0,      22,      28,    2077,   0,   0
cpuset:                  72,      0,     223,     127,     361,   0,   0
cyclic_id_cache:         64,      0,       0,       0,       0,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,     256,     268,    1114,   0,   0
mbuf:                   256,      0,       1,     267,    1544,   0,   0
mbuf_cluster:          2048,  33792,     512,      12,     512,   0,   0
mbuf_jumbo_page:       4096,  16896,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216, 300000,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 128000,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
dtrace_state_cache:    4096,      0,       0,       0,       0,   0,   0
g_bio:                  232,      0,       0,     176,   18939,   0,   0
ttyinq:                 160,      0,     120,      24,     255,   0,   0
ttyoutq:                256,      0,      64,      26,     136,   0,   0
ata_request:            328,      0,       0,       0,       0,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
VNODE:                  480,      0,     667,      13,     693,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,      36,   11437,   0,   0
S VFS Cache:            108,      0,     645,      81,    1457,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,       0,       0,       0,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,      54,       2,      54,   0,   0
NFSMOUNT:               632,      0,       0,       0,       0,   0,   0
NFSNODE:                656,      0,       0,       0,       0,   0,   0
Mountpoints:            792,      0,       6,       9,       6,   0,   0
pipe:                   728,      0,       5,      25,    1488,   0,   0
ksiginfo:               112,      0,      41,    1015,      46,   0,   0
itimer:                 344,      0,       0,       0,       0,   0,   0
pfsrctrpl:              152,  10000,       0,       0,       0,   0,   0
pfrulepl:               936,      0,       1,       7,       1,   0,   0
pfstatepl:              288,  10010,       0,       0,       0,   0,   0
pfstatekeypl:           288,      0,       0,       0,       0,   0,   0
pfstateitempl:          288,      0,       0,       0,       0,   0,   0
pfaltqpl:               240,      0,       0,       0,       0,   0,   0
pfpooladdrpl:            88,      0,       1,      83,       1,   0,   0
pfrktable:             1296,   1002,       0,       0,       0,   0,   0
pfrkentry:              160, 200016,       0,       0,       0,   0,   0
pfrkcounters:            64,      0,       0,       0,       0,   0,   0
pffrent:                 32,   5050,       0,       0,       0,   0,   0
pffrag:                  80,      0,       0,       0,       0,   0,   0
pffrcache:               80,  10035,       0,       0,       0,   0,   0
pffrcent:                24,  50022,       0,       0,       0,   0,   0
pfstatescrub:            40,      0,       0,       0,       0,   0,   0
pfiaddrpl:              120,      0,       0,       0,       0,   0,   0
pfospfen:               112,      0,     700,      26,     700,   0,   0
pfosfp:                  40,      0,     410,      94,     410,   0,   0
KNOTE:                  128,      0,       8,      79,    2936,   0,   0
socket:                 680,  33792,      25,      47,     966,   0,   0
ipq:                     56,   1071,       0,       0,       0,   0,   0
udp_inpcb:              392,  33800,       3,      47,     909,   0,   0
udpcb:                   16,  33936,       3,     333,     909,   0,   0
tcp_inpcb:              392,  33800,       8,      22,      12,   0,   0
tcpcb:                  976,  33792,       8,       4,      12,   0,   0
tcptw:                   72,   6800,       0,       0,       0,   0,   0
syncache:               152,  15375,       0,       0,       0,   0,   0
hostcache:              136,  15372,       0,       0,       0,   0,   0
tcpreass:                40,   2184,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
sctp_ep:               1376,  33792,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     216,       4,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  33800,       0,       0,       0,   0,   0
unpcb:                  240,  33792,       8,      40,      32,   0,   0
rtentry:                200,      0,      17,      40,      17,   0,   0
pfsync:                  88,      0,       0,       0,       0,   0,   0
selfd:                   56,      0,      29,     160,    3889,   0,   0
SWAPMETA:               288, 116519,       0,       0,       0,   0,   0
FFS inode:              168,      0,     629,      31,     653,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,     629,      46,     653,   0,   0
NetGraph items:          72,   4118,       0,      87,       7,   0,   0
NetGraph data items:     72,    522,       0,      87,       4,   0,   0
taskq_zone:              48,      0,       0,     216,       8,   0,   0
zio_cache:              880,      0,       8,    1160,    3552,   0,   0
zio_link_cache:          48,      0,       6,    2154,    5326,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,       0,       0,       0,   0,   0
dnode_t:                856,      0,     132,      12,     208,   0,   0
dmu_buf_impl_t:         224,      0,     114,      39,     177,   0,   0
arc_buf_hdr_t:          216,      0,     129,      33,     197,   0,   0
arc_buf_t:              104,      0,     129,      51,     232,   0,   0
zil_lwb_cache:          192,      0,       0,       0,       0,   0,   0
zfs_znode_cache:        400,      0,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq16: hdac0 ohci0+                  184          4
irq17: ehci0                           2          0
irq18: ohci2 ohci3+                    4          0
irq22: ahci0                        5777        148
irq256: hpet0:t0                   12537        321
irq257: hpet0:t1                    3810         97
irq259: re0                          461         11
Total                              22775        583

------------------------------------------------------------------------
pstat -T

 77/16424 files
0M/10239M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/ada0p2       20971264        0 20971264     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
            ada0             ada1             ada2             cpu
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
 74.49 130  9.49  30.76   7  0.21  30.46   7  0.21   1  0  6  0 92

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
        msgmax:        16384    (max characters in a message)
        msgmni:           40    (# of message queues)
        msgmnb:         2048    (max characters in a message queue)
        msgtql:           40    (max # of messages in system)
        msgssz:            8    (size of a message segment)
        msgseg:         2048    (# of message segments in system)

shminfo:
        shmmax:    536870912    (max shared memory segment size)
        shmmin:            1    (min shared memory segment size)
        shmmni:          192    (max number of shared memory identifiers)
        shmseg:          128    (max shared memory segments per process)
        shmall:       131072    (max amount of shared memory in pages)

seminfo:
        semmni:           50    (# of semaphore identifiers)
        semmns:          340    (# of semaphores in system)
        semmnu:          150    (# of undo structures in system)
        semmsl:          340    (max # of semaphores per id)
        semopm:          100    (max # of operations per semop call)
        semume:           50    (max # of undo entries per process)
        semusz:          632    (size in bytes of undo structure)
        semvmx:        32767    (semaphore maximum value)
        semaem:        16384    (adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
        0 packets sent
                0 data packets (0 bytes)
                0 data packets (0 bytes) retransmitted
                0 data packets unnecessarily retransmitted
                0 resends initiated by MTU discovery
                0 ack-only packets (0 delayed)
                0 URG only packets
                0 window probe packets
                0 window update packets
                0 control packets
        4 packets received
                0 acks (for 0 bytes)
                0 duplicate acks
                0 acks for unsent data
                0 packets (0 bytes) received in-sequence
                0 completely duplicate packets (0 bytes)
                0 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                0 out-of-order packets (0 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                0 window update packets
                0 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
                0 discarded due to memory problems
        0 connection requests
        0 connection accepts
        0 bad connection attempts
        0 listen queue overflows
        0 ignored RSTs in the windows
        0 connections established (including accepts)
        4 connections closed (including 0 drops)
                0 connections updated cached RTT on close
                0 connections updated cached RTT variance on close
                0 connections updated cached ssthresh on close
        0 embryonic connections dropped
        0 segments updated rtt (of 0 attempts)
        0 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 Connections (fin_wait_2) dropped because of timeout
        0 keepalive timeouts
                0 keepalive probes sent
                0 connections dropped by keepalive
        0 correct ACK header predictions
        0 correct data packet header predictions
        0 syncache entries added
                0 retransmitted
                0 dupsyn
                0 dropped
                0 completed
                0 bucket overflow
                0 cache overflow
                0 reset
                0 stale
                0 aborted
                0 badack
                0 unreach
                0 zone failures
        0 cookies sent
        0 cookies received
        0 hostcache entries added
                0 bucket overflow
        0 SACK recovery episodes
        0 segment rexmits in SACK recovery episodes
        0 byte rexmits in SACK recovery episodes
        0 SACK options (SACK blocks) received
        0 SACK options (SACK blocks) sent
        0 SACK scoreboard overflow
        0 packets with ECN CE bit set
        0 packets with ECN ECT(0) bit set
        0 packets with ECN ECT(1) bit set
        0 successful ECN handshakes
        0 times ECN reduced the congestion window
udp:
        177 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        1 dropped due to no socket
        130 broadcast/multicast datagrams undelivered
        0 dropped due to full socket buffers
        0 not for hashed pcb
        46 delivered
        507 datagrams output
        0 times multicast source filter matched
ip:
        193 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets reassembled ok
        191 packets for this host
        2 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        0 packets not forwardable
        0 packets received for unknown multicast group
        0 redirects sent
        522 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        1 call to icmp_error
        0 errors not generated in response to an icmp message
        Output histogram:
                echo reply: 10
                destination unreachable: 1
        0 messages with bad code fields
        0 messages less than the minimum length
        0 messages with bad checksum
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                destination unreachable: 2
                echo: 10
        10 message responses generated
        0 invalid return addresses
        0 no return routes
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with wrong TTL
        0 messages received with bad checksum
        0 V1/V2 membership queries received
        0 V3 membership queries received
        0 membership queries received with invalid field(s)
        0 general queries received
        0 group queries received
        0 group-source queries received
        0 group-source queries dropped
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 V3 reports received without Router Alert
        0 membership reports sent
pfsync:
        0 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                0 failed state lookup/inserts
        0 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                0 send error
arp:
        14 ARP requests sent
        3 ARP replies sent
        54 ARP requests received
        9 ARP replies received
        63 ARP packets received
        462 total packets dropped due to no ARP entry
        0 ARP entrys timed out
        0 Duplicate IPs seen
ip6:
        0 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        0 packets for this host
        0 packets forwarded
        0 packets not forwardable
        0 redirects sent
        5 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        396 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        0 multicast packets which we don't join
        Mbuf statistics:
                0 one mbuf
                0 one ext mbuf
                0 two or more ext mbuf
        0 packets whose headers are not contiguous
        0 tunneling packets that can't find gif
        0 packets discarded because of too many headers
        0 failures of source address selection
        Source addresses selection rule applied:
icmp6:
        0 calls to icmp6_error
        0 errors not generated in response to an icmp6 message
        0 errors not generated because of rate limitation
        Output histogram:
                neighbor solicitation: 1
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 bad redirect messages
        0 path MTU changes
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output

------------------------------------------------------------------------
netstat -m

257/535/792 mbufs in use (current/cache/total)
244/280/524/33792 mbuf clusters in use (current/cache/total/max)
256/268 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/16896 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/300000 9k jumbo clusters in use (current/cache/total/max)
0/0/0/128000 16k jumbo clusters in use (current/cache/total/max)
552K/693K/1246K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
re0    1500 <Link#1>      f4:6d:04:29:a0:d8      226     0     0       45     0     0    0 
re0    1500 10.5.5.0      backup                  56     -     -      490     -     -    - 
re0    1500 fe80::f66d:4f fe80::f66d:4ff:fe        0     -     -        2     -     -    - 
usbus     0 <Link#2>                               0     0     0        0     0     0    0 
usbus     0 <Link#3>                               0     0     0        0     0     0    0 
usbus     0 <Link#4>                               0     0     0        0     0     0    0 
usbus     0 <Link#5>                               0     0     0        0     0     0    0 
usbus     0 <Link#6>                               0     0     0        0     0     0    0 
usbus     0 <Link#7>                               0     0     0        0     0     0    0 
sk0*   1500 <Link#8>      00:21:91:8e:84:91        0     0     0        0     0     0    0 
usbus     0 <Link#9>                               0     0     0        0     0     0    0 
plip0  1500 <Link#10>                              0     0     0        0     0     0    0 
pflog 33152 <Link#11>                              0     0     0        0     0     0    0 
pfsyn  1500 <Link#12>                              0     0     0        0     0     0    0 
lo0   16384 <Link#13>                             32     0     0       32     0     0    0 
lo0   16384 localhost     ::1                      0     -     -        0     -     -    - 
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 
lo0   16384 your-net      localhost               32     -     -       32     -     -    - 

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            10.5.5.240         UGS         0      455    re0
10.5.5.0/24        link#1             U           0       35    re0
10.5.5.6           link#1             UHS         0        0    lo0
127.0.0.1          link#13            UH          0       32    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#13                       UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%re0/64                     link#1                        U           re0
fe80::f66d:4ff:fe29:a0d8%re0      link#1                        UHS         lo0
fe80::%lo0/64                     link#13                       U           lo0
fe80::1%lo0                       link#13                       UHS         lo0
ff01::%re0/32                     fe80::f66d:4ff:fe29:a0d8%re0  U           re0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%re0/32                     fe80::f66d:4ff:fe29:a0d8%re0  U           re0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffffe0009e537a0 tcp4       0      0 *.22               *.*                LISTEN
fffffe0009e53b70 tcp6       0      0 *.22               *.*                LISTEN
fffffe0009e68000 tcp4       0      0 *.25               *.*                LISTEN
fffffe0009e683d0 tcp6       0      0 *.25               *.*                LISTEN
fffffe0009e687a0 tcp6       0      0 ::1.953            *.*                LISTEN
fffffe0009e527a0 tcp4       0      0 127.0.0.1.953      *.*                LISTEN
fffffe0009e52000 tcp4       0      0 127.0.0.1.53       *.*                LISTEN
fffffe0009e523d0 tcp4       0      0 10.5.5.6.53        *.*                LISTEN
fffffe0009b36930 udp4       0      0 10.5.5.6.1701      *.*                
fffffe0009b36dc8 udp4       0      0 127.0.0.1.53       *.*                
fffffe0009b37000 udp4       0      0 10.5.5.6.53        *.*                
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffffe0009c37c30 stream      0      0 fffffe0009b6fb40        0        0        0 /var/run/devd.pipe
fffffe0009c373c0 dgram       0      0        0 fffffe0009c245a0        0 fffffe0009c242d0
fffffe0009c242d0 dgram       0      0        0 fffffe0009c245a0        0 fffffe0009c37690
fffffe0009c37690 dgram       0      0        0 fffffe0009c245a0        0        0
fffffe0009c243c0 dgram       0      0 fffffe0009d38d20        0        0        0 /var/named/var/run/log
fffffe0009c244b0 dgram       0      0 fffffe0009d53d20        0        0        0 /var/run/log
fffffe0009c245a0 dgram       0      0 fffffe0009d54000        0 fffffe0009c373c0        0 /var/run/logpriv
fffffe0009c24690 dgram       0      0 fffffe0009ad1780        0        0        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address         
tcp4  0/0/128        *.ssh                  
tcp6  0/0/128        *.ssh                  
tcp4  0/0/20         *.smtp                 
tcp6  0/0/20         *.smtp                 
tcp6  0/0/128        localhost.rndc         
tcp4  0/0/128        localhost.rndc         
tcp4  0/0/3          localhost.domain       
tcp4  0/0/3          backup.domain          
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     zfskern     2095 root /             2 drwxr-xr-x    1024  r
root     zfskern     2095   wd /             2 drwxr-xr-x    1024  r
root     zpool       2094 root /             2 drwxr-xr-x    1024  r
root     zpool       2094   wd /         32900 drwxr-xr-x     512  r
root     zpool       2094 text /         66069 -r-xr-xr-x   92480  r
root     zpool       2094 ctty /dev         56 crw-------   ttyv0 rw
root     zpool       2094    0 /dev         56 crw-------   ttyv0 rw
root     zpool       2094    1 /dev         56 crw-------   ttyv0 rw
root     zpool       2094    2 /dev         56 crw-------   ttyv0 rw
root     zpool       2094    3 /dev        141 crw-rw-rw-     zfs rw
root     zpool       2094    4 /dev         20 crw-rw-rw-    zero  r
root     zpool       2094    5 /         32932 -rw-r--r--       0  r
root     csh         2092 root /             2 drwxr-xr-x    1024  r
root     csh         2092   wd /         32900 drwxr-xr-x     512  r
root     csh         2092 text /         66038 -r-xr-xr-x  382264  r
root     csh         2092 ctty /dev         56 crw-------   ttyv0 rw
root     csh         2092   15 /dev         56 crw-------   ttyv0 rw
root     csh         2092   16 /dev         56 crw-------   ttyv0 rw
root     csh         2092   17 /dev         56 crw-------   ttyv0 rw
root     csh         2092   18 /dev         56 crw-------   ttyv0 rw
root     csh         2092   19 /dev         56 crw-------   ttyv0 rw
root     getty       2091 root /             2 drwxr-xr-x    1024  r
root     getty       2091   wd /             2 drwxr-xr-x    1024  r
root     getty       2091 text /usr     4594020 -r-xr-xr-x   28040  r
root     getty       2091 ctty /dev         63 crw-------   ttyv7 rw
root     getty       2091    0 /dev         63 crw-------   ttyv7 rw
root     getty       2091    1 /dev         63 crw-------   ttyv7 rw
root     getty       2091    2 /dev         63 crw-------   ttyv7 rw
root     getty       2090 root /             2 drwxr-xr-x    1024  r
root     getty       2090   wd /             2 drwxr-xr-x    1024  r
root     getty       2090 text /usr     4594020 -r-xr-xr-x   28040  r
root     getty       2090 ctty /dev         62 crw-------   ttyv6 rw
root     getty       2090    0 /dev         62 crw-------   ttyv6 rw
root     getty       2090    1 /dev         62 crw-------   ttyv6 rw
root     getty       2090    2 /dev         62 crw-------   ttyv6 rw
root     getty       2089 root /             2 drwxr-xr-x    1024  r
root     getty       2089   wd /             2 drwxr-xr-x    1024  r
root     getty       2089 text /usr     4594020 -r-xr-xr-x   28040  r
root     getty       2089 ctty /dev         61 crw-------   ttyv5 rw
root     getty       2089    0 /dev         61 crw-------   ttyv5 rw
root     getty       2089    1 /dev         61 crw-------   ttyv5 rw
root     getty       2089    2 /dev         61 crw-------   ttyv5 rw
root     getty       2088 root /             2 drwxr-xr-x    1024  r
root     getty       2088   wd /             2 drwxr-xr-x    1024  r
root     getty       2088 text /usr     4594020 -r-xr-xr-x   28040  r
root     getty       2088 ctty /dev         60 crw-------   ttyv4 rw
root     getty       2088    0 /dev         60 crw-------   ttyv4 rw
root     getty       2088    1 /dev         60 crw-------   ttyv4 rw
root     getty       2088    2 /dev         60 crw-------   ttyv4 rw
root     getty       2087 root /             2 drwxr-xr-x    1024  r
root     getty       2087   wd /             2 drwxr-xr-x    1024  r
root     getty       2087 text /usr     4594020 -r-xr-xr-x   28040  r
root     getty       2087 ctty /dev         59 crw-------   ttyv3 rw
root     getty       2087    0 /dev         59 crw-------   ttyv3 rw
root     getty       2087    1 /dev         59 crw-------   ttyv3 rw
root     getty       2087    2 /dev         59 crw-------   ttyv3 rw
root     getty       2086 root /             2 drwxr-xr-x    1024  r
root     getty       2086   wd /             2 drwxr-xr-x    1024  r
root     getty       2086 text /usr     4594020 -r-xr-xr-x   28040  r
root     getty       2086 ctty /dev         58 crw-------   ttyv2 rw
root     getty       2086    0 /dev         58 crw-------   ttyv2 rw
root     getty       2086    1 /dev         58 crw-------   ttyv2 rw
root     getty       2086    2 /dev         58 crw-------   ttyv2 rw
root     getty       2085 root /             2 drwxr-xr-x    1024  r
root     getty       2085   wd /             2 drwxr-xr-x    1024  r
root     getty       2085 text /usr     4594020 -r-xr-xr-x   28040  r
root     getty       2085 ctty /dev         57 crw-------   ttyv1 rw
root     getty       2085    0 /dev         57 crw-------   ttyv1 rw
root     getty       2085    1 /dev         57 crw-------   ttyv1 rw
root     getty       2085    2 /dev         57 crw-------   ttyv1 rw
root     login       2084 root /             2 drwxr-xr-x    1024  r
root     login       2084   wd /         32900 drwxr-xr-x     512  r
root     login       2084 text /usr     5304431 -r-sr-xr-x   25096  r
root     login       2084 ctty /dev         56 crw-------   ttyv0 rw
root     login       2084    0 /dev         56 crw-------   ttyv0 rw
root     login       2084    1 /dev         56 crw-------   ttyv0 rw
root     login       2084    2 /dev         56 crw-------   ttyv0 rw
root     login       2084    3* local dgram fffffe0009c373c0 <-> fffffe0009c245a0
root     sleep       2082 root /             2 drwxr-xr-x    1024  r
root     sleep       2082   wd /             2 drwxr-xr-x    1024  r
root     sleep       2082 text /         66080 -r-xr-xr-x    5512  r
root     sleep       2082    0 /dev         19 crw-rw-rw-    null  r
root     sleep       2082    1* pipe fffffe0009d2fcb8 <-> fffffe0009d2fb60      0 rw
root     sleep       20
Comment 1 skeletor 2013-06-26 14:07:45 UTC
This is only FreeBSD bug, because from Solaris i have successfully 
accessed to this pool.
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2013-06-27 00:10:55 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-zfs-devel

Over to maintainer(s).
Comment 3 skeletor 2013-12-12 13:54:58 UTC
This can happens with disk in pool, that have different buffer size. 
After replace such disk in pool - pool started normal work.
So, this is a bug.
Comment 4 emz 2017-12-01 09:28:30 UTC
Same thing on 11.0-STABLE r315957.
12 panics in a row on during ~ one month.

I have crashdumps in case someone needs them.

Typical bt:

savecore: reboot after panic:                                                                                                               Solaris(panic): zfs: allocating allocated segment(offset=964258332672 size=16384)
Nov 30 03:58:03 san savecore: reboot after panic:                                                                                                               Solaris(panic): zfs: allocating allocated segment(offset=964258332672 size=16384)
savecore: writing core to /var/crash/vmcore.9
Writing crash summary to /var/crash/core.txt.9.
Clearing /tmp (X related).
Starting ctld.
NFS on reserved port only=YES
NFSv4 is disabled
Starting rpcbind.
/etc/rc.d/mountd: WARNING: /etc/exports is not readable.
Starting mountd.
Starting nfsd.
Starting statd.
Starting lockd.
Processing uwsgi profile: api
Starting uwsgi.
[uWSGI] getting INI configuration from /usr/local/etc/uwsgi_api.ini
Processing uwsgi profile: zfsreplica
Starting uwsgi.
[uWSGI] getting INI configuration from /usr/local/etc/uwsgi_zfsreplica.ini
Updating motd:.
Mounting late filesystems:.
Starting ntpd.
Performing sanity check on nginx configuration:
Nov 30 03:58:32 san ntpd[1083]: restrict: ignoring line 21, address/host 'fd00::/16' unusable.
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
Starting bird.
Configuring vt: blanktime.
Performing sanity check on sshd configuration.
Starting sshd.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.
Starting background file system checks in 60 seconds.

Thu Nov 30 03:58:33 +05 2017
Nov 30 04:51:35 san ctld[2278]: 192.168.1.72 (iqn.2016-04.net.playkey:test-kvm): read: Connection reset by peer
Nov 30 11:07:02 san ctld[2537]: 192.168.1.72 (iqn.2016-04.net.playkey:test-kvm): read: Connection reset by peer
WARNING: 192.168.1.57 (iqn.2010-04.org.ipxe:03000200-0400-0500-0006-000700080009): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.1.57 (iqn.2010-04.org.ipxe:03000200-0400-0500-0006-000700080009): no ping reply (NOP-Out) after 5 seconds; dropping connection
Nov 30 17:29:48 san su: failed to get password: authentication error
Nov 30 17:29:50 san su: emz to root on /dev/pts/1
WARNING: 192.168.1.57 (iqn.2010-04.org.ipxe:03000200-0400-0500-0006-000700080009): no ping reply (NOP-Out) after 5 seconds; dropping connection
panic: Solaris(panic): zfs: allocating allocated segment(offset=1102832721920 size=16384)

cpuid = 4
KDB: stack backtrace:
#0 0xffffffff80a8c7c7 at kdb_backtrace+0x67
#1 0xffffffff80a49ae6 at vpanic+0x186
#2 0xffffffff80a49953 at panic+0x43
#3 0xffffffff82432192 at vcmn_err+0xc2
#4 0xffffffff821a4b1a at zfs_panic_recover+0x5a
#5 0xffffffff8218e035 at range_tree_add+0x235
#6 0xffffffff8218cb86 at metaslab_free_dva+0x276
#7 0xffffffff8218cc31 at metaslab_free+0x81
#8 0xffffffff821cfbda at zio_dva_free+0x1a
#9 0xffffffff821ccb8d at zio_execute+0x23d
#10 0xffffffff80a9d447 at taskqueue_run_locked+0x117
#11 0xffffffff80a9e548 at taskqueue_thread_loop+0xc8
#12 0xffffffff80a0dd05 at fork_exit+0x85
#13 0xffffffff80e92f6e at fork_trampoline+0xe
Uptime: 13h42m11s
Dumping 2638 out of 24531 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

This is also related to zfs corruption on the pool, which happens after these panics. May be this is somehow related to the fact that this host runs iSCSI target, probably panics occur when working with specific data which iSCSI target serves.