I have a problem pool, with which a have a loop kernel panic at boot process panic: Solaris(panic): zfs: allocating allocated segment(offset=34630775808 size=4096) If i disable zfs (in /etc/rc.conf, /boot/loader.conf) then boot continue without any errors. If after boot i try to run one of this command #zpool list #zpool status i received a kernel panic. Below a full chash report Unread portion of the kernel message buffer: panic: Solaris(panic): zfs: allocating allocated segment(offset=34630775808 size=4096) cpuid = 0 KDB: stack backtrace: #0 0xffffffff809916b6 at kdb_backtrace+0x66 #1 0xffffffff80958a9e at panic+0x1ce #2 0xffffffff816d615b at vcmn_err+0x7b #3 0xffffffff81c886aa at zfs_panic_recover+0x7a #4 0xffffffff81c8aec4 at space_map_add+0xe4 #5 0xffffffff81c8b3b4 at space_map_load+0x1a4 #6 0xffffffff81c7567c at metaslab_activate+0xdc #7 0xffffffff81c76432 at metaslab_alloc+0x7b2 #8 0xffffffff81cb036a at zio_dva_allocate+0x9a #9 0xffffffff81cad063 at zio_execute+0xc3 #10 0xffffffff8099e145 at taskqueue_run_locked+0x85 #11 0xffffffff8099f0c6 at taskqueue_thread_loop+0x46 #12 0xffffffff8092733f at fork_exit+0x11f #13 0xffffffff80c40e9e at fork_trampoline+0xe Uptime: 50s Dumping 508 out of 7909 MB:..4%..13%..23%..32%..41%..51%..63%..73%..82%..92% Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtraceall.ko Reading symbols from /boot/kernel/cyclic.ko...Reading symbols from /boot/kernel/cyclic.ko.symbols...done. done. Loaded symbols for /boot/kernel/cyclic.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /boot/kernel/dtrace.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtrace.ko Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtmalloc.ko Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtnfscl.ko Reading symbols from /boot/kernel/dtnfsclient.ko...Reading symbols from /boot/kernel/dtnfsclient.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtnfsclient.ko Reading symbols from /boot/kernel/nfsclient.ko...Reading symbols from /boot/kernel/nfsclient.ko.symbols...done. done. Loaded symbols for /boot/kernel/nfsclient.ko Reading symbols from /boot/kernel/nfs_common.ko...Reading symbols from /boot/kernel/nfs_common.ko.symbols...done. done. Loaded symbols for /boot/kernel/nfs_common.ko Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /boot/kernel/fbt.ko.symbols...done. done. Loaded symbols for /boot/kernel/fbt.ko Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. done. Loaded symbols for /boot/kernel/fasttrap.ko Reading symbols from /boot/kernel/lockstat.ko...Reading symbols from /boot/kernel/lockstat.ko.symbols...done. done. Loaded symbols for /boot/kernel/lockstat.ko Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /boot/kernel/sdt.ko.symbols...done. done. Loaded symbols for /boot/kernel/sdt.ko Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /boot/kernel/systrace.ko.symbols...done. done. Loaded symbols for /boot/kernel/systrace.ko Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. done. Loaded symbols for /boot/kernel/systrace_freebsd32.ko Reading symbols from /boot/kernel/profile.ko...Reading symbols from /boot/kernel/profile.ko.symbols...done. done. Loaded symbols for /boot/kernel/profile.ko Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_socket.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_mppc.ko Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done. done. Loaded symbols for /boot/kernel/rc4.ko Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 #1 0xffffffff80958581 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 #2 0xffffffff80958a77 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:636 #3 0xffffffff816d615b in vcmn_err (ce=0, fmt=0xffffffff81d325a0 "zfs: allocating allocated segment(offset=%llu size=%llu)\n", adx=0xffffff80974cf6a0) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:54 #4 0xffffffff81c886aa in zfs_panic_recover (fmt=Variable "fmt" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c:1220 #5 0xffffffff81c8aec4 in space_map_add (sm=0xfffffe0009974d00, start=34630775808, size=4096) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c:102 #6 0xffffffff81c8b3b4 in space_map_load (sm=0xfffffe0009974d00, ops=0xffffffff81d3de20, maptype=Variable "maptype" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c:326 #7 0xffffffff81c7567c in metaslab_activate (msp=0xfffffe0009974800, activation_weight=9223372036854775808) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:886 #8 0xffffffff81c76432 in metaslab_alloc (spa=0xfffffe0009ea6000, mc=0xfffffe01f90791c0, psize=512, bp=0xfffffe01f93c3080, ndvas=2, txg=3759540, hintbp=0x0, flags=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1259 #9 0xffffffff81cb036a in zio_dva_allocate (zio=0xfffffe01f929ba50) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2170 #10 0xffffffff81cad063 in zio_execute (zio=0xfffffe01f929ba50) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1196 #11 0xffffffff8099e145 in taskqueue_run_locked (queue=0xfffffe0009d5fd80) at /usr/src/sys/kern/subr_taskqueue.c:308 #12 0xffffffff8099f0c6 in taskqueue_thread_loop (arg=Variable "arg" is not available. ) at /usr/src/sys/kern/subr_taskqueue.c:497 #13 0xffffffff8092733f in fork_exit ( callout=0xffffffff8099f080 <taskqueue_thread_loop>, arg=0xfffffe0009a28840, frame=0xffffff80974cfc40) at /usr/src/sys/kern/kern_fork.c:992 #14 0xffffffff80c40e9e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #15 0x0000000000000000 in ?? () #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0xfffffe01f90b78b0 in ?? () #41 0xfffffe01f90b7470 in ?? () #42 0x0000000000000000 in ?? () #43 0xffffff80974cf600 in ?? () #44 0xffffff80974cf5a8 in ?? () #45 0xfffffe00024d4470 in ?? () #46 0xffffffff8098425d in sched_switch (td=0xffffffff812ee700, newtd=0xfffffe0009a28840, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1921 Previous frame inner to this frame (corrupt stack?) (kgdb) ------------------------------------------------------------------------ ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 -8 0 0 0 fffffe0009d61900 DLs ?? 0:00.02 [kernel] 0 1 0 0 52 0 6276 0 wait DLs ?? 0:00.00 [init] 0 2 0 0 -16 0 0 0 ctl_work DL ?? 0:00.00 [ctl_thrd] 0 3 0 0 -16 0 0 0 pftm DL ?? 0:00.00 [pfpurge] 0 4 0 0 -16 0 0 0 waiting_ DL ?? 0:00.00 [sctp_iter 0 5 0 0 -16 0 0 0 ccb_scan DL ?? 0:00.00 [xpt_thrd] 0 6 0 0 -16 0 0 0 psleep DL ?? 0:00.00 [pagedaemo 0 7 0 0 -16 0 0 0 psleep DL ?? 0:00.00 [vmdaemon] 0 8 0 0 155 0 0 0 pgzero DL ?? 0:00.00 [pagezero] 0 9 0 0 -16 0 0 0 psleep DL ?? 0:00.00 [bufdaemon 0 10 0 0 -16 0 0 0 audit_wo DL ?? 0:00.00 [audit] 0 11 0 0 155 0 0 0 - RL ?? 0:14.44 [idle] 0 12 0 0 -72 0 0 0 - WL ?? 0:00.04 [intr] 0 13 0 0 -8 0 0 0 - DL ?? 0:00.04 [geom] 0 14 0 0 -16 0 0 0 - DL ?? 0:00.00 [yarrow] 0 15 0 0 -68 0 0 0 - DL ?? 0:00.00 [usb] 0 16 0 0 16 0 0 0 syncer DL ?? 0:00.00 [syncer] 0 17 0 0 -16 0 0 0 vlruwt DL ?? 0:00.00 [vnlru] 0 18 0 0 -16 0 0 0 sdflush DL ?? 0:00.00 [softdepfl 0 129 1 0 52 0 9920 0 pause Ds ?? 0:00.00 [adjkerntz 0 1643 1 0 20 0 10376 0 select Ds ?? 0:00.00 [devd] 0 1800 1 0 20 0 12052 0 select Ds ?? 0:00.00 [syslogd] 53 1897 1 0 20 0 51224 0 kqread Ds ?? 0:00.00 [named] 0 1936 1 0 52 0 41564 0 select Ds ?? 0:00.03 [mpd5] 0 1972 0 0 -16 0 0 0 sleep DL ?? 0:00.00 [ng_queue] 26 1999 1 0 52 0 23124 0 select Ds ?? 0:00.00 [exim-4.80 0 2029 1 0 34 0 46744 0 select Ds ?? 0:00.00 [sshd] 0 2033 1 0 52 0 14128 0 nanslp Ds ?? 0:00.00 [cron] 0 2080 1 0 52 0 14504 0 wait D ?? 0:00.00 [sh] 0 2081 1 0 52 0 9912 0 piperd D ?? 0:00.00 [logger] 0 2082 2080 0 52 0 3776 0 nanslp D ?? 0:00.00 [sleep] 0 2084 1 0 20 0 41164 0 wait Ds ?? 0:00.00 [login] 0 2085 1 0 52 0 12052 0 ttyin Ds+ ?? 0:00.00 [getty] 0 2086 1 0 52 0 12052 0 ttyin Ds+ ?? 0:00.00 [getty] 0 2087 1 0 52 0 12052 0 ttyin Ds+ ?? 0:00.00 [getty] 0 2088 1 0 52 0 12052 0 ttyin Ds+ ?? 0:00.00 [getty] 0 2089 1 0 52 0 12052 0 ttyin Ds+ ?? 0:00.00 [getty] 0 2090 1 0 52 0 12052 0 ttyin Ds+ ?? 0:00.00 [getty] 0 2091 1 0 52 0 12052 0 ttyin Ds+ ?? 0:00.00 [getty] 0 2092 2084 0 20 0 17532 0 pause D ?? 0:00.00 [csh] 0 2094 2092 0 43 0 35476 0 tx->tx_s D+ ?? 0:00.00 [zpool] 0 2095 0 0 -8 0 0 0 zio->io_ DL ?? 0:00.00 [zfskern] ------------------------------------------------------------------------ vmstat -s 89990 cpu context switches 22775 device interrupts 4274 software interrupts 140816 traps 190784 system calls 20 kernel threads created 2066 fork() calls 9 vfork() calls 0 rfork() calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 626 vnode pager pageins 4855 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out 0 page daemon wakeups 0 pages examined by the page daemon 0 pages reactivated 60165 copy-on-write faults 237 copy-on-write optimized faults 48790 zero fill pages zeroed 0 zero fill pages prezeroed 1 intransit blocking page faults 157145 total VM faults taken 0 pages affected by kernel thread creation 1050577 pages affected by fork() 4776 pages affected by vfork() 0 pages affected by rfork() 0 pages cached 176632 pages freed 0 pages freed by daemon 0 pages freed by exiting processes 6824 pages active 8252 pages inactive 0 pages in VM cache 59163 pages wired down 1885116 pages free 4096 bytes per page 25089 total name lookups cache hits (86% pos + 7% neg) system 0% per-directory deletions 0%, falsehits 0%, toolong 0% ------------------------------------------------------------------------ vmstat -m Type InUse MemUse HighUse Requests Size(s) md_nvidia_data 0 0K - 31 512 mixer 3 12K - 3 4096 md_sii_data 0 0K - 31 512 acpi_perf 2 1K - 2 128 CAM dev queue 8 1K - 8 128 CAM XPT 122 138K - 311 16,32,64,128,1024,2048 isadev 7 1K - 7 128 UART 3 2K - 3 16,512,1024 cdev 7 2K - 7 256 sigio 1 1K - 1 64 filedesc 49 50K - 2106 16,32,64,128,512,1024,2048,4096 kdtrace 298 67K - 2491 64,256 kenv 84 12K - 100 16,32,64,128 kqueue 2 9K - 19 256,2048 proc-args 22 2K - 590 32,64,128,256 hhook 2 1K - 2 128 pfsync 1 1K - 1 512 ithread 90 14K - 90 32,128,256 KTRACE 100 13K - 100 128 acpiintr 1 1K - 1 64 linker 354 908K - 452 16,32,64,128,256,512,1024,2048,4096 lockf 14 2K - 90 64,128 loginclass 2 1K - 2 64 ip6ndp 15 2K - 16 64,128 ip6opt 0 0K - 198 256 temp 47 30K - 6066 16,32,64,128,256,512,1024,2048,4096 devbuf 36192 94194K - 36326 16,32,64,128,256,512,1024,2048,4096 module 508 64K - 508 128 mtx_pool 2 16K - 2 osd 3 1K - 4 16,64 USBdev 37 5K - 37 64,128,256,512 pmchooks 1 1K - 1 128 USB 59 39K - 60 16,32,64,128,256,2048,4096 subproc 94 194K - 2148 512,4096 proc 2 32K - 2 session 18 3K - 18 128 pgrp 20 3K - 20 128 cred 36 6K - 5188 64,256 uidinfo 4 5K - 6 128,4096 plimit 12 3K - 152 256 acpica 5964 604K - 64893 16,32,64,128,256,512,1024,2048,4096 acpitask 1 2K - 1 2048 sysctltmp 0 0K - 365 16,32,64,128,256,4096 sysctloid 4457 219K - 4562 16,32,64,128 sysctl 0 0K - 510 16,32,64 tidhash 1 32K - 1 callout 1 512K - 1 umtx 516 65K - 516 128 p1003.1b 1 1K - 1 16 SWAP 2 2189K - 2 64 bus-sc 88 267K - 3611 16,32,64,128,256,512,1024,2048,4096 bus 1272 104K - 6402 16,32,64,128,256,512,1024 devstat 12 25K - 12 32,4096 eventhandler 97 8K - 97 64,128 kobj 335 1340K - 666 4096 Per-cpu 1 1K - 1 32 CAM queue 30 2K - 152 16,32,256 rman 246 30K - 661 16,32,128 sbuf 0 0K - 1266 16,32,64,128,256,512,1024,2048,4096 acpisem 24 3K - 24 128 ppbusdev 3 1K - 3 256 stack 0 0K - 2 256 taskqueue 47 5K - 77 16,32,64,128,1024 Unitno 15 1K - 383 32,64 iov 0 0K - 2073 16,64,128,256,512 select 8 1K - 8 128 ioctlops 1 8K - 1893 16,32,64,128,256,512,1024,2048,4096 msg 4 30K - 4 2048,4096 sem 4 106K - 4 2048,4096 shm 1 20K - 1 tty 20 20K - 22 1024,2048 accf 2 1K - 2 64 mbuf_tag 0 0K - 720 32,64 shmfd 1 8K - 1 entropy 1024 64K - 1024 64 pcb 24 157K - 714 16,32,64,128,1024,2048,4096 soname 5 1K - 2318 16,32,128 vfscache 1 2048K - 1 vfs_hash 1 1024K - 1 CAM SIM 8 2K - 8 256 vnodes 2 1K - 2 256 DEVFS1 131 66K - 148 512 mount 74 4K - 177 16,32,64,128,256 vnodemarker 0 0K - 32 512 BPF 13 2K - 13 128 ether_multi 33 2K - 38 16,32,64 ifaddr 136 33K - 136 32,64,128,256,512,2048,4096 ifnet 14 27K - 14 128,2048 DEVFS3 305 77K - 326 256 clone 8 32K - 8 4096 arpcom 2 1K - 2 16 lltable 39 17K - 39 256,512 DEVFS2 130 3K - 130 16 DEVFS_RULE 55 26K - 55 64,512 DEVFS 54 2K - 55 16,128 routetbl 35 5K - 188 32,64,128,256,512 igmp 13 4K - 13 256 ipid 2 24K - 2 in_multi 2 1K - 2 256 sctp_iter 0 0K - 3 256 sctp_ifn 2 1K - 2 128 sctp_ifa 5 1K - 5 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 3 16 hostcache 1 28K - 1 syncache 1 96K - 1 in6_multi 20 3K - 20 32,256 mld 13 2K - 13 128 rpc 2 1K - 2 256 audit_evclass 179 6K - 218 32 jblocks 8 2K - 8 128,256 savedino 0 0K - 24 256 sbdep 0 0K - 7 64 jsegdep 2065 130K - 2154 64 jseg 16 2K - 16 128 jfreefrag 0 0K - 3 128 jnewblk 0 0K - 2060 128 jremref 0 0K - 45 128 jaddref 0 0K - 46 128 freework 7 1K - 22 128 newdirblk 4 1K - 6 64 dirrem 6 1K - 33 128 mkdir 4 1K - 12 128 diradd 13 2K - 34 128 freefile 17 2K - 24 64 freeblks 6 1K - 21 128 freefrag 0 0K - 3 128 indirdep 3 1K - 6 128 newblk 2030 764K - 2061 256 bmsafemap 7 10K - 48 256 inodedep 41 1044K - 72 512 pagedep 10 259K - 17 256 ufs_dirhash 39 8K - 39 16,32,64,128,512 ufs_mount 12 50K - 13 512,4096 vm_pgdata 2 129K - 2 128 memdesc 1 4K - 1 4096 atkbddev 2 1K - 2 64 pfs_nodes 21 6K - 21 256 GEOM 87 17K - 1140 16,32,64,128,256,512,1024,2048 CAM periph 8 2K - 32 16,32,64,128,256 ctlmem 5062 10113K - 5062 128,2048 hdaa 5 41K - 5 1024,2048 hdac 1 1K - 1 1024 hdacc 1 1K - 1 32 ctlblk 200 1600K - 200 ramdisk 1 4096K - 1 acpidev 36 3K - 36 64 apmdev 1 1K - 1 128 madt_table 0 0K - 1 4096 kbdmux 7 18K - 7 16,512,1024,2048 feeder 19 2K - 24 32,128 ctlpool 532 142K - 532 32,512 raid_data 0 0K - 186 32,128,256 io_apic 1 2K - 1 2048 MCA 6 1K - 6 128 msi 4 1K - 4 128 nexusdev 3 1K - 3 16 pci_link 16 2K - 16 16,32,128 cyclic 8 1K - 8 16,64,128 solaris 1519 25596K - 9962 16,32,64,128,256,512,1024,2048,4096 fbt 1 256K - 1 netgraph_msg 0 0K - 5 64,128,256 netgraph_node 5 1K - 7 128,256 netgraph_hook 0 0K - 2 128 netgraph_sock 6 1K - 9 32,128 netgraph_path 0 0K - 4 16 netgraph_mppc 0 0K - 1 1024 kstat_data 4 1K - 4 64 ------------------------------------------------------------------------ vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 208, 0, 209, 12, 209, 0, 0 UMA Zones: 640, 0, 209, 1, 209, 0, 0 UMA Slabs: 568, 0, 4365, 164, 6054, 0, 0 UMA RCntSlabs: 568, 0, 262, 4, 262, 0, 0 UMA Hash: 256, 0, 82, 8, 82, 0, 0 16 Bucket: 152, 0, 69, 6, 69, 0, 0 32 Bucket: 280, 0, 67, 3, 67, 0, 0 64 Bucket: 536, 0, 48, 1, 48, 57, 0 128 Bucket: 1048, 0, 91, 2, 91, 638, 0 VM OBJECT: 232, 0, 1204, 28, 25078, 0, 0 MAP: 232, 0, 7, 25, 7, 0, 0 KMAP ENTRY: 120, 277791, 100, 179, 22165, 0, 0 MAP ENTRY: 120, 0, 559, 123, 57139, 0, 0 fakepg: 120, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 348, 42, 348, 0, 0 16: 16, 0, 3026, 334, 37397, 0, 0 32: 32, 0, 2528, 199, 12778, 0, 0 64: 64, 0, 23347, 61, 55907, 0, 0 128: 128, 0, 8835, 648, 14855, 0, 0 256: 256, 0, 3061, 89, 8664, 0, 0 512: 512, 0, 876, 55, 7155, 0, 0 1024: 1024, 0, 79, 1093, 6083, 0, 0 2048: 2048, 0, 5368, 94, 5933, 0, 0 4096: 4096, 0, 473, 48, 7428, 0, 0 Files: 80, 0, 77, 103, 7930, 0, 0 TURNSTILE: 136, 0, 259, 41, 259, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1184, 0, 41, 10, 2095, 0, 0 THREAD: 1128, 0, 255, 3, 394, 0, 0 SLEEPQUEUE: 80, 0, 259, 31, 259, 0, 0 VMSPACE: 392, 0, 22, 28, 2077, 0, 0 cpuset: 72, 0, 223, 127, 361, 0, 0 cyclic_id_cache: 64, 0, 0, 0, 0, 0, 0 audit_record: 960, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 256, 268, 1114, 0, 0 mbuf: 256, 0, 1, 267, 1544, 0, 0 mbuf_cluster: 2048, 33792, 512, 12, 512, 0, 0 mbuf_jumbo_page: 4096, 16896, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 300000, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 128000, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 dtrace_state_cache: 4096, 0, 0, 0, 0, 0, 0 g_bio: 232, 0, 0, 176, 18939, 0, 0 ttyinq: 160, 0, 120, 24, 255, 0, 0 ttyoutq: 256, 0, 64, 26, 136, 0, 0 ata_request: 328, 0, 0, 0, 0, 0, 0 ata_composite: 336, 0, 0, 0, 0, 0, 0 VNODE: 480, 0, 667, 13, 693, 0, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 36, 11437, 0, 0 S VFS Cache: 108, 0, 645, 81, 1457, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NCLNODE: 568, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 54, 2, 54, 0, 0 NFSMOUNT: 632, 0, 0, 0, 0, 0, 0 NFSNODE: 656, 0, 0, 0, 0, 0, 0 Mountpoints: 792, 0, 6, 9, 6, 0, 0 pipe: 728, 0, 5, 25, 1488, 0, 0 ksiginfo: 112, 0, 41, 1015, 46, 0, 0 itimer: 344, 0, 0, 0, 0, 0, 0 pfsrctrpl: 152, 10000, 0, 0, 0, 0, 0 pfrulepl: 936, 0, 1, 7, 1, 0, 0 pfstatepl: 288, 10010, 0, 0, 0, 0, 0 pfstatekeypl: 288, 0, 0, 0, 0, 0, 0 pfstateitempl: 288, 0, 0, 0, 0, 0, 0 pfaltqpl: 240, 0, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 1, 83, 1, 0, 0 pfrktable: 1296, 1002, 0, 0, 0, 0, 0 pfrkentry: 160, 200016, 0, 0, 0, 0, 0 pfrkcounters: 64, 0, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 0, 0, 0, 0 pffrag: 80, 0, 0, 0, 0, 0, 0 pffrcache: 80, 10035, 0, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 0, 0, 0, 0, 0 pfospfen: 112, 0, 700, 26, 700, 0, 0 pfosfp: 40, 0, 410, 94, 410, 0, 0 KNOTE: 128, 0, 8, 79, 2936, 0, 0 socket: 680, 33792, 25, 47, 966, 0, 0 ipq: 56, 1071, 0, 0, 0, 0, 0 udp_inpcb: 392, 33800, 3, 47, 909, 0, 0 udpcb: 16, 33936, 3, 333, 909, 0, 0 tcp_inpcb: 392, 33800, 8, 22, 12, 0, 0 tcpcb: 976, 33792, 8, 4, 12, 0, 0 tcptw: 72, 6800, 0, 0, 0, 0, 0 syncache: 152, 15375, 0, 0, 0, 0, 0 hostcache: 136, 15372, 0, 0, 0, 0, 0 tcpreass: 40, 2184, 0, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0 sctp_ep: 1376, 33792, 0, 0, 0, 0, 0 sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 216, 4, 0, 0 sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 ripcb: 392, 33800, 0, 0, 0, 0, 0 unpcb: 240, 33792, 8, 40, 32, 0, 0 rtentry: 200, 0, 17, 40, 17, 0, 0 pfsync: 88, 0, 0, 0, 0, 0, 0 selfd: 56, 0, 29, 160, 3889, 0, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0, 0 FFS inode: 168, 0, 629, 31, 653, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 629, 46, 653, 0, 0 NetGraph items: 72, 4118, 0, 87, 7, 0, 0 NetGraph data items: 72, 522, 0, 87, 4, 0, 0 taskq_zone: 48, 0, 0, 216, 8, 0, 0 zio_cache: 880, 0, 8, 1160, 3552, 0, 0 zio_link_cache: 48, 0, 6, 2154, 5326, 0, 0 zio_buf_512: 512, 0, 0, 0, 0, 0, 0 zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0 zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0 zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0 zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0 zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0 zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0 zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0 zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0 zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0 zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0 zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0 zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0 zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0 zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0 zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0 zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0 zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0 zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0 zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0 zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0 zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0 zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0 zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0 zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0 zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0 zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0 zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0 zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0 zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0 zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0 zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0 zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0 zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0 zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0 zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0 zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0 zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0 zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0 zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0 zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0 zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0 zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0 zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0 zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0 zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0 zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0 zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0 zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0 zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0 zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0 zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0 zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0 zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0 zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0 zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0 zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0 zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0 zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0 zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0 zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0 zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0 zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0 zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0 zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0 zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0 zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0 zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0 zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0 zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0 zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0 zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0 zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0 zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0 zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0 zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0 zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0 zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0 zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0 zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0 zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0 zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0 zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0 zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0 zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0 zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0 zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0 zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0 sa_cache: 80, 0, 0, 0, 0, 0, 0 dnode_t: 856, 0, 132, 12, 208, 0, 0 dmu_buf_impl_t: 224, 0, 114, 39, 177, 0, 0 arc_buf_hdr_t: 216, 0, 129, 33, 197, 0, 0 arc_buf_t: 104, 0, 129, 51, 232, 0, 0 zil_lwb_cache: 192, 0, 0, 0, 0, 0, 0 zfs_znode_cache: 400, 0, 0, 0, 0, 0, 0 ------------------------------------------------------------------------ vmstat -i interrupt total rate irq16: hdac0 ohci0+ 184 4 irq17: ehci0 2 0 irq18: ohci2 ohci3+ 4 0 irq22: ahci0 5777 148 irq256: hpet0:t0 12537 321 irq257: hpet0:t1 3810 97 irq259: re0 461 11 Total 22775 583 ------------------------------------------------------------------------ pstat -T 77/16424 files 0M/10239M swap space ------------------------------------------------------------------------ pstat -s Device 512-blocks Used Avail Capacity /dev/ada0p2 20971264 0 20971264 0% ------------------------------------------------------------------------ iostat iostat: kvm_read(_tk_nin): invalid address (0x0) iostat: disabling TTY statistics ada0 ada1 ada2 cpu KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 74.49 130 9.49 30.76 7 0.21 30.46 7 0.21 1 0 6 0 92 ------------------------------------------------------------------------ ipcs -a Message Queues: T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME Shared Memory: T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Semaphores: T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME ------------------------------------------------------------------------ ipcs -T msginfo: msgmax: 16384 (max characters in a message) msgmni: 40 (# of message queues) msgmnb: 2048 (max characters in a message queue) msgtql: 40 (max # of messages in system) msgssz: 8 (size of a message segment) msgseg: 2048 (# of message segments in system) shminfo: shmmax: 536870912 (max shared memory segment size) shmmin: 1 (min shared memory segment size) shmmni: 192 (max number of shared memory identifiers) shmseg: 128 (max shared memory segments per process) shmall: 131072 (max amount of shared memory in pages) seminfo: semmni: 50 (# of semaphore identifiers) semmns: 340 (# of semaphores in system) semmnu: 150 (# of undo structures in system) semmsl: 340 (max # of semaphores per id) semopm: 100 (max # of operations per semop call) semume: 50 (max # of undo entries per process) semusz: 632 (size in bytes of undo structure) semvmx: 32767 (semaphore maximum value) semaem: 16384 (adjust on exit max value) ------------------------------------------------------------------------ nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 0 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 0 0 0 0 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 0 0 0 0 0 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 0 0 0 ------------------------------------------------------------------------ netstat -s tcp: 0 packets sent 0 data packets (0 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 0 ack-only packets (0 delayed) 0 URG only packets 0 window probe packets 0 window update packets 0 control packets 4 packets received 0 acks (for 0 bytes) 0 duplicate acks 0 acks for unsent data 0 packets (0 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 0 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 0 connection requests 0 connection accepts 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 0 connections established (including accepts) 4 connections closed (including 0 drops) 0 connections updated cached RTT on close 0 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 0 embryonic connections dropped 0 segments updated rtt (of 0 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 0 correct ACK header predictions 0 correct data packet header predictions 0 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 0 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 0 cookies sent 0 cookies received 0 hostcache entries added 0 bucket overflow 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 177 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 with no checksum 1 dropped due to no socket 130 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 46 delivered 507 datagrams output 0 times multicast source filter matched ip: 193 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 191 packets for this host 2 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 522 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 1 call to icmp_error 0 errors not generated in response to an icmp message Output histogram: echo reply: 10 destination unreachable: 1 0 messages with bad code fields 0 messages less than the minimum length 0 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: destination unreachable: 2 echo: 10 10 message responses generated 0 invalid return addresses 0 no return routes igmp: 0 messages received 0 messages received with too few bytes 0 messages received with wrong TTL 0 messages received with bad checksum 0 V1/V2 membership queries received 0 V3 membership queries received 0 membership queries received with invalid field(s) 0 general queries received 0 group queries received 0 group-source queries received 0 group-source queries dropped 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 V3 reports received without Router Alert 0 membership reports sent pfsync: 0 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for bad ttl 0 packets shorter than header 0 packets discarded for bad version 0 packets discarded for bad HMAC 0 packets discarded for bad action 0 packets discarded for short packet 0 states discarded for bad values 0 stale states 0 failed state lookup/inserts 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 send error arp: 14 ARP requests sent 3 ARP replies sent 54 ARP requests received 9 ARP replies received 63 ARP packets received 462 total packets dropped due to no ARP entry 0 ARP entrys timed out 0 Duplicate IPs seen ip6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 5 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 396 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Mbuf statistics: 0 one mbuf 0 one ext mbuf 0 two or more ext mbuf 0 packets whose headers are not contiguous 0 tunneling packets that can't find gif 0 packets discarded because of too many headers 0 failures of source address selection Source addresses selection rule applied: icmp6: 0 calls to icmp6_error 0 errors not generated in response to an icmp6 message 0 errors not generated because of rate limitation Output histogram: neighbor solicitation: 1 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output ------------------------------------------------------------------------ netstat -m 257/535/792 mbufs in use (current/cache/total) 244/280/524/33792 mbuf clusters in use (current/cache/total/max) 256/268 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/16896 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/300000 9k jumbo clusters in use (current/cache/total/max) 0/0/0/128000 16k jumbo clusters in use (current/cache/total/max) 552K/693K/1246K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ------------------------------------------------------------------------ netstat -id Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop re0 1500 <Link#1> f4:6d:04:29:a0:d8 226 0 0 45 0 0 0 re0 1500 10.5.5.0 backup 56 - - 490 - - - re0 1500 fe80::f66d:4f fe80::f66d:4ff:fe 0 - - 2 - - - usbus 0 <Link#2> 0 0 0 0 0 0 0 usbus 0 <Link#3> 0 0 0 0 0 0 0 usbus 0 <Link#4> 0 0 0 0 0 0 0 usbus 0 <Link#5> 0 0 0 0 0 0 0 usbus 0 <Link#6> 0 0 0 0 0 0 0 usbus 0 <Link#7> 0 0 0 0 0 0 0 sk0* 1500 <Link#8> 00:21:91:8e:84:91 0 0 0 0 0 0 0 usbus 0 <Link#9> 0 0 0 0 0 0 0 plip0 1500 <Link#10> 0 0 0 0 0 0 0 pflog 33152 <Link#11> 0 0 0 0 0 0 0 pfsyn 1500 <Link#12> 0 0 0 0 0 0 0 lo0 16384 <Link#13> 32 0 0 32 0 0 0 lo0 16384 localhost ::1 0 - - 0 - - - lo0 16384 fe80::1%lo0 fe80::1 0 - - 0 - - - lo0 16384 your-net localhost 32 - - 32 - - - ------------------------------------------------------------------------ netstat -anr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.5.5.240 UGS 0 455 re0 10.5.5.0/24 link#1 U 0 35 re0 10.5.5.6 link#1 UHS 0 0 lo0 127.0.0.1 link#13 UH 0 32 lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 ::1 link#13 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 fe80::/10 ::1 UGRS lo0 fe80::%re0/64 link#1 U re0 fe80::f66d:4ff:fe29:a0d8%re0 link#1 UHS lo0 fe80::%lo0/64 link#13 U lo0 fe80::1%lo0 link#13 UHS lo0 ff01::%re0/32 fe80::f66d:4ff:fe29:a0d8%re0 U re0 ff01::%lo0/32 ::1 U lo0 ff02::/16 ::1 UGRS lo0 ff02::%re0/32 fe80::f66d:4ff:fe29:a0d8%re0 U re0 ff02::%lo0/32 ::1 U lo0 ------------------------------------------------------------------------ netstat -anA Active Internet connections (including servers) Tcpcb Proto Recv-Q Send-Q Local Address Foreign Address (state) fffffe0009e537a0 tcp4 0 0 *.22 *.* LISTEN fffffe0009e53b70 tcp6 0 0 *.22 *.* LISTEN fffffe0009e68000 tcp4 0 0 *.25 *.* LISTEN fffffe0009e683d0 tcp6 0 0 *.25 *.* LISTEN fffffe0009e687a0 tcp6 0 0 ::1.953 *.* LISTEN fffffe0009e527a0 tcp4 0 0 127.0.0.1.953 *.* LISTEN fffffe0009e52000 tcp4 0 0 127.0.0.1.53 *.* LISTEN fffffe0009e523d0 tcp4 0 0 10.5.5.6.53 *.* LISTEN fffffe0009b36930 udp4 0 0 10.5.5.6.1701 *.* fffffe0009b36dc8 udp4 0 0 127.0.0.1.53 *.* fffffe0009b37000 udp4 0 0 10.5.5.6.53 *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr fffffe0009c37c30 stream 0 0 fffffe0009b6fb40 0 0 0 /var/run/devd.pipe fffffe0009c373c0 dgram 0 0 0 fffffe0009c245a0 0 fffffe0009c242d0 fffffe0009c242d0 dgram 0 0 0 fffffe0009c245a0 0 fffffe0009c37690 fffffe0009c37690 dgram 0 0 0 fffffe0009c245a0 0 0 fffffe0009c243c0 dgram 0 0 fffffe0009d38d20 0 0 0 /var/named/var/run/log fffffe0009c244b0 dgram 0 0 fffffe0009d53d20 0 0 0 /var/run/log fffffe0009c245a0 dgram 0 0 fffffe0009d54000 0 fffffe0009c373c0 0 /var/run/logpriv fffffe0009c24690 dgram 0 0 fffffe0009ad1780 0 0 0 /var/run/log ------------------------------------------------------------------------ netstat -aL Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address tcp4 0/0/128 *.ssh tcp6 0/0/128 *.ssh tcp4 0/0/20 *.smtp tcp6 0/0/20 *.smtp tcp6 0/0/128 localhost.rndc tcp4 0/0/128 localhost.rndc tcp4 0/0/3 localhost.domain tcp4 0/0/3 backup.domain unix 0/0/4 /var/run/devd.pipe ------------------------------------------------------------------------ fstat USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root zfskern 2095 root / 2 drwxr-xr-x 1024 r root zfskern 2095 wd / 2 drwxr-xr-x 1024 r root zpool 2094 root / 2 drwxr-xr-x 1024 r root zpool 2094 wd / 32900 drwxr-xr-x 512 r root zpool 2094 text / 66069 -r-xr-xr-x 92480 r root zpool 2094 ctty /dev 56 crw------- ttyv0 rw root zpool 2094 0 /dev 56 crw------- ttyv0 rw root zpool 2094 1 /dev 56 crw------- ttyv0 rw root zpool 2094 2 /dev 56 crw------- ttyv0 rw root zpool 2094 3 /dev 141 crw-rw-rw- zfs rw root zpool 2094 4 /dev 20 crw-rw-rw- zero r root zpool 2094 5 / 32932 -rw-r--r-- 0 r root csh 2092 root / 2 drwxr-xr-x 1024 r root csh 2092 wd / 32900 drwxr-xr-x 512 r root csh 2092 text / 66038 -r-xr-xr-x 382264 r root csh 2092 ctty /dev 56 crw------- ttyv0 rw root csh 2092 15 /dev 56 crw------- ttyv0 rw root csh 2092 16 /dev 56 crw------- ttyv0 rw root csh 2092 17 /dev 56 crw------- ttyv0 rw root csh 2092 18 /dev 56 crw------- ttyv0 rw root csh 2092 19 /dev 56 crw------- ttyv0 rw root getty 2091 root / 2 drwxr-xr-x 1024 r root getty 2091 wd / 2 drwxr-xr-x 1024 r root getty 2091 text /usr 4594020 -r-xr-xr-x 28040 r root getty 2091 ctty /dev 63 crw------- ttyv7 rw root getty 2091 0 /dev 63 crw------- ttyv7 rw root getty 2091 1 /dev 63 crw------- ttyv7 rw root getty 2091 2 /dev 63 crw------- ttyv7 rw root getty 2090 root / 2 drwxr-xr-x 1024 r root getty 2090 wd / 2 drwxr-xr-x 1024 r root getty 2090 text /usr 4594020 -r-xr-xr-x 28040 r root getty 2090 ctty /dev 62 crw------- ttyv6 rw root getty 2090 0 /dev 62 crw------- ttyv6 rw root getty 2090 1 /dev 62 crw------- ttyv6 rw root getty 2090 2 /dev 62 crw------- ttyv6 rw root getty 2089 root / 2 drwxr-xr-x 1024 r root getty 2089 wd / 2 drwxr-xr-x 1024 r root getty 2089 text /usr 4594020 -r-xr-xr-x 28040 r root getty 2089 ctty /dev 61 crw------- ttyv5 rw root getty 2089 0 /dev 61 crw------- ttyv5 rw root getty 2089 1 /dev 61 crw------- ttyv5 rw root getty 2089 2 /dev 61 crw------- ttyv5 rw root getty 2088 root / 2 drwxr-xr-x 1024 r root getty 2088 wd / 2 drwxr-xr-x 1024 r root getty 2088 text /usr 4594020 -r-xr-xr-x 28040 r root getty 2088 ctty /dev 60 crw------- ttyv4 rw root getty 2088 0 /dev 60 crw------- ttyv4 rw root getty 2088 1 /dev 60 crw------- ttyv4 rw root getty 2088 2 /dev 60 crw------- ttyv4 rw root getty 2087 root / 2 drwxr-xr-x 1024 r root getty 2087 wd / 2 drwxr-xr-x 1024 r root getty 2087 text /usr 4594020 -r-xr-xr-x 28040 r root getty 2087 ctty /dev 59 crw------- ttyv3 rw root getty 2087 0 /dev 59 crw------- ttyv3 rw root getty 2087 1 /dev 59 crw------- ttyv3 rw root getty 2087 2 /dev 59 crw------- ttyv3 rw root getty 2086 root / 2 drwxr-xr-x 1024 r root getty 2086 wd / 2 drwxr-xr-x 1024 r root getty 2086 text /usr 4594020 -r-xr-xr-x 28040 r root getty 2086 ctty /dev 58 crw------- ttyv2 rw root getty 2086 0 /dev 58 crw------- ttyv2 rw root getty 2086 1 /dev 58 crw------- ttyv2 rw root getty 2086 2 /dev 58 crw------- ttyv2 rw root getty 2085 root / 2 drwxr-xr-x 1024 r root getty 2085 wd / 2 drwxr-xr-x 1024 r root getty 2085 text /usr 4594020 -r-xr-xr-x 28040 r root getty 2085 ctty /dev 57 crw------- ttyv1 rw root getty 2085 0 /dev 57 crw------- ttyv1 rw root getty 2085 1 /dev 57 crw------- ttyv1 rw root getty 2085 2 /dev 57 crw------- ttyv1 rw root login 2084 root / 2 drwxr-xr-x 1024 r root login 2084 wd / 32900 drwxr-xr-x 512 r root login 2084 text /usr 5304431 -r-sr-xr-x 25096 r root login 2084 ctty /dev 56 crw------- ttyv0 rw root login 2084 0 /dev 56 crw------- ttyv0 rw root login 2084 1 /dev 56 crw------- ttyv0 rw root login 2084 2 /dev 56 crw------- ttyv0 rw root login 2084 3* local dgram fffffe0009c373c0 <-> fffffe0009c245a0 root sleep 2082 root / 2 drwxr-xr-x 1024 r root sleep 2082 wd / 2 drwxr-xr-x 1024 r root sleep 2082 text / 66080 -r-xr-xr-x 5512 r root sleep 2082 0 /dev 19 crw-rw-rw- null r root sleep 2082 1* pipe fffffe0009d2fcb8 <-> fffffe0009d2fb60 0 rw root sleep 20
This is only FreeBSD bug, because from Solaris i have successfully accessed to this pool.
Responsible Changed From-To: freebsd-bugs->freebsd-zfs-devel Over to maintainer(s).
This can happens with disk in pool, that have different buffer size. After replace such disk in pool - pool started normal work. So, this is a bug.
Same thing on 11.0-STABLE r315957. 12 panics in a row on during ~ one month. I have crashdumps in case someone needs them. Typical bt: savecore: reboot after panic: Solaris(panic): zfs: allocating allocated segment(offset=964258332672 size=16384) Nov 30 03:58:03 san savecore: reboot after panic: Solaris(panic): zfs: allocating allocated segment(offset=964258332672 size=16384) savecore: writing core to /var/crash/vmcore.9 Writing crash summary to /var/crash/core.txt.9. Clearing /tmp (X related). Starting ctld. NFS on reserved port only=YES NFSv4 is disabled Starting rpcbind. /etc/rc.d/mountd: WARNING: /etc/exports is not readable. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Processing uwsgi profile: api Starting uwsgi. [uWSGI] getting INI configuration from /usr/local/etc/uwsgi_api.ini Processing uwsgi profile: zfsreplica Starting uwsgi. [uWSGI] getting INI configuration from /usr/local/etc/uwsgi_zfsreplica.ini Updating motd:. Mounting late filesystems:. Starting ntpd. Performing sanity check on nginx configuration: Nov 30 03:58:32 san ntpd[1083]: restrict: ignoring line 21, address/host 'fd00::/16' unusable. nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful Starting nginx. Starting bird. Configuring vt: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Starting background file system checks in 60 seconds. Thu Nov 30 03:58:33 +05 2017 Nov 30 04:51:35 san ctld[2278]: 192.168.1.72 (iqn.2016-04.net.playkey:test-kvm): read: Connection reset by peer Nov 30 11:07:02 san ctld[2537]: 192.168.1.72 (iqn.2016-04.net.playkey:test-kvm): read: Connection reset by peer WARNING: 192.168.1.57 (iqn.2010-04.org.ipxe:03000200-0400-0500-0006-000700080009): no ping reply (NOP-Out) after 5 seconds; dropping connection WARNING: 192.168.1.57 (iqn.2010-04.org.ipxe:03000200-0400-0500-0006-000700080009): no ping reply (NOP-Out) after 5 seconds; dropping connection Nov 30 17:29:48 san su: failed to get password: authentication error Nov 30 17:29:50 san su: emz to root on /dev/pts/1 WARNING: 192.168.1.57 (iqn.2010-04.org.ipxe:03000200-0400-0500-0006-000700080009): no ping reply (NOP-Out) after 5 seconds; dropping connection panic: Solaris(panic): zfs: allocating allocated segment(offset=1102832721920 size=16384) cpuid = 4 KDB: stack backtrace: #0 0xffffffff80a8c7c7 at kdb_backtrace+0x67 #1 0xffffffff80a49ae6 at vpanic+0x186 #2 0xffffffff80a49953 at panic+0x43 #3 0xffffffff82432192 at vcmn_err+0xc2 #4 0xffffffff821a4b1a at zfs_panic_recover+0x5a #5 0xffffffff8218e035 at range_tree_add+0x235 #6 0xffffffff8218cb86 at metaslab_free_dva+0x276 #7 0xffffffff8218cc31 at metaslab_free+0x81 #8 0xffffffff821cfbda at zio_dva_free+0x1a #9 0xffffffff821ccb8d at zio_execute+0x23d #10 0xffffffff80a9d447 at taskqueue_run_locked+0x117 #11 0xffffffff80a9e548 at taskqueue_thread_loop+0xc8 #12 0xffffffff80a0dd05 at fork_exit+0x85 #13 0xffffffff80e92f6e at fork_trampoline+0xe Uptime: 13h42m11s Dumping 2638 out of 24531 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% This is also related to zfs corruption on the pool, which happens after these panics. May be this is somehow related to the fact that this host runs iSCSI target, probably panics occur when working with specific data which iSCSI target serves.
Posting a followup since I'm experiencing a very similar error. Machine is a laptop with only one disk. FreeBSD 13-head, r361728 The machine is configures with geli and zfs boot, as created by the FreeBSD installer. At some point the machine has had a panic+reboot while shutting down [1] and on reboot, after inputting the geli password, the machine panicked. With a debug kernel I discovered the reason: panic: Solaris(panic): zfs: allocating allocated segment(offset=433818959872 size=4096) of (offset=433818021200 size=32768) The panic appears to happen in range_tree_add_impl, called from space_map_load_callback. After this I was able to make the machine boot by setting vfs.zfs.recover=1 in loader.conf. At boot it now gives these error messages: Solaris: WARNING: zfs: allocating allocated segment(offset=433818959872 size=4096) of (offset=433818931200 size=32768) Solaris: WARNING: zfs: allocating allocated segment(offset=433815740416 size=4096) of (offset=433815736320 size=8192) Once booted the machine works fine, for the little use I've put it up after this event. I also tried to run some more diagnostic zdb gave me this: # zdb -c zroot Traversing all blocks to verify metadata checksums and verify nothing leaked ... loading concrete vdev 0, metaslab 101 of 113 ...Assertion failed: rs->rs_start <= start (0x65019be000 <= 0x65019bd000), file /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c, line 410. Abort (core dumped) Anny other diagnostic I could try to run? [1] this is actually speculation since it happened when I commanded a shutdown from another PC in another room, and found the machine at thew geli password prompt instead of turned off.
Reassign.
This may be related: https://github.com/openzfs/zfs/issues/10667
I have reinstalled the machine giving me this problem from scratch. Before getting this issue the machine was crashing on boot or shutdown but I never grabbed one of those. After reinstalling it it is having sporadic panics related to OpenZFS I've created bug #249985 with data on these periodic panics. I suspect the two issues to be related in some way, for example the panicss I'm now getting could end up putting the pool in the state to create the bug described here.
Resolved per bug #249985. Pease re-open if not resolved.