Bug 176636

Summary: Occasional crashes with ZFS in 9.1-RELEASE
Product: Base System Reporter: Rasmus Skaarup <freebsd>
Component: kernAssignee: Graham Perrin <grahamperrin>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: fs, grahamperrin
Priority: Normal Keywords: crash
Version: 9.1-RELEASE   
Hardware: Any   
OS: Any   

Description Rasmus Skaarup 2013-03-04 13:10:00 UTC
I've been having trouble with one of our virtual machines running FreeBSD on CentOS with KVM. The other machine is running without issues, but this one keeps crashing. We have tried moving it to another physical machine to eliminate hardware issues - and the crashes are still occurring. 

Today the machine crashed three times. 

Crash 0 (2nd of March):

Unread portion of the kernel message buffer:
panic: bad pte
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bcffe8 at pmap_remove_pages+0x3a8
#3 0xffffffff80b49d9a at vmspace_exit+0x9a
#4 0xffffffff808b9d69 at exit1+0x379
#5 0xffffffff808bac3e at sys_sys_exit+0xe
#6 0xffffffff80bd7ae6 at amd64_syscall+0x546
#7 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 3d4h17m27s

(gdb) l *pmap_remove_pages+0x3a8
0xffffffff80bcffe8 is in pmap_remove_pages (/usr/src/sys/amd64/amd64/pmap.c:4183).
4178	
4179					/*
4180					 * Update the vm_page_t clean/reference bits.
4181					 */
4182					if ((tpte & (PG_M | PG_RW)) == (PG_M | PG_RW)) {
4183						if ((tpte & PG_PS) != 0) {
4184							for (mt = m; mt < &m[NBPDR / PAGE_SIZE]; mt++)
4185								vm_page_dirty(mt);
4186						} else
4187							vm_page_dirty(m);
(gdb) 


Crash 1 (4th of March):

panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff80b49d9a at vmspace_exit+0x9a
#7 0xffffffff808b9d69 at exit1+0x379
#8 0xffffffff808bac3e at sys_sys_exit+0xe
#9 0xffffffff80bd7ae6 at amd64_syscall+0x546
#10 0xffffffff80bc3447 at Xfast_syscall+0xf7

(gdb) l *vmspace_exit+0x9a
0xffffffff80b49d9a is in vmspace_exit (/usr/src/sys/vm/vm_map.c:427).
422			pmap_remove_pages(vmspace_pmap(vm));
423			/* Switch now since this proc will free vmspace */
424			PROC_VMSPACE_LOCK(p);
425			p->p_vmspace = &vmspace0;
426			PROC_VMSPACE_UNLOCK(p);
427			pmap_activate(td);
428			vmspace_dofree(vm);
429		}
430		vmspace_container_reset(p);
431	}
(gdb) 

Crash 2 (also 4th of March):

cpuid = 0
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80b50923 at vm_page_free_toq+0x273
#3 0xffffffff816bd5ba at zfs_freebsd_read+0x62a
#4 0xffffffff8099113d at vn_rdwr+0x1ad
#5 0xffffffff8095c9dd at kern_sendfile+0xdad
#6 0xffffffff8095d12c at do_sendfile+0xdc
#7 0xffffffff80bd7ae6 at amd64_syscall+0x546
#8 0xffffffff80bc3447 at Xfast_syscall+0xf7

(gdb) l *vm_page_free_toq+0x273
0xffffffff80b50923 is in vm_page_free_toq (/usr/src/sys/vm/vm_page.c:1886).
1881	
1882		m->valid = 0;
1883		vm_page_undirty(m);
1884	
1885		if (m->wire_count != 0)
1886			panic("vm_page_free: freeing wired page %p", m);
1887		if (m->hold_count != 0) {
1888			m->flags &= ~PG_ZERO;
1889			vm_page_lock_queues();
1890			vm_page_enqueue(PQ_HOLD, m);
(gdb) 


Crash 3 (also 4th of March):

panic: vm_page_free: freeing busy page 0xfffffe00d5d2db38
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80b50923 at vm_page_free_toq+0x273
#3 0xffffffff816bd5ba at zfs_freebsd_read+0x62a
#4 0xffffffff8099113d at vn_rdwr+0x1ad
#5 0xffffffff8095c9dd at kern_sendfile+0xdad
#6 0xffffffff8095d12c at do_sendfile+0xdc
#7 0xffffffff80bd7ae6 at amd64_syscall+0x546
#8 0xffffffff80bc3447 at Xfast_syscall+0xf7


The last two seems similar, but otherwise no pattern is obvious to me.

core and info text files are available zipped (98KB) for all crashes here:

http://gal.dk/text-files.zip

Cores are availble upon request, their sizes vary:

-rw-------   1 root  wheel  2482114560 Mar  2 15:04 vmcore.0
-rw-------   1 root  wheel  2597650432 Mar  4 03:08 vmcore.1
-rw-------   1 root  wheel  1687846912 Mar  4 12:48 vmcore.2
-rw-------   1 root  wheel  1076121600 Mar  4 13:06 vmcore.3

How-To-Repeat: Occurs by regular production. Machine functions as web and mail server (with MySQL also running on localhost).
Comment 1 Andriy Gapon freebsd_committer freebsd_triage 2013-03-05 10:55:22 UTC
Please try the following patch.

--- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
+++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
@@ -543,7 +543,9 @@ mappedread_sf(vnode_t *vp, int nbytes, uio_t *uio)
 			vm_page_io_finish(pp);
 			vm_page_lock(pp);
 			if (error) {
-				vm_page_free(pp);
+				if (pp->wire_count == 0 && pp->valid == 0 &&
+				    pp->busy == 0 && !(pp->oflags & VPO_BUSY))
+					vm_page_free(pp);
 			} else {
 				pp->valid = VM_PAGE_BITS_ALL;
 				vm_page_activate(pp);

Additionally, if possible, I would like to see the following information from
"Crash 3":
- value of 'pg' in kern_sendfile frame
- output of 'info reg' command in zfs_freebsd_read frame

Thank you.
-- 
Andriy Gapon
Comment 2 Rasmus Skaarup 2013-03-05 16:38:13 UTC
#   kgdb /boot/kernel/kernel /home/crash/vmcore.3
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain =
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for =
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: vm_page_free: freeing busy page 0xfffffe00d5d2db38
cpuid =3D 1
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80b50923 at vm_page_free_toq+0x273
#3 0xffffffff816bd5ba at zfs_freebsd_read+0x62a
#4 0xffffffff8099113d at vn_rdwr+0x1ad
#5 0xffffffff8095c9dd at kern_sendfile+0xdad
#6 0xffffffff8095d12c at do_sendfile+0xdc
#7 0xffffffff80bd7ae6 at amd64_syscall+0x546
#8 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 17m54s
Dumping 1026 out of 5100 =
MB:..2%..11%..21%..32%..41%..52%..61%..71%..82%..91%

Reading symbols from /boot/modules/virtio.ko...Reading symbols from =
/boot/modules/virtio.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio.ko
Reading symbols from /boot/modules/virtio_pci.ko...Reading symbols from =
/boot/modules/virtio_pci.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_pci.ko
Reading symbols from /boot/modules/virtio_blk.ko...Reading symbols from =
/boot/modules/virtio_blk.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_blk.ko
Reading symbols from /boot/modules/if_vtnet.ko...Reading symbols from =
/boot/modules/if_vtnet.ko.symbols...done.
done.
Loaded symbols for /boot/modules/if_vtnet.ko
Reading symbols from /boot/modules/virtio_balloon.ko...Reading symbols =
from /boot/modules/virtio_balloon.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_balloon.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from =
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from =
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from =
/boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from =
/boot/kernel/ipdivert.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipdivert.ko
#0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) bt
#0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=3D260)
    at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of bounds>)
    at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80b50923 in vm_page_free_toq (m=3D0xfffffe00d5d2db38)
    at /usr/src/sys/vm/vm_page.c:1862
#4  0xffffffff816bd5ba in zfs_freebsd_read (ap=3DVariable "ap" is not =
available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_vnops.c:476
#5  0xffffffff8099113d in vn_rdwr (rw=3DUIO_READ, vp=3D0xfffffe01058db780,=
=20
    base=3D0x0, len=3DVariable "len" is not available.
) at vnode_if.h:384
#6  0xffffffff8095c9dd in kern_sendfile (td=3D0xfffffe00986bf8e0,=20
    uap=3D0xffffff8162323bb0, hdr_uio=3D0xfffffe002cf6da00, trl_uio=3D0x0,=
 compat=3D0)
    at /usr/src/sys/kern/uipc_syscalls.c:2115
#7  0xffffffff8095d12c in do_sendfile (td=3D0xfffffe00986bf8e0,=20
    uap=3D0xffffff8162323bb0, compat=3D0) at =
/usr/src/sys/kern/uipc_syscalls.c:1799
#8  0xffffffff80bd7ae6 in amd64_syscall (td=3D0xfffffe00986bf8e0, =
traced=3D0)
    at subr_syscall.c:135
#9  0xffffffff80bc3447 in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:387
#10 0x0000000801feb5cc in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit
[root@dentredje ~]#   kgdb /boot/kernel/kernell /home/crash/vmcore.3
kernel          kernel.symbols =20
[root@dentredje ~]#   kgdb /boot/kernel/kernell /home/crash/vmcore.3
kernel          kernel.symbols =20
[root@dentredje ~]#   kgdb /boot/kernel/kernel /home/crash/vmcore.3
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain =
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for =
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: vm_page_free: freeing busy page 0xfffffe00d5d2db38
cpuid =3D 1
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80b50923 at vm_page_free_toq+0x273
#3 0xffffffff816bd5ba at zfs_freebsd_read+0x62a
#4 0xffffffff8099113d at vn_rdwr+0x1ad
#5 0xffffffff8095c9dd at kern_sendfile+0xdad
#6 0xffffffff8095d12c at do_sendfile+0xdc
#7 0xffffffff80bd7ae6 at amd64_syscall+0x546
#8 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 17m54s
Dumping 1026 out of 5100 =
MB:..2%..11%..21%..32%..41%..52%..61%..71%..82%..91%

Reading symbols from /boot/modules/virtio.ko...Reading symbols from =
/boot/modules/virtio.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio.ko
Reading symbols from /boot/modules/virtio_pci.ko...Reading symbols from =
/boot/modules/virtio_pci.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_pci.ko
Reading symbols from /boot/modules/virtio_blk.ko...Reading symbols from =
/boot/modules/virtio_blk.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_blk.ko
Reading symbols from /boot/modules/if_vtnet.ko...Reading symbols from =
/boot/modules/if_vtnet.ko.symbols...done.
done.
Loaded symbols for /boot/modules/if_vtnet.ko
Reading symbols from /boot/modules/virtio_balloon.ko...Reading symbols =
from /boot/modules/virtio_balloon.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_balloon.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from =
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from =
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from =
/boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from =
/boot/kernel/ipdivert.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipdivert.ko
#0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) bt
#0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=3D260) at =
/usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of bounds>) =
at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80b50923 in vm_page_free_toq (m=3D0xfffffe00d5d2db38) at =
/usr/src/sys/vm/vm_page.c:1862
#4  0xffffffff816bd5ba in zfs_freebsd_read (ap=3DVariable "ap" is not =
available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_vnops.c:476
#5  0xffffffff8099113d in vn_rdwr (rw=3DUIO_READ, vp=3D0xfffffe01058db780,=
 base=3D0x0, len=3DVariable "len" is not available.
) at vnode_if.h:384
#6  0xffffffff8095c9dd in kern_sendfile (td=3D0xfffffe00986bf8e0, =
uap=3D0xffffff8162323bb0, hdr_uio=3D0xfffffe002cf6da00, trl_uio=3D0x0,=20=

    compat=3D0) at /usr/src/sys/kern/uipc_syscalls.c:2115
#7  0xffffffff8095d12c in do_sendfile (td=3D0xfffffe00986bf8e0, =
uap=3D0xffffff8162323bb0, compat=3D0)
    at /usr/src/sys/kern/uipc_syscalls.c:1799
#8  0xffffffff80bd7ae6 in amd64_syscall (td=3D0xfffffe00986bf8e0, =
traced=3D0) at subr_syscall.c:135
#9  0xffffffff80bc3447 in Xfast_syscall () at =
/usr/src/sys/amd64/amd64/exception.S:387
#10 0x0000000801feb5cc in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 6
#6  0xffffffff8095c9dd in kern_sendfile (td=3D0xfffffe00986bf8e0, =
uap=3D0xffffff8162323bb0, hdr_uio=3D0xfffffe002cf6da00, trl_uio=3D0x0,=20=

    compat=3D0) at /usr/src/sys/kern/uipc_syscalls.c:2115
warning: Source file is more recent than executable.

2115					error =3D vn_rdwr(UIO_READ, vp, =
NULL, MAXBSIZE,
(kgdb) frame 6
#6  0xffffffff8095c9dd in kern_sendfile (td=3D0xfffffe00986bf8e0, =
uap=3D0xffffff8162323bb0, hdr_uio=3D0xfffffe002cf6da00, trl_uio=3D0x0,=20=

    compat=3D0) at /usr/src/sys/kern/uipc_syscalls.c:2115
2115					error =3D vn_rdwr(UIO_READ, vp, =
NULL, MAXBSIZE,
(kgdb) print pg
$1 =3D (struct vm_page *) 0xfffffe00d5d2db38
(kgdb) info reg
rax            0x0	0
rbx            0x0	0
rcx            0x0	0
rdx            0x0	0
rsi            0x0	0
rdi            0x0	0
rbp            0xffffff8162323ab0	0xffffff8162323ab0
rsp            0xffffff8162323910	0xffffff8162323910
r8             0x0	0
r9             0x0	0
r10            0x0	0
r11            0x0	0
r12            0xfffffe00d5d2db38	-2195435889864
r13            0xfffffe002c731600	-2198277515776
r14            0xffffff8162323bb0	-543813387344
r15            0xfffffe000e2ea3a0	-2198785317984
rip            0xffffffff8095c9dd	0xffffffff8095c9dd =
<kern_sendfile+3501>
eflags         0x0	0
cs             0x0	0
ss             0x0	0
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
(kgdb)=20




Best regards,
Rasmus skaarup


On 05/03/2013, at 11.55, Andriy Gapon <avg@FreeBSD.org> wrote:

>=20
>=20
> Please try the following patch.
>=20
> --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
> +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
> @@ -543,7 +543,9 @@ mappedread_sf(vnode_t *vp, int nbytes, uio_t *uio)
> 			vm_page_io_finish(pp);
> 			vm_page_lock(pp);
> 			if (error) {
> -				vm_page_free(pp);
> +				if (pp->wire_count =3D=3D 0 && pp->valid =
=3D=3D 0 &&
> +				    pp->busy =3D=3D 0 && !(pp->oflags & =
VPO_BUSY))
> +					vm_page_free(pp);
> 			} else {
> 				pp->valid =3D VM_PAGE_BITS_ALL;
> 				vm_page_activate(pp);
>=20
> Additionally, if possible, I would like to see the following =
information from
> "Crash 3":
> - value of 'pg' in kern_sendfile frame
> - output of 'info reg' command in zfs_freebsd_read frame
>=20
> Thank you.
> --=20
> Andriy Gapon
>=20
Comment 3 Andriy Gapon freebsd_committer freebsd_triage 2013-03-05 16:51:20 UTC
I need 'info reg' from zfs_freebsd_read frame, which frame 4 in kgdb.


-- 
Andriy Gapon
Comment 4 Rasmus Skaarup 2013-03-05 16:56:05 UTC
kgdb) frame 4
#4  0xffffffff816bd5ba in zfs_freebsd_read (ap=3DVariable "ap" is not =
available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_vnops.c:476
warning: Source file is more recent than executable.

476					if (pp->wire_count =3D=3D 0 && =
pp->valid =3D=3D 0 &&
(kgdb) info reg
rax            0x0	0
rbx            0x7a	122
rcx            0x0	0
rdx            0x0	0
rsi            0x0	0
rdi            0x0	0
rbp            0xffffff8162323810	0xffffff8162323810
rsp            0xffffff8162323710	0xffffff8162323710
r8             0x0	0
r9             0x0	0
r10            0x0	0
r11            0x0	0
r12            0xfffffe015123cc80	-2193366987648
r13            0xfffffe0006aa1000	-2198911447040
r14            0xfffffe0006aa10e8	-2198911446808
r15            0xffffff8162323850	-543813388208
rip            0xffffffff816bd5ba	0xffffffff816bd5ba =
<zfs_freebsd_read+1578>
eflags         0x0	0
cs             0x0	0
ss             0x0	0
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
(kgdb)=20


I guess line 476 changed after your patch, but I guess you can use the =
output anyway.

Br
Rasmus Skaarup

On 05/03/2013, at 17.51, Andriy Gapon <avg@FreeBSD.org> wrote:

>=20
> I need 'info reg' from zfs_freebsd_read frame, which frame 4 in kgdb.
>=20
>=20
> --=20
> Andriy Gapon
>=20
Comment 5 Rasmus Skaarup 2013-03-06 12:40:53 UTC
The system has become highly unstable now. It crashes very often, and =
almost every time I issue a zpool command.=20

I tried booting with the patched kernel, but it still crashes. Here is =
the info from the latest crash:

This command crashed the system:

***
[root@dentredje rc.d]# zpool status -v
  pool: data2
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 0h15m with 8 errors on Wed Mar  6 11:28:12 =
2013
config:

	NAME        STATE     READ WRITE CKSUM
	data2       ONLINE       0     0     1
	  vtbd1p1   ONLINE       0     0     2
	  vtbd2p1   ONLINE       0     0     2

errors: Permanent errors have been detected in the following files:

        data2/home:<0x0>
        =
/data2/home/jacobk/www.secretsounds.dk/40days/final_tracks_for_5.1/bob_5/b=
ob_5-start_05_01_46_11-5_of_5.wav
        /data2/home/skaarup/mysql-backup.sql
        data2/src:<0x0>
        data2/src:<0xb>
***

info.txt.6:

[root@dentredje crash]# cat info.6=20
Dump header from device /dev/vtbd0p4
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 893067264B (851 MB)
  Blocksize: 512
  Dumptime: Wed Mar  6 13:29:20 2013
  Hostname: dentredje.dvconsulting.dk
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 9.1-RELEASE #0: Wed Mar  6 11:30:02 CET 2013
    root@denfjerde.dvconsulting.dk:/usr/obj/usr/src/sys/GENERIC
  Panic String: page fault
  Dump Parity: 2869697314
  Bounds: 6
  Dump Status: good
[root@dentredje crash]#=20

core.txt.6

***
[root@dentredje crash]# cat core.txt.6=20
dentredje.dvconsulting.dk dumped core - see /var/crash/vmcore.6

Wed Mar  6 13:32:32 CET 2013

FreeBSD dentredje.dvconsulting.dk 9.1-RELEASE FreeBSD 9.1-RELEASE #0: =
Wed Mar  6 11:30:02 CET 2013     =
root@denfjerde.dvconsulting.dk:/usr/obj/usr/src/sys/GENERIC  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain =
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for =
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address	=3D 0x60
fault code		=3D supervisor read data, page not present
instruction pointer	=3D 0x20:0xffffffff8162e4f0
stack pointer	        =3D 0x28:0xffffff8161b726e0
frame pointer	        =3D 0x28:0xffffff8161b727d0
code segment		=3D base rx0, limit 0xfffff, type 0x1b
			=3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	=3D interrupt enabled, resume, IOPL =3D 0
current process		=3D 2999 (zpool)
trap number		=3D 12
panic: page fault
cpuid =3D 3
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff81673975 at sa_handle_get_from_db+0x95
#7 0xffffffff81673a38 at sa_handle_get+0x48
#8 0xffffffff8169f516 at zfs_grab_sa_handle+0x96
#9 0xffffffff8169faca at zfs_obj_to_path+0x6a
#10 0xffffffff816b8c75 at zfs_ioc_obj_to_path+0x75
#11 0xffffffff816bad46 at zfsdev_ioctl+0xe6
#12 0xffffffff807db28b at devfs_ioctl_f+0x7b
#13 0xffffffff80932325 at kern_ioctl+0x115
#14 0xffffffff8093255d at sys_ioctl+0xfd
#15 0xffffffff80bd7ae6 at amd64_syscall+0x546
#16 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 2m10s
Dumping 851 out of 5100 =
MB:..2%..12%..21%..31%..42%..51%..61%..72%..81%..91%

Reading symbols from /boot/modules/virtio.ko...Reading symbols from =
/boot/modules/virtio.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio.ko
Reading symbols from /boot/modules/virtio_pci.ko...Reading symbols from =
/boot/modules/virtio_pci.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_pci.ko
Reading symbols from /boot/modules/virtio_blk.ko...Reading symbols from =
/boot/modules/virtio_blk.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_blk.ko
Reading symbols from /boot/modules/if_vtnet.ko...Reading symbols from =
/boot/modules/if_vtnet.ko.symbols...done.
done.
Loaded symbols for /boot/modules/if_vtnet.ko
Reading symbols from /boot/modules/virtio_balloon.ko...Reading symbols =
from /boot/modules/virtio_balloon.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_balloon.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from =
/boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from =
/boot/kernel/ipdivert.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipdivert.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from =
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from =
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
#0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=3D260)
    at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of bounds>)
    at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80bd8240 in trap_fatal (frame=3D0xc, eva=3DVariable "eva" =
is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff80bd857d in trap_pfault (frame=3D0xffffff8161b72630, =
usermode=3D0)
    at /usr/src/sys/amd64/amd64/trap.c:773
#5  0xffffffff80bd8b9e in trap (frame=3D0xffffff8161b72630)
    at /usr/src/sys/amd64/amd64/trap.c:456
#6  0xffffffff80bc315f in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff8162e4f0 in list_remove_tail ()
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/list=
.c:151
#8  0xffffffff816736af in sa_build_index (hdl=3D0xfffffe010ac98690,=20
    buftype=3DSA_BONUS)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1490
#9  0xffffffff81673975 in sa_handle_get_from_db (os=3D0xfffffe0132baf000,=20=

    db=3D0xfffffe010ac968c0, userp=3D0x0, hdl_type=3DVariable "hdl_type" =
is not available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1367
#10 0xffffffff81673a38 in sa_handle_get (objset=3D0xfffffe0132baf000, =
objid=3DVariable "objid" is not available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1392
#11 0xffffffff8169f516 in zfs_grab_sa_handle (osp=3D0xfffffe0132baf000,=20=

    obj=3D230809, hdlp=3D0xffffff8161b72908, db=3D0xffffff8161b72900,=20
    tag=3D0xffffffff81722e10)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_znode.c:2031
#12 0xffffffff8169faca in zfs_obj_to_path (osp=3D0xfffffe0132baf000,=20
    obj=3D230809, buf=3D0xffffff8008757400 "data2/src", len=3D2048)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_znode.c:2184
#13 0xffffffff816b8c75 in zfs_ioc_obj_to_path (zc=3D0xffffff8008757000)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_ioctl.c:1537
#14 0xffffffff816bad46 in zfsdev_ioctl (dev=3DVariable "dev" is not =
available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_ioctl.c:5218
#15 0xffffffff807db28b in devfs_ioctl_f (fp=3D0xfffffe006bdf8690,=20
    com=3D3583531558, data=3DVariable "data" is not available.
) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
#16 0xffffffff80932325 in kern_ioctl (td=3DVariable "td" is not =
available.
) at file.h:293
#17 0xffffffff8093255d in sys_ioctl (td=3D0xfffffe00033e58e0,=20
    uap=3D0xffffff8161b72bb0) at /usr/src/sys/kern/sys_generic.c:691
#18 0xffffffff80bd7ae6 in amd64_syscall (td=3D0xfffffe00033e58e0, =
traced=3D0)
    at subr_syscall.c:135
#19 0xffffffff80bc3447 in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:387
#20 0x0000000801debcdc in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)=20

------------------------------------------------------------------------
ps -axl

ps: can't read sigacts at 0xfffffe00a1c64000

------------------------------------------------------------------------
vmstat -s

   392168 cpu context switches
    44454 device interrupts
     8725 software interrupts
  1299960 traps
  1763303 system calls
       19 kernel threads created
     2839  fork() calls
      141 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     4580 vnode pager pageins
    19695 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
      426 pages reactivated
   206213 copy-on-write faults
      470 copy-on-write optimized faults
   947712 zero fill pages zeroed
        0 zero fill pages prezeroed
       86 intransit blocking page faults
  1274475 total VM faults taken
        0 pages affected by kernel thread creation
  1413892 pages affected by  fork()
    70812 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
  1157439 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
   107649 pages active
   203197 pages inactive
       59 pages in VM cache
   180298 pages wired down
   771791 pages free
     4096 bytes per page
   990965 total name lookups
          cache hits (89% pos + 5% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
CAM dev queue     4     1K       -        4  128
  md_sii_data     0     0K       -       42  512
      CAM XPT    29    16K       -       95  32,64,128,1024,2048
       isadev     9     2K       -        9  128
     acpiintr     1     1K       -        1  64
         cdev     8     2K       -        8  256
       acpica  1707   188K       -    21034  =
16,32,64,128,256,512,1024,2048
        sigio     1     1K       -        1  64
     filedesc   234   963K       -     3180  16,32,64,512,1024,2048,4096
         kenv    66    11K       -       70  16,32,64,128
       kqueue    24    27K       -      250  256,512,2048
    proc-args   141    13K       -     1766  16,32,64,128,256
        hhook     2     1K       -        2  128
     acpitask     1     2K       -        1  2048
      ithread    74    12K       -       74  32,128,256
    CAM queue    12     1K       -       71  16,32
       KTRACE   100    13K       -      100  128
       linker   261   897K       -      303  =
16,32,64,128,256,512,1024,2048,4096
        lockf   148    15K       -     1787  64,128,256
   loginclass     2     1K       -        6  64
       ip6ndp    10     1K       -       13  64,128
         temp    20    13K       -    14346  =
16,32,64,128,256,512,1024,2048,4096
       devbuf  3564 27445K       -     3609  =
16,32,64,128,256,512,1024,2048,4096
       module   489    62K       -      489  128
     mtx_pool     2    16K       -        2 =20
       USBdev     4     1K       -        4  64,128
          osd     3     1K       -       16  16,64
          USB     7     3K       -        7  16,32,128,2048
     pmchooks     1     1K       -        1  128
      subproc   410   835K       -     3230  512,4096
         proc     2    16K       -        2 =20
      session    28     4K       -      143  128
         pgrp    38     5K       -      219  128
         cred   170    27K       -    58670  64,256
      uidinfo    24     5K       -      257  128,2048
       plimit    21     6K       -      201  256
      ata_pci     1     1K       -        1  64
    sysctltmp     0     0K       -      890  16,32,64,128,4096
    sysctloid  2923   145K       -     2998  16,32,64,128
       sysctl     0     0K       -      840  16,32,64
      tidhash     1    16K       -        1 =20
      callout     3  1536K       -        3 =20
         umtx   948   119K       -      948  128
     p1003.1b     1     1K       -        1  16
         SWAP     2   549K       -        2  64
       bus-sc    46   105K       -     2000  =
16,32,128,256,512,1024,2048,4096
          bus  1036    83K       -     3883  16,32,64,128,256,1024
      devstat    16    33K       -       16  32,4096
 eventhandler    90     8K       -       90  64,128
      acpisem    16     2K       -       16  128
         kobj   334  1336K       -      692  4096
      CAM SIM     4     1K       -        4  256
      Per-cpu     1     1K       -        1  32
   CAM periph     2     1K       -       18  16,32,64,128,256
         rman   100    12K       -      508  16,32,128
         sbuf     0     0K       -     2357  =
16,32,64,128,256,512,1024,2048,4096
       sglist     4     4K       -        4  1024
      entropy  1024    64K       -     1024  64
       ctlmem  5062 10113K       -     5062  128,2048
        stack     0     0K       -        2  256
    taskqueue    61     6K       -       91  16,32,64,128,1024
       Unitno    16     1K       -     2152  32,64
          iov     0     0K       -     4365  16,64,128,256,512
       select   195    25K       -      195  128
     ioctlops     1     8K       -    15314  =
16,32,64,128,256,512,1024,2048,4096
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        3  2048
          tty    20    20K       -       22  1024,2048
          pts     1     1K       -        1  256
     mbuf_tag     0     0K       -        7  32
        shmfd     1     8K       -        1 =20
          pcb    52   158K       -      809  16,32,128,1024,2048,4096
       soname    11     2K       -     7276  16,32,128
     vfscache     1  2048K       -        1 =20
   cl_savebuf     0     0K       -      310  64
     vfs_hash     1  1024K       -        1 =20
       DEVFS1    80    40K       -      102  512
       vnodes    28     3K       -      108  64,256
       DEVFS3   103    26K       -      132  256
        mount    86     4K       -      169  16,32,64,128,256
  vnodemarker     0     0K       -      120  512
          BPF     6     1K       -        6  128
  ether_multi    79     5K       -       94  16,32,64
       ifaddr    80    21K       -       80  32,64,128,256,512,4096
        ifnet     7    13K       -        7  128,2048
        clone     6    24K       -        6  4096
       arpcom     4     1K       -        4  16
      lltable    26    10K       -       26  256,512
        DEVFS    11     1K       -       12  16,128
     routetbl    56     7K       -      221  32,64,128,256,512
         igmp     6     2K       -        6  256
     in_multi     4     1K       -        4  256
    sctp_iter     0     0K       -       12  256
     sctp_ifn     4     1K       -        4  128
     sctp_ifa    14     2K       -       14  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -       12  16
    hostcache     1    28K       -        1 =20
     syncache     1    96K       -        1 =20
    in6_multi    36     5K       -       36  32,256
          mld     6     1K       -        6  128
          rpc     2     1K       -        2  256
audit_evclass   179     6K       -      218  32
      jblocks     6     2K       -        6  128,256
     savedino     0     0K       -      129  256
        sbdep     0     0K       -       17  64
      jsegdep   106     7K       -    25650  64
         jseg   270    34K       -      315  128
    jfreefrag     0     0K       -      829  128
      jnewblk     0     0K       -    23303  128
      jremref     8     1K       -      735  128
      jaddref     0     0K       -      783  128
      freedep     0     0K       -        4  64
     freework    21     3K       -      583  64,128
    newdirblk     0     0K       -       36  64
       dirrem    46     6K       -      663  128
        mkdir     0     0K       -       72  128
       diradd     1     1K       -      711  128
     freefile     9     1K       -      411  64
     freeblks    17     3K       -      438  128
     freefrag     0     0K       -      829  128
     indirdep     0     0K       -       12  128
       newblk     5   129K       -    23304  256
    bmsafemap     4     9K       -      654  256
     inodedep   113  1080K       -      919  512
      pagedep    52   141K       -      401  256
  ufs_dirhash    93    20K       -       94  16,32,64,128,256,512
    ufs_mount    15    67K       -       15  512,4096
    vm_pgdata     2   129K       -        2  128
      UMAHash     2     1K       -        2  512
      memdesc     1     4K       -        1  4096
     atkbddev     2     1K       -        2  64
    pfs_nodes    21     6K       -       21  256
       ctlblk   200  1600K       -      200 =20
         GEOM   127    26K       -     1541  =
16,32,64,128,256,512,1024,2048
      ramdisk     1  4096K       -        1 =20
      acpidev    18     2K       -       18  64
      ctlpool   532   142K       -      532  32,512
       kbdmux     6    18K       -        6  16,512,1024,2048
          LED     4     1K       -        4  16,128
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
       feeder     7     1K       -        7  32
     pci_link    10     1K       -       10  16,128
    raid_data     0     0K       -      252  32,128,256
      io_apic     1     2K       -        1  2048
          MCA    10     2K       -       10  128
     nexusdev     3     1K       -        3  16
md_nvidia_data     0     0K       -       42  512
  IpFw/IpAcct    20     5K       -       35  16,32,64,128,1024
      solaris 42732 117071K       -   516982  =
16,32,64,128,256,512,1024,2048,4096
   kstat_data     4     1K       -        4  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     188,      16,     188,   0,   0
UMA Zones:              896,      0,     188,       0,     188,   0,   0
UMA Slabs:              568,      0,   10842,       1,   17497,   0,   0
UMA RCntSlabs:          568,      0,     811,       1,     811,   0,   0
UMA Hash:               256,      0,      79,      11,      81,   0,   0
16 Bucket:              152,      0,     140,      10,     140,   0,   0
32 Bucket:              280,      0,     119,       7,     119,   2,   0
64 Bucket:              536,      0,     125,       1,     125,  57,   0
128 Bucket:            1048,      0,     492,       0,     492,1050,   0
VM OBJECT:              232,      0,    8228,     204,   60846,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 182559,      99,     521,   22390,   0,   0
MAP ENTRY:              120,      0,    8567,     330,  147448,   0,   0
fakepg:                 120,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     331,      42,     331,   0,   0
16:                      16,      0,    3858,     342,   53852,   0,   0
32:                      32,      0,    4201,     647,   46199,   0,   0
64:                      64,      0,   23763,   15269,  271306,   0,   0
128:                    128,      0,    6587,     431,  130489,   0,   0
256:                    256,      0,    6884,   15661,  117624,   0,   0
512:                    512,      0,    4884,     128,   76796,   0,   0
1024:                  1024,      0,     845,    1207,    7696,   0,   0
2048:                  2048,      0,    5580,     330,    6845,   0,   0
4096:                  4096,      0,    1816,     131,   39969,   0,   0
Files:                   80,      0,     969,     246,   48815,   0,   0
TURNSTILE:              136,      0,     475,      85,     475,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1184,      0,     179,      49,    2999,   0,   0
THREAD:                1128,      0,     429,      45,     591,   0,   0
SLEEPQUEUE:              80,      0,     475,      76,     475,   0,   0
VMSPACE:                392,      0,     161,      49,    2985,   0,   0
cpuset:                  72,      0,     206,     194,     348,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,     602,     813,   32072,   0,   0
mbuf:                   256,      0,      14,     633,   36538,   0,   0
mbuf_cluster:          2048,  25600,    1408,      60,    1408,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,      77,    1138,   0,   0
mbuf_jumbo_9k:         9216,  19200,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  12800,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,     672,     185,   0,   0
g_bio:                  232,      0,       0,     640,   95668,   0,   0
ttyinq:                 160,      0,     135,      81,     270,   0,   0
ttyoutq:                256,      0,      72,      48,     144,   0,   0
ata_request:            328,      0,       0,       0,       0,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtblk_request:           48,      0,     512,      64,     512,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
VNODE:                  480,      0,    6185,      87,    7601,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
S VFS Cache:            108,      0,    5866,     140,   46996,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,     411,     177,    3463,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       1,      67,  210673,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,     206,      34,     206,   0,   0
Mountpoints:            792,      0,       7,       8,       7,   0,   0
pipe:                   728,      0,      53,      52,    2023,   0,   0
ksiginfo:               112,      0,     214,     842,     290,   0,   0
itimer:                 344,      0,       1,      21,       1,   0,   0
KNOTE:                  128,      0,      69,     105,     342,   0,   0
socket:                 680,  25602,     289,      65,    2277,   0,   0
unpcb:                  240,  25600,     164,      92,     687,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,      29,      71,    1126,   0,   0
udpcb:                   16,  25704,      29,     643,    1126,   0,   0
tcp_inpcb:              392,  25600,     120,      60,     440,   0,   0
tcpcb:                  976,  25600,      85,      27,     440,   0,   0
tcptw:                   72,   5150,      35,     215,      37,   0,   0
syncache:               152,  15375,       1,     124,     186,   0,   0
hostcache:              136,  15372,      57,     111,      57,   0,   0
tcpreass:                40,   1680,      17,     403,     398,   0,   0
sackhole:                32,      0,       0,     303,      26,   0,   0
sctp_ep:               1376,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     288,      13,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       0,      50,      18,   0,   0
rtentry:                200,      0,      34,      80,      34,   0,   0
selfd:                   56,      0,     432,     387,   92002,   0,   0
SWAPMETA:               288, 116519,       0,       0,       0,   0,   0
FFS inode:              168,      0,    3499,     109,    3934,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,    3499,     161,    3926,   0,   0
IPFW dynamic rule:      120,      0,       0,       0,       0,   0,   0
divcb:                  392,  25600,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,     360,      64,   0,   0
zio_cache:              880,      0,       1,    1599,   75260,   0,   0
zio_link_cache:          48,      0,       0,    2448,   37923,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,    2647,     143,    3636,   0,   0
dnode_t:                856,      0,    3641,     111,    4026,   0,   0
dmu_buf_impl_t:         224,      0,    7859,     250,   10606,   0,   0
arc_buf_hdr_t:          216,      0,    6225,      93,    8090,   0,   0
arc_buf_t:              104,      0,    6135,     165,    8127,   0,   0
zil_lwb_cache:          192,      0,       1,      99,      23,   0,   0
zfs_znode_cache:        400,      0,    2646,      45,    3625,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                           8          0
irq10: em0 em1+                    21867        173
irq11: re0 uhci0++*                22579        179
cpu0:timer                         43190        342
cpu3:timer                         31518        250
cpu2:timer                         35619        282
cpu1:timer                         44648        354
Total                             199429       1582

------------------------------------------------------------------------
pstat -T

969/12328 files
0M/4095M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/vtbd0p4       8388352        0  8388352     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
           vtbd0            vtbd1            vtbd2             cpu
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
 83.94 168 13.73   7.24  39  0.28  11.92  30  0.35  13  0  7  1 79

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
CGROUP                 CBYTES                 QNUM               QBYTES  =
      LSPID        LRPID STIME    RTIME    CTIME  =20

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    =
DTIME    CTIME  =20
m        65536   1667461225 --rw-rw-rw- nobody   munin    nobody   munin =
              0        65536         1601         2199 13:28:39 13:28:39 =
13:27:56

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
CGROUP          NSEMS OTIME    CTIME  =20
s        65536   1667461225 --rw-rw-rw- nobody   munin    nobody   munin =
              3 13:28:39 13:27:56


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory =
identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    =
Remove
        0         0         0         0         0         0         0    =
     0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    =
Access
        0         0         0         0         0         0         0    =
     0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    =
Misses
        0         0         0         0         0         0         0    =
     0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    =
Misses
        0         0         0         0         0         0         0    =
     0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    =
Remove
        0         0         0         0         0         0         0    =
     0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    =
Access
        0         0         0         0         0         0         0    =
     0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
	9385 packets sent
		6192 data packets (6331579 bytes)
		99 data packets (127875 bytes) retransmitted
		4 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		2119 ack-only packets (214 delayed)
		0 URG only packets
		0 window probe packets
		768 window update packets
		209 control packets
	10672 packets received
		4817 acks (for 6305478 bytes)
		222 duplicate acks
		0 acks for unsent data
		4579 packets (4011692 bytes) received in-sequence
		34 completely duplicate packets (43596 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		398 out-of-order packets (533435 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		91 window update packets
		78 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		12 discarded due to memory problems
	86 connection requests
	185 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	8 ignored RSTs in the windows
	245 connections established (including accepts)
	321 connections closed (including 40 drops)
		153 connections updated cached RTT on close
		153 connections updated cached RTT variance on close
		7 connections updated cached ssthresh on close
	26 embryonic connections dropped
	4733 segments updated rtt (of 3794 attempts)
	27 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	359 correct ACK header predictions
	2854 correct data packet header predictions
	186 syncache entries added
		0 retransmitted
		1 dupsyn
		0 dropped
		185 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	186 cookies sent
	0 cookies received
	57 hostcache entries added
		0 bucket overflow
	3 SACK recovery episodes
	36 segment rexmits in SACK recovery episodes
	52128 byte rexmits in SACK recovery episodes
	92 SACK options (SACK blocks) received
	503 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	3761 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	1 with no checksum
	1279 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	2482 delivered
	2483 datagrams output
	0 times multicast source filter matched
ip:
	14453 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	14435 packets for this host
	12 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	0 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	13495 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	387 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	746 calls to icmp_error
	0 errors not generated in response to an icmp message
	Output histogram:
		echo reply: 2
		destination unreachable: 746
	0 messages with bad code fields
	0 messages less than the minimum length
	0 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	Input histogram:
		destination unreachable: 7
		echo: 2
	2 message responses generated
	0 invalid return addresses
	0 no return routes
igmp:
	5 messages received
	0 messages received with too few bytes
	0 messages received with wrong TTL
	0 messages received with bad checksum
	5 V1/V2 membership queries received
	0 V3 membership queries received
	0 membership queries received with invalid field(s)
	5 general queries received
	0 group queries received
	0 group-source queries received
	0 group-source queries dropped
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 V3 reports received without Router Alert
	0 membership reports sent
arp:
	9 ARP requests sent
	0 ARP replies sent
	66 ARP requests received
	1 ARP reply received
	67 ARP packets received
	0 total packets dropped due to no ARP entry
	0 ARP entrys timed out
	0 Duplicate IPs seen
ip6:
	0 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	0 packets for this host
	0 packets forwarded
	0 packets not forwardable
	0 redirects sent
	11 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	0 multicast packets which we don't join
	Mbuf statistics:
		0 one mbuf
		3 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not contiguous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	0 failures of source address selection
	Source addresses selection rule applied:
		16 same address
icmp6:
	0 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	Output histogram:
		neighbor solicitation: 3
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		0 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	0 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output

------------------------------------------------------------------------
netstat -m

616/1446/2062 mbufs in use (current/cache/total)
595/873/1468/25600 mbuf clusters in use (current/cache/total/max)
602/813 mbuf+clusters out of packet secondary zone in use =
(current/cache)
0/77/77/12800 4k (page size) jumbo clusters in use =
(current/cache/total/max)
0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
1344K/2415K/3759K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
59 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs Idrop    =
Opkts Oerrs  Coll Drop
usbus     0 <Link#1>                               0     0     0        =
0     0     0    0=20
re0    1500 <Link#2>      00:16:8e:1a:f3:a9        2     0     0        =
1     0     0    0=20
re0    1500 192.168.122.0 192.168.122.3            0     -     -        =
0     -     -    -=20
re0    1500 fe80::216:8ef fe80::216:8eff:fe        0     -     -        =
2     -     -    -=20
em0    1500 <Link#3>      52:54:00:b2:12:8c    13686     0     0    =
12271     0     0    0=20
em0    1500 195.234.155.0 dentredje              921     -     -    =
12265     -     -    -=20
em0    1500 fe80::5054:ff fe80::5054:ff:feb        0     -     -        =
1     -     -    -=20
em0    1500 195.234.155.0 skaarup.org           9594     -     -        =
0     -     -    -=20
em0    1500 195.234.155.0 ns11.skaarupaps.d        6     -     -        =
0     -     -    -=20
em0    1500 195.234.155.0 195.234.155.50           0     -     -        =
0     -     -    -=20
em0    1500 195.234.155.0 ntp.gal.dk            3079     -     -        =
0     -     -    -=20
em0    1500 195.234.155.0 195.234.155.75           0     -     -        =
0     -     -    -=20
em1    1500 <Link#4>      52:54:00:b2:12:a2       15     0     0        =
1     0     0    0=20
em1    1500 10.0.0.0      dentredjeb               0     -     -        =
0     -     -    -=20
em1    1500 fe80::5054:ff fe80::5054:ff:feb        0     -     -        =
1     -     -    -=20
lo0   16384 <Link#5>                             842     0     0      =
842     0     0    0=20
lo0   16384 localhost     ::1                      0     -     -        =
0     -     -    -=20
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        =
0     -     -    -=20
lo0   16384 your-net      localhost              842     -     -      =
842     -     -    -=20
ipfw0 65536 <Link#6>                               0     0     0        =
0     0     0    0=20

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif =
Expire
default            195.234.155.1      UGS         1    12267    em0
10.0.0.0/24        link#4             U           0        0    em1
10.0.0.3           link#4             UHS         0        0    lo0
127.0.0.1          link#5             UH          0      842    lo0
192.168.122.0/24   link#2             U           0        0    re0
192.168.122.3      link#2             UHS         0        0    lo0
195.234.155.0/24   link#3             U           0        0    em0
195.234.155.50     link#3             UHS         0        0    lo0
195.234.155.70     link#3             UHS         0        0    lo0
195.234.155.75     link#3             UHS         0        0    lo0
195.234.155.102    link#3             UHS         0        0    lo0
195.234.155.124    link#3             UHS         0        0    lo0
195.234.155.143    link#3             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags    =
  Netif Expire
::/96                             ::1                           UGRS     =
   lo0
::1                               link#5                        UH       =
   lo0
::ffff:0.0.0.0/96                 ::1                           UGRS     =
   lo0
fe80::/10                         ::1                           UGRS     =
   lo0
fe80::%re0/64                     link#2                        U        =
   re0
fe80::216:8eff:fe1a:f3a9%re0      link#2                        UHS      =
   lo0
fe80::%em0/64                     link#3                        U        =
   em0
fe80::5054:ff:feb2:128c%em0       link#3                        UHS      =
   lo0
fe80::%em1/64                     link#4                        U        =
   em1
fe80::5054:ff:feb2:12a2%em1       link#4                        UHS      =
   lo0
fe80::%lo0/64                     link#5                        U        =
   lo0
fe80::1%lo0                       link#5                        UHS      =
   lo0
ff01::%re0/32                     fe80::216:8eff:fe1a:f3a9%re0  U        =
   re0
ff01::%em0/32                     fe80::5054:ff:feb2:128c%em0   U        =
   em0
ff01::%em1/32                     fe80::5054:ff:feb2:12a2%em1   U        =
   em1
ff01::%lo0/32                     ::1                           U        =
   lo0
ff02::/16                         ::1                           UGRS     =
   lo0
ff02::%re0/32                     fe80::216:8eff:fe1a:f3a9%re0  U        =
   re0
ff02::%em0/32                     fe80::5054:ff:feb2:128c%em0   U        =
   em0
ff02::%em1/32                     fe80::5054:ff:feb2:12a2%em1   U        =
   em1
ff02::%lo0/32                     ::1                           U        =
   lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address  =
  (state)
fffffe010a292000 tcp4       0      0 195.234.155.143.20 =
208.83.137.114.270 ESTABLISHED
fffffe010a28c7a0 tcp4       0      0 195.234.155.143.25 =
81.19.253.131.6340 CLOSE_WAIT
fffffe00a1069000 tcp4       0      0 195.234.155.143.25 =
208.83.137.117.270 CLOSED
fffffe00a106a3d0 tcp4       0      0 195.234.155.70.25  =
81.30.22.111.11067 ESTABLISHED
fffffe01326973d0 tcp4       0      6 195.234.155.143.43 173.194.69.26.25 =
  FIN_WAIT_1
fffffe010a2933d0 tcp4       0      0 195.234.155.143.50 78.156.102.15.25 =
  ESTABLISHED
fffffe00a106a000 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.387 ESTABLISHED
fffffe013219a7a0 tcp4       0      0 127.0.0.1.26       127.0.0.1.11423  =
  ESTABLISHED
fffffe006b41ab70 tcp4       0      0 127.0.0.1.11423    127.0.0.1.26     =
  ESTABLISHED
fffffe00a10693d0 tcp4       0      0 127.0.0.1.26       127.0.0.1.32356  =
  ESTABLISHED
fffffe010a2927a0 tcp4       0      0 127.0.0.1.32356    127.0.0.1.26     =
  ESTABLISHED
fffffe01326977a0 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.329 ESTABLISHED
fffffe010a292b70 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.394 ESTABLISHED
fffffe00a10697a0 tcp4       0      0 195.234.155.143.26 81.19.253.131.25 =
  LAST_ACK
fffffe00a1069b70 tcp4       0      0 195.234.155.70.143 =
178.249.49.187.366 ESTABLISHED
fffffe006bd383d0 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.450 ESTABLISHED
fffffe013214d7a0 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.392 ESTABLISHED
fffffe0006670000 tcp4    1448      0 195.234.155.70.993 =
178.249.49.187.484 ESTABLISHED
fffffe00a1d72b70 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.381 ESTABLISHED
fffffe010a293000 tcp4       0      0 195.234.155.70.143 =
178.249.49.187.400 ESTABLISHED
fffffe01326d0000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59864 ESTABLISHED
fffffe010a28cb70 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59863 ESTABLISHED
fffffe010a2937a0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59862 ESTABLISHED
fffffe010a28d000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59861 ESTABLISHED
fffffe01327306c0 tcp4       0      0 195.234.155.70.25  =
83.77.53.106.12850 TIME_WAIT
fffffe010a28c3d0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59858 ESTABLISHED
fffffe0132730708 tcp4       0      0 195.234.155.70.25  =
85.49.162.228.5748 TIME_WAIT
fffffe010a28d3d0 tcp4       0      0 195.234.155.70.25  =
93.163.242.142.457 ESTABLISHED
fffffe0132730750 tcp4       0      0 195.234.155.70.25  =
78.131.218.227.329 TIME_WAIT
fffffe0132730798 tcp4       0      0 195.234.155.70.25  =
189.6.215.52.60453 TIME_WAIT
fffffe01327307e0 tcp4       0      0 195.234.155.70.25  =
77.229.19.23.55285 TIME_WAIT
fffffe0132730828 tcp4       0      0 195.234.155.70.80  =
88.208.217.154.452 TIME_WAIT
fffffe00066707a0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62467 ESTABLISHED
fffffe00a1d74000 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62466 ESTABLISHED
fffffe00a1d743d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62465 ESTABLISHED
fffffe0132730870 tcp4       0      0 195.234.155.70.25  =
91.39.55.104.31089 TIME_WAIT
fffffe006bd373d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62464 ESTABLISHED
fffffe006bd393d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62463 ESTABLISHED
fffffe006bd387a0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62462 ESTABLISHED
fffffe00a106c3d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62461 ESTABLISHED
fffffe00a106c7a0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62460 ESTABLISHED
fffffe00a1d72000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59837 FIN_WAIT_2
fffffe00a1d723d0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59836 FIN_WAIT_2
fffffe00a1d727a0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59835 FIN_WAIT_2
fffffe00a1d73000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59834 FIN_WAIT_2
fffffe00a1d733d0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59833 FIN_WAIT_2
fffffe00a1d737a0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59832 FIN_WAIT_2
fffffe0132730a20 tcp4       0      0 195.234.155.70.25  =
190.179.18.226.264 TIME_WAIT
fffffe01327309d8 tcp4       0      0 195.234.155.70.25  =
187.54.5.117.2375  TIME_WAIT
fffffe010a473288 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15139  TIME_WAIT
fffffe01327308b8 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15138  TIME_WAIT
fffffe0132730900 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15136  TIME_WAIT
fffffe0132730990 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15135  TIME_WAIT
fffffe0132730948 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15129  TIME_WAIT
fffffe01326db7a0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
fffffe0132684000 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
fffffe006bd377a0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
fffffe0132730a68 tcp4       0      0 195.234.155.70.25  =
84.228.228.52.2942 TIME_WAIT
fffffe01326da7a0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
fffffe01326d03d0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1037 LAST_ACK
fffffe0132730af8 tcp4       0      0 195.234.155.70.25  =
81.202.62.25.61332 TIME_WAIT
fffffe0132730ab0 tcp4       0      0 195.234.155.70.25  =
201.212.179.57.332 TIME_WAIT
fffffe0132730b88 tcp4       0      0 195.234.155.70.80  =
157.55.36.49.20092 TIME_WAIT
fffffe0132730b40 tcp4       0      0 195.234.155.70.25  =
81.34.117.66.50711 TIME_WAIT
fffffe0132730bd0 tcp4       0      0 195.234.155.70.80  =
157.55.35.76.14663 TIME_WAIT
fffffe0132730c18 tcp4       0      0 195.234.155.70.80  =
199.30.20.103.3187 TIME_WAIT
fffffe0132730c60 tcp4       0      0 195.234.155.70.110 =
188.183.53.110.607 TIME_WAIT
fffffe0132730ca8 tcp4       0      0 195.234.155.70.80  =
131.253.38.99.6317 TIME_WAIT
fffffe006bd37b70 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.455 ESTABLISHED
fffffe01326d1dc8 tcp4       0      0 195.234.155.70.25  79.42.1.16.21112 =
  TIME_WAIT
fffffe0132730cf0 tcp4       0      0 195.234.155.70.25  =
5.22.76.122.15314  TIME_WAIT
fffffe006bd38b70 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.332 ESTABLISHED
fffffe006bd39000 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.446 ESTABLISHED
fffffe006bd397a0 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.432 ESTABLISHED
fffffe006bd39b70 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.457 ESTABLISHED
fffffe0132697000 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.448 ESTABLISHED
fffffe0132684b70 tcp4       0      0 195.234.155.70.993 =
87.72.76.248.50286 ESTABLISHED
fffffe0132730d38 tcp4       0      0 195.234.155.70.25  =
189.6.215.52.61730 TIME_WAIT
fffffe010a473240 tcp4       0      0 195.234.155.70.25  =
91.99.166.18.9811  TIME_WAIT
fffffe013214d000 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.422 ESTABLISHED
fffffe0006d95000 tcp4       0      0 127.0.0.1.26       *.*              =
  LISTEN
fffffe01321993d0 tcp4       0      0 *.25               *.*              =
  LISTEN
fffffe01326da000 tcp4       0      0 127.0.0.1.53       *.*              =
  LISTEN
fffffe0006f7db70 tcp4       0      0 *.587              *.*              =
  LISTEN
fffffe013219ab70 tcp4       0      0 127.0.0.1.3306     *.*              =
  LISTEN
fffffe01321997a0 tcp4       0      0 195.234.155.70.25  =
83.49.89.31.58370  FIN_WAIT_2
fffffe00066703d0 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.374 ESTABLISHED
fffffe0006670b70 tcp4       0      0 10.0.0.3.4949      *.*              =
  LISTEN
fffffe01326dab70 tcp46      0      0 *.110              *.*              =
  LISTEN
fffffe006b41a7a0 tcp46      0      0 *.993              *.*              =
  LISTEN
fffffe013214db70 tcp46      0      0 *.143              *.*              =
  LISTEN
fffffe006b469000 tcp4       0      0 195.234.155.70.80  =
65.55.215.54.65193 TIME_WAIT
fffffe0132730dc8 tcp4       0      0 195.234.155.70.25  =
80.122.8.131.65026 TIME_WAIT
fffffe0132730d80 tcp4       0      0 195.234.155.70.25  =
190.179.62.166.249 TIME_WAIT
fffffe01326d07a0 tcp4       0      0 *.443              *.*              =
  LISTEN
fffffe01326d0b70 tcp6       0      0 *.443              *.*              =
  LISTEN
fffffe006b419000 tcp4       0      0 *.82               *.*              =
  LISTEN
fffffe006b4193d0 tcp6       0      0 *.82               *.*              =
  LISTEN
fffffe006b4197a0 tcp4       0      0 *.80               *.*              =
  LISTEN
fffffe006b419b70 tcp6       0      0 *.80               *.*              =
  LISTEN
fffffe006b469048 tcp4       0      0 195.234.155.70.110 =
85.81.181.86.49004 TIME_WAIT
fffffe006b4690d8 tcp4       0      0 195.234.155.70.110 =
85.81.181.86.38322 TIME_WAIT
fffffe006b469090 tcp4       0      0 195.234.155.70.110 =
85.81.181.86.47880 TIME_WAIT
fffffe006b469168 tcp4       0      0 195.234.155.70.25  =
190.138.186.235.19 TIME_WAIT
fffffe006b469120 tcp4       0      0 195.234.155.70.25  =
201.188.10.227.302 TIME_WAIT
fffffe0132199000 tcp4       0     95 195.234.155.70.25  =
212.156.16.66.2593 ESTABLISHED
fffffe013219a000 tcp4       0      0 195.234.155.70.22  =
85.233.252.158.373 ESTABLISHED
fffffe01326db000 tcp4       0      0 *.21               *.*              =
  LISTEN
fffffe013219a3d0 tcp4       0      0 127.0.0.1.51629    127.0.0.1.51635  =
  ESTABLISHED
fffffe0006d953d0 tcp4       0      0 127.0.0.1.51635    127.0.0.1.51629  =
  ESTABLISHED
fffffe01326db3d0 tcp4       0      0 *.13782            *.*              =
  LISTEN
fffffe0132697b70 tcp4       0      0 127.0.0.1.51629    127.0.0.1.51632  =
  ESTABLISHED
fffffe01326dbb70 tcp4       0      0 127.0.0.1.51632    127.0.0.1.51629  =
  ESTABLISHED
fffffe0006f7d3d0 tcp4       0      0 *.13724            *.*              =
  LISTEN
fffffe01326843d0 tcp4       0      0 *.22               *.*              =
  LISTEN
fffffe01326847a0 tcp6       0      0 *.22               *.*              =
  LISTEN
fffffe0006f7d000 tcp4       0      0 127.0.0.1.51629    *.*              =
  LISTEN
fffffe0132153000 tcp4       0      0 127.0.0.1.1557     *.*              =
  LISTEN
fffffe0006d95b70 tcp4       0      0 *.1556             *.*              =
  LISTEN
fffffe01321533d0 tcp6       0      0 *.1556             *.*              =
  LISTEN
fffffe0006342dc8 udp4       0      0 *.15949            *.*              =
 =20
fffffe0006dcf930 udp6       0      0 *.*                *.*              =
 =20
fffffe0006da2dc8 udp4       0      0 *.25350            *.*              =
 =20
fffffe006b328498 udp6       0      0 *.*                *.*              =
 =20
fffffe0006da2c40 udp4       0      0 195.234.155.143.58 8.8.8.8.53       =
 =20
fffffe0006370000 udp4     631      0 195.234.155.143.95 8.8.8.8.53       =
 =20
fffffe0006da2ab8 udp4       0      0 127.0.0.1.53       *.*              =
 =20
fffffe0006375ab8 udp4       0      0 195.234.155.102.53 *.*              =
 =20
fffffe00063497a8 udp4     149      0 *.56674            *.*              =
 =20
fffffe0006370dc8 udp4     115      0 *.64883            *.*              =
 =20
fffffe0006342498 udp4     128      0 *.43982            *.*              =
 =20
fffffe0006dcfc40 udp4       0      0 127.0.0.1.123      *.*              =
 =20
fffffe0006dcfdc8 udp6       0      0 fe80:5::1.123      *.*              =
 =20
fffffe0006da2188 udp6       0      0 ::1.123            *.*              =
 =20
fffffe0006da2498 udp6       0      0 fe80:4::5054:ff:.1 *.*              =
 =20
fffffe0006371620 udp4       0      0 10.0.0.3.123       *.*              =
 =20
fffffe0006371ab8 udp4       0      0 195.234.155.75.123 *.*              =
 =20
fffffe0006375310 udp4       0      0 195.234.155.124.12 *.*              =
 =20
fffffe0006375188 udp4       0      0 195.234.155.50.123 *.*              =
 =20
fffffe0006375000 udp4       0      0 195.234.155.102.12 *.*              =
 =20
fffffe0006371dc8 udp4       0      0 195.234.155.70.123 *.*              =
 =20
fffffe0006371c40 udp6       0      0 fe80:3::5054:ff:.1 *.*              =
 =20
fffffe0006371930 udp4       0      0 195.234.155.143.12 *.*              =
 =20
fffffe00063717a8 udp6       0      0 fe80:2::216:8eff.1 *.*              =
 =20
fffffe0006371188 udp4       0      0 192.168.122.3.123  *.*              =
 =20
fffffe0006371310 udp6       0      0 *.123              *.*              =
 =20
fffffe0006371498 udp4       0      0 *.123              *.*              =
 =20
fffffe0006375c40 udp4       0      0 *.514              *.*              =
 =20
fffffe0006375dc8 udp6       0      0 *.514              *.*              =
 =20
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffffe010a3f0690 stream      0      0        0 fffffe006be66e10        0 =
       0
fffffe006be66e10 stream      0      0        0 fffffe010a3f0690        0 =
       0
fffffe006be66c30 stream      0      0        0        0        0        =
0
fffffe010ab1dd20 stream      0      0        0 fffffe010ab1de10        0 =
       0
fffffe010ab1de10 stream      0      0        0 fffffe010ab1dd20        0 =
       0
fffffe010ab1e0f0 stream      0      0        0        0        0        =
0
fffffe006be67000 stream      0      0        0 fffffe006be674b0        0 =
       0
fffffe006be674b0 stream      0      0        0 fffffe006be67000        0 =
       0
fffffe006be67e10 stream      0      0        0        0        0        =
0
fffffe010ab69000 stream      0      0        0 fffffe010ab690f0        0 =
       0
fffffe010ab690f0 stream      0      0        0 fffffe010ab69000        0 =
       0
fffffe000662ac30 stream      0      0        0        0        0        =
0
fffffe000662a690 stream      0      0        0        0        0        =
0
fffffe006be665a0 stream      0      0        0        0        0        =
0
fffffe006be66780 stream      0      0        0 fffffe006be67780        0 =
       0
fffffe006be67780 stream      0      0        0 fffffe006be66780        0 =
       0
fffffe006b45d960 stream      0      0        0 fffffe00065703c0        0 =
       0
fffffe00065703c0 stream      0      0        0 fffffe006b45d960        0 =
       0
fffffe006be67870 stream      0      0        0 fffffe000656f4b0        0 =
       0
fffffe000656f4b0 stream      0      0        0 fffffe006be67870        0 =
       0
fffffe0006570960 stream      0      0        0        0        0        =
0
fffffe006be67d20 stream      0      0        0        0        0        =
0
fffffe01321d13c0 stream      0      0        0        0        0        =
0
fffffe006be66960 stream      0      0        0 fffffe006be661e0        0 =
       0
fffffe006be661e0 stream      0      0        0 fffffe006be66960        0 =
       0
fffffe01321d1a50 stream      0      0        0 fffffe006be67960        0 =
       0
fffffe006be67960 stream      0      0        0 fffffe01321d1a50        0 =
       0
fffffe006be662d0 stream      0      0        0 fffffe000656fd20        0 =
       0
fffffe000656fd20 stream      0      0        0 fffffe006be662d0        0 =
       0
fffffe01321d12d0 stream      0      0        0 fffffe01321d1960        0 =
       0
fffffe01321d1960 stream      0      0        0 fffffe01321d12d0        0 =
       0
fffffe010a3f04b0 stream      0      0        0        0        0        =
0
fffffe00065705a0 stream      0      0        0 fffffe01321d1d20        0 =
       0
fffffe01321d1d20 stream      0      0        0 fffffe00065705a0        0 =
       0
fffffe00066e3870 stream      0      0        0        0        0        =
0
fffffe006be66690 stream      0      0        0        0        0        =
0
fffffe010a3f00f0 stream      0      0        0 fffffe010a3f01e0        0 =
       0
fffffe010a3f01e0 stream      0      0        0 fffffe010a3f00f0        0 =
       0
fffffe006be66d20 stream      0      0        0        0        0        =
0
fffffe010a3f02d0 stream      0      0        0 fffffe010a3f03c0        0 =
       0
fffffe010a3f03c0 stream      0      0        0 fffffe010a3f02d0        0 =
       0
fffffe010a3f05a0 stream      0      0        0        0        0        =
0
fffffe00066e34b0 stream      0      0        0 fffffe006b45d3c0        0 =
       0
fffffe006b45d3c0 stream      0      0        0 fffffe00066e34b0        0 =
       0
fffffe00066e3000 stream      0      0        0 fffffe000662a5a0        0 =
       0
fffffe000662a5a0 stream      0      0        0 fffffe00066e3000        0 =
       0
fffffe010a3f0780 stream      0      0        0        0        0        =
0
fffffe006b45dc30 stream      0      0        0        0        0        =
0
fffffe006b45dd20 stream      0      0        0 fffffe006b45de10        0 =
       0
fffffe006b45de10 stream      0      0        0 fffffe006b45dd20        0 =
       0
fffffe010a3f0870 stream      0      0        0 fffffe010a3f0960        0 =
       0
fffffe010a3f0960 stream      0      0        0 fffffe010a3f0870        0 =
       0
fffffe0132a5dc30 stream      0      0        0        0        0        =
0
fffffe010a3f0a50 stream      0      0        0 fffffe010a3f0b40        0 =
       0
fffffe010a3f0b40 stream      0      0        0 fffffe010a3f0a50        0 =
       0
fffffe00066e33c0 stream      0      0        0        0        0        =
0
fffffe00065700f0 stream      0      0        0 fffffe006b45d4b0        0 =
       0
fffffe006b45d4b0 stream      0      0        0 fffffe00065700f0        0 =
       0
fffffe006b45d870 stream      0      0        0        0        0        =
0
fffffe006be663c0 stream      0      0        0 fffffe006be664b0        0 =
       0
fffffe006be664b0 stream      0      0        0 fffffe006be663c0        0 =
       0
fffffe00064cf4b0 stream      0      0        0        0        0        =
0
fffffe000662a4b0 stream      0      0        0 fffffe000662ab40        0 =
       0
fffffe000662ab40 stream      0      0        0 fffffe000662a4b0        0 =
       0
fffffe006be66a50 stream      0      0        0        0        0        =
0
fffffe000656fa50 stream      0      0        0 fffffe00066e30f0        0 =
       0
fffffe00066e30f0 stream      0      0        0 fffffe000656fa50        0 =
       0
fffffe00064cfd20 stream      0      0        0        0        0        =
0
fffffe006b45d5a0 stream      0      0        0 fffffe0132a5d1e0        0 =
       0
fffffe0132a5d1e0 stream      0      0        0 fffffe006b45d5a0        0 =
       0
fffffe0006570e10 stream      0      0        0        0        0        =
0
fffffe006b45da50 stream      0      0        0        0        0        =
0
fffffe00064cf2d0 stream      0      0        0 fffffe00064cf1e0        0 =
       0
fffffe00064cf1e0 stream      0      0        0 fffffe00064cf2d0        0 =
       0
fffffe006b45d690 stream      0      0        0        0        0        =
0
fffffe006be671e0 stream      0      0        0 fffffe006be672d0        0 =
       0
fffffe006be672d0 stream      0      0        0 fffffe006be671e0        0 =
       0
fffffe0006570b40 stream      0      0        0 fffffe00066e35a0        0 =
       0
fffffe00066e35a0 stream      0      0        0 fffffe0006570b40        0 =
       0
fffffe000662a3c0 stream      0      0        0        0        0        =
0
fffffe0132a5db40 stream      0      0        0 fffffe00066e3690        0 =
       0
fffffe00066e3690 stream      0      0        0 fffffe0132a5db40        0 =
       0
fffffe0006570c30 stream      0      0        0        0        0        =
0
fffffe0132a5d000 stream      0      0        0 fffffe01321d1e10        0 =
       0
fffffe01321d1e10 stream      0      0        0 fffffe0132a5d000        0 =
       0
fffffe0006570d20 stream      0      0        0        0        0        =
0
fffffe00066e31e0 stream      0      0        0 fffffe0006570a50        0 =
       0
fffffe0006570a50 stream      0      0        0 fffffe00066e31e0        0 =
       0
fffffe00064cf5a0 stream      0      0        0        0        0        =
0
fffffe000662a2d0 stream      0      0        0 fffffe01321d10f0        0 =
       0
fffffe01321d10f0 stream      0      0        0 fffffe000662a2d0        0 =
       0
fffffe000656f2d0 stream      0      0        0 fffffe000656f1e0        0 =
       0
fffffe000656f1e0 stream      0      0        0 fffffe000656f2d0        0 =
       0
fffffe00064cf0f0 stream      0      0        0 fffffe00064cf000        0 =
       0
fffffe00064cf000 stream      0      0        0 fffffe00064cf0f0        0 =
       0
fffffe000662ae10 stream      0      0        0 fffffe000662ad20        0 =
       0
fffffe000662ad20 stream      0      0        0 fffffe000662ae10        0 =
       0
fffffe0132a5e0f0 stream      0      0        0 fffffe0132a5e000        0 =
       0
fffffe0132a5e000 stream      0      0        0 fffffe0132a5e0f0        0 =
       0
fffffe000656f780 stream      0      0        0 fffffe000662a1e0        0 =
       0
fffffe000662a1e0 stream      0      0        0 fffffe000656f780        0 =
       0
fffffe00065704b0 stream      0      0        0 fffffe0132a5d5a0        0 =
       0
fffffe0132a5d5a0 stream      0      0        0 fffffe00065704b0        0 =
       0
fffffe000656f0f0 stream      0      0        0 fffffe00064cfa50        0 =
       0
fffffe00064cfa50 stream      0      0        0 fffffe000656f0f0        0 =
       0
fffffe00064cfe10 stream      0      0        0 fffffe00064cfb40        0 =
       0
fffffe00064cfb40 stream      0      0        0 fffffe00064cfe10        0 =
       0
fffffe0132a5d4b0 stream      0      0        0 fffffe0132a5d3c0        0 =
       0
fffffe0132a5d3c0 stream      0      0        0 fffffe0132a5d4b0        0 =
       0
fffffe0132a5d2d0 stream      0      0        0 fffffe000656f000        0 =
       0
fffffe000656f000 stream      0      0        0 fffffe0132a5d2d0        0 =
       0
fffffe000662aa50 stream      0      0        0 fffffe000662a960        0 =
       0
fffffe000662a960 stream      0      0        0 fffffe000662aa50        0 =
       0
fffffe000662a870 stream      0      0        0 fffffe000662a780        0 =
       0
fffffe000662a780 stream      0      0        0 fffffe000662a870        0 =
       0
fffffe0132a5e2d0 stream      0      0        0 fffffe0132a5e3c0        0 =
       0
fffffe0132a5e3c0 stream      0      0        0 fffffe0132a5e2d0        0 =
       0
fffffe00065702d0 stream      0      0        0 fffffe01321d14b0        0 =
       0
fffffe01321d14b0 stream      0      0        0 fffffe00065702d0        0 =
       0
fffffe0132a5d960 stream      0      0        0 fffffe00066e3a50        0 =
       0
fffffe00066e3a50 stream      0      0        0 fffffe0132a5d960        0 =
       0
fffffe0132a5e4b0 stream      0      0        0 fffffe0132a5e5a0        0 =
       0
fffffe0132a5e5a0 stream      0      0        0 fffffe0132a5e4b0        0 =
       0
fffffe00066e3b40 stream      0      0        0 fffffe000662a0f0        0 =
       0
fffffe000662a0f0 stream      0      0        0 fffffe00066e3b40        0 =
       0
fffffe0132a5e690 stream      0      0        0 fffffe0132a5e780        0 =
       0
fffffe0132a5e780 stream      0      0        0 fffffe0132a5e690        0 =
       0
fffffe000656fc30 stream      0      0        0 fffffe0132a5da50        0 =
       0
fffffe0132a5da50 stream      0      0        0 fffffe000656fc30        0 =
       0
fffffe00064cf3c0 stream      0      0        0 fffffe0132a5de10        0 =
       0
fffffe0132a5de10 stream      0      0        0 fffffe00064cf3c0        0 =
       0
fffffe0132a5ee10 stream      0      0        0 fffffe006b45d1e0        0 =
       0
fffffe006b45d1e0 stream      0      0        0 fffffe0132a5ee10        0 =
       0
fffffe006b45d0f0 stream      0      0        0 fffffe01321d11e0        0 =
       0
fffffe01321d11e0 stream      0      0        0 fffffe006b45d0f0        0 =
       0
fffffe0132a5e870 stream      0      0        0 fffffe0132a5e960        0 =
       0
fffffe0132a5e960 stream      0      0        0 fffffe0132a5e870        0 =
       0
fffffe0132a5ea50 stream      0      0        0 fffffe0132a5eb40        0 =
       0
fffffe0132a5eb40 stream      0      0        0 fffffe0132a5ea50        0 =
       0
fffffe0132a5ec30 stream      0      0 fffffe006b4e83c0        0        0 =
       0 /tmp/mysql.sock
fffffe000656fb40 stream      0      0        0        0        0        =
0
fffffe0132a5e1e0 stream      0      0        0 fffffe0132a5d780        0 =
       0
fffffe0132a5d780 stream      0      0        0 fffffe0132a5e1e0        0 =
       0
fffffe01321d1b40 stream      0      0 fffffe006b4e5d20        0        0 =
       0 /var/run/authdaemond/socket.tmp
fffffe0132a5d690 stream      0      0 fffffe006b1fc000        0        0 =
       0 /var/run/clamav/clamd
fffffe000662a000 stream      0      0        0 fffffe01321d1870        0 =
       0
fffffe01321d1870 stream      0      0        0 fffffe000662a000        0 =
       0
fffffe00066e3c30 stream      0      0        0 fffffe00066e3d20        0 =
       0 /tmp/PBXPIPEbpcd
fffffe00066e3d20 stream      0      0        0 fffffe00066e3c30        0 =
       0
fffffe000656f690 stream      0      0 fffffe0132732780        0        0 =
       0 /usr/openv/var/vnetd/bpcd.uds
fffffe01321d1690 stream      0      0 fffffe01326b53c0        0        0 =
       0 /usr/openv/var/vnetd/terminate_bpcd.uds
fffffe00064cf960 stream      0      0        0 fffffe00065701e0        0 =
       0 /tmp/PBXPIPEvnetd
fffffe00065701e0 stream      0      0        0 fffffe00064cf960        0 =
       0
fffffe0006570000 stream      0      0 fffffe0132652000        0        0 =
       0 /usr/openv/var/vnetd/terminate_vnetd.uds
fffffe00066e3e10 stream      0      0        0 fffffe01321d1000        0 =
       0
fffffe01321d1000 stream      0      0        0 fffffe00066e3e10        0 =
       0
fffffe000656fe10 stream      0      0 fffffe0006381960        0        0 =
       0 /var/run/devd.pipe
fffffe00064cf780 dgram       0      0        0 fffffe000656f870        0 =
fffffe00066e3780
fffffe00066e3780 dgram       0      0        0 fffffe000656f870        0 =
fffffe0132a5dd20
fffffe0132a5dd20 dgram       0      0        0 fffffe000656f870        0 =
fffffe00064cf870
fffffe00064cf870 dgram       0      0        0 fffffe000656f870        0 =
fffffe000656f5a0
fffffe000656f5a0 dgram       0      0        0 fffffe000656f870        0 =
       0
fffffe000656f870 dgram       0      0 fffffe00067035a0        0 =
fffffe00064cf780        0 /var/run/logpriv
fffffe000656f960 dgram       0      0 fffffe0006703780        0        0 =
       0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address        =20
tcp4  0/0/20         localhost.26          =20
tcp4  0/0/20         *.smtp                =20
tcp4  0/0/20         localhost.domain      =20
tcp4  0/0/50         *.submission          =20
tcp4  0/0/50         localhost.mysql       =20
tcp4  0/0/128        dentredjeb.4949       =20
tcp46 0/0/128        *.pop3                =20
tcp46 0/0/128        *.imaps               =20
tcp46 0/0/128        *.imap                =20
tcp4  0/0/128        *.https               =20
tcp6  0/0/128        *.https               =20
tcp4  0/0/128        *.xfer                =20
tcp6  0/0/128        *.xfer                =20
tcp4  0/0/128        *.http                =20
tcp6  0/0/128        *.http                =20
tcp4  0/0/64         *.ftp                 =20
tcp4  0/0/5          *.13782               =20
tcp4  0/0/5          *.13724               =20
tcp4  0/0/128        *.ssh                 =20
tcp6  0/0/128        *.ssh                 =20
tcp4  0/0/5          localhost.51629       =20
tcp4  0/0/5          localhost.1557        =20
tcp4  0/0/5          *.1556                =20
tcp6  0/0/5          *.1556                =20
unix  0/0/50         /tmp/mysql.sock
unix  0/0/128        /var/run/authdaemond/socket.tmp
unix  0/0/128        /var/run/clamav/clamd
unix  0/0/5          /usr/openv/var/vnetd/bpcd.uds
unix  0/0/1          /usr/openv/var/vnetd/terminate_bpcd.uds
unix  0/0/1          /usr/openv/var/vnetd/terminate_vnetd.uds
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat
[root@dentredje crash]#
***


On 05/03/2013, at 17.56, Rasmus Skaarup <freebsd@gal.dk> wrote:

>=20
> kgdb) frame 4
> #4  0xffffffff816bd5ba in zfs_freebsd_read (ap=3DVariable "ap" is not =
available.
> )
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_vnops.c:476
> warning: Source file is more recent than executable.
>=20
> 476					if (pp->wire_count =3D=3D 0 && =
pp->valid =3D=3D 0 &&
> (kgdb) info reg
> rax            0x0	0
> rbx            0x7a	122
> rcx            0x0	0
> rdx            0x0	0
> rsi            0x0	0
> rdi            0x0	0
> rbp            0xffffff8162323810	0xffffff8162323810
> rsp            0xffffff8162323710	0xffffff8162323710
> r8             0x0	0
> r9             0x0	0
> r10            0x0	0
> r11            0x0	0
> r12            0xfffffe015123cc80	-2193366987648
> r13            0xfffffe0006aa1000	-2198911447040
> r14            0xfffffe0006aa10e8	-2198911446808
> r15            0xffffff8162323850	-543813388208
> rip            0xffffffff816bd5ba	0xffffffff816bd5ba =
<zfs_freebsd_read+1578>
> eflags         0x0	0
> cs             0x0	0
> ss             0x0	0
> ds             0x0	0
> es             0x0	0
> fs             0x0	0
> gs             0x0	0
> (kgdb)=20
>=20
>=20
> I guess line 476 changed after your patch, but I guess you can use the =
output anyway.
>=20
> Br
> Rasmus Skaarup
>=20
> On 05/03/2013, at 17.51, Andriy Gapon <avg@FreeBSD.org> wrote:
>=20
>>=20
>> I need 'info reg' from zfs_freebsd_read frame, which frame 4 in kgdb.
>>=20
>>=20
>> --=20
>> Andriy Gapon
>>=20
>=20
Comment 6 Rasmus Skaarup 2013-03-06 12:44:11 UTC
More info:

[root@dentredje ~]# kgdb /boot/kernel/kernel /var/crash/vmcore.6=20
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain =
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for =
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address	=3D 0x60
fault code		=3D supervisor read data, page not present
instruction pointer	=3D 0x20:0xffffffff8162e4f0
stack pointer	        =3D 0x28:0xffffff8161b726e0
frame pointer	        =3D 0x28:0xffffff8161b727d0
code segment		=3D base rx0, limit 0xfffff, type 0x1b
			=3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	=3D interrupt enabled, resume, IOPL =3D 0
current process		=3D 2999 (zpool)
trap number		=3D 12
panic: page fault
cpuid =3D 3
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff81673975 at sa_handle_get_from_db+0x95
#7 0xffffffff81673a38 at sa_handle_get+0x48
#8 0xffffffff8169f516 at zfs_grab_sa_handle+0x96
#9 0xffffffff8169faca at zfs_obj_to_path+0x6a
#10 0xffffffff816b8c75 at zfs_ioc_obj_to_path+0x75
#11 0xffffffff816bad46 at zfsdev_ioctl+0xe6
#12 0xffffffff807db28b at devfs_ioctl_f+0x7b
#13 0xffffffff80932325 at kern_ioctl+0x115
#14 0xffffffff8093255d at sys_ioctl+0xfd
#15 0xffffffff80bd7ae6 at amd64_syscall+0x546
#16 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 2m10s
Dumping 851 out of 5100 =
MB:..2%..12%..21%..31%..42%..51%..61%..72%..81%..91%

Reading symbols from /boot/modules/virtio.ko...Reading symbols from =
/boot/modules/virtio.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio.ko
Reading symbols from /boot/modules/virtio_pci.ko...Reading symbols from =
/boot/modules/virtio_pci.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_pci.ko
Reading symbols from /boot/modules/virtio_blk.ko...Reading symbols from =
/boot/modules/virtio_blk.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_blk.ko
Reading symbols from /boot/modules/if_vtnet.ko...Reading symbols from =
/boot/modules/if_vtnet.ko.symbols...done.
done.
Loaded symbols for /boot/modules/if_vtnet.ko
Reading symbols from /boot/modules/virtio_balloon.ko...Reading symbols =
from /boot/modules/virtio_balloon.ko.symbols...done.
done.
Loaded symbols for /boot/modules/virtio_balloon.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from =
/boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from =
/boot/kernel/ipdivert.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipdivert.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from =
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from =
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
#0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) bt
#0  doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=3D260) at =
/usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of bounds>)
    at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80bd8240 in trap_fatal (frame=3D0xc, eva=3DVariable "eva" =
is not available.
) at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff80bd857d in trap_pfault (frame=3D0xffffff8161b72630, =
usermode=3D0)
    at /usr/src/sys/amd64/amd64/trap.c:773
#5  0xffffffff80bd8b9e in trap (frame=3D0xffffff8161b72630) at =
/usr/src/sys/amd64/amd64/trap.c:456
#6  0xffffffff80bc315f in calltrap () at =
/usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff8162e4f0 in list_remove_tail ()
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/list=
.c:151
#8  0xffffffff816736af in sa_build_index (hdl=3D0xfffffe010ac98690, =
buftype=3DSA_BONUS)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1490
#9  0xffffffff81673975 in sa_handle_get_from_db (os=3D0xfffffe0132baf000, =
db=3D0xfffffe010ac968c0,=20
    userp=3D0x0, hdl_type=3DVariable "hdl_type" is not available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1367
#10 0xffffffff81673a38 in sa_handle_get (objset=3D0xfffffe0132baf000, =
objid=3DVariable "objid" is not available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1392
#11 0xffffffff8169f516 in zfs_grab_sa_handle (osp=3D0xfffffe0132baf000, =
obj=3D230809,=20
    hdlp=3D0xffffff8161b72908, db=3D0xffffff8161b72900, =
tag=3D0xffffffff81722e10)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_znode.c:2031
#12 0xffffffff8169faca in zfs_obj_to_path (osp=3D0xfffffe0132baf000, =
obj=3D230809,=20
    buf=3D0xffffff8008757400 "data2/src", len=3D2048)
---Type <return> to continue, or q <return> to quit---q
 at /usr/src/sys/modules/zfs/../../cddl/contrib/oQuit
(kgdb) frame 9
#9  0xffffffff81673975 in sa_handle_get_from_db (os=3D0xfffffe0132baf000, =
db=3D0xfffffe010ac968c0,=20
    userp=3D0x0, hdl_type=3DVariable "hdl_type" is not available.
)
    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1367
warning: Source file is more recent than executable.

1367			error =3D sa_build_index(handle, SA_BONUS);
(kgdb) info reg
rax            0x50	80
rbx            0xfffffe010ac98690	-2194547308912
rcx            0xfffffe00a1a45300	-2196311354624
rdx            0xffffff8161b72798	-543821453416
rsi            0xfffffe00a1a45300	-2196311354624
rdi            0x50	80
rbp            0xffffff8161b72810	0xffffff8161b72810
rsp            0xffffff8161b727e0	0xffffff8161b727e0
r8             0x700000	7340032
r9             0x720000	7471104
r10            0x10700000	275775488
r11            0xffffffff81675190	-2123935344
r12            0xfffffe010ac968c0	-2194547316544
r13            0x1	1
r14            0x0	0
r15            0xfffffe0132baf000	-2193877176320
rip            0xffffffff81673975	0xffffffff81673975 =
<sa_handle_get_from_db+149>
eflags         0x10282	66178
cs             0x20	32
ss             0x28	40
ds             0x0	0
es             0x0	0
fs             0x0	0
---Type <return> to continue, or q <return> to quit---=20
gs             0x0	0
(kgdb)=20


Best regards,
Rasmus skaarup


On 06/03/2013, at 13.40, Rasmus Skaarup <freebsd@gal.dk> wrote:

>=20
> The system has become highly unstable now. It crashes very often, and =
almost every time I issue a zpool command.=20
>=20
> I tried booting with the patched kernel, but it still crashes. Here is =
the info from the latest crash:
>=20
> This command crashed the system:
>=20
> ***
> [root@dentredje rc.d]# zpool status -v
>  pool: data2
> state: ONLINE
> status: One or more devices has experienced an error resulting in data
> 	corruption.  Applications may be affected.
> action: Restore the file in question if possible.  Otherwise restore =
the
> 	entire pool from backup.
>   see: http://illumos.org/msg/ZFS-8000-8A
>  scan: scrub repaired 0 in 0h15m with 8 errors on Wed Mar  6 11:28:12 =
2013
> config:
>=20
> 	NAME        STATE     READ WRITE CKSUM
> 	data2       ONLINE       0     0     1
> 	  vtbd1p1   ONLINE       0     0     2
> 	  vtbd2p1   ONLINE       0     0     2
>=20
> errors: Permanent errors have been detected in the following files:
>=20
>        data2/home:<0x0>
>        =
/data2/home/jacobk/www.secretsounds.dk/40days/final_tracks_for_5.1/bob_5/b=
ob_5-start_05_01_46_11-5_of_5.wav
>        /data2/home/skaarup/mysql-backup.sql
>        data2/src:<0x0>
>        data2/src:<0xb>
> ***
>=20
> info.txt.6:
>=20
> [root@dentredje crash]# cat info.6=20
> Dump header from device /dev/vtbd0p4
>  Architecture: amd64
>  Architecture Version: 2
>  Dump Length: 893067264B (851 MB)
>  Blocksize: 512
>  Dumptime: Wed Mar  6 13:29:20 2013
>  Hostname: dentredje.dvconsulting.dk
>  Magic: FreeBSD Kernel Dump
>  Version String: FreeBSD 9.1-RELEASE #0: Wed Mar  6 11:30:02 CET 2013
>    root@denfjerde.dvconsulting.dk:/usr/obj/usr/src/sys/GENERIC
>  Panic String: page fault
>  Dump Parity: 2869697314
>  Bounds: 6
>  Dump Status: good
> [root@dentredje crash]#=20
>=20
> core.txt.6
>=20
> ***
> [root@dentredje crash]# cat core.txt.6=20
> dentredje.dvconsulting.dk dumped core - see /var/crash/vmcore.6
>=20
> Wed Mar  6 13:32:32 CET 2013
>=20
> FreeBSD dentredje.dvconsulting.dk 9.1-RELEASE FreeBSD 9.1-RELEASE #0: =
Wed Mar  6 11:30:02 CET 2013     =
root@denfjerde.dvconsulting.dk:/usr/obj/usr/src/sys/GENERIC  amd64
>=20
> panic: page fault
>=20
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and =
you are
> welcome to change it and/or distribute copies of it under certain =
conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for =
details.
> This GDB was configured as "amd64-marcel-freebsd"...
>=20
> Unread portion of the kernel message buffer:
>=20
>=20
> Fatal trap 12: page fault while in kernel mode
> cpuid =3D 0; apic id =3D 00
> fault virtual address	=3D 0x60
> fault code		=3D supervisor read data, page not present
> instruction pointer	=3D 0x20:0xffffffff8162e4f0
> stack pointer	        =3D 0x28:0xffffff8161b726e0
> frame pointer	        =3D 0x28:0xffffff8161b727d0
> code segment		=3D base rx0, limit 0xfffff, type 0x1b
> 			=3D DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags	=3D interrupt enabled, resume, IOPL =3D 0
> current process		=3D 2999 (zpool)
> trap number		=3D 12
> panic: page fault
> cpuid =3D 3
> KDB: stack backtrace:
> #0 0xffffffff809208a6 at kdb_backtrace+0x66
> #1 0xffffffff808ea8be at panic+0x1ce
> #2 0xffffffff80bd8240 at trap_fatal+0x290
> #3 0xffffffff80bd857d at trap_pfault+0x1ed
> #4 0xffffffff80bd8b9e at trap+0x3ce
> #5 0xffffffff80bc315f at calltrap+0x8
> #6 0xffffffff81673975 at sa_handle_get_from_db+0x95
> #7 0xffffffff81673a38 at sa_handle_get+0x48
> #8 0xffffffff8169f516 at zfs_grab_sa_handle+0x96
> #9 0xffffffff8169faca at zfs_obj_to_path+0x6a
> #10 0xffffffff816b8c75 at zfs_ioc_obj_to_path+0x75
> #11 0xffffffff816bad46 at zfsdev_ioctl+0xe6
> #12 0xffffffff807db28b at devfs_ioctl_f+0x7b
> #13 0xffffffff80932325 at kern_ioctl+0x115
> #14 0xffffffff8093255d at sys_ioctl+0xfd
> #15 0xffffffff80bd7ae6 at amd64_syscall+0x546
> #16 0xffffffff80bc3447 at Xfast_syscall+0xf7
> Uptime: 2m10s
> Dumping 851 out of 5100 =
MB:..2%..12%..21%..31%..42%..51%..61%..72%..81%..91%
>=20
> Reading symbols from /boot/modules/virtio.ko...Reading symbols from =
/boot/modules/virtio.ko.symbols...done.
> done.
> Loaded symbols for /boot/modules/virtio.ko
> Reading symbols from /boot/modules/virtio_pci.ko...Reading symbols =
from /boot/modules/virtio_pci.ko.symbols...done.
> done.
> Loaded symbols for /boot/modules/virtio_pci.ko
> Reading symbols from /boot/modules/virtio_blk.ko...Reading symbols =
from /boot/modules/virtio_blk.ko.symbols...done.
> done.
> Loaded symbols for /boot/modules/virtio_blk.ko
> Reading symbols from /boot/modules/if_vtnet.ko...Reading symbols from =
/boot/modules/if_vtnet.ko.symbols...done.
> done.
> Loaded symbols for /boot/modules/if_vtnet.ko
> Reading symbols from /boot/modules/virtio_balloon.ko...Reading symbols =
from /boot/modules/virtio_balloon.ko.symbols...done.
> done.
> Loaded symbols for /boot/modules/virtio_balloon.ko
> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from =
/boot/kernel/ipfw.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/ipfw.ko
> Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from =
/boot/kernel/ipdivert.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/ipdivert.ko
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from =
/boot/kernel/zfs.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko
> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols =
from /boot/kernel/opensolaris.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/opensolaris.ko
> #0  doadump (textdump=3DVariable "textdump" is not available.
> ) at pcpu.h:224
> 224	pcpu.h: No such file or directory.
> 	in pcpu.h
> (kgdb) #0  doadump (textdump=3DVariable "textdump" is not available.
> ) at pcpu.h:224
> #1  0xffffffff808ea3a1 in kern_reboot (howto=3D260)
>    at /usr/src/sys/kern/kern_shutdown.c:448
> #2  0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of =
bounds>)
>    at /usr/src/sys/kern/kern_shutdown.c:636
> #3  0xffffffff80bd8240 in trap_fatal (frame=3D0xc, eva=3DVariable =
"eva" is not available.
> )
>    at /usr/src/sys/amd64/amd64/trap.c:857
> #4  0xffffffff80bd857d in trap_pfault (frame=3D0xffffff8161b72630, =
usermode=3D0)
>    at /usr/src/sys/amd64/amd64/trap.c:773
> #5  0xffffffff80bd8b9e in trap (frame=3D0xffffff8161b72630)
>    at /usr/src/sys/amd64/amd64/trap.c:456
> #6  0xffffffff80bc315f in calltrap ()
>    at /usr/src/sys/amd64/amd64/exception.S:228
> #7  0xffffffff8162e4f0 in list_remove_tail ()
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/list=
.c:151
> #8  0xffffffff816736af in sa_build_index (hdl=3D0xfffffe010ac98690,=20
>    buftype=3DSA_BONUS)
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1490
> #9  0xffffffff81673975 in sa_handle_get_from_db =
(os=3D0xfffffe0132baf000,=20
>    db=3D0xfffffe010ac968c0, userp=3D0x0, hdl_type=3DVariable =
"hdl_type" is not available.
> )
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1367
> #10 0xffffffff81673a38 in sa_handle_get (objset=3D0xfffffe0132baf000, =
objid=3DVariable "objid" is not available.
> )
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
sa.c:1392
> #11 0xffffffff8169f516 in zfs_grab_sa_handle (osp=3D0xfffffe0132baf000,=20=

>    obj=3D230809, hdlp=3D0xffffff8161b72908, db=3D0xffffff8161b72900,=20=

>    tag=3D0xffffffff81722e10)
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_znode.c:2031
> #12 0xffffffff8169faca in zfs_obj_to_path (osp=3D0xfffffe0132baf000,=20=

>    obj=3D230809, buf=3D0xffffff8008757400 "data2/src", len=3D2048)
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_znode.c:2184
> #13 0xffffffff816b8c75 in zfs_ioc_obj_to_path (zc=3D0xffffff8008757000)
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_ioctl.c:1537
> #14 0xffffffff816bad46 in zfsdev_ioctl (dev=3DVariable "dev" is not =
available.
> )
>    at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_ioctl.c:5218
> #15 0xffffffff807db28b in devfs_ioctl_f (fp=3D0xfffffe006bdf8690,=20
>    com=3D3583531558, data=3DVariable "data" is not available.
> ) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
> #16 0xffffffff80932325 in kern_ioctl (td=3DVariable "td" is not =
available.
> ) at file.h:293
> #17 0xffffffff8093255d in sys_ioctl (td=3D0xfffffe00033e58e0,=20
>    uap=3D0xffffff8161b72bb0) at /usr/src/sys/kern/sys_generic.c:691
> #18 0xffffffff80bd7ae6 in amd64_syscall (td=3D0xfffffe00033e58e0, =
traced=3D0)
>    at subr_syscall.c:135
> #19 0xffffffff80bc3447 in Xfast_syscall ()
>    at /usr/src/sys/amd64/amd64/exception.S:387
> #20 0x0000000801debcdc in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb)=20
>=20
> =
------------------------------------------------------------------------
> ps -axl
>=20
> ps: can't read sigacts at 0xfffffe00a1c64000
>=20
> =
------------------------------------------------------------------------
> vmstat -s
>=20
>   392168 cpu context switches
>    44454 device interrupts
>     8725 software interrupts
>  1299960 traps
>  1763303 system calls
>       19 kernel threads created
>     2839  fork() calls
>      141 vfork() calls
>        0 rfork() calls
>        0 swap pager pageins
>        0 swap pager pages paged in
>        0 swap pager pageouts
>        0 swap pager pages paged out
>     4580 vnode pager pageins
>    19695 vnode pager pages paged in
>        0 vnode pager pageouts
>        0 vnode pager pages paged out
>        0 page daemon wakeups
>        0 pages examined by the page daemon
>      426 pages reactivated
>   206213 copy-on-write faults
>      470 copy-on-write optimized faults
>   947712 zero fill pages zeroed
>        0 zero fill pages prezeroed
>       86 intransit blocking page faults
>  1274475 total VM faults taken
>        0 pages affected by kernel thread creation
>  1413892 pages affected by  fork()
>    70812 pages affected by vfork()
>        0 pages affected by rfork()
>        0 pages cached
>  1157439 pages freed
>        0 pages freed by daemon
>        0 pages freed by exiting processes
>   107649 pages active
>   203197 pages inactive
>       59 pages in VM cache
>   180298 pages wired down
>   771791 pages free
>     4096 bytes per page
>   990965 total name lookups
>          cache hits (89% pos + 5% neg) system 0% per-directory
>          deletions 0%, falsehits 0%, toolong 0%
>=20
> =
------------------------------------------------------------------------
> vmstat -m
>=20
>         Type InUse MemUse HighUse Requests  Size(s)
> CAM dev queue     4     1K       -        4  128
>  md_sii_data     0     0K       -       42  512
>      CAM XPT    29    16K       -       95  32,64,128,1024,2048
>       isadev     9     2K       -        9  128
>     acpiintr     1     1K       -        1  64
>         cdev     8     2K       -        8  256
>       acpica  1707   188K       -    21034  =
16,32,64,128,256,512,1024,2048
>        sigio     1     1K       -        1  64
>     filedesc   234   963K       -     3180  =
16,32,64,512,1024,2048,4096
>         kenv    66    11K       -       70  16,32,64,128
>       kqueue    24    27K       -      250  256,512,2048
>    proc-args   141    13K       -     1766  16,32,64,128,256
>        hhook     2     1K       -        2  128
>     acpitask     1     2K       -        1  2048
>      ithread    74    12K       -       74  32,128,256
>    CAM queue    12     1K       -       71  16,32
>       KTRACE   100    13K       -      100  128
>       linker   261   897K       -      303  =
16,32,64,128,256,512,1024,2048,4096
>        lockf   148    15K       -     1787  64,128,256
>   loginclass     2     1K       -        6  64
>       ip6ndp    10     1K       -       13  64,128
>         temp    20    13K       -    14346  =
16,32,64,128,256,512,1024,2048,4096
>       devbuf  3564 27445K       -     3609  =
16,32,64,128,256,512,1024,2048,4096
>       module   489    62K       -      489  128
>     mtx_pool     2    16K       -        2 =20
>       USBdev     4     1K       -        4  64,128
>          osd     3     1K       -       16  16,64
>          USB     7     3K       -        7  16,32,128,2048
>     pmchooks     1     1K       -        1  128
>      subproc   410   835K       -     3230  512,4096
>         proc     2    16K       -        2 =20
>      session    28     4K       -      143  128
>         pgrp    38     5K       -      219  128
>         cred   170    27K       -    58670  64,256
>      uidinfo    24     5K       -      257  128,2048
>       plimit    21     6K       -      201  256
>      ata_pci     1     1K       -        1  64
>    sysctltmp     0     0K       -      890  16,32,64,128,4096
>    sysctloid  2923   145K       -     2998  16,32,64,128
>       sysctl     0     0K       -      840  16,32,64
>      tidhash     1    16K       -        1 =20
>      callout     3  1536K       -        3 =20
>         umtx   948   119K       -      948  128
>     p1003.1b     1     1K       -        1  16
>         SWAP     2   549K       -        2  64
>       bus-sc    46   105K       -     2000  =
16,32,128,256,512,1024,2048,4096
>          bus  1036    83K       -     3883  16,32,64,128,256,1024
>      devstat    16    33K       -       16  32,4096
> eventhandler    90     8K       -       90  64,128
>      acpisem    16     2K       -       16  128
>         kobj   334  1336K       -      692  4096
>      CAM SIM     4     1K       -        4  256
>      Per-cpu     1     1K       -        1  32
>   CAM periph     2     1K       -       18  16,32,64,128,256
>         rman   100    12K       -      508  16,32,128
>         sbuf     0     0K       -     2357  =
16,32,64,128,256,512,1024,2048,4096
>       sglist     4     4K       -        4  1024
>      entropy  1024    64K       -     1024  64
>       ctlmem  5062 10113K       -     5062  128,2048
>        stack     0     0K       -        2  256
>    taskqueue    61     6K       -       91  16,32,64,128,1024
>       Unitno    16     1K       -     2152  32,64
>          iov     0     0K       -     4365  16,64,128,256,512
>       select   195    25K       -      195  128
>     ioctlops     1     8K       -    15314  =
16,32,64,128,256,512,1024,2048,4096
>          msg     4    30K       -        4  2048,4096
>          sem     4   106K       -        4  2048,4096
>          shm     1    20K       -        3  2048
>          tty    20    20K       -       22  1024,2048
>          pts     1     1K       -        1  256
>     mbuf_tag     0     0K       -        7  32
>        shmfd     1     8K       -        1 =20
>          pcb    52   158K       -      809  16,32,128,1024,2048,4096
>       soname    11     2K       -     7276  16,32,128
>     vfscache     1  2048K       -        1 =20
>   cl_savebuf     0     0K       -      310  64
>     vfs_hash     1  1024K       -        1 =20
>       DEVFS1    80    40K       -      102  512
>       vnodes    28     3K       -      108  64,256
>       DEVFS3   103    26K       -      132  256
>        mount    86     4K       -      169  16,32,64,128,256
>  vnodemarker     0     0K       -      120  512
>          BPF     6     1K       -        6  128
>  ether_multi    79     5K       -       94  16,32,64
>       ifaddr    80    21K       -       80  32,64,128,256,512,4096
>        ifnet     7    13K       -        7  128,2048
>        clone     6    24K       -        6  4096
>       arpcom     4     1K       -        4  16
>      lltable    26    10K       -       26  256,512
>        DEVFS    11     1K       -       12  16,128
>     routetbl    56     7K       -      221  32,64,128,256,512
>         igmp     6     2K       -        6  256
>     in_multi     4     1K       -        4  256
>    sctp_iter     0     0K       -       12  256
>     sctp_ifn     4     1K       -        4  128
>     sctp_ifa    14     2K       -       14  128
>     sctp_vrf     1     1K       -        1  64
>    sctp_a_it     0     0K       -       12  16
>    hostcache     1    28K       -        1 =20
>     syncache     1    96K       -        1 =20
>    in6_multi    36     5K       -       36  32,256
>          mld     6     1K       -        6  128
>          rpc     2     1K       -        2  256
> audit_evclass   179     6K       -      218  32
>      jblocks     6     2K       -        6  128,256
>     savedino     0     0K       -      129  256
>        sbdep     0     0K       -       17  64
>      jsegdep   106     7K       -    25650  64
>         jseg   270    34K       -      315  128
>    jfreefrag     0     0K       -      829  128
>      jnewblk     0     0K       -    23303  128
>      jremref     8     1K       -      735  128
>      jaddref     0     0K       -      783  128
>      freedep     0     0K       -        4  64
>     freework    21     3K       -      583  64,128
>    newdirblk     0     0K       -       36  64
>       dirrem    46     6K       -      663  128
>        mkdir     0     0K       -       72  128
>       diradd     1     1K       -      711  128
>     freefile     9     1K       -      411  64
>     freeblks    17     3K       -      438  128
>     freefrag     0     0K       -      829  128
>     indirdep     0     0K       -       12  128
>       newblk     5   129K       -    23304  256
>    bmsafemap     4     9K       -      654  256
>     inodedep   113  1080K       -      919  512
>      pagedep    52   141K       -      401  256
>  ufs_dirhash    93    20K       -       94  16,32,64,128,256,512
>    ufs_mount    15    67K       -       15  512,4096
>    vm_pgdata     2   129K       -        2  128
>      UMAHash     2     1K       -        2  512
>      memdesc     1     4K       -        1  4096
>     atkbddev     2     1K       -        2  64
>    pfs_nodes    21     6K       -       21  256
>       ctlblk   200  1600K       -      200 =20
>         GEOM   127    26K       -     1541  =
16,32,64,128,256,512,1024,2048
>      ramdisk     1  4096K       -        1 =20
>      acpidev    18     2K       -       18  64
>      ctlpool   532   142K       -      532  32,512
>       kbdmux     6    18K       -        6  16,512,1024,2048
>          LED     4     1K       -        4  16,128
>       apmdev     1     1K       -        1  128
>   madt_table     0     0K       -        1  4096
>       feeder     7     1K       -        7  32
>     pci_link    10     1K       -       10  16,128
>    raid_data     0     0K       -      252  32,128,256
>      io_apic     1     2K       -        1  2048
>          MCA    10     2K       -       10  128
>     nexusdev     3     1K       -        3  16
> md_nvidia_data     0     0K       -       42  512
>  IpFw/IpAcct    20     5K       -       35  16,32,64,128,1024
>      solaris 42732 117071K       -   516982  =
16,32,64,128,256,512,1024,2048,4096
>   kstat_data     4     1K       -        4  64
>=20
> =
------------------------------------------------------------------------
> vmstat -z
>=20
> ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL =
SLEEP
>=20
> UMA Kegs:               208,      0,     188,      16,     188,   0,   =
0
> UMA Zones:              896,      0,     188,       0,     188,   0,   =
0
> UMA Slabs:              568,      0,   10842,       1,   17497,   0,   =
0
> UMA RCntSlabs:          568,      0,     811,       1,     811,   0,   =
0
> UMA Hash:               256,      0,      79,      11,      81,   0,   =
0
> 16 Bucket:              152,      0,     140,      10,     140,   0,   =
0
> 32 Bucket:              280,      0,     119,       7,     119,   2,   =
0
> 64 Bucket:              536,      0,     125,       1,     125,  57,   =
0
> 128 Bucket:            1048,      0,     492,       0,     492,1050,   =
0
> VM OBJECT:              232,      0,    8228,     204,   60846,   0,   =
0
> MAP:                    232,      0,       7,      25,       7,   0,   =
0
> KMAP ENTRY:             120, 182559,      99,     521,   22390,   0,   =
0
> MAP ENTRY:              120,      0,    8567,     330,  147448,   0,   =
0
> fakepg:                 120,      0,       0,       0,       0,   0,   =
0
> mt_zone:               4112,      0,     331,      42,     331,   0,   =
0
> 16:                      16,      0,    3858,     342,   53852,   0,   =
0
> 32:                      32,      0,    4201,     647,   46199,   0,   =
0
> 64:                      64,      0,   23763,   15269,  271306,   0,   =
0
> 128:                    128,      0,    6587,     431,  130489,   0,   =
0
> 256:                    256,      0,    6884,   15661,  117624,   0,   =
0
> 512:                    512,      0,    4884,     128,   76796,   0,   =
0
> 1024:                  1024,      0,     845,    1207,    7696,   0,   =
0
> 2048:                  2048,      0,    5580,     330,    6845,   0,   =
0
> 4096:                  4096,      0,    1816,     131,   39969,   0,   =
0
> Files:                   80,      0,     969,     246,   48815,   0,   =
0
> TURNSTILE:              136,      0,     475,      85,     475,   0,   =
0
> umtx pi:                 96,      0,       0,       0,       0,   0,   =
0
> MAC labels:              40,      0,       0,       0,       0,   0,   =
0
> PROC:                  1184,      0,     179,      49,    2999,   0,   =
0
> THREAD:                1128,      0,     429,      45,     591,   0,   =
0
> SLEEPQUEUE:              80,      0,     475,      76,     475,   0,   =
0
> VMSPACE:                392,      0,     161,      49,    2985,   0,   =
0
> cpuset:                  72,      0,     206,     194,     348,   0,   =
0
> audit_record:           960,      0,       0,       0,       0,   0,   =
0
> mbuf_packet:            256,      0,     602,     813,   32072,   0,   =
0
> mbuf:                   256,      0,      14,     633,   36538,   0,   =
0
> mbuf_cluster:          2048,  25600,    1408,      60,    1408,   0,   =
0
> mbuf_jumbo_page:       4096,  12800,       0,      77,    1138,   0,   =
0
> mbuf_jumbo_9k:         9216,  19200,       0,       0,       0,   0,   =
0
> mbuf_jumbo_16k:       16384,  12800,       0,       0,       0,   0,   =
0
> mbuf_ext_refcnt:          4,      0,       0,     672,     185,   0,   =
0
> g_bio:                  232,      0,       0,     640,   95668,   0,   =
0
> ttyinq:                 160,      0,     135,      81,     270,   0,   =
0
> ttyoutq:                256,      0,      72,      48,     144,   0,   =
0
> ata_request:            328,      0,       0,       0,       0,   0,   =
0
> ata_composite:          336,      0,       0,       0,       0,   0,   =
0
> vtblk_request:           48,      0,     512,      64,     512,   0,   =
0
> vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   =
0
> VNODE:                  480,      0,    6185,      87,    7601,   0,   =
0
> VNODEPOLL:              112,      0,       0,       0,       0,   0,   =
0
> S VFS Cache:            108,      0,    5866,     140,   46996,   0,   =
0
> STS VFS Cache:          148,      0,       0,       0,       0,   0,   =
0
> L VFS Cache:            328,      0,     411,     177,    3463,   0,   =
0
> LTS VFS Cache:          368,      0,       0,       0,       0,   0,   =
0
> NAMEI:                 1024,      0,       1,      67,  210673,   0,   =
0
> NCLNODE:                568,      0,       0,       0,       0,   0,   =
0
> DIRHASH:               1024,      0,     206,      34,     206,   0,   =
0
> Mountpoints:            792,      0,       7,       8,       7,   0,   =
0
> pipe:                   728,      0,      53,      52,    2023,   0,   =
0
> ksiginfo:               112,      0,     214,     842,     290,   0,   =
0
> itimer:                 344,      0,       1,      21,       1,   0,   =
0
> KNOTE:                  128,      0,      69,     105,     342,   0,   =
0
> socket:                 680,  25602,     289,      65,    2277,   0,   =
0
> unpcb:                  240,  25600,     164,      92,     687,   0,   =
0
> ipq:                     56,    819,       0,       0,       0,   0,   =
0
> udp_inpcb:              392,  25600,      29,      71,    1126,   0,   =
0
> udpcb:                   16,  25704,      29,     643,    1126,   0,   =
0
> tcp_inpcb:              392,  25600,     120,      60,     440,   0,   =
0
> tcpcb:                  976,  25600,      85,      27,     440,   0,   =
0
> tcptw:                   72,   5150,      35,     215,      37,   0,   =
0
> syncache:               152,  15375,       1,     124,     186,   0,   =
0
> hostcache:              136,  15372,      57,     111,      57,   0,   =
0
> tcpreass:                40,   1680,      17,     403,     398,   0,   =
0
> sackhole:                32,      0,       0,     303,      26,   0,   =
0
> sctp_ep:               1376,  25600,       0,       0,       0,   0,   =
0
> sctp_asoc:             2288,  40000,       0,       0,       0,   0,   =
0
> sctp_laddr:              48,  80064,       0,     288,      13,   0,   =
0
> sctp_raddr:             704,  80000,       0,       0,       0,   0,   =
0
> sctp_chunk:             136, 400008,       0,       0,       0,   0,   =
0
> sctp_readq:             104, 400032,       0,       0,       0,   0,   =
0
> sctp_stream_msg_out:    112, 400026,       0,       0,       0,   0,   =
0
> sctp_asconf:             40, 400008,       0,       0,       0,   0,   =
0
> sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   =
0
> ripcb:                  392,  25600,       0,      50,      18,   0,   =
0
> rtentry:                200,      0,      34,      80,      34,   0,   =
0
> selfd:                   56,      0,     432,     387,   92002,   0,   =
0
> SWAPMETA:               288, 116519,       0,       0,       0,   0,   =
0
> FFS inode:              168,      0,    3499,     109,    3934,   0,   =
0
> FFS1 dinode:            128,      0,       0,       0,       0,   0,   =
0
> FFS2 dinode:            256,      0,    3499,     161,    3926,   0,   =
0
> IPFW dynamic rule:      120,      0,       0,       0,       0,   0,   =
0
> divcb:                  392,  25600,       0,       0,       0,   0,   =
0
> taskq_zone:              48,      0,       0,     360,      64,   0,   =
0
> zio_cache:              880,      0,       1,    1599,   75260,   0,   =
0
> zio_link_cache:          48,      0,       0,    2448,   37923,   0,   =
0
> zio_buf_512:            512,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_512:       512,      0,       0,       0,       0,   0,   =
0
> zio_buf_1024:          1024,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   =
0
> zio_buf_1536:          1536,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   =
0
> zio_buf_2048:          2048,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   =
0
> zio_buf_2560:          2560,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   =
0
> zio_buf_3072:          3072,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   =
0
> zio_buf_3584:          3584,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   =
0
> zio_buf_4096:          4096,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   =
0
> zio_buf_5120:          5120,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   =
0
> zio_buf_6144:          6144,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   =
0
> zio_buf_7168:          7168,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   =
0
> zio_buf_8192:          8192,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   =
0
> zio_buf_10240:        10240,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   =
0
> zio_buf_12288:        12288,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   =
0
> zio_buf_14336:        14336,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   =
0
> zio_buf_16384:        16384,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   =
0
> zio_buf_20480:        20480,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   =
0
> zio_buf_24576:        24576,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   =
0
> zio_buf_28672:        28672,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   =
0
> zio_buf_32768:        32768,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   =
0
> zio_buf_36864:        36864,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   =
0
> zio_buf_40960:        40960,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   =
0
> zio_buf_45056:        45056,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   =
0
> zio_buf_49152:        49152,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   =
0
> zio_buf_53248:        53248,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   =
0
> zio_buf_57344:        57344,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   =
0
> zio_buf_61440:        61440,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   =
0
> zio_buf_65536:        65536,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   =
0
> zio_buf_69632:        69632,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   =
0
> zio_buf_73728:        73728,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   =
0
> zio_buf_77824:        77824,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   =
0
> zio_buf_81920:        81920,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   =
0
> zio_buf_86016:        86016,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   =
0
> zio_buf_90112:        90112,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   =
0
> zio_buf_94208:        94208,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   =
0
> zio_buf_98304:        98304,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   =
0
> zio_buf_102400:      102400,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   =
0
> zio_buf_106496:      106496,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   =
0
> zio_buf_110592:      110592,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   =
0
> zio_buf_114688:      114688,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   =
0
> zio_buf_118784:      118784,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   =
0
> zio_buf_122880:      122880,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   =
0
> zio_buf_126976:      126976,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   =
0
> zio_buf_131072:      131072,      0,       0,       0,       0,   0,   =
0
> zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   =
0
> sa_cache:                80,      0,    2647,     143,    3636,   0,   =
0
> dnode_t:                856,      0,    3641,     111,    4026,   0,   =
0
> dmu_buf_impl_t:         224,      0,    7859,     250,   10606,   0,   =
0
> arc_buf_hdr_t:          216,      0,    6225,      93,    8090,   0,   =
0
> arc_buf_t:              104,      0,    6135,     165,    8127,   0,   =
0
> zil_lwb_cache:          192,      0,       1,      99,      23,   0,   =
0
> zfs_znode_cache:        400,      0,    2646,      45,    3625,   0,   =
0
>=20
>=20
> =
------------------------------------------------------------------------
> vmstat -i
>=20
> interrupt                          total       rate
> irq1: atkbd0                           8          0
> irq10: em0 em1+                    21867        173
> irq11: re0 uhci0++*                22579        179
> cpu0:timer                         43190        342
> cpu3:timer                         31518        250
> cpu2:timer                         35619        282
> cpu1:timer                         44648        354
> Total                             199429       1582
>=20
> =
------------------------------------------------------------------------
> pstat -T
>=20
> 969/12328 files
> 0M/4095M swap space
>=20
> =
------------------------------------------------------------------------
> pstat -s
>=20
> Device          512-blocks     Used    Avail Capacity
> /dev/vtbd0p4       8388352        0  8388352     0%
>=20
> =
------------------------------------------------------------------------
> iostat
>=20
> iostat: kvm_read(_tk_nin): invalid address (0x0)
> iostat: disabling TTY statistics
>           vtbd0            vtbd1            vtbd2             cpu
>  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
> 83.94 168 13.73   7.24  39  0.28  11.92  30  0.35  13  0  7  1 79
>=20
> =
------------------------------------------------------------------------
> ipcs -a
>=20
> Message Queues:
> T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
CGROUP                 CBYTES                 QNUM               QBYTES  =
      LSPID        LRPID STIME    RTIME    CTIME  =20
>=20
> Shared Memory:
> T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    =
DTIME    CTIME  =20
> m        65536   1667461225 --rw-rw-rw- nobody   munin    nobody   =
munin               0        65536         1601         2199 13:28:39 =
13:28:39 13:27:56
>=20
> Semaphores:
> T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
CGROUP          NSEMS OTIME    CTIME  =20
> s        65536   1667461225 --rw-rw-rw- nobody   munin    nobody   =
munin               3 13:28:39 13:27:56
>=20
>=20
> =
------------------------------------------------------------------------
> ipcs -T
>=20
> msginfo:
> 	msgmax:        16384	(max characters in a message)
> 	msgmni:           40	(# of message queues)
> 	msgmnb:         2048	(max characters in a message queue)
> 	msgtql:           40	(max # of messages in system)
> 	msgssz:            8	(size of a message segment)
> 	msgseg:         2048	(# of message segments in system)
>=20
> shminfo:
> 	shmmax:    536870912	(max shared memory segment size)
> 	shmmin:            1	(min shared memory segment size)
> 	shmmni:          192	(max number of shared memory =
identifiers)
> 	shmseg:          128	(max shared memory segments per process)
> 	shmall:       131072	(max amount of shared memory in pages)
>=20
> seminfo:
> 	semmni:           50	(# of semaphore identifiers)
> 	semmns:          340	(# of semaphores in system)
> 	semmnu:          150	(# of undo structures in system)
> 	semmsl:          340	(max # of semaphores per id)
> 	semopm:          100	(max # of operations per semop call)
> 	semume:           50	(max # of undo entries per process)
> 	semusz:          632	(size in bytes of undo structure)
> 	semvmx:        32767	(semaphore maximum value)
> 	semaem:        16384	(adjust on exit max value)
>=20
>=20
> =
------------------------------------------------------------------------
> nfsstat
>=20
> Client Info:
> Rpc Counts:
>  Getattr   Setattr    Lookup  Readlink      Read     Write    Create   =
 Remove
>        0         0         0         0         0         0         0   =
      0
>   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus   =
 Access
>        0         0         0         0         0         0         0   =
      0
>    Mknod    Fsstat    Fsinfo  PathConf    Commit
>        0         0         0         0         0
> Rpc Info:
> TimedOut   Invalid X Replies   Retries  Requests
>        0         0         0         0         0
> Cache Info:
> Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits  =
  Misses
>        0         0         0         0         0         0         0   =
      0
> BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits  =
  Misses
>        0         0         0         0         0         0         0   =
      0
>=20
> Server Info:
>  Getattr   Setattr    Lookup  Readlink      Read     Write    Create   =
 Remove
>        0         0         0         0         0         0         0   =
      0
>   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus   =
 Access
>        0         0         0         0         0         0         0   =
      0
>    Mknod    Fsstat    Fsinfo  PathConf    Commit
>        0         0         0         0         0
> Server Ret-Failed
>                0
> Server Faults
>            0
> Server Cache Stats:
>   Inprog      Idem  Non-idem    Misses
>        0         0         0         0
> Server Write Gathering:
> WriteOps  WriteRPC   Opsaved
>        0         0         0
>=20
> =
------------------------------------------------------------------------
> netstat -s
>=20
> tcp:
> 	9385 packets sent
> 		6192 data packets (6331579 bytes)
> 		99 data packets (127875 bytes) retransmitted
> 		4 data packets unnecessarily retransmitted
> 		0 resends initiated by MTU discovery
> 		2119 ack-only packets (214 delayed)
> 		0 URG only packets
> 		0 window probe packets
> 		768 window update packets
> 		209 control packets
> 	10672 packets received
> 		4817 acks (for 6305478 bytes)
> 		222 duplicate acks
> 		0 acks for unsent data
> 		4579 packets (4011692 bytes) received in-sequence
> 		34 completely duplicate packets (43596 bytes)
> 		0 old duplicate packets
> 		0 packets with some dup. data (0 bytes duped)
> 		398 out-of-order packets (533435 bytes)
> 		0 packets (0 bytes) of data after window
> 		0 window probes
> 		91 window update packets
> 		78 packets received after close
> 		0 discarded for bad checksums
> 		0 discarded for bad header offset fields
> 		0 discarded because packet too short
> 		12 discarded due to memory problems
> 	86 connection requests
> 	185 connection accepts
> 	0 bad connection attempts
> 	0 listen queue overflows
> 	8 ignored RSTs in the windows
> 	245 connections established (including accepts)
> 	321 connections closed (including 40 drops)
> 		153 connections updated cached RTT on close
> 		153 connections updated cached RTT variance on close
> 		7 connections updated cached ssthresh on close
> 	26 embryonic connections dropped
> 	4733 segments updated rtt (of 3794 attempts)
> 	27 retransmit timeouts
> 		0 connections dropped by rexmit timeout
> 	0 persist timeouts
> 		0 connections dropped by persist timeout
> 	0 Connections (fin_wait_2) dropped because of timeout
> 	0 keepalive timeouts
> 		0 keepalive probes sent
> 		0 connections dropped by keepalive
> 	359 correct ACK header predictions
> 	2854 correct data packet header predictions
> 	186 syncache entries added
> 		0 retransmitted
> 		1 dupsyn
> 		0 dropped
> 		185 completed
> 		0 bucket overflow
> 		0 cache overflow
> 		0 reset
> 		0 stale
> 		0 aborted
> 		0 badack
> 		0 unreach
> 		0 zone failures
> 	186 cookies sent
> 	0 cookies received
> 	57 hostcache entries added
> 		0 bucket overflow
> 	3 SACK recovery episodes
> 	36 segment rexmits in SACK recovery episodes
> 	52128 byte rexmits in SACK recovery episodes
> 	92 SACK options (SACK blocks) received
> 	503 SACK options (SACK blocks) sent
> 	0 SACK scoreboard overflow
> 	0 packets with ECN CE bit set
> 	0 packets with ECN ECT(0) bit set
> 	0 packets with ECN ECT(1) bit set
> 	0 successful ECN handshakes
> 	0 times ECN reduced the congestion window
> udp:
> 	3761 datagrams received
> 	0 with incomplete header
> 	0 with bad data length field
> 	0 with bad checksum
> 	1 with no checksum
> 	1279 dropped due to no socket
> 	0 broadcast/multicast datagrams undelivered
> 	0 dropped due to full socket buffers
> 	0 not for hashed pcb
> 	2482 delivered
> 	2483 datagrams output
> 	0 times multicast source filter matched
> ip:
> 	14453 total packets received
> 	0 bad header checksums
> 	0 with size smaller than minimum
> 	0 with data size < data length
> 	0 with ip length > max ip packet size
> 	0 with header length < data size
> 	0 with data length < header length
> 	0 with bad options
> 	0 with incorrect version number
> 	0 fragments received
> 	0 fragments dropped (dup or out of space)
> 	0 fragments dropped after timeout
> 	0 packets reassembled ok
> 	14435 packets for this host
> 	12 packets for unknown/unsupported protocol
> 	0 packets forwarded (0 packets fast forwarded)
> 	0 packets not forwardable
> 	0 packets received for unknown multicast group
> 	0 redirects sent
> 	13495 packets sent from this host
> 	0 packets sent with fabricated ip header
> 	0 output packets dropped due to no bufs, etc.
> 	387 output packets discarded due to no route
> 	0 output datagrams fragmented
> 	0 fragments created
> 	0 datagrams that can't be fragmented
> 	0 tunneling packets that can't find gif
> 	0 datagrams with bad address in header
> icmp:
> 	746 calls to icmp_error
> 	0 errors not generated in response to an icmp message
> 	Output histogram:
> 		echo reply: 2
> 		destination unreachable: 746
> 	0 messages with bad code fields
> 	0 messages less than the minimum length
> 	0 messages with bad checksum
> 	0 messages with bad length
> 	0 multicast echo requests ignored
> 	0 multicast timestamp requests ignored
> 	Input histogram:
> 		destination unreachable: 7
> 		echo: 2
> 	2 message responses generated
> 	0 invalid return addresses
> 	0 no return routes
> igmp:
> 	5 messages received
> 	0 messages received with too few bytes
> 	0 messages received with wrong TTL
> 	0 messages received with bad checksum
> 	5 V1/V2 membership queries received
> 	0 V3 membership queries received
> 	0 membership queries received with invalid field(s)
> 	5 general queries received
> 	0 group queries received
> 	0 group-source queries received
> 	0 group-source queries dropped
> 	0 membership reports received
> 	0 membership reports received with invalid field(s)
> 	0 membership reports received for groups to which we belong
> 	0 V3 reports received without Router Alert
> 	0 membership reports sent
> arp:
> 	9 ARP requests sent
> 	0 ARP replies sent
> 	66 ARP requests received
> 	1 ARP reply received
> 	67 ARP packets received
> 	0 total packets dropped due to no ARP entry
> 	0 ARP entrys timed out
> 	0 Duplicate IPs seen
> ip6:
> 	0 total packets received
> 	0 with size smaller than minimum
> 	0 with data size < data length
> 	0 with bad options
> 	0 with incorrect version number
> 	0 fragments received
> 	0 fragments dropped (dup or out of space)
> 	0 fragments dropped after timeout
> 	0 fragments that exceeded limit
> 	0 packets reassembled ok
> 	0 packets for this host
> 	0 packets forwarded
> 	0 packets not forwardable
> 	0 redirects sent
> 	11 packets sent from this host
> 	0 packets sent with fabricated ip header
> 	0 output packets dropped due to no bufs, etc.
> 	0 output packets discarded due to no route
> 	0 output datagrams fragmented
> 	0 fragments created
> 	0 datagrams that can't be fragmented
> 	0 packets that violated scope rules
> 	0 multicast packets which we don't join
> 	Mbuf statistics:
> 		0 one mbuf
> 		3 one ext mbuf
> 		0 two or more ext mbuf
> 	0 packets whose headers are not contiguous
> 	0 tunneling packets that can't find gif
> 	0 packets discarded because of too many headers
> 	0 failures of source address selection
> 	Source addresses selection rule applied:
> 		16 same address
> icmp6:
> 	0 calls to icmp6_error
> 	0 errors not generated in response to an icmp6 message
> 	0 errors not generated because of rate limitation
> 	Output histogram:
> 		neighbor solicitation: 3
> 	0 messages with bad code fields
> 	0 messages < minimum length
> 	0 bad checksums
> 	0 messages with bad length
> 	Histogram of error messages to be generated:
> 		0 no route
> 		0 administratively prohibited
> 		0 beyond scope
> 		0 address unreachable
> 		0 port unreachable
> 		0 packet too big
> 		0 time exceed transit
> 		0 time exceed reassembly
> 		0 erroneous header field
> 		0 unrecognized next header
> 		0 unrecognized option
> 		0 redirect
> 		0 unknown
> 	0 message responses generated
> 	0 messages with too many ND options
> 	0 messages with bad ND options
> 	0 bad neighbor solicitation messages
> 	0 bad neighbor advertisement messages
> 	0 bad router solicitation messages
> 	0 bad router advertisement messages
> 	0 bad redirect messages
> 	0 path MTU changes
> rip6:
> 	0 messages received
> 	0 checksum calculations on inbound
> 	0 messages with bad checksum
> 	0 messages dropped due to no socket
> 	0 multicast messages dropped due to no socket
> 	0 messages dropped due to full socket buffers
> 	0 delivered
> 	0 datagrams output
>=20
> =
------------------------------------------------------------------------
> netstat -m
>=20
> 616/1446/2062 mbufs in use (current/cache/total)
> 595/873/1468/25600 mbuf clusters in use (current/cache/total/max)
> 602/813 mbuf+clusters out of packet secondary zone in use =
(current/cache)
> 0/77/77/12800 4k (page size) jumbo clusters in use =
(current/cache/total/max)
> 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
> 1344K/2415K/3759K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 59 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
>=20
> =
------------------------------------------------------------------------
> netstat -id
>=20
> Name    Mtu Network       Address              Ipkts Ierrs Idrop    =
Opkts Oerrs  Coll Drop
> usbus     0 <Link#1>                               0     0     0       =
 0     0     0    0=20
> re0    1500 <Link#2>      00:16:8e:1a:f3:a9        2     0     0       =
 1     0     0    0=20
> re0    1500 192.168.122.0 192.168.122.3            0     -     -       =
 0     -     -    -=20
> re0    1500 fe80::216:8ef fe80::216:8eff:fe        0     -     -       =
 2     -     -    -=20
> em0    1500 <Link#3>      52:54:00:b2:12:8c    13686     0     0    =
12271     0     0    0=20
> em0    1500 195.234.155.0 dentredje              921     -     -    =
12265     -     -    -=20
> em0    1500 fe80::5054:ff fe80::5054:ff:feb        0     -     -       =
 1     -     -    -=20
> em0    1500 195.234.155.0 skaarup.org           9594     -     -       =
 0     -     -    -=20
> em0    1500 195.234.155.0 ns11.skaarupaps.d        6     -     -       =
 0     -     -    -=20
> em0    1500 195.234.155.0 195.234.155.50           0     -     -       =
 0     -     -    -=20
> em0    1500 195.234.155.0 ntp.gal.dk            3079     -     -       =
 0     -     -    -=20
> em0    1500 195.234.155.0 195.234.155.75           0     -     -       =
 0     -     -    -=20
> em1    1500 <Link#4>      52:54:00:b2:12:a2       15     0     0       =
 1     0     0    0=20
> em1    1500 10.0.0.0      dentredjeb               0     -     -       =
 0     -     -    -=20
> em1    1500 fe80::5054:ff fe80::5054:ff:feb        0     -     -       =
 1     -     -    -=20
> lo0   16384 <Link#5>                             842     0     0      =
842     0     0    0=20
> lo0   16384 localhost     ::1                      0     -     -       =
 0     -     -    -=20
> lo0   16384 fe80::1%lo0   fe80::1                  0     -     -       =
 0     -     -    -=20
> lo0   16384 your-net      localhost              842     -     -      =
842     -     -    -=20
> ipfw0 65536 <Link#6>                               0     0     0       =
 0     0     0    0=20
>=20
> =
------------------------------------------------------------------------
> netstat -anr
>=20
> Routing tables
>=20
> Internet:
> Destination        Gateway            Flags    Refs      Use  Netif =
Expire
> default            195.234.155.1      UGS         1    12267    em0
> 10.0.0.0/24        link#4             U           0        0    em1
> 10.0.0.3           link#4             UHS         0        0    lo0
> 127.0.0.1          link#5             UH          0      842    lo0
> 192.168.122.0/24   link#2             U           0        0    re0
> 192.168.122.3      link#2             UHS         0        0    lo0
> 195.234.155.0/24   link#3             U           0        0    em0
> 195.234.155.50     link#3             UHS         0        0    lo0
> 195.234.155.70     link#3             UHS         0        0    lo0
> 195.234.155.75     link#3             UHS         0        0    lo0
> 195.234.155.102    link#3             UHS         0        0    lo0
> 195.234.155.124    link#3             UHS         0        0    lo0
> 195.234.155.143    link#3             UHS         0        0    lo0
>=20
> Internet6:
> Destination                       Gateway                       Flags  =
    Netif Expire
> ::/96                             ::1                           UGRS   =
     lo0
> ::1                               link#5                        UH     =
     lo0
> ::ffff:0.0.0.0/96                 ::1                           UGRS   =
     lo0
> fe80::/10                         ::1                           UGRS   =
     lo0
> fe80::%re0/64                     link#2                        U      =
     re0
> fe80::216:8eff:fe1a:f3a9%re0      link#2                        UHS    =
     lo0
> fe80::%em0/64                     link#3                        U      =
     em0
> fe80::5054:ff:feb2:128c%em0       link#3                        UHS    =
     lo0
> fe80::%em1/64                     link#4                        U      =
     em1
> fe80::5054:ff:feb2:12a2%em1       link#4                        UHS    =
     lo0
> fe80::%lo0/64                     link#5                        U      =
     lo0
> fe80::1%lo0                       link#5                        UHS    =
     lo0
> ff01::%re0/32                     fe80::216:8eff:fe1a:f3a9%re0  U      =
     re0
> ff01::%em0/32                     fe80::5054:ff:feb2:128c%em0   U      =
     em0
> ff01::%em1/32                     fe80::5054:ff:feb2:12a2%em1   U      =
     em1
> ff01::%lo0/32                     ::1                           U      =
     lo0
> ff02::/16                         ::1                           UGRS   =
     lo0
> ff02::%re0/32                     fe80::216:8eff:fe1a:f3a9%re0  U      =
     re0
> ff02::%em0/32                     fe80::5054:ff:feb2:128c%em0   U      =
     em0
> ff02::%em1/32                     fe80::5054:ff:feb2:12a2%em1   U      =
     em1
> ff02::%lo0/32                     ::1                           U      =
     lo0
>=20
> =
------------------------------------------------------------------------
> netstat -anA
>=20
> Active Internet connections (including servers)
> Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign =
Address    (state)
> fffffe010a292000 tcp4       0      0 195.234.155.143.20 =
208.83.137.114.270 ESTABLISHED
> fffffe010a28c7a0 tcp4       0      0 195.234.155.143.25 =
81.19.253.131.6340 CLOSE_WAIT
> fffffe00a1069000 tcp4       0      0 195.234.155.143.25 =
208.83.137.117.270 CLOSED
> fffffe00a106a3d0 tcp4       0      0 195.234.155.70.25  =
81.30.22.111.11067 ESTABLISHED
> fffffe01326973d0 tcp4       0      6 195.234.155.143.43 =
173.194.69.26.25   FIN_WAIT_1
> fffffe010a2933d0 tcp4       0      0 195.234.155.143.50 =
78.156.102.15.25   ESTABLISHED
> fffffe00a106a000 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.387 ESTABLISHED
> fffffe013219a7a0 tcp4       0      0 127.0.0.1.26       =
127.0.0.1.11423    ESTABLISHED
> fffffe006b41ab70 tcp4       0      0 127.0.0.1.11423    127.0.0.1.26   =
    ESTABLISHED
> fffffe00a10693d0 tcp4       0      0 127.0.0.1.26       =
127.0.0.1.32356    ESTABLISHED
> fffffe010a2927a0 tcp4       0      0 127.0.0.1.32356    127.0.0.1.26   =
    ESTABLISHED
> fffffe01326977a0 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.329 ESTABLISHED
> fffffe010a292b70 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.394 ESTABLISHED
> fffffe00a10697a0 tcp4       0      0 195.234.155.143.26 =
81.19.253.131.25   LAST_ACK
> fffffe00a1069b70 tcp4       0      0 195.234.155.70.143 =
178.249.49.187.366 ESTABLISHED
> fffffe006bd383d0 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.450 ESTABLISHED
> fffffe013214d7a0 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.392 ESTABLISHED
> fffffe0006670000 tcp4    1448      0 195.234.155.70.993 =
178.249.49.187.484 ESTABLISHED
> fffffe00a1d72b70 tcp4       0      0 195.234.155.70.993 =
178.249.49.187.381 ESTABLISHED
> fffffe010a293000 tcp4       0      0 195.234.155.70.143 =
178.249.49.187.400 ESTABLISHED
> fffffe01326d0000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59864 ESTABLISHED
> fffffe010a28cb70 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59863 ESTABLISHED
> fffffe010a2937a0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59862 ESTABLISHED
> fffffe010a28d000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59861 ESTABLISHED
> fffffe01327306c0 tcp4       0      0 195.234.155.70.25  =
83.77.53.106.12850 TIME_WAIT
> fffffe010a28c3d0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59858 ESTABLISHED
> fffffe0132730708 tcp4       0      0 195.234.155.70.25  =
85.49.162.228.5748 TIME_WAIT
> fffffe010a28d3d0 tcp4       0      0 195.234.155.70.25  =
93.163.242.142.457 ESTABLISHED
> fffffe0132730750 tcp4       0      0 195.234.155.70.25  =
78.131.218.227.329 TIME_WAIT
> fffffe0132730798 tcp4       0      0 195.234.155.70.25  =
189.6.215.52.60453 TIME_WAIT
> fffffe01327307e0 tcp4       0      0 195.234.155.70.25  =
77.229.19.23.55285 TIME_WAIT
> fffffe0132730828 tcp4       0      0 195.234.155.70.80  =
88.208.217.154.452 TIME_WAIT
> fffffe00066707a0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62467 ESTABLISHED
> fffffe00a1d74000 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62466 ESTABLISHED
> fffffe00a1d743d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62465 ESTABLISHED
> fffffe0132730870 tcp4       0      0 195.234.155.70.25  =
91.39.55.104.31089 TIME_WAIT
> fffffe006bd373d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62464 ESTABLISHED
> fffffe006bd393d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62463 ESTABLISHED
> fffffe006bd387a0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62462 ESTABLISHED
> fffffe00a106c3d0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62461 ESTABLISHED
> fffffe00a106c7a0 tcp4       0      0 195.234.155.70.143 =
80.162.6.187.62460 ESTABLISHED
> fffffe00a1d72000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59837 FIN_WAIT_2
> fffffe00a1d723d0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59836 FIN_WAIT_2
> fffffe00a1d727a0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59835 FIN_WAIT_2
> fffffe00a1d73000 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59834 FIN_WAIT_2
> fffffe00a1d733d0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59833 FIN_WAIT_2
> fffffe00a1d737a0 tcp4       0      0 195.234.155.70.80  =
193.88.216.6.59832 FIN_WAIT_2
> fffffe0132730a20 tcp4       0      0 195.234.155.70.25  =
190.179.18.226.264 TIME_WAIT
> fffffe01327309d8 tcp4       0      0 195.234.155.70.25  =
187.54.5.117.2375  TIME_WAIT
> fffffe010a473288 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15139  TIME_WAIT
> fffffe01327308b8 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15138  TIME_WAIT
> fffffe0132730900 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15136  TIME_WAIT
> fffffe0132730990 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15135  TIME_WAIT
> fffffe0132730948 tcp4       0      0 195.234.155.70.80  =
80.63.33.34.15129  TIME_WAIT
> fffffe01326db7a0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
> fffffe0132684000 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
> fffffe006bd377a0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
> fffffe0132730a68 tcp4       0      0 195.234.155.70.25  =
84.228.228.52.2942 TIME_WAIT
> fffffe01326da7a0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1038 ESTABLISHED
> fffffe01326d03d0 tcp4       0      0 195.234.155.70.993 =
5.103.173.126.1037 LAST_ACK
> fffffe0132730af8 tcp4       0      0 195.234.155.70.25  =
81.202.62.25.61332 TIME_WAIT
> fffffe0132730ab0 tcp4       0      0 195.234.155.70.25  =
201.212.179.57.332 TIME_WAIT
> fffffe0132730b88 tcp4       0      0 195.234.155.70.80  =
157.55.36.49.20092 TIME_WAIT
> fffffe0132730b40 tcp4       0      0 195.234.155.70.25  =
81.34.117.66.50711 TIME_WAIT
> fffffe0132730bd0 tcp4       0      0 195.234.155.70.80  =
157.55.35.76.14663 TIME_WAIT
> fffffe0132730c18 tcp4       0      0 195.234.155.70.80  =
199.30.20.103.3187 TIME_WAIT
> fffffe0132730c60 tcp4       0      0 195.234.155.70.110 =
188.183.53.110.607 TIME_WAIT
> fffffe0132730ca8 tcp4       0      0 195.234.155.70.80  =
131.253.38.99.6317 TIME_WAIT
> fffffe006bd37b70 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.455 ESTABLISHED
> fffffe01326d1dc8 tcp4       0      0 195.234.155.70.25  =
79.42.1.16.21112   TIME_WAIT
> fffffe0132730cf0 tcp4       0      0 195.234.155.70.25  =
5.22.76.122.15314  TIME_WAIT
> fffffe006bd38b70 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.332 ESTABLISHED
> fffffe006bd39000 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.446 ESTABLISHED
> fffffe006bd397a0 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.432 ESTABLISHED
> fffffe006bd39b70 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.457 ESTABLISHED
> fffffe0132697000 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.448 ESTABLISHED
> fffffe0132684b70 tcp4       0      0 195.234.155.70.993 =
87.72.76.248.50286 ESTABLISHED
> fffffe0132730d38 tcp4       0      0 195.234.155.70.25  =
189.6.215.52.61730 TIME_WAIT
> fffffe010a473240 tcp4       0      0 195.234.155.70.25  =
91.99.166.18.9811  TIME_WAIT
> fffffe013214d000 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.422 ESTABLISHED
> fffffe0006d95000 tcp4       0      0 127.0.0.1.26       *.*            =
    LISTEN
> fffffe01321993d0 tcp4       0      0 *.25               *.*            =
    LISTEN
> fffffe01326da000 tcp4       0      0 127.0.0.1.53       *.*            =
    LISTEN
> fffffe0006f7db70 tcp4       0      0 *.587              *.*            =
    LISTEN
> fffffe013219ab70 tcp4       0      0 127.0.0.1.3306     *.*            =
    LISTEN
> fffffe01321997a0 tcp4       0      0 195.234.155.70.25  =
83.49.89.31.58370  FIN_WAIT_2
> fffffe00066703d0 tcp4       0      0 195.234.155.70.993 =
85.233.252.158.374 ESTABLISHED
> fffffe0006670b70 tcp4       0      0 10.0.0.3.4949      *.*            =
    LISTEN
> fffffe01326dab70 tcp46      0      0 *.110              *.*            =
    LISTEN
> fffffe006b41a7a0 tcp46      0      0 *.993              *.*            =
    LISTEN
> fffffe013214db70 tcp46      0      0 *.143              *.*            =
    LISTEN
> fffffe006b469000 tcp4       0      0 195.234.155.70.80  =
65.55.215.54.65193 TIME_WAIT
> fffffe0132730dc8 tcp4       0      0 195.234.155.70.25  =
80.122.8.131.65026 TIME_WAIT
> fffffe0132730d80 tcp4       0      0 195.234.155.70.25  =
190.179.62.166.249 TIME_WAIT
> fffffe01326d07a0 tcp4       0      0 *.443              *.*            =
    LISTEN
> fffffe01326d0b70 tcp6       0      0 *.443              *.*            =
    LISTEN
> fffffe006b419000 tcp4       0      0 *.82               *.*            =
    LISTEN
> fffffe006b4193d0 tcp6       0      0 *.82               *.*            =
    LISTEN
> fffffe006b4197a0 tcp4       0      0 *.80               *.*            =
    LISTEN
> fffffe006b419b70 tcp6       0      0 *.80               *.*            =
    LISTEN
> fffffe006b469048 tcp4       0      0 195.234.155.70.110 =
85.81.181.86.49004 TIME_WAIT
> fffffe006b4690d8 tcp4       0      0 195.234.155.70.110 =
85.81.181.86.38322 TIME_WAIT
> fffffe006b469090 tcp4       0      0 195.234.155.70.110 =
85.81.181.86.47880 TIME_WAIT
> fffffe006b469168 tcp4       0      0 195.234.155.70.25  =
190.138.186.235.19 TIME_WAIT
> fffffe006b469120 tcp4       0      0 195.234.155.70.25  =
201.188.10.227.302 TIME_WAIT
> fffffe0132199000 tcp4       0     95 195.234.155.70.25  =
212.156.16.66.2593 ESTABLISHED
> fffffe013219a000 tcp4       0      0 195.234.155.70.22  =
85.233.252.158.373 ESTABLISHED
> fffffe01326db000 tcp4       0      0 *.21               *.*            =
    LISTEN
> fffffe013219a3d0 tcp4       0      0 127.0.0.1.51629    =
127.0.0.1.51635    ESTABLISHED
> fffffe0006d953d0 tcp4       0      0 127.0.0.1.51635    =
127.0.0.1.51629    ESTABLISHED
> fffffe01326db3d0 tcp4       0      0 *.13782            *.*            =
    LISTEN
> fffffe0132697b70 tcp4       0      0 127.0.0.1.51629    =
127.0.0.1.51632    ESTABLISHED
> fffffe01326dbb70 tcp4       0      0 127.0.0.1.51632    =
127.0.0.1.51629    ESTABLISHED
> fffffe0006f7d3d0 tcp4       0      0 *.13724            *.*            =
    LISTEN
> fffffe01326843d0 tcp4       0      0 *.22               *.*            =
    LISTEN
> fffffe01326847a0 tcp6       0      0 *.22               *.*            =
    LISTEN
> fffffe0006f7d000 tcp4       0      0 127.0.0.1.51629    *.*            =
    LISTEN
> fffffe0132153000 tcp4       0      0 127.0.0.1.1557     *.*            =
    LISTEN
> fffffe0006d95b70 tcp4       0      0 *.1556             *.*            =
    LISTEN
> fffffe01321533d0 tcp6       0      0 *.1556             *.*            =
    LISTEN
> fffffe0006342dc8 udp4       0      0 *.15949            *.*            =
   =20
> fffffe0006dcf930 udp6       0      0 *.*                *.*            =
   =20
> fffffe0006da2dc8 udp4       0      0 *.25350            *.*            =
   =20
> fffffe006b328498 udp6       0      0 *.*                *.*            =
   =20
> fffffe0006da2c40 udp4       0      0 195.234.155.143.58 8.8.8.8.53     =
   =20
> fffffe0006370000 udp4     631      0 195.234.155.143.95 8.8.8.8.53     =
   =20
> fffffe0006da2ab8 udp4       0      0 127.0.0.1.53       *.*            =
   =20
> fffffe0006375ab8 udp4       0      0 195.234.155.102.53 *.*            =
   =20
> fffffe00063497a8 udp4     149      0 *.56674            *.*            =
   =20
> fffffe0006370dc8 udp4     115      0 *.64883            *.*            =
   =20
> fffffe0006342498 udp4     128      0 *.43982            *.*            =
   =20
> fffffe0006dcfc40 udp4       0      0 127.0.0.1.123      *.*            =
   =20
> fffffe0006dcfdc8 udp6       0      0 fe80:5::1.123      *.*            =
   =20
> fffffe0006da2188 udp6       0      0 ::1.123            *.*            =
   =20
> fffffe0006da2498 udp6       0      0 fe80:4::5054:ff:.1 *.*            =
   =20
> fffffe0006371620 udp4       0      0 10.0.0.3.123       *.*            =
   =20
> fffffe0006371ab8 udp4       0      0 195.234.155.75.123 *.*            =
   =20
> fffffe0006375310 udp4       0      0 195.234.155.124.12 *.*            =
   =20
> fffffe0006375188 udp4       0      0 195.234.155.50.123 *.*            =
   =20
> fffffe0006375000 udp4       0      0 195.234.155.102.12 *.*            =
   =20
> fffffe0006371dc8 udp4       0      0 195.234.155.70.123 *.*            =
   =20
> fffffe0006371c40 udp6       0      0 fe80:3::5054:ff:.1 *.*            =
   =20
> fffffe0006371930 udp4       0      0 195.234.155.143.12 *.*            =
   =20
> fffffe00063717a8 udp6       0      0 fe80:2::216:8eff.1 *.*            =
   =20
> fffffe0006371188 udp4       0      0 192.168.122.3.123  *.*            =
   =20
> fffffe0006371310 udp6       0      0 *.123              *.*            =
   =20
> fffffe0006371498 udp4       0      0 *.123              *.*            =
   =20
> fffffe0006375c40 udp4       0      0 *.514              *.*            =
   =20
> fffffe0006375dc8 udp6       0      0 *.514              *.*            =
   =20
> Active UNIX domain sockets
> Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
> fffffe010a3f0690 stream      0      0        0 fffffe006be66e10        =
0        0
> fffffe006be66e10 stream      0      0        0 fffffe010a3f0690        =
0        0
> fffffe006be66c30 stream      0      0        0        0        0       =
 0
> fffffe010ab1dd20 stream      0      0        0 fffffe010ab1de10        =
0        0
> fffffe010ab1de10 stream      0      0        0 fffffe010ab1dd20        =
0        0
> fffffe010ab1e0f0 stream      0      0        0        0        0       =
 0
> fffffe006be67000 stream      0      0        0 fffffe006be674b0        =
0        0
> fffffe006be674b0 stream      0      0        0 fffffe006be67000        =
0        0
> fffffe006be67e10 stream      0      0        0        0        0       =
 0
> fffffe010ab69000 stream      0      0        0 fffffe010ab690f0        =
0        0
> fffffe010ab690f0 stream      0      0        0 fffffe010ab69000        =
0        0
> fffffe000662ac30 stream      0      0        0        0        0       =
 0
> fffffe000662a690 stream      0      0        0        0        0       =
 0
> fffffe006be665a0 stream      0      0        0        0        0       =
 0
> fffffe006be66780 stream      0      0        0 fffffe006be67780        =
0        0
> fffffe006be67780 stream      0      0        0 fffffe006be66780        =
0        0
> fffffe006b45d960 stream      0      0        0 fffffe00065703c0        =
0        0
> fffffe00065703c0 stream      0      0        0 fffffe006b45d960        =
0        0
> fffffe006be67870 stream      0      0        0 fffffe000656f4b0        =
0        0
> fffffe000656f4b0 stream      0      0        0 fffffe006be67870        =
0        0
> fffffe0006570960 stream      0      0        0        0        0       =
 0
> fffffe006be67d20 stream      0      0        0        0        0       =
 0
> fffffe01321d13c0 stream      0      0        0        0        0       =
 0
> fffffe006be66960 stream      0      0        0 fffffe006be661e0        =
0        0
> fffffe006be661e0 stream      0      0        0 fffffe006be66960        =
0        0
> fffffe01321d1a50 stream      0      0        0 fffffe006be67960        =
0        0
> fffffe006be67960 stream      0      0        0 fffffe01321d1a50        =
0        0
> fffffe006be662d0 stream      0      0        0 fffffe000656fd20        =
0        0
> fffffe000656fd20 stream      0      0        0 fffffe006be662d0        =
0        0
> fffffe01321d12d0 stream      0      0        0 fffffe01321d1960        =
0        0
> fffffe01321d1960 stream      0      0        0 fffffe01321d12d0        =
0        0
> fffffe010a3f04b0 stream      0      0        0        0        0       =
 0
> fffffe00065705a0 stream      0      0        0 fffffe01321d1d20        =
0        0
> fffffe01321d1d20 stream      0      0        0 fffffe00065705a0        =
0        0
> fffffe00066e3870 stream      0      0        0        0        0       =
 0
> fffffe006be66690 stream      0      0        0        0        0       =
 0
> fffffe010a3f00f0 stream      0      0        0 fffffe010a3f01e0        =
0        0
> fffffe010a3f01e0 stream      0      0        0 fffffe010a3f00f0        =
0        0
> fffffe006be66d20 stream      0      0        0        0        0       =
 0
> fffffe010a3f02d0 stream      0      0        0 fffffe010a3f03c0        =
0        0
> fffffe010a3f03c0 stream      0      0        0 fffffe010a3f02d0        =
0        0
> fffffe010a3f05a0 stream      0      0        0        0        0       =
 0
> fffffe00066e34b0 stream      0      0        0 fffffe006b45d3c0        =
0        0
> fffffe006b45d3c0 stream      0      0        0 fffffe00066e34b0        =
0        0
> fffffe00066e3000 stream      0      0        0 fffffe000662a5a0        =
0        0
> fffffe000662a5a0 stream      0      0        0 fffffe00066e3000        =
0        0
> fffffe010a3f0780 stream      0      0        0        0        0       =
 0
> fffffe006b45dc30 stream      0      0        0        0        0       =
 0
> fffffe006b45dd20 stream      0      0        0 fffffe006b45de10        =
0        0
> fffffe006b45de10 stream      0      0        0 fffffe006b45dd20        =
0        0
> fffffe010a3f0870 stream      0      0        0 fffffe010a3f0960        =
0        0
> fffffe010a3f0960 stream      0      0        0 fffffe010a3f0870        =
0        0
> fffffe0132a5dc30 stream      0      0        0        0        0       =
 0
> fffffe010a3f0a50 stream      0      0        0 fffffe010a3f0b40        =
0        0
> fffffe010a3f0b40 stream      0      0        0 fffffe010a3f0a50        =
0        0
> fffffe00066e33c0 stream      0      0        0        0        0       =
 0
> fffffe00065700f0 stream      0      0        0 fffffe006b45d4b0        =
0        0
> fffffe006b45d4b0 stream      0      0        0 fffffe00065700f0        =
0        0
> fffffe006b45d870 stream      0      0        0        0        0       =
 0
> fffffe006be663c0 stream      0      0        0 fffffe006be664b0        =
0        0
> fffffe006be664b0 stream      0      0        0 fffffe006be663c0        =
0        0
> fffffe00064cf4b0 stream      0      0        0        0        0       =
 0
> fffffe000662a4b0 stream      0      0        0 fffffe000662ab40        =
0        0
> fffffe000662ab40 stream      0      0        0 fffffe000662a4b0        =
0        0
> fffffe006be66a50 stream      0      0        0        0        0       =
 0
> fffffe000656fa50 stream      0      0        0 fffffe00066e30f0        =
0        0
> fffffe00066e30f0 stream      0      0        0 fffffe000656fa50        =
0        0
> fffffe00064cfd20 stream      0      0        0        0        0       =
 0
> fffffe006b45d5a0 stream      0      0        0 fffffe0132a5d1e0        =
0        0
> fffffe0132a5d1e0 stream      0      0        0 fffffe006b45d5a0        =
0        0
> fffffe0006570e10 stream      0      0        0        0        0       =
 0
> fffffe006b45da50 stream      0      0        0        0        0       =
 0
> fffffe00064cf2d0 stream      0      0        0 fffffe00064cf1e0        =
0        0
> fffffe00064cf1e0 stream      0      0        0 fffffe00064cf2d0        =
0        0
> fffffe006b45d690 stream      0      0        0        0        0       =
 0
> fffffe006be671e0 stream      0      0        0 fffffe006be672d0        =
0        0
> fffffe006be672d0 stream      0      0        0 fffffe006be671e0        =
0        0
> fffffe0006570b40 stream      0      0        0 fffffe00066e35a0        =
0        0
> fffffe00066e35a0 stream      0      0        0 fffffe0006570b40        =
0        0
> fffffe000662a3c0 stream      0      0        0        0        0       =
 0
> fffffe0132a5db40 stream      0      0        0 fffffe00066e3690        =
0        0
> fffffe00066e3690 stream      0      0        0 fffffe0132a5db40        =
0        0
> fffffe0006570c30 stream      0      0        0        0        0       =
 0
> fffffe0132a5d000 stream      0      0        0 fffffe01321d1e10        =
0        0
> fffffe01321d1e10 stream      0      0        0 fffffe0132a5d000        =
0        0
> fffffe0006570d20 stream      0      0        0        0        0       =
 0
> fffffe00066e31e0 stream      0      0        0 fffffe0006570a50        =
0        0
> fffffe0006570a50 stream      0      0        0 fffffe00066e31e0        =
0        0
> fffffe00064cf5a0 stream      0      0        0        0        0       =
 0
> fffffe000662a2d0 stream      0      0        0 fffffe01321d10f0        =
0        0
> fffffe01321d10f0 stream      0      0        0 fffffe000662a2d0        =
0        0
> fffffe000656f2d0 stream      0      0        0 fffffe000656f1e0        =
0        0
> fffffe000656f1e0 stream      0      0        0 fffffe000656f2d0        =
0        0
> fffffe00064cf0f0 stream      0      0        0 fffffe00064cf000        =
0        0
> fffffe00064cf000 stream      0      0        0 fffffe00064cf0f0        =
0        0
> fffffe000662ae10 stream      0      0        0 fffffe000662ad20        =
0        0
> fffffe000662ad20 stream      0      0        0 fffffe000662ae10        =
0        0
> fffffe0132a5e0f0 stream      0      0        0 fffffe0132a5e000        =
0        0
> fffffe0132a5e000 stream      0      0        0 fffffe0132a5e0f0        =
0        0
> fffffe000656f780 stream      0      0        0 fffffe000662a1e0        =
0        0
> fffffe000662a1e0 stream      0      0        0 fffffe000656f780        =
0        0
> fffffe00065704b0 stream      0      0        0 fffffe0132a5d5a0        =
0        0
> fffffe0132a5d5a0 stream      0      0        0 fffffe00065704b0        =
0        0
> fffffe000656f0f0 stream      0      0        0 fffffe00064cfa50        =
0        0
> fffffe00064cfa50 stream      0      0        0 fffffe000656f0f0        =
0        0
> fffffe00064cfe10 stream      0      0        0 fffffe00064cfb40        =
0        0
> fffffe00064cfb40 stream      0      0        0 fffffe00064cfe10        =
0        0
> fffffe0132a5d4b0 stream      0      0        0 fffffe0132a5d3c0        =
0        0
> fffffe0132a5d3c0 stream      0      0        0 fffffe0132a5d4b0        =
0        0
> fffffe0132a5d2d0 stream      0      0        0 fffffe000656f000        =
0        0
> fffffe000656f000 stream      0      0        0 fffffe0132a5d2d0        =
0        0
> fffffe000662aa50 stream      0      0        0 fffffe000662a960        =
0        0
> fffffe000662a960 stream      0      0        0 fffffe000662aa50        =
0        0
> fffffe000662a870 stream      0      0        0 fffffe000662a780        =
0        0
> fffffe000662a780 stream      0      0        0 fffffe000662a870        =
0        0
> fffffe0132a5e2d0 stream      0      0        0 fffffe0132a5e3c0        =
0        0
> fffffe0132a5e3c0 stream      0      0        0 fffffe0132a5e2d0        =
0        0
> fffffe00065702d0 stream      0      0        0 fffffe01321d14b0        =
0        0
> fffffe01321d14b0 stream      0      0        0 fffffe00065702d0        =
0        0
> fffffe0132a5d960 stream      0      0        0 fffffe00066e3a50        =
0        0
> fffffe00066e3a50 stream      0      0        0 fffffe0132a5d960        =
0        0
> fffffe0132a5e4b0 stream      0      0        0 fffffe0132a5e5a0        =
0        0
> fffffe0132a5e5a0 stream      0      0        0 fffffe0132a5e4b0        =
0        0
> fffffe00066e3b40 stream      0      0        0 fffffe000662a0f0        =
0        0
> fffffe000662a0f0 stream      0      0        0 fffffe00066e3b40        =
0        0
> fffffe0132a5e690 stream      0      0        0 fffffe0132a5e780        =
0        0
> fffffe0132a5e780 stream      0      0        0 fffffe0132a5e690        =
0        0
> fffffe000656fc30 stream      0      0        0 fffffe0132a5da50        =
0        0
> fffffe0132a5da50 stream      0      0        0 fffffe000656fc30        =
0        0
> fffffe00064cf3c0 stream      0      0        0 fffffe0132a5de10        =
0        0
> fffffe0132a5de10 stream      0      0        0 fffffe00064cf3c0        =
0        0
> fffffe0132a5ee10 stream      0      0        0 fffffe006b45d1e0        =
0        0
> fffffe006b45d1e0 stream      0      0        0 fffffe0132a5ee10        =
0        0
> fffffe006b45d0f0 stream      0      0        0 fffffe01321d11e0        =
0        0
> fffffe01321d11e0 stream      0      0        0 fffffe006b45d0f0        =
0        0
> fffffe0132a5e870 stream      0      0        0 fffffe0132a5e960        =
0        0
> fffffe0132a5e960 stream      0      0        0 fffffe0132a5e870        =
0        0
> fffffe0132a5ea50 stream      0      0        0 fffffe0132a5eb40        =
0        0
> fffffe0132a5eb40 stream      0      0        0 fffffe0132a5ea50        =
0        0
> fffffe0132a5ec30 stream      0      0 fffffe006b4e83c0        0        =
0        0 /tmp/mysql.sock
> fffffe000656fb40 stream      0      0        0        0        0       =
 0
> fffffe0132a5e1e0 stream      0      0        0 fffffe0132a5d780        =
0        0
> fffffe0132a5d780 stream      0      0        0 fffffe0132a5e1e0        =
0        0
> fffffe01321d1b40 stream      0      0 fffffe006b4e5d20        0        =
0        0 /var/run/authdaemond/socket.tmp
> fffffe0132a5d690 stream      0      0 fffffe006b1fc000        0        =
0        0 /var/run/clamav/clamd
> fffffe000662a000 stream      0      0        0 fffffe01321d1870        =
0        0
> fffffe01321d1870 stream      0      0        0 fffffe000662a000        =
0        0
> fffffe00066e3c30 stream      0      0        0 fffffe00066e3d20        =
0        0 /tmp/PBXPIPEbpcd
> fffffe00066e3d20 stream      0      0        0 fffffe00066e3c30        =
0        0
> fffffe000656f690 stream      0      0 fffffe0132732780        0        =
0        0 /usr/openv/var/vnetd/bpcd.uds
> fffffe01321d1690 stream      0      0 fffffe01326b53c0        0        =
0        0 /usr/openv/var/vnetd/terminate_bpcd.uds
> fffffe00064cf960 stream      0      0        0 fffffe00065701e0        =
0        0 /tmp/PBXPIPEvnetd
> fffffe00065701e0 stream      0      0        0 fffffe00064cf960        =
0        0
> fffffe0006570000 stream      0      0 fffffe0132652000        0        =
0        0 /usr/openv/var/vnetd/terminate_vnetd.uds
> fffffe00066e3e10 stream      0      0        0 fffffe01321d1000        =
0        0
> fffffe01321d1000 stream      0      0        0 fffffe00066e3e10        =
0        0
> fffffe000656fe10 stream      0      0 fffffe0006381960        0        =
0        0 /var/run/devd.pipe
> fffffe00064cf780 dgram       0      0        0 fffffe000656f870        =
0 fffffe00066e3780
> fffffe00066e3780 dgram       0      0        0 fffffe000656f870        =
0 fffffe0132a5dd20
> fffffe0132a5dd20 dgram       0      0        0 fffffe000656f870        =
0 fffffe00064cf870
> fffffe00064cf870 dgram       0      0        0 fffffe000656f870        =
0 fffffe000656f5a0
> fffffe000656f5a0 dgram       0      0        0 fffffe000656f870        =
0        0
> fffffe000656f870 dgram       0      0 fffffe00067035a0        0 =
fffffe00064cf780        0 /var/run/logpriv
> fffffe000656f960 dgram       0      0 fffffe0006703780        0        =
0        0 /var/run/log
>=20
> =
------------------------------------------------------------------------
> netstat -aL
>=20
> Current listen queue sizes (qlen/incqlen/maxqlen)
> Proto Listen         Local Address        =20
> tcp4  0/0/20         localhost.26          =20
> tcp4  0/0/20         *.smtp                =20
> tcp4  0/0/20         localhost.domain      =20
> tcp4  0/0/50         *.submission          =20
> tcp4  0/0/50         localhost.mysql       =20
> tcp4  0/0/128        dentredjeb.4949       =20
> tcp46 0/0/128        *.pop3                =20
> tcp46 0/0/128        *.imaps               =20
> tcp46 0/0/128        *.imap                =20
> tcp4  0/0/128        *.https               =20
> tcp6  0/0/128        *.https               =20
> tcp4  0/0/128        *.xfer                =20
> tcp6  0/0/128        *.xfer                =20
> tcp4  0/0/128        *.http                =20
> tcp6  0/0/128        *.http                =20
> tcp4  0/0/64         *.ftp                 =20
> tcp4  0/0/5          *.13782               =20
> tcp4  0/0/5          *.13724               =20
> tcp4  0/0/128        *.ssh                 =20
> tcp6  0/0/128        *.ssh                 =20
> tcp4  0/0/5          localhost.51629       =20
> tcp4  0/0/5          localhost.1557        =20
> tcp4  0/0/5          *.1556                =20
> tcp6  0/0/5          *.1556                =20
> unix  0/0/50         /tmp/mysql.sock
> unix  0/0/128        /var/run/authdaemond/socket.tmp
> unix  0/0/128        /var/run/clamav/clamd
> unix  0/0/5          /usr/openv/var/vnetd/bpcd.uds
> unix  0/0/1          /usr/openv/var/vnetd/terminate_bpcd.uds
> unix  0/0/1          /usr/openv/var/vnetd/terminate_vnetd.uds
> unix  0/0/4          /var/run/devd.pipe
>=20
> =
------------------------------------------------------------------------
> fstat
> [root@dentredje crash]#
> ***
>=20
>=20
> On 05/03/2013, at 17.56, Rasmus Skaarup <freebsd@gal.dk> wrote:
>=20
>>=20
>> kgdb) frame 4
>> #4  0xffffffff816bd5ba in zfs_freebsd_read (ap=3DVariable "ap" is not =
available.
>> )
>>   at =
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
zfs_vnops.c:476
>> warning: Source file is more recent than executable.
>>=20
>> 476					if (pp->wire_count =3D=3D 0 && =
pp->valid =3D=3D 0 &&
>> (kgdb) info reg
>> rax            0x0	0
>> rbx            0x7a	122
>> rcx            0x0	0
>> rdx            0x0	0
>> rsi            0x0	0
>> rdi            0x0	0
>> rbp            0xffffff8162323810	0xffffff8162323810
>> rsp            0xffffff8162323710	0xffffff8162323710
>> r8             0x0	0
>> r9             0x0	0
>> r10            0x0	0
>> r11            0x0	0
>> r12            0xfffffe015123cc80	-2193366987648
>> r13            0xfffffe0006aa1000	-2198911447040
>> r14            0xfffffe0006aa10e8	-2198911446808
>> r15            0xffffff8162323850	-543813388208
>> rip            0xffffffff816bd5ba	0xffffffff816bd5ba =
<zfs_freebsd_read+1578>
>> eflags         0x0	0
>> cs             0x0	0
>> ss             0x0	0
>> ds             0x0	0
>> es             0x0	0
>> fs             0x0	0
>> gs             0x0	0
>> (kgdb)=20
>>=20
>>=20
>> I guess line 476 changed after your patch, but I guess you can use =
the output anyway.
>>=20
>> Br
>> Rasmus Skaarup
>>=20
>> On 05/03/2013, at 17.51, Andriy Gapon <avg@FreeBSD.org> wrote:
>>=20
>>>=20
>>> I need 'info reg' from zfs_freebsd_read frame, which frame 4 in =
kgdb.
>>>=20
>>>=20
>>> --=20
>>> Andriy Gapon
>>>=20
>>=20
>=20
Comment 7 Andriy Gapon freebsd_committer freebsd_triage 2013-03-06 13:36:38 UTC
Is this on the same physical machine where you got the vm_page_free_toq panic or
on a different machine?
Please test the change on a machine different from the machine where you were
getting all the different panics that you described in the other/previous PR.

-- 
Andriy Gapon
Comment 8 Rasmus Skaarup 2013-03-06 14:24:27 UTC
Yes, I have tried to move the virtual machine to two different other =
physical hosts. Exactly the same crashes there.

Is it a failure in the ZFS filesystem that causes the panics?

Best regards,
Rasmus Skaarup=
Comment 9 Andriy Gapon freebsd_committer freebsd_triage 2013-03-06 14:40:49 UTC
on 06/03/2013 16:24 Rasmus Skaarup said the following:
> 
> Yes, I have tried to move the virtual machine to two different other physical hosts. Exactly the same crashes there.
> 
> Is it a failure in the ZFS filesystem that causes the panics?

Since the panics look so random I can not really say.

-- 
Andriy Gapon
Comment 10 Rasmus Skaarup 2013-03-06 14:55:50 UTC
The panics are all zfs related. I occurs like clock work when I type =
"zpool status".

Br
Rasmus Skaarup

On 06/03/2013, at 15.40, Andriy Gapon <avg@FreeBSD.org> wrote:

> on 06/03/2013 16:24 Rasmus Skaarup said the following:
>>=20
>> Yes, I have tried to move the virtual machine to two different other =
physical hosts. Exactly the same crashes there.
>>=20
>> Is it a failure in the ZFS filesystem that causes the panics?
>=20
> Since the panics look so random I can not really say.
>=20
> --=20
> Andriy Gapon
>=20
Comment 11 Andriy Gapon freebsd_committer freebsd_triage 2013-03-06 14:59:49 UTC
on 06/03/2013 16:55 Rasmus Skaarup said the following:
> 
> The panics are all zfs related. I occurs like clock work when I type "zpool status".

They do happen in random places all over the zfs code.

-- 
Andriy Gapon
Comment 12 Rasmus Skaarup 2013-03-06 15:22:59 UTC
Tonight I will migrate all data to a UFS drive and let it run for a few =
days. If it works, I will create a new ZFS pool and move data back and =
see what happens.

Is there any information you need before I proceed?

Br
Rasmus Skaarup=
Comment 13 Rasmus Skaarup 2013-03-07 05:00:46 UTC
This is the only kind of panic I get - after your patch:

Fatal trap 12: page fault while in kernel mode
cpuid =3D 1; apic id =3D 01
fault virtual address   =3D 0x60
fault code              =3D supervisor read data, page not present
instruction pointer     =3D 0x20:0xffffffff8162e4f0
stack pointer           =3D 0x28:0xffffff81624726e0
frame pointer           =3D 0x28:0xffffff81624727d0
code segment            =3D base rx0, limit 0xfffff, type 0x1b
                        =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
current process         =3D 26068 (zpool)
trap number             =3D 12
panic: page fault
cpuid =3D 1
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff81673975 at sa_handle_get_from_db+0x95
#7 0xffffffff81673a38 at sa_handle_get+0x48
#8 0xffffffff8169f516 at zfs_grab_sa_handle+0x96
#9 0xffffffff8169faca at zfs_obj_to_path+0x6a
#10 0xffffffff816b8c75 at zfs_ioc_obj_to_path+0x75
#11 0xffffffff816bad46 at zfsdev_ioctl+0xe6
#12 0xffffffff807db28b at devfs_ioctl_f+0x7b
#13 0xffffffff80932325 at kern_ioctl+0x115
#14 0xffffffff8093255d at sys_ioctl+0xfd
#15 0xffffffff80bd7ae6 at amd64_syscall+0x546
#16 0xffffffff80bc3447 at Xfast_syscall+0xf7

Br
Rasmus

On 06/03/2013, at 16.22, Rasmus Skaarup <freebsd@gal.dk> wrote:

>=20
> Tonight I will migrate all data to a UFS drive and let it run for a =
few days. If it works, I will create a new ZFS pool and move data back =
and see what happens.
>=20
> Is there any information you need before I proceed?
>=20
> Br
> Rasmus Skaarup
Comment 14 Rasmus Skaarup 2013-03-09 23:05:35 UTC
After moving to UFS I no longer experience crashes at all.

Let me know if there is anything I can do.

Br
Rasmus Skaarup
Comment 15 Andriy Gapon freebsd_committer freebsd_triage 2013-03-10 11:01:32 UTC
on 07/03/2013 07:00 Rasmus Skaarup said the following:
> 
> This is the only kind of panic I get - after your patch:
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 01
> fault virtual address   = 0x60
> fault code              = supervisor read data, page not present
> instruction pointer     = 0x20:0xffffffff8162e4f0
> stack pointer           = 0x28:0xffffff81624726e0
> frame pointer           = 0x28:0xffffff81624727d0
> code segment            = base rx0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 26068 (zpool)
> trap number             = 12
> panic: page fault
> cpuid = 1
> KDB: stack backtrace:
> #0 0xffffffff809208a6 at kdb_backtrace+0x66
> #1 0xffffffff808ea8be at panic+0x1ce
> #2 0xffffffff80bd8240 at trap_fatal+0x290
> #3 0xffffffff80bd857d at trap_pfault+0x1ed
> #4 0xffffffff80bd8b9e at trap+0x3ce
> #5 0xffffffff80bc315f at calltrap+0x8
> #6 0xffffffff81673975 at sa_handle_get_from_db+0x95
> #7 0xffffffff81673a38 at sa_handle_get+0x48
> #8 0xffffffff8169f516 at zfs_grab_sa_handle+0x96
> #9 0xffffffff8169faca at zfs_obj_to_path+0x6a
> #10 0xffffffff816b8c75 at zfs_ioc_obj_to_path+0x75
> #11 0xffffffff816bad46 at zfsdev_ioctl+0xe6
> #12 0xffffffff807db28b at devfs_ioctl_f+0x7b
> #13 0xffffffff80932325 at kern_ioctl+0x115
> #14 0xffffffff8093255d at sys_ioctl+0xfd
> #15 0xffffffff80bd7ae6 at amd64_syscall+0x546
> #16 0xffffffff80bc3447 at Xfast_syscall+0xf7

It is possible that while there were the memory corruptions (either because of
the bug for which I sent you the patch or for some other reason), some bad /
corrupted ZFS metadata was written to the stable storage.  Now that corrupted
data could be causing further panics.  It would be interesting to re-create a
pool from scratch and see how that behaves.  If you do that, please use the patch.

-- 
Andriy Gapon
Comment 16 Rasmus Skaarup 2013-03-10 11:45:30 UTC
I am deciding whether to do this or not, the users had been impacted for =
a couple of days, and I wouldn't become very popular if I introduce the =
same errors again.

I did a 'zfs scrub' after migrating to a healthy system, but apparently =
that didn't fix the corruption.

Br
Rasmus Skaarup

On 10/03/2013, at 12.01, Andriy Gapon <avg@FreeBSD.org> wrote:

> on 07/03/2013 07:00 Rasmus Skaarup said the following:
>>=20
>> This is the only kind of panic I get - after your patch:
>>=20
>> Fatal trap 12: page fault while in kernel mode
>> cpuid =3D 1; apic id =3D 01
>> fault virtual address   =3D 0x60
>> fault code              =3D supervisor read data, page not present
>> instruction pointer     =3D 0x20:0xffffffff8162e4f0
>> stack pointer           =3D 0x28:0xffffff81624726e0
>> frame pointer           =3D 0x28:0xffffff81624727d0
>> code segment            =3D base rx0, limit 0xfffff, type 0x1b
>>                        =3D DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
>> current process         =3D 26068 (zpool)
>> trap number             =3D 12
>> panic: page fault
>> cpuid =3D 1
>> KDB: stack backtrace:
>> #0 0xffffffff809208a6 at kdb_backtrace+0x66
>> #1 0xffffffff808ea8be at panic+0x1ce
>> #2 0xffffffff80bd8240 at trap_fatal+0x290
>> #3 0xffffffff80bd857d at trap_pfault+0x1ed
>> #4 0xffffffff80bd8b9e at trap+0x3ce
>> #5 0xffffffff80bc315f at calltrap+0x8
>> #6 0xffffffff81673975 at sa_handle_get_from_db+0x95
>> #7 0xffffffff81673a38 at sa_handle_get+0x48
>> #8 0xffffffff8169f516 at zfs_grab_sa_handle+0x96
>> #9 0xffffffff8169faca at zfs_obj_to_path+0x6a
>> #10 0xffffffff816b8c75 at zfs_ioc_obj_to_path+0x75
>> #11 0xffffffff816bad46 at zfsdev_ioctl+0xe6
>> #12 0xffffffff807db28b at devfs_ioctl_f+0x7b
>> #13 0xffffffff80932325 at kern_ioctl+0x115
>> #14 0xffffffff8093255d at sys_ioctl+0xfd
>> #15 0xffffffff80bd7ae6 at amd64_syscall+0x546
>> #16 0xffffffff80bc3447 at Xfast_syscall+0xf7
>=20
> It is possible that while there were the memory corruptions (either =
because of
> the bug for which I sent you the patch or for some other reason), some =
bad /
> corrupted ZFS metadata was written to the stable storage.  Now that =
corrupted
> data could be causing further panics.  It would be interesting to =
re-create a
> pool from scratch and see how that behaves.  If you do that, please =
use the patch.
>=20
> --=20
> Andriy Gapon
>=20
Comment 17 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 07:58:55 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped
Comment 18 Graham Perrin freebsd_committer freebsd_triage 2022-10-17 12:40:30 UTC
Keyword: 

    patch
or  patch-ready

– in lieu of summary line prefix: 

    [patch]

* bulk change for the keyword
* summary lines may be edited manually (not in bulk). 

Keyword descriptions and search interface: 

    <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>
Comment 19 Graham Perrin freebsd_committer freebsd_triage 2022-12-23 10:56:53 UTC
We might reasonably assume that the type of crash that occurred in 2013 is no longer reproducible with ZFS or OpenZFS in 2022.