Hi guys, My server is crashing either after some days our hours up, here is debugging. Do you you have any idea what's going on? I'm running 10.3-STABLE, but before it was running with 10.2-STABLE with same issue. Thank you. # kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 7; apic id = 08 fault virtual address = 0x10 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff818f198c stack pointer = 0x28:0xfffffe0237ac2970 frame pointer = 0x28:0xfffffe0237ac29c0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 6 (trim zroot) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff808df650 at kdb_backtrace+0x60 #1 0xffffffff808a5d56 at vpanic+0x126 #2 0xffffffff808a5c23 at panic+0x43 #3 0xffffffff80cfd9ab at trap_fatal+0x36b #4 0xffffffff80cfdbec at trap_pfault+0x22c #5 0xffffffff80cfd32a at trap+0x47a #6 0xffffffff80ce45bc at calltrap+0x8 #7 0xffffffff818f1dd8 at avl_remove+0x1c8 #8 0xffffffff819c6395 at trim_map_commit+0x235 #9 0xffffffff819c6226 at trim_map_commit+0xc6 #10 0xffffffff819c5fa2 at trim_thread+0xf2 #11 0xffffffff80876788 at fork_exit+0xc8 #12 0xffffffff80ce4afe at fork_trampoline+0xe Uptime: 9h32m0s Dumping 705 out of 8152 MB: (CTRL-C to abort) ..3%..12%..21%..32%..41%..53%..62%..71%..82%..91% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/ng_UI.ko.symbols...done. Loaded symbols for /boot/kernel/ng_UI.ko.symbols Reading symbols from /boot/kernel/netgraph.ko.symbols...done. Loaded symbols for /boot/kernel/netgraph.ko.symbols Reading symbols from /boot/kernel/ng_async.ko.symbols...done. Loaded symbols for /boot/kernel/ng_async.ko.symbols Reading symbols from /boot/kernel/ng_bpf.ko.symbols...done. Loaded symbols for /boot/kernel/ng_bpf.ko.symbols Reading symbols from /boot/kernel/ng_cisco.ko.symbols...done. Loaded symbols for /boot/kernel/ng_cisco.ko.symbols Reading symbols from /boot/kernel/ng_echo.ko.symbols...done. Loaded symbols for /boot/kernel/ng_echo.ko.symbols Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. Loaded symbols for /boot/kernel/ng_ether.ko.symbols Reading symbols from /boot/kernel/ng_frame_relay.ko.symbols...done. Loaded symbols for /boot/kernel/ng_frame_relay.ko.symbols Reading symbols from /boot/kernel/ng_hole.ko.symbols...done. Loaded symbols for /boot/kernel/ng_hole.ko.symbols Reading symbols from /boot/kernel/ng_iface.ko.symbols...done. Loaded symbols for /boot/kernel/ng_iface.ko.symbols Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done. Loaded symbols for /boot/kernel/ng_ksocket.ko.symbols Reading symbols from /boot/kernel/ng_l2tp.ko.symbols...done. Loaded symbols for /boot/kernel/ng_l2tp.ko.symbols Reading symbols from /boot/kernel/ng_lmi.ko.symbols...done. Loaded symbols for /boot/kernel/ng_lmi.ko.symbols Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done. Loaded symbols for /boot/kernel/ng_mppc.ko.symbols Reading symbols from /boot/kernel/rc4.ko.symbols...done. Loaded symbols for /boot/kernel/rc4.ko.symbols Reading symbols from /boot/kernel/ng_netflow.ko.symbols...done. Loaded symbols for /boot/kernel/ng_netflow.ko.symbols Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done. Loaded symbols for /boot/kernel/ng_ppp.ko.symbols Reading symbols from /boot/kernel/ng_pppoe.ko.symbols...done. Loaded symbols for /boot/kernel/ng_pppoe.ko.symbols Reading symbols from /boot/kernel/ng_pptpgre.ko.symbols...done. Loaded symbols for /boot/kernel/ng_pptpgre.ko.symbols Reading symbols from /boot/kernel/ng_rfc1490.ko.symbols...done. Loaded symbols for /boot/kernel/ng_rfc1490.ko.symbols Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. Loaded symbols for /boot/kernel/ng_socket.ko.symbols Reading symbols from /boot/kernel/ng_tee.ko.symbols...done. Loaded symbols for /boot/kernel/ng_tee.ko.symbols Reading symbols from /boot/kernel/ng_tty.ko.symbols...done. Loaded symbols for /boot/kernel/ng_tty.ko.symbols Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done. Loaded symbols for /boot/kernel/ng_vjc.ko.symbols Reading symbols from /boot/kernel/coretemp.ko.symbols...done. Loaded symbols for /boot/kernel/coretemp.ko.symbols Reading symbols from /boot/modules/plcm.ko...done. Loaded symbols for /boot/modules/plcm.ko Reading symbols from /boot/kernel/ng_tcpmss.ko.symbols...done. Loaded symbols for /boot/kernel/ng_tcpmss.ko.symbols Reading symbols from /boot/kernel/ng_nat.ko.symbols...done. Loaded symbols for /boot/kernel/ng_nat.ko.symbols Reading symbols from /boot/kernel/ng_car.ko.symbols...done. Loaded symbols for /boot/kernel/ng_car.ko.symbols #0 doadump (textdump=<value optimized out>) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) list *0xffffffff818f198c 0xffffffff818f198c is in avl_rotation (/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:316). 311 avl_node_t *cright; 312 avl_node_t *gchild; 313 avl_node_t *gright; 314 avl_node_t *gleft; 315 int which_child = AVL_XCHILD(node); 316 int child_bal = AVL_XBALANCE(child); 317 318 /* BEGIN CSTYLED */ 319 /* 320 * case 1 : node is overly left heavy, the left child is balanced or Current language: auto; currently minimal (kgdb) bt #0 doadump (textdump=<value optimized out>) at pcpu.h:219 #1 0xffffffff808a5a2c in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486 #2 0xffffffff808a5d95 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889 #3 0xffffffff808a5c23 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:818 #4 0xffffffff80cfd9ab in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:858 #5 0xffffffff80cfdbec in trap_pfault (frame=0xfffffe0237ac28c0, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:681 #6 0xffffffff80cfd32a in trap (frame=0xfffffe0237ac28c0) at /usr/src/sys/amd64/amd64/trap.c:447 #7 0xffffffff80ce45bc in calltrap () at /usr/src/sys/amd64/amd64/exception.S:238 #8 0xffffffff818f198c in avl_rotation (tree=0xfffff800112f6020, node=0xfffff8014b6c0880, balance=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:315 #9 0xffffffff818f1dd8 in avl_remove (tree=0xfffff800112f6020, data=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:818 #10 0xffffffff819c6395 in trim_map_commit (spa=0xfffff8000ebfa000, zio=0xfffff800110b93b8, vd=0xfffff800110bf000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:472 #11 0xffffffff819c6226 in trim_map_commit (spa=0xfffff8000ebfa000, zio=0xfffff800110b93b8, vd=0xfffff8000eaf8800) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:539 #12 0xffffffff819c5fa2 in trim_thread (arg=0xfffff8000ebfa000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:586 #13 0xffffffff80876788 in fork_exit (callout=0xffffffff819c5eb0 <trim_thread>, arg=0xfffff8000ebfa000, frame=0xfffffe0237ac2c00) at /usr/src/sys/kern/kern_fork.c:1027 #14 0xffffffff80ce4afe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:613 #15 0x0000000000000000 in ?? ()