Fri Jun 19 17:44:09 MSK 2015 FreeBSD server.dc3.instatfootball.tv 10.1-RELEASE-p13 FreeBSD 10.1-RELEASE-p13 #2: Thu Jun 18 21:51:24 MSK 2015 admin@server.dc3.instatfootball.tv:/usr/obj/usr/src/sys/CORV1 amd64 panic: general protection fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid = 4; apic id = 04 instruction pointer = 0x20:0xffffffff80aff484 stack pointer = 0x28:0xfffffe08367bb820 frame pointer = 0x28:0xfffffe08367bb860 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi4: clock) trap number = 9 panic: general protection fault cpuid = 4 KDB: stack backtrace: #0 0xffffffff80967d80 at kdb_backtrace+0x60 #1 0xffffffff8092ca55 at panic+0x155 #2 0xffffffff80d7344f at trap_fatal+0x38f #3 0xffffffff80d730ac at trap+0x75c #4 0xffffffff80d58cb2 at calltrap+0x8 #5 0xffffffff80affaef at tcp_tw_2msl_scan+0x8f #6 0xffffffff80afdaa5 at tcp_slowtimo+0x95 #7 0xffffffff809989d4 at pfslowtimo+0x54 #8 0xffffffff80941f17 at softclock_call_cc+0x177 #9 0xffffffff80942354 at softclock+0x94 #10 0xffffffff808ff5eb at intr_event_execute_handlers+0xab #11 0xffffffff808ffa36 at ithread_loop+0x96 #12 0xffffffff808fd13a at fork_exit+0x9a #13 0xffffffff80d591ee at fork_trampoline+0xe Uptime: 19h23m41s Dumping 2904 out of 32603 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/if_re.ko.symbols...done. Loaded symbols for /boot/kernel/if_re.ko.symbols Reading symbols from /boot/kernel/tmpfs.ko.symbols...done. Loaded symbols for /boot/kernel/tmpfs.ko.symbols Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. Loaded symbols for /boot/kernel/fdescfs.ko.symbols #0 doadump (textdump=<value optimized out>) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219 #1 0xffffffff8092c6d2 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff8092ca94 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80d7344f in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865 #4 0xffffffff80d730ac in trap (frame=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:203 #5 0xffffffff80d58cb2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #6 0xffffffff80aff484 in tcp_twclose (tw=0xfffff804e5baf268, reuse=0) at /usr/src/sys/netinet/tcp_timewait.c:631 #7 0xffffffff80affaef in tcp_tw_2msl_scan (reuse=0) at /usr/src/sys/netinet/tcp_timewait.c:645 #8 0xffffffff80afdaa5 in tcp_slowtimo () at /usr/src/sys/netinet/tcp_timer.c:148 #9 0xffffffff809989d4 in pfslowtimo (arg=0xfffff804e5baf268) at /usr/src/sys/kern/uipc_domain.c:508 #10 0xffffffff80941f17 in softclock_call_cc (c=0xffffffff816288d0, cc=0xffffffff816a5c80, direct=0) at /usr/src/sys/kern/kern_timeout.c:682 #11 0xffffffff80942354 in softclock (arg=0xffffffff816a5c80) at /usr/src/sys/kern/kern_timeout.c:810 #12 0xffffffff808ff5eb in intr_event_execute_handlers ( p=<value optimized out>, ie=0xfffff80005e5f700) at /usr/src/sys/kern/kern_intr.c:1263 #13 0xffffffff808ffa36 in ithread_loop (arg=0xfffff8000835e000) at /usr/src/sys/kern/kern_intr.c:1276 #14 0xffffffff808fd13a in fork_exit ( callout=0xffffffff808ff9a0 <ithread_loop>, arg=0xfffff8000835e000, frame=0xfffffe08367bbac0) at /usr/src/sys/kern/kern_fork.c:996 #15 0xffffffff80d591ee in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #16 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb)
Another panic on another machine with same kernel and same hardware: Tue Jun 23 03:04:35 MSK 2015 FreeBSD server.dc5.instatfootball.tv 10.1-RELEASE-p13 FreeBSD 10.1-RELEASE-p13 #2: Thu Jun 18 23:17:06 MSK 2015 root@server.dc5.instatfootball.tv:/usr/obj/usr/src/sys/CORV1 amd64 panic: page fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x6aeb00000378 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80934ceb stack pointer = 0x28:0xfffffe085935f050 frame pointer = 0x28:0xfffffe085935f0f0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 22645 (find) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff80967d80 at kdb_backtrace+0x60 #1 0xffffffff8092ca55 at panic+0x155 #2 0xffffffff80d7344f at trap_fatal+0x38f #3 0xffffffff80d73768 at trap_pfault+0x308 #4 0xffffffff80d72dca at trap+0x47a #5 0xffffffff80d58cb2 at calltrap+0x8 #6 0xffffffff8093486d at _sx_xlock+0x5d #7 0xffffffff81a06e59 at dnode_hold_impl+0x2f9 #8 0xffffffff819f746d at dmu_bonus_hold+0x1d #9 0xffffffff81a5a15c at zfs_zget+0xdc #10 0xffffffff81a71054 at zfs_dirent_lock+0x474 #11 0xffffffff81a7130d at zfs_dirlook+0x17d #12 0xffffffff81a86810 at zfs_lookup+0x280 #13 0xffffffff81a86c8d at zfs_freebsd_lookup+0x6d #14 0xffffffff80e90231 at VOP_CACHEDLOOKUP_APV+0xa1 #15 0xffffffff809baef6 at vfs_cache_lookup+0xd6 #16 0xffffffff80e90121 at VOP_LOOKUP_APV+0xa1 #17 0xffffffff809c33fc at lookup+0x59c Uptime: 4d2h53m41s Dumping 4403 out of 32603 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/if_re.ko.symbols...done. Loaded symbols for /boot/kernel/if_re.ko.symbols Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. Loaded symbols for /boot/kernel/if_lagg.ko.symbols Reading symbols from /boot/kernel/tmpfs.ko.symbols...done. Loaded symbols for /boot/kernel/tmpfs.ko.symbols Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. Loaded symbols for /boot/kernel/fdescfs.ko.symbols #0 doadump (textdump=<value optimized out>) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219 #1 0xffffffff8092c6d2 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff8092ca94 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80d7344f in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865 #4 0xffffffff80d73768 in trap_pfault (frame=0xfffffe085935efa0, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676 #5 0xffffffff80d72dca in trap (frame=0xfffffe085935efa0) at /usr/src/sys/amd64/amd64/trap.c:440 #6 0xffffffff80d58cb2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #7 0xffffffff80934ceb in _sx_xlock_hard (sx=0xfffff800bbb439c8, tid=18446735278531930400, opts=<value optimized out>, file=0x0, line=1496707472) at /usr/src/sys/kern/kern_sx.c:556 #8 0xffffffff8093486d in _sx_xlock (sx=0xfffff800368fe920, opts=0, file=<value optimized out>, line=0) at sx.h:152 #9 0xffffffff81a06e59 in dnode_hold_impl (os=<value optimized out>, object=<value optimized out>, flag=1, tag=<value optimized out>, dnp=0xfffffe085935f190) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1152 #10 0xffffffff819f746d in dmu_bonus_hold (os=0xfffff800bbb439c8, object=18446735278531930400, tag=<value optimized out>, dbp=0xfffffe085935f1d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:275 #11 0xffffffff81a5a15c in zfs_zget (zfsvfs=<value optimized out>, obj_num=13510, zpp=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1135 #12 0xffffffff81a71054 in zfs_dirent_lock (dlpp=0xfffffe085935f340, dzp=<value optimized out>, name=<value optimized out>, zpp=0x0, flag=<value optimized out>, direntflags=<value optimized out>, realpnp=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:318 #13 0xffffffff81a7130d in zfs_dirlook (dzp=0xfffff8001bf905c0, name=0xfffffe085935f420 "libcrypt.so", vpp=0xfffffe085935f868, flags=<value optimized out>, deflg=0x0, rpnp=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:426 #14 0xffffffff81a86810 in zfs_lookup () at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1490 #15 0xffffffff81a86c8d in zfs_freebsd_lookup (ap=0xfffffe085935f588) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6136 #16 0xffffffff80e90231 in VOP_CACHEDLOOKUP_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:197 #17 0xffffffff809baef6 in vfs_cache_lookup (ap=<value optimized out>) at vnode_if.h:80 #18 0xffffffff80e90121 in VOP_LOOKUP_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:129 #19 0xffffffff809c33fc in lookup (ndp=0xfffffe085935f808) at vnode_if.h:54 #20 0xffffffff809c2b64 in namei (ndp=0xfffffe085935f808) at /usr/src/sys/kern/vfs_lookup.c:302 #21 0xffffffff809d6e5e in kern_statat_vnhook (td=0xfffff800368fe920, flag=<value optimized out>, fd=-100, path=0x801093e48 <Address 0x801093e48 out of bounds>, pathseg=UIO_USERSPACE, sbp=0xfffffe085935f920, hook=0x6aeb00000000) at /usr/src/sys/kern/vfs_syscalls.c:2284 #22 0xffffffff809d6ff0 in sys_lstat (td=0xfffff800bbb439c8, uap=0xfffffe085935fa40) at /usr/src/sys/kern/vfs_syscalls.c:2264 #23 0xffffffff80d73d81 in amd64_syscall (td=0xfffff800368fe920, traced=0) at subr_syscall.c:134 #24 0xffffffff80d58f9b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 #25 0x000000080095b37a in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb)
Another panic on another machine with same kernel and same hardware: Wed Jun 24 20:30:56 MSK 2015 FreeBSD server.dc4.instatfootball.tv 10.1-RELEASE-p13 FreeBSD 10.1-RELEASE-p13 #2: Thu Jun 18 23:17:00 MSK 2015 root@server.dc4.instatfootball.tv:/usr/obj/usr/src/sys/CORV1 amd64 panic: general protection fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...kgdb: kvm_read: invalid address (0xffeff80015c45490) Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid = 4; apic id = 04 instruction pointer = 0x20:0xffffffff80931b40 stack pointer = 0x28:0xfffffe0859465830 frame pointer = 0x28:0xfffffe0859465900 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 44457 (httpd) trap number = 9 panic: general protection fault cpuid = 4 KDB: stack backtrace: #0 0xffffffff80967d80 at kdb_backtrace+0x60 #1 0xffffffff8092ca55 at panic+0x155 #2 0xffffffff80d7344f at trap_fatal+0x38f #3 0xffffffff80d730ac at trap+0x75c #4 0xffffffff80d58cb2 at calltrap+0x8 #5 0xffffffff808f849c at exit1+0x10dc #6 0xffffffff808f73be at sys_sys_exit+0xe #7 0xffffffff80d73d81 at amd64_syscall+0x351 #8 0xffffffff80d58f9b at Xfast_syscall+0xfb Uptime: 5d20h52m14s Dumping 4361 out of 32603 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/if_re.ko.symbols...done. Loaded symbols for /boot/kernel/if_re.ko.symbols Reading symbols from /boot/kernel/tmpfs.ko.symbols...done. Loaded symbols for /boot/kernel/tmpfs.ko.symbols Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. Loaded symbols for /boot/kernel/fdescfs.ko.symbols #0 doadump (textdump=<value optimized out>) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219 #1 0xffffffff8092c6d2 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff8092ca94 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80d7344f in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865 #4 0xffffffff80d730ac in trap (frame=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:203 #5 0xffffffff80d58cb2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #6 0xffffffff80931b40 in tdsendsignal (p=0xfffff80015bf44c0, td=<value optimized out>, sig=20, ksi=0xfffff8000dfd54d0) at /usr/src/sys/kern/kern_sig.c:1971 #7 0xffffffff808f849c in exit1 (td=0xfffff80248d01920, rv=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:566 #8 0xffffffff808f73be in sys_sys_exit (td=0xfffff80015bf4848, uap=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:153 #9 0xffffffff80d73d81 in amd64_syscall (td=0xfffff80248d01920, traced=0) at subr_syscall.c:134 #10 0xffffffff80d58f9b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 #11 0x0000000801c511ea in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb)