Bug 191882 - Multiple panics when ctld is running
Summary: Multiple panics when ctld is running
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Edward Tomasz Napierala
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-07-15 16:38 UTC by Steven Peterson
Modified: 2014-10-16 09:13 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Steven Peterson 2014-07-15 16:38:18 UTC
We are experiencing panics on multiple systems, running the in kernel iscsi target

Systems all go through a panic loop started with a page fault, then 2 - 3 General Protection Fault Panics before the system returns to normal

Issue seams to be caused by heavy IO on the system not exclusively iscsi.
But ctld must be running for the panic to occur.

Up times very from a few minutes to a few days
I have a set of dumps for this most recent iteration on my home test box (it is experiencing the same issue as my other boxes)

We have tried reinstalling the system and a build world to no avail.

Here is the text of the inital panic, I can up load the tar ball of the rest of the data if needed (total size as a tbz2 is 977mb)

plfs.pl.mainstream.net dumped core - see /var/crash/vmcore.1

Tue Jul 15 08:38:40 EDT 2014

FreeBSD plfs.pl.mainstream.net 10.0-RELEASE-p6 FreeBSD 10.0-RELEASE-p6 #0: Tue Jun 24 07:47:37 UTC 2014     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x0
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff808c474e
stack pointer           = 0x28:0xfffffe0838f936e0
frame pointer           = 0x28:0xfffffe0838f93770
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 11 (idle: cpu0)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff808e7e90 at kdb_backtrace+0x60
#1 0xffffffff808af975 at panic+0x155
#2 0xffffffff80c8e762 at trap_fatal+0x3a2
#3 0xffffffff80c8ea39 at trap_pfault+0x2c9
#4 0xffffffff80c8e1c6 at trap+0x5e6
#5 0xffffffff80c75462 at calltrap+0x8
#6 0xffffffff80d65770 at handleevents+0x180
#7 0xffffffff80d66098 at timercb+0x308
#8 0xffffffff80d9073c at lapic_handle_timer+0x9c
#9 0xffffffff80c75fbc at Xtimerint+0x8c
#10 0xffffffff80359e77 at acpi_cpu_idle+0x137
#11 0xffffffff80c7972f at cpu_idle_acpi+0x3f
#12 0xffffffff80c797d3 at cpu_idle+0x93
#13 0xffffffff808d8b2e at sched_idletd+0x1ee
#14 0xffffffff80881a4a at fork_exit+0x9a
#15 0xffffffff80c7599e at fork_trampoline+0xe
Uptime: 3m14s
Dumping 1504 out of 32645 MB:..2%..11%..21%..31%..41%..52%..61%..71%..81%..91%

Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
Loaded symbols for /boot/kernel/nullfs.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/geom_linux_lvm.ko.symbols...done.
Loaded symbols for /boot/kernel/geom_linux_lvm.ko.symbols
Reading symbols from /boot/kernel/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel/uhid.ko.symbols
Reading symbols from /boot/kernel/ctl.ko.symbols...done.
Loaded symbols for /boot/kernel/ctl.ko.symbols
Reading symbols from /boot/kernel/iscsi.ko.symbols...done.
Loaded symbols for /boot/kernel/iscsi.ko.symbols
Reading symbols from /boot/kernel/kgssapi_krb5.ko.symbols...done.
Loaded symbols for /boot/kernel/kgssapi_krb5.ko.symbols
Reading symbols from /boot/kernel/kgssapi.ko.symbols...done.
Loaded symbols for /boot/kernel/kgssapi.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/rc4.ko.symbols...done.
Loaded symbols for /boot/kernel/rc4.ko.symbols
Reading symbols from /boot/kernel/nfsserver.ko.symbols...done.
Loaded symbols for /boot/kernel/nfsserver.ko.symbols
Reading symbols from /boot/kernel/nfs_common.ko.symbols...done.
Loaded symbols for /boot/kernel/nfs_common.ko.symbols
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff808af5f0 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff808af9b4 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80c8e762 in trap_fatal (frame=<value optimized out>, 
    eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:882
#4  0xffffffff80c8ea39 in trap_pfault (frame=0xfffffe0838f93630, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:699
#5  0xffffffff80c8e1c6 in trap (frame=0xfffffe0838f93630)
    at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80c75462 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff808c474e in callout_process (now=835132764813)
    at /usr/src/sys/kern/kern_timeout.c:458
#8  0xffffffff80d65770 in handleevents (now=835132764813, fake=0)
    at /usr/src/sys/kern/kern_clocksource.c:222
#9  0xffffffff80d66098 in timercb (et=<value optimized out>, 
    arg=<value optimized out>) at /usr/src/sys/kern/kern_clocksource.c:365
#10 0xffffffff80d9073c in lapic_handle_timer (frame=0xfffffe0838f93870)
    at /usr/src/sys/x86/x86/local_apic.c:816
#11 0xffffffff80c75fbc in Xtimerint () at apic_vector.S:109
#12 0xffffffff80c72ae6 in acpi_cpu_c1 ()
    at /usr/src/sys/amd64/acpica/acpi_machdep.c:95
#13 0xffffffff80359e77 in acpi_cpu_idle (sbt=<value optimized out>)
    at /usr/src/sys/dev/acpica/acpi_cpu.c:1026
#14 0xffffffff80c7972f in cpu_idle_acpi (sbt=79351405)
    at /usr/src/sys/amd64/amd64/machdep.c:680
#15 0xffffffff80c797d3 in cpu_idle (busy=0)
    at /usr/src/sys/amd64/amd64/machdep.c:835
#16 0xffffffff808d8b2e in sched_idletd (dummy=<value optimized out>)
    at /usr/src/sys/kern/sched_ule.c:2673
#17 0xffffffff80881a4a in fork_exit (
    callout=0xffffffff808d8940 <sched_idletd>, arg=0x0, 
    frame=0xfffffe0838f93ac0) at /usr/src/sys/kern/kern_fork.c:995
#18 0xffffffff80c7599e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:606
#19 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)
Comment 1 Steven Peterson 2014-07-15 18:10:40 UTC
Sample of General Protection Fault Panic:

plfs.pl.mainstream.net dumped core - see /var/crash/vmcore.4

Tue Jul 15 14:07:00 EDT 2014

FreeBSD plfs.pl.mainstream.net 10.0-RELEASE-p6 FreeBSD 10.0-RELEASE-p6 #0 r268165: Tue Jul 15 13:15:23 EDT 2014     root@plfs.pl.mainstream.net:/usr/obj/usr/src/sys/PLFS  amd64

panic: general protection fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 9 with interrupts disabled


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x20:0xffffffff808c4593
stack pointer           = 0x28:0xfffffe0838f936e0
frame pointer           = 0x28:0xfffffe0838f93770
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 11 (idle: cpu0)
trap number             = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff808e7ce0 at kdb_backtrace+0x60
#1 0xffffffff808af7c5 at panic+0x155
#2 0xffffffff80c8de32 at trap_fatal+0x3a2
#3 0xffffffff80c8da6f at trap+0x7bf
#4 0xffffffff80c74b32 at calltrap+0x8
#5 0xffffffff80d64e40 at handleevents+0x180
#6 0xffffffff80d65768 at timercb+0x308
#7 0xffffffff80d8fe0c at lapic_handle_timer+0x9c
#8 0xffffffff80c7568c at Xtimerint+0x8c
#9 0xffffffff80359cc7 at acpi_cpu_idle+0x137
#10 0xffffffff80c78dff at cpu_idle_acpi+0x3f
#11 0xffffffff80c78ea3 at cpu_idle+0x93
#12 0xffffffff808d897e at sched_idletd+0x1ee
#13 0xffffffff8088189a at fork_exit+0x9a
#14 0xffffffff80c7506e at fork_trampoline+0xe
Uptime: 1m12s
Dumping 1436 out of 32645 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
Loaded symbols for /boot/kernel/nullfs.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/geom_linux_lvm.ko.symbols...done.
Loaded symbols for /boot/kernel/geom_linux_lvm.ko.symbols
Reading symbols from /boot/kernel/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel/uhid.ko.symbols
Reading symbols from /boot/kernel/ctl.ko.symbols...done.
Loaded symbols for /boot/kernel/ctl.ko.symbols
Reading symbols from /boot/kernel/iscsi.ko.symbols...done.
Loaded symbols for /boot/kernel/iscsi.ko.symbols
Reading symbols from /boot/kernel/kgssapi_krb5.ko.symbols...done.
Loaded symbols for /boot/kernel/kgssapi_krb5.ko.symbols
Reading symbols from /boot/kernel/kgssapi.ko.symbols...done.
Loaded symbols for /boot/kernel/kgssapi.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/rc4.ko.symbols...done.
Loaded symbols for /boot/kernel/rc4.ko.symbols
Reading symbols from /boot/kernel/nfsserver.ko.symbols...done.
Loaded symbols for /boot/kernel/nfsserver.ko.symbols
Reading symbols from /boot/kernel/nfs_common.ko.symbols...done.
Loaded symbols for /boot/kernel/nfs_common.ko.symbols
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff808af440 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff808af804 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80c8de32 in trap_fatal (frame=<value optimized out>, 
    eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:882
#4  0xffffffff80c8da6f in trap (frame=<value optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:224
#5  0xffffffff80c74b32 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#6  0xffffffff808c4593 in callout_process (now=309634963277)
    at /usr/src/sys/kern/kern_timeout.c:458
#7  0xffffffff80d64e40 in handleevents (now=309634963277, fake=0)
    at /usr/src/sys/kern/kern_clocksource.c:222
#8  0xffffffff80d65768 in timercb (et=<value optimized out>, 
    arg=<value optimized out>) at /usr/src/sys/kern/kern_clocksource.c:365
#9  0xffffffff80d8fe0c in lapic_handle_timer (frame=0xfffffe0838f93870)
    at /usr/src/sys/x86/x86/local_apic.c:816
#10 0xffffffff80c7568c in Xtimerint () at apic_vector.S:109
#11 0xffffffff80c721b6 in acpi_cpu_c1 ()
    at /usr/src/sys/amd64/acpica/acpi_machdep.c:95
#12 0xffffffff80359cc7 in acpi_cpu_idle (sbt=<value optimized out>)
    at /usr/src/sys/dev/acpica/acpi_cpu.c:1026
#13 0xffffffff80c78dff in cpu_idle_acpi (sbt=106894436)
    at /usr/src/sys/amd64/amd64/machdep.c:680
#14 0xffffffff80c78ea3 in cpu_idle (busy=0)
    at /usr/src/sys/amd64/amd64/machdep.c:835
#15 0xffffffff808d897e in sched_idletd (dummy=<value optimized out>)
    at /usr/src/sys/kern/sched_ule.c:2673
#16 0xffffffff8088189a in fork_exit (
    callout=0xffffffff808d8790 <sched_idletd>, arg=0x0, 
    frame=0xfffffe0838f93ac0) at /usr/src/sys/kern/kern_fork.c:995
#17 0xffffffff80c7506e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:606
#18 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)
Comment 2 Steven Peterson 2014-07-15 18:11:26 UTC
Additional Sample of Page Fault Panic:
plfs.pl.mainstream.net dumped core - see /var/crash/vmcore.2

Tue Jul 15 14:00:31 EDT 2014

FreeBSD plfs.pl.mainstream.net 10.0-RELEASE-p6 FreeBSD 10.0-RELEASE-p6 #0 r268165: Tue Jul 15 13:15:23 EDT 2014     root@plfs.pl.mainstream.net:/usr/obj/usr/src/sys/PLFS  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0x378
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff8089c531
stack pointer           = 0x28:0xfffffe0838fbb7e0
frame pointer           = 0x28:0xfffffe0838fbb860
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi4: clock)
trap number             = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff808e7ce0 at kdb_backtrace+0x60
#1 0xffffffff808af7c5 at panic+0x155
#2 0xffffffff80c8de32 at trap_fatal+0x3a2
#3 0xffffffff80c8e109 at trap_pfault+0x2c9
#4 0xffffffff80c8d896 at trap+0x5e6
#5 0xffffffff80c74b32 at calltrap+0x8
#6 0xffffffff8089c36d at __mtx_lock_flags+0x5d
#7 0xffffffff81c49aab at icl_pdu_queue+0x2b
#8 0xffffffff81c2b81a at cfiscsi_pdu_queue+0x9a
#9 0xffffffff808c483c at softclock_call_cc+0x16c
#10 0xffffffff808c4c84 at softclock+0x94
#11 0xffffffff80883beb at intr_event_execute_handlers+0xab
#12 0xffffffff80884036 at ithread_loop+0x96
#13 0xffffffff8088189a at fork_exit+0x9a
#14 0xffffffff80c7506e at fork_trampoline+0xe
Uptime: 1m14s
Dumping 1417 out of 32645 MB:..2%..11%..21%..31%..41%..51%..61%..72%..81%..91%

Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
Loaded symbols for /boot/kernel/nullfs.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/geom_linux_lvm.ko.symbols...done.
Loaded symbols for /boot/kernel/geom_linux_lvm.ko.symbols
Reading symbols from /boot/kernel/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel/uhid.ko.symbols
Reading symbols from /boot/kernel/ctl.ko.symbols...done.
Loaded symbols for /boot/kernel/ctl.ko.symbols
Reading symbols from /boot/kernel/iscsi.ko.symbols...done.
Loaded symbols for /boot/kernel/iscsi.ko.symbols
Reading symbols from /boot/kernel/kgssapi_krb5.ko.symbols...done.
Loaded symbols for /boot/kernel/kgssapi_krb5.ko.symbols
Reading symbols from /boot/kernel/kgssapi.ko.symbols...done.
Loaded symbols for /boot/kernel/kgssapi.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/rc4.ko.symbols...done.
Loaded symbols for /boot/kernel/rc4.ko.symbols
Reading symbols from /boot/kernel/nfsserver.ko.symbols...done.
Loaded symbols for /boot/kernel/nfsserver.ko.symbols
Reading symbols from /boot/kernel/nfs_common.ko.symbols...done.
Loaded symbols for /boot/kernel/nfs_common.ko.symbols
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff808af440 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff808af804 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80c8de32 in trap_fatal (frame=<value optimized out>, 
    eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:882
#4  0xffffffff80c8e109 in trap_pfault (frame=0xfffffe0838fbb730, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:699
#5  0xffffffff80c8d896 in trap (frame=0xfffffe0838fbb730)
    at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80c74b32 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff8089c531 in __mtx_lock_sleep (c=0xfffff80262b42de0, 
    tid=18446735277754427680, opts=-2117811456, file=0xfffff80008382900 "", 
    line=2) at /usr/src/sys/kern/kern_mutex.c:433
#8  0xffffffff8089c36d in __mtx_lock_flags (c=<value optimized out>, 
    opts=137898272, 
    file=0xffffffff81c4c300 "NING: %s: failed to allocate %zd bytes\n", 
    line=137898240) at /usr/src/sys/kern/kern_mutex.c:224
#9  0xffffffff81c49aab in icl_pdu_queue (ip=0xfffff80262a38420)
    at /usr/src/sys/modules/iscsi/../../dev/iscsi/icl.c:955
#10 0xffffffff81c2b81a in cfiscsi_pdu_queue (response=0xfffff80262a38420)
    at /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:390
#11 0xffffffff808c483c in softclock_call_cc (c=0xfffff800085c4c60, 
    cc=0xffffffff814f3180, direct=0) at /usr/src/sys/kern/kern_timeout.c:682
#12 0xffffffff808c4c84 in softclock (arg=0xffffffff814f3180)
    at /usr/src/sys/kern/kern_timeout.c:810
#13 0xffffffff80883beb in intr_event_execute_handlers (
    p=<value optimized out>, ie=0xfffff80008360800)
    at /usr/src/sys/kern/kern_intr.c:1263
#14 0xffffffff80884036 in ithread_loop (arg=0xfffff80008366de0)
    at /usr/src/sys/kern/kern_intr.c:1276
#15 0xffffffff8088189a in fork_exit (
    callout=0xffffffff80883fa0 <ithread_loop>, arg=0xfffff80008366de0, 
    frame=0xfffffe0838fbbac0) at /usr/src/sys/kern/kern_fork.c:995
#16 0xffffffff80c7506e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:606
#17 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)
Comment 3 Steven Peterson 2014-07-15 18:20:07 UTC
for the page fault ran kgdb got the following (following the instructions in the handbook)
 (kgdb) backtrace
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff808af440 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff808af804 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80c8de32 in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:882
#4  0xffffffff80c8e109 in trap_pfault (frame=0xfffffe0838fbb730, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699
#5  0xffffffff80c8d896 in trap (frame=0xfffffe0838fbb730) at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80c74b32 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff8089c531 in __mtx_lock_sleep (c=0xfffff80262b42de0, tid=18446735277754427680, opts=-2117811456, file=0xfffff80008382900 "", line=2) at /usr/src/sys/kern/kern_mutex.c:433
#8  0xffffffff8089c36d in __mtx_lock_flags (c=<value optimized out>, opts=137898272, file=0xffffffff81c4c300 "NING: %s: failed to allocate %zd bytes\n", line=137898240)
    at /usr/src/sys/kern/kern_mutex.c:224
#9  0xffffffff81c49aab in icl_pdu_queue (ip=0xfffff80262a38420) at /usr/src/sys/modules/iscsi/../../dev/iscsi/icl.c:955
#10 0xffffffff81c2b81a in cfiscsi_pdu_queue (response=0xfffff80262a38420) at /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:390
#11 0xffffffff808c483c in softclock_call_cc (c=0xfffff800085c4c60, cc=0xffffffff814f3180, direct=0) at /usr/src/sys/kern/kern_timeout.c:682
#12 0xffffffff808c4c84 in softclock (arg=0xffffffff814f3180) at /usr/src/sys/kern/kern_timeout.c:810
#13 0xffffffff80883beb in intr_event_execute_handlers (p=<value optimized out>, ie=0xfffff80008360800) at /usr/src/sys/kern/kern_intr.c:1263
#14 0xffffffff80884036 in ithread_loop (arg=0xfffff80008366de0) at /usr/src/sys/kern/kern_intr.c:1276
#15 0xffffffff8088189a in fork_exit (callout=0xffffffff80883fa0 <ithread_loop>, arg=0xfffff80008366de0, frame=0xfffffe0838fbbac0) at /usr/src/sys/kern/kern_fork.c:995
#16 0xffffffff80c7506e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606
#17 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) list *0xffffffff8089c531
0xffffffff8089c531 is in __mtx_lock_sleep (/usr/src/sys/kern/kern_mutex.c:434).
429			 * owner stops running or the state of the lock changes.
430			 */
431			v = m->mtx_lock;
432			if (v != MTX_UNOWNED) {
433				owner = (struct thread *)(v & ~MTX_FLAGMASK);
434				if (TD_IS_RUNNING(owner)) {
435					if (LOCK_LOG_TEST(&m->lock_object, 0))
436						CTR3(KTR_LOCK,
437						    "%s: spinning on %p held by %p",
438						    __func__, m, owner);
Comment 4 Edward Tomasz Napierala freebsd_committer 2014-09-09 14:39:03 UTC
Does the problem still occur under recent 10-STABLE (10.1-PRERELEASE)?
Comment 5 Edward Tomasz Napierala freebsd_committer 2014-10-16 09:13:39 UTC
I believe the bug that caused this was fixed in r269197.