Bug 244975 - Panic: deadlres_td_sleep_q: possible deadlock detected
Summary: Panic: deadlres_td_sleep_q: possible deadlock detected
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-22 12:24 UTC by Gordon Bergling
Modified: 2020-11-26 16:21 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gordon Bergling freebsd_committer freebsd_triage 2020-03-22 12:24:51 UTC
I recently got this kernel panic while reporting the system. A kernel dump is available on request.

FreeBSD fbsd-dev.0xfce3.net 13.0-CURRENT FreeBSD 13.0-CURRENT #0 979585e9a6e-c267389(master): Fri Mar 13 21:31:16 CET 2020     root@lion.0xfce3.net:/boiler/nfs/obj/boiler/nfs/src/amd64.amd64/sys/GENERIC  amd64

panic: deadlres_td_sleep_q: possible deadlock detected for 0xfffffe0001efda00, blocked for 180105 ticks

Unread portion of the kernel message buffer:
<3>pid 10000 (find), jid 0, uid 0, was killed: out of swap space
<3>pid 14819 (ntpd), jid 0, uid 123, was killed: out of swap space
<3>pid 47356 (hv_kvp_daemon), jid 0, uid 0, was killed: out of swap space
<3>pid 61093 (devd), jid 0, uid 0, was killed: out of swap space
<3>pid 5427 (rpc.statd), jid 0, uid 0, was killed: out of swap space
<3>pid 99994 (auditd), jid 0, uid 0, was killed: out of swap space
<3>pid 6158 (rpc.lockd), jid 0, uid 0, was killed: out of swap space
<3>pid 78549 (rpcbind), jid 0, uid 0, was killed: out of swap space
<3>pid 498 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 727 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 10600 (wc), jid 0, uid 0, was killed: out of swap space
<3>pid 77620 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 8561 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 9550 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 1444 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 3054 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 78120 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 79779 (sh), jid 0, uid 0, was killed: out of swap space
<3>pid 48007 (hv_vss_daemon), jid 0, uid 0, was killed: out of swap space
<3>pid 3487 (mail), jid 0, uid 0, was killed: out of swap space
<3>pid 80132 (mail), jid 0, uid 0, was killed: out of swap space
<3>pid 36313 (getty), jid 0, uid 0, was killed: out of swap space
<3>pid 36249 (getty), jid 0, uid 0, was killed: out of swap space
<3>pid 10298 (tee), jid 0, uid 0, was killed: out of swap space
<3>pid 77183 (cron), jid 0, uid 0, was killed: out of swap space
<3>pid 1246 (lockf), jid 0, uid 0, was killed: out of swap space
<3>pid 77777 (lockf), jid 0, uid 0, was killed: out of swap space
panic: deadlres_td_sleep_q: possible deadlock detected for 0xfffffe0001efda00, blocked for 180105 ticks

cpuid = 0
time = 1584153669
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0001c969c0
vpanic() at vpanic+0x182/frame 0xfffffe0001c96a10
panic() at panic+0x43/frame 0xfffffe0001c96a70
deadlkres() at deadlkres+0x360/frame 0xfffffe0001c96ab0
fork_exit() at fork_exit+0x80/frame 0xfffffe0001c96af0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0001c96af0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic

No symbol "zombproc" in current context.
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/mac_ntpd.ko...Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mac_ntpd.ko
#0  doadump (textdump=0) at src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  doadump (textdump=0) at src/sys/amd64/include/pcpu_aux.h:55
#1  0xffffffff8049a3ea in db_dump (dummy=<value optimized out>,
    dummy2=<value optimized out>, dummy3=<value optimized out>,
    dummy4=<value optimized out>) at /boiler/nfs/src/sys/ddb/db_command.c:575
#2  0xffffffff8049a1ac in db_command (cmd_table=<value optimized out>,
    dopager=1) at /boiler/nfs/src/sys/ddb/db_command.c:482
#3  0xffffffff80499f1d in db_command_loop ()
    at /boiler/nfs/src/sys/ddb/db_command.c:535
#4  0xffffffff8049d118 in db_trap (type=<value optimized out>,
    code=<value optimized out>) at /boiler/nfs/src/sys/ddb/db_main.c:253
#5  0xffffffff80c07c74 in kdb_trap (type=3, code=0, tf=<value optimized out>)
    at /boiler/nfs/src/sys/kern/subr_kdb.c:699
#6  0xffffffff8105cc08 in trap (frame=0xfffffe0001c968f0)
    at /boiler/nfs/src/sys/amd64/amd64/trap.c:579
#7  0xffffffff81032ac8 in calltrap ()
    at /boiler/nfs/src/sys/amd64/amd64/exception.S:289
#8  0xffffffff80c07407 in kdb_enter (why=0xffffffff811ec1dd "panic",
    msg=<value optimized out>) at src/sys/amd64/include/cpufunc.h:65
#9  0xffffffff80bbccee in vpanic (fmt=<value optimized out>,
    ap=<value optimized out>) at /boiler/nfs/src/sys/kern/kern_shutdown.c:902
#10 0xffffffff80bbca83 in panic (fmt=<value optimized out>)
    at /boiler/nfs/src/sys/kern/kern_shutdown.c:839
#11 0xffffffff80b53ef0 in deadlkres ()
    at /boiler/nfs/src/sys/kern/kern_clock.c:242
#12 0xffffffff80b7a760 in fork_exit (callout=0xffffffff80b53b90 <deadlkres>,
    arg=0x0, frame=0xfffffe0001c96b00)
    at /boiler/nfs/src/sys/kern/kern_fork.c:1061
#13 0xffffffff81033b0e in fork_trampoline ()
    at /boiler/nfs/src/sys/amd64/amd64/exception.S:1080
#14 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)
Comment 1 Gordon Bergling freebsd_committer freebsd_triage 2020-04-09 20:45:36 UTC
I had a similar kernel panic today with the following backtrace:

__curthread () at /boiler/nfs/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /boiler/nfs/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=0) at /boiler/nfs/src/sys/kern/kern_shutdown.c:394
#2  0xffffffff8049b43a in db_dump (dummy=<optimized out>,
    dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>)
    at /boiler/nfs/src/sys/ddb/db_command.c:575
#3  0xffffffff8049b1fc in db_command (last_cmdp=<optimized out>,
    cmd_table=<optimized out>, dopager=1)
    at /boiler/nfs/src/sys/ddb/db_command.c:482
#4  0xffffffff8049af6d in db_command_loop ()
    at /boiler/nfs/src/sys/ddb/db_command.c:535
#5  0xffffffff8049e168 in db_trap (type=<optimized out>, code=<optimized out>)
    at /boiler/nfs/src/sys/ddb/db_main.c:253
#6  0xffffffff80c07e24 in kdb_trap (type=3, code=0, tf=<optimized out>)
    at /boiler/nfs/src/sys/kern/subr_kdb.c:699
#7  0xffffffff8105cc68 in trap (frame=0xfffffe0001c968f0)
    at /boiler/nfs/src/sys/amd64/amd64/trap.c:578
#8  <signal handler called>
#9  kdb_enter (why=0xffffffff811ec7d7 "panic", msg=<optimized out>)
    at /boiler/nfs/src/sys/kern/subr_kdb.c:486
#10 0xffffffff80bbd8fe in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /boiler/nfs/src/sys/kern/kern_shutdown.c:902
#11 0xffffffff80bbd693 in panic (
    fmt=0xffffffff81c8db28 <cnputs_mtx> "\363\a\033\201\377\377\377\377")
    at /boiler/nfs/src/sys/kern/kern_shutdown.c:839
#12 0xffffffff80b54cfd in deadlres_td_sleep_q (p=0xfffff800033a7528,
    td=0xfffffe0001effa00, slpticks=180000)
    at /boiler/nfs/src/sys/kern/kern_clock.c:243
#13 deadlkres () at /boiler/nfs/src/sys/kern/kern_clock.c:288
#14 0xffffffff80b7b490 in fork_exit (callout=0xffffffff80b54970 <deadlkres>,
    arg=0x0, frame=0xfffffe0001c96b00)
    at /boiler/nfs/src/sys/kern/kern_fork.c:1051
#15 <signal handler called>

A core dump is also available.
Comment 2 Mark Johnston freebsd_committer freebsd_triage 2020-11-24 18:21:28 UTC
Is this still occurring?  If so, we'll want to see the backtrace of the thread referenced in the panic message.  For example, 0xfffffe0001efda00, in which case try:

(kgdb) p ((struct thread *)0xfffffe0001efda00)->td_tid
$1 = 123456
(kgdb) tid 123456
(kgdb) bt
Comment 3 Gordon Bergling freebsd_committer freebsd_triage 2020-11-26 16:21:55 UTC
I haven't seen this panic recently so I would think that this bug can be closed.