Bug 245217 - panic: spin lock held too long
Summary: panic: spin lock held too long
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-31 17:58 UTC by emz
Modified: 2021-02-06 18:01 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description emz 2020-03-31 17:58:14 UTC
Env: 
====
FreeBSD gw1.qwerty.perm.ru 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  amd64
Running as VM under Hyper-V/Windows.

Host system isn't starving neither on disk (SSDs) nor memory ot CPU.

Will be glad ttto provide any additional info in case itt's needed.

So far I can provide:
http://mail.qwerty.perm.ru/files/core.txt.0
http://mail.qwerty.perm.ru/files/info.0
http://mail.qwerty.perm.ru/files/vmcore.0 (8 Gigs)

Backtrace:
==========
[root@gw1:/var/crash]# /usr/libexec/kgdb /boot/kernel/kernel vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
spin lock 0xfffff801727ca6c0 (turnstile lock) held by 0xfffff8001ecc7000 (tid 100628) too long
panic: spin lock held too long
cpuid = 11
time = 1585676097
KDB: stack backtrace:
#0 0xffffffff80be7977 at kdb_backtrace+0x67
#1 0xffffffff80b9b563 at vpanic+0x1a3
#2 0xffffffff80b9b3b3 at panic+0x43
#3 0xffffffff80b7b6ec at _mtx_lock_indefinite_check+0x6c
#4 0xffffffff80b7b2b4 at _mtx_lock_spin_cookie+0xd4
#5 0xffffffff80bfd5b5 at turnstile_trywait+0xe5
#6 0xffffffff80b9674f at __rw_wlock_hard+0x2ef
#7 0xffffffff80dcc0cc at tcp_usr_send+0x81c
#8 0xffffffff80c325e6 at sosend_generic+0x586
#9 0xffffffff80c328f0 at sosend+0x50
#10 0xffffffff80c0d6e9 at soo_write+0x49
#11 0xffffffff80c04862 at dofilewrite+0xb2
#12 0xffffffff80c04760 at sys_writev+0x70
#13 0xffffffff81075449 at amd64_syscall+0x369
#14 0xffffffff8104fd1d at fast_syscall_common+0x101
Uptime: 4d10h59m7s
Dumping 8227 out of 24533 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ether.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/carp.ko...Reading symbols from /usr/lib/debug//boot/kernel/carp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/carp.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/intpm.ko...Reading symbols from /usr/lib/debug//boot/kernel/intpm.ko.debug...done.
done.
Loaded symbols for /boot/kernel/intpm.ko
Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug...done.
done.
Loaded symbols for /boot/kernel/smbus.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/pfsync.ko...Reading symbols from /usr/lib/debug//boot/kernel/pfsync.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pfsync.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_mppc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_mppc.ko
Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /usr/lib/debug//boot/kernel/rc4.ko.debug...done.
done.
Loaded symbols for /boot/kernel/rc4.ko
Reading symbols from /boot/kernel/ng_tee.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_tee.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_tee.ko
Reading symbols from /boot/kernel/ng_pptpgre.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_pptpgre.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_pptpgre.ko
Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ksocket.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_ksocket.ko
Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_iface.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_iface.ko
Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ppp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_ppp.ko
Reading symbols from /boot/kernel/ng_tcpmss.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_tcpmss.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_tcpmss.ko
Reading symbols from /boot/kernel/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_netflow.ko
Reading symbols from /boot/kernel/smbfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/smbfs.ko
Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from /usr/lib/debug//boot/kernel/libiconv.ko.debug...done.
done.
Loaded symbols for /boot/kernel/libiconv.ko
Reading symbols from /boot/kernel/libmchain.ko...Reading symbols from /usr/lib/debug//boot/kernel/libmchain.ko.debug...done.
done.
Loaded symbols for /boot/kernel/libmchain.ko
Reading symbols from /boot/kernel/ng_l2tp.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_l2tp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_l2tp.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /usr/lib/debug//boot/kernel/pflog.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:230
230     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:230
#1  0xffffffff80b9b14b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:446
#2  0xffffffff80b9b5c3 in vpanic (fmt=<value optimized out>, ap=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:872
#3  0xffffffff80b9b3b3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:799
#4  0xffffffff80b7b6ec in _mtx_lock_indefinite_check (m=<value optimized out>, ldap=<value optimized out>)
    at /usr/src/sys/kern/kern_mutex.c:1222
#5  0xffffffff80b7b2b4 in _mtx_lock_spin_cookie (c=0xfffff801727ca6d8, v=18446735278133243904)
    at /usr/src/sys/kern/kern_mutex.c:748
#6  0xffffffff80bfd5b5 in turnstile_trywait (lock=0xfffff800d46ef3f0) at /usr/src/sys/kern/subr_turnstile.c:562
#7  0xffffffff80b9674f in __rw_wlock_hard (c=0xfffff800d46ef408, v=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:1060
#8  0xffffffff80dcc0cc in tcp_usr_send (so=0xfffff80387bd1000, flags=0, m=<value optimized out>,
    nam=<value optimized out>, control=<value optimized out>, td=0xbe) at /usr/src/sys/netinet/tcp_usrreq.c:923
#9  0xffffffff80c325e6 in sosend_generic (so=0xfffff80387bd1000, addr=0x0, uio=0xfffff804a19dd700,
    top=0xfffff8012a2aee00, control=<value optimized out>, flags=0, td=0xfffff8005dbbc580)
    at /usr/src/sys/kern/uipc_socket.c:1579
#10 0xffffffff80c328f0 in sosend (so=<value optimized out>, addr=<value optimized out>, uio=<value optimized out>,
    top=<value optimized out>, control=0x0, flags=<value optimized out>, td=0xfffff8005dbbc580)
    at /usr/src/sys/kern/uipc_socket.c:1625
#11 0xffffffff80c0d6e9 in soo_write (fp=<value optimized out>, uio=0xfffff804a19dd700,
    active_cred=<value optimized out>, flags=<value optimized out>, td=<value optimized out>)
    at /usr/src/sys/kern/sys_socket.c:148
#12 0xffffffff80c04862 in dofilewrite (td=0x0, fd=658, fp=0xfffff80368e119b0, auio=0xfffff804a19dd700,
    offset=<value optimized out>, flags=<value optimized out>) at file.h:314
#13 0xffffffff80c04760 in sys_writev (td=0xfffff8005dbbc580, uap=0xfffff8005dbbc940)
    at /usr/src/sys/kern/sys_generic.c:491
#14 0xffffffff81075449 in amd64_syscall (td=0xfffff8005dbbc580, traced=0) at subr_syscall.c:135
#15 0xffffffff8104fd1d in fast_syscall_common () at /usr/src/sys/amd64/amd64/exception.S:504
#16 0x0000000800b51e3a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)
Comment 1 Mark Johnston freebsd_committer freebsd_triage 2020-03-31 18:29:45 UTC
Can you show output from

(kgdb) tid 100628
(kgdb) bt

?

This could be caused by FreeBSD-EN-19:14.epoch, fixed in 12.0-p9.