NOTE ==== The described panic occurs post to building the kernel after introducing the changes in review D15381. While the audit daemon is running and the system wide audit mask is set as "pc", i.e 'process-control'. On executing "exa" from shell causes a consistent and reproducible kernel panic. I have set the following alias, which causes the issue. * alias l='exa -iBghlSu --git --all' Here is the panic stack trace: ================================================= panic: acquiring blockable sleep lock with spinlock or critical section held (rw) evclass_lock @ /usr/src/sys/security/audit/audit_bsm_db.c:119 cpuid = 1 time = 1527095030 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00005f96d0 vpanic() at vpanic+0x1a3/frame 0xfffffe00005f9730 doadump() at doadump/frame 0xfffffe00005f97b0 witness_checkorder() at witness_checkorder+0x15c/frame 0xfffffe00005f9840 __rw_rlock_int() at __rw_rlock_int+0x8b/frame 0xfffffe00005f9870 au_event_class() at au_event_class+0x21/frame 0xfffffe00005f9890 audit_commit() at audit_commit+0x13a/frame 0xfffffe00005f98d0 audit_syscall_exit() at audit_syscall_exit+0x28/frame 0xfffffe00005f98f0 thread_exit() at thread_exit+0xdb/frame 0xfffffe00005f9930 kern_thr_exit() at kern_thr_exit+0x11e/frame 0xfffffe00005f9960 sys_thr_exit() at sys_thr_exit+0x67/frame 0xfffffe00005f9980 amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe00005f9ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00005f9ab0 --- syscall (431, FreeBSD ELF64, sys_thr_exit), rip = 0x8017128ca, rsp = 0x7fffdfbfba78, rbp = 0x7fffdfbfba90 --- KDB: enter: panic ================================================== Additional Information ---------------------------------- On trying to get the full debug information on the panic. When I try to do "sysctl debug.kdb.panic=1" I get another panic irrespective of whether auditd is running or what audit mask is set.
What version of FreeBSD were you using? If CURRENT, what svn rev? Please supply the output of "uname -a"
> uname -a > FreeBSD bsdtest 12.0-CURRENT FreeBSD 12.0-CURRENT #1: Mon May 21 20:10:48 IST 2018 root@bsdtest:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
I reproduced this bug in a different way. Simply doing a kernel build while "pc" is in the audit mask and auditd is running will do it. I also found the root cause: 1) kern_thr_exit locks the process's spin lock with PROC_SLOCK(p) 2) thread_exit() calls AUDIT_SYSCALL_EXIT while the spin lock is still held 3) a few stack frames later, au_event_class tries to lock a rwlock with EVCLASS_RLOCK. However, witness forbids grabbing an rwlock while holding a spin lock. The likeliest solution is to fix thread_exit so that AUDIT_SYSCALL_EXIT isn't called with the spin lock held.
A commit references this bug: Author: asomers Date: Wed Jul 11 19:38:42 UTC 2018 New revision: 336205 URL: https://svnweb.freebsd.org/changeset/base/336205 Log: Don't acquire evclass_lock with a spinlock held When the "pc" audit class is enabled and auditd is running, witness will panic during thread exit because au_event_class tries to lock an rwlock while holding a spinlock acquired upstack by thread_exit. To fix this, move AUDIT_SYSCALL_EXIT futher upstack, before the spinlock is acquired. Of thread_exit's 16 callers, it's only necessary to call AUDIT_SYSCALL_EXIT from two, exit1 (for exiting processes) and kern_thr_exit (for exiting threads). The other callers are all kernel threads, which needen't call AUDIT_SYSCALL_EXIT because since they can't make syscalls there will be nothing to audit. And exit1 already does call AUDIT_SYSCALL_EXIT, making the second call in thread_exit redundant for that case. PR: 228444 Reported by: aniketp Reviewed by: aniketp, kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D16210 Changes: head/sys/kern/kern_thr.c head/sys/kern/kern_thread.c
A commit references this bug: Author: asomers Date: Fri Aug 3 14:05:23 UTC 2018 New revision: 337242 URL: https://svnweb.freebsd.org/changeset/base/337242 Log: MFC r336205: Don't acquire evclass_lock with a spinlock held When the "pc" audit class is enabled and auditd is running, witness will panic during thread exit because au_event_class tries to lock an rwlock while holding a spinlock acquired upstack by thread_exit. To fix this, move AUDIT_SYSCALL_EXIT futher upstack, before the spinlock is acquired. Of thread_exit's 16 callers, it's only necessary to call AUDIT_SYSCALL_EXIT from two, exit1 (for exiting processes) and kern_thr_exit (for exiting threads). The other callers are all kernel threads, which needen't call AUDIT_SYSCALL_EXIT because since they can't make syscalls there will be nothing to audit. And exit1 already does call AUDIT_SYSCALL_EXIT, making the second call in thread_exit redundant for that case. PR: 228444 Reported by: aniketp Reviewed by: aniketp, kib Differential Revision: https://reviews.freebsd.org/D16210 Changes: _U stable/11/ stable/11/sys/kern/kern_thr.c stable/11/sys/kern/kern_thread.c
A commit references this bug: Author: asomers Date: Fri Aug 3 14:45:54 UTC 2018 New revision: 337258 URL: https://svnweb.freebsd.org/changeset/base/337258 Log: MFC r336205: Don't acquire evclass_lock with a spinlock held When the "pc" audit class is enabled and auditd is running, witness will panic during thread exit because au_event_class tries to lock an rwlock while holding a spinlock acquired upstack by thread_exit. To fix this, move AUDIT_SYSCALL_EXIT futher upstack, before the spinlock is acquired. Of thread_exit's 16 callers, it's only necessary to call AUDIT_SYSCALL_EXIT from two, exit1 (for exiting processes) and kern_thr_exit (for exiting threads). The other callers are all kernel threads, which needen't call AUDIT_SYSCALL_EXIT because since they can't make syscalls there will be nothing to audit. And exit1 already does call AUDIT_SYSCALL_EXIT, making the second call in thread_exit redundant for that case. PR: 228444 Reported by: aniketp Reviewed by: aniketp, kib Differential Revision: https://reviews.freebsd.org/D16210 Changes: _U stable/10/ stable/10/sys/kern/kern_thr.c stable/10/sys/kern/kern_thread.c