Bug 240853 - Any moderate I/O panics the kernel when MAC_BIBA + multilabel is enabled
Summary: Any moderate I/O panics the kernel when MAC_BIBA + multilabel is enabled
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords: crash, needs-qa
Depends on:
Blocks:
 
Reported: 2019-09-27 06:37 UTC by kayront
Modified: 2019-09-28 09:42 UTC (History)
3 users (show)

See Also:


Attachments
dmesg output as requested (5.82 KB, text/plain)
2019-09-28 09:42 UTC, kayront
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description kayront 2019-09-27 06:37:56 UTC
/dev/vtbd0p2 on / (ufs, local, noatime, journaled soft-updates, multilabel)


Any moderate I/O, such as running a find /, installing a package larger than some arbitrary (and changing) size, or even reloading a tmux session with a couple of windows and panes is enough to trigger this.

cpuid = 0
time = 1569485155
KDB: stack backtrace:
#0 0xffffffff80be78d7 at kdb_backtrace+0x67
#1 0xffffffff80b9b4b3 at vpanic+0x1a3
#2 0xffffffff80b9b303 at panic+0x43
#3 0xffffffff80e979f6 at softdep_deallocate_dependencies+0x76
#4 0xffffffff80c4a27f at brelse+0x16f
#5 0xffffffff80c4e7fd at buf_recycle+0x2ad
#6 0xffffffff80c48a3d at bufkva_reclaim+0x3d
#7 0xffffffff80c01899 at vmem_xalloc+0x469
#8 0xffffffff80c013f1 at vmem_alloc+0x31
#9 0xffffffff80c4f805 at bufkva_alloc+0xa5
#10 0xffffffff80c4c429 at getnewbuf+0x319
#11 0xffffffff80c49944 at getblkx+0x484
#12 0xffffffff80c4c0f5 at getblk+0x15
#13 0xffffffff80e8b71c at ffs_balloc_ufs2+0xd1c
#14 0xffffffff80ebb496 at ffs_close_ea+0x1b6
#15 0xffffffff811fc93e at VOP_SETEXTATTR_APV+0x7e
#16 0xffffffff80c7d05e at vn_extattr_set+0x13e
#17 0xffffffff826abbf6 at biba_vnode_create_extattr+0x126
Uptime: 40s


Since 12.1-RELEASE is coming, I figured it would be a good time to report this. The presence of this bug makes MAC_BIBA unusable, which is a shame.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-09-27 07:07:52 UTC
Thank you for your report

Could you please include more information, including:

- Exact FreeBSD version (uname -a)
- /var/run/dmesg.boot output (as an attachment)
- Check and confirm that the file system is clean (fsck -fy in single user), or report back if it is not
- See if you can obtain a kernel panic backtrace [1], and in particular try to catch the initial "panic message"
- Clarify whether or not the panic is reproducible *without* MAC_BIBA and/or multilabel enabled (test each combination)

[1] https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html#kerneldebug-obtain
Comment 2 Peter Holm freebsd_committer 2019-09-27 10:11:37 UTC
Just loading mac_mls and rebooting causes:
root@x4:~ # kldstat
Id Refs Address        Size Name
 1    4   0x800000  1a39ed4 kernel
 2    1  0x223a000     d364 mac_mls.ko
root@x4:~ # init 6
panic: mac_label_get: NULL label
cpuid = 1
time = 1569578887
KDB: stack backtrace:
db_trace_self_wrapper(e6d46d,1bf0e68,0,221cf7d4,bd8ea1,...) at db_trace_self_wrapper+0x2a/frame 0x221cf7a8
kdb_backtrace(d,1,1,0,0,...) at kdb_backtrace+0x2e/frame 0x221cf808
vpanic(16350f7,221cf850,221cf850,221cf850,126c8ad,...) at vpanic+0x121/frame 0x221cf830
panic(16350f7,221cf864,223e183,0,0,...) at panic+0x14/frame 0x221cf844
mac_label_get(0,0,21994000,221cf89c,1271f07,...) at mac_label_get+0x1d/frame 0x221cf850
mls_destroy_label(0) at mls_destroy_label+0x13/frame 0x221cf864
mac_socket_destroy(21994000) at mac_socket_destroy+0xd7/frame 0x221cf89c
sodealloc(21994000) at sodealloc+0x96/frame 0x221cf8cc
soclose(21994000) at soclose+0x3d2/frame 0x221cf904
soo_close(189f60a8,22619000) at soo_close+0x1f/frame 0x221cf910
_fdrop(189f60a8,22619000) at _fdrop+0x18/frame 0x221cf928
closef(189f60a8,22619000) at closef+0x1f7/frame 0x221cf97c
fdescfree_fds(1) at fdescfree_fds+0xc8/frame 0x221cf9a4
fdescfree(22619000) at fdescfree+0x429/frame 0x221cfa0c
exit1(22619000,0,f,17fffc80,21d30b20,...) at exit1+0x473/frame 0x221cfa3c
sigexit(22619000,f,225d9ab8,0,225d9ab8,...) at sigexit+0xc32/frame 0x221cfc08
postsig(f) at postsig+0x2ac/frame 0x221cfca4
ast(221cfce8,3b,3b,3b,ffbfe638,...) at ast+0x526/frame 0x221cfcdc
zone_mls(0,d4460163,0,d4461163,0,...) at 0xffc055ad/frame 0xffbfe00c
(null)() at 0
KDB: enter: panic
[ thread pid 745 tid 100189 ]
Stopped at      kdb_enter+0x35: movl    $0,kdb_why
db> x/s version
version:        FreeBSD 13.0-CURRENT #0 r352789: Fri Sep 27 09:35:31 CEST 2019\012    pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO\012
db>
Comment 3 kayront 2019-09-28 09:40:11 UTC
- Exact FreeBSD version (uname -a)

FreeBSD fbsdtest 12.0-RELEASE-p10 FreeBSD 12.0-RELEASE-p10 GENERIC  amd64

- /var/run/dmesg.boot output (as an attachment)

Will attach.

- Check and confirm that the file system is clean (fsck -fy in single user), or report back if it is not

/dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS

- See if you can obtain a kernel panic backtrace [1], and in particular try to catch the initial "panic message"

It is: softdep_deallocate_dependencies: dangling deps

Funnily (or tragically, I guess) enough with dumps enabled it then proceeds to panic while writing the crash summary:

Writing crash summary to /var/crash/core.txt.0.
panic: softdep_deallocate_dependencies: dangling deps
cpuid = 0
time = 1569663157
KDB: stack backtrace:
#0 0xffffffff80be78d7 at kdb_backtrace+0x67
#1 0xffffffff80b9b4b3 at vpanic+0x1a3
#2 0xffffffff80b9b303 at panic+0x43
#3 0xffffffff80e979f6 at softdep_deallocate_dependencies+0x76
#4 0xffffffff80c4a27f at brelse+0x16f
#5 0xffffffff80c4e7fd at buf_recycle+0x2ad
#6 0xffffffff80c4ee57 at bufspace_daemon+0xe7
#7 0xffffffff80b5be83 at fork_exit+0x83
#8 0xffffffff8105061e at fork_trampoline+0xe
Uptime: 9s
Dumping 102 out of 472 MB:..16%..32%..47%..63%..78%..94%


- Clarify whether or not the panic is reproducible *without* MAC_BIBA and/or multilabel enabled (test each combination)

Just tested, and:

root@fbsdtest:~ # kldstat | grep biba
root@fbsdtest:~ # mount|grep label
/dev/vtbd0p2 on / (ufs, local, noatime, journaled soft-updates, multilabel, acls)
devfs on /dev (devfs, local, multilabel)

Ran a find /, installed some packages, and no problem; this seems to make sense if you look at the original message, which included:

#17 0xffffffff826abbf6 at biba_vnode_create_extattr+0x126

(this is not present if the module is not loaded)

Without multilabel is how I had the system before and it was fine, so I did not test again, but let me know if that would add any new useful information.
Comment 4 kayront 2019-09-28 09:42:29 UTC
Created attachment 207907 [details]
dmesg output as requested

Dmesg output
Comment 5 kayront 2019-09-28 09:42:49 UTC
Attachment added - https://bugs.freebsd.org/bugzilla/attachment.cgi?id=207907