Bug 209816 - [lor] lock order reversal with devfs, tmpfs and ufs
Summary: [lor] lock order reversal with devfs, tmpfs and ufs
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-05-28 20:22 UTC by Marcus von Appen
Modified: 2021-11-02 18:28 UTC (History)
2 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Marcus von Appen freebsd_committer freebsd_triage 2016-05-28 20:22:49 UTC 
uname -a:
FreeBSD ... 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r300409M: Wed May 25 22:39:21 CEST 2016     ...:/usr/obj/usr/src/sys/GENERIC  amd64


Running synth install or synth upgrade-system always produces the following lock order reversal:

lock order reversal:
1st 0xfffff800149ddd50 tmpfs (tmpfs) @ /usr/src/sys/kern/vfs_mount.c:848
2nd 0xfffff800140d2d50 ufs (ufs) @ /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:670
stack backtrace:
#0 0xffffffff80a9b7c0 at witness_debugger+0x70
#1 0xffffffff80a9b6b4 at witness_checkorder+0xe54
#2 0xffffffff80a155c6 at __lockmgr_args+0x4d6
#3 0xffffffff80cf8466 at ffs_lock+0xa6
#4 0xffffffff81002f30 at VOP_LOCK1_APV+0x100
#5 0xffffffff823f7891 at null_lock+0xd1
#6 0xffffffff81002f30 at VOP_LOCK1_APV+0x100
#7 0xffffffff80b0b73a at _vn_lock+0x9a
#8 0xffffffff823f6b51 at nullfs_root+0x41
#9 0xffffffff80af2f90 at vfs_donmount+0x1430
#10 0xffffffff80af1b32 at sys_nmount+0x72
#11 0xffffffff80ea6a2b at amd64_syscall+0x2db
#12 0xffffffff80e86d9b at Xfast_syscall+0xfb
lock order reversal:
1st 0xfffff80014c699a0 tmpfs (tmpfs) @ /usr/src/sys/kern/vfs_mount.c:848
2nd 0xfffff80014cac7c8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2498
stack backtrace:
#0 0xffffffff80a9b7c0 at witness_debugger+0x70
#1 0xffffffff80a9b6b4 at witness_checkorder+0xe54
#2 0xffffffff80a155c6 at __lockmgr_args+0x4d6
#3 0xffffffff80aeab0c at vop_stdlock+0x3c
#4 0xffffffff81002f30 at VOP_LOCK1_APV+0x100
#5 0xffffffff80b0b73a at _vn_lock+0x9a
#6 0xffffffff80afbbb3 at vget+0x63
#7 0xffffffff809114fd at devfs_allocv+0xcd
#8 0xffffffff80911013 at devfs_root+0x43
#9 0xffffffff80af2f90 at vfs_donmount+0x1430
#10 0xffffffff80af1b32 at sys_nmount+0x72
#11 0xffffffff80ea6a2b at amd64_syscall+0x2db
#12 0xffffffff80e86d9b at Xfast_syscall+0xfb
Comment 1 ml 2017-06-16 17:02:14 UTC 
Hello.

I've just seen the same on a 10.3p18/amd64:

lock order reversal:
1st 0xfffff803b3908a28 tmpfs (tmpfs) @ /usr/src/sys/kern/vfs_mount.c:848
2nd 0xfffff803b3908c98 ufs (ufs) @ /usr/src/sys/fs/nullfs/null_vnops.c:670
KDB: stack backtrace:
#0 0xffffffff804c72b0 at kdb_backtrace+0x60
#1 0xffffffff804e7204 at witness_checkorder+0xe24
#2 0xffffffff80468a79 at __lockmgr_args+0x9d9
#3 0xffffffff806b2162 at ffs_lock+0x92
#4 0xffffffff8077ad5c at VOP_LOCK1_APV+0xfc
#5 0xffffffff803c2dd0 at null_lock+0xe0
#6 0xffffffff8077ad5c at VOP_LOCK1_APV+0xfc
#7 0xffffffff80547492 at _vn_lock+0xd2
#8 0xffffffff803c2010 at nullfs_root+0x50
#9 0xffffffff805306d8 at vfs_donmount+0x13b8
#10 0xffffffff8052f2ff at sys_nmount+0x4f
#11 0xffffffff80712f3c at amd64_syscall+0x2ac
#12 0xffffffff806f6b0b at Xfast_syscall+0xfb
lock order reversal:
1st 0xfffff803a337c2d8 tmpfs (tmpfs) @ /usr/src/sys/kern/vfs_mount.c:848
2nd 0xfffff8024f106a28 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2253
KDB: stack backtrace:
#0 0xffffffff804c72b0 at kdb_backtrace+0x60
#1 0xffffffff804e7204 at witness_checkorder+0xe24
#2 0xffffffff80468a79 at __lockmgr_args+0x9d9
#3 0xffffffff80528a5c at vop_stdlock+0x3c
#4 0xffffffff8077ad5c at VOP_LOCK1_APV+0xfc
#5 0xffffffff80547492 at _vn_lock+0xd2
#6 0xffffffff805387e7 at vget+0x67
#7 0xffffffff8038e8cd at devfs_allocv+0xfd
#8 0xffffffff8038e353 at devfs_root+0x43
#9 0xffffffff805306d8 at vfs_donmount+0x13b8
#10 0xffffffff8052f2ff at sys_nmount+0x4f
#11 0xffffffff80712f3c at amd64_syscall+0x2ac
#12 0xffffffff806f6b0b at Xfast_syscall+0xfb



The second one looks like and old friend:
http://sources.zabbadoz.net/freebsd/lor/280.html

I couldn't fine the first one on that list, though.



This happened while running Poudriere on a full UFS+SU (no ZFS) system.
In fact Poudriere has brough my box to a complete hang several times (forcing an hardware reset) and that's why I enabled WITNESS, INVARIANTS, etc...
After such an hang, the system is very likely to panic over and over again until I boot into single user mode and "fsck -y" every filesystem (including a full fsck after replaying the journal on UFS+SU+J partitions).



On another run only the second LOR showed up without the first.