Bug 210907 - [lor] lock order reversal in /usr/src/sys/kern/kern_jail.c:1031
Summary: [lor] lock order reversal in /usr/src/sys/kern/kern_jail.c:1031
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-07-08 04:52 UTC by Paul Armstrong
Modified: 2020-01-21 11:00 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Armstrong 2016-07-08 04:52:24 UTC
lock order reversal:
 1st 0xffffffff81dc4118 allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:1031
 2nd 0xffffffff81de9c68 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ /usr/src/sys/net/vnet.c:574
stack backtrace:
#0 0xffffffff80aae180 at witness_debugger+0x70
#1 0xffffffff80aae074 at witness_checkorder+0xe54
#2 0xffffffff80a56e73 at _sx_slock+0x73
#3 0xffffffff80b6a451 at vnet_alloc+0xf1
#4 0xffffffff80a16d2e at kern_jail_set+0x1d8e
#5 0xffffffff80a18801 at sys_jail_set+0x41
#6 0xffffffff80f16b2b at amd64_syscall+0x2db
#7 0xffffffff80ef68cb at Xfast_syscall+0xfb
ifa_maintain_loopback_route: deletion failed for interface lo0: 48


Noticed when shutting down a jail.
11.0 Alpha 6

Delta from GENERIC:
device pf
device pflog
device pfsync

options VIMAGE
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_CDNR
options ALTQ_PRIQ
options ALTQ_NOPCC
Comment 1 Mark Millard 2018-10-18 04:15:50 UTC
I'm see this followed by a crash on a head -r339076
based build for powerpc64. It was during a kyua
test:

sys/netinet/reuseport_lb:basic_ipv4  ->  failed: /usr/src/tests/sys/netinet/reuseport_lb.c:165: bind() failed: Address already in use  [0.022s]
sys/netinet/reuseport_lb:basic_ipv6  ->  failed: /usr/src/tests/sys/netinet/reuseport_lb.c:221: bind() failed: Address already in use  [0.022s]
sys/netipsec/tunnel/aes_cbc_128_hmac_sha1:v4  ->

Console/logging:

epair3a: Ethernet address: 02:60:27:70:4b:0a
epair3b: Ethernet address: 02:60:27:70:4b:0b
epair3a: link state changed to UP
epair3b: link state changed to UP
lock order reversal:
 1st 0x13be260 allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:960
 2nd 0x15964a0 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ /usr/src/sys/net/vnet.c:575
stack backtrace:
#0 0x6f6520 at witness_debugger+0xf4
#1 0x6f8440 at witness_checkorder+0xa1c
#2 0x675690 at _sx_slock_int+0x70
#3 0x675810 at _sx_slock+0x1c
#4 0x7f4338 at vnet_sysinit+0x38
#5 0x7f44dc at vnet_alloc+0x118
#6 0x62ab84 at kern_jail_set+0x3274
#7 0x62b62c at sys_jail_set+0x8c
#8 0xa8a798 at trap+0x9a0
#9 0xa7e660 at powerpc_interrupt+0x140

fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0xc00000008df1df30
   dsisr           = 0x42000000
   srr0            = 0xe000000047854e98 (0xe000000047854e98)
   srr1            = 0x9000000000009032
   current msr     = 0x9000000000009032
   lr              = 0xe000000047854e90 (0xe000000047854e90)
   curthread       = 0xc0000000206b6000
          pid = 9464, comm = jail

(Hand transcribed from here on:)

[ thread pid 9464 tid 100296 ]
Stopped at vnet_epair_init+0x78: stdx r3,r29,r30
db:0:kdb.enter.default> bt
Tracing pid 9464 tid 100296 td 0xc0000000206b6000
0xe000000047274240: at vnet_sysinit+0x70
0xe000000047274270: at vnet_alloc+0x118
0xe000000047274300: at kern_jail_set+0x32740
0xe000000047274610: at sys_jail_set+08c
0xe000000047274660: at trap+0x9a0
0xe000000047274790: at powerpc_interrupt+0x140
0xe000000047274820: user sc trap by 0x81016a888
srr1 = 0x900000000000f032
r1   = 0x3fffffffffffd080
cr   = 0x28002482
xer  = 0x20000000
ctr  = 0x81016a880
r2   = 0x810322300
Comment 2 Mark Millard 2018-10-18 06:21:26 UTC
(In reply to Mark Millard from comment #1)

Typo fixes:

0x32740 should have been: 0x3274
08c should have been 0x8c

Given those . . .

The Lock Order Reversal's back trace and the
crash's back trace are partially the same,
name for name and offset for offset:

#5 0x7f44dc at vnet_alloc+0x118
#6 0x62ab84 at kern_jail_set+0x3274
#7 0x62b62c at sys_jail_set+0x8c
#8 0xa8a798 at trap+0x9a0
#9 0xa7e660 at powerpc_interrupt+0x140

and:

0xe000000047274270: at vnet_alloc+0x118
0xe000000047274300: at kern_jail_set+0x3274
0xe000000047274610: at sys_jail_set+0x8c
0xe000000047274660: at trap+0x9a0
0xe000000047274790: at powerpc_interrupt+0x140

But for:

#4 0x7f4338 at vnet_sysinit+0x38
vs.
0xe000000047274240: at vnet_sysinit+0x70

the offsets are different.