There was recent work in the stack to not hold non-sleepable locks around NIC driver ioctls. It seems that vlan(4) still does. begin_synchronized_op with the following non-sleepable locks held: exclusive rm vlan (vlan) r = 0 (0xfffff8000459e388) locked @ /root/ws/head/sys/net/if_vlan.c:1917 shared rm vlan_rm (vlan_rm) r = 0 (0xffffffff827e9480) locked @ /root/ws/head/sys/net/if_vlan.c:1914 stack backtrace: #0 0xffffffff81238331 at witness_debugger+0xf1 #1 0xffffffff812398a8 at witness_warn+0x3b8 #2 0xffffffff8326296c at begin_synchronized_op+0x6c #3 0xffffffff83275b72 at cxgbe_ioctl+0x382 #4 0xffffffff8132e24a at if_addmulti+0x38a #5 0xffffffff8134ebf4 at vlan_setmulti+0x384 #6 0xffffffff8134dbc7 at vlan_ioctl+0x797 #7 0xffffffff8132e24a at if_addmulti+0x38a #8 0xffffffff814181c8 at in_getmulti+0x178 #9 0xffffffff81417ee8 at in_joingroup_locked+0xb8 #10 0xffffffff81417e0a at in_joingroup+0x4a #11 0xffffffff8140f326 at in_aifaddr_ioctl+0xe46 #12 0xffffffff8140d6f8 at in_control+0x228 #13 0xffffffff8132abdd at ifioctl+0x7fd #14 0xffffffff8124e4ce at soo_ioctl+0x79e #15 0xffffffff8124053c at fo_ioctl+0x4c #16 0xffffffff81240291 at kern_ioctl+0x361 #17 0xffffffff8123fe6a at sys_ioctl+0x2ea
We also see this when using iflib drivers: lock order reversal: (sleepable after non-sleepable) 1st 0xfffff807b4884d08 vlan (vlan) @ /usr/src/sys/net/if_vlan.c:1931 2nd 0xfffff80549c87560 iflib ctx lock (iflib ctx lock) @ /usr/src/sys/net/iflib.c:4218 stack backtrace: #0 0xffffffff80bc1c83 at witness_debugger+0x73 #1 0xffffffff80bc19d3 at witness_checkorder+0xab3 #2 0xffffffff80b62e68 at _sx_xlock+0x68 #3 0xffffffff80c7a2bc at iflib_if_ioctl+0x8c #4 0xffffffff80c5beca at if_addmulti+0x24a #5 0xffffffff80c6c9c6 at vlan_setmulti+0x196 #6 0xffffffff80c6bb89 at vlan_ioctl+0xa9 #7 0xffffffff80c5beca at if_addmulti+0x24a #8 0xffffffff80ce0075 at in_joingroup_locked+0x235 #9 0xffffffff80cdfe12 at in_joingroup+0x42 #10 0xffffffff80cdb358 at in_control+0xb08 #11 0xffffffff80c5af73 at ifioctl+0x1853 #12 0xffffffff80bc6bba at kern_ioctl+0x2ba #13 0xffffffff80bc688e at sys_ioctl+0x15e #14 0xffffffff8102ce0c at amd64_syscall+0x28c #15 0xffffffff810095fd at fast_syscall_common+0x101
I believe this is fixed in r338850. *** This bug has been marked as a duplicate of bug 230510 ***