Bug 271768 - panic: _sx_xlock_hard: recursed on non-recursive sx iflib ctx lock @ /usr/src/sys/net/iflib.c:2582
Summary: panic: _sx_xlock_hard: recursed on non-recursive sx iflib ctx lock @ /usr/src...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: crash, iflib
Depends on:
Blocks:
 
Reported: 2023-06-01 15:53 UTC by Piotr Kubaj
Modified: 2023-07-20 23:04 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Piotr Kubaj freebsd_committer freebsd_triage 2023-06-01 15:53:36 UTC
I run CURRENT revision baef3a5b585f. my kernel config is:
include GENERIC
ident VAL
nodevice em
nodevice ixl
nodevice iavf
nodevice ice
nodevice ix
nodevice ixv

When I try to kldunload if_ice, I get:
panic: _sx_xlock_hard: recursed on non-recursive sx iflib ctx lock @ /usr/src/sys/net/iflib.c:2582

cpuid = 3
time = 1685630930
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe010b8475d0
vpanic() at vpanic+0x150/frame 0xfffffe010b847620
panic() at panic+0x43/frame 0xfffffe010b847680
_sx_xlock_hard() at _sx_xlock_hard+0x82d/frame 0xfffffe010b847720
_sx_xlock() at _sx_xlock+0xb3/frame 0xfffffe010b847760
iflib_media_status() at iflib_media_status+0x32/frame 0xfffffe010b847790
ifmedia_ioctl() at ifmedia_ioctl+0x16a/frame 0xfffffe010b8477c0
dump_iface() at dump_iface+0x12f/frame 0xfffffe010b847840
rtnl_handle_ifevent() at rtnl_handle_ifevent+0xab/frame 0xfffffe010b8478c0
if_attach_internal() at if_attach_internal+0x41e/frame 0xfffffe010b847910
ether_ifattach() at ether_ifattach+0x29/frame 0xfffffe010b847940
iflib_device_register() at iflib_device_register+0xbf7/frame 0xfffffe010b8479d0
iflib_device_attach() at iflib_device_attach+0xb5/frame 0xfffffe010b847a00
device_attach() at device_attach+0x3c0/frame 0xfffffe010b847a40
device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xfffffe010b847a70
pci_driver_added() at pci_driver_added+0xf2/frame 0xfffffe010b847ab0
devclass_driver_added() at devclass_driver_added+0x39/frame 0xfffffe010b847af0
device_do_deferred_actions() at device_do_deferred_actions+0x3a/frame 0xfffffe010b847b10
devctl2_ioctl() at devctl2_ioctl+0x531/frame 0xfffffe010b847be0
devfs_ioctl() at devfs_ioctl+0xcc/frame 0xfffffe010b847c30
vn_ioctl() at vn_ioctl+0xc0/frame 0xfffffe010b847ca0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe010b847cc0
kern_ioctl() at kern_ioctl+0x1fe/frame 0xfffffe010b847d30
sys_ioctl() at sys_ioctl+0x154/frame 0xfffffe010b847e00
amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe010b847f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010b847f30
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x1da51d7cee6a, rsp = 0x1da51b4eb188, rbp = 0x1da51b4eb240 ---
Comment 1 Piotr Kubaj freebsd_committer freebsd_triage 2023-06-01 16:10:18 UTC
This issue also happens if I disable devmatch and load the ice module manually. For some reason it doesn't happen during autoloading with enabled devmatch.
Comment 2 Przemysław Lewandowski 2023-06-02 06:34:35 UTC
I added "devmatch_blocklist="if_ice"" in /etc/rc.conf and automatically reload doesn't occur. It helps to confirm that panic exists only during load driver.

It looks like the CTX_LOCK is already locked in iflib_media_status.
I remember that a few weeks ago disabling NETLINK was helping but not now.
Comment 3 Przemysław Lewandowski 2023-06-02 13:58:01 UTC
CTX is locked first in iflib_device_register and second in iflib_media_status(here panic).
SIOCGIFMEDIA is received in iflib_if_ioctl(then ifmedia_ioctl -> iflib_media_status) but on 13.2 it is not sent from get_operstate_ether() in netlink module. On 14.0 is sent from netlink.
Can issue be related to ifnet_arrival_event? It needs to be investigated more.
Comment 4 Przemysław Lewandowski 2023-06-15 07:07:25 UTC
Prepared fix for this: https://reviews.freebsd.org/D40557
Comment 5 commit-hook freebsd_committer freebsd_triage 2023-07-19 23:38:52 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=a52f23f4c49e4766fb9eb0bf460cc83c5f63f17d

commit a52f23f4c49e4766fb9eb0bf460cc83c5f63f17d
Author:     Eric Joyner <erj@FreeBSD.org>
AuthorDate: 2023-07-19 22:40:46 +0000
Commit:     Eric Joyner <erj@FreeBSD.org>
CommitDate: 2023-07-19 22:51:26 +0000

    iflib: Unlock ctx lock around call to ether_ifattach()

    Panic occurs during loading driver using kldload. It exists since netlink is
    enabled.  There is problem with double locking ctx. This fix allows to call
    ether_ifattach() without locked ctx.

    Signed-off-by: Eric Joyner <erj@FreeBSD.org>

    PR:             271768
    Reviewed by:    erj@, jhb@
    MFC after:      1 day
    Sponsored by:   Intel Corporation
    Differential Revision:  https://reviews.freebsd.org/D40557

 sys/net/iflib.c | 6 ++++++
 1 file changed, 6 insertions(+)
Comment 6 commit-hook freebsd_committer freebsd_triage 2023-07-20 22:48:30 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=93f39b4675865094b298e1fca5bcfa788348dd36

commit 93f39b4675865094b298e1fca5bcfa788348dd36
Author:     Przemyslaw Lewandowski <przemyslawx.lewandowski@intel.com>
AuthorDate: 2023-07-19 22:40:46 +0000
Commit:     Eric Joyner <erj@FreeBSD.org>
CommitDate: 2023-07-20 22:42:22 +0000

    iflib: Unlock ctx lock around call to ether_ifattach()

    Panic occurs during loading driver using kldload. It exists since netlink is
    enabled.  There is problem with double locking ctx. This fix allows to call
    ether_ifattach() without locked ctx.

    Signed-off-by: Eric Joyner <erj@FreeBSD.org>

    PR:             271768
    Reviewed by:    erj@, jhb@
    Sponsored by:   Intel Corporation
    Differential Revision:  https://reviews.freebsd.org/D40557

    (cherry picked from commit a52f23f4c49e4766fb9eb0bf460cc83c5f63f17d)

 sys/net/iflib.c | 6 ++++++
 1 file changed, 6 insertions(+)
Comment 7 commit-hook freebsd_committer freebsd_triage 2023-07-20 22:57:33 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=2617bca5a8c98e9e84fba3f2ebd9302e8e893b8f

commit 2617bca5a8c98e9e84fba3f2ebd9302e8e893b8f
Author:     Przemyslaw Lewandowski <przemyslawx.lewandowski@intel.com>
AuthorDate: 2023-07-19 22:40:46 +0000
Commit:     Eric Joyner <erj@FreeBSD.org>
CommitDate: 2023-07-20 22:56:41 +0000

    iflib: Unlock ctx lock around call to ether_ifattach()

    Panic occurs during loading driver using kldload. It exists since netlink is
    enabled.  There is problem with double locking ctx. This fix allows to call
    ether_ifattach() without locked ctx.

    Signed-off-by: Eric Joyner <erj@FreeBSD.org>

    PR:             271768
    Reviewed by:    erj@, jhb@
    Sponsored by:   Intel Corporation
    Differential Revision:  https://reviews.freebsd.org/D40557

    (cherry picked from commit a52f23f4c49e4766fb9eb0bf460cc83c5f63f17d)

 sys/net/iflib.c | 6 ++++++
 1 file changed, 6 insertions(+)
Comment 8 Eric Joyner freebsd_committer freebsd_triage 2023-07-20 23:04:04 UTC
At least at Intel, Przemyslaw's fix did fix this issue for us.