Created attachment 235531 [details] Core text from cfrash dump Adding INVARIANTS & INVARIANT_SUPPORT to GENERIC kernel configuration file compiles, but panics system during the next boot cycle. Panic string is similar to what is reported in BuG ID 259707. panic: mutex mrsas_sim_lock not owned at /usr/src/sys/kern/kern_mutex.c:204 Attaching core.txt.2 from crash dump.
Warner, are you in contact with the maintainer of this driver at Broadcom?
(In reply to Mark Johnston from comment #1) I've forwarded this to them and see if they are still the right point of contact.
Update: I re-loaded the same system with the latest -STABLE branch release (07/29) and I am not seeing the issue.
(In reply to Terry Beck from comment #3) Is this still with INVARIANTS added? If not, then that's probably because the assertion in question simply isn't enabled.
yes, INVARIANTS are enabled. I used the same conf file as with the -RELEASE installation.
(In reply to Terry Beck from comment #5) Looking at the code, I'm pretty sure that the problem is still there in stable/13, so I can't explain why it's not happening anymore.
(In reply to Mark Johnston from comment #6) Hello, I faced the same issue running HardenedBSD (13-stable and current) on a HP Server DL360 Gen 10 plus with BROADCOM AERO-10E2 SAS Raid Controller. kernel panic occurs as soon as a write access is attempted on /dev/da0. So I decided to test on FreeBSD and I can confirm that : - On FreeBSD-13.2-RC3 -> it works - On FreeBSD-14.0-CURRENT -> kernel panic According to me this problem is related to the "INVARIANTS" options in kernel : the mrsas driver source code has not been modified to deal with these options. kernel panic occurs when trying to unlock the "sim_lock" mutex. With a custom GENERIC kernel without these options (tested on HardenedBSD), it works. As a proof of concept, I patched sys/kern/kern_mutex.c to not panic in case of wrong assertion (!mtx_owned(m)) on the mutex "mrsas_sim_lock" used in sys/dev/mrsas/mrsas.c. With this workaround in place, even with INVARIANTS options, no problem with mrsas driver anymore. Hope it can help to fix this issue, I would be happy to help testing a patch or anything, do not hesitate.
(In reply to Jérémie Jourdin from comment #7) Forgot to mention that problem occurs under RAID configuration (R1 in my use case). No problem with JBOD.
Created attachment 241093 [details] proposed patch Please give the attached patch a try with INVARIANTS enabled.
(In reply to Mark Johnston from comment #9) Thank you. The proposed patch is working for me : no kernel panic anymore, even with INVARIANTS* options enabled in kernel. Tested on : - HPE DL360 Gen10+ (Raid 1) - HardenedBSD 13-stable - Commit 02f071b2d30b0bfb4a674c9577892668772698bd
https://reviews.freebsd.org/D39559
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4640df1b0a49697840b81f6bcd269a483514c6aa commit 4640df1b0a49697840b81f6bcd269a483514c6aa Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-10-07 00:31:03 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-10-07 00:31:03 +0000 mrsas: Fix callout locking in mrsas_complete_cmd() callout_stop() requires the associated lock to be held. This is a bit hacky, but I believe it's safe since the subsequent mrsas_cmd_done() call will also acquire the SIM lock to stop a different callout. PR: 265484 Reviewed by: imp Tested by: Jérémie Jourdin <jeremie.jourdin@advens.fr> MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39559 sys/dev/mrsas/mrsas.c | 2 ++ 1 file changed, 2 insertions(+)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1b934004c04c64e96ed08b1a0c560225def21d94 commit 1b934004c04c64e96ed08b1a0c560225def21d94 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-10-07 00:31:03 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-10-14 15:28:48 +0000 mrsas: Fix callout locking in mrsas_complete_cmd() callout_stop() requires the associated lock to be held. This is a bit hacky, but I believe it's safe since the subsequent mrsas_cmd_done() call will also acquire the SIM lock to stop a different callout. PR: 265484 Reviewed by: imp Tested by: Jérémie Jourdin <jeremie.jourdin@advens.fr> MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39559 (cherry picked from commit 4640df1b0a49697840b81f6bcd269a483514c6aa) sys/dev/mrsas/mrsas.c | 2 ++ 1 file changed, 2 insertions(+)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=29de7af6ee47a3a251763e4220203a0d960ea532 commit 29de7af6ee47a3a251763e4220203a0d960ea532 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-10-07 00:31:03 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-10-14 15:29:11 +0000 mrsas: Fix callout locking in mrsas_complete_cmd() callout_stop() requires the associated lock to be held. This is a bit hacky, but I believe it's safe since the subsequent mrsas_cmd_done() call will also acquire the SIM lock to stop a different callout. PR: 265484 Reviewed by: imp Tested by: Jérémie Jourdin <jeremie.jourdin@advens.fr> MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39559 (cherry picked from commit 4640df1b0a49697840b81f6bcd269a483514c6aa) sys/dev/mrsas/mrsas.c | 2 ++ 1 file changed, 2 insertions(+)
This will be included in 14.0 as well.
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=630daf68d5bd73136e981610449bf094ca00a4a3 commit 630daf68d5bd73136e981610449bf094ca00a4a3 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-10-07 00:31:03 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-10-15 13:16:12 +0000 mrsas: Fix callout locking in mrsas_complete_cmd() callout_stop() requires the associated lock to be held. This is a bit hacky, but I believe it's safe since the subsequent mrsas_cmd_done() call will also acquire the SIM lock to stop a different callout. Approved by: re (gjb) PR: 265484 Reviewed by: imp Tested by: Jérémie Jourdin <jeremie.jourdin@advens.fr> MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39559 (cherry picked from commit 4640df1b0a49697840b81f6bcd269a483514c6aa) (cherry picked from commit 29de7af6ee47a3a251763e4220203a0d960ea532) sys/dev/mrsas/mrsas.c | 2 ++ 1 file changed, 2 insertions(+)