Created attachment 262016 [details] procstat output On the fresh ( https://github.com/freebsd/freebsd-src/commit/f6e5bcd4716c2974c7ceb1fbbfc0ef8db8e13d8b ) CURRENT I started getting random processes to hang at random times. When the process hangs, `top` reports its state as [zfs]. Here is the procstat output for the hanged system state: https://termbin.com/o8sf (also attached). In that case the hanged processes were - yakuake - nextcloud - kalendarac - plasmashell - pulseaudio All of these processes are waiting on mi_switch+0x172 sleepq_switch+0x109 sleeplk+0x110 lockmgr_slock_hard+0x3c3 VOP_LOCK1_APV+0x2a _vn_lock+0x53 vfs_lookup+0x14c namei+0x35d kern_accessat+0x1c0 amd64_syscall+0x169 fast_syscall_common+0xf8 Interestingly, there is one more process waiting on this lock, but in the "select" state - xdg-desktop-portal: mi_switch+0x172 sleepq_switch+0x109 sleeplk+0x110 lockmgr_slock_hard+0x3c3 VOP_LOCK1_APV+0x2a _vn_lock+0x53 vfs_lookup+0x14c namei+0x35d vn_inotify_add_watch+0x1f5 VOP_INOTIFY_ADD_WATCH_APV+0x2a kern_inotify_add_watch+0x34a amd64_syscall+0x169 fast_syscall_common+0xf8 This made me think that recent inotify changes might be related to this hang. After the deadlock resolver kicked in I was able to print the location: https://github.com/freebsd/freebsd-src/blob/78935fa40eee14aab1c6de676a1017bcabdf5fa1/sys/kern/vfs_subr.c#L3384
Created attachment 262018 [details] Another example of hang Attached another example of the hang. This time xdg-desktop-portal has no mentions of inotify, so maybe it is indeed unrelated.
All "zfs" thjreads wait for some zfs vnode lock. It is crucial to understand which lock is it, and what does it owner do. I do not see an obvious owner candidate from the backtraces, but there are a lot, so it is not hard to miss it. Follow https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/#kerneldebug-deadlocks to gather the required debugging info.
In fact it might be as simple as https://reviews.freebsd.org/D51233
I can confirm that https://reviews.freebsd.org/D51233 fixed issue with processes that stuck in _vn_lock vfs_lookup namei vn_inotify_add_watch sequence Thank you
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=63389aea24f15fe823c775960d77017f1a35f54b commit 63389aea24f15fe823c775960d77017f1a35f54b Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2025-07-10 07:15:13 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2025-07-10 14:34:45 +0000 inotify: do not call into namei() with a locked vnode PR: 288127 Reviewed by: markj Fixes: f1f230439fa48581f40a57f095627f667a9713c3 Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D51233 sys/kern/vfs_inotify.c | 2 ++ 1 file changed, 2 insertions(+)
I haven't seen a hang yet after applying this patch too. But it feels weird to me that the bug happens during the mount attempt, because for me most of hangs were happening during the KDE Plasma startup - and nothing is mounted during the process. I will monitor for hangs for a few days more.
(In reply to Gleb Popov from comment #6) It is not about mount action, but about passing the mount point during lookup.
I think we can close this, I see no hangs after applying Konstantin's fix. Thanks kib for quickly fixing this!
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e8c54222acdbb977c4164667627b6ee569d2eb90 commit e8c54222acdbb977c4164667627b6ee569d2eb90 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2025-07-10 07:15:13 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2026-08-05 15:15:16 +0000 inotify: do not call into namei() with a locked vnode PR: 288127 Reviewed by: markj Fixes: f1f230439fa48581f40a57f095627f667a9713c3 Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D51233 (cherry picked from commit 63389aea24f15fe823c775960d77017f1a35f54b) sys/kern/vfs_inotify.c | 2 ++ 1 file changed, 2 insertions(+)