Kernel: 14.3-RELEASE-p2 Unread portion of the kernel message buffer: rdi: fffffe13be744c70 rsi: 0000000000000001 rdx: fffff85ee4b83201 rcx: 0000000000000000 r8: 0000000000000201 r9: fffff857376a7900 rax: 0000000000000000 rbx: fffff85ee4b831c0 rbp: fffffe13be744c30 r10: 0000000000000218 r11: 0000000000000c01 r12: fffff89ffa412000 r13: fffff88551397000 r14: fffff8564a211600 r15: fffffe143354b100 trap number = 12 panic: page fault cpuid = 43 time = 1757978146 KDB: stack backtrace: #0 0xffffffff80ba8f1d at kdb_backtrace+0x5d #1 0xffffffff80b5aa11 at vpanic+0x161 #2 0xffffffff80b5a8a3 at panic+0x43 #3 0xffffffff8104dbfa at trap_pfault+0x3da #4 0xffffffff81023d98 at calltrap+0x8 #5 0xffffffff811102dd at VOP_CLOSE_APV+0x1d #6 0xffffffff80c5aa0c at vn_close1+0x14c #7 0xffffffff80c58e0d at vn_closefile+0x3d #8 0xffffffff80af6e61 at _fdrop+0x11 #9 0xffffffff80c0df29 at unp_dispose+0x269 #10 0xffffffff80c0476e at soshutdown+0x10e #11 0xffffffff80c0bb21 at kern_shutdown+0x51 #12 0xffffffff8104e547 at amd64_syscall+0x117 #13 0xffffffff810246ab at fast_syscall_common+0xf8 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xffffffff80b5a56b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:523 #3 0xffffffff80b5aa69 in vpanic (fmt=0xffffffff811a03c8 "%s", ap=ap@entry=0xfffffe13be744a00) at /usr/src/sys/kern/kern_shutdown.c:967 #4 0xffffffff80b5a8a3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:891 #5 0xffffffff8104dbfa in trap_fatal (frame=<optimized out>, eva=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:1000 #6 0xffffffff8104dbfa in trap_pfault (frame=0xfffffe13be744a80, usermode=false, signo=<optimized out>, ucode=<optimized out>) #7 <signal handler called> #8 0xffffffff8418d4a5 in fuse_vnop_close (ap=0xfffffe13be744c70) at /usr/src/sys/fs/fuse/fuse_vnops.c:286 #9 0xffffffff811102dd in VOP_CLOSE_APV (vop=0xffffffff84196d88 <fuse_vnops>, a=a@entry=0xfffffe13be744c70) at vnode_if.c:496 #10 0xffffffff80c5aa0c in VOP_CLOSE (vp=0xfffff85ee4b831c0, fflag=1, cred=0xfffff8564a211600, td=0x0) at ./vnode_if.h:247 #11 vn_close1 (vp=vp@entry=0xfffff85ee4b831c0, flags=1, file_cred=0xfffff8564a211600, td=0x0, keep_ref=false) at /usr/src/sys/kern/vfs_vnops.c:543 #12 0xffffffff80c58e0d in vn_closefile (fp=0xfffff93fb97d42d0, td=0x0) at /usr/src/sys/kern/vfs_vnops.c:1872 #13 0xffffffff80af6e61 in fo_close (fp=0xfffffe13be744c70, td=0x1, td@entry=0x0) at /usr/src/sys/sys/file.h:396 #14 _fdrop (fp=0xfffffe13be744c70, td=0x1, td@entry=0x0) at /usr/src/sys/kern/kern_descrip.c:3711 #15 0xffffffff80afa73a in closef_nothread (fp=0xfffffe13be744c70) at /usr/src/sys/kern/kern_descrip.c:2852 #16 0xffffffff80c0e208 in unp_discard (fp=0xfffffe13be744c70, fp@entry=0xfffff93fb97d42d0) at /usr/src/sys/kern/uipc_usrreq.c:2921 #17 0xffffffff80c0df29 in unp_freerights (fdep=0xfffff8c087015830, fdcount=1) at /usr/src/sys/kern/uipc_usrreq.c:2431 #18 unp_scan (m0=0xfffff941ae168500, op=<optimized out>) at /usr/src/sys/kern/uipc_usrreq.c:3333 #19 unp_dispose (so=<optimized out>) at /usr/src/sys/kern/uipc_usrreq.c:3302 #20 0xffffffff80c0476e in soshutdown (so=<optimized out>, how=how@entry=2) at /usr/src/sys/kern/uipc_socket.c:3661 #21 0xffffffff80c0bb21 in kern_shutdown (td=0xfffff89ffa412000, s=<optimized out>, how=2) at /usr/src/sys/kern/uipc_syscalls.c:1210 #22 0xffffffff8104e547 in syscallenter (td=0xfffff89ffa412000) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193 #23 amd64_syscall (td=0xfffff89ffa412000, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1241 #24 <signal handler called> #25 0x0000000825796a7a in ?? () Backtrace stopped: Cannot access memory at address 0x8456c4ef8 Looks like the problem is in sys/fs/fuse/fuse_vnops.c:788 where ap->a_td returned NULL: 788 struct thread *td = ap->a_td; In line 789, it's trying to dereference td->td_proc->p_pid and crashes because td is NULL. Based on my modest knowledge, the fix is quite simple. At line 789: - pid_t pid = td->td_proc->p_pid; + pid_t pid = (td != NULL) ? td->td_proc->p_pid : 0; Looking deeper into the code, it's allowed to have a pid equal to 0, and it shouldn't crash.
The second pointer dereference is in line 828: 828 fuse_vnode_savesize(vp, cred, td->td_proc->p_pid); I guess it can be replaced by: 828 fuse_vnode_savesize(vp, cred, pid);
Good catch! Can you please tell me how to reproduce it? What kind of file was this? A unix domain named socket? And what kind of process was it? It's weird for a_td to be NULL.
On Thu Sep 18 21:23:03 2025 UTC, asomers@FreeBSD.org wrote: > Good catch! Can you please tell me how to reproduce it? What kind of file was > this? A unix domain named socket? And what kind of process was it? It's weird > for a_td to be NULL. Yes, that was unix(4) socket, that explicitly calls closef_nothread().
(In reply to Alan Somers from comment #2) The remote storage was mounted via SSHFS. The Clam AntiVirus daemon (clamd) service performed recursive scanning beginning from the root directory and had some files mounted from the FUSE-mounted directory. The panic occurred when clamd was restarted. I think the issue is between stopping the Clamd daemon and killing all its child processes/threads, and closing the file descriptor of a file/socket located in the Fuse-mounted directory.
Is there any possibility that ClamAV opened two unix-domain sockets, then attempted to send one of them through the other using sendmsg with SCM_RIGHTS? That's my best theory for why a_td was NULL, especially in light of git rev 0cb64678bc (svn r216150).
(In reply to Alan Somers from comment #5) Sorry, I have no idea how it attempted to send. I cannot reproduce this behavior.
This happened not in the deferred mech, straight through the syscall: KDB: stack backtrace: #0 0xffffffff80ba8f1d at kdb_backtrace+0x5d #1 0xffffffff80b5aa11 at vpanic+0x161 #2 0xffffffff80b5a8a3 at panic+0x43 #3 0xffffffff8104dbfa at trap_pfault+0x3da #4 0xffffffff81023d98 at calltrap+0x8 #5 0xffffffff811102dd at VOP_CLOSE_APV+0x1d #6 0xffffffff80c5aa0c at vn_close1+0x14c #7 0xffffffff80c58e0d at vn_closefile+0x3d #8 0xffffffff80af6e61 at _fdrop+0x11 #9 0xffffffff80c0df29 at unp_dispose+0x269 #10 0xffffffff80c0476e at soshutdown+0x10e #11 0xffffffff80c0bb21 at kern_shutdown+0x51 #12 0xffffffff8104e547 at amd64_syscall+0x117 #13 0xffffffff810246ab at fast_syscall_common+0xf8 I think DDB lost a few frames here, should be: ... _fdrop closef_nothread unp_discard unp_freerights unp_scan unp_dispose ...
I've managed to reproduce the crash. The trick was that the file that causes the crash _wasn't_ the socket. Rather, the conditions that lead to the crash are: * Two processes are communicating via Unix domain sockets. * One process opens a _regular_file_ on a fusefs file system. * The sending process sends the file descriptor through a socket via SCM_RIGHTS * The sending process then closes its regular file descriptor * The receiving process closes its Unix domain socket, or shuts it down, without reading the file descriptor from it. If the receiving process crashes, that would probably work too. Since the last open copy of the file was stored within a socket's receive buffer, it ends up getting closed with no associated thread. I'll work on a patch.
I've opened a code review (linked) with a patch.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e043af9ca59608309cac2fd222c17f989ba0d35e commit e043af9ca59608309cac2fd222c17f989ba0d35e Author: Alan Somers <asomers@FreeBSD.org> AuthorDate: 2025-09-19 16:02:25 +0000 Commit: Alan Somers <asomers@FreeBSD.org> CommitDate: 2025-09-19 22:38:42 +0000 fusefs: fix a kernel panic regarding SCM_RIGHTS If the last copy of an open file resides within the socket buffer of a unix-domain socket, then VOP_CLOSE will be called with no thread information. Fix fusefs to handle that case, and add a regression test. Also add a test case for writes to a file that lies within a sockbuf. Along with close, a write from the writeback cache is the only other operation I can think of that might apply to a file residing in a sockbuf. PR: 289686 Reported by: iron.udjin@gmail.com MFC after: 1 week Sponsored by: ConnectWise Reviewed by: glebius, markj Differential Revision: https://reviews.freebsd.org/D52625 sys/fs/fuse/fuse_vnops.c | 10 ++++-- tests/sys/fs/fusefs/release.cc | 54 +++++++++++++++++++++++++++++++ tests/sys/fs/fusefs/write.cc | 73 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 134 insertions(+), 3 deletions(-)
I've fixed it in main. Ordinarily I would wait for the reporter to have a chance to test the patch before committing it. But since 15.0-RELEASE is so close I am being hasty.
(In reply to Alan Somers from comment #11) I'm not sure that I reproduced the issue correctly, but I patched the sources, built the kernel, restarted, mounted SSHFS, tried to scan the remote directory with Clamd, and restarted it a few times. No issue appears. Looks like it fixed.
Great news, iron.udjin .
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=5206983e76c0bd42893eae98955bc8168a0a8d0a commit 5206983e76c0bd42893eae98955bc8168a0a8d0a Author: Alan Somers <asomers@FreeBSD.org> AuthorDate: 2025-09-19 16:02:25 +0000 Commit: Alan Somers <asomers@FreeBSD.org> CommitDate: 2025-09-26 18:11:37 +0000 fusefs: fix a kernel panic regarding SCM_RIGHTS If the last copy of an open file resides within the socket buffer of a unix-domain socket, then VOP_CLOSE will be called with no thread information. Fix fusefs to handle that case, and add a regression test. Also add a test case for writes to a file that lies within a sockbuf. Along with close, a write from the writeback cache is the only other operation I can think of that might apply to a file residing in a sockbuf. PR: 289686 Reported by: iron.udjin@gmail.com Sponsored by: ConnectWise Reviewed by: glebius, markj Differential Revision: https://reviews.freebsd.org/D52625 (cherry picked from commit e043af9ca59608309cac2fd222c17f989ba0d35e) sys/fs/fuse/fuse_vnops.c | 10 ++++-- tests/sys/fs/fusefs/release.cc | 54 +++++++++++++++++++++++++++++++ tests/sys/fs/fusefs/write.cc | 73 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 134 insertions(+), 3 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3152a2f5dd0447daba80466f70c66ea794400015 commit 3152a2f5dd0447daba80466f70c66ea794400015 Author: Alan Somers <asomers@FreeBSD.org> AuthorDate: 2025-09-19 16:02:25 +0000 Commit: Alan Somers <asomers@FreeBSD.org> CommitDate: 2025-09-27 14:30:13 +0000 fusefs: fix a kernel panic regarding SCM_RIGHTS If the last copy of an open file resides within the socket buffer of a unix-domain socket, then VOP_CLOSE will be called with no thread information. Fix fusefs to handle that case, and add a regression test. Also add a test case for writes to a file that lies within a sockbuf. Along with close, a write from the writeback cache is the only other operation I can think of that might apply to a file residing in a sockbuf. PR: 289686 Reported by: iron.udjin@gmail.com Sponsored by: ConnectWise Reviewed by: glebius, markj Differential Revision: https://reviews.freebsd.org/D52625 (cherry picked from commit e043af9ca59608309cac2fd222c17f989ba0d35e) sys/fs/fuse/fuse_vnops.c | 10 ++++-- tests/sys/fs/fusefs/release.cc | 54 +++++++++++++++++++++++++++++++ tests/sys/fs/fusefs/write.cc | 73 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 134 insertions(+), 3 deletions(-)