Apr 28 04:59:57 xxhost kernel: newnfs: server '192.168.15.57' error: fileid changed. fsid 2f67004f:509582de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) Apr 28 14:33:05 xxhost syslogd: kernel boot file is /boot/kernel/kernel Apr 28 14:33:05 xxhost kernel: Apr 28 14:33:05 xxhost syslogd: last message repeated 1 times Apr 28 14:33:05 xxhost kernel: Fatal trap 12: page fault while in kernel mode Apr 28 14:33:05 xxhost kernel: cpuid = 0; apic id = 00 Apr 28 14:33:05 xxhost kernel: fault virtual address = 0x0 Apr 28 14:33:05 xxhost kernel: fault code = supervisor read data, page not present Apr 28 14:33:05 xxhost kernel: instruction pointer = 0x20:0xffffffff81075393 Apr 28 14:33:05 xxhost kernel: stack pointer = 0x28:0xfffffe00486bf790 Apr 28 14:33:05 xxhost kernel: frame pointer = 0x28:0xfffffe00486bf790 Apr 28 14:33:05 xxhost kernel: code segment = base rx0, limit 0xfffff, type 0x1b Apr 28 14:33:05 xxhost kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Apr 28 14:33:05 xxhost kernel: processor eflags = interrupt enabled, resume, IOPL = 0 Apr 28 14:33:05 xxhost kernel: current process = 72368 (nfscl) Apr 28 14:33:05 xxhost kernel: rdi: fffff8002dbb9de4 rsi: 0000000000000000 rdx: 0000000000000010 Apr 28 14:33:05 xxhost kernel: rcx: 0000000000000000 r8: 000000000000002c r9: fffffe00486c0000 Apr 28 14:33:05 xxhost kernel: rax: 0000000000000000 rbx: fffff8002dbb9c00 rbp: fffffe00486bf790 Apr 28 14:33:05 xxhost kernel: r10: 0000000000000001 r11: 000000000000044c r12: fffff800056c1000 Apr 28 14:33:05 xxhost kernel: r13: fffff80012b06800 r14: fffffe00486bf9a8 r15: fffff80012b06818 Apr 28 14:33:05 xxhost kernel: trap number = 12 Apr 28 14:33:05 xxhost kernel: panic: page fault Apr 28 14:33:05 xxhost kernel: cpuid = 0 Apr 28 14:33:05 xxhost kernel: time = 1777348587 Apr 28 14:33:05 xxhost kernel: KDB: stack backtrace: Apr 28 14:33:05 xxhost kernel: #0 0xffffffff80bbe1ed at kdb_backtrace+0x5d Apr 28 14:33:05 xxhost kernel: #1 0xffffffff80b71576 at vpanic+0x136 Apr 28 14:33:05 xxhost kernel: #2 0xffffffff80b71433 at panic+0x43 Apr 28 14:33:05 xxhost kernel: #3 0xffffffff81079f69 at trap_pfault+0x3c9 Apr 28 14:33:05 xxhost kernel: #4 0xffffffff81050008 at calltrap+0x8 Apr 28 14:33:05 xxhost kernel: #5 0xffffffff80a0320d at newnfs_request+0xfbd Apr 28 14:33:05 xxhost kernel: #6 0xffffffff80a11aca at nfsrpc_destroysession+0x10a Apr 28 14:33:05 xxhost kernel: #7 0xffffffff80a05ed7 at nfsv4_setsequence+0x77 Apr 28 14:33:05 xxhost kernel: #8 0xffffffff80a054af at nfscl_reqstart+0x32f Apr 28 14:33:05 xxhost kernel: #9 0xffffffff80a383d7 at nfsrpc_renew+0x87 Apr 28 14:33:05 xxhost kernel: #10 0xffffffff80a1c6bf at nfscl_renewthread+0x76f Apr 28 14:33:05 xxhost kernel: #11 0xffffffff80a53e34 at start_nfscl+0x14 Apr 28 14:33:05 xxhost kernel: #12 0xffffffff80b2786b at fork_exit+0x7b Apr 28 14:33:05 xxhost kernel: #13 0xffffffff8105102e at fork_trampoline+0xe Apr 28 14:33:05 xxhost kernel: Uptime: 42d21h4m28s Apr 28 14:33:05 xxhost kernel: Automatic reboot in 15 seconds - press a key on the console to abort Apr 28 14:33:05 xxhost kernel: Rebooting... Apr 28 14:33:05 xxhost kernel: ---<<BOOT>>--- Apr 28 14:33:05 xxhost kernel: Copyright (c) 1992-2025 The FreeBSD Project. Apr 28 14:33:05 xxhost kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Apr 28 14:33:05 xxhost kernel: The Regents of the University of California. All rights reserved. Apr 28 14:33:05 xxhost kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Apr 28 14:33:05 xxhost kernel: FreeBSD 15.0-RELEASE-p6 GENERIC amd64
note 192.168.15/24 is a wireguard tunnel network, there is some unstabilities these days. Some /var/log/messages before panic: Apr 27 11:44:41 dog2 kernel: nfs server 192.168.15.57:/: not responding Apr 27 11:47:01 dog2 kernel: nfs server 192.168.15.57:/: is alive again Apr 27 16:58:32 dog2 kernel: nfs server 192.168.15.57:/: not responding Apr 27 17:00:33 dog2 kernel: nfs server 192.168.15.57:/: is alive again Apr 27 17:40:04 dog2 kernel: nfs server 192.168.15.57:/: not responding Apr 27 17:40:04 dog2 kernel: nfs server 192.168.15.57:/: is alive again Apr 27 17:52:04 dog2 kernel: nfs server 192.168.15.57:/: not responding mount cmd: mount -t nfs -o nfsv4,soft,intr,retrans=6,retrycnt=3,nosuid,noexec,nosymfollow,noatime,timeout=200 192.168.15.57:/ /mnt nfs server config: nfs_server_enable="YES" nfsv4_server_enable="YES" nfsv4_server_only="YES" nfs_server_flags="-t --minthreads 1 --maxthreads 4 -h 192.168.15.57" mountd_enable="YES" cat /etc/exports V4: /exports /exports -maproot=nobody:nobody -alldirs cat /etc/sysctl.conf | grep nfs vfs.nfsd.server_min_nfsvers=4 vfs.nfsd.server_min_minorversion4=2 #nfsv4 server vfs.nfs.enable_uidtostring=1 vfs.nfsd.enable_stringtouid=1
(In reply to Jov from comment #1) You cannot use "soft" or "intr" options on NFSv4 mounts. (See the BUGS section of mount_nfs(8).) Having said that, it shouldn't panic(). I will take a look at that. (Since "soft" and/or "intr" will corrupt the session, a crash in destroy session is probably a side effect of that.)
(In reply to Rick Macklem from comment #2) Hi Rick, Thanks for your work! I got a panic again on another host. I know soft/intr bugs for nfsv4.2, but I do not want to hang my client when the server can not be connected. May 24 15:46:29 cq6 syslogd: last message repeated 1 times May 24 15:46:29 cq6 kernel: Fatal trap 12: page fault while in kernel mode May 24 15:46:29 cq6 kernel: cpuid = 0; apic id = 00 May 24 15:46:29 cq6 kernel: fault virtual address = 0x0 May 24 15:46:29 cq6 kernel: fault code = supervisor read data, page not present May 24 15:46:29 cq6 kernel: instruction pointer = 0x20:0xffffffff81075393 May 24 15:46:29 cq6 kernel: stack pointer = 0x28:0xfffffe0053d3f790 May 24 15:46:29 cq6 kernel: frame pointer = 0x28:0xfffffe0053d3f790 May 24 15:46:29 cq6 kernel: code segment = base rx0, limit 0xfffff, type 0x1b May 24 15:46:29 cq6 kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 May 24 15:46:29 cq6 kernel: processor eflags = interrupt enabled, resume, IOPL = 0 May 24 15:46:29 cq6 kernel: current process = 99462 (nfscl) May 24 15:46:29 cq6 kernel: rdi: fffff8003d81e9e4 rsi: 0000000000000000 rdx: 0000000000000010 May 24 15:46:29 cq6 kernel: rcx: 0000000000000000 r8: 0000000000000025 r9: fffffe0053d40000 May 24 15:46:29 cq6 kernel: rax: 0000000000000000 rbx: fffff8003d81e800 rbp: fffffe0053d3f790 May 24 15:46:29 cq6 kernel: r10: 0000000000000001 r11: 000000000000044c r12: fffff8007a476780 May 24 15:46:29 cq6 kernel: r13: fffff8006114e000 r14: fffffe0053d3f9a8 r15: fffff8006114e018 May 24 15:46:29 cq6 kernel: trap number = 12 May 24 15:46:29 cq6 kernel: panic: page fault May 24 15:46:29 cq6 kernel: cpuid = 0 May 24 15:46:29 cq6 kernel: time = 1779608142 May 24 15:46:29 cq6 kernel: KDB: stack backtrace: May 24 15:46:29 cq6 kernel: #0 0xffffffff80bbe1ed at kdb_backtrace+0x5d May 24 15:46:29 cq6 kernel: #1 0xffffffff80b71576 at vpanic+0x136 May 24 15:46:29 cq6 kernel: #2 0xffffffff80b71433 at panic+0x43 May 24 15:46:29 cq6 kernel: #3 0xffffffff81079f69 at trap_pfault+0x3c9 May 24 15:46:29 cq6 kernel: #4 0xffffffff8104ff18 at calltrap+0x8 May 24 15:46:29 cq6 kernel: #5 0xffffffff80a0320d at newnfs_request+0xfbd May 24 15:46:29 cq6 kernel: #6 0xffffffff80a11aca at nfsrpc_destroysession+0x10a May 24 15:46:29 cq6 kernel: #7 0xffffffff80a05ed7 at nfsv4_setsequence+0x77 May 24 15:46:29 cq6 kernel: #8 0xffffffff80a054af at nfscl_reqstart+0x32f May 24 15:46:29 cq6 kernel: #9 0xffffffff80a383d7 at nfsrpc_renew+0x87 May 24 15:46:29 cq6 kernel: #10 0xffffffff80a1c6bf at nfscl_renewthread+0x76f May 24 15:46:29 cq6 kernel: #11 0xffffffff80a53e34 at start_nfscl+0x14 May 24 15:46:29 cq6 kernel: #12 0xffffffff80b2786b at fork_exit+0x7b May 24 15:46:29 cq6 kernel: #13 0xffffffff81050f3e at fork_trampoline+0xe May 24 15:46:29 cq6 kernel: Uptime: 148d19h12m31s May 24 15:46:29 cq6 kernel: Dumping 374 out of 2005 MB:..5%..13%..22%..35%..43%..52%..65%..73%..82%..94% May 24 15:46:29 cq6 kernel: Dump complete May 24 15:46:29 cq6 kernel: Automatic reboot in 15 seconds - press a key on the console to abort May 24 15:46:29 cq6 kernel: Rebooting... May 24 15:46:29 cq6 kernel: ---<<BOOT>>--- May 24 15:46:29 cq6 kernel: Copyright (c) 1992-2025 The FreeBSD Project.
(In reply to Jov from comment #3) Do you have a kernel.debug? (They are in somewhere like /usr/obj/usr/src/amd64.amd64/sys/GENERIC after you have built a kernel from sources.) If you haven't built a kernel from sources, but have the sources in /usr/src, you can just # cd /usr/src # make buildkernel to get that done. Then... # nm kernel.debug | fgrep newnfs_request - add 0xfbd to the value from above and then # addr2line -e kernel.debug 0x... (the value + 0xfbd) That's the only way I know of to get the source line# for newnfs_request+0xfbd, which is what I need to try and track this down. I know basically what is happening.. - Timeouts waiting for RPCs make session slots invalid - Once all are invalid, the code does a DestroySession to get rid of the, now broken, session. --> What I don't know is why it would crash in newnfs_request() when doing this.
(In reply to Rick Macklem from comment #4) I use 15.0-generic kernel,so I pkg install FreeBSD-kernel-generic-dbg-15.0p9 Please note that the paniced kernel was not patch level 9, so I do not konw if these info are useful. #nm kernel.debug | fgrep newnfs_request ffffffff80a02210 T newnfs_request # addr2line -e kernel.debug 0xffffffff80a031cd /usr/src/sys/fs/nfs/nfs_commonkrpc.c:1252 https://github.com/freebsd/freebsd-src/blob/releng/15.0/sys/fs/nfs/nfs_commonkrpc.c#L1252 if (bcmp(sep->nfsess_sessionid, nd->nd_sequence, NFSX_V4SESSIONID) == 0) { printf("Initiate recovery. If server " "has not rebooted, " "check NFS clients for unique " "/etc/hostid's\n");
Created attachment 271122 [details] nfscl: Fix crashes caused by wrong struct field being used Ouch! This is an embarrassing one. It should be comparing with nd_sessionid and not nd_sequence. (In this case, nd_sequence would be NULL.) When I looked, nd_sessionid wasn't even being set properly, so this code "worked" by random luck when nd_sequence was set to point to something. (I have tested recovery in the past, so it did work?) However, I only tested what you are seeing artificially, because I didn't have a real way to get the code to where you end up. Anyhow, this patch should fix it. I'll commit it right away and it should make it in 15.1 (Then I'll see about a 15.0 errata for it.) Thanks for reporting this and figuring out where it crashed in the sources. (I looked in this area when it was first reported, but didn't spot the typo bug.)
Oh, and since you now have built a kernel from sources, maybe you could apply the patch in the attachment, build it and run it, to see how it behaves in your setup. Just # cd /usr/src # patch -p0 < patch-in-attachment # make buildkernel # make installkernel (In case you haven't run a kernel built from sources.)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4d80d4913e79c8b5918b1f04c1c7b38e6c76b9b4 commit 4d80d4913e79c8b5918b1f04c1c7b38e6c76b9b4 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-05-25 19:22:32 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-05-25 19:22:32 +0000 nfs: Fix argument typo to avoid a crash A typo resulted in the wrong argument for a bytewise comparison that could result in a crash if the incorrect argument was not a valid pointer. This patch fixes the argument. While investigating this, I noticed that the correct argument was not being filled in as required, so this patch fixes that, as well. Somehow, recovery from a NFSv4.1/4.2 server crash worked during testing, so this was not detected. The bug/patch only affects NFS client mounts using NFSv4.1/4.2. PR: 294925 Reported by: Jov <amutu@amutu.com> MFC after: 3 days sys/fs/nfs/nfs_commonkrpc.c | 5 +++-- sys/fs/nfs/nfs_commonsubs.c | 3 +++ 2 files changed, 6 insertions(+), 2 deletions(-)
Patch has been committed and will be MFC'd.
Created attachment 271260 [details] nfs: Check for recovery in progress to avoid recovery flood This patch can be applied after attachment#271122 [details]. glebius@ reported that his NFS server reboot involved multiple recovery cycles. This patch only starts a recovery if recovery is not in progress. Hopefully that will reduce the likelyhood of multiple recovery cycles?
Created attachment 271262 [details] nfs: Check for recovery in progress to avoid recovery flood This patch can be applied after attachment#271122 [details]. glebius@ reported that his NFS server reboot involved multiple recovery cycles. This patch only starts a recovery if recovery is not in progress. Hopefully that will reduce the likelyhood of multiple recovery cycles?
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ea4886f2829bf33866c8c0c60b14a9641fc54b40 commit ea4886f2829bf33866c8c0c60b14a9641fc54b40 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-06-04 22:02:48 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-06-04 22:02:48 +0000 nfs_commonkrpc.c: Improve handling of NFSv4.1/4.2 recovery Commit 4d80d4913e79 fixed a long standing bug in the recovery code. However. glebius@ reported seeing multiple recovery cycles with this patch during an NFSv4.1/4.2 server reboot. This commit should minimize the risk of multiple recovery cycles. PR: 294925 Reported by: Jov <amutu@amutu.com> MFC after: 2 weeks Fixes: 4d80d4913e79 ("nfs: Fix argument typo to avoid a crash") sys/fs/nfs/nfs_commonkrpc.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=84043284ad1f6e7c0423555f28c9133efd37b6a8 commit 84043284ad1f6e7c0423555f28c9133efd37b6a8 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-05-25 19:22:32 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-06-26 01:28:58 +0000 nfs: Fix argument typo to avoid a crash A typo resulted in the wrong argument for a bytewise comparison that could result in a crash if the incorrect argument was not a valid pointer. This patch fixes the argument. While investigating this, I noticed that the correct argument was not being filled in as required, so this patch fixes that, as well. Somehow, recovery from a NFSv4.1/4.2 server crash worked during testing, so this was not detected. The bug/patch only affects NFS client mounts using NFSv4.1/4.2. PR: 294925 (cherry picked from commit 4d80d4913e79c8b5918b1f04c1c7b38e6c76b9b4) sys/fs/nfs/nfs_commonkrpc.c | 5 +++-- sys/fs/nfs/nfs_commonsubs.c | 3 +++ 2 files changed, 6 insertions(+), 2 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=28ae0d8491187bd7c4b244a68d79634dd8f9a698 commit 28ae0d8491187bd7c4b244a68d79634dd8f9a698 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-05-25 19:22:32 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-06-26 01:33:54 +0000 nfs: Fix argument typo to avoid a crash A typo resulted in the wrong argument for a bytewise comparison that could result in a crash if the incorrect argument was not a valid pointer. This patch fixes the argument. While investigating this, I noticed that the correct argument was not being filled in as required, so this patch fixes that, as well. Somehow, recovery from a NFSv4.1/4.2 server crash worked during testing, so this was not detected. The bug/patch only affects NFS client mounts using NFSv4.1/4.2. PR: 294925 (cherry picked from commit 4d80d4913e79c8b5918b1f04c1c7b38e6c76b9b4) sys/fs/nfs/nfs_commonkrpc.c | 5 +++-- sys/fs/nfs/nfs_commonsubs.c | 3 +++ 2 files changed, 6 insertions(+), 2 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=afc113696ff6652f132e2893e8efacb669860667 commit afc113696ff6652f132e2893e8efacb669860667 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-06-04 22:02:48 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-06-26 01:35:27 +0000 nfs_commonkrpc.c: Improve handling of NFSv4.1/4.2 recovery Commit 4d80d4913e79 fixed a long standing bug in the recovery code. However. glebius@ reported seeing multiple recovery cycles with this patch during an NFSv4.1/4.2 server reboot. This commit should minimize the risk of multiple recovery cycles. PR: 294925 (cherry picked from commit ea4886f2829bf33866c8c0c60b14a9641fc54b40) sys/fs/nfs/nfs_commonkrpc.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ae9f66486df64edcd0f2bef70afa67d245916835 commit ae9f66486df64edcd0f2bef70afa67d245916835 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-06-04 22:02:48 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-06-26 01:39:35 +0000 nfs_commonkrpc.c: Improve handling of NFSv4.1/4.2 recovery Commit 4d80d4913e79 fixed a long standing bug in the recovery code. However. glebius@ reported seeing multiple recovery cycles with this patch during an NFSv4.1/4.2 server reboot. This commit should minimize the risk of multiple recovery cycles. PR: 294925 (cherry picked from commit ea4886f2829bf33866c8c0c60b14a9641fc54b40) sys/fs/nfs/nfs_commonkrpc.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
Both patches have been MFC'd. If the reporter experiences the problem on a system with both patches applied, they can re-open the PR. (The reporter has not confirmed whether or not the patches fix their problem.)