Created attachment 273400 [details] crashinfo output Now that bug #296224 is fixed, I tried again doing a "make -j12 buildworld" over NFS. The system panicked different this time: Fatal trap 9: general protection fault while in kernel mode cpuid = 4; apic id = 04 instruction pointer = 0x20:0xffffffff80ac1caa stack pointer = 0x28:0xfffffe0197b1ef20 frame pointer = 0x28:0xfffffe0197b1efc0 processor eflags = interrupt enabled, resume, IOPL = 0 current thread = 34908/101210 (sh/sh) rdi: fffff80177aff340 rsi: 0000000000000002 rdx: ffffffff81e585b0 rcx: 0000000000000001 r8: ffffffff81ad3a20 r9: 0000000000000002 rax: 0000000000000000 rbx: deadc0dedeadc0de rbp: fffffe0197b1efc0 r10: 0000000000000000 r11: 0000000000000001 r12: 0000000000000000 r13: deadc0dedeadc0de r14: deadc0dedeadc0de r15: fffff8010648c497 panic: general protection fault cpuid = 4 time = 1785657295 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe0197b1eca0 vpanic() at vpanic+0x149/frame 0xfffffe0197b1edd0 panic() at panic+0x43/frame 0xfffffe0197b1ee30 trap_fatal() at trap_fatal+0x6f/frame 0xfffffe0197b1ee50 calltrap() at calltrap+0x8/frame 0xfffffe0197b1ee50 --- trap 0x9, rip = 0xffffffff80ac1caa, rsp = 0xfffffe0197b1ef20, rbp = 0xfffffe0197b1efc0 --- nfscl_expireclient() at nfscl_expireclient+0xcda/frame 0xfffffe0197b1efc0 nfscl_hasexpired() at nfscl_hasexpired+0x293/frame 0xfffffe0197b1f010 nfsrpc_read() at nfsrpc_read+0x69e/frame 0xfffffe0197b1f210 ncl_readrpc() at ncl_readrpc+0xc8/frame 0xfffffe0197b1f330 ncl_doio() at ncl_doio+0xec/frame 0xfffffe0197b1f3d0 nfs_strategy() at nfs_strategy+0xb9/frame 0xfffffe0197b1f400 vop_sigdefer() at vop_sigdefer+0x30/frame 0xfffffe0197b1f430 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0x5d/frame 0xfffffe0197b1f460 bufstrategy() at bufstrategy+0x4e/frame 0xfffffe0197b1f4a0 breadn_flags() at breadn_flags+0x1b5/frame 0xfffffe0197b1f520 vfs_bio_getpages() at vfs_bio_getpages+0x339/frame 0xfffffe0197b1f610 ncl_getpages() at ncl_getpages+0x1b0/frame 0xfffffe0197b1f6d0 vop_sigdefer() at vop_sigdefer+0x30/frame 0xfffffe0197b1f700 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x51/frame 0xfffffe0197b1f730 vnode_pager_getpages() at vnode_pager_getpages+0x42/frame 0xfffffe0197b1f780 vm_pager_get_pages() at vm_pager_get_pages+0x59/frame 0xfffffe0197b1f7d0 vm_page_grab_valid_iter() at vm_page_grab_valid_iter+0x503/frame 0xfffffe0197b1f930 vm_page_grab_valid_unlocked() at vm_page_grab_valid_unlocked+0xef/frame 0xfffffe0197b1f990 exec_map_first_page() at exec_map_first_page+0xab/frame 0xfffffe0197b1f9d0 kern_execve() at kern_execve+0x78f/frame 0xfffffe0197b1fd70 sys_execve() at sys_execve+0x5f/frame 0xfffffe0197b1fdf0 amd64_syscall() at amd64_syscall+0x17c/frame 0xfffffe0197b1ff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0197b1ff30 --- syscall (59, FreeBSD ELF64, execve), rip = 0x1ee27e4a52a, rsp = 0x1ee2621d3f8, rbp = 0x1ee2621d540 --- The NFS server is running illumos. Here's the output of nfsstat -m: # nfsstat -m blackbird:/export on /net/blackbird/export nfsv4,minorversion=0,tcp,resvport,nconnect=1,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2147483647 # ps ax | fgrep nfscbd 6485 - Is 0:00.00 nfscbd: master (nfscbd) 6486 - S 0:00.03 nfscbd: server (nfscbd)
Since you are using minorversion=0, try a mount without nfscbd running. (Callbacks and delegations are never going to work correctly for NFSv4.0.) The expired case should never really happen. It indicates that there was a network patitioning between client and server that exceeded the lease duration (or the Illumos server is broken and replied NFS4ERR_EXPIRED when it shouldn't have). The code in nfscl_hasexpired() is 20years old and never gets exercised. The crash appears to be a messed up linked list for Opens, but I wouldn't dare to guess why that might happen. If the crash happens again when nfscbd(8) is not running (and not running when the mount is done), please add a comment here.
I disabled nfscbd and rebooted, then tried again. The results look a bit similar to bug #296224: Memory modified after free 0xfffff80b37ec8180 (384, malloc-384, 0xffffffff81aaef30) + 72 = deadc0dedeadc0dd panic: Memory modified after free 0xfffff80b37ec8180 (384, malloc-384, NFSCL deleg) + 72 = deadc0dedeadc0dd cpuid = 2 time = 1785750276 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe01f7668f90 vpanic() at vpanic+0x149/frame 0xfffffe01f76690c0 panic() at panic+0x43/frame 0xfffffe01f7669120 mtrash_ctor() at mtrash_ctor+0x147/frame 0xfffffe01f7669160 item_ctor() at item_ctor+0x12c/frame 0xfffffe01f76691b0 malloc() at malloc+0x8f/frame 0xfffffe01f7669200 nfsrpc_openrpc() at nfsrpc_openrpc+0x4db/frame 0xfffffe01f76694b0 nfsrpc_openrpc() at nfsrpc_openrpc+0x979/frame 0xfffffe01f76697c0 nfsrpc_open() at nfsrpc_open+0x29c/frame 0xfffffe01f76698d0 nfs_open() at nfs_open+0x75/frame 0xfffffe01f76699e0 vop_sigdefer() at vop_sigdefer+0x30/frame 0xfffffe01f7669a10 VOP_OPEN_APV() at VOP_OPEN_APV+0x52/frame 0xfffffe01f7669a40 vn_open_vnode() at vn_open_vnode+0x1ae/frame 0xfffffe01f7669ac0 vn_open_cred() at vn_open_cred+0x6e3/frame 0xfffffe01f7669c50 openatfp() at openatfp+0x2ed/frame 0xfffffe01f7669dc0 sys_openat() at sys_openat+0x3d/frame 0xfffffe01f7669df0 amd64_syscall() at amd64_syscall+0x17c/frame 0xfffffe01f7669f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01f7669f30 --- syscall (499, FreeBSD ELF64, openat), rip = 0x1e268cc9a42a, rsp = 0x1e268006ea98, rbp = 0x1e268006ead0 ---
Hmm. This Illumos server appears to be broken. When there is no working callback path, the server is not supposed to issue delegations, but it appears to have done so. Does your kernel have commit 71ac1ec5c9? (It was committed to main on Apr. 30, 2026, so if your kernel is built from sources after that date, it does.) --> This patch was meant to ensure that NFSv4.0 servers did not issue delegations, by specifying a callback path address of 0.0.0.0. If you have this patch, then the NFSv4.0 server definitely will not be able to do callbacks. If the Illumos 4.0 (not 4.1 or 4.2) server is issuing delegations in this case, I can't think of a way to fix things. (Maybe Illumos has some "turn off delegations" switch for their server?) If you cannot figure out how to disable issuance of delegations for the Illumos server, you could try reverting 71ac1ec5c9 (or using a different server that supports NFSv4.1).
Or drop back to NFSv3 mounts.
Oh, and if you choose to revert 71ac1ec5c9, re-enable nfscbd as well.
Regardless of the protocol violations or unsupported features, crashing the kernel, esp. with page fault or general protection fault, should not happen. Do I miss something?
(In reply to Konstantin Belousov from comment #6) I will take a look someday, but the honest answer is that a broken server will always be able to find ways to break the client and not mounting that server is the only fix, although it would be nice if a server cannot crash the client system. (This case is an extremely rare "should never happen" one, where the client loses its lease and tries to recover the opens. It cannot recover the locks. This recovery isn't even covered in the RFCs, because there is no good solution. Maybe the code should have just forced a dismount for this case?) But I will look at it and try and figure out where the delegation gets free'd prematurely, but my "to do" list is pretty long these days.
My FreeBSD test system has commit 71ac1ec5c9, I'll try reverting it to see whether this changes things. That being said, if you have more information on how exactly you believe the illumos NFS server to be broken, I'd be happy to help you file an illumos bug or file one for you myself. But since my expertise on NFS is quite limited I'll need a pretty good explanation of the issue. :-) Regarding the FreeBSD panic I'm firmly with Konstantin Belousov here: FreeBSD should not crash in the NFS client code regardless what any random broken NFS server does or does not.
With 71ac1ec5c9 reverted, it happened again with a different stack: Memory modified after free 0xfffff801066bf900 (384, malloc-384, 0xffffffff81aaefa0) + 72 = deadc0dedeadc0dd panic: Memory modified after free 0xfffff801066bf900 (384, malloc-384, NFSCL deleg) + 72 = deadc0dedeadc0dd cpuid = 5 time = 1785776804 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe0197a33f90 vpanic() at vpanic+0x149/frame 0xfffffe0197a340c0 panic() at panic+0x43/frame 0xfffffe0197a34120 mtrash_ctor() at mtrash_ctor+0x147/frame 0xfffffe0197a34160 item_ctor() at item_ctor+0x12c/frame 0xfffffe0197a341b0 malloc() at malloc+0x8f/frame 0xfffffe0197a34200 nfsrpc_openrpc() at nfsrpc_openrpc+0x4db/frame 0xfffffe0197a344b0 nfsrpc_openrpc() at nfsrpc_openrpc+0x979/frame 0xfffffe0197a347c0 nfsrpc_open() at nfsrpc_open+0x29c/frame 0xfffffe0197a348d0 nfs_open() at nfs_open+0x75/frame 0xfffffe0197a349e0 vop_sigdefer() at vop_sigdefer+0x30/frame 0xfffffe0197a34a10 VOP_OPEN_APV() at VOP_OPEN_APV+0x52/frame 0xfffffe0197a34a40 vn_open_vnode() at vn_open_vnode+0x1ae/frame 0xfffffe0197a34ac0 vn_open_cred() at vn_open_cred+0x6e3/frame 0xfffffe0197a34c50 openatfp() at openatfp+0x2ed/frame 0xfffffe0197a34dc0 sys_openat() at sys_openat+0x3d/frame 0xfffffe0197a34df0 amd64_syscall() at amd64_syscall+0x17c/frame 0xfffffe0197a34f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0197a34f30 --- syscall (499, FreeBSD ELF64, openat), rip = 0x61f6ac7f42a, rsp = 0x61f55c92c48, rbp = 0x61f55c92c80 ---
(In reply to Hans Rosenfeld from comment #8) > That being said, if you have more information on how exactly you believe the > illumos NFS server to be broken, I'd be happy to help you file an illumos > bug or file one for you myself. But since my expertise on NFS is quite limited > I'll need a pretty good explanation of the issue. :-) It appears that the Illumos server is issuing delegations when the callback path is not established. This should never happen. (You can check to be sure by capturing packets for one of these mounts and looking at the Open replies in wireshark. If any of them have a delegation, the server is broken.) Now, having said the above, I don't see much point in your doing so. NFSv4.0 is deprecated (the Linux folk are discussing disabling it and may have already done so). For example, here's what Netapp says on their web site: BEST PRACTICE: NFSv4.1 or NFSv4.2 should be used instead of NFSv4.0. NFSv4.0 is an older version of the NFS protocol and is no longer actively developed by the NFS community. (You can google "NFSv4.0 is deprecated" and get lots more like this.) > Regarding the FreeBSD panic I'm firmly with Konstantin Belousov here: > FreeBSD should not crash in the NFS client code regardless what any random > broken NFS server does or does not. Don't bet on it. It there was an engineering team working on the FreeBSD NFS client, maybe. But, basically all you've got is one old retired guy who doesn't get paid any $$$ for doing NFS/FreeBSD work. (The code the first crash is in was written over 20years ago and this is the first time I've seen a report of a crash in it. I doubt I'll spot where the prematuure free() is, but I will take a look one of these days.) I'll also note that I think "Affects many people" is an exaggeration, given that I've never seen a crash report from anyone else. If you really want to track this down, you can comment out "free()" calls until the crashes go away and then bisect down to the "free()" that causes the crash, but that is going to be time consuming.
A faster way to figure out where the use-after-free is happening would be to try reproducing the crash with a GENERIC-KASAN kernel. kasan(9) will trigger a panic when the use-after-free actually occurs, rather than when the affected object is recycled by the kernel memory allocator.
Created attachment 273448 [details] nfs_clrpcops.c: Comment out an oddball case of nfsrpc_openrpc() recursion You could try this little patch. I just noticed that, for your crashes except the first one, the code is calling nfsrpc_openrpc() recursively. This code path will not have been exercised much (if ever) and really is cruft that I should remove from the sources even if it doesn't help w.r.t. these crashes. (The patch just comments out the code.)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8f20299b473af6132e0f146d7f634640993aeb81 commit 8f20299b473af6132e0f146d7f634640993aeb81 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-08-04 15:55:24 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-08-04 15:55:24 +0000 nfs_commonkrpc.c: Get rid of NFSv4.0 delegation cruft Delegations in NFSv4.0 never worked well and, since the NFSv4.0 protocol is now deprecated, use of delegations for NFSv4.0 is disabled as far as the client can do so. It turns out that some Illumos NFSv4.0 server issues delegations anyhow (even when the callback path is specified as 0.0.0.0) and this can cause use after free problems. This patch deleted some cruft that did an nfsrpc_openrpc() call recursively when an NFSv4.0 server failed to issue a delegation when it had previously done so. This code was only meant to be an optimization and would have been rarely exercised. Since this recursive call of nfsrpc_openrpc() is in some of the backtraces in the bugzilla PR, getting rid of the cruft makes sense. It is not known if this helps w.r.t. the use after free problems at this time. PR: 297233 MFC after: 3 days sys/fs/nfsclient/nfs_clrpcops.c | 27 --------------------------- 1 file changed, 27 deletions(-)
As far as I can tell your patch fixes this problem. The build used to fail within 5 minutes, now it's been building on NFS for over an hour. Thank you!
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6154eec89acb3aa6561f2e8ba365115a8056591d commit 6154eec89acb3aa6561f2e8ba365115a8056591d Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-08-04 15:55:24 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-08-06 19:46:22 +0000 nfs_commonkrpc.c: Get rid of NFSv4.0 delegation cruft Delegations in NFSv4.0 never worked well and, since the NFSv4.0 protocol is now deprecated, use of delegations for NFSv4.0 is disabled as far as the client can do so. It turns out that some Illumos NFSv4.0 server issues delegations anyhow (even when the callback path is specified as 0.0.0.0) and this can cause use after free problems. This patch deleted some cruft that did an nfsrpc_openrpc() call recursively when an NFSv4.0 server failed to issue a delegation when it had previously done so. This code was only meant to be an optimization and would have been rarely exercised. Since this recursive call of nfsrpc_openrpc() is in some of the backtraces in the bugzilla PR, getting rid of the cruft makes sense. It is not known if this helps w.r.t. the use after free problems at this time. PR: 297233 (cherry picked from commit 8f20299b473af6132e0f146d7f634640993aeb81) sys/fs/nfsclient/nfs_clrpcops.c | 27 --------------------------- 1 file changed, 27 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=bdea6e971844ac2e0a3dfab47fc6e678a11e5d19 commit bdea6e971844ac2e0a3dfab47fc6e678a11e5d19 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-08-04 15:55:24 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-08-06 19:52:36 +0000 nfs_commonkrpc.c: Get rid of NFSv4.0 delegation cruft Delegations in NFSv4.0 never worked well and, since the NFSv4.0 protocol is now deprecated, use of delegations for NFSv4.0 is disabled as far as the client can do so. It turns out that some Illumos NFSv4.0 server issues delegations anyhow (even when the callback path is specified as 0.0.0.0) and this can cause use after free problems. This patch deleted some cruft that did an nfsrpc_openrpc() call recursively when an NFSv4.0 server failed to issue a delegation when it had previously done so. This code was only meant to be an optimization and would have been rarely exercised. Since this recursive call of nfsrpc_openrpc() is in some of the backtraces in the bugzilla PR, getting rid of the cruft makes sense. It is not known if this helps w.r.t. the use after free problems at this time. PR: 297233 (cherry picked from commit 8f20299b473af6132e0f146d7f634640993aeb81) sys/fs/nfsclient/nfs_clrpcops.c | 27 --------------------------- 1 file changed, 27 deletions(-)
The patch that gets rid of the cruft is now committed and MFC'd. I will leave this open until the reporter confirms whether or not the crashes still occur.
As far as I can tell the crashes no longer happen with this change.
(In reply to Hans Rosenfeld from comment #18) Ok. So are you comfortable with the bug report being closed, given that the patch is committed and MFC'd?
(In reply to Rick Macklem from comment #19) Yes :)
The patch that is now committed to main and MFC'd to stable/15 and stable/14 seems to have stopped the crashes.