Created attachment 273424 [details] patch Solving AWS EFS mount problem with my EC2 instances, with the help of claude code. An NFSv4.1 mount can wedge permanently: every process touching it blocks in uninterruptible D state (WCHAN "nfsbadse") and only a reboot recovers. It is triggered when the server returns NFS4ERR_BADSESSION on SEQUENCE while the ClientID is also stale (NFS4ERR_STALE_CLIENTID). Observed against Amazon EFS, and reproduced without EFS against a plain local nfsd (see below). Root cause Two guards that each prevent a recovery storm together form a trap with no exit: 1. sys/fs/nfs/nfs_commonkrpc.c, BADSESSION handler in newnfs_request(): recovery is only (re)initiated when the live MDS session has nfsess_defunct == 0. Once the live session is already defunct, every further SEQUENCE->NFS4ERR_BADSESSION is silently swallowed and recovery is never triggered again. 2. sys/fs/nfs/nfs_commonsubs.c, nfsv4_sequencelookup(): returns NFS4ERR_BADSESSION for any op on a defunct session, so foreground ops spin the 1s "Badsession looping" retry forever. How the live session is left defunct with no replacement: The recover_done_time gate in nfscl_renewthread() (sys/fs/nfsclient/nfs_clstate.c) allows only one full recovery per nfsc_renew (lease/2). A second BADSESSION within that window forces nfscl_recover() to call nfsrpc_setclient() with retok=true, i.e. CreateSession with the extant ClientID only. Against a stale ClientID that CreateSession fails, nfsrpc_setclient() returns NFSERR_IO (sys/fs/nfsclient/nfs_clrpcops.c), which nfscl_recover() does not retry. But the BADSESSION handler had already marked the live session defunct, so it stays at the head of nm_sess with no non-defunct replacement, and guard 1 then ensures recovery is never attempted again. Evidence (live EFS capture, vfs.nfs.debuglevel=1): Recovery fires and succeeds 4x via the ExchangeID+CreateSession fallback, then after two recoveries 45s apart the live MDS session is defunct with no replacement; from there the renew thread hits SEQUENCE->NFS4ERR_BADSESSION every ~45s for 2h17m with zero "Initiate recovery", then foreground access spins "Badsession looping" at 1/s until reboot. nfsstat -c freezes (ExchangeId/CreateSess pinned). Reproduce without EFS (plain local nfsd + a fault-injecting TCP proxy that returns NFS4ERR_BADSESSION on SEQUENCE and NFS4ERR_STALE_CLIENTID on CREATE_SESSION): sysctl vfs.nfsd.server_max_minorversion4=2 vfs.nfsd.nfs_privport=0 sysctl vfs.nfs.debuglevel=1 # export /export over "V4: /export"; run proxy on :2050 -> :2049, arm 4s/15s mount_nfs -o nfsv4,minorversion=1,hard,retrans=2,port=2050 127.0.0.1:/ /mnt/t # drive concurrent ls/stat on /mnt/t in a loop The unpatched client wedges in ~3 min (frozen nfsstat, "Badsession looping" at 1/s with no "Initiate recovery", D-state "nfsbadse"). Fix: In the BADSESSION handler in newnfs_request(), drop the nfsess_defunct == 0 term from the re-trigger condition. The NFSCLFLAGS_RECVRINPROG | NFSCLFLAGS_RECOVER check already present is what prevents a recovery storm; the nfsess_defunct == 0 test is redundant for that purpose and is what makes the wedge permanent. With it removed, a BADSESSION on an already-defunct live session re-arms NFSCLFLAGS_RECOVER. Setting nfsess_defunct = 1 is idempotent. This self-heals: CreateSession-only retries repeat one BADSESSION apart until the nfsc_renew window rolls over, after which a full ExchangeID+CreateSession recovery succeeds.
Ouch!!, my bad. I added that check in 4d80d49 just to reduce the likelyhood of the message being spewed out multiple times, which I then handled differently in ea4886f anyhow. The fix looks fine and really just reverts a bogus part of 4d80d49. You can commit this reviewed by rmacklem. Thanks, rick ps: I can commit it, if for some reason you would prefer that?
(In reply to Rick Macklem from comment #1) Yes I prefer you commit it: This patch was fully written by Claude code, so I trust you a lot more :-)
(In reply to Olivier Cochard from comment #2) Ok. It's really just a one line change that reverts a little part of the commit that I broke it with. Maybe I'll trim back the comment, if that's ok with you?
(In reply to Rick Macklem from comment #3) Sure, the comment was a lot too long anyway.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=5ab48fb9f7ac43fb1242a678312a02df5d4d5b53 commit 5ab48fb9f7ac43fb1242a678312a02df5d4d5b53 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-08-03 23:57:38 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-08-04 00:02:47 +0000 nfs_commonkrpc.c: Fix recovery that was broken by 4d80d4913e79 Commit 4d80d4913e79 added a check for nfsess_defunct already being set. This was incorrect because, once set, nfsess_defunct remains set and an additional recovery might be needed. This patch reverts this part of 4d80d4913e79. PR: 297252 Suggested by: olivier MFC after: 3 days Fixes: 4d80d4913e79 ("nfs: Fix argument typo to avoid a crash") sys/fs/nfs/nfs_commonkrpc.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
A variant of the patch has been committed and will be MFC'd.
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=037232c2001528995d75556c8e2007c686e50134 commit 037232c2001528995d75556c8e2007c686e50134 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-08-03 23:57:38 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-08-06 19:45:02 +0000 nfs_commonkrpc.c: Fix recovery that was broken by 4d80d4913e79 Commit 4d80d4913e79 added a check for nfsess_defunct already being set. This was incorrect because, once set, nfsess_defunct remains set and an additional recovery might be needed. This patch reverts this part of 4d80d4913e79. PR: 297252 (cherry picked from commit 5ab48fb9f7ac43fb1242a678312a02df5d4d5b53) sys/fs/nfs/nfs_commonkrpc.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ca0dcfd1a8ecbd7f4f99b06adcb5d67c8742679c commit ca0dcfd1a8ecbd7f4f99b06adcb5d67c8742679c Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2026-08-03 23:57:38 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2026-08-06 19:51:27 +0000 nfs_commonkrpc.c: Fix recovery that was broken by 4d80d4913e79 Commit 4d80d4913e79 added a check for nfsess_defunct already being set. This was incorrect because, once set, nfsess_defunct remains set and an additional recovery might be needed. This patch reverts this part of 4d80d4913e79. PR: 297252 (cherry picked from commit 5ab48fb9f7ac43fb1242a678312a02df5d4d5b53) sys/fs/nfs/nfs_commonkrpc.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
The patch has been committed and MFC'd.