On a server where user's $HOME is automounted through the following maps: /etc/auto_master: /home auto_home /etc/auto_home: * -nfsv4 server:/home/& trying to "svn checkout" a new working copy from any SVN server yields: joerg@daemon ~/tmp% svn co https://svn.freebsd.org/base svn: E000005: Can't check path '/home/.svn/wc.db': Input/output error The respective ktrace output is: 8779 svn NAMI "/home/joerg/.svn" 8779 svn RET fstatat -1 errno 2 No such file or directory 8779 svn CALL fstatat(AT_FDCWD,0x801abc0d0,0x7fffffffdfa0,0x200<AT_SYMLINK_NOFOLLOW>) 8779 svn NAMI "/home/.svn" 8779 svn STRU struct stat {dev=18446744072887533315, ino=4, mode=040755, nlink=3, uid=0, gid=0, rdev=0, atime=1579769649.589640074, mtime=1579769649.589640074, ctime=1579769649.589640074, birthtime=1579769649.589640074, size=512, blksize=4096, blocks=1, flags=0x0 } 8779 svn RET fstatat 0 8779 svn CALL fstatat(AT_FDCWD,0x8013e6d80,0x7fffffffdf30,0x200<AT_SYMLINK_NOFOLLOW>) 8779 svn NAMI "/home/.svn/wc.db" 8779 svn RET fstatat -1 errno 5 Input/output error 8779 svn CALL write(0x2,0x80137e1e0,0x46) 8779 svn GIO fd 2 wrote 70 bytes "svn: E000005: Can't check path '/home/.svn/wc.db': Input/output error " So SVN traverses directories upwards for .svn/wc.db entries. Surprisingly enough, stat(2) on /home/.svn does not yield the expected ENOENT but a valid result. (NB: /home/.svn does *not* exist on the NFS server.) However, trying to access wc.db under that ficticous directory then yields EIO. The automountd debug log looks like that: Jan 23 21:34:44 daemon automountd[8692]: map "auto_home" maps to "/etc/auto_home" Jan 23 21:34:44 daemon automountd[8692]: done parsing map "auto_home" Jan 23 21:34:44 daemon automountd[8692]: map may contain wildcard entries Jan 23 21:34:44 daemon automountd[8692]: found node defined at auto_home:1; it is a mountpoint Jan 23 21:34:44 daemon automountd[8692]: fstype not specified in options; defaulting to "nfs" Jan 23 21:34:44 daemon automountd[8692]: retrycnt not specified in options; defaulting to 1 Jan 23 21:34:44 daemon automountd[8692]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/joerg /home/joerg/" as pid 8693 Jan 23 21:34:44 daemon automountd[8692]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/joerg /home/joerg/", pid 8693, terminated gracefully Jan 23 21:34:44 daemon automountd[8692]: mount done; exiting Jan 23 21:34:44 daemon automountd[8692]: completing request 100 with error 0 Jan 23 21:34:44 daemon automountd[8675]: child process 8692 terminated gracefully Jan 23 21:34:44 daemon automountd[8675]: waiting for request from the kernel Jan 23 21:35:26 daemon automountd[8675]: got request; forking child process #0 Jan 23 21:35:26 daemon automountd[8675]: waiting for request from the kernel Jan 23 21:35:26 daemon automountd[8705]: got request 101: from map auto_home, path /home/.svn/, prefix "/home", key ".svn", options "" Jan 23 21:35:26 daemon automountd[8705]: parsing map "auto_home" Jan 23 21:35:26 daemon automountd[8705]: map "auto_home" maps to "/etc/auto_home" Jan 23 21:35:26 daemon automountd[8705]: done parsing map "auto_home" Jan 23 21:35:26 daemon automountd[8705]: map may contain wildcard entries Jan 23 21:35:26 daemon automountd[8705]: found node defined at auto_home:1; it is a mountpoint Jan 23 21:35:26 daemon automountd[8705]: fstype not specified in options; defaulting to "nfs" Jan 23 21:35:26 daemon automountd[8705]: retrycnt not specified in options; defaulting to 1 Jan 23 21:35:26 daemon automountd[8705]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/.svn /home/.svn/" as pid 8706 Jan 23 21:35:26 daemon automountd[8705]: completing request 101 with error 5 Jan 23 21:35:26 daemon automountd[8675]: child process 8705 terminated with exit status 1 Jan 23 21:35:26 daemon automountd[8675]: waiting for request from the kernel Jan 23 21:35:27 daemon automountd[8675]: got request; forking child process #0 Jan 23 21:35:27 daemon automountd[8675]: waiting for request from the kernel Jan 23 21:35:27 daemon automountd[8708]: got request 102: from map auto_home, path /home/.svn/, prefix "/home", key ".svn", options "" Jan 23 21:35:27 daemon automountd[8708]: parsing map "auto_home" Jan 23 21:35:27 daemon automountd[8708]: map "auto_home" maps to "/etc/auto_home" Jan 23 21:35:27 daemon automountd[8708]: done parsing map "auto_home" Jan 23 21:35:27 daemon automountd[8708]: map may contain wildcard entries Jan 23 21:35:27 daemon automountd[8708]: found node defined at auto_home:1; it is a mountpoint Jan 23 21:35:27 daemon automountd[8708]: fstype not specified in options; defaulting to "nfs" Jan 23 21:35:27 daemon automountd[8708]: retrycnt not specified in options; defaulting to 1 Jan 23 21:35:27 daemon automountd[8708]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/.svn /home/.svn/" as pid 8709 Jan 23 21:35:27 daemon automountd[8708]: completing request 102 with error 5 Jan 23 21:35:27 daemon automountd[8675]: child process 8708 terminated with exit status 1 Jan 23 21:35:27 daemon automountd[8675]: waiting for request from the kernel Jan 23 21:35:28 daemon automountd[8675]: got request; forking child process #0 Jan 23 21:35:28 daemon automountd[8675]: waiting for request from the kernel Jan 23 21:35:28 daemon automountd[8711]: got request 103: from map auto_home, path /home/.svn/, prefix "/home", key ".svn", options "" Jan 23 21:35:28 daemon automountd[8711]: parsing map "auto_home" Jan 23 21:35:28 daemon automountd[8711]: map "auto_home" maps to "/etc/auto_home" Jan 23 21:35:28 daemon automountd[8711]: done parsing map "auto_home" Jan 23 21:35:28 daemon automountd[8711]: map may contain wildcard entries Jan 23 21:35:28 daemon automountd[8711]: found node defined at auto_home:1; it is a mountpoint Jan 23 21:35:28 daemon automountd[8711]: fstype not specified in options; defaulting to "nfs" Jan 23 21:35:28 daemon automountd[8711]: retrycnt not specified in options; defaulting to 1 Jan 23 21:35:28 daemon automountd[8711]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/.svn /home/.svn/" as pid 8712 Jan 23 21:35:28 daemon automountd[8711]: completing request 103 with error 5 Jan 23 21:35:28 daemon automountd[8675]: child process 8711 terminated with exit status 1 Jan 23 21:35:28 daemon automountd[8675]: waiting for request from the kernel Me thinks stat()ing /home/.svn ought to immediately return ENOENT rather than pretending that directory might exist.
^Triage: to submitter: is this still relevant in the post-SVN world?
(In reply to Mark Linimon from comment #1) IMHO, SVN just uncovered a bug that is actually somewhere deeper in the system. So, in terms of SVN, this could probably be closed, but the bug would remain.
^Triage: incorporate feedback. I was only doing PR triage, so I won't be able to tackle this one myself, sorry.
Edward any suggestions on next steps for obtaining diagnostic information?
For a simple test, use something like this: #include <sys/stat.h> #include <stdio.h> int main(void) { struct stat sb; int rv = stat("/home/foobar", &sb); printf("stat() got %d\n", rv); return 0; } /home/foobar is supposed to be a directory that would never exist since there is no such user "foobar". After running the above on an automounted /home, /home/foobar has been created. Trying to access anything there yields Jun 23 21:03:51 daemon automountd[33777]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/foobar /home/foobar/", pid 33833, terminated with exit status 1 Jun 23 21:03:51 daemon kernel: WARNING: autofs_trigger_one: request for /home/foobar/ completed with error 5 Jun 23 21:03:51 daemon automountd[33777]: mount failed Jun 23 21:03:53 daemon automountd[34841]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/foobar /home/foobar/", pid 35155, terminated with exit status 1 Jun 23 21:03:53 daemon kernel: WARNING: autofs_trigger_one: request for /home/foobar/ completed with error 5 Jun 23 21:03:53 daemon automountd[34841]: mount failed Jun 23 21:03:54 daemon automountd[35511]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/foobar /home/foobar/", pid 35610, terminated with exit status 1 Jun 23 21:03:54 daemon kernel: WARNING: autofs_trigger_one: request for /home/foobar/ completed with error 5 Jun 23 21:03:54 daemon automountd[35511]: mount failed
I agree with Joerg's diagnosis: it appears automountd(8) creates the mountpoint even though it shouldn't, and then (correctly) fails to mount it. I'm not sure why it's doing that, though.
(In reply to Edward Tomasz Napierala from comment #6) Sure would be nice to have the errno from the failing mount program as it exited. I *think* this is what's happening: 1. automountd creates /home/foobar because it needs somewhere to mount the NFS filesystem. 2. automountd attempts to mount_nfs /home/foobar from the nfs server. 3. The nfs server replies with ENOENT (because /home/foobar doesn't exist on server). 4. the mount program fails and exits with status code 1. 5. automountd doesn't remove /home/foobar after the failed mount. One thought is; make a special case for NFS that checks if the target directory is exported from the NFS server before attempting to mount it. Not sure how feasible this idea is, I'd have to look into it..or maybe one of you know off the top of your head? I imagine an issue with doing a special case for NFS is that other automounted filesystems may have similar problems as well (but in a different way). Still, given that NFS seems to be the largest use-case for automount, it may be worth making a special case for NFS, even if the problem remains for other automounted filesystems. This is why I mentioned it would be nice to have the errno from the failing mount program as it *might* work as general solution for other automounted filesystems.
We already have a mechanism for this kind of check: special maps. Kind of how special_hosts works, but looking up usernames instead of hosts. I've done a quick test, and it appears that "cd /net/nonexistent" does not result in that directory being created. So, special case could consist of a new map, say, special_users, which would first verify the username using getent(1). Another thing we could do is to remove the (newly created) mountpoint if mount fails, before releasing the process which triggered the mount.
(In reply to Edward Tomasz Napierala from comment #8) To me, the latter (remove the temporary mountpoint before proceeding) sounds like the right way to go.
When using a direct map with a wildcard entry and doing "cd /net/nonexistent", a new directory will be created. W.R.T. a special_users map, just because the user exists on the system doesn't necessarily mean that the users home directory is being exported via NFS, no? The temporary mountpoint should definitely be cleaned up on failure. I suppose this cleanup should happen after autofs has exhausted its retry attempts? Do you think autofs or automountd (via request from autofs) should do the clean up? Quick glance through the code; looks like it would be a bit of work to make the mount_nfs bits public that would allow automountd to retrieve the error code that was returned from the NFS server.. There's not a reliable way to discover exported directories for NFSv3/NFSv4? I think in a perfect world, the program flow would be: 1. request to automountd to mount from a NFS server 2. automountd checks if that directory is exported from the NFS server 3. if yes, create the temporary mountpoint 4. attempt the mount 5. if error, remove temporary mountpoint and return the error code to autofs 6. let autofs, based on the error code, decide if it wants to retry to mount. But, it's not a perfect world, is it?