Bug 243551 - Cannot checkout src tree in automounted $HOME
Summary: Cannot checkout src tree in automounted $HOME
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-23 20:47 UTC by Joerg Wunsch
Modified: 2021-07-18 20:01 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joerg Wunsch freebsd_committer 2020-01-23 20:47:38 UTC
On a server where user's $HOME is automounted through the following maps:

/etc/auto_master:

/home           auto_home

/etc/auto_home:

*       -nfsv4  server:/home/&

trying to "svn checkout" a new working copy from any SVN server yields:

joerg@daemon ~/tmp% svn co https://svn.freebsd.org/base
svn: E000005: Can't check path '/home/.svn/wc.db': Input/output error

The respective ktrace output is:

  8779 svn      NAMI  "/home/joerg/.svn"
  8779 svn      RET   fstatat -1 errno 2 No such file or directory
  8779 svn      CALL  fstatat(AT_FDCWD,0x801abc0d0,0x7fffffffdfa0,0x200<AT_SYMLINK_NOFOLLOW>)
  8779 svn      NAMI  "/home/.svn"
  8779 svn      STRU  struct stat {dev=18446744072887533315, ino=4, mode=040755, nlink=3, uid=0, gid=0, rdev=0, atime=1579769649.589640074, mtime=1579769649.589640074, ctime=1579769649.589640074, birthtime=1579769649.589640074, size=512, blksize=4096, blocks=1, flags=0x0 }
  8779 svn      RET   fstatat 0
  8779 svn      CALL  fstatat(AT_FDCWD,0x8013e6d80,0x7fffffffdf30,0x200<AT_SYMLINK_NOFOLLOW>)
  8779 svn      NAMI  "/home/.svn/wc.db"
  8779 svn      RET   fstatat -1 errno 5 Input/output error
  8779 svn      CALL  write(0x2,0x80137e1e0,0x46)
  8779 svn      GIO   fd 2 wrote 70 bytes
       "svn: E000005: Can't check path '/home/.svn/wc.db': Input/output error
       "

So SVN traverses directories upwards for .svn/wc.db entries. Surprisingly enough, stat(2) on /home/.svn does not yield the expected ENOENT but a valid result. (NB: /home/.svn does *not* exist on the NFS server.) However, trying to access wc.db under that ficticous directory then yields EIO.

The automountd debug log looks like that:

Jan 23 21:34:44 daemon automountd[8692]: map "auto_home" maps to "/etc/auto_home"
Jan 23 21:34:44 daemon automountd[8692]: done parsing map "auto_home"
Jan 23 21:34:44 daemon automountd[8692]: map may contain wildcard entries
Jan 23 21:34:44 daemon automountd[8692]: found node defined at auto_home:1; it is a mountpoint
Jan 23 21:34:44 daemon automountd[8692]: fstype not specified in options; defaulting to "nfs"
Jan 23 21:34:44 daemon automountd[8692]: retrycnt not specified in options; defaulting to 1
Jan 23 21:34:44 daemon automountd[8692]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/joerg /home/joerg/" as pid 8693
Jan 23 21:34:44 daemon automountd[8692]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/joerg /home/joerg/", pid 8693, terminated gracefully
Jan 23 21:34:44 daemon automountd[8692]: mount done; exiting
Jan 23 21:34:44 daemon automountd[8692]: completing request 100 with error 0
Jan 23 21:34:44 daemon automountd[8675]: child process 8692 terminated gracefully
Jan 23 21:34:44 daemon automountd[8675]: waiting for request from the kernel
Jan 23 21:35:26 daemon automountd[8675]: got request; forking child process #0
Jan 23 21:35:26 daemon automountd[8675]: waiting for request from the kernel
Jan 23 21:35:26 daemon automountd[8705]: got request 101: from map auto_home, path /home/.svn/, prefix "/home", key ".svn", options ""
Jan 23 21:35:26 daemon automountd[8705]: parsing map "auto_home"
Jan 23 21:35:26 daemon automountd[8705]: map "auto_home" maps to "/etc/auto_home"
Jan 23 21:35:26 daemon automountd[8705]: done parsing map "auto_home"
Jan 23 21:35:26 daemon automountd[8705]: map may contain wildcard entries
Jan 23 21:35:26 daemon automountd[8705]: found node defined at auto_home:1; it is a mountpoint
Jan 23 21:35:26 daemon automountd[8705]: fstype not specified in options; defaulting to "nfs"
Jan 23 21:35:26 daemon automountd[8705]: retrycnt not specified in options; defaulting to 1
Jan 23 21:35:26 daemon automountd[8705]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/.svn /home/.svn/" as pid 8706
Jan 23 21:35:26 daemon automountd[8705]: completing request 101 with error 5
Jan 23 21:35:26 daemon automountd[8675]: child process 8705 terminated with exit status 1
Jan 23 21:35:26 daemon automountd[8675]: waiting for request from the kernel
Jan 23 21:35:27 daemon automountd[8675]: got request; forking child process #0
Jan 23 21:35:27 daemon automountd[8675]: waiting for request from the kernel
Jan 23 21:35:27 daemon automountd[8708]: got request 102: from map auto_home, path /home/.svn/, prefix "/home", key ".svn", options ""
Jan 23 21:35:27 daemon automountd[8708]: parsing map "auto_home"
Jan 23 21:35:27 daemon automountd[8708]: map "auto_home" maps to "/etc/auto_home"
Jan 23 21:35:27 daemon automountd[8708]: done parsing map "auto_home"
Jan 23 21:35:27 daemon automountd[8708]: map may contain wildcard entries
Jan 23 21:35:27 daemon automountd[8708]: found node defined at auto_home:1; it is a mountpoint
Jan 23 21:35:27 daemon automountd[8708]: fstype not specified in options; defaulting to "nfs"
Jan 23 21:35:27 daemon automountd[8708]: retrycnt not specified in options; defaulting to 1
Jan 23 21:35:27 daemon automountd[8708]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/.svn /home/.svn/" as pid 8709
Jan 23 21:35:27 daemon automountd[8708]: completing request 102 with error 5
Jan 23 21:35:27 daemon automountd[8675]: child process 8708 terminated with exit status 1
Jan 23 21:35:27 daemon automountd[8675]: waiting for request from the kernel
Jan 23 21:35:28 daemon automountd[8675]: got request; forking child process #0
Jan 23 21:35:28 daemon automountd[8675]: waiting for request from the kernel
Jan 23 21:35:28 daemon automountd[8711]: got request 103: from map auto_home, path /home/.svn/, prefix "/home", key ".svn", options ""
Jan 23 21:35:28 daemon automountd[8711]: parsing map "auto_home"
Jan 23 21:35:28 daemon automountd[8711]: map "auto_home" maps to "/etc/auto_home"
Jan 23 21:35:28 daemon automountd[8711]: done parsing map "auto_home"
Jan 23 21:35:28 daemon automountd[8711]: map may contain wildcard entries
Jan 23 21:35:28 daemon automountd[8711]: found node defined at auto_home:1; it is a mountpoint
Jan 23 21:35:28 daemon automountd[8711]: fstype not specified in options; defaulting to "nfs"
Jan 23 21:35:28 daemon automountd[8711]: retrycnt not specified in options; defaulting to 1
Jan 23 21:35:28 daemon automountd[8711]: executing "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/.svn /home/.svn/" as pid 8712
Jan 23 21:35:28 daemon automountd[8711]: completing request 103 with error 5
Jan 23 21:35:28 daemon automountd[8675]: child process 8711 terminated with exit status 1
Jan 23 21:35:28 daemon automountd[8675]: waiting for request from the kernel

Me thinks stat()ing /home/.svn ought to immediately return ENOENT rather than pretending that directory might exist.
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2021-06-22 00:52:43 UTC
^Triage: to submitter: is this still relevant in the post-SVN world?
Comment 2 Joerg Wunsch freebsd_committer 2021-06-22 04:06:21 UTC
(In reply to Mark Linimon from comment #1)

IMHO, SVN just uncovered a bug that is actually somewhere deeper in the system.

So, in terms of SVN, this could probably be closed, but the bug would remain.
Comment 3 Mark Linimon freebsd_committer freebsd_triage 2021-06-22 20:53:06 UTC
^Triage: incorporate feedback.  I was only doing PR triage, so I won't be able to tackle this one myself, sorry.
Comment 4 Ed Maste freebsd_committer 2021-06-23 17:11:22 UTC
Edward any suggestions on next steps for obtaining diagnostic information?
Comment 5 Joerg Wunsch freebsd_committer 2021-06-23 19:05:20 UTC
For a simple test, use something like this:

#include <sys/stat.h>
#include <stdio.h>

int
main(void)
{
        struct stat sb;

        int rv = stat("/home/foobar", &sb);

        printf("stat() got %d\n", rv);

        return 0;
}

/home/foobar is supposed to be a directory that would never exist since there is no such user "foobar". After running the above on an automounted /home, /home/foobar has been created. Trying to access anything there yields

Jun 23 21:03:51 daemon automountd[33777]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/foobar /home/foobar/", pid 33833, terminated with exit status 1
Jun 23 21:03:51 daemon kernel: WARNING: autofs_trigger_one: request for /home/foobar/ completed with error 5
Jun 23 21:03:51 daemon automountd[33777]: mount failed
Jun 23 21:03:53 daemon automountd[34841]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/foobar /home/foobar/", pid 35155, terminated with exit status 1
Jun 23 21:03:53 daemon kernel: WARNING: autofs_trigger_one: request for /home/foobar/ completed with error 5
Jun 23 21:03:53 daemon automountd[34841]: mount failed
Jun 23 21:03:54 daemon automountd[35511]: "mount -t nfs -o nfsv4,automounted,retrycnt=1 alfred.sax.de:/home/foobar /home/foobar/", pid 35610, terminated with exit status 1
Jun 23 21:03:54 daemon kernel: WARNING: autofs_trigger_one: request for /home/foobar/ completed with error 5
Jun 23 21:03:54 daemon automountd[35511]: mount failed
Comment 6 Edward Tomasz Napierala freebsd_committer 2021-07-13 12:02:51 UTC
I agree with Joerg's diagnosis: it appears automountd(8) creates the mountpoint even though it shouldn't, and then (correctly) fails to mount it.  I'm not sure why it's doing that, though.
Comment 7 Robert Wing freebsd_committer 2021-07-16 20:27:43 UTC
(In reply to Edward Tomasz Napierala from comment #6)

Sure would be nice to have the errno from the failing mount program                  
as it exited.                                                           
                                                                        
I *think* this is what's happening:                                     
                                                                        
1. automountd creates /home/foobar because it needs somewhere to mount the
   NFS filesystem.
2. automountd attempts to mount_nfs /home/foobar from the nfs server.   
3. The nfs server replies with ENOENT (because /home/foobar doesn't exist on
   server).
4. the mount program fails and exits with status code 1.                
5. automountd doesn't remove /home/foobar after the failed mount.       
                                                                        
One thought is; make a special case for NFS that checks if the target             
directory is exported from the NFS server before attempting to mount it.         
Not sure how feasible this idea is, I'd have to look into it..or maybe one  
of you know off the top of your head?                                   
                                                                        
I imagine an issue with doing a special case for NFS is that other automounted filesystems may have similar problems as well (but in a different way). Still, given that NFS seems to be the largest use-case for automount, it may be worth making a special case for NFS, even if the problem remains for other automounted filesystems. This is why I mentioned it would be nice to have the errno from the failing mount program as it *might* work as general solution for other automounted filesystems.
Comment 8 Edward Tomasz Napierala freebsd_committer 2021-07-17 08:47:50 UTC
We already have a mechanism for this kind of check: special maps.  Kind of how special_hosts works, but looking up usernames instead of hosts.  I've done a quick test, and it appears that "cd /net/nonexistent" does not result in that directory being created.  So, special case could consist of a new map, say, special_users, which would first verify the username using getent(1).

Another thing we could do is to remove the (newly created) mountpoint if mount fails, before releasing the process which triggered the mount.
Comment 9 Joerg Wunsch freebsd_committer 2021-07-17 08:51:50 UTC
(In reply to Edward Tomasz Napierala from comment #8)

To me, the latter (remove the temporary mountpoint before proceeding) sounds like the right way to go.
Comment 10 Robert Wing freebsd_committer 2021-07-18 20:01:20 UTC
When using a direct map with a wildcard entry and doing "cd /net/nonexistent", a new directory will be created. W.R.T. a special_users map, just because the user exists on the system doesn't necessarily mean that the users home directory is being exported via NFS, no?

The temporary mountpoint should definitely be cleaned up on failure. I suppose this cleanup should happen after autofs has exhausted its retry attempts? Do you think autofs or automountd (via request from autofs) should do the clean up?

Quick glance through the code; looks like it would be a bit of work to make the mount_nfs bits public that would allow automountd to retrieve the error code that was returned from the NFS server..

There's not a reliable way to discover exported directories for NFSv3/NFSv4?

I think in a perfect world, the program flow would be:

1. request to automountd to mount from a NFS server
2. automountd checks if that directory is exported from the NFS server
3. if yes, create the temporary mountpoint
4. attempt the mount
5. if error, remove temporary mountpoint and return the error code to autofs
6. let autofs, based on the error code, decide if it wants to retry to mount.

But, it's not a perfect world, is it?