Bug 256592 - automountd invoking /etc/autofs/special_hosts with non-hostnames as first argument
Summary: automountd invoking /etc/autofs/special_hosts with non-hostnames as first arg...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 13.0-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Edward Tomasz Napierala
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-06-14 08:36 UTC by Tim Foster
Modified: 2021-06-25 22:56 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Foster 2021-06-14 08:36:21 UTC
In a holdover from our days on Solaris, we use NFS automountd frequently.

However, browsing /net mountpoints with the following /etc/auto_master entry generally causes a 5 second lag when visiting each directory before our bash prompt returns:

/net            -hosts          -nosuid,intr

In /var/adm/messages, we typically see:

WARNING: autofs_task: request 94 for /net/ timed out after 30 seconds
WARNING: autofs_trigger_one: request for /net/ completed with error 60, pid 3311 (git)
WARNING: autofs_task: request 95 for /net/ timed out after 30 seconds
WARNING: autofs_trigger_one: request for /net/ completed with error 60, pid 3311 (git)
WARNING: autofs_task: request 96 for /net/ timed out after 30 seconds
WARNING: autofs_trigger_one: request for /net/ completed with error 60, pid 3311 (git)
WARNING: autofs_task: request 97 for /net/ timed out after 30 seconds
WARNING: autofs_trigger_one: request for /net/ completed with error 60, pid 3311 (git)


Seeing "git" there looks a bit odd. In our ~/.bashrc, we have this code to automatically show the git branch of any directory we visit in case it's a git repository:

===
parse_git_branch() {
    git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/'
}

function bash_prompt {
    if [[ "$(id -u)" == 0 ]]; then
      return
    fi
    echo "\[\033[00;34m\]$(id -un)@\$(uname -n)\$(parse_git_branch)\[\033[00m\] "
}

PS1=$(bash_prompt)
===

so that seems like it might have some relevance. Sure enough, when running automountd with some debug flags, we see:

root@puroto:~/.ssh # automountd -d -v
automountd: waiting for request from the kernel
automountd: not forking due to -d flag; will exit after servicing a single request
automountd: got request 91: from map -hosts, path /net/, prefix "/net", key ".git", options "nosuid,intr"
automountd: parsing map "-hosts"
automountd: executing "/etc/autofs/special_hosts .git" as pid 3319
RPC: Unknown host
showmount: can't do exports rpc
automountd: "/etc/autofs/special_hosts .git", pid 3319, terminated with exit status 1
automountd: failed to handle special map "-hosts"
automountd: completing request 91 with error 5


So why does automountd think ".git" is a hostname we should be passing to the special_hosts script, which otherwise ought to be just running 'showmount -e' on the NFS server? Something must be passing ".git" as an argument to /etc/autofs/special_hosts?

For now, we've hacked around the problem with this change to /etc/autofs/special_hosts on the NFS client to see if we can resolve the host at all. This isn't an ideal fix, but seems to work well enough for now.

===
    13  getent hosts $1 > /dev/null
    14  if [ $? -ne 0 ]; then
    15      exit 0
    16  fi
    17
    18  out=`showmount -E "$1"`
    19  [ $? -eq 0 ] || exit 1
    20  echo "$out" | awk -v host="$1" \
    21      '{ printf "\"%s\"\t\"%s:%s\" ", $0, host, $0 } END { printf "\n" }'
    22

===

There seems to be a similar instance of this sort of thing reported at
https://mail-index.netbsd.org/current-users/2019/11/28/msg037105.html

We're running:

FreeBSD puroto 13.0-RELEASE-p1 FreeBSD 13.0-RELEASE-p1 #0: Wed May 26 22:15:09 UTC 2021     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
Comment 1 Tim Foster 2021-06-14 08:55:07 UTC
In the case of the top-level "/net" directory, it makes sense that we're trying to mount "/net/.git" since our bash script is trying to determine if "/net" is a git repository.

However as we browse down the mountpoints from our server, it looks like we're trying to effectively run this multiple times, which seems odd. Adding a logger call to special_hosts, 

===
    13  getent hosts $1 > /dev/null
    14  if [ $? -ne 0 ]; then
    15      logger -t info "tried to access a missing host $1"
    16      exit 0
    17  fi
    18
    19  out=`showmount -E "$1"`
    20  [ $? -eq 0 ] || exit 1
    21  echo "$out" | awk -v host="$1" \
    22      '{ printf "\"%s\"\t\"%s:%s\" ", $0, host, $0 } END { printf "\n" }'
    23
===

we then did a tail -f /var/log/messages in one window, and tried browsing the
mountpoint in another. Here, we'll interleave our shell commands with the log output:

===
timf@puroto bg
[1]+ tail -f /var/log/messages &
timf@puroto
timf@puroto cd /net/linn
Jun 14 09:51:49 puroto info[4884]: tried to access a missing host .git
Jun 14 09:51:49 puroto info[4891]: tried to access a missing host .git
Jun 14 09:51:49 puroto info[4899]: tried to access a missing host HEAD
timf@puroto cd mnt
Jun 14 09:51:54 puroto info[4910]: tried to access a missing host .git
Jun 14 09:51:54 puroto info[4917]: tried to access a missing host .git
Jun 14 09:51:54 puroto info[4925]: tried to access a missing host HEAD
timf@puroto cd tank
Jun 14 09:51:57 puroto info[4943]: tried to access a missing host .git
Jun 14 09:51:57 puroto info[4950]: tried to access a missing host .git
Jun 14 09:51:57 puroto info[4958]: tried to access a missing host HEAD
timf@puroto cd archive
Jun 14 09:51:59 puroto info[4978]: tried to access a missing host .git
Jun 14 09:51:59 puroto info[4985]: tried to access a missing host .git
Jun 14 09:51:59 puroto info[4993]: tried to access a missing host HEAD
timf@puroto ls -d .
25 ./
timf@puroto cd mail
timf@puroto
===

It looks like we're trying to run special_hosts a few times for each level of the hierarchy, where "linn" is our nfs server, and "mnt/tank/archive" is the first shared filesystem, with "mail" just being a normal directory in that share.
Comment 2 Edward Tomasz Napierala freebsd_committer freebsd_triage 2021-06-25 20:44:17 UTC
Can you try something like this:

# dtrace -n 'autofs_trigger_vn:entry  { stack(); ustack(); }'

And then run your test in another terminal and observe the output?  It should make it evident what is triggering it.
Comment 3 Tim Foster 2021-06-25 22:56:05 UTC
Yep. Commenting out my workaround to special_hosts, and running that script,
we're seeing this when my user does a 'cd /net/linn;' 


  7  74603          autofs_trigger_vn:entry
              autofs.ko`autofs_lookup+0x1af
              kernel`lookup+0x68c
              kernel`namei+0x487
              kernel`kern_statat+0xcf
              kernel`sys_fstatat+0x2f
              kernel`amd64_syscall+0x10c
              kernel`0xffffffff8106229e

              libc.so.7`0x80077c2ea
              git`0x493c01
              git`0x494cc2
              git`0x494fcb
              git`0x2adf82
              git`0x2ad6bc
              git`0x2accbf
              git`0x363e2f
              git`0x2ac770
              `0x800510008

 12  74603          autofs_trigger_vn:entry
              autofs.ko`autofs_lookup+0x1af
              kernel`lookup+0x68c
              kernel`namei+0x487
              kernel`kern_statat+0xcf
              kernel`sys_fstatat+0x2f
              kernel`amd64_syscall+0x10c
              kernel`0xffffffff8106229e

              libc.so.7`0x80077c2ea
              git`0x43abfa
              git`0x4938d5
              git`0x494cf6
              git`0x494fcb
              git`0x2adf82
              git`0x2ad6bc
              git`0x2accbf
              git`0x363e2f
              git`0x2ac770
              `0x800510008

  8  74603          autofs_trigger_vn:entry
              autofs.ko`autofs_lookup+0x1af
              kernel`lookup+0x68c
              kernel`namei+0x487
              kernel`kern_statat+0xcf
              kernel`sys_fstatat+0x2f
              kernel`amd64_syscall+0x10c
              kernel`0xffffffff8106229e

              libc.so.7`0x80077c2ea
              git`0x43abfa
              git`0x4938d5
              git`0x494d70
              git`0x494fcb
              git`0x2adf82
              git`0x2ad6bc
              git`0x2accbf
              git`0x363e2f
              git`0x2ac770
              `0x800510008


and then when we cd down to /net/linn/mnt, we get a similar:


  3  74603          autofs_trigger_vn:entry
              autofs.ko`autofs_lookup+0x1af
              kernel`lookup+0x68c
              kernel`namei+0x487
              kernel`kern_statat+0xcf
              kernel`sys_fstatat+0x2f
              kernel`amd64_syscall+0x10c
              kernel`0xffffffff8106229e

              libc.so.7`0x80077c2ea
              git`0x493c01
              git`0x494cc2
              git`0x494fcb
              git`0x2adf82
              git`0x2ad6bc
              git`0x2accbf
              git`0x363e2f
              git`0x2ac770
              `0x800510008

  5  74603          autofs_trigger_vn:entry
              autofs.ko`autofs_lookup+0x1af
              kernel`lookup+0x68c
              kernel`namei+0x487
              kernel`kern_statat+0xcf
              kernel`sys_fstatat+0x2f
              kernel`amd64_syscall+0x10c
              kernel`0xffffffff8106229e

              libc.so.7`0x80077c2ea
              git`0x43abfa
              git`0x4938d5
              git`0x494cf6
              git`0x494fcb
              git`0x2adf82
              git`0x2ad6bc
              git`0x2accbf
              git`0x363e2f
              git`0x2ac770
              `0x800510008

  0  74603          autofs_trigger_vn:entry
              autofs.ko`autofs_lookup+0x1af
              kernel`lookup+0x68c
              kernel`namei+0x487
              kernel`kern_statat+0xcf
              kernel`sys_fstatat+0x2f
              kernel`amd64_syscall+0x10c
              kernel`0xffffffff8106229e

              libc.so.7`0x80077c2ea
              git`0x43abfa
              git`0x4938d5
              git`0x494d70
              git`0x494fcb
              git`0x2adf82
              git`0x2ad6bc
              git`0x2accbf
              git`0x363e2f
              git`0x2ac770
              `0x800510008


Notably, when I browse to the the top-level of a shared mountpoint or a
directory that exists in that share, e.g.

cd /net/linn/mnt/tank/projects
or
cd /net/mnt/tank/projects/triton.git

we don't get the delay, and either no output from the dtrace script if it's
a directory in that mountpoint, or if it's the mountpoint itself
(/net/linn/mnt/tank/projects) we see:


  5  74603          autofs_trigger_vn:entry
              autofs.ko`autofs_lookup+0x1af
              kernel`lookup+0x68c
              kernel`namei+0x487
              kernel`kern_statat+0xcf
              kernel`sys_fstatat+0x2f
              kernel`amd64_syscall+0x10c
              kernel`0xffffffff8106229e

dtrace: sysctl(kern.proc): No such process
              0x8005512ea
              0x2b43e8
              0x2b4c8b
              0x2b48db
              0x2b46d8
              0x26c977
              0x268d1d
              0x26bc7c
              0x26800e
              0x26738b
              0x26b111
              0x2690ab
              0x2bf772
              0x2904ed
              0x294f44
              0x28bf38
              0x28b453
              0x257a3a
              0x255e86
              0x25379a

Git seems to be trying to look for a few different directories each time it gets
triggered. While I can understand that we get confused when looking for

/net/<hostname>.git/ or /net/hostname.SomeOtherGitSpecificThing

it doesn't make sense to me that

/net/<hostname>/<first-level-of-share>.git or /net/<hostname>/<first-level-of-share>.SomeOtherGitSpecificThing is also slow.

On my nfs server, these are the shares configured:

timf@puroto showmount -e linn
Exports list on linn:
/mnt/tank/music                    Everyone
/mnt/tank/timf                     Everyone
/mnt/tank/iorangi                  Everyone
/mnt/tank/notes                    Everyone
/mnt/tank/archive                  Everyone
/mnt/tank/igyo                     Everyone
/mnt/tank/projects                 Everyone
/mnt/tank/calendars                Everyone
/mnt/tank/pictures                 Everyone
/mnt/tank/movies                   Everyone
/mnt/tank/ellaf                    Everyone

Notably, all shares are under /mnt/tank/<something>