Bug 212608

Summary: sockstat(1) and lsof(8) can not identity the owner of listening UDP ports when it is the kernel
Product: Base System Reporter: Ben Woods <woodsb02>
Component: kernAssignee: freebsd-bugs mailing list <bugs>
Status: Open ---    
Severity: Affects Some People CC: cem, nowak
Priority: --- Keywords: needs-patch
Version: CURRENTFlags: koobs: mfc-stable11?
koobs: mfc-stable12?
Hardware: Any   
OS: Any   

Description Ben Woods freebsd_committer 2016-09-12 14:33:33 UTC
When the NFS server is running, nfsd listens on TCP and UDP port 2049, but this is incorrectly reported in both sockstat(1) and lsof(8).

$ pgrep -lf nfs
888 nfsd: server
886 nfsd: master

sockstat -l4p 2049
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     nfsd       886   5  tcp4   *:2049                *:*
?        ?          ?     ?  udp4   *:2049                *:*

$ netstat -4na
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 *.2049                 *.*                    LISTEN
udp4       0      0 *.2049                 *.*                    

$ lsof -i TCP:2049 -i UDP:2049
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nfsd    886 root    5u  IPv4 0xfffff8010cdaf830      0t0  TCP *:nfsd (LISTEN)
nfsd    886 root    6u  IPv6 0xfffff8010cdaf418      0t0  TCP *:nfsd (LISTEN)

Perceived problems with the above output:
1. sockstat output does not show user, command, PID, FD details for udp4:2049
2. lsof does not identify anything listening on UDP port 2049

It is quite disconcerting to see output from sockstat(1) that shows an unknown process is listening on a port.

Note: netstat does not show state for UDP ports, so a missing "LISTEN" is not perceived as a bug.


Steps to repeat:

NFS shares enabled in /etc/rc.conf, and /etc/exports or with ZFS exports as per the handbook:
https://www.freebsd.org/doc/handbook/network-nfs.html

# service nfsd start
# service mountd reload
$ sockstat -l4p 2049
$ netstat -4na
# pkg install sysutils/lsof
$ lsof -i TCP:2049
$ lsof -i UDP:2049


Potentially relevant notes from manpage:
     -a      Specifies that nfsd should bind to the wildcard IP address.  This
             is the default if no -h options are given.  It may also be
             specified in addition to any -h options given.  Note that NFS/UDP
             does not operate properly when bound to the wildcard IP address
             whether you use -a or do not use -h.
Comment 1 nowak 2016-09-17 11:24:54 UTC
I believe this is because udp nfs socket is opened by a kernel thread so there is no user/pid/command/fd to report.
Comment 2 Ben Woods freebsd_committer 2016-09-18 07:27:57 UTC
(In reply to nowak from comment #1)
So the TCP socket is opened by userspace, but the UDP socket is opened by the kernel?
Comment 3 Conrad Meyer freebsd_committer 2017-02-16 17:14:00 UTC
It seems the UDP socket is opened by userspace too:

https://github.com/freebsd/freebsd/blob/master/usr.sbin/nfsd/nfsd.c#L463-L491

However, we pass the UDP socket into the kernel with nfssvc_addsock and close the userspace copy.  We keep the TCP one open in order to do accept(2).
Comment 4 Ben Woods freebsd_committer 2017-02-16 22:33:25 UTC
Hi Conrad,
Thanks for that description - it makes sense.
It's still surprising from a user's point of view to see ? in the output of sockstat. Can you think of any way this could be improved? Could sockstat report it is being used by the kernel, or even better nfs within the kernel?
Comment 5 Michael Bueker 2017-06-18 15:15:48 UTC
I can confirm this bug on 11.0, but only for the UDP port:

# sockstat -4l | grep 2049
root     nfsd       646   5  tcp4   *:2049                *:*
?        ?          ?     ?  udp4   *:2049                *:*
Comment 6 Ben Woods freebsd_committer 2019-02-21 22:47:37 UTC
This issue has also been raised on the freebsd-hackers mailing list:
https://lists.freebsd.org/pipermail/freebsd-hackers/2019-February/054146.html