Bug 212608

Summary:	sockstat(1) and lsof(8) can not identity the owner of listening UDP ports when it is the kernel
Product:	Base System	Reporter:	Ben Woods <woodsb02>
Component:	kern	Assignee:	freebsd-bugs (Nobody) <bugs>
Status:	Open ---
Severity:	Affects Some People	CC:	cem, nowak
Priority:	---	Keywords:	needs-patch
Version:	CURRENT	Flags:	koobs: mfc-stable12? koobs: mfc-stable11?
Hardware:	Any
OS:	Any

Description Ben Woods freebsd_committer

2016-09-12 14:33:33 UTC

When the NFS server is running, nfsd listens on TCP and UDP port 2049, but this is incorrectly reported in both sockstat(1) and lsof(8).

$ pgrep -lf nfs
888 nfsd: server
886 nfsd: master

sockstat -l4p 2049
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root nfsd 886 5 tcp4 *:2049 *:*
? ? ? ? udp4 *:2049 *:*

$ netstat -4na
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 *.2049 *.* LISTEN
udp4 0 0 *.2049 *.*

$ lsof -i TCP:2049 -i UDP:2049
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nfsd 886 root 5u IPv4 0xfffff8010cdaf830 0t0 TCP *:nfsd (LISTEN)
nfsd 886 root 6u IPv6 0xfffff8010cdaf418 0t0 TCP *:nfsd (LISTEN)

Perceived problems with the above output:
1. sockstat output does not show user, command, PID, FD details for udp4:2049
2. lsof does not identify anything listening on UDP port 2049

It is quite disconcerting to see output from sockstat(1) that shows an unknown process is listening on a port.

Note: netstat does not show state for UDP ports, so a missing "LISTEN" is not perceived as a bug.

Steps to repeat:

NFS shares enabled in /etc/rc.conf, and /etc/exports or with ZFS exports as per the handbook:
https://www.freebsd.org/doc/handbook/network-nfs.html

# service nfsd start
# service mountd reload
$ sockstat -l4p 2049
$ netstat -4na
# pkg install sysutils/lsof
$ lsof -i TCP:2049
$ lsof -i UDP:2049

Potentially relevant notes from manpage:
-a Specifies that nfsd should bind to the wildcard IP address. This
is the default if no -h options are given. It may also be
specified in addition to any -h options given. Note that NFS/UDP
does not operate properly when bound to the wildcard IP address
whether you use -a or do not use -h.

Comment 1 nowak 2016-09-17 11:24:54 UTC

I believe this is because udp nfs socket is opened by a kernel thread so there is no user/pid/command/fd to report.

Comment 2 Ben Woods freebsd_committer

2016-09-18 07:27:57 UTC

(In reply to nowak from comment #1)
So the TCP socket is opened by userspace, but the UDP socket is opened by the kernel?

Comment 3 Conrad Meyer freebsd_committer

2017-02-16 17:14:00 UTC

It seems the UDP socket is opened by userspace too:

https://github.com/freebsd/freebsd/blob/master/usr.sbin/nfsd/nfsd.c#L463-L491

However, we pass the UDP socket into the kernel with nfssvc_addsock and close the userspace copy.  We keep the TCP one open in order to do accept(2).

Comment 4 Ben Woods freebsd_committer

2017-02-16 22:33:25 UTC

Hi Conrad,
Thanks for that description - it makes sense.
It's still surprising from a user's point of view to see ? in the output of sockstat. Can you think of any way this could be improved? Could sockstat report it is being used by the kernel, or even better nfs within the kernel?

Comment 5 Michael Büker 2017-06-18 15:15:48 UTC

I can confirm this bug on 11.0, but only for the UDP port:

# sockstat -4l | grep 2049
root     nfsd       646   5  tcp4   *:2049                *:*
?        ?          ?     ?  udp4   *:2049                *:*

Comment 6 Ben Woods freebsd_committer

2019-02-21 22:47:37 UTC

This issue has also been raised on the freebsd-hackers mailing list:
https://lists.freebsd.org/pipermail/freebsd-hackers/2019-February/054146.html