Created attachment 207418 [details]
I just noticed that our production servers seems to be accumulating FIN_WAIT_2 state entries in PF (atleast IPv6 ones).
# pfctl -ss | egrep FIN_WAIT_2 | egrep -v 2001 | wc -l
# pfctl -ss | egrep FIN_WAIT_2 | egrep 2001 | wc -l
Using workaround from bug 222126 seems to wake up the "pf purge" kernel thread again
(atleast for a while):
# echo "set timeout interval 5" | pfctl -mf -
Dell PowerEdge R730xd
(NFS & SMB fileservers)
*** Bug 240533 has been marked as a duplicate of this bug. ***
Can you repeat the tests from #222126?
Specifically the dtrace script in comment #1 and the procstat in comment #6?
I ran those tests you mention and some other stuff, and it looks like the "accumulation" was due to Linux (Ubuntu 18.04 for what it's worth) NFS clients bombarding our servers with new TCP connections (which they just as quickly teared down). Like 200-400 new requests/s with unique client source port numbers - no surprise those states accumulated quickly.
Exactly why those Linux clients are doing this is a bit unclear but it looks like it have something to do with users having their home directory mounted via NFSv4 with sec=krb5, and then their Kerberos tickets expiring (on the client). Possibly while they were running "evolution" which has a number of files/databases opened on the users home directory. This seems to cause "rpc.gssd" (on the client) to go into a spin (100% CPU) and somehow causes this endless stream of new TCP connections...
The stream of IP packets we are seeing look like this:
0.001280 Client -> Server SYN
0.001289 Server -> Client SYN+ACK
0.001516 Client -> Server ACK
0.003609 Client -> Server FIN+ACK
0.003615 Server -> Client ACK
0.003620 Server -> Client FIN+ACK
0.003841 Client -> Server ACK
<repeat 400 times/s>
Anyway I don't think this is a problem in FreeBSD/pf so we can close this bug. Looking more like (yet another) Linux bug.
(I wonder if it would be possible to throttle misbehaving clients like these somehow, perhaps some rate-limiting in PF could do the trick? Hmm...)
Not a FreeBSD/pf issue, as per comment #3.