From FreeBSD 5.x on, it is possible to allocate local ports for TCP connections even if PCB for given socket is in TIME_WAIT state. This causes execessive packet drops in scenarios where client initiates lots of outgoing connections toward server with stateful firewall (pf).

This behavior is not present in 4.x releases.

Fix: 

There is number of possibilities:

- convert stateful firewall ruleset to stateless (really not an option in my setup)
- tweak timeouts for firewall states (also not an option)
- remove INP_TIMEWAIT logic from FreeBSD 5.x TCP stack altogether
- use following patch:
  http://techie.devnull.cz/public/noTW.patch
  - this patch adds new sysctl net.inet.ip.portrange.twalloc which can be used to revert the INP_TIMEWAIT behavior to the old state. the patch will preserve current bind(2)/connect(2) behavior in FreeBSD 5.x by default.
How-To-Repeat: 1. download and compile
   http://techie.devnull.cz/public/tcpclient-bind.c
   [ or use any other basic TCP client which can bind(2) local port ]
2. connect to TCP service running on server with stateful firewall.
3. wait for tcpclient-bind to end
4. display socket state for the connection from step nr. 2
5. connect to the same TCP service
6. watch server logs (dmesg) for 'pf BAD state' entries

For example:

lada@ipfw2$ ./tcpclient-bind SERVER 21
connecting
connected
src addr = CLIENT-IP
src port = 1234
closing connection
lada@ipfw2$ netstat -an |grep SERVER-IP
tcp4       0      0  CLIENT-IP.1234     SERVER-IP.21      TIME_WAIT
lada@ipfw2$ ./tcpclient-bind SERVER 21
connecting
tcpclient-bind: connect: Operation timed out
lada@ipfw2$ 

the 'BAD state' messages then look like this:

Oct  3 13:53:54 SERVER kernel: pf: BAD state: TCP SERVER-IP:21 SERVER-IP:21 CLIENT-IP:4094 [lo=1074578133 high=1074644741 win=33304 modulator=0 wscale=1] [lo=819316103 high=819382710 win=33304 modulator=0 wscale=1] 9:9 S seq=1087912989 ack=819316103 len=0 ackskew=0 pkts=12:19 dir=in,fwd
Oct  3 13:53:54 SERVER kernel: pf: State failure on: 1       | 5  


example of stateful rule:

  pass in quick on $ext_if proto tcp from any to ($ext_if) \
        port = ftp \
        flags S/S keep state \
        label "EXT-ingress--ftp-cmd"


This problem is always reproducible.

One of the scenarios where this problems could manifest itself is e.g. proxy+backend setup where proxy issues big number of connections to backend server running stateful firewall.