Bug 196087

Summary: pf loses states during rdr
Product: Base System Reporter: Vitalii <nucleus.mails>
Component: kernAssignee: freebsd-pf (Nobody) <pf>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: kp
Priority: ---    
Version: 10.0-STABLE   
Hardware: amd64   
OS: Any   

Description Vitalii 2014-12-18 09:09:21 UTC
FreeBSD Packet Filter (pf) "loses" states it is intended to keep.
My router uses pf, the external interface is ix0 (192.168.0.1/24), internal ix1 (10.0.0.1/24) (real ips in this bug report are replaced with 192.168.0.0/24 and 10.0.0.0/24). The pf.conf:

ext="ix0"
int="ix1"

set optimization normal
set block-policy drop
set debug urgent

#set timeout {tcp.first 120, tcp.opening 60, tcp.established 86400}
set timeout {tcp.first 30, tcp.opening 15, tcp.established 16000 }

set timeout { adaptive.start 20000, adaptive.end 120000 }
set limit {states 24000000, frags 80000, src-nodes 240000, table-entries 240000 }

rdr pass on $ext proto { tcp,udp } from any to 10.0.0.100 port 26 -> 10.0.0.200 port 25

The tcpdump lookes like this at ix0:

03:30:23.993221 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [S], seq 3193007607, win 29200, options [mss 1460,sackOK,TS val 7462962 ecr 0,nop,wscale 7], length 0
03:30:23.993489 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [S.], seq 2683690666, ack 3193007608, win 5792, options [mss 1460,sackOK,TS val 3356240738 ecr 7462962,nop,wscale 6], length 0
03:30:24.001833 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [.], ack 1, win 229, options [nop,nop,TS val 7462964 ecr 3356240738], length 0
03:30:24.004494 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 1:68, ack 1, win 91, options [nop,nop,TS val 3356240741 ecr 7462964], length 67
03:30:24.013028 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [.], ack 68, win 229, options [nop,nop,TS val 7462967 ecr 3356240741], length 0
03:30:24.020436 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1:21, ack 68, win 229, options [nop,nop,TS val 7462969 ecr 3356240741], length 20
03:30:24.020565 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], ack 21, win 91, options [nop,nop,TS val 3356240745 ecr 7462969], length 0
03:30:24.020625 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 68:257, ack 21, win 91, options [nop,nop,TS val 3356240745 ecr 7462969], length 189
03:30:24.031660 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 21:31, ack 257, win 237, options [nop,nop,TS val 7462972 ecr 3356240745], length 10
03:30:24.032104 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 257:287, ack 31, win 91, options [nop,nop,TS val 3356240748 ecr 7462972], length 30
03:30:24.079129 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [.], ack 287, win 237, options [nop,nop,TS val 7462984 ecr 3356240748], length 0
03:30:24.115438 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 31:548, ack 287, win 237, options [nop,nop,TS val 7462993 ecr 3356240748], length 517
03:30:24.128042 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], seq 287:1735, ack 548, win 108, options [nop,nop,TS val 3356240772 ecr 7462993], length 1448
03:30:24.128054 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], seq 1735:3183, ack 548, win 108, options [nop,nop,TS val 3356240772 ecr 7462993], length 1448
03:30:24.128059 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 3183:4028, ack 548, win 108, options [nop,nop,TS val 3356240772 ecr 7462993], length 845
03:30:24.138631 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [.], ack 1735, win 260, options [nop,nop,TS val 7462998 ecr 3356240772], length 0
03:30:24.139007 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [.], ack 3183, win 282, options [nop,nop,TS val 7462999 ecr 3356240772], length 0
03:30:24.139100 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [.], ack 4028, win 305, options [nop,nop,TS val 7462999 ecr 3356240772], length 0
03:30:24.140425 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 548:746, ack 4028, win 305, options [nop,nop,TS val 7462999 ecr 3356240772], length 198
03:30:24.143985 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4028:4294, ack 746, win 124, options [nop,nop,TS val 3356240776 ecr 7462999], length 266
03:30:24.153969 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 746:836, ack 4294, win 327, options [nop,nop,TS val 7463002 ecr 3356240776], length 90
03:30:24.154221 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4294:4507, ack 836, win 124, options [nop,nop,TS val 3356240778 ecr 7463002], length 213
03:30:24.171670 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 836:958, ack 4507, win 350, options [nop,nop,TS val 7463007 ecr 3356240778], length 122
03:30:24.173312 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4507:4576, ack 958, win 124, options [nop,nop,TS val 3356240783 ecr 7463007], length 69
03:30:24.184091 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 958:1064, ack 4576, win 350, options [nop,nop,TS val 7463009 ecr 3356240783], length 106
03:30:24.184637 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4576:4629, ack 1064, win 124, options [nop,nop,TS val 3356240786 ecr 7463009], length 53
03:30:24.198860 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1064:1170, ack 4629, win 350, options [nop,nop,TS val 7463014 ecr 3356240786], length 106
03:30:24.236252 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4629:4682, ack 1170, win 124, options [nop,nop,TS val 3356240799 ecr 7463014], length 53
03:30:24.250760 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1170:1276, ack 4682, win 350, options [nop,nop,TS val 7463026 ecr 3356240799], length 106
03:30:24.290819 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], ack 1276, win 124, options [nop,nop,TS val 3356240813 ecr 7463026], length 0
03:30:25.277871 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4682:4735, ack 1276, win 124, options [nop,nop,TS val 3356241059 ecr 7463026], length 53
03:30:25.286685 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1276:1382, ack 4735, win 350, options [nop,nop,TS val 7463286 ecr 3356241059], length 106
03:30:25.286884 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], ack 1382, win 124, options [nop,nop,TS val 3356241062 ecr 7463286], length 0
03:30:26.324700 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4735:4788, ack 1382, win 124, options [nop,nop,TS val 3356241321 ecr 7463286], length 53
03:30:26.335513 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1382:1488, ack 4788, win 350, options [nop,nop,TS val 7463547 ecr 3356241321], length 106
03:30:26.335675 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], ack 1488, win 124, options [nop,nop,TS val 3356241324 ecr 7463547], length 0
03:30:27.374006 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4788:4841, ack 1488, win 124, options [nop,nop,TS val 3356241583 ecr 7463547], length 53
03:30:27.400642 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1488:1594, ack 4841, win 350, options [nop,nop,TS val 7463814 ecr 3356241583], length 106
03:30:27.400824 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], ack 1594, win 124, options [nop,nop,TS val 3356241590 ecr 7463814], length 0
03:30:28.438606 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4841:4894, ack 1594, win 124, options [nop,nop,TS val 3356241849 ecr 7463814], length 53
03:30:28.449875 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1594:1700, ack 4894, win 350, options [nop,nop,TS val 7464076 ecr 3356241849], length 106
03:30:28.450080 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], ack 1700, win 124, options [nop,nop,TS val 3356241852 ecr 7464076], length 0
03:30:29.484062 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [P.], seq 4894:4947, ack 1700, win 124, options [nop,nop,TS val 3356242111 ecr 7464076], length 53
03:30:29.497751 IP 192.168.0.2.53143 > 10.0.0.100.26: Flags [P.], seq 1700:1806, ack 4947, win 350, options [nop,nop,TS val 7464338 ecr 3356242111], length 106
03:30:29.497931 IP 10.0.0.100.26 > 192.168.0.2.53143: Flags [.], ack 1806, win 124, options [nop,nop,TS val 3356242114 ecr 7464338], length 0
03:30:30.535817 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 2683695613:2683695666, ack 3193009413, win 124, options [nop,nop,TS val 3356242374 ecr 7464338], length 53
03:30:30.746681 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 0:53, ack 1, win 124, options [nop,nop,TS val 3356242427 ecr 7464338], length 53
03:30:31.171215 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 0:53, ack 1, win 124, options [nop,nop,TS val 3356242533 ecr 7464338], length 53
03:30:32.018917 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 0:53, ack 1, win 124, options [nop,nop,TS val 3356242745 ecr 7464338], length 53
03:30:33.714629 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 0:53, ack 1, win 124, options [nop,nop,TS val 3356243169 ecr 7464338], length 53
03:30:37.106577 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 0:53, ack 1, win 124, options [nop,nop,TS val 3356244017 ecr 7464338], length 53
03:30:43.890433 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 0:53, ack 1, win 124, options [nop,nop,TS val 3356245713 ecr 7464338], length 53
03:30:57.458197 IP 10.0.0.200.25 > 192.168.0.2.53143: Flags [P.], seq 0:53, ack 1, win 124, options [nop,nop,TS val 3356249105 ecr 7464338], length 53

As you can see after some number of packets pf loses states and forgets that it should return packets from 10.0.0.200:25 as 10.0.0.100.26. This causes the problems with timeouts during tcp connection at 192.168.0.2 and as result at 10.0.0.200.

Please fix the bug.
Comment 1 Vitalii 2014-12-18 09:19:57 UTC
P.S. Server works as gateway and totally 50-100 kpps passes through it, but the number of packets that fit this rdr rule is relatively small, only 10-1000 pps.
Comment 2 Vitalii 2014-12-18 09:21:31 UTC
The problem was noticed after moving from another server with freebsd 9 amd64 to this server with FreeBSD 10.0-STABLE amd64.
Comment 3 Kristof Provost freebsd_committer freebsd_triage 2019-03-23 06:54:12 UTC
Does this problem still happen on 12.0?

The problem description doesn't immediately ring any bells with me, so unless we can reproduce it or get access to a failing setup I don't think there's much we can do.