Bug 164402
| Summary: | [pf] pf crashes with a particular set of rules when first matching packet arrives | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Eugene M. Zheganin <eugene> |
| Component: | kern | Assignee: | freebsd-bugs (Nobody) <bugs> |
| Status: | Closed Overcome By Events | ||
| Severity: | Affects Only Me | CC: | kp |
| Priority: | Normal | ||
| Version: | 9.0-RELEASE | ||
| Hardware: | Any | ||
| OS: | Any | ||
Responsible Changed From-To: freebsd-bugs->freebsd-pf Over to maintainer(s). Hi,
I have a vague suspicion on what is happening. Your description of
the problem looks like if a packet processing in the kernel has entered
an endless loop.
Looking at pf_route() I see such possibility. From OpenBSD we have
this protection against endless looping:
if ((*m)->m_pkthdr.pf.routed++ > 3) {
m0 = *m;
*m = NULL;
goto bad;
}
In our code this transforms to:
if (pd->pf_mtag->routed++ > 3) {
m0 = *m;
*m = NULL;
goto bad;
}
The root difference between storing the tag on mbuf and on pfdesc
is that we lose pfdesc, and thus the tag, when we enter pf_test()
recursively. And pf_route() does this recursion:
if (oifp != ifp) {
if (pf_test(PF_OUT, ifp, &m0, NULL) != PF_PASS) {
goto bad;
....
--
Totus tuus, Glebius.
On Sun, Apr 15, 2012 at 11:10:03AM +0000, Gleb Smirnoff wrote:
T> I have a vague suspicion on what is happening. Your description of
T> the problem looks like if a packet processing in the kernel has entered
T> an endless loop.
T>
T> Looking at pf_route() I see such possibility. From OpenBSD we have
T> this protection against endless looping:
T>
T> if ((*m)->m_pkthdr.pf.routed++ > 3) {
T> m0 = *m;
T> *m = NULL;
T> goto bad;
T> }
T>
T> In our code this transforms to:
T>
T> if (pd->pf_mtag->routed++ > 3) {
T> m0 = *m;
T> *m = NULL;
T> goto bad;
T> }
T>
T> The root difference between storing the tag on mbuf and on pfdesc
T> is that we lose pfdesc, and thus the tag, when we enter pf_test()
T> recursively. And pf_route() does this recursion:
T>
T> if (oifp != ifp) {
T> if (pf_test(PF_OUT, ifp, &m0, NULL) != PF_PASS) {
T> goto bad;
T> ....
On second look I see that my suspicion may not be true. In the
beginning of pf_test() we do pf_get_mtag() which preserves already
present tag if there is one.
--
Totus tuus, Glebius.
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped FreeBSD 9.0 is no longer supported. If this problem can be reproduced in 12.0 or 11.2 please re-open this bug. |
This is a router with numerous ISPs connected. This router is running pf with a set of rules and some kind of route-to/reply-to rules to make it answer from each ISP source address via corresponding ISP channel. This router is known to hang quite frequently. It's previous incarnation running on another machine used to hang too (8.2-R, 8.2-S), I was suspecting the hardware, so I moved it here and installed a 9.0. This is an IBM x3560 machine, with all last firmware/BIOS fixes from IBM applied. Eventually I managed to figure the exact set of rules which makes it to hang. Set of rules is attached below. This router is running INVARIANTS/WITNESS/stuff kernel, but still it hangs, not traps. Sometimes though, it may eventually show a very large amount of kernel messages at enormous speed, after that it remains unresponsive to keyboard, to DDB entering and stuff until rebooted. Now I should mention the exact place in the set of rules which makes it hang. This is the place: ===Cut=== 1. # http, server 2. # outer world 3. pass in on $oif reply-to ($oif $picgw) proto tcp from !$picnet to $oip port { 80, 443 } 4. #pass in on $oif proto tcp from any to $oip port { 80, 443 } no state 5. #pass out on $asif route-to ($oif $picgw) proto tcp from $oip port { 80, 443 } to any no state 6. #pass out on $oif proto tcp from $oip port { 80, 443 } to any no state 7. # our servers on this link 8. pass in on $oif proto tcp from $picnet to $oip port { 80, 443 } 9. pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port { 80, 443 } 10. pass in on $asif proto tcp from any to $asip port { 80, 443 } ===Cut=== If I comment out line 3, and uncomment lines 4,5,6 (ofc with pfctl -f /etc/pf.myrules) the system will hang on first matching packets. This reproduceable (I tried 6 to 8 times, got the exact behaviour). By the way, the reason of initiate line no.3 commenting was the fact that running with reply-to rule diminishes the speed from 100Mbit to average 6Kbytes/sec (and this is weird, as you can see there are no queues). A reference to a screen capture of these messages is attached below: http://tech.norma.perm.ru/files/pf-panic03.jpeg This server is equipped with an ipkvm, I can give access to it if needed. Also I can provide further information if needed/any possible help. pf set of rules: [emz@taiga:/etc]# cat pf.taiga iifs = "{" vlan1 vlan2 vlan5 vlan9 vlan10 vlan12 vlan15 vlan19 "}" oif = "vlan104" oip = "89.250.210.67" oif2 = "vlan818" oip2 = "86.109.196.3" oip3 = "86.109.196.5" asif = "vlan23" asip = "128.127.144.3" picgw = "89.250.210.65" picnet = "89.250.210.64/28" syngw = "86.109.196.1" defgw = "128.127.144.5" localpubwifiifs = "{" vlan11 vlan21 "}" table <localpubwifinets> { 192.168.8.192/26, 192.168.9.0/26 } rdpip = "192.168.3.16" oip6if = "gif0" oip6ip = "2001:470:1f08:14c0::2" tunbroker = "216.66.80.26" iip6if = "vlan22" vpnpool = "192.168.248.0/26" hqmbxip = "192.168.3.32" table <publicwifinets> { 192.168.8.192/26, 192.168.93.32/27, 192.168.93.64/27, 192.168.93.128/27, 192.168.93.96/27, 192.168.9.0/26, 192.168.93.192/27, 192.168.93.224/27, 192.168.93.160/27 } table <rfc1918> { 192.168.0.0/16, 172.16.0.0/16, 10.0.0.0/8, 224.0.0.0/8, fd00::/16, fe80::16 } table <moscowrsa> { 94.159.37.117, 94.159.37.118 } no rdr on $oif proto tcp from 192.168.0.0/16 to <moscowrsa> port { 80, 443 } rdr on $oif2 proto tcp from !<rfc1918> to $oip3 port 443 -> $hqmbxip port 443 rdr on $oif proto tcp from !<rfc1918> to $oip port 3389 -> $rdpip port 3389 rdr on $asif proto tcp from !<rfc1918> to $asip port 3389 -> $rdpip port 3389 rdr on $iifs proto tcp from 192.168.0.0/16 to !192.168.0.0/16 port { 80, 443 } -> 127.0.0.1 port 3129 no nat on $asif proto gre all nat on $oif proto { tcp, udp, icmp } from 192.168.0.0/16 to !192.168.0.0/16 -> $oif nat on $oif2 proto { tcp, udp, icmp } from 192.168.0.0/16 to !192.168.0.0/16 -> $oif2 nat on $asif proto { tcp, udp, icmp } from 192.168.0.0/16 to !192.168.0.0/16 -> $asif # default blocking block log all # localhost pass quick on lo0 no state # internal interfaces pass quick on $iifs no state # icmp pass quick proto icmp from any to any pass quick proto icmp6 from any to any # AS ospf pass quick on $asif proto ospf no state # carp pass quick proto vrrp from any to 224.0.0.18 pass in quick on carp0 proto { icmp, tcp, udp } from any to any pass out quick on carp0 proto { icmp, tcp, udp } from any to any # rdp pass in on $oif reply-to ($oif $picgw) proto tcp from !<rfc1918> to $rdpip pass in on $asif proto tcp from !<rfc1918> to $asip port 3389 pass in on $asif proto tcp from !<rfc1918> to $rdpip port 3389 # redirect to exchange, SSL pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from !<rfc1918> to $oip3 port 443 pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from !<rfc1918> to $hqmbxip port 443 # smtp # blocking end-client smtp outside block log on $iifs proto tcp from <rfc1918> to !<rfc1918> # server # outer world pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 25 pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 25 pass in on $asif proto tcp from any to $asip port 25 # client pass out on $oif proto tcp from $oip to any port 25 pass out on $oif2 proto tcp from $oip2 to any port 25 pass out on $asif proto tcp from $asip to any port 25 # ssh, server # outer world pass in on $oif reply-to ($oif $picgw) proto tcp from !$picnet to $oip port 22 # our servers on this link pass in on $oif proto tcp from $picnet to $oip port 22 pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 22 pass in on $asif proto tcp from any to $asip port 22 # ssh, client pass out on $oif proto tcp from $oip to any port 22 pass out on $oif2 proto tcp from $oip2 to any port 22 pass out on $asif proto tcp from $asip to any port 22 # http, server # outer world pass in on $oif reply-to ($oif $picgw) proto tcp from !$picnet to $oip port { 80, 443 } #pass in on $oif proto tcp from any to $oip port { 80, 443 } no state #pass out on $asif route-to ($oif $picgw) proto tcp from $oip port { 80, 443 } to any no state #pass out on $oif proto tcp from $oip port { 80, 443 } to any no state # our servers on this link pass in on $oif proto tcp from $picnet to $oip port { 80, 443 } pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port { 80, 443 } pass in on $asif proto tcp from any to $asip port { 80, 443 } # http, client pass out on $oif proto tcp from $oip to any port { 80, 443 } pass out on $oif2 proto tcp from $oip2 to any port { 80, 443 } pass out on $asif proto tcp from $asip to any port { 80, 443 } # ftp, client pass out on $oif proto tcp from $oip to any port 21 pass out on $oif2 proto tcp from $oip2 to any port 21 pass out on $asif proto tcp from $asip to any port 21 # ftp, client, data, passive pass in on $oif proto tcp from any port 20 to $oip pass in on $oif2 proto tcp from any port 20 to $oip2 pass in on $asif proto tcp from any port 20 to $asip # ftp, server # data pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 21 pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 21 pass in on $asif proto tcp from any to $asip port 21 # ftp, server, data, passive pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 49152:65535 pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 49152:65535 pass in on $asif proto tcp from any to $asip port 49152:65535 # ftp, server, data, active pass out on $oif route-to ($oif $picgw) proto tcp from $oip port 20 to any pass out on $oif2 route-to ($oif2 $syngw) proto tcp from $oip port 20 to any pass out on $asif proto tcp from $oip port 20 to any # dns, zone transfers, client pass out on $oif proto tcp from $oip to any port 53 pass out on $oif2 proto tcp from $oip2 to any port 53 pass out on $asif proto tcp from $asip to any port 53 # dns, zone transfers, server pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 53 pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 53 pass in on $asif proto tcp from any to $asip port 53 # dns, server pass in on $oif reply-to ($oif $picgw) proto udp from any to $oip port 53 pass in on $oif2 reply-to ($oif2 $syngw) proto udp from any to $oip2 port 53 pass in on $asif proto udp from any to $asip port 53 # dns, client pass out on $oif proto udp from $oip to any port 53 pass out on $oif2 proto udp from $oip2 to any port 53 pass out on $asif proto udp from $asip to any port 53 # ntpd pass out on $oif proto udp from $oip to any port 123 pass out on $oif2 proto udp from $oip2 to any port 123 pass out on $asif proto udp from $asip to any port 123 # mpd, l2tp pass in on $oif reply-to ($oif $picgw) proto udp from any to $oip port 1701 pass in on $oif2 reply-to ($oif2 $syngw) proto udp from any to $oip2 port 1701 pass in on $asif reply-to ($asif $defgw) proto udp from any to $asip port 1701 # mpd, pptp # control pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 1723 pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 1723 pass in on $asif proto tcp from any to $asip port 1723 # gre pass in on $oif reply-to ($oif $picgw) proto gre from any to $oip no state pass out on $asif route-to ($oif $picgw) proto gre from $oip to any no state pass out on $oif proto gre from $oip to any no state pass in on $oif2 reply-to ($oif2 $syngw) proto gre from any to $oip2 pass out on $asif route-to ($oif2 $syngw) proto gre from $oip2 to any pass out on $oif2 route-to ($oif2 $syngw) proto gre from $oip2 to any pass in on $asif proto gre from any to $asip no state pass out on $asif proto gre from $asip to any no state # group interfaces pass on vpn inet proto { icmp, tcp, udp } all no state # cvsup pass out on $oif proto tcp from $oip to any port 5999 pass out on $oif2 proto tcp from $oip2 to any port 5999 pass out on $asif proto tcp from $asip to any port 5999 # cvs pass out on $oif proto tcp from $oip to any port 2401 pass out on $oif2 proto tcp from $oip2 to any port 2401 pass out on $asif proto tcp from $asip to any port 2401 # rdp pass out on $oif proto tcp from $oip to any port 3389 pass out on $oif2 proto tcp from $oip2 to any port 3389 pass out on $asif proto tcp from $asip to any port 3389 # Mdm pass out on $oif proto tcp from $oip to 195.161.112.97 port 4434 pass out on $oif2 proto tcp from $oip2 to 195.161.112.97 port 4434 pass out on $asif proto tcp from $asip to 195.161.112.97 port 4434 # unicredit pass out on $oif proto tcp from $oip to any port 7235 pass out on $oif2 proto tcp from $oip2 to any port 7235 pass out on $asif proto tcp from $asip to any port 7235 # some other bank pass out on $oif proto tcp from $oip to 212.120.165.203 port 110 pass out on $oif2 proto tcp from $oip2 to 212.120.165.203 port 110 pass out on $asif proto tcp from $asip to 212.120.165.203 port 110 # sber pass out on $oif proto tcp from $oip to 94.51.87.70 port { 664, 666, 668, 670 } pass out on $oif2 proto tcp from $oip2 to 94.51.87.70 port { 664, 666, 668, 670 } pass out on $asif proto tcp from $asip to 94.51.87.70 port { 664, 666, 668, 670 } pass out on $oif proto tcp from $oip to 194.186.207.124 port 9443 pass out on $oif2 proto tcp from $oip2 to 194.186.207.124 port 9443 pass out on $asif proto tcp from $asip to 194.186.207.124 port 9443 # synterra billing pass on $oif proto tcp from $oip to 217.115.85.6 port 8443 pass on $oif2 proto tcp from $oip2 to 217.115.85.6 port 8443 pass on $asif proto tcp from $asip to 217.115.85.6 port 8443 # irc pass out on $oif proto tcp from $oip to any port { 6667, 6669 } pass out on $oif2 proto tcp from $oip2 to any port { 6667, 6669 } pass out on $asif proto tcp from $asip to any port { 6667, 6669 } # public wifi # local net pass in on $localpubwifiifs proto { tcp, udp, icmp } from <localpubwifinets> to <localpubwifinets> tag pubwifi pass out on $localpubwifiifs proto { tcp, udp, icmp } from <localpubwifinets> to <localpubwifinets> tag pubwifi pass in on $localpubwifiifs proto udp from <localpubwifinets> to <localpubwifinets> port { 67, 68 } tag pubwifi pass in on $localpubwifiifs proto udp from 0.0.0.0 port 68 to 255.255.255.255 port 67 tag pubwifi pass in on $localpubwifiifs proto { tcp, udp, icmp } from <localpubwifinets> to !<rfc1918> tag pubwifi # other networks pass in on $iifs proto { tcp, udp, icmp } from <pubwifinets> to !<rfc1918> tag pubwifi pass in on $oif proto { tcp, udp, icmp } from !<rfc1918> to <pubwifinets> tag pubwifi pass in on $asif proto { tcp, udp, icmp } from !<rfc1918> to <pubwifinets> tag pubwifi # turn loose the swans pass out on $oif all tagged pubwifi pass out on $asif all tagged pubwifi # razor pass out on $oif proto tcp from $oip to any port 2703 pass out on $oif2 proto tcp from $oip2 to any port 2703 pass out on $asif proto tcp from $asip to any port 2703 # kassy pass out on $oif proto tcp from $oip to 212.33.228.45 port 5000 pass out on $oif2 proto tcp from $oip2 to 212.33.228.45 port 5000 pass out on $asif proto tcp from $asip to 212.33.228.45 port 5000 # jabber pass out on $oif proto tcp from $oip to any port { 5222, 5223 } pass out on $oif2 proto tcp from $oip2 to any port { 5222, 5223 } pass out on $asif proto tcp from $asip to any port { 5222, 5223 } # squid passive ftp pass out on $oif proto tcp from $oip to any user squid pass out on $oif2 proto tcp from $oip2 to any user squid pass out on $asif proto tcp from $asip to any user squid # moscow rsa pass out on $oif proto tcp from $oip to <moscowrsa> port { 1044, 1045, 2000 } pass out on $oif proto udp from $oip to <moscowrsa> port 2000 pass out on $oif2 proto tcp from $oip2 to <moscowrsa> port { 1044, 1045, 2000 } pass out on $oif2 proto udp from $oip2 to <moscowrsa> port 2000 pass out on $asif proto tcp from $asip to <moscowrsa> port { 1044, 1045, 2000 } pass out on $asif proto udp from $asip to <moscowrsa> port 2000 # ipv6 # internal interface pass on $iip6if all no state # gif traffic # rule below breaks everything pass out on $asif route-to ($oif $picgw) proto ipv6 from $oip to $tunbroker no state pass out on $oif proto ipv6 from $oip to $tunbroker no state pass in on $oif proto ipv6 from $tunbroker to $oip no state # icmp pass on $oip6if inet6 proto icmp6 from any to any no state ## client services # dns pass out on $oip6if inet6 proto udp from $oip6ip to any port 53 # http pass out on $oip6if inet6 proto tcp from $oip6ip to any port { 80, 443 } # our server pass in on $oip6if inet6 proto tcp from any to $oip6ip port { 80, 443 } ## server pass in on $oip6if inet6 proto udp from any to $oip6ip port 53 How-To-Repeat: Get a FreeBSD 9.0 server with numerous ISP, build a pf set of rules and a policy-based routing.