It appears pf keeps incorrect state for packets routed via a GRE tunnel secured via IPSec. GRE tunnel works correctly without IPSec, but with IPSec (transport mode) enabled between the hosts and pf enabled, traffic does not flow as expected. Traffic between hosts (not going over GRE tunnel) with IPSec appears to work as expected. ICMP echo requests / replies and TCP SYN packets make it through, even if rules are put in place to prevent the traffic. TCP replies are NOT let through, even if rules are put in place to allow the traffic. When showing states with pfctl, it seems that the states are both in one direction when using IPSec, as opposed to in opposite directions without IPSec. CORRECT (GRE tunnel w/o IPSEC): > pfctl -ss | grep 10.6.0.10 | grep 10.1.0.1 gre0 icmp 10.6.0.10:63271 <- 10.1.0.1:63271 0:0 vmx1 icmp 10.1.0.1:63271 -> 10.6.0.10:63271 0:0 BAD (GRE tunnel w/ IPSEC): > pfctl -ss | grep 10.6.0.10 | grep 10.1.0.1 vmx1 icmp 10.1.0.1:588 -> 10.6.0.10:588 0:0 gre0 icmp 10.6.0.10:588 -> 10.1.0.1:588 0:0 Config: host1 10.10.10.1 external interface, 10.6.0.1 internal interface host2 10.10.10.2 host1 (freebsd router) ifconfig gre0 tunnel: inet 10.10.10.1 -> 10.10.10.2 inet 10.1.0.1 --> 10.1.0.2 netmask 0xfffffffc host2 (freebsd client) ifconfig gre0 tunnel: inet 10.10.10.2 -> 10.10.10.1 inet 10.1.0.2 --> 10.1.0.1 netmask 0xfffffffc host2: route add -net 10.6.0.0/23 10.1.0.1 Originally came across this in downstream pfSense based on 11.1-RELEASE-p6, but I've reproduced this 12-CURRENT snapshot r330034.
Can you add your network and pf configuration? Ideally minimised to the smallest reproducing test-case.
################# HOST 1 - ROUTER ################################### root@fbsd12test1:~ # cat /etc/rc.conf hostname="fbsd12test1" ifconfig_vmx0="inet 10.10.10.1 netmask 255.255.255.0" ifconfig_vmx1="inet 10.6.0.1 netmask 255.255.254.0" #lan is a /23 cloned_interfaces="gre0" ifconfig_gre0="inet 10.1.0.1 10.1.0.2 netmask 255.255.255.252 tunnel 10.10.10.1 10.10.10.2 link0 up" gateway_enable="YES" pf_enable="YES" pflog_enable="YES" sshd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO" strongswan_enable="YES" root@fbsd12test1:~ # cat /etc/pf.conf set block-policy drop set loginterface egress set skip on lo0 set skip on gre0 ### has no apparent effect? ### #default block rules block log all # allow in from other host pass in quick on vmx0 from 10.10.10.0/24 # allow all in from lan subnet pass in quick from 10.6.0.0/23 # pass all output packets pass out quick root@fbsd12test1:~ # cat /usr/local/etc/ipsec.conf config setup uniqueids = yes conn bypasslan leftsubnet = 10.6.0.0/23 rightsubnet = 10.6.0.0/23 authby = never type = passthrough auto = route conn con1 fragmentation = yes keyexchange = ike reauth = yes forceencaps = no mobike = no rekey = yes installpolicy = yes type = transport dpdaction = restart dpddelay = 10s dpdtimeout = 60s auto = route left = 10.10.10.1 right = 10.10.10.2 leftid = 10.10.10.1 ikelifetime = 28800s lifetime = 3600s ike = aes256-sha256-modp4096! esp = aes256-sha256-modp4096! leftauth = psk rightauth = psk rightid = 10.10.10.2 aggressive = no rightsubnet = 10.10.10.2 leftsubnet = 10.10.10.1 root@fbsd12test1:~ # cat /usr/local/etc/ipsec.secrets # ipsec.secrets - strongSwan IPsec secrets file %any 10.10.10.2 : PSK "testingtestingtesting" ################# HOST 2 - CLIENT ################################### root@fbsd12test2:~ # cat /etc/rc.conf hostname="fbsd12test2" ifconfig_vmx0="inet 10.10.10.2 netmask 255.255.255.0" cloned_interfaces="gre0" ifconfig_gre0="inet 10.1.0.2 10.1.0.1 netmask 255.255.255.252 tunnel 10.10.10.2 10.10.10.1 link0 up" pf_enable="NO" sshd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO" strongswan_enable="YES" root@fbsd12test2:~ # cat /usr/local/etc/ipsec.conf config setup uniqueids = yes conn con1 fragmentation = yes keyexchange = ike reauth = yes forceencaps = no mobike = no rekey = yes installpolicy = yes type = transport dpdaction = restart dpddelay = 10s dpdtimeout = 60s auto = route left = 10.10.10.2 right = 10.10.10.1 leftid = 10.10.10.2 ikelifetime = 28800s lifetime = 3600s ike = aes256-sha256-modp4096! esp = aes256-sha256-modp4096! leftauth = psk rightauth = psk rightid = 10.10.10.1 aggressive = no rightsubnet = 10.10.10.1 leftsubnet = 10.10.10.2 root@fbsd12test2:~ # cat /usr/local/etc/ipsec.secrets # ipsec.secrets - strongSwan IPsec secrets file %any 10.10.10.1 : PSK "testingtestingtesting" root@fbsd12test2:~ # route add -net 10.6.0.0/23 10.1.0.2 Test tcp connection to host on 10.6.0.0/23 (for instance, 10.6.0.10). SYN packet gets through to, ACK doesn't get back. Also, ICMP traffic let through even if you remove the pass rules that let it go through.
Just for reference, here's the downstream bug: https://redmine.pfsense.org/issues/4479
Correction/clarification: Setting "set skip on gre0" indeed allows the traffic to go through, but doesn't allow any control of traffic over the tunnel. The following /etc/pf.conf configuration does NOT work as expected: set block-policy drop set loginterface egress set skip on lo0 #default block rules block log all # allow in from other host pass in quick on vmx0 from 10.10.10.0/24 # pass in from gre0 pass in quick on gre0 to 10.6.0.0/23 # allow all in from lan subnet pass in quick from 10.6.0.0/23 # pass all output packets pass out quick
There is an implementation available for fixing this in Darwin kernel. Very easy for some one with enough time to copy paste their fixes on it.
(In reply to Ermal Luçi from comment #5) Do you have a pointer to the relevant bits? I can probably take a look at some point.
@(In reply to Ermal Luçi from comment #5) Very curious to see a pointer to those bits - I poked around the Darwin XNU sources, but I'm way out of my depth :)
(In reply to obrienmd from comment #7) You can start here: https://github.com/apple/darwin-xnu/blob/master/bsd/net/pf.c#L317 Its very easy to bring it to FreeBSD, it is only about available time.
Hey all, A friend of mine was curious about getting this bug resolved. He still sees it today on some of his systems. Are there any updates? Thanks for the hard work!
Sadly not. This bug is actually still on my todo list, so I haven't forgotten about it yet, but it's some way down the list right now. If you're friend is very motivated a test case we can plug into the existing pf tests would likely be very helpful.