Bug 226411

Summary: PF does not properly keep state with GRE in IPSec
Product: Base System Reporter: Eric Dombroski <eric>
Component: kernAssignee: freebsd-pf (Nobody) <pf>
Status: New ---    
Severity: Affects Some People CC: dani-bsd, eri, franco, ian, jorgito1412, kp, obrienmd, shawn.webb
Priority: ---    
Version: CURRENT   
Hardware: amd64   
OS: Any   

Description Eric Dombroski 2018-03-07 02:57:01 UTC
It appears pf keeps incorrect state for packets routed via a GRE tunnel secured via IPSec.  GRE tunnel works correctly without IPSec, but with IPSec (transport mode) enabled between the hosts and pf enabled, traffic does not flow as expected.  Traffic between hosts (not going over GRE tunnel) with IPSec appears to work as expected.

ICMP echo requests / replies and TCP SYN packets make it through, even if  rules are put in place to prevent the traffic.  TCP replies are NOT let through, even if rules are put in place to allow the traffic.  

When showing states with pfctl, it seems that the states are both in one direction when using IPSec, as opposed to in opposite directions without IPSec.



CORRECT (GRE tunnel w/o IPSEC):
> pfctl -ss | grep 10.6.0.10 | grep 10.1.0.1
gre0 icmp 10.6.0.10:63271 <- 10.1.0.1:63271       0:0
vmx1 icmp 10.1.0.1:63271 -> 10.6.0.10:63271       0:0


BAD (GRE tunnel w/ IPSEC):
> pfctl -ss | grep 10.6.0.10 | grep 10.1.0.1
vmx1 icmp 10.1.0.1:588 -> 10.6.0.10:588       0:0
gre0 icmp 10.6.0.10:588 -> 10.1.0.1:588       0:0


Config:

host1 10.10.10.1 external interface, 10.6.0.1 internal interface
host2 10.10.10.2

host1 (freebsd router)
ifconfig gre0
        tunnel: inet 10.10.10.1 -> 10.10.10.2
        inet 10.1.0.1 --> 10.1.0.2 netmask 0xfffffffc

host2 (freebsd client)
ifconfig gre0
        tunnel: inet 10.10.10.2 -> 10.10.10.1
        inet 10.1.0.2 --> 10.1.0.1 netmask 0xfffffffc

host2:  route add -net 10.6.0.0/23 10.1.0.1

Originally came across this in downstream pfSense based on 11.1-RELEASE-p6, but I've reproduced this 12-CURRENT snapshot r330034.
Comment 1 Kristof Provost freebsd_committer freebsd_triage 2018-03-11 04:40:51 UTC
Can you add your network and pf configuration? Ideally minimised to the smallest reproducing test-case.
Comment 2 Eric Dombroski 2018-03-13 01:05:53 UTC
################# HOST 1 - ROUTER ###################################
root@fbsd12test1:~ # cat /etc/rc.conf
hostname="fbsd12test1"
ifconfig_vmx0="inet 10.10.10.1 netmask 255.255.255.0"
ifconfig_vmx1="inet 10.6.0.1 netmask 255.255.254.0"  #lan is a /23
cloned_interfaces="gre0"
ifconfig_gre0="inet 10.1.0.1 10.1.0.2 netmask 255.255.255.252 tunnel 10.10.10.1 10.10.10.2 link0 up"
gateway_enable="YES"
pf_enable="YES"
pflog_enable="YES"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
strongswan_enable="YES"





root@fbsd12test1:~ # cat /etc/pf.conf
set block-policy drop
set loginterface egress
set skip on lo0
set skip on gre0  ### has no apparent effect? ###


#default block rules
block log all

# allow in from other host
pass in quick on vmx0 from 10.10.10.0/24

# allow all in from lan subnet
pass in quick from 10.6.0.0/23

# pass all output packets
pass out quick



root@fbsd12test1:~ # cat /usr/local/etc/ipsec.conf
config setup
        uniqueids = yes

conn bypasslan
        leftsubnet = 10.6.0.0/23
        rightsubnet = 10.6.0.0/23
        authby = never
        type = passthrough
        auto = route

conn con1
        fragmentation = yes
        keyexchange = ike
        reauth = yes
        forceencaps = no
        mobike = no
        rekey = yes
        installpolicy = yes
        type = transport
        dpdaction = restart
        dpddelay = 10s
        dpdtimeout = 60s
        auto = route
        left = 10.10.10.1
        right = 10.10.10.2
        leftid = 10.10.10.1
        ikelifetime = 28800s
        lifetime = 3600s
        ike = aes256-sha256-modp4096!
        esp = aes256-sha256-modp4096!
        leftauth = psk
        rightauth = psk
        rightid = 10.10.10.2
        aggressive = no
        rightsubnet = 10.10.10.2
        leftsubnet = 10.10.10.1


root@fbsd12test1:~ # cat /usr/local/etc/ipsec.secrets
# ipsec.secrets - strongSwan IPsec secrets file
%any 10.10.10.2 : PSK "testingtestingtesting"



















################# HOST 2 - CLIENT ###################################
root@fbsd12test2:~ # cat /etc/rc.conf
hostname="fbsd12test2"
ifconfig_vmx0="inet 10.10.10.2 netmask 255.255.255.0"
cloned_interfaces="gre0"
ifconfig_gre0="inet 10.1.0.2 10.1.0.1 netmask 255.255.255.252 tunnel 10.10.10.2 10.10.10.1 link0 up"
pf_enable="NO"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
strongswan_enable="YES"


root@fbsd12test2:~ # cat /usr/local/etc/ipsec.conf
config setup
        uniqueids = yes

conn con1
        fragmentation = yes
        keyexchange = ike
        reauth = yes
        forceencaps = no
        mobike = no
        rekey = yes
        installpolicy = yes
        type = transport
        dpdaction = restart
        dpddelay = 10s
        dpdtimeout = 60s
        auto = route
        left = 10.10.10.2
        right = 10.10.10.1
        leftid = 10.10.10.2
        ikelifetime = 28800s
        lifetime = 3600s
        ike = aes256-sha256-modp4096!
        esp = aes256-sha256-modp4096!
        leftauth = psk
        rightauth = psk
        rightid = 10.10.10.1
        aggressive = no
        rightsubnet = 10.10.10.1
        leftsubnet = 10.10.10.2


root@fbsd12test2:~ # cat /usr/local/etc/ipsec.secrets
# ipsec.secrets - strongSwan IPsec secrets file
%any 10.10.10.1 : PSK "testingtestingtesting"


root@fbsd12test2:~ # route add -net 10.6.0.0/23 10.1.0.2




Test tcp connection to host on 10.6.0.0/23 (for instance, 10.6.0.10). SYN packet gets through to, ACK doesn't get back.  Also, ICMP traffic let through even if you remove the pass rules that let it go through.
Comment 3 Daniel B. 2018-03-13 10:54:44 UTC
Just for reference, here's the downstream bug: https://redmine.pfsense.org/issues/4479
Comment 4 Eric Dombroski 2018-03-13 20:00:53 UTC
Correction/clarification:

Setting "set skip on gre0" indeed allows the traffic to go through, but doesn't allow any control of traffic over the tunnel.  

The following /etc/pf.conf configuration does NOT work as expected:


set block-policy drop
set loginterface egress
set skip on lo0

#default block rules
block log all

# allow in from other host
pass in quick on vmx0 from 10.10.10.0/24

# pass in from gre0
pass in quick on gre0 to 10.6.0.0/23

# allow all in from lan subnet
pass in quick from 10.6.0.0/23

# pass all output packets
pass out quick
Comment 5 Ermal Luçi freebsd_committer freebsd_triage 2018-03-18 01:28:38 UTC
There is an implementation available for fixing this in Darwin kernel.
Very easy for some one with enough time to copy paste their fixes on it.
Comment 6 Kristof Provost freebsd_committer freebsd_triage 2018-03-18 09:27:26 UTC
(In reply to Ermal Luçi from comment #5)
Do you have a pointer to the relevant bits? I can probably take a look at some point.
Comment 7 obrienmd 2018-05-03 17:09:44 UTC
@(In reply to Ermal Luçi from comment #5)

Very curious to see a pointer to those bits - I poked around the Darwin XNU sources, but I'm way out of my depth :)
Comment 8 Ermal Luçi freebsd_committer freebsd_triage 2018-05-03 23:15:36 UTC
(In reply to obrienmd from comment #7)

You can start here:
https://github.com/apple/darwin-xnu/blob/master/bsd/net/pf.c#L317

Its very easy to bring it to FreeBSD, it is only about available time.
Comment 9 Shawn Webb 2019-03-15 15:18:15 UTC
Hey all,

A friend of mine was curious about getting this bug resolved. He still sees it today on some of his systems. Are there any updates?

Thanks for the hard work!
Comment 10 Kristof Provost freebsd_committer freebsd_triage 2019-03-15 15:26:34 UTC
Sadly not. This bug is actually still on my todo list, so I haven't forgotten about it yet, but it's some way down the list right now.

If you're friend is very motivated a test case we can plug into the existing pf tests would likely be very helpful.