Bug 215006 - [ipsec] Unable to use pf RDR on enc0 in transport mode
Summary: [ipsec] Unable to use pf RDR on enc0 in transport mode
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Andrey V. Elsukov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-12-02 15:04 UTC by Jérôme-Charles LALLEMAND
Modified: 2016-12-07 11:32 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jérôme-Charles LALLEMAND 2016-12-02 15:04:38 UTC
I'm trying to nat packet incoming from enc0 to an other machine.

The RDR statement in pf works for the incoming packet, but the reply from the other machine is forwarded back to the issuer without encryption.

It might be because of the state matching on the reply, witch send back the reply bypassing the SPD rules.

This is working flawlessly with encryption.

I’m in a gateway setup, is there any chances to get the RDR working with ipsec in transport mode ?


Computer1 em0 | ----->IPSEC-----> | em0 Computer2 em1 | -----> | em0 Computer3 |
        10.11.1.3               172.31.0.1     10.56.1.10    10.56.1.224


Here is my setup :

ifconfig
em0 : 172.31.0.1/24
em1 : 10.56.1.10/24

pf.conf :
rdr on enc0 inet from 10.11.0.0/16 to 172.31.0.1 tag "balance-1" -> 10.56.1.224
pass all

setkey.conf :
add -4 10.11.1.3 172.31.0.1 esp 0x100 -m transport -E rijndael-cbc "This is secret AES 256 bits key!" -A hmac-sha2-256 "This is secret HMAC 256 bits key";
add -4 172.31.0.1 10.11.1.3 esp 0x101 -m transport -E rijndael-cbc "This is secret AES 256 bits key!" -A hmac-sha2-256 "This is secret HMAC 256 bits key";

spdadd 10.11.1.3 172.31.0.1 any -P in ipsec esp/transport//require;
spdadd 172.31.0.1 10.11.1.3 any -P out ipsec esp/transport//require;


Results:
ping -W1 -c1 -S 10.11.1.3 172.31.0.1
PING 172.31.0.1 (172.31.0.1) from 10.11.1.3: 56 data bytes

tcpdump -ni em0
11:43:31.276852 IP 10.11.1.3 > 172.31.0.1: ESP(spi=0x00000100,seq=0x16), length 120
11:43:31.277594 IP 172.31.0.1 > 10.11.1.3: ICMP echo reply, id 49496, seq 0, length 64

Thank you for your help.
Comment 1 Andrey V. Elsukov freebsd_committer 2016-12-02 21:52:09 UTC
(In reply to Jérôme-Charles LALLEMAND from comment #0)
> I'm trying to nat packet incoming from enc0 to an other machine.
> 
> The RDR statement in pf works for the incoming packet, but the reply from
> the other machine is forwarded back to the issuer without encryption.
> 
> It might be because of the state matching on the reply, witch send back the
> reply bypassing the SPD rules.

No, this doesn't work, because PF consumes packet when it does "rdr". PF does output directly, thus a packet does not coming back to the IPsec.

> This is working flawlessly with encryption.
> 
> I’m in a gateway setup, is there any chances to get the RDR working with
> ipsec in transport mode ?

11:43:31.277594 IP 172.31.0.1 > 10.11.1.3: ICMP echo reply, id 49496, seq 0, length 64

I guess you have another rdr rule to make this work? I'm not quite familiar with PF, but probably you can use route via loopback interface and make rdr via it.
You need a route to the 10.11.1.3, so when encrypted packet will be routed via loopback interface, you can forward it to needed address..
Comment 2 Andrey V. Elsukov freebsd_committer 2016-12-02 22:01:11 UTC
(In reply to Andrey V. Elsukov from comment #1)
> (In reply to Jérôme-Charles LALLEMAND from comment #0)
> > I'm trying to nat packet incoming from enc0 to an other machine.

Ah, I missed "nat" keyword here. So, replies just translated back by NAT?
Can you show route table (netstat -rnf inet) from Computer2?
Comment 3 Jérôme-Charles LALLEMAND 2016-12-05 14:33:58 UTC
Thank you for your answers.
Yes, it seems that the answers, go directly using the nat state.

I got a route to 10.11.1.3, via em0.

172.31.0.1 is a loopback, advertised through em0

The routing table of Computer 2 follow:

netstat -rnf inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
10.56.0.0/16       10.56.1.1          UGS         em1
10.56.1.0/24       link#2             U           em1
10.56.1.10         link#2             UHS         lo0
10.11.1.0/24       10.112.1.9         UGS         em0
10.11.1.8/29       link#1             U           em0
10.11.1.11         link#1             UHS         lo0
127.0.0.1          link#6             UH          lo0
172.31.0.1         link#9             UH          lo0
192.168.0.0/16     10.56.1.1          UGS         em1
Comment 4 Jérôme-Charles LALLEMAND 2016-12-05 18:09:39 UTC
I've run some other tests, using a default gateway toward lo0 and route-to rules to force ipsec to apply on lo0 with the correct src/dst.

I don't understand where exactly the SPD apply :

I've tested à setup with 2 SPD rules f 
0.0.0.0/0 0.0.0.0/0 any -P in ipsec esp/transport//require;
0.0.0.0/0 0.0.0.0/0 any -P out ipsec esp/transport//require;

And still, some packets are leaving m'y box unencrypted, despite the require flag
Comment 5 Jérôme-Charles LALLEMAND 2016-12-05 18:10:22 UTC
I've run some other tests, using a default gateway toward lo0 and route-to rules to force ipsec to apply on lo0 with the correct src/dst.

I don't understand where exactly the SPD apply :

I've tested à setup with 2 SPD rules f 
0.0.0.0/0 0.0.0.0/0 any -P in ipsec esp/transport//require;
0.0.0.0/0 0.0.0.0/0 any -P out ipsec esp/transport//require;

And still, some packets are leaving m'y box unencrypted, despite the require flag
Comment 6 Andrey V. Elsukov freebsd_committer 2016-12-05 18:32:05 UTC
(In reply to Jérôme-Charles LALLEMAND from comment #5)
> I've run some other tests, using a default gateway toward lo0 and route-to
> rules to force ipsec to apply on lo0 with the correct src/dst.
> 
> I don't understand where exactly the SPD apply :
> 
> I've tested à setup with 2 SPD rules f 
> 0.0.0.0/0 0.0.0.0/0 any -P in ipsec esp/transport//require;
> 0.0.0.0/0 0.0.0.0/0 any -P out ipsec esp/transport//require;
> 
> And still, some packets are leaving m'y box unencrypted, despite the require
> flag

This how PF works. For some type of actions (route-to, nat, probably some others) it takes mbuf from the FreeBSD's network stack and then don't return it back. It does direct if_output() into network interface.

SPD applied in the ip_output().

ip_output()
  ip_ipsec_output()
    IPsec processing begins:
       call to if_enc()
          PF handles packet and does direct if_output
    IPsec thinks that packet was dropped by PF and didn't finish its work
----------

You may try to use ipfw(4), it reuses ip_output() for NAT'ed and forwarded packets. For ipfw this will work like:
ip_output()
  ip_ipsec_output()
    IPsec processing begins:
       call to if_enc()
          ipfw handles packet, do NAT then return to IPSec
    IPsec do encryption
    IPsec call ip_output() with encrypted packet
      ip_ipsec_output() decides that no IPsec required
      ipfw checks ESP packet
      if_output() using interface determined by route lookup
Comment 7 Andrey V. Elsukov freebsd_committer 2016-12-05 18:36:47 UTC
AFAIK, Franco Fichtner <franco at lastsummer dot de> is working on the solution, where PF will work like ipfw does.
Comment 8 Jérôme-Charles LALLEMAND 2016-12-07 10:19:48 UTC
I've tried IPFW and I get better results but I got still a problem with the reply packet.

On my gateway the SPD for the reply match correctly, because when I disable it, the reply is forwarded encrypted. But when I enable the SPD in rule, the packet disappear, I don't know where.

The SA corresponding seems OK, and I got no error log (ipsec.debug=1)

Here is my setup :


             10.160.64.3           10.112.1.3                172.31.0.1
                     LAN\         /WAN0                             em0\
                         \       /                                      \
|Client| ---->clear----> |Gateway| ---->ESP----> |Router| ---->ESP----> |Server|
       \                  FreeBSD                /      \
        \em0                                eth0/        \eth1
         10.160.64.100                10.112.1.1          172.31.0.254


The gateway is NATing source and destination. The setup works without encryption. But with encryption I can't get unencrypted reply to the client.
With encryption on, the rule 211 get no more match.



routing:
========
netstat -rnf inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.56.1.1          UGS         em1
10.56.1.0/24       link#2             U           em1
10.56.1.46         link#2             UHS         lo0
10.112.1.0/29      link#1             U          WAN0
10.112.1.3         link#1             UHS         lo0
10.160.64.0/24     link#3             U           LAN
10.160.64.3        link#3             UHS         lo0
127.0.0.1          link#5             UH          lo0
172.31.0.1         10.112.1.1         UGHS       WAN0

sysctl values:
==============

sysctl net.inet.ip.forwarding=1
sysctl net.inet.ip.fw.one_pass=0
sysctl net.enc.out.ipsec_bpf_mask=1
sysctl net.enc.out.ipsec_filter_mask=1
sysctl net.enc.in.ipsec_bpf_mask=2
sysctl net.enc.in.ipsec_filter_mask=2


ipfw rules:
===========

ipfw -q -f flush
ipfw -q add 100 allow ip from any to any via lo0
# Admin iface
ipfw -q add 101 allow ip from any to any via em1

ipfw nat 10 config if LAN redirect_addr 172.31.0.1 10.160.64.3
ipfw nat 11 config ip 10.112.1.3

ipfw add 200 nat 10 tag 10 ip4 from any to 10.160.64.3 in recv LAN
ipfw add 201 nat 11 tagged 10 out xmit WAN0
ipfw add 210 nat 11 tag 11 ip4 from any to 10.112.1.3 in recv WAN0
ipfw add 211 nat 10 out xmit LAN

ipfw add 10000 allow ip4 from any to any


setkey.conf:
============

flush;
spdflush;

add -4 10.160.64.100 172.31.0.1 esp 0x100 -m transport -E rijndael-cbc "This is secret AES 256 bits key0";
add -4 172.31.0.1 10.160.64.100 esp 0x101 -m transport -E rijndael-cbc "This is secret AES 256 bits key1";

spdadd 10.160.64.100 172.31.0.1 any -P out ipsec esp/transport//require;
spdadd 172.31.0.1 10.160.64.100 any -P in ipsec esp/transport//require;
Comment 9 Andrey V. Elsukov freebsd_committer 2016-12-07 11:15:35 UTC
(In reply to Jérôme-Charles LALLEMAND from comment #8)
> I've tried IPFW and I get better results but I got still a problem with the
> reply packet.

If I understand correctly, do you want to see on the WAN ESP packets with following addresses 10.112.1.3 <-> 172.31.0.1?

> On my gateway the SPD for the reply match correctly, because when I disable
> it, the reply is forwarded encrypted. But when I enable the SPD in rule, the
> packet disappear, I don't know where.
> 
> The SA corresponding seems OK, and I got no error log (ipsec.debug=1)
> 
> Here is my setup :
> 
> 
>              10.160.64.3           10.112.1.3                172.31.0.1
>                      LAN\         /WAN0                             em0\
>                          \       /                                      \
> |Client| ---->clear----> |Gateway| ---->ESP----> |Router| ---->ESP---->
> |Server|
>        \                  FreeBSD                /      \
>         \em0                                eth0/        \eth1
>          10.160.64.100                10.112.1.1          172.31.0.254
> 
> 
> The gateway is NATing source and destination. The setup works without
> encryption. But with encryption I can't get unencrypted reply to the client.
> With encryption on, the rule 211 get no more match.
> 
> routing:
> ========
> netstat -rnf inet
> Routing tables
> 
> Internet:
> Destination        Gateway            Flags     Netif Expire
> default            10.56.1.1          UGS         em1
> 10.56.1.0/24       link#2             U           em1
> 10.56.1.46         link#2             UHS         lo0
> 10.112.1.0/29      link#1             U          WAN0
> 10.112.1.3         link#1             UHS         lo0
> 10.160.64.0/24     link#3             U           LAN
> 10.160.64.3        link#3             UHS         lo0
> 127.0.0.1          link#5             UH          lo0
> 172.31.0.1         10.112.1.1         UGHS       WAN0
> 
> sysctl values:
> ==============
> 
> sysctl net.inet.ip.forwarding=1
> sysctl net.inet.ip.fw.one_pass=0
> sysctl net.enc.out.ipsec_bpf_mask=1
> sysctl net.enc.out.ipsec_filter_mask=1
> sysctl net.enc.in.ipsec_bpf_mask=2
> sysctl net.enc.in.ipsec_filter_mask=2
> 
> 
> ipfw rules:
> ===========
> 
> ipfw -q -f flush
> ipfw -q add 100 allow ip from any to any via lo0
> # Admin iface
> ipfw -q add 101 allow ip from any to any via em1
> 
> ipfw nat 10 config if LAN redirect_addr 172.31.0.1 10.160.64.3
> ipfw nat 11 config ip 10.112.1.3
> 
> ipfw add 200 nat 10 tag 10 ip4 from any to 10.160.64.3 in recv LAN
> ipfw add 201 nat 11 tagged 10 out xmit WAN0
> ipfw add 210 nat 11 tag 11 ip4 from any to 10.112.1.3 in recv WAN0
> ipfw add 211 nat 10 out xmit LAN
> 
> ipfw add 10000 allow ip4 from any to any
> 
> 
> setkey.conf:
> ============
> 
> flush;
> spdflush;
> 
> add -4 10.160.64.100 172.31.0.1 esp 0x100 -m transport -E rijndael-cbc "This
> is secret AES 256 bits key0";
> add -4 172.31.0.1 10.160.64.100 esp 0x101 -m transport -E rijndael-cbc "This
> is secret AES 256 bits key1";
> 
> spdadd 10.160.64.100 172.31.0.1 any -P out ipsec esp/transport//require;
> spdadd 172.31.0.1 10.160.64.100 any -P in ipsec esp/transport//require;

Inbound traffic is handled using corresponding SA. If ESP packet was not destined to your IP, it will not be decrypted. After decryption it will be checked to match with security policy.

So, If I understand your needs correctly, you need something like:

spdadd 10.160.64.100 172.31.0.1 any -P out ipsec  esp/transport//require;
spdadd 172.31.0.1 10.112.1.3 any -P in ipsec esp/transport//require;

add -4 10.160.64.100 172.31.0.1 esp 0x100 -m transport -E rijndael-cbc "This is secret AES 256 bits key0";
add -4 172.31.0.1 10.112.1.3 esp 0x101 -m transport -E rijndael-cbc "This is secret AES 256 bits key1";

Then configure NAT:
ipfw nat 11 config ip 10.112.1.3
ipfw add nat 11 ip from any to any via enc0

Configure if_enc:
net.enc.out.ipsec_filter_mask=1
net.enc.in.ipsec_filter_mask=1

How it should work:

ping -S 10.160.64.100 172.31.0.1

Gateway receives packet on the LAN interface. It routes packet via the WAN0 interface. Then it founds security policy and does IPsec transform. It does lookup for corresponding SA, this is SA with SPI 0x100. Then it calls to ipfw, where packet will be translated to ICMP 10.112.1.3 -> 172.31.0.1, then we should continue IPsec processing and encrypt packet. Then it just will be routed to 172.31.0.1.

When we receive the reply, it should be ESP packet from 172.31.0.1 to 10.112.1.3 with SPI 0x101.
Since the packet is destined to us, ESP protocol handler will try to decrypt it. It will do SA lookup for ESP proto with SPI 0x101 and destination 10.112.1.3. And it should find our SA. Then packet will be decrypted and
after that it will be passed to ipfw. ipfw nat should translate it back to the ICMP 172.31.0.1->10.160.64.100 and return to IPsec processing. Since there no additional IPsec processing required, packet will be routed to 10.160.64.100.

Something similar should be done on the 172.31.0.1 side.
Comment 10 Jérôme-Charles LALLEMAND 2016-12-07 11:32:08 UTC
> Inbound traffic is handled using corresponding SA. If ESP packet was 
> not destined to your IP, it will not be decrypted. After decryption it 
> will be checked to match with security policy.

Thank you for your useful explanations. I understand way better now how it works.

> If I understand correctly, do you want to see on the WAN ESP packets
> with following addresses 10.112.1.3 <-> 172.31.0.1?

Exactly my needs.


I want to do RDR and NAT on the gateway.

My test command on the client is : ping -S 10.160.64.100 10.160.64.3

I'll adapt your solution to do:
1/ RDR on LAN
2/ ipsec encryption
3/ NAT on enc0