Bug 121642 - [ipsec] [gif]: When using IPSec, tcpdump shows non-decapsulated packets on gif interface
Summary: [ipsec] [gif]: When using IPSec, tcpdump shows non-decapsulated packets on gi...
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 7.0-RELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-03-12 11:50 UTC by Alexander Zagrebin
Modified: 2019-05-20 11:09 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Zagrebin 2008-03-12 11:50:00 UTC
I'm using ipsec-protected tunnel between 2 hosts.
The first one (HostA) is running 6.3-RELEASE and second one (HostB) - 7.0-RELEASE.
The network topology:

     ------- public network ------- 
        |                      |
        |                      |
     A.A.A.A                B.B.B.B
      Host A                 Host B
   6.3-RELEASE            7.0-RELEASE
   192.168.0.2            192.168.1.1
        |                      |
        |                      |
     private                private
net 192.168.0.0/24    net 192.168.1.0/24

After creating the tunnel using gif interfaces without IPSec, I can successfully
ping host 192.168.0.10 from 192.168.1.2.
tcpdump on gif interfaces shows:

HostA# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:21:20.861465 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
21:21:20.862060 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40

HostB# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:21:25.012390 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
21:21:25.014457 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40

All right.

Now I'm starting IPSec.

ipsec.conf on HostA:
spdadd B.B.B.B/32 A.A.A.A/32 ipencap
    -P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
spdadd A.A.A.A/32 B.B.B.B/32 ipencap
    -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;

ipsec.conf on HostB:
spdadd A.A.A.A/32 B.B.B.B/32 ipencap
    -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
spdadd B.B.B.B/32 A.A.A.A/32 ipencap
    -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;

Pinging 192.168.0.10 from 192.168.1.2 is successful again, but
tcpdump shows:

HostA# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:24:12.429778 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
21:24:12.430246 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 51969, length 40

HostB# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2: 
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ICMP echo reply, id 512, seq 51969, length 40 (ipip-proto-4)

Oops!
tcpdump shows non-decapsulated packets, but tunnel works fine.

How-To-Repeat: See above
Comment 1 Remko Lodder freebsd_committer 2008-03-12 12:24:14 UTC
Responsible Changed
From-To: freebsd-bugs->gnn

Hi George, can you have a look at this one please?
Comment 2 Remko Lodder freebsd_committer 2008-03-12 12:42:07 UTC
Responsible Changed
From-To: gnn->bz

Reassign to Bjoern on his request
Comment 3 Volker 2008-03-12 16:52:15 UTC
> Now I'm starting IPSec.
> 
> ipsec.conf on HostA:
> spdadd B.B.B.B/32 A.A.A.A/32 ipencap
> -P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
> spdadd A.A.A.A/32 B.B.B.B/32 ipencap
> -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
> 
> ipsec.conf on HostB:
> spdadd A.A.A.A/32 B.B.B.B/32 ipencap
> -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
> spdadd B.B.B.B/32 A.A.A.A/32 ipencap
> -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
> 
> Pinging 192.168.0.10 from 192.168.1.2 is successful again, but
> tcpdump shows:
> 
> HostA# tcpdump -ni gif0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
> 21:24:12.429778 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
> 21:24:12.430246 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 51969, length 40
> 
> HostB# tcpdump -ni gif0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
> 21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
> 21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2:

hmm, I'm wondering if `ping -S A.A.A.A B.B.B.B' (executed at Host A) may
serve your expectations?

Actually I can smell some misconfigurations here. ;)

Volker
Comment 4 Volker 2008-03-12 17:24:49 UTC
Sorry, I've been giving just half of an answer with my previous post (as
you're trying to use gif).

Say, you're using a policy like:

spdadd 192.168.0.0/24 192.168.1.0/24 ipencap
 -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;

then trying if `ping -S 192.168.0.1 192.168.1.1' may magically let you
see encrypted traffic.
Comment 5 Bjoern A. Zeeb freebsd_committer 2008-03-12 23:42:41 UTC
On Wed, 12 Mar 2008, Alexander Zagrebin wrote:

Hi,

> I'm using ipsec-protected tunnel between 2 hosts.
> The first one (HostA) is running 6.3-RELEASE and second one (HostB) - 7.0-RELEASE.
> The network topology:
>
>     ------- public network -------
>        |                      |
>        |                      |
>     A.A.A.A                B.B.B.B
>      Host A                 Host B
>   6.3-RELEASE            7.0-RELEASE
>   192.168.0.2            192.168.1.1
>        |                      |
>        |                      |
>     private                private
> net 192.168.0.0/24    net 192.168.1.0/24
>
> After creating the tunnel using gif interfaces without IPSec, I can successfully
> ping host 192.168.0.10 from 192.168.1.2.
> tcpdump on gif interfaces shows:
>
> HostA# tcpdump -ni gif0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
> 21:21:20.861465 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
> 21:21:20.862060 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40
>
> HostB# tcpdump -ni gif0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
> 21:21:25.012390 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
> 21:21:25.014457 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40
>
> All right.
>
> Now I'm starting IPSec.
>
> ipsec.conf on HostA:
> spdadd B.B.B.B/32 A.A.A.A/32 ipencap
>    -P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
> spdadd A.A.A.A/32 B.B.B.B/32 ipencap
>    -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
>
> ipsec.conf on HostB:
> spdadd A.A.A.A/32 B.B.B.B/32 ipencap
>    -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
> spdadd B.B.B.B/32 A.A.A.A/32 ipencap
>    -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;



If you are only using IPsec to protect the 'gif tunnel'
you could use transport mode.

If you do not need the real interface for a link state
routing protocol or similar you could simply use tunnel
mode for the entire 'private nets' without the gif tunnel.


Now that's not your problem, let's see..


> Pinging 192.168.0.10 from 192.168.1.2 is successful again, but
> tcpdump shows:
>
> HostA# tcpdump -ni gif0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
> 21:24:12.429778 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
> 21:24:12.430246 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 51969, length 40
>
> HostB# tcpdump -ni gif0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
> 21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
> 21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2:
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ICMP echo reply, id 512, seq 51969, length 40 (ipip-proto-4)
>
> Oops!
> tcpdump shows non-decapsulated packets, but tunnel works fine.

ok, so that's the input path where you are seeing this. let's see what
happens there... ( I simplified things the further on I got as it's
long ...)

ip_input (IP, ESP)
ip_ipsec_filtertunnel() false -> cont.
pfil_run_hooks
ip_ipsec_input
ipsec_in_reject (cont)
dispatch(ESPv4)
esp4_input
ipsec4_common_input
ipsec_common_input
dispatch(XF_ESP)
esp_input
crypto_dispatch
esp_input_cb	[ strip esp header -> outer IPIP header now ]
ipsec_common_input_cb
ipsec4_common_input_cb
enc(BEFORE)
netisr_queue(NETISR_IP)
ip_input
filter (here we see the ipencap from the ipsec on the incoming if)
dispatch(ipencap)
encap4_input
ip_input
_ipip_input (remove outer IP header (the one from ouside ESP) ->
              inner ESP IP header + ULP which is inner gif IP header left)
              -> IP:ULP == IP:IP again)
enc(AFTER)
netisr_queue
ip_input
filter
dispatch(upper layer protocol == IP -> ipencap)
encap4_input
in_gif_input
m_adj(m, off) [should cut off the first IP header -> IP:ULP == IP:ICMP left]
gif_input
BPF << there you should see it with one IP header left.
dispatch(IP)
ip_input
...


I wonder what I am missing atm, it's a bit late here. I'll try to find
the clue tomo^H^H^H^Hlater today.

I knew there are things that happen differently, especially the IPIP
seen in ip_input before the call to _ipip_input but I wasn't aware
that there were later problems as well and basically should not be.


-- 
Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
Software is harder than hardware  so better get it right the first time.
Comment 6 Bjoern A. Zeeb freebsd_committer 2008-03-13 09:55:23 UTC
On Wed, 12 Mar 2008, Bjoern A. Zeeb wrote:

Hi,

I had suspected the right thing while thinking about this after
writing the reply but I now also read the code to validate this.

Basically things seem to work as expected.

I wonder if you are also seeing the inner packet only as well
(two packets in the outgoing direction)?

Can you confirm this?



I'll add comments inline further down.


> > HostB# tcpdump -ni gif0
> > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
> > 21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
> > 21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2:
> >                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > ICMP echo reply, id 512, seq 51969, length 40 (ipip-proto-4)
> >
> > Oops!
> > tcpdump shows non-decapsulated packets, but tunnel works fine.
>
> ok, so that's the input path where you are seeing this. let's see what
> happens there... ( I simplified things the further on I got as it's
> long ...)
>
> ip_input (IP, ESP)
> ip_ipsec_filtertunnel() false -> cont.
> pfil_run_hooks
> ip_ipsec_input
> ipsec_in_reject (cont)
> dispatch(ESPv4)
> esp4_input
> ipsec4_common_input
> ipsec_common_input
> dispatch(XF_ESP)
> esp_input
> crypto_dispatch
> esp_input_cb	[ strip esp header -> outer IPIP header now ]
> ipsec_common_input_cb
> ipsec4_common_input_cb
> enc(BEFORE)
> netisr_queue(NETISR_IP)
> ip_input
> filter (here we see the ipencap from the ipsec on the incoming if)
> dispatch(ipencap)
> encap4_input

This is walking through checking all possible candidates, which would
be:

- ipe4_protosw (_ipip_input from xform_ipip.c)
- in_gif_protosw (gif_encapcheck from in_gif.c/if_gif.c)

the gif is returning the higher priority thus it gets the packet.

so from here on it goes like this:

in_gif_input
m_adj(m, off) [should cut off the first IP header -> IP:ULP == IPIP left]
gif_input
BPF << here you are now seeing the IPIP as above
dispatch(IP)
ip_input
filter (ipencap again)
dispatch(ipencap)	with just esxactly the same IPIP headers again as above
and to my understanding it now should go through 
encap4_input
in_gif_input
...
again and you should see it a 2nd time in BPF.

Can you confirm this?




So basically this is a general problem with the IPsec/ipencap code.

I have on my todo list:
9 ipsec improvements: do not go through ip_input three times


There are multiple solutions to fix that but all have to be thought of
carefully to not break other assumptions.


The most important problem with the current code is that for
if_enc you will not see the packet for filtering but you will see it
in ip_input more often.



> ip_input
   ^^ this should have been ip4_input (from xform_ipip.c btw)
> _ipip_input (remove outer IP header (the one from ouside ESP) ->
>               inner ESP IP header + ULP which is inner gif IP header left)
>               -> IP:ULP == IP:IP again)
> enc(AFTER)
> netisr_queue
> ip_input
> filter
> dispatch(upper layer protocol == IP -> ipencap)
> encap4_input
> in_gif_input
> m_adj(m, off) [should cut off the first IP header -> IP:ULP == IP:ICMP left]
> gif_input
> BPF << there you should see it with one IP header left.
> dispatch(IP)
> ip_input
> ...

-- 
Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
Software is harder than hardware  so better get it right the first time.
Comment 7 Bjoern A. Zeeb freebsd_committer 2008-03-13 09:58:00 UTC
State Changed
From-To: open->feedback

Wait for feedback whether things go through gif/bpf twice or not.
Comment 8 Bjoern A. Zeeb freebsd_committer 2008-03-13 14:30:59 UTC
On Thu, 13 Mar 2008, Alexander Zagrebin wrote:

Hi,

>> I wonder if you are also seeing the inner packet only as well
>> (two packets in the outgoing direction)?
>>
>> Can you confirm this?
>
> If I have correctly understood, you suppose, that tcpdump may show 2
> outgoing packets instead 1?
> No. It shows one packet on each outgoing one.

hmm, ok , then something else is going on still. I guess I need to
really trace packets.


> If it can help, after switching the tunnel to transport mode,
> tcpdump shows traffic via gif absolutely correctly.
> So, the problem exists when using gif and ipsec in tunnel mode only.

Makes perfectly sense.

With tunnel mode you end up with:

IP | ESP | IP | IP | ULP
reducing to
IP | IP | IP | ULP (1st ipip decap|gif, gif wins)
                    (in_gif_input removes the first header)
BPF happens
IP | IP | ULP      (2nd ipip decap|gif --- I thought gif wins
 			again and we'd see it a 2nd time in bpf)
IP | ULP


With transport mode it's like:
IP | ESP | IP | ULP
IP | IP | ULP      (1st ipip decap|gif, gif wins)
IP | ULP           (in_gif_input removes the first header)
BPF happens

-- 
Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
Software is harder than hardware  so better get it right the first time.
Comment 9 Bjoern A. Zeeb freebsd_committer 2008-03-13 14:41:43 UTC
State Changed
From-To: feedback->open

Feedback recevied. Needs for analysing.
Comment 10 Andrey V. Elsukov freebsd_committer 2014-04-04 10:35:45 UTC
Responsible Changed
From-To: bz->ae

I'm working on this now.
Comment 11 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 07:58:54 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped
Comment 12 Andrey V. Elsukov freebsd_committer 2019-05-20 11:09:51 UTC
I tested many different types of tunnels w/ and w/o IPsec, and I believe all possible configuration should work correctly. Of course there is still be possible some ambiguous configurations, but an administrator always can explicitly specify IPsec mode for SAs, and then it will work as expected.