Bug 202885

Summary:	IPsec performance problems with fragmented ESP and packet loss
Product:	Base System	Reporter:	cmb
Component:	kern	Assignee:	George V. Neville-Neil <gnn>
Status:	Closed Unable to Reproduce
Severity:	Affects Many People	CC:	gnn, jim, marco, meyer.sydney, ports
Priority:	---
Version:	CURRENT
Hardware:	Any
OS:	Any

Description cmb 2015-09-04 03:15:43 UTC

Take the following setup: 

[LAN A] -- (host A) -- [WAN] -- (host B) -- [LAN B] 

Where "host A" and "host B" are 11-CURRENT boxes as of Sept 2, running GENERIC-NODEBUG. Setup IPsec tunnel between the two, to route between LAN A and B. Doesn't matter what specifics are used in the IPsec config. 

Where hosts on the LAN sides have their MTU at 1500, so ESP traffic ends up being fragmented, and there is packet loss on WAN, serious performance problems are encountered. With no loss on WAN, no issue. 

Do the following on host A: 
ipfw pipe 1 config 100Mbit/s
ipfw pipe 2 config 100Mbit/s
ipfw add 10 pipe 1 ip from any to 1.2.3.4
ipfw add 11 pipe 2 ip from 1.2.3.4 to any 

where its WAN IP is 1.2.3.4, and push some traffic over the VPN. I'm doing 'fetch -o /dev/null ...' from a web server on the opposite side. Your throughput drops off to near 0 repeatedly. Wireshark I/O graph showing this: 
https://files.pfsense.org/cmb/100m-limit-frag.png

change the MTU on the LAN hosts to 1350 to eliminate ESP fragmentation and you get the expected steady throughput: 
https://files.pfsense.org/cmb/100m-limit-no-frag.png

Comment 1 marco 2015-09-19 09:52:35 UTC

I have a case that I think is related.

This is my setup:

Tunnel using gif interface between 2 hosts to do IPv6 over IPv4.

Lets call the host providing the IPv6 tunnel Host A.
Lets call the host receiving IPv6 through the tunnel Host B.

Host A kernel is: r286965.
Host A $(freebsd-version) is: 10.2-RELEASE-p2

Host B kernel is: r286823M
Host B $(freebsd-version) is: 10.2-RELEASE-p2

In a default scenario these are the traffic properties:
- IPv4 Host A => Host B - wire speed
- IPv4 Host B => Host A - wire speed
- IPv6 Host A => Host B - wire speed
- IPv6 Host B => Host A - Lots of retransmits and ~4KB/s

When I add this route to host B the problem for traffic "IPv6 Host B => Host A" gets solved:
route -6 add 2000::/3 2a02:898:57:ff00::4 -mtu 1279

The default mtu of course is 1280.  This smells like an off-by-one somewhere no? :-)

Thanks in advance for looking at this, and please do let me know when you need more information.

Comment 2 George V. Neville-Neil freebsd_committer

2015-12-04 02:47:50 UTC

Howdy,

Can you put up a full netstat -s both before and after you try this?

Also, can you post a copy of the tcpdump output?

Thanks

Comment 3 George V. Neville-Neil freebsd_committer

2015-12-04 02:57:14 UTC

One more request, please take a look at the fragment sysctls:

net.inet.ip.maxfragpackets
net.inet.ip.fragpackets

Comment 4 George V. Neville-Neil freebsd_committer

2015-12-04 02:57:14 UTC

One more request, please take a look at the fragment sysctls:

net.inet.ip.maxfragpackets
net.inet.ip.fragpackets

Comment 5 cmb 2015-12-04 10:02:49 UTC

gnn: will do, and I'll get you into the test setup as well. Need to recreate it.

Comment 6 marco 2015-12-04 15:17:14 UTC

Hi, thanks for looking into this.

From my end of this, in the mean time my host has restarted which made the issue go away.  This is very unexpected as I usually take great care in making sure a reboot will not result in a different configuration.

I am expecting a bit that what I was seeing may have been caused by a counter running over or similar, although I did not do a great deal of traffic across the tunnel.

Long story short: Unfortunately I cannot recreate the situation for which I'm sorry :S

Thanks again,

Marco van Tol

Comment 7 George V. Neville-Neil freebsd_committer

2016-08-29 16:15:22 UTC

Reporter cannot reproduce.