Bug 202885 - IPsec performance problems with fragmented ESP and packet loss
Summary: IPsec performance problems with fragmented ESP and packet loss
Status: Closed Unable to Reproduce
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: George V. Neville-Neil
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-09-04 03:15 UTC by cmb
Modified: 2016-08-29 16:15 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description cmb 2015-09-04 03:15:43 UTC
Take the following setup: 

[LAN A] -- (host A) -- [WAN] -- (host B) -- [LAN B] 

Where "host A" and "host B" are 11-CURRENT boxes as of Sept 2, running GENERIC-NODEBUG. Setup IPsec tunnel between the two, to route between LAN A and B. Doesn't matter what specifics are used in the IPsec config. 

Where hosts on the LAN sides have their MTU at 1500, so ESP traffic ends up being fragmented, and there is packet loss on WAN, serious performance problems are encountered. With no loss on WAN, no issue. 

Do the following on host A: 
ipfw pipe 1 config 100Mbit/s
ipfw pipe 2 config 100Mbit/s
ipfw add 10 pipe 1 ip from any to 1.2.3.4
ipfw add 11 pipe 2 ip from 1.2.3.4 to any 

where its WAN IP is 1.2.3.4, and push some traffic over the VPN. I'm doing 'fetch -o /dev/null ...' from a web server on the opposite side. Your throughput drops off to near 0 repeatedly. Wireshark I/O graph showing this: 
https://files.pfsense.org/cmb/100m-limit-frag.png

change the MTU on the LAN hosts to 1350 to eliminate ESP fragmentation and you get the expected steady throughput: 
https://files.pfsense.org/cmb/100m-limit-no-frag.png
Comment 1 marco 2015-09-19 09:52:35 UTC
I have a case that I think is related.

This is my setup:

Tunnel using gif interface between 2 hosts to do IPv6 over IPv4.

Lets call the host providing the IPv6 tunnel Host A.
Lets call the host receiving IPv6 through the tunnel Host B.

Host A kernel is: r286965.
Host A $(freebsd-version) is: 10.2-RELEASE-p2

Host B kernel is: r286823M
Host B $(freebsd-version) is: 10.2-RELEASE-p2

In a default scenario these are the traffic properties:
- IPv4 Host A => Host B - wire speed
- IPv4 Host B => Host A - wire speed
- IPv6 Host A => Host B - wire speed
- IPv6 Host B => Host A - Lots of retransmits and ~4KB/s

When I add this route to host B the problem for traffic "IPv6 Host B => Host A" gets solved:
route -6 add 2000::/3 2a02:898:57:ff00::4 -mtu 1279

The default mtu of course is 1280.  This smells like an off-by-one somewhere no? :-)

Thanks in advance for looking at this, and please do let me know when you need more information.
Comment 2 George V. Neville-Neil freebsd_committer freebsd_triage 2015-12-04 02:47:50 UTC
Howdy,

Can you put up a full netstat -s both before and after you try this?

Also, can you post a copy of the tcpdump output?

Thanks
Comment 3 George V. Neville-Neil freebsd_committer freebsd_triage 2015-12-04 02:57:14 UTC
One more request, please take a look at the fragment sysctls:

net.inet.ip.maxfragpackets
net.inet.ip.fragpackets
Comment 4 George V. Neville-Neil freebsd_committer freebsd_triage 2015-12-04 02:57:14 UTC
One more request, please take a look at the fragment sysctls:

net.inet.ip.maxfragpackets
net.inet.ip.fragpackets
Comment 5 cmb 2015-12-04 10:02:49 UTC
gnn: will do, and I'll get you into the test setup as well. Need to recreate it.
Comment 6 marco 2015-12-04 15:17:14 UTC
Hi, thanks for looking into this.

From my end of this, in the mean time my host has restarted which made the issue go away.  This is very unexpected as I usually take great care in making sure a reboot will not result in a different configuration.

I am expecting a bit that what I was seeing may have been caused by a counter running over or similar, although I did not do a great deal of traffic across the tunnel.

Long story short: Unfortunately I cannot recreate the situation for which I'm sorry :S

Thanks again,

Marco van Tol
Comment 7 George V. Neville-Neil freebsd_committer freebsd_triage 2016-08-29 16:15:22 UTC
Reporter cannot reproduce.