Take the following setup: [LAN A] -- (host A) -- [WAN] -- (host B) -- [LAN B] Where "host A" and "host B" are 11-CURRENT boxes as of Sept 2, running GENERIC-NODEBUG. Setup IPsec tunnel between the two, to route between LAN A and B. Doesn't matter what specifics are used in the IPsec config. Where hosts on the LAN sides have their MTU at 1500, so ESP traffic ends up being fragmented, and there is packet loss on WAN, serious performance problems are encountered. With no loss on WAN, no issue. Do the following on host A: ipfw pipe 1 config 100Mbit/s ipfw pipe 2 config 100Mbit/s ipfw add 10 pipe 1 ip from any to 1.2.3.4 ipfw add 11 pipe 2 ip from 1.2.3.4 to any where its WAN IP is 1.2.3.4, and push some traffic over the VPN. I'm doing 'fetch -o /dev/null ...' from a web server on the opposite side. Your throughput drops off to near 0 repeatedly. Wireshark I/O graph showing this: https://files.pfsense.org/cmb/100m-limit-frag.png change the MTU on the LAN hosts to 1350 to eliminate ESP fragmentation and you get the expected steady throughput: https://files.pfsense.org/cmb/100m-limit-no-frag.png
I have a case that I think is related. This is my setup: Tunnel using gif interface between 2 hosts to do IPv6 over IPv4. Lets call the host providing the IPv6 tunnel Host A. Lets call the host receiving IPv6 through the tunnel Host B. Host A kernel is: r286965. Host A $(freebsd-version) is: 10.2-RELEASE-p2 Host B kernel is: r286823M Host B $(freebsd-version) is: 10.2-RELEASE-p2 In a default scenario these are the traffic properties: - IPv4 Host A => Host B - wire speed - IPv4 Host B => Host A - wire speed - IPv6 Host A => Host B - wire speed - IPv6 Host B => Host A - Lots of retransmits and ~4KB/s When I add this route to host B the problem for traffic "IPv6 Host B => Host A" gets solved: route -6 add 2000::/3 2a02:898:57:ff00::4 -mtu 1279 The default mtu of course is 1280. This smells like an off-by-one somewhere no? :-) Thanks in advance for looking at this, and please do let me know when you need more information.
Howdy, Can you put up a full netstat -s both before and after you try this? Also, can you post a copy of the tcpdump output? Thanks
One more request, please take a look at the fragment sysctls: net.inet.ip.maxfragpackets net.inet.ip.fragpackets
gnn: will do, and I'll get you into the test setup as well. Need to recreate it.
Hi, thanks for looking into this. From my end of this, in the mean time my host has restarted which made the issue go away. This is very unexpected as I usually take great care in making sure a reboot will not result in a different configuration. I am expecting a bit that what I was seeing may have been caused by a counter running over or similar, although I did not do a great deal of traffic across the tunnel. Long story short: Unfortunately I cannot recreate the situation for which I'm sorry :S Thanks again, Marco van Tol
Reporter cannot reproduce.