Bug 237197 - ipnat dropping connections after upgrade to 11.2
Summary: ipnat dropping connections after upgrade to 11.2
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Cy Schubert
URL:
Keywords: regression
Depends on: 208566
Blocks:
  Show dependency treegraph
 
Reported: 2019-04-11 09:19 UTC by Mike
Modified: 2019-04-17 03:06 UTC (History)
4 users (show)

See Also:
koobs: mfc-stable12+
koobs: mfc-stable11+


Attachments
ipfstat dump (2.57 KB, text/plain)
2019-04-11 10:38 UTC, Mike
no flags Details
ipnat -s dump (2.11 KB, text/plain)
2019-04-11 10:38 UTC, Mike
no flags Details
ipfstat with stock kernel (2.57 KB, text/plain)
2019-04-11 14:14 UTC, Mike
no flags Details
ipnat -s with stock kernel (2.11 KB, text/plain)
2019-04-11 14:15 UTC, Mike
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike 2019-04-11 09:19:42 UTC
Hello,

I am runnning VPN gateways based on openvpn / Freebsd / ipnat.

This setup worked flawlessy for years until I upgraded from Freebsd 9 to Freebsd 11.2 or 12.0.

Without changing anything in the software setup, some connections are getting dropped, the nat drop count is steadily increasing over time (a few hours from reboot) and most people are complaining about speed issues .... Not good.

Hardware is the same, the only thing changed is the O.S version.

Kernel is stock kernel with the #define LARGE_NAT in the ip_nat.h.

After spending a week trying to chase ghosts, I'm quite puzzled.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-11 10:03:10 UTC
@Mike If you could include the full network configuration of the system (as attachments), that would be appreciated. 

Please sanitize where necessary (openvpn authentication details, ip's, etc). This should include:

Is the issue reproducible with a stock (GENERIC) kernel without the LARGE_NAT define? If you haven't tested that, please do.
Comment 2 Mike 2019-04-11 10:38:30 UTC
Created attachment 203582 [details]
ipfstat dump
Comment 3 Mike 2019-04-11 10:38:58 UTC
Created attachment 203583 [details]
ipnat -s dump
Comment 4 Mike 2019-04-11 10:41:07 UTC
Attached ipnat -s and ipfstat dumps 

I will try with a genuine kernel without any modifications and let you know if that improves things.

What puzzles me is that the environment has not changed a bit. Same openvpn scripts, same nat rules, same hardware and IP addresses ....

Thank you :-)
Comment 5 frank 2019-04-11 11:43:15 UTC
Please try:

   /sbin/ipf -T nat_maxbucket=2047

This solved it for me (I do have a bunch of ipfilter patches in my 11.2 kernel from Cy Schubert)
Comment 6 Mike 2019-04-11 14:13:54 UTC
Tried your nat_maxbucket increase. It did not help

Thanks anyways :-)
Comment 7 Mike 2019-04-11 14:14:31 UTC
Created attachment 203593 [details]
ipfstat with stock kernel
Comment 8 Mike 2019-04-11 14:15:05 UTC
Created attachment 203594 [details]
ipnat -s with stock kernel

Not really better :-)
Comment 9 Mike 2019-04-12 07:41:03 UTC
Well, in a desperate move, I have upgraded to 12.0 and the problem magically disappeared ...
Comment 10 frank 2019-04-12 08:22:48 UTC
That makes sense, Cy did not back-port all ipfilter patches into 11.2-RELEASE (not sure about 11-STABLE).
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-16 23:11:15 UTC
Can we identify a set of bugs (PR's) and/or commits that can/should be merged?
Comment 12 Cy Schubert freebsd_committer freebsd_triage 2019-04-17 01:44:23 UTC
11.2-RELEASE does not have nor will it have the ipnat patches applied. You must wait for 11.3-RELEASE, which IIRC should be GA over summer sometime.

You can svnup your sources to 11-STABLE, which has the fix, or install the patch directly to your 11.2-RELEASE instead.

All ipfilter patches have been MFCed to 11-STABLE and 12-STABLE. Some have been MFCed to 10-STABLE however since -CURRENT has diverged too much from 10-STABLE I an no longer MFCing to it any more.

Wait for 11.3-RELEASE.
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-17 02:16:40 UTC
(In reply to Cy Schubert from comment #12)

I understood the request as being for any changes not yet merged (if any were identified), to be merged to stable/11, not releng/11.2 (where only major 'errata/security' bugs are merged). From your comment, that seems that the fix to this issue has already been resolved, and merged to both (11, 12) stable branches, with fixes to be included in 11.3-RELEASE.

Do any other PR exist that references the relevant ipnat commits, so this bug can be set to depend on them?

With @triage hat:

- Set resolution FIXED: with resolution: update to stable/11 or stable/12
- Assign to committer that resolved (thank you Cy!)
- Set mfc-* flags to track that resolution was merged to both (11, 12) stable branches
Comment 14 Cy Schubert freebsd_committer freebsd_triage 2019-04-17 02:33:16 UTC
This was merged to stable/12 and stable/11 by r338047 (I merge to stable/11 and stable/12 in the same commit).

PR/208566 originally documented this bug and is resolved. This PR is now dependent on that one.