Bug 257106 - Local-origin connections matching 'pass out' rules with 'route-to' fail
Summary: Local-origin connections matching 'pass out' rules with 'route-to' fail
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.2-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-pf (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-07-11 10:28 UTC by Mark C
Modified: 2021-08-07 16:32 UTC (History)
1 user (show)

See Also:


Attachments
Extended version of the route_to.sh test script, including multiwanlocal test (7.21 KB, text/plain)
2021-07-13 15:31 UTC, Mark C
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mark C 2021-07-11 10:28:01 UTC
Running FreeBSD 12-stable, I'm running into a problem with commit 369643 (41063b40168b69b38e92d8da3af3b45e58fd98ca) and later from Subversion, which appears to have been.  The previous revision (369642 in SVN) works fine.

I have a rule that says:

pass out log (all) quick route-to (em1 172.29.29.2) inet proto tcp from 172.29.29.1 to any flags S/SA keep state

This is to route traffic that originates on the local machine, with a source address of 172.29.29.1, to the router at 172.29.29.2  My particular use-case is to divide local traffic between WAN connections based on their source addresses.

Note that this, while similar, differs from the case mentioned in 8ca8248886af583fa2010badfe03e472d8505db8 (r369645) in that the matched connections originate locally rather than remotely, so the rule is an OUT rule rather than an IN rule.

Connections that match this rule get established and exchange a bit of data, but then stall.  The corresponding state gets stuck as follows:

all tcp 172.29.29.1:25564 -> 172.67.29.179:443       ESTABLISHED:SYN_SENT

The address 172.67.29.179 is an arbitrary remote address.  The remote end never goes to ESTABLISHED.  Replies from the remote end are matching the rule and correctly and getting logged, until the connection stalls.

This is the if() statement that changed in 41063b40168b69b38e92d8da3af3b45e58fd98ca.  It seems to make sense that this would break with packets that only go out?

    if (dir == PF_IN) {
        if (pf_test(PF_OUT, 0, ifp, &m0, inp) != PF_PASS)
            goto bad;


The tests in route_to.sh from r369646 (8ca8248886af583fa2010badfe03e472d8505db8) pass.

sys/netpfil/pf/route_to:multiwan  ->  passed  [0.109s]
sys/netpfil/pf/route_to:v4  ->  passed  [3.045s]
sys/netpfil/pf/route_to:v6  ->  passed  [3.105s]

I should be able to conjure up an updated route_to.sh test script to test for the local-origin + route-to scenario, which I'll attach in due course.

I'm happy to test fixes, provide packet dumps, etc.

Thanks
Mark
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2021-07-11 10:32:17 UTC
^Triage: assign.

While here, note that we are phasing out [tags] in Summaries.
Comment 2 Mark C 2021-07-13 15:31:27 UTC
Created attachment 226431 [details]
Extended version of the route_to.sh test script, including multiwanlocal test

Adding an updated version of the original route_to.sh test script to illustrate the issue.  The new multiwanlocal test passes in r369642, and fails from 369646 on.
Comment 3 Kristof Provost freebsd_committer 2021-07-14 12:42:45 UTC
That test case was *extremely* useful.

I've got a fix ready for review here:
https://reviews.freebsd.org/D31177

It should just apply to stable/12 as well.

I've lightly edited your test case here:
https://reviews.freebsd.org/D31178
Comment 4 Mark C 2021-07-14 16:31:37 UTC
Great.  Thank-you for the quick turnaround. I learned more than I have in 20 years of FreeBSD use regarding jails while writing that test.

I've tested the patch with r369646 and it works.

I'm busy building the HEAD of 12-STABLE (with the patch applied) currently, and should be ready to test in the morning.
Comment 5 Mark C 2021-07-15 10:23:26 UTC
Thanks for the assistance Kristoff.  I've tested this patch with the HEAD of 12, and it works.

If you need anything further from me on this to get it committed, let me know.

Mark
Comment 6 Kristof Provost freebsd_committer 2021-07-15 11:14:52 UTC
(In reply to Mark C from comment #5)
I don't think so. I'm going to give this a little bit more time for review and then commit in a day or two.
Comment 7 commit-hook freebsd_committer 2021-07-17 14:30:17 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f

commit 2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2021-07-14 10:17:03 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2021-07-17 12:28:07 +0000

    pf: locally originating connections with 'route-to' fail

    Similar to the REPLY_TO shortcut (6d786845cf) we also can't shortcut
    ROUTE_TO. If we do we will fail to apply transformations or update the
    state, which can lead to premature termination of the connections.

    PR:             257106
    MFC after:      3 weeks
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D31177

 sys/netpfil/pf/pf.c | 6 ------
 1 file changed, 6 deletions(-)
Comment 8 commit-hook freebsd_committer 2021-07-17 14:30:18 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b

commit f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2021-07-14 10:33:37 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2021-07-17 12:28:08 +0000

    pf tests: test locally originated connections with route-to

    PR:             257106
    Submitted by:   Mark Cammidge <mark@peralex.com>
    MFC after:      3 weeks
    Differential Revision:  https://reviews.freebsd.org/D31178

 tests/sys/netpfil/pf/route_to.sh | 70 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
Comment 9 commit-hook freebsd_committer 2021-08-07 16:30:43 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=e650ef36ba9ef7ec994d3e7048a56fea761f9c7e

commit e650ef36ba9ef7ec994d3e7048a56fea761f9c7e
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2021-07-14 10:17:03 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2021-08-07 07:08:09 +0000

    pf: locally originating connections with 'route-to' fail

    Similar to the REPLY_TO shortcut (6d786845cf) we also can't shortcut
    ROUTE_TO. If we do we will fail to apply transformations or update the
    state, which can lead to premature termination of the connections.

    PR:             257106
    MFC after:      3 weeks
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D31177

    (cherry picked from commit 2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f)

 sys/netpfil/pf/pf.c | 6 ------
 1 file changed, 6 deletions(-)
Comment 10 commit-hook freebsd_committer 2021-08-07 16:30:44 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=3bedd8a3b6d7816341cf6eebc3f91633a175320d

commit 3bedd8a3b6d7816341cf6eebc3f91633a175320d
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2021-07-14 10:33:37 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2021-08-07 07:08:19 +0000

    pf tests: test locally originated connections with route-to

    PR:             257106
    Submitted by:   Mark Cammidge <mark@peralex.com>
    MFC after:      3 weeks
    Differential Revision:  https://reviews.freebsd.org/D31178

    (cherry picked from commit f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b)

 tests/sys/netpfil/pf/route_to.sh | 70 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
Comment 11 commit-hook freebsd_committer 2021-08-07 16:30:45 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=bd28b2d73223b898169986eb5f04ee6045929612

commit bd28b2d73223b898169986eb5f04ee6045929612
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2021-07-14 10:33:37 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2021-08-07 07:11:28 +0000

    pf tests: test locally originated connections with route-to

    PR:             257106
    Submitted by:   Mark Cammidge <mark@peralex.com>
    MFC after:      3 weeks
    Differential Revision:  https://reviews.freebsd.org/D31178

    (cherry picked from commit f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b)

 tests/sys/netpfil/pf/route_to.sh | 70 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
Comment 12 commit-hook freebsd_committer 2021-08-07 16:30:46 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=44c47bc6d61ea295c6bb955a40f32c93a589f3ea

commit 44c47bc6d61ea295c6bb955a40f32c93a589f3ea
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2021-07-14 10:17:03 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2021-08-07 07:09:35 +0000

    pf: locally originating connections with 'route-to' fail

    Similar to the REPLY_TO shortcut (6d786845cf) we also can't shortcut
    ROUTE_TO. If we do we will fail to apply transformations or update the
    state, which can lead to premature termination of the connections.

    PR:             257106
    MFC after:      3 weeks
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D31177

    (cherry picked from commit 2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f)

 sys/netpfil/pf/pf.c | 6 ------
 1 file changed, 6 deletions(-)