Running FreeBSD 12-stable, I'm running into a problem with commit 369643 (41063b40168b69b38e92d8da3af3b45e58fd98ca) and later from Subversion, which appears to have been. The previous revision (369642 in SVN) works fine. I have a rule that says: pass out log (all) quick route-to (em1 172.29.29.2) inet proto tcp from 172.29.29.1 to any flags S/SA keep state This is to route traffic that originates on the local machine, with a source address of 172.29.29.1, to the router at 172.29.29.2 My particular use-case is to divide local traffic between WAN connections based on their source addresses. Note that this, while similar, differs from the case mentioned in 8ca8248886af583fa2010badfe03e472d8505db8 (r369645) in that the matched connections originate locally rather than remotely, so the rule is an OUT rule rather than an IN rule. Connections that match this rule get established and exchange a bit of data, but then stall. The corresponding state gets stuck as follows: all tcp 172.29.29.1:25564 -> 172.67.29.179:443 ESTABLISHED:SYN_SENT The address 172.67.29.179 is an arbitrary remote address. The remote end never goes to ESTABLISHED. Replies from the remote end are matching the rule and correctly and getting logged, until the connection stalls. This is the if() statement that changed in 41063b40168b69b38e92d8da3af3b45e58fd98ca. It seems to make sense that this would break with packets that only go out? if (dir == PF_IN) { if (pf_test(PF_OUT, 0, ifp, &m0, inp) != PF_PASS) goto bad; The tests in route_to.sh from r369646 (8ca8248886af583fa2010badfe03e472d8505db8) pass. sys/netpfil/pf/route_to:multiwan -> passed [0.109s] sys/netpfil/pf/route_to:v4 -> passed [3.045s] sys/netpfil/pf/route_to:v6 -> passed [3.105s] I should be able to conjure up an updated route_to.sh test script to test for the local-origin + route-to scenario, which I'll attach in due course. I'm happy to test fixes, provide packet dumps, etc. Thanks Mark
^Triage: assign. While here, note that we are phasing out [tags] in Summaries.
Created attachment 226431 [details] Extended version of the route_to.sh test script, including multiwanlocal test Adding an updated version of the original route_to.sh test script to illustrate the issue. The new multiwanlocal test passes in r369642, and fails from 369646 on.
That test case was *extremely* useful. I've got a fix ready for review here: https://reviews.freebsd.org/D31177 It should just apply to stable/12 as well. I've lightly edited your test case here: https://reviews.freebsd.org/D31178
Great. Thank-you for the quick turnaround. I learned more than I have in 20 years of FreeBSD use regarding jails while writing that test. I've tested the patch with r369646 and it works. I'm busy building the HEAD of 12-STABLE (with the patch applied) currently, and should be ready to test in the morning.
Thanks for the assistance Kristoff. I've tested this patch with the HEAD of 12, and it works. If you need anything further from me on this to get it committed, let me know. Mark
(In reply to Mark C from comment #5) I don't think so. I'm going to give this a little bit more time for review and then commit in a day or two.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f commit 2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2021-07-14 10:17:03 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2021-07-17 12:28:07 +0000 pf: locally originating connections with 'route-to' fail Similar to the REPLY_TO shortcut (6d786845cf) we also can't shortcut ROUTE_TO. If we do we will fail to apply transformations or update the state, which can lead to premature termination of the connections. PR: 257106 MFC after: 3 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D31177 sys/netpfil/pf/pf.c | 6 ------ 1 file changed, 6 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b commit f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2021-07-14 10:33:37 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2021-07-17 12:28:08 +0000 pf tests: test locally originated connections with route-to PR: 257106 Submitted by: Mark Cammidge <mark@peralex.com> MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D31178 tests/sys/netpfil/pf/route_to.sh | 70 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e650ef36ba9ef7ec994d3e7048a56fea761f9c7e commit e650ef36ba9ef7ec994d3e7048a56fea761f9c7e Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2021-07-14 10:17:03 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2021-08-07 07:08:09 +0000 pf: locally originating connections with 'route-to' fail Similar to the REPLY_TO shortcut (6d786845cf) we also can't shortcut ROUTE_TO. If we do we will fail to apply transformations or update the state, which can lead to premature termination of the connections. PR: 257106 MFC after: 3 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D31177 (cherry picked from commit 2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f) sys/netpfil/pf/pf.c | 6 ------ 1 file changed, 6 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3bedd8a3b6d7816341cf6eebc3f91633a175320d commit 3bedd8a3b6d7816341cf6eebc3f91633a175320d Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2021-07-14 10:33:37 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2021-08-07 07:08:19 +0000 pf tests: test locally originated connections with route-to PR: 257106 Submitted by: Mark Cammidge <mark@peralex.com> MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D31178 (cherry picked from commit f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b) tests/sys/netpfil/pf/route_to.sh | 70 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=bd28b2d73223b898169986eb5f04ee6045929612 commit bd28b2d73223b898169986eb5f04ee6045929612 Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2021-07-14 10:33:37 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2021-08-07 07:11:28 +0000 pf tests: test locally originated connections with route-to PR: 257106 Submitted by: Mark Cammidge <mark@peralex.com> MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D31178 (cherry picked from commit f808bb9b7e5ee2243e5a2aaad2275a78cdbe981b) tests/sys/netpfil/pf/route_to.sh | 70 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=44c47bc6d61ea295c6bb955a40f32c93a589f3ea commit 44c47bc6d61ea295c6bb955a40f32c93a589f3ea Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2021-07-14 10:17:03 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2021-08-07 07:09:35 +0000 pf: locally originating connections with 'route-to' fail Similar to the REPLY_TO shortcut (6d786845cf) we also can't shortcut ROUTE_TO. If we do we will fail to apply transformations or update the state, which can lead to premature termination of the connections. PR: 257106 MFC after: 3 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D31177 (cherry picked from commit 2c0d115bbc8f8ee3f011a5c4a69bcbf58c4b721f) sys/netpfil/pf/pf.c | 6 ------ 1 file changed, 6 deletions(-)