Bug 255678 - security/strongswan cant add routes via RTM_ADD via PF_ROUTE socket
Summary: security/strongswan cant add routes via RTM_ADD via PF_ROUTE socket
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Alexander V. Chernikov
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2021-05-07 07:41 UTC by martin.larsson2
Modified: 2022-10-11 10:15 UTC (History)
11 users (show)

See Also:
linimon: maintainer-feedback? (strongswan)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description martin.larsson2 2021-05-07 07:41:20 UTC
since freebsd 13.0 strongswan cant add routes via RTM_ADD via PF_ROUTE socket anymore. 

The error I see is  PF_ROUTE route failed: Operation not supported

If this is related to the big route rewrite, it seems undocumented.

link to strongswan issue -> https://github.com/strongswan/strongswan/issues/275
Comment 1 Francois ten Krooden 2021-05-10 08:50:59 UTC
I had a look at the strongSwan issue on github.
I also geuss it might be related to the changes made to the routing in FreeBSD.

I am just a bit occupied currently to debug and see if I can identify the issue.
Comment 2 Marek Zarychta 2021-05-13 14:12:11 UTC
(In reply to martin.larsson2 from comment #0)
Have you tried to check if it works on 13.0-STABLE? There is a couple of fixes committed to stable/13 which maybe solved the issue. Please take a look at bug 255089 and bug 255273. See commits de703e98e6c863874aa6012e3ce1a61eee58c846 and 6f1e5d9169a9cbd65b480cb7c1012c3360d2ba06.
https://cgit.freebsd.org/src/commit/?id=de703e98e6c863874aa6012e3ce1a61eee58c846
https://cgit.freebsd.org/src/commit/?id=6f1e5d9169a9cbd65b480cb7c1012c3360d2ba06
Comment 3 martin.larsson2 2021-05-17 13:44:45 UTC
I installed 13.0-STABLE kernel and /boot only which seemed to work.

result is that I can see the route being added now and no errors.
but regardless of settings in kernel-pfkey.conf it always add external route now.
and thus it doesnt work anyways.

Internet:
Destination        Gateway            Flags     Netif Expire
default            213.80.11.16     UGS        igb1
10.11.12.0/24      213.80.11.16     US         igb1

my cfg

kernel-pfkey {

    # Size of the receive buffer for the event socket (0 for default size).
    # events_buffer_size = 0

    # Whether to load the plugin. Can also be an integer to increase the
    # priority of this plugin.
    load = yes
    route_via_internal = yes
}
Comment 4 Tobias Brunner 2021-05-17 13:51:46 UTC
Please increase the log level for KNL to 2. Then check the log message starting with "installing route:". Does it list the correct (internal) interface?
Comment 5 martin.larsson2 2021-05-17 14:03:13 UTC
Mon, 2021-05-17, 15:58:12 07[KNL] adding policy 10.11.12.0/24 === 192.168.5.0/24 in
Mon, 2021-05-17, 15:58:12 07[KNL] adding policy 192.168.5.0/24 === 10.11.12.0/24 out
Mon, 2021-05-17, 15:58:12 07[KNL] getting a local address in traffic selector 192.168.5.0/24
Mon, 2021-05-17, 15:58:12 07[KNL] using host 192.168.5.10
Mon, 2021-05-17, 15:58:12 07[KNL] using 213.80.11.16 as nexthop to reach 19.4.13.19
Mon, 2021-05-17, 15:58:12 07[KNL] 192.168.5.10 is on interface igb0
Mon, 2021-05-17, 15:58:12 07[KNL] installing route: 10.11.12.0/24 via 213.80.11.16 src 192.168.5.10 dev igb0
Mon, 2021-05-17, 15:58:12 07[CHD] CHILD_SA rolfen{1} state change: CREATED => ROUTED

seems internal interface is correctly detected
Comment 6 Marek Zarychta 2021-05-17 14:07:48 UTC
You can try to increase net.route.algo.debug_level by changing this sysctl, but it might be dependent on kernel "options FIB_ALGO" which is enabled by default only in CURRENT.
Comment 7 Tobias Brunner 2021-05-17 14:09:17 UTC
OK, that interface name is passed via RTA_IFP attribute to the kernel when installing the route. Maybe it doesn't like that the gateway (presumably) is not reachable via that interface, so it changes it.
Comment 8 Kristof Provost freebsd_committer freebsd_triage 2021-05-17 14:10:55 UTC
Net bug, not a pf issue. (In this context PF is protocol family, not packet filter)
Comment 9 Marek Zarychta 2021-05-17 14:12:22 UTC
Also disabling multipath at this stage will probably help: sysctl net.route.multipath=0
Comment 10 martin.larsson2 2021-05-17 14:13:39 UTC
setting sysctl net.route.multipath=0 didnt change anything in this case
Comment 11 Alexander V. Chernikov freebsd_committer freebsd_triage 2021-05-30 10:21:30 UTC
Apologies for the late reactionm, I missed this PR.

Martin, would it be possible if you could:

1) share netstat -rnW, netstat -4onW output
2) run route -n monitor, run strongswan and share the output?
3) describe the desired outcome in a bit more details? E.g. is the problem that igb1 is selected instead of igb0, multiple default routes while there should be the only one, something else?
Comment 12 martin.larsson2 2021-05-30 19:09:45 UTC
yes, thanks for looking in to this.

here is what you requested I hope.

https://dumpinen.com/rbBJLFGuX-n

and yes, what I expect is that my two remote networks being 10.11.12.0/24 and 192.168.17.0/24 to be routed via igb0 as it did on earlier versions of freebsd.
Comment 13 martin.larsson2 2021-05-31 13:44:21 UTC
FYI, output is from 13.0-release with kernel and /boot from 13.0-STABLE
Comment 14 Alexander V. Chernikov freebsd_committer freebsd_triage 2021-05-31 20:02:12 UTC
(In reply to martin.larsson2 from comment #13)
Okay. So let me try to describe it to check if my understanding is correct:

There are 2 interfaces, igb0 (192.168.5.0/24) and igb1 (213.80.111.160/27).
The default route points towards 213.80.111.161, which is directly reachable via igb0.

Stronswan tries to install the route with nexthop 213.80.11.16 (which is indirectly reachable via igb0 default) and specifying interface igb0.

The expected behaviour is that for this route, the system will consider 213.80.11.16 directly reachable via igb0, correct?
Comment 15 Alexander V. Chernikov freebsd_committer freebsd_triage 2021-05-31 20:09:18 UTC
(In reply to Alexander V. Chernikov from comment #14)
Whoops. Wrote igb0 instead of igb1. Copying the last comment with proper interface names:


There are 2 interfaces, igb0 (192.168.5.0/24) and igb1 (213.80.111.160/27).
The default route points towards 213.80.111.161, which is directly reachable via igb1.

Stronswan tries to install the route with nexthop 213.80.11.16 (which is indirectly reachable via igb1 default) and specifying interface igb0.

The expected behaviour is that for this route, the system will consider 213.80.11.16 directly reachable via igb0, correct?
Comment 16 martin.larsson2 2021-06-01 07:41:39 UTC
is it Tobias?
Comment 17 Tobias Brunner 2021-06-01 09:54:44 UTC
> The expected behaviour is that for this route, the system will consider 213.80.11.16 directly reachable via igb0, correct?

Not exactly. The end goal is to install a route that causes the kernel to select the "internal" IP address (192.168.5.10 on igb0) as source when reaching the remote VPN subnet (10.11.12.0/24).  Because the IPsec policy is between 192.168.5.0/24 and 10.11.12.0/24, selecting the address on the external interface (213.80.111.176) would cause the traffic to get sent unprotected (unless there was a second IPsec policy that covered traffic between that IP and the remote subnet).

By default, strongSwan installs the route for the remote subnet via outbound interface (i.e. over which the IKE peer is reachable). However, like the default route, this would cause the IP on the "external" interface (213.80.111.176) to get selected as source. So we added an option (charon.plugins.kernel-pfkey.route_via_internal) that causes the installation of the route via "internal" interface (the next hop is still the one to reach the IKE peer, though, maybe we should remove that?). As Martin reported, this worked previously.

For comparison, on Linux, we install a route for the remote subnet via external interface but we set the RTA_PREFSRC attribute to the internal IP address, which causes it to get selected when traffic to the remote subnet is generated (we also install that route in a separate routing table that takes precedence over the main table and allows us to even override the default route without conflicts). AFAIK, there is nothing similar on FreeBSD.
Comment 18 Alexander V. Chernikov freebsd_committer freebsd_triage 2021-06-03 22:11:17 UTC
(In reply to Tobias Brunner from comment #17)
> Not exactly. The end goal is to install a route that causes the kernel to select the "internal" IP address (192.168.5.10 on igb0) as source when reaching the remote VPN subnet (10.11.12.0/24).
Got it.

> For comparison, on Linux, we install a route for the remote subnet via external interface but we set the RTA_PREFSRC attribute to the internal IP address, which causes it to get selected when traffic to the remote subnet is generated (we also install that route in a separate routing table that takes precedence over the main table and allows us to even override the default route without conflicts). AFAIK, there is nothing similar on FreeBSD.
*BSD has RTAX_IFA rtsock option allowing to choose the preferred source address.
FreeBSD has support for multiple routing tables (net.fibs), though there may be some rough edges.

I'll be able to look and hopefully fix the issue on the weekend.
Re optimal way of specifying the source address - IMO having an explicit RTAX_IFA + RTAX_IFP (specified by an ifindex) should be more bulletproof, but let me fix the bug first & verify the proper RTAX_IFA operations.
Comment 19 Tobias Brunner 2021-06-04 07:45:38 UTC
> *BSD has RTAX_IFA rtsock option allowing to choose the preferred source address.

Great. Weirdly, we already parse this when we check the routes to determine a source address, not sure exactly why we never added it to the routes.

Does that also work if the address in RTAX_IFA is on a different interface than the one configured with RTAX_IFP? (In Martin's case the address is on igb0 while the interface is igb1.)

> FreeBSD has support for multiple routing tables (net.fibs), though there may be some rough edges.

Are these managed via PF_ROUTE interface? Or how does that work/interoperate exactly?
Comment 20 martin.larsson2 2021-07-14 08:16:24 UTC
did you get any chance to look at it?
Comment 21 commit-hook freebsd_committer freebsd_triage 2021-08-29 20:14:11 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=d98954e229812eee2fa6bf97714fecbbdcc56e4c

commit d98954e229812eee2fa6bf97714fecbbdcc56e4c
Author:     Alexander V. Chernikov <melifaro@FreeBSD.org>
AuthorDate: 2021-08-29 19:51:28 +0000
Commit:     Alexander V. Chernikov <melifaro@FreeBSD.org>
CommitDate: 2021-08-29 20:05:14 +0000

    routing: Bring back the ability to specify transmit interface via its name.

    Some software references outgoing interfaces by specifying name instead of
     index.

    Use rti_ifp from rt_addrinfo if provided instead of always using
     address interface when constructing nexthop.

    PR:             255678
    Reported by:    martin.larsson2 at gmail.com
    MFC after:      1 week

 sys/net/route.c          | 45 ++++++++++++++++++++++++++++++++++++---------
 sys/net/route/nhop_ctl.c |  2 +-
 2 files changed, 37 insertions(+), 10 deletions(-)
Comment 22 martin.larsson2 2021-08-31 13:38:52 UTC
thanks alot for your work alexander!

Would this apply to stable if I want to to try?
Comment 23 commit-hook freebsd_committer freebsd_triage 2021-09-07 21:31:09 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=f3d69003374a291e2ccb4a1ecc318e90827bac09

commit f3d69003374a291e2ccb4a1ecc318e90827bac09
Author:     Alexander V. Chernikov <melifaro@FreeBSD.org>
AuthorDate: 2021-08-29 19:51:28 +0000
Commit:     Alexander V. Chernikov <melifaro@FreeBSD.org>
CommitDate: 2021-09-07 21:25:24 +0000

    routing: Bring back the ability to specify transmit interface via its name.

    Some software references outgoing interfaces by specifying name instead of
     index.

    Use rti_ifp from rt_addrinfo if provided instead of always using
     address interface when constructing nexthop.

    PR:             255678
    Reported by:    martin.larsson2 at gmail.com

    (cherry picked from commit d98954e229812eee2fa6bf97714fecbbdcc56e4c)

 sys/net/route.c          | 45 ++++++++++++++++++++++++++++++++++++---------
 sys/net/route/nhop_ctl.c |  2 +-
 2 files changed, 37 insertions(+), 10 deletions(-)
Comment 24 martin.larsson2 2021-09-19 04:25:51 UTC
I have now tested this on 13-STABLE kernel and can confirm that it works, thanks again Alexander and you others.
Comment 25 crest 2021-09-27 11:47:30 UTC
Will this fix make it into a future errata for FreeBSD 13.0?