since freebsd 13.0 strongswan cant add routes via RTM_ADD via PF_ROUTE socket anymore. The error I see is PF_ROUTE route failed: Operation not supported If this is related to the big route rewrite, it seems undocumented. link to strongswan issue -> https://github.com/strongswan/strongswan/issues/275
I had a look at the strongSwan issue on github. I also geuss it might be related to the changes made to the routing in FreeBSD. I am just a bit occupied currently to debug and see if I can identify the issue.
(In reply to martin.larsson2 from comment #0) Have you tried to check if it works on 13.0-STABLE? There is a couple of fixes committed to stable/13 which maybe solved the issue. Please take a look at bug 255089 and bug 255273. See commits de703e98e6c863874aa6012e3ce1a61eee58c846 and 6f1e5d9169a9cbd65b480cb7c1012c3360d2ba06. https://cgit.freebsd.org/src/commit/?id=de703e98e6c863874aa6012e3ce1a61eee58c846 https://cgit.freebsd.org/src/commit/?id=6f1e5d9169a9cbd65b480cb7c1012c3360d2ba06
I installed 13.0-STABLE kernel and /boot only which seemed to work. result is that I can see the route being added now and no errors. but regardless of settings in kernel-pfkey.conf it always add external route now. and thus it doesnt work anyways. Internet: Destination Gateway Flags Netif Expire default 213.80.11.16 UGS igb1 10.11.12.0/24 213.80.11.16 US igb1 my cfg kernel-pfkey { # Size of the receive buffer for the event socket (0 for default size). # events_buffer_size = 0 # Whether to load the plugin. Can also be an integer to increase the # priority of this plugin. load = yes route_via_internal = yes }
Please increase the log level for KNL to 2. Then check the log message starting with "installing route:". Does it list the correct (internal) interface?
Mon, 2021-05-17, 15:58:12 07[KNL] adding policy 10.11.12.0/24 === 192.168.5.0/24 in Mon, 2021-05-17, 15:58:12 07[KNL] adding policy 192.168.5.0/24 === 10.11.12.0/24 out Mon, 2021-05-17, 15:58:12 07[KNL] getting a local address in traffic selector 192.168.5.0/24 Mon, 2021-05-17, 15:58:12 07[KNL] using host 192.168.5.10 Mon, 2021-05-17, 15:58:12 07[KNL] using 213.80.11.16 as nexthop to reach 19.4.13.19 Mon, 2021-05-17, 15:58:12 07[KNL] 192.168.5.10 is on interface igb0 Mon, 2021-05-17, 15:58:12 07[KNL] installing route: 10.11.12.0/24 via 213.80.11.16 src 192.168.5.10 dev igb0 Mon, 2021-05-17, 15:58:12 07[CHD] CHILD_SA rolfen{1} state change: CREATED => ROUTED seems internal interface is correctly detected
You can try to increase net.route.algo.debug_level by changing this sysctl, but it might be dependent on kernel "options FIB_ALGO" which is enabled by default only in CURRENT.
OK, that interface name is passed via RTA_IFP attribute to the kernel when installing the route. Maybe it doesn't like that the gateway (presumably) is not reachable via that interface, so it changes it.
Net bug, not a pf issue. (In this context PF is protocol family, not packet filter)
Also disabling multipath at this stage will probably help: sysctl net.route.multipath=0
setting sysctl net.route.multipath=0 didnt change anything in this case
Apologies for the late reactionm, I missed this PR. Martin, would it be possible if you could: 1) share netstat -rnW, netstat -4onW output 2) run route -n monitor, run strongswan and share the output? 3) describe the desired outcome in a bit more details? E.g. is the problem that igb1 is selected instead of igb0, multiple default routes while there should be the only one, something else?
yes, thanks for looking in to this. here is what you requested I hope. https://dumpinen.com/rbBJLFGuX-n and yes, what I expect is that my two remote networks being 10.11.12.0/24 and 192.168.17.0/24 to be routed via igb0 as it did on earlier versions of freebsd.
FYI, output is from 13.0-release with kernel and /boot from 13.0-STABLE
(In reply to martin.larsson2 from comment #13) Okay. So let me try to describe it to check if my understanding is correct: There are 2 interfaces, igb0 (192.168.5.0/24) and igb1 (213.80.111.160/27). The default route points towards 213.80.111.161, which is directly reachable via igb0. Stronswan tries to install the route with nexthop 213.80.11.16 (which is indirectly reachable via igb0 default) and specifying interface igb0. The expected behaviour is that for this route, the system will consider 213.80.11.16 directly reachable via igb0, correct?
(In reply to Alexander V. Chernikov from comment #14) Whoops. Wrote igb0 instead of igb1. Copying the last comment with proper interface names: There are 2 interfaces, igb0 (192.168.5.0/24) and igb1 (213.80.111.160/27). The default route points towards 213.80.111.161, which is directly reachable via igb1. Stronswan tries to install the route with nexthop 213.80.11.16 (which is indirectly reachable via igb1 default) and specifying interface igb0. The expected behaviour is that for this route, the system will consider 213.80.11.16 directly reachable via igb0, correct?
is it Tobias?
> The expected behaviour is that for this route, the system will consider 213.80.11.16 directly reachable via igb0, correct? Not exactly. The end goal is to install a route that causes the kernel to select the "internal" IP address (192.168.5.10 on igb0) as source when reaching the remote VPN subnet (10.11.12.0/24). Because the IPsec policy is between 192.168.5.0/24 and 10.11.12.0/24, selecting the address on the external interface (213.80.111.176) would cause the traffic to get sent unprotected (unless there was a second IPsec policy that covered traffic between that IP and the remote subnet). By default, strongSwan installs the route for the remote subnet via outbound interface (i.e. over which the IKE peer is reachable). However, like the default route, this would cause the IP on the "external" interface (213.80.111.176) to get selected as source. So we added an option (charon.plugins.kernel-pfkey.route_via_internal) that causes the installation of the route via "internal" interface (the next hop is still the one to reach the IKE peer, though, maybe we should remove that?). As Martin reported, this worked previously. For comparison, on Linux, we install a route for the remote subnet via external interface but we set the RTA_PREFSRC attribute to the internal IP address, which causes it to get selected when traffic to the remote subnet is generated (we also install that route in a separate routing table that takes precedence over the main table and allows us to even override the default route without conflicts). AFAIK, there is nothing similar on FreeBSD.
(In reply to Tobias Brunner from comment #17) > Not exactly. The end goal is to install a route that causes the kernel to select the "internal" IP address (192.168.5.10 on igb0) as source when reaching the remote VPN subnet (10.11.12.0/24). Got it. > For comparison, on Linux, we install a route for the remote subnet via external interface but we set the RTA_PREFSRC attribute to the internal IP address, which causes it to get selected when traffic to the remote subnet is generated (we also install that route in a separate routing table that takes precedence over the main table and allows us to even override the default route without conflicts). AFAIK, there is nothing similar on FreeBSD. *BSD has RTAX_IFA rtsock option allowing to choose the preferred source address. FreeBSD has support for multiple routing tables (net.fibs), though there may be some rough edges. I'll be able to look and hopefully fix the issue on the weekend. Re optimal way of specifying the source address - IMO having an explicit RTAX_IFA + RTAX_IFP (specified by an ifindex) should be more bulletproof, but let me fix the bug first & verify the proper RTAX_IFA operations.
> *BSD has RTAX_IFA rtsock option allowing to choose the preferred source address. Great. Weirdly, we already parse this when we check the routes to determine a source address, not sure exactly why we never added it to the routes. Does that also work if the address in RTAX_IFA is on a different interface than the one configured with RTAX_IFP? (In Martin's case the address is on igb0 while the interface is igb1.) > FreeBSD has support for multiple routing tables (net.fibs), though there may be some rough edges. Are these managed via PF_ROUTE interface? Or how does that work/interoperate exactly?
did you get any chance to look at it?
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d98954e229812eee2fa6bf97714fecbbdcc56e4c commit d98954e229812eee2fa6bf97714fecbbdcc56e4c Author: Alexander V. Chernikov <melifaro@FreeBSD.org> AuthorDate: 2021-08-29 19:51:28 +0000 Commit: Alexander V. Chernikov <melifaro@FreeBSD.org> CommitDate: 2021-08-29 20:05:14 +0000 routing: Bring back the ability to specify transmit interface via its name. Some software references outgoing interfaces by specifying name instead of index. Use rti_ifp from rt_addrinfo if provided instead of always using address interface when constructing nexthop. PR: 255678 Reported by: martin.larsson2 at gmail.com MFC after: 1 week sys/net/route.c | 45 ++++++++++++++++++++++++++++++++++++--------- sys/net/route/nhop_ctl.c | 2 +- 2 files changed, 37 insertions(+), 10 deletions(-)
thanks alot for your work alexander! Would this apply to stable if I want to to try?
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f3d69003374a291e2ccb4a1ecc318e90827bac09 commit f3d69003374a291e2ccb4a1ecc318e90827bac09 Author: Alexander V. Chernikov <melifaro@FreeBSD.org> AuthorDate: 2021-08-29 19:51:28 +0000 Commit: Alexander V. Chernikov <melifaro@FreeBSD.org> CommitDate: 2021-09-07 21:25:24 +0000 routing: Bring back the ability to specify transmit interface via its name. Some software references outgoing interfaces by specifying name instead of index. Use rti_ifp from rt_addrinfo if provided instead of always using address interface when constructing nexthop. PR: 255678 Reported by: martin.larsson2 at gmail.com (cherry picked from commit d98954e229812eee2fa6bf97714fecbbdcc56e4c) sys/net/route.c | 45 ++++++++++++++++++++++++++++++++++++--------- sys/net/route/nhop_ctl.c | 2 +- 2 files changed, 37 insertions(+), 10 deletions(-)
I have now tested this on 13-STABLE kernel and can confirm that it works, thanks again Alexander and you others.
Will this fix make it into a future errata for FreeBSD 13.0?
^Triage: note that FreeBSD 13.0 is out of support, so the MFC is OBE.