# service wireguard start [#] ifconfig wg create name wg0 (...) [+] Backgrounding route monitor # service wireguard stop [#] ifconfig wg0 destroy # ps | grep route 2625 4 S+ 0:00.00 grep route 85368 4 I 0:00.00 route -n monitor
Hi all! I've independently investigated this case and, at first, it seemed to me that the problem lies within wg-quick bash script, to be more specific - at monitor_daemon() function. But now it seems to me that the real cause is at route -n monitor base command. It's output changed (due to netlink transition), and now scripts have totally no input from it. See how its output differs from 13.3 то 14.0: under 13.3-RELEASE (GENERIC amd64) $ bash -c 'route -n monitor | while read event; do echo "bash: $event"; done;' & [1] 24167 $ ifconfig gif0 create && ifconfig gif0 destroy bash: bash: got message of size 24 on Mon Apr 8 16:47:51 2024 bash: RTM_IFANNOUNCE: interface arrival/departure: len 24, if# 6, what: arrival bash: bash: got message of size 168 on Mon Apr 8 16:47:51 2024 bash: RTM_IFINFO: iface status change: len 168, if# 6, link: down, flags:<PTP,MULTICAST> bash: bash: got message of size 168 on Mon Apr 8 16:47:51 2024 bash: RTM_IFINFO: iface status change: len 168, if# 6, link: down, flags:<PTP,MULTICAST> bash: bash: got message of size 24 on Mon Apr 8 16:47:51 2024 bash: RTM_IFANNOUNCE: interface arrival/departure: len 24, if# 6, what: departure $ under 14.0-RELEASE-p6 $ bash -c 'route -n monitor | while read event; do echo "bash: $event"; done;' & [4] 15153 $ ifconfig gif0 create && ifconfig gif0 destroy $ as you may see there's not a line from monitor went to read, but route -n monitor alone (on 14.0) do print messages on console: $ route -nt monitor & [4] 15202 $ ifconfig gif0 create && ifconfig gif0 destroy 22:07:05.940 PID 0 add/repl iface iface#3 gif0 admin DOWN oper DOWN mtu 1280 22:07:05.943 PID 0 add/repl iface iface#3 gif0 admin DOWN oper DOWN mtu 1280 22:07:05.943 PID 0 add/repl iface iface#3 gif0 admin DOWN oper DOWN mtu 1280 22:07:05.946 PID 0 add/repl iface iface#3 gif0 admin DOWN oper DOWN mtu 1280 22:07:05.946 PID 0 delete iface iface#3 gif0 admin DOWN oper DOWN mtu 1280
Created attachment 249862 [details] script simplifcation due to 14.0-RELEASE route messages format the main problem was route itself, so first use my patch and recompile /sbin/route (see bug #278265). And then you can patch wg-quick
Created attachment 249863 [details] another script simplification. The same effect but it's more in original author's style
Comment on attachment 249863 [details] another script simplification. The same effect but it's more in original author's style >--- /usr/local/bin/wg-quick.org 2024-04-05 18:52:38.093753000 +0300 >+++ /usr/local/bin/wg-quick 2024-04-09 19:30:08.214518000 +0300 >@@ -284,19 +284,17 @@ > (make_temp > trap 'del_routes; clean_temp; exit 0' INT TERM EXIT > exec >/dev/null 2>&1 >- exec 19< <(exec route -n monitor) >- local event pid=$! > # TODO: this should also check to see if the endpoint actually changes > # in response to incoming packets, and then call set_endpoint_direct_route > # then too. That function should be able to gracefully cleanup if the > # endpoints change. >- while read -u 19 -r event; do >- [[ $event == RTM_* ]] || continue >+ while read -r event; do >+ # == due to 14.0 NETLINK messages == [[ $event == RTM_* ]] || continue > ifconfig "$INTERFACE" >/dev/null 2>&1 || break > [[ $AUTO_ROUTE4 -eq 1 || $AUTO_ROUTE6 -eq 1 ]] && set_endpoint_direct_route > # TODO: set the mtu as well, but only if up >- done >- kill $pid) & disown >+ done < <(exec route -n monitor) >+ ) & disown > } > > HAVE_SET_DNS=0
I also found this problem under GhostBSD GhostBSD-24.10.1, because after 'wg-quick down wg0' the local network did not work correctly anymore. I tested the two patches under 14.2-STABLE. The network route is now removed, but the process 'route -n monitor' unfortunately does not terminate.
I believe that I am facing the same issue. --- After a server restart: > ps -aux | grep wg root 3123 0.0 0.0 15584 4328 0 S 00:11 0:00.00 /usr/local/bin/bash /usr/local/bin/wg-quick up wg0 root 7633 0.0 0.0 13836 2384 0 S+ 00:11 0:00.00 grep wg > ps -aux | grep route root 3486 0.0 0.0 13820 2244 0 I 00:11 0:00.00 route -n monitor --- Then if I stop wireguard: > service wireguard stop [#] ifconfig wg0 destroy > ps -aux | grep wg root 3123 0.0 0.0 15584 4328 0 I 00:11 0:00.00 /usr/local/bin/bash /usr/local/bin/wg-quick up wg0 > ps -aux | grep route root 3486 0.0 0.0 13820 2364 0 I 00:11 0:00.00 route -n monitor
I'm on FreeBSD 14.2. It looks to me like "route -n monitor" doesn't output unless stdout is a tty. wg-quick's route monitor won't ever work.
(In reply to cweimann from comment #7) don't know why it's no output in your case. For real usage @wg-quick look at the monitor_daemon() within that script At one of my 14.3-RELEASE-p1 this cmd $ daemon -fo /tmp/test.log route -n monitor put lots of routing output in /tmp/test.log
(In reply to Oleg Streejak from comment #8) Interesting. You are right that does work but it seems very buffered. Try watching the log with tail -F /tmp/test.log. For me % route -n monitor > rtest.log produces nothing in rtest.log. I would have thought when I hit ctrl-c to stop it stdout would flush but it seems not. I did fiddle with the relevent bit of code from monitor_daemon and sent $event to /dev/stdout and got nothing. When I just run 'route -n monitor' from the shell (on my firewall box) I get a pretty steady flow. At least one entry every 5 seconds or so. But if I leave "route -n monitor > rtest.log" running in a window I get nothing after 5 minutes. Ah!!! I finally got output in rtest.log. The file is 32768 bytes so I'm guessing that is the buffer size. This must result in large delays in wg-quick's monitor_daemon function noticing anything. I'll give your patch a try. I'm curious why your 'daemon' line, although buffered, is much less buffered than a simple redirect to a file.
The monitor_daemon function watches 'route -n monitor' for events. When an event occurs it checks to see if the interface exists. Since those events are delayed by buffering I think it is likely that in the case of a wireguard restart the interface would exist again. usr/bin/route needs a fix. % git diff -p diff --git a/sbin/route/route_netlink.c b/sbin/route/route_netlink.c index 28766b062e18..b81a9eb7187d 100644 --- a/sbin/route/route_netlink.c +++ b/sbin/route/route_netlink.c @@ -780,6 +780,8 @@ monitor_nl(int fib) print_nlmsg(&h, hdr, &attrs); snl_clear_lb(&h.ss_cmd); snl_clear_lb(&ss_event); + fflush(stdout); + } snl_free(&ss_event); %
(In reply to cweimann from comment #10) that another bug - see my comment #2
(In reply to Oleg Streejak from comment #11) Oops. I nead to read more carefully. Thank you!!
Thanks a lot to everyone involved to keep debugging this issue! I kind of dropped this issue because - honestly this bash script is the pain and it's not the first time this kind of issue came up. So to make some progress it looks like we need to get this patch for route committed first. Can't do it myself but I will find someone do it. Then we can fix the port.
The route issue has been addressed in FreeBSD-EN-25:14.route: A commit in branch releng/14.2 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=168703212b618f32b42ed1f9e062a74d7f0cc534 commit 168703212b618f32b42ed1f9e062a74d7f0cc534 Author: Oleg Streejak <oleg@pcbtech.ru> AuthorDate: 2025-07-17 17:06:50 +0000 Commit: Gordon Tetlow <gordon@FreeBSD.org> CommitDate: 2025-08-07 23:23:56 +0000 route: fix `route -n monitor` when its output is redirected This is a small oversight in the transition to netlink; the non-netlink implementation would explicitly flush its stdout as necessary to avoid apparent long stalls in output when we end up fully-buffered. Adjust the netlink implementation to do the same. This was noticed while trying to triage failures in the wg-quick script. Commit message by kevans, patch by author. PR: 278265 Fixes: 091fec1188929 ("route: switch transport protocol [...]") Approved by: so Security: FreeBSD-EN-25:14.route (cherry picked from commit 97b61b22edba74c62adba1d022fb73541aa5ff93) (cherry picked from commit dd695839efd80fe81143cd6c7a552c30df8448f6) sbin/route/route_netlink.c | 1 +
(In reply to Oleg Streejak from comment #4) Your simplification removed the necessary kill command to stop the route monitor command (otherwise it will keep running until some route event happens). See below for an improved patch that should also handle 13.x gracefully. I will attach a patch to the port to this PR shortly. --- wg-quick.orig 2025-09-24 11:03:23.474144000 +0000 +++ wg-quick 2025-09-26 10:50:30.610481000 +0000 @@ -27,6 +27,7 @@ CONFIG_FILE="" PROGRAM="${0##*/}" ARGS=( "$@" ) +FREEBSD_MAJOR_VERSION=$(freebsd-version | cut -d. -f1) cmd() { echo "[#] $*" >&3 @@ -284,19 +285,18 @@ (make_temp trap 'del_routes; clean_temp; exit 0' INT TERM EXIT exec >/dev/null 2>&1 - exec 19< <(exec route -n monitor) - local event pid=$! # TODO: this should also check to see if the endpoint actually changes # in response to incoming packets, and then call set_endpoint_direct_route # then too. That function should be able to gracefully cleanup if the # endpoints change. - while read -u 19 -r event; do - [[ $event == RTM_* ]] || continue + while read -r event; do + [[ "$FREEBSD_MAJOR_VERSION" -lt 14 && $event != RTM_* ]] && continue ifconfig "$INTERFACE" >/dev/null 2>&1 || break [[ $AUTO_ROUTE4 -eq 1 || $AUTO_ROUTE6 -eq 1 ]] && set_endpoint_direct_route # TODO: set the mtu as well, but only if up - done - kill $pid) & disown + done < <(exec route -n monitor) + pkill -g0 "route -n monitor" || true + ) & disown } HAVE_SET_DNS=0
Created attachment 264098 [details] Fix route monitor on FreeBSD 14 Please find attached a patch to the port that should fix the issue for 14 without breaking 13.
Created attachment 264107 [details] Corrected patch (pkill was missing -f) Sorry, uploaded an incomplete patch beforehand :(
Thank you for the improvements to wg-quick. I was concerned that the number of route monitors kept increasing whenever I restarted the WireGuard service. On the other hand, the route command on FreeBSD 14 and later now uses Netlink, which has increased the output of route -n monitor. Specifically, at my home environment, I'm seeing output like the following, about one or two lines per second: route -n monitor 10:28:11.990 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 10:28:12.991 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 10:28:13.928 PID 0 add/repl neigh 192.168.22.222 state REACHABLE lladdr 00:50:b6:mm:mm:mm iface net0 10:28:13.992 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 10:28:14.883 PID 0 add/repl neigh 192.168.22.222 state REACHABLE lladdr 00:50:b6:mm:mm:mm iface net0 10:28:15.361 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 10:28:15.883 PID 0 add/repl neigh 192.168.22.222 state REACHABLE lladdr 00:50:b6:mm:mm:mm iface net0 10:28:17.296 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 10:28:18.298 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 10:28:18.856 PID 0 add/repl neigh 192.168.22.222 state REACHABLE lladdr 00:50:b6:mm:mm:mm iface net0 10:28:19.299 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 10:28:19.383 PID 0 add/repl neigh 192.168.22.222 state REACHABLE lladdr 00:50:b6:mm:mm:mm iface net0 10:28:20.384 PID 0 add/repl neigh 192.168.22.222 state REACHABLE lladdr 00:50:b6:mm:mm:mm iface net0 10:28:20.474 PID 0 add/repl neigh 192.168.22.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface net0 Additionally, on a single FreeBSD box, I have about 10 WireGuard VPNs set up for network redundancy, and wg-quick for all of them was running this process once or twice per second. Because of this, after updating from FreeBSD 13.x to 14.x, I found that even simple file editing with vi felt slow with the same settings. I was initially puzzled by the cause but found that the above was the culprit. On FreeBSD 13.x, route -n monitor usually had very little output, so it didn't cause a performance issue. My personal suggestions are: Plan A: For FreeBSD 14.x and later, change the wg-quick monitoring method to something other than route -n monitor. Use Netlink to retrieve only the necessary information. Plan A': Base the solution on the existing route code, but reduce the number of queries to Netlink, and use this 'Yet Another route' version within wg-quick. Specifically, I suggest reviewing the monitor_nl function in the sbin/route/route_netlink.c code: void monitor_nl(int fib) { struct snl_state ss_event = {}; struct nl_helper h; nl_init_socket(&ss_event); nl_helper_init(&h); int groups[] = { RTNLGRP_LINK, RTNLGRP_NEIGH, <-- This RTNLGRP_NEXTHOP, #ifdef INET RTNLGRP_IPV4_IFADDR, RTNLGRP_IPV4_ROUTE, #endif #ifdef INET6 RTNLGRP_IPV6_IFADDR, RTNLGRP_IPV6_ROUTE, #endif I think Plans A and A' would be quite difficult to implement. Plan B: Allow users to select whether to run monitor_daemon in a wg.conf file or similar. Currently, I'm not experiencing any issues even when the monitor_daemon isn't running. I wanted to share this since there seems to be an opportunity to re-evaluate the wg-quick monitor_daemon. By the way, on my end, I'm currently operating by killing route -n monitor.
Created attachment 264115 [details] Improved patch based on feedback (In reply to MIHIRA Sanpei Yoshiro from comment #18) Thank you for your feedback - I wonder if the load issues you're facing are actually due to route monitor itself or all the things the script does in reaction to it? I altered the patch to only check for interface events on the wireguard interface in case auto routing is disabled. Therefore it will still catch if the interface is deleted and call del_routes. Technically, this feels like something devd should do, but using it here would require quite some refactoring. It would be nice if you could test if this fixes your load issues. I'm not sure how to optimize this in case auto routing is enabled - like, which events should actually trigger set_endpoint_direct_route. Also, if it wasn't for del_routes, we could probably just avoid running the monitoring daemon automatically in case auto routing is not used. I also wonder if auto routing could actually cause some kind of feedback loop if adding the routes creates more events - so if this is the case, these would need to be filtered? So many questions :) I also noticed that there was an issue how kill worked when having multiple connections, so I refined that part of the patch.
(In reply to Michael Gmelin from comment #19) Thank you for the wg-quick fix. PROBLEM 1: Slow Performance The load has decreased. Original wg-quick (on a machine running five Wireguard instances; bash was handling the process) 85 processes: 2 running, 80 sleeping, 3 stopped CPU: 4.9% user, 0.0% nice, 6.1% system, 10.2% interrupt, 78.8% idle Mem: 51M Active, 1198M Inact, 72M Laundry, 448M Wired, 201M Buf, 210M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 6803 sanpei 4 20 0 31M 7564K uwait 3 48:29 14.05% iperf 1641 smokeping 1 20 0 78M 47M select 3 0:58 0.39% perl 33207 root 1 20 0 14M 3468K CPU0 0 0:00 0.25% top 33063 root 1 20 0 15M 4260K piperd 1 0:00 0.18% bash 32783 root 1 20 0 15M 4408K piperd 1 0:00 0.17% bash 32939 root 1 20 0 15M 4404K piperd 3 0:00 0.17% bash 32858 root 1 20 0 15M 4408K piperd 2 0:00 0.17% bash 33023 root 1 20 0 15M 4420K piperd 1 0:00 0.17% bash 33067 root 1 20 0 12M 1960K sbwait 2 0:00 0.10% route 33027 root 1 20 0 12M 1956K sbwait 3 0:00 0.08% route 32788 root 1 20 0 12M 1968K sbwait 1 0:00 0.08% route 32863 root 1 20 0 12M 1976K sbwait 3 0:00 0.07% route 32949 root 1 20 0 12M 1964K sbwait 2 0:00 0.07% route Improved wg-quick (now only uses route and grep [OK]) 93 processes: 6 running, 84 sleeping, 3 stopped CPU: 0.2% user, 0.0% nice, 1.6% system, 1.8% interrupt, 96.4% idle Mem: 52M Active, 1198M Inact, 72M Laundry, 448M Wired, 201M Buf, 208M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1540 root 1 20 0 12M 1636K select 3 8:04 1.54% powerd 33857 root 1 20 0 14M 3364K CPU1 1 0:00 0.32% top 33760 root 1 20 0 12M 1960K RUN 2 0:00 0.18% route 33459 root 1 20 0 12M 1972K RUN 3 0:00 0.17% route 33546 root 1 20 0 12M 1960K RUN 0 0:00 0.17% route 33718 root 1 20 0 12M 1960K CPU3 3 0:00 0.17% route 33625 root 1 20 0 12M 1968K RUN 2 0:00 0.16% route 33787 root 1 20 0 25M 11M select 0 0:00 0.07% ospfd 33767 root 1 20 0 24M 7900K select 2 0:00 0.03% watchfrr 33784 root 5 20 0 51M 13M select 3 0:00 0.03% zebra 30855 sanpei 1 20 0 24M 10M select 2 0:04 0.03% sshd-session 33761 root 1 20 0 12M 1948K piperd 1 0:00 0.02% grep 33627 root 1 20 0 12M 1948K piperd 2 0:00 0.02% grep 33548 root 1 20 0 12M 1956K piperd 0 0:00 0.02% grep 33461 root 1 20 0 12M 1960K piperd 3 0:00 0.02% grep 33719 root 1 20 0 12M 1956K piperd 0 0:00 0.02% grep 1492 root 1 20 0 13M 2004K select 3 0:16 0.02% syslogd 30860 sanpei 1 20 0 19M 8372K select 3 0:00 0.01% sudo PROBLEM 2: Does the route persist after a reboot? I've confirmed that the route restarts upon reboot on my end as well [OK]. Regarding PROBLEM 1, I think the fundamental issue is that the route command is slow, so it might be a good idea to propose a change to /sbin/route. The route command showing neighbor entries isn't something I saw in the output with the FreeBSD 13 route command. Also, there is a known issue where the USB Ethernet link periodically goes down and up[*1] when monitored via route -n monitor using NetLink. This is due to a section of the USB Ethernet's MII code that doesn't account for SMP (Symmetric Multiprocessing). While the root fix is to modify the USB Ethernet MII section, a workaround could be to stop looking at neighbor entries with route -n monitor. I plan to make time to propose changes for the USB Ethernet MII and the behavior of route -n monitor. I hope you can move forward with applying the wg-quick improvements to the ports. *1 Bug 252165 - usb network and mii bus media status race condition https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252165
Thanks grembo@ for looking into this issue! It might not be the perfect solution but if you think it improves the situation I approve the patch.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=06dfe625ddce3df823fc392c60c81881c36d13a1 commit 06dfe625ddce3df823fc392c60c81881c36d13a1 Author: Michael Gmelin <grembo@FreeBSD.org> AuthorDate: 2025-09-26 11:00:55 +0000 Commit: Michael Gmelin <grembo@FreeBSD.org> CommitDate: 2025-09-29 12:14:18 +0000 net/wireguard-tools: Fix route monitor on FreeBSD 14 PR: 275026 Reported by: Vedran Miletic <vedran@miletic.net> Approved by: decke (maintainer) Co-authored-by: Oleg Streejak <oleg@pcbtech.ru> net/wireguard-tools/Makefile | 2 +- .../files/patch-wg-quick_freebsd.bash (new) | 43 ++++++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-)