Created attachment 246568 [details] Change made to wg-quick to work around this problem. Hi there I've noticed a behavior change that breaks wireguard setups after upgrade from 13.2-p5 to 14-RELEASE. After updating them from 13.2-p5 to 14-RELEASE the wireguard service just stopped working. I noticed that in FreeBSD 14 you are not able to add a route to a wireguard interface that doesn't have an address assigned. Adding a wg interface with an IP address to /etc/rc.conf doesn't help because the script notices the interface is there and doesn't configure it. This is repeatable (tried on both my wireguard hosts, and firewalls and a service router that were all upgraded to 14.0 this evening) and the output looks like this (using wg1 to not interrupt existing traffic on wg0): # freebsd-version 14.0-RELEASE # ifconfig wg1 create # ifconfig wg1 up # route -n add 10.2.0.0/24 -interface wg1 add net 10.2.0.0: gateway wg1 fib 0: Invalid argument # ifconfig wg1 inet 10.0.0.3/32 # route -n add 10.2.0.0/24 -interface wg1 add net 10.2.0.0: gateway wg1 This behavior does not happen in 13.2 (a test VM): # freebsd-version 13.2-RELEASE-p1 # ifconfig wg1 create # ifconfig wg1 up # route -n add 10.2.0.0/24 -interface wg1 add net 10.2.0.0: gateway wg1 Notice if the device is created and up, the route command fails with 'invalid argument.' Once I add an address (any address at all) the route command works but that doesn't fix the wireguard service. To fix it I added an address to the created interface in /usr/local/bin/wg-quick specifically after line 119 where I added: 'cmd ifconfig $INTERFACE inet 127.0.0.200/32' (added diff for better context). This is NOT a good fix at all, it just got me working and back into production again. This would impact anyone upgrading a system running wireguard using the wireguard tools from v13 to v14. It may also be repeatable on tunnel or other devices but I've not seen that in this troubleshooting. Note when broaching this on discord, jlduran pointed out this link appears to be very similar to this but I can't relate it 100% as I don't see any kernel logging in dmesg/syslog (probably because I need to increase verbosity): https://reviews.freebsd.org/D41330
Comment on attachment 246568 [details] Change made to wg-quick to work around this problem. Sorry the first part is the only one that matters, the rest (@@ -315 and forward) is chaff from my troubleshooting.
I can verify that review D41330 fixes the problem. Assuming 14.0-RELEASE sources in /usr/src: # git clone -b releng/14.0 https://git.freebsd.org/src.git /usr/src Apply the patch: # cd /usr/src # patch < fetch https://reviews.freebsd.org/D41330.diff Build and install the kernel: # make buildkernel # make installkernel Reboot: # shutdown -r now The old kernel is under /boot/kernel.old. It should work as expected. Try again: # ifconfig wg1 create # ifconfig wg1 up # route -n add 10.2.0.0/24 -interface wg1 add net 10.2.0.0: gateway wg1 Verify: # netstat -4nr
Sorry, the patch step should have read: # cd /usr/src # fetch https://reviews.freebsd.org/D41330.diff # patch < D41330.diff
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f818559774cb0c1516364c4beca361480fd68b5b commit f818559774cb0c1516364c4beca361480fd68b5b Author: KUROSAWA Takahiro <takahiro.kurosawa@gmail.com> AuthorDate: 2023-11-28 18:14:50 +0000 Commit: R. Christian McDonald <rcm@FreeBSD.org> CommitDate: 2023-11-28 21:11:55 +0000 netlink: fix adding an interface route route add <host> -iface <netif>" for a netif without an IPv4/IPv6 address fails with EINVAL. Need to use a link-level ifaddr for gw if an ifaddr for dst is not found as the rtsock-based implementation does. PR: 275341 Reported by: Sean Cody <sean@tinfoilhat.ca> Reviewed by: rcm Tested by: rcm Approved by: kp (mentor) MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D41330 sys/netlink/route/rt.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
Thanks for the report!
This fix has not been commited to stable/14 for now. I'm waiting a patch for 14.0-RELEASE... Thanks,
This bug is caused by https://cgit.freebsd.org/src/commit/?id=cc3793b1c54847e26001f42026778703970fa570 . The commit has not been MFCed to stable/13 so stable/13 and release/13.3 are not affected right now. Reopen since the fix has not been in stable/14. Link another report: https://lists.freebsd.org/archives/freebsd-net/2024-March/004700.html .
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7a4a122f9731c589831e1c00d85588ff9813b338 commit 7a4a122f9731c589831e1c00d85588ff9813b338 Author: KUROSAWA Takahiro <takahiro.kurosawa@gmail.com> AuthorDate: 2023-11-28 18:14:50 +0000 Commit: Zhenlei Huang <zlei@FreeBSD.org> CommitDate: 2024-03-14 08:27:42 +0000 netlink: fix adding an interface route route add <host> -iface <netif>" for a netif without an IPv4/IPv6 address fails with EINVAL. Need to use a link-level ifaddr for gw if an ifaddr for dst is not found as the rtsock-based implementation does. PR: 275341 Reported by: Sean Cody <sean@tinfoilhat.ca> Reviewed by: rcm Tested by: rcm Approved by: kp (mentor) MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D41330 (cherry picked from commit f818559774cb0c1516364c4beca361480fd68b5b) sys/netlink/route/rt.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
I've tested the change with bird OSPFV3 and the problem is still there after the patch is applied. Until I set explicitely a dummy IP interface on the point to point interface I get the following error: ``` "Mar 31 07:33:01 freebsd bird[17645]: ospfv4: Cannot find next hop addre ss for vlan200" ``` Once you set a "dummy /32" ipv4 on vlan200 the issue disappear. Eg. of configuration: ``` protocol ospf v3 ospfv4 { ipv4 { import filter ospf_import; export filter ospf_export; }; area 0 { interface "lo1" { stub yes; }; interface "vlan200" { type ptp; cost 10; }; }; } ``` ``` ifconfig vlan200 vlan200: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 150 0 options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG> ether 9a:c7:bd:e8:62:dc inet6 fe80::98c7:bdff:fee8:62dc%vlan200 prefixlen 64 scopeid 0x4 groups: vlan vlan: 200 vlanproto: 802.1q vlanpcp: 0 parent interface: iavf0 media: Ethernet autoselect (10Gbase-SR <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> ```
(In reply to benoitc from comment #9) > I've tested the change with bird OSPFV3 and the problem is still there after the patch is applied. That is weird. Can you please share your version? Is that 14.0-RELEASE or stable/14 ?
(In reply to Zhenlei Huang from comment #10) this was on 14-RELEASE with patch applied. I will retry on 14-STABLE
(In reply to benoitc from comment #11) Please feel free to try using BGP, for OSPFv3 it's not implemented yet. Maybe with the next version of BIRD it will work[1]. 1. https://bird.network.cz/pipermail/bird-users/2024-April/017555.html