I am on OpnSense 25.7.6 that uses FreeBSD 14.3-stable. I got Wireguard working. Then I added CARP interfaces on LAN and WAN side. Then handshake does not complete. I see this in the pflog0 traffic on VPN initiation: tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 256 bytes 00:00:00.000000 rule 191/0(match): pass in on igc0: (tos 0x0, ttl 56, id 50850, offset 0, flags [DF], proto UDP (17), length 176) a.b.c.d.29106 > 192.168.178.2.51820: UDP, length 148 00:00:00.000028 rule 137/0(match): pass out on igc0: (tos 0x0, ttl 55, id 50850, offset 0, flags [DF], proto UDP (17), length 176) a.b.c.d.29106 > 192.168.178.2.51820: UDP, length 148 a.b.c.d is the public IP address of my peer. 192.168.178.2 is my CARP interface on the WAN side. The pass in is the handshake initiation from the peer. The pass out is the response thereof. Notice, the src and dst are reversed. That's why handshake does not complete (if it started at all). I need help in diving deep and undoing src and dst reversal. Snippet from netstat -rnfinet 10.251.0.0/16 link#23 U wg1 10.251.0.1 link#7 UHS lo0 10.251.0.2 link#23 UHS wg1 10.251.0.3 link#23 UHS wg1 Enable debugging on wg1 gives me: wg1: Handshake for peer 1 did not complete after 5 seconds, retrying (retry 15) wg1: Sending handshake initiation to peer 1 wg1: Unable to send packet: 47 I looked into the FreeBSD wg(4) driver source code: https://cgit.freebsd.org/src/tree/sys/dev/wg/if_wg.c?h=stable/14&id=684dd4e8c0085f4e64016a44d4cd699b2fc29681 line 1014 prints "Unable to send packet: " https://cgit.freebsd.org/src/tree/sys/sys/errno.h?h=stable/14 in line 110 I found this: #define EAFNOSUPPORT 47 /* Address family not supported by protocol family */ 47 means it cannot handle the address family. This is weird to me because I use IPv4 only. IPv6 is disabled. From looking into the code, I suppose the stacktrace looks like this: wg_send_initiation() line 1290 wg_peer_send_buf() line 1278 wg_send_buf() line 987 wg_send() line 935 return EAFNOSUPPORT; line 960
Could be related to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285650
More verbose tcpdump output: tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on igc0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:23:13.401182 e0:28:6d:89:6b:02 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 190: a.b.c.d.18457 > 192.168.178.2.51820: UDP, length 148 17:23:13.401227 00:90:27:e6:33:13 > e0:28:6d:89:6b:02, ethertype IPv4 (0x0800), length 190: a.b.c.d.18457 > 192.168.178.2.51820: UDP, length 148 a.b.c.d is the public IP address of my peer. 192.168.178.2 is my CARP address on the WAN side. The first line is the handshake initiation from the peer. The second line is the response thereof. Notice, the src IP and dst IP are reversed while the MAC-Addresses are ok. Layer 2 is ok. The problem is in layer 3. Maybe there is a bcopy() or memcpy() somewhere with src and dst arguments swapped?
This is the WAN interface. I hope this is helpful, too. igc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: igc0_WAN (wan) options=4e427bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 00:90:27:e6:33:13 inet 192.168.178.10 netmask 0xffffff00 broadcast 192.168.178.255 inet 192.168.178.2 netmask 0xffffff00 broadcast 192.168.178.255 vhid 1 carp: MASTER vhid 1 advbase 1 advskew 0 peer 224.0.0.18 peer6 ff02::12 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
What's unclear is where CARP might be breaking an assumption that wg(4) is making. The source address is set with IP_SENDSRCADDR using udp_in[1] that we got via udp_input() -> udp_append() -> tunnel func -- is that address wrong, or is it getting dorked up on udp_send()? Actually, I'm kind of eyeballing this commit[0] by glebius and wondering if it's still a problem in 15.0 / main. The wg(4) socket has previously been sobind() to 0.0.0.0:<port>, which presumably takes us into this branch: if (inp->inp_laddr.s_addr == INADDR_ANY || inp->inp_lport == 0 || sin->sin_addr.s_addr == INADDR_ANY || sin->sin_addr.s_addr == INADDR_BROADCAST) { So we call in_pcbconnect_setup(), which at a minimum probably clobbers our specified src address in laddr.s_addr. I don't know where CARP comes in or if this ends up somehow clobbering in this precise way, but finding a way to test this on main or stable/15 would be a good start towards working on a resolution -- at the very least, if udp_send() wasn't wreaking some havoc there, we can try a custom kernel with more diagnostics to figure out if it's messing up on the way *in* to if_wg(4) or on the way out. [0] https://cgit.freebsd.org/src/commit/?id=69c05f42871406b4b2b2dac00a268d1da0cacd3e
I run three instances of tcpdump on wg1, one capturing any tcp, one capturing any udp and one capturing any icmp traffic. Then I initiated the wireguard handshake from my peer. The result of all three tcpdump instances: 0 packets captured 0 packets received by filter 0 packets dropped by kernel If the problem were on the way out of if_wg(4) then tcpdump would have catched any tcp, udp and/or icmp traffic. But there is no traffic within if_wg(4). Therefore, the problem must be on the way *in* to if_wg(4). P.S. Let me know if I should run tcpdump with a specific filter.
(In reply to relvy from comment #5) I think packets get tapped out to bpf simultaneously way too early and too late to draw any more useful conclusions from tcpdump. Entire layers of stack of opportunities to screw things up. That said, we probably can conclude it's a problem on the way out because we actually associated incoming with a peer and seemingly tried twice with the same IP involved- so we successful recorded the source as the peer's endpoint.
(In reply to Kyle Evans from comment #4) I tested the mentioned commit. The tcpdump output: tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on igc0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:55:41.268806 e0:28:6d:89:6b:02 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 190: a.b.c.d.60586 > 192.168.178.2.51820: UDP, length 148 17:55:41.268880 00:90:27:e6:33:13 > e0:28:6d:89:6b:02, ethertype IPv4 (0x0800), length 190: a.b.c.d.60586 > 192.168.178.2.51820: UDP, length 148 17:55:41.873931 00:90:27:e6:33:13 > e0:28:6d:89:6b:02, ethertype IPv4 (0x0800), length 190: 192.168.178.10.51820 > a.b.c.d.51820: UDP, length 148 17:55:46.336997 e0:28:6d:89:6b:02 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 190: a.b.c.d.60586 > 192.168.178.2.51820: UDP, length 148 17:55:46.337014 00:90:27:e6:33:13 > e0:28:6d:89:6b:02, ethertype IPv4 (0x0800), length 190: a.b.c.d.60586 > 192.168.178.2.51820: UDP, length 148 17:55:47.062936 00:90:27:e6:33:13 > e0:28:6d:89:6b:02, ethertype IPv4 (0x0800), length 190: 192.168.178.10.51820 > a.b.c.d.51820: UDP, length 148 And in dmesg I see: [78] wg1: Handshake for peer 1 did not complete after 5 seconds, retrying (try 13) [78] wg1: Sending handshake initiation to peer 1 [78] wg1: Unable to send packet: 47 So this indeed does fix the src / dst IP reversal. There are two issues left: First, the destination *port* in the reply is not correct Second, in the message "Unable to send packet: 47" the 47 means unknown address family.
Created attachment 265312 [details] git(1) diff against base I suspect we need this at a minimum because udp_send() may modify our sockaddr (and if there's some translation after we sosend(), we probably want to re-apply the transformation every time). I don't think it'll be a complete solution, though.
(In reply to Kyle Evans from comment #8) The output of `wg show` for this peer would also be useful, specifically noting if the `endpoint` depicts the address correctly (both address and port) with the patched kernel. As soon as we match an incoming handshake attempt from a.b.c.d, we record that as the new endpoint.
(In reply to Kyle Evans from comment #8) I tested the patch on top of the other one. tcpdump output: 08:29:57.135466 e0:28:6d:89:6b:02 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 190: a.b.c.d.23423 > 192.168.178.2.51820: UDP, length 148 08:29:57.135523 00:90:27:e6:33:13 > e0:28:6d:89:6b:02, ethertype IPv4 (0x0800), length 190: a.b.c.d.23423 > 192.168.178.2.51820: UDP, length 148 08:30:02.246290 e0:28:6d:89:6b:02 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 190: a.b.c.d.23423 > 192.168.178.2.51820: UDP, length 148 08:30:02.246315 00:90:27:e6:33:13 > e0:28:6d:89:6b:02, ethertype IPv4 (0x0800), length 190: a.b.c.d.23423 > 192.168.178.2.51820: UDP, length 148 I see no incoming traffic when I initiate the VPN from my peer. I see only the reply with correct src / dst IP and with reversed src / dst port. In dmesg I still see the error code 47 (EAFNOSUPPORT) [330] wg1: Handshake for peer 1 did not complete after 5 seconds, retrying (try 12) [330] wg1: Sending handshake initiation to peer 1 [330] wg1: Unable to send packet: 47 "wg show" shows a difference with this patch. Without this patch: $ wg show interface: wg1 public key: <secret> listening port: 51820 peer: <secret> endpoint: a.b.c.d:51820 allowed ips: 10.251.0.3/32 transfer: 0 B received, 261.17 KiB sent persistent keepalive: every 30 seconds With this patch there is no endpoint recorded because the incoming traffic is "missing": $ wg show interface: wg1 public key: <secret> listening port: 51820 peer: <secret> allowed ips: 10.251.0.3/32 transfer: 0 B received, 11.27 KiB sent persistent keepalive: every 30 seconds
(In reply to Kyle Evans from comment #8) Concering transformation: Yes, I do an outbound NAT mapping from the WAN interface to the CARP address.
(In reply to relvy from comment #10) > I see no incoming traffic when I initiate the VPN from my peer. > I see only the reply with correct src / dst IP and with reversed src / dst port. I have to correct myself. tcpdump shows incoming traffic but no outgoing traffic.
(In reply to relvy from comment #12) While I rub a few brain-cells together to try and understand what's happening: a.b.c.d is external to your local network, correct? Is there NAT involved for a.b.c.d to pass a packet in to 192.168.178.2?
(In reply to Kyle Evans from comment #13) > a.b.c.d is external to your local network, correct? Is there NAT involved for a.b.c.d to pass a packet in to 192.168.178.2? Infrastructure: LAN opnsense WAN ------------ opnsense WAN CARP -------------- Gateway -------- Internet ----- peer (road warrior) igc0 192.168.178.2 a.b.c.d peer initiates handshake LAN opnsense WAN ------------ opnsense WAN CARP -------------- Gateway -------- Internet ----- peer (road warrior) igc0 192.168.178.2:51820 <----- Port forwarding <---------------- a.b.c.d outgoing traffic: LAN opnsense WAN ------------ opnsense WAN CARP -------------- Gateway -------- Internet ----- peer (road warrior) igc0 -- outbound NAT --> 192.168.178.2 ---------------------------------------------> a.b.c.d
> Can you try again initiating from the peer and add relevant lines from pfctl -s states for a.b.c.d, please? all udp 192.168.178.2:51820 <- a.b.c.d:23423 NO_TRAFFIC:SINGLE
> Can you try again initiating from the peer and add relevant lines from pfctl -s states for a.b.c.d, please? I wish I could edit comments :) all udp 192.168.178.2:51820 <- a.b.c.d:23423 NO_TRAFFIC:SINGLE all udp a.b.c.d:23423 -> 192.168.178.2:51820 SINGLE:NO_TRAFFIC
Are there any updates? If not I would like to bring up the EAFNOSUPPORT issue that is still there: wg1: Handshake for peer 1 did not complete after 5 seconds, retrying (retry 15) wg1: Sending handshake initiation to peer 1 wg1: Unable to send packet: 47 The "Unable to send packet" debug message comes from if_wg.c:wg_send_buf(). wg_send_buf() -> wg_send() -> return EAFNOSUPPORT; -> wg_send_buf() -> DPRINT(sc, "Unable to send packet: %d\n", ret); This means e->e_remote.r_sa.sa_family is neither AF_INET nor AF_INET6. I suppose carp zeroes that out and we run the error path all the way along.
Update: To reproduce the issue the CARP backup is not necessary. Only configure the CARP master with wireguard is necessary to reproduce.
(In reply to relvy from comment #17) I don't really see how that's possible at the moment; we get that addrerss one of a few different ways: 1.) wg(8) set it via the ioctl interface, but it only does that for AF_INET / AF_INET6 2.) When we receive a valid handshake initiation 3.) When we receive a valid handshake response 4.) When we receive a valid packet over the tunnel For #2 - 4, those are pulled from the `pkt->p_endpoint`, which is populated in wg_input(). If we didn't get an AF_INET / AF_INET6 address *there*, the packet is dropped. You say that IPv6 is disabled, but I don't think opnsense produces any kernel configs with it completely ripped out, so EAFNOSUPPORT from it being an AF_INET6 is likely not what's happened -- something presumably went wrong in the sosend() path instead.
(In reply to relvy from comment #18) So, to trigger the issue it is enough to configure CARP master? can you provide short instruction how to reproduce the issue?
(In reply to Kyle Evans from comment #19) > I don't really see how that's possible at the moment; we get that addrerss one of a few different ways: > 1.) wg(8) set it via the ioctl interface, but it only does that for AF_INET / AF_INET6 > 2.) When we receive a valid handshake initiation > 3.) When we receive a valid handshake response > 4.) When we receive a valid packet over the tunnel For road warrior configuration I do not use #1. "Disabling IPv6" in OpnSense is to block all IPv6 traffic via PF rules.
Created attachment 265617 [details] Print family address in error path Add debug prints in error path to figure out what family address is actually used.
(In reply to vova from comment #20) I took out the carp backup machine from production environment and put it into a testbed with a single subnet 10.2.0.0/16. I am able to reproduce it there, too. I stripped it down to figure out what is necessary to reproduce. I removed the LAN CARP: issue is still reproduceable. I removed wg0 and kept wg1: issue is still reproduceable. What I have now is: Infrastructure: opnsense WAN ------------ opnsense WAN CARP ------- peer (road warrior) igc0 10.2.1.30 a.b.c.d peer initiates handshake: opnsense WAN ------------ opnsense WAN CARP ----- peer (road warrior) igc0 10.2.1.30:51820 <------- a.b.c.d outgoing traffic: LAN opnsense WAN ------------ opnsense WAN CARP ----- peer (road warrior) igc0 -- outbound NAT --> 10.2.1.30 -----------> a.b.c.d a.b.c.d is the peer address, an DHCP address from 10.2.0.x/16 # pfctl -s states | grep 51820 all udp 10.2.1.30:51820 <- a.b.c.d:53003 NO_TRAFFIC:SINGLE all udp a.b.c.d:53003 -> 10.2.1.30:51820 SINGLE:NO_TRAFFIC
(In reply to vova from comment #20) > can you provide short instruction how to reproduce the issue? Install & setup Wireguard instance on an wg interface. Add a CARP virtual IP to your ethernet interface (your outgoing interface if you have more than one). Add an outgoing NAT rule that does Source NAT from your ethernet IP address to your CARP address. Setup a Wireguard peer. Do *not* use wg(8) to set the peers endpoint address. When setup is done, initiate the connection from the peer. On the peer you should *not* see any incoming traffic. On your wg instance, see tcpdump -n -i <interface> udp and port 51820 see ifconfig <interface> debug and dmesg see wg show see pfctl -s states | grep 51820
(In reply to relvy from comment #24) It appears to be an advanced and complex setup, resulting in a hard-to-reproduce issue. I wonder - was it ever supported ? Perhaps it's not WireGuard, but a carp(4) or a firewall issue ? Was PF or IPFW used? Or perhaps IPF in this case ? Could you post a detailed or, even better, a scripted way to reproduce ?
(In reply to Marek Zarychta from comment #25) I can answer one question: > Was PF or IPFW used? Or perhaps IPF in this case ? OpnSense uses pf for filter/nat/scrub and ipfw for dummynet shaper (if shaper rules are enabled)
(In reply to relvy from comment #26) > > Was PF or IPFW used? Or perhaps IPF in this case ? > OpnSense uses pf for filter/nat/scrub and ipfw for dummynet shaper (if shaper rules are enabled) I realized that I use traffic shaper to target bufferbloat. In the testbed I removed the traffic shaper and still reproduced the issue. At the very least this rules out IPFW and IPF. PF is still on. There are two PF commits that look interesting to me: https://cgit.freebsd.org/src/commit/sys?id=c12013f5bb3819e64499f02ecd199a635003c7ce https://cgit.freebsd.org/src/commit/sys?id=7dedc3c21436bb5a1220f8901992d2772a163f78 Let me know what you think.
(In reply to relvy from comment #22) With the family address debug print patch applied I see in dmesg: wg1: Sending handshake initiation to peer 1 wg1: wg_send: Unknown family: 0 wg1: Unable to send packet: 47 I do not see the wg_input debug output. The family address in the sosend() path is 0.
(In reply to relvy from comment #28) Right, that was pretty expected... as I noted back in #19, it must be sosend() itself based on how the address is populated. Sorry, I haven't had much time to push this further- I have a test case that just exercises CARP + wg with a no-backup, but I haven't yet been able to work pf into it to see if that breaks upstream.
(sorry, looked at this too early in the morning; let me think about this some more)
I have tested this patch [0] and with options INVARIANTS, INVARIANTS_SUPPORT, KASSERT_PANIC_OPTIONAL, KDTRACE_HOOS, KDTRACE_FRAME and DDB_CTF enabled. With this I see nothing in dmesg. Nothing after boot and nothing after I initiated handshake from the peer. In tcpdump I see 19:00:34.416053 IP a.b.c.d.47305 > 10.2.1.30.51820: UDP, length 148 19:00:34.416057 IP a.b.c.d.47305 > 10.2.1.30.51820: UDP, length 148 19:00:34.418392 IP a.b.c.d.47305 > 10.2.1.30.51820: UDP, length 148 pfctl -s states | grep 51820 all udp 10.2.1.30:51820 <- a.b.c.d:45350 NO_TRAFFIC:SINGLE all udp a.b.c.d:45350 -> 10.2.1.30:51820 SINGLE:NO_TRAFFIC No assertion is triggert. [0] https://people.freebsd.org/~kevans/wg-keepalive.diff