Hi Kristof, I still owe you a proper bug ticket for the issue we shortly talked about in Coimbra. During the winter holidays when I planned to work on "open source things" I fell ill with covid - so that was that. Anyway, the issue is still quite easily reproducible in 14.0-p5. I have this virtual machine at Vultr located in New Jersey that I use as a wireguard (kernel module as standard in releng/14) endpoint for obvious reasons. The uplink has a single public IPv4 and a single public IPv6 address so I use tunnel networks for both protocols and NAT both outbound to the interface address. Wireguard config: ---- [Interface] Address = 192.168.254.1/24,2003:a:d59:3840::1/64 PrivateKey = *** ListenPort = 51820 # PMH [Peer] PublicKey = *** AllowedIPs = 192.168.254.2/32,2003:a:d59:3840::2/128 ---- The client (Mac OS) has 0.0.0.0/0 and ::/0 in its AllowedIPs routing the entire traffic through the tunnel. For IPv4 I use an RFC 1918 network, obviously. For IPv6 I use a GUA /64 from my allocation at home, because ULA are essentially useless the way current desktop OSes treat them. pf NAT config: ---- nat on vtnet0 from 192.168.254.0/24 to any -> 140.82.8.233 nat on vtnet0 from 2003:a:d59:3840::/64 to any -> 2001:19f0:5:2634:5400:3ff:fefa:33f9 pass all no state ---- I am currently sitting behind a local 100 Mbit/s symmetrical uplink. Using Ookla Speedtest I get: Local connection without VPN enabled: 90/90 Mbit/s VPN connection with vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6: 80/80-ish Mbit/s, no noticeable slowdown VPN connection with vtnet0 without -rxcsum -txcsum -rxcsum6 -txcsum6: 0.3 Mbit/s download while upload seems to be unaffected I can give you root access to the server if required. Kind regards, Patrick
Looks like the bug is addressed directly to Kristof.
Unfortunately, the main problem is that all our NAT implementations do not support offload. Therefore, you need to turn it off.
From an operational point of view understood and perfectly valid. E.g. OPNsense comes with hardware offloading switched off by default. Kristof claimed in private conversation that this *should* work and I promised to file a ticket with a reproducible case.
(In reply to Alexander Fedorov from comment #2) > Unfortunately, the main problem is that all our NAT implementations do not support offload. Iirc. that's the case on ipfw, but not on pf. I fixed a checksum issue in pf many years ago: https://cgit.freebsd.org/src/commit/?id=c110fc49da2995d10d60d908af0838ecb4be9bee (PR 154428, 193579, 198868). > 0.3 Mbit/s download while upload seems to be unaffected From what perspective is that? As in: is the issue with packets the relevant machine sends or receives? Did you tcpdump to confirm that it's a checksum issue?
I encountered the same issue when I was using AppJail apply nat to jail on my hetzner VPS. You can see the full context here: https://github.com/DtxdF/AppJail/issues/8 Just like Patrick M. Hausen's case, the upload speed is been affected but not that much(1659.30Mbps inside jail vs 2853.11Mbps outside jail, 40% slower but still fast enough), but the download speed is disastrous(19.49Mbps inside jail vs 6298.97Mbps outside jail, 323x times slower.)
(In reply to Orville Q. Song from comment #5) After setting hw.vtnet.csum_disable="1" in /boot/loader.conf, the download and upload speeds inside the jail are the same as outside the jail now.
Hi Kristof, I apologize for not getting back to you sooner. I first missed your update of the issue and then I had indeed a hard time reproducing it in a local environment. Turns out that this seems to be an issue in KVM with VirtIO network and FreeBSD as a guest and can only be fixed on the KVM side. I received this info from Antranig Vartanian, who promised to update this issue with more detailed information. Allegedly he is working with vultr.com on this. Just to give you an update. As far as I'm concerned we might also just close it. hw.vtnet.csum_disable="1" is good enough for me. Kind regards, Patrick
Is this still an issue? If yes, is understood what is happening and what causes the issue?
Hi Michael, this is exactly one of the dozen or so bug reports all related to partial checksums and offloading in KVM guest situations. You may close it. When will we see your latest work fixing that eternal issue in a release? Will it be MFC'd to releng/14? Kind regards, Patrick
Is this documented anywhere? I spent days trying to figure out why NAT performance fell over while setting up a VPN server on Vultr and the only way I found out about it was via this and other bug reports. Could a note about KVM land in vtnet(4)?
No note needed - it's fixed. By Michael.
Just to be clear, this is a fix in the KVM side (someone above states that's the only place it can be fixed) or a workaround (disabling checksums?) on the FreeBSD side?
(In reply to spork from comment #12) We have improved the handling of freebsd when receiving packets which do not have a valid checksum yet. This avoids sending packets with incorrect checksums. review D52684 will disable hardware LRO in the default setup. This is what physical NICs also do when they support hardware and software LRO. This avoids that oversized packets are sent and dropped, which requires timer based retransmissions which kill the performance. The reason I am asking is that I don't have a setup to test that the issue is resolved when vtnet interfaces are used in combination with NAT. That is the reason I am asking if someone experiencing the problem can test stable/14 or stable/15 or current?
(In reply to punkt.de Hosting Team from comment #9) Hi Patrick, most of the fixes are in stable/14, stable/15, current. review D52684 is missing, but could be tested by putting hw.vtnet.lro_disable=1 in /boot/loader.conf. This disables hardware LRO, but allows for software LRO. Best regards Michael
I can test in a couple of days - after EuroBSDCon ;-) So let's leave it open until then.
(In reply to punkt.de Hosting Team from comment #15) Perfect, thank you very much! Enjoy EuroBSDCon! Best regards Michael
(In reply to punkt.de Hosting Team from comment #15) All fixes are now in current, stable/15, and stable/14. Please let me know if there are any problems with the default setup.
(In reply to punkt.de Hosting Team from comment #15) Did you had a chance to test?
(In reply to punkt.de Hosting Team from comment #15) ping
(In reply to punkt.de Hosting Team from comment #15) Can you retest? Or let me know that you can't.
Apologies for reporting back so late. I am currently a bit puzzled. I deployed a fresh VPC at Vultr.com in London, installed FreeBSD 14.3 and updated to 14.3-RELEASE-p7. I recreated the setup from my initial post: - wireguard tunnel from my desktop Mac to this machine - outbound NAT with pf so I can use this tunnel as an exit point to the Internet 1. iperf3 from my desktop to the VPC, no tunnel or anything active 30 Mbit/s up, 30 Mbit/s down 2. Tunnel active, iperf3 from the inner tunnel address of my Mac to the inner tunnel address of the VPC 30 Mbit/s up, 30 Mbit/s down 3. Tunnel active, default gateway to the tunnel, so NAT outbound from the VPC, iperf3 from my Mac to a different system at Vultr in the UK so geographically close 30 Mbit/s up, 30 Mbit/s down Numbers rounded up a bit, but this looks quite well. Is this expected with 14.3-p7? Or has the problem vanished for some other reason? Also for now I only tested IPv4 ... IPv6 next.
Now the puzzling thing is, when I run Ookla Speedtest I get: 30 Mbit/s up, 0.2 Mbit/s down But with hw.vtnet.csum_disable="1" in /boot/loader.conf Speedtest gives these results: 30 Mbit/s up, 1.5 Mbit/s down So I am not quite sure if I am even reproducing the original problem, anymore or measuring some other effect. MTU on the tunnel is forced to 1280 on the Mac side, so I can rule out MTU problems due to stacked encapsulation. Any suggestion what to try next? My use case is obviously a fast tunnelled connection with outbound NAT.
(In reply to Patrick M. Hausen from comment #22) So vtnet fixes are in stable/14, not releng/14.3. Is it possible for you to upgrade to stable/14? Then we can look at the bad results from Ookla Speedtest. But I would prefer to debug existing problems instead of problem which might already be fixed.
Buildworld running ...
(In reply to Patrick M. Hausen from comment #24) Thanks. Kernel would be good enough, I think. But it is always better to have a consistent state... Thanks for helping to nail down the issue.
Isn't buildworld before buildkernel mandatory if you just checked out a fresh source tree and there is no populated /usr/obj? I haven't installed/updated from source in years thanks to freebsd-update and soon pkgbase. Anyway, all done and all looks well. Up: 30 Mbit/s, Down: 80 Mbit/s Through WG tunnel and pf NAT. No tunables whatsoever.
root@uk:~ # freebsd-version 14.3-STABLE root@uk:~ # uname -a FreeBSD uk.hausen.com 14.3-STABLE FreeBSD 14.3-STABLE stable/14-a5c6c9563119 GENERIC amd64
(In reply to Patrick M. Hausen from comment #26) When doing kernel development, I usually only compile the kernel and use the older userland. Usually this works and in this case I do not expect that the userland has an impact. I am glad to hear that the problem is fixed in stable/14. Thank you very much for testing! Are you fine with me closing this bug report?
Of course - about time it is closed. One last question for the record: is it also fixed in releng/15.0 or in stable/15 only just like with stable/14? Because I originally started with a pkgbase 15.0 installation and got completely inconsistent results, then started over with 14.3-p7, which did not improve anything. Explicably so. Thanks! Patrick
Ah ... and: > When doing kernel development, I usually only compile the kernel and use the older userland. I know that at least in the past you could not do a "make buildkernel" without a "make buildworld" first. At least not with an empty /usr/obj.
(In reply to Patrick M. Hausen from comment #29) I think stable/14, stable/15 and releng/15.0 are in sync with respect to vtnet. If you observe a difference between them, we should look into it. It might be related to something else. Please open a new bug report in that case.
(In reply to Patrick M. Hausen from comment #30) I think I do that all the times. Checkout the freebsd-src tree, make kernel, reboot...