Created attachment 223365 [details] default virtio iperf3 speed ipv6 FreeBSD 13.0-RC2 I installed FreeBSD 13.0-RC2 on some of my database virtual machines using the VirtIO network adapter. The VMs only use IPv6. After the upgrade MySQL threads kept filling up with data waiting to be sent. The same setup worked flawlessly with 12.2. Upon investigation I noticed that the IPv6 transfer speed from the VM to any other device is in the range of 46Kbps. Usually it is 15-20Gbps from VM to VM on the same bare metal machine. If IPv4 is used no issues at all on FreeBSD 13.0-RC2. This issue was also recently discussed on the freebsd-net list under the "severe ipv6 tcp transfer issues" subject. The problem can be easily reproduced by setting up two virtual machines running the default FreeBSD 13.0-RC2 release candidate. (Note the problem already exists in BETA4). Configure the network device using virtio and setup IPv6. Run a simple iperf3 session from VM to VM on IPv6 and the slowness shows up every time in both directions. I tested it with virt-manager as well as using Proxmox and both show that issue. A quick fix was to issue ifconfig vtnet0 -tso6. That brings transfer speeds back to acceptable levels in the Gbps range. However it is still much slower for vm-vm-same-host than IPv4. About 4Gbps on IPv6 vs 15-20Gbps IPv4. FreeBSD 12.2 has the same 15-20Gbps speed IPv4/IPv6. My thoughts were that the recent virtio changes committed by Bryan Venteicher were the culprit. I started bisecting the changes submitted 2021-01-19. After building a few kernels it is the following commit causing the issue: 475a60aec7e if_vtnet: Misc Tx path cleanup review D27926 Building a kernel without that commit brings back IPv4/IPv6 on the same level as before. Not much of kernel geek so I'm not sure what part of the code is causing the problem to show up in IPv6 only. Attached are a couple iperf3 & ifconfig commands to show the slowness. If anything else is needed, please let me know, but all you need is stock FreeBSD 13.0-RC2 VM using virtio and ipv6.
Created attachment 223366 [details] iperf3 transfer speed ipv4 on FreeBSD 13.0-RC2
Created attachment 223367 [details] iperf3 ipv6 with ifconfig vtnet0 -tso6 FreeBSD 13.0-RC2
Created attachment 223368 [details] ifconfig defaults on FreeBSD 13.0-RC2 virtio
Created attachment 223369 [details] dmesg Also it doesn't matter if the machine configuration is q35 or i440fx.
>After building a few kernels it is the following commit causing the issue: 475a60aec7e if_vtnet: Misc Tx path cleanup Did you just remove this commit, or rolled back the branch to the commit before 475a60aec7e? I tried to reproduce this bug with bhyve - no success. FreeBSD-13-RC1 <-> FreeBSD-13-RC1 - 10-20 Gbit/s. FreeBSD-13-Rc1 <-> Ubuntu - 10-20 Gbit/s.
What is the output of sysctl dev.vtnet after you did a transfer showing the problem?
I only removed that single 475a60aec7e commit. I haven't tried it with bhyve. Maybe the issue only shows up if the hypervisor runs on a Linux system? I got a copy of Ubuntu Desktop 20.04.2.0 (and updated) and setup FreeBSD 13 using virt-manager. Same thing happens with a copy of the latest Proxmox setting up FreeBSD 13. I'll run another test and get the sysctl dev.vtnet output. I'll get two outputs one with the regular RC2 kernel and the other with the 475a60aec7e commit removed.
guess the same issue was reported on freebsd-net@ by Blake Hartshorn. The .pcap files provided (privately) indicated that all segments sent larger than the MTU were dropped by the host OS. So it is a TSO issue...
I guess so too. That's why also ifconfig vtnet0 -tso6 brings back acceptable speeds. But without the TSO the vm-to-vm throughput on the same host is not anywhere near IPv4 or what it was on FreeBSD 12 for IPv6.
(In reply to Michael Tratz from comment #9) I'm not arguing that there is a bug. I'm just trying to figure out what is wrong with the patch. Also trying to reproduce this locally...
Created attachment 223398 [details] sysctl dev.vtnet (slow) No worries Michael! Note: vtnet1 is the interface I'm using for testing. So look at that output. The VM was freshly started for the test. The iperf3 was run for 30 seconds.
Created attachment 223399 [details] Fast iperf3 without 475a60aec7e Here is the same output running the kernel without the 475a60aec7e commit.
OK, I know what the problem is. I can reproduce it locally. Thanks for the information. Will post a patch soon.
Created attachment 223401 [details] Proposed fix
(In reply to Michael Tratz from comment #12) Can you try the Proposed fix?
Thank you so much Michael for the quick fix. Everything is back to normal. IPv6 is speedy again.
(In reply to Michael Tratz from comment #16) Thanks for reporting back! The path is under review: https://reviews.freebsd.org/D29331
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d4697a6b56168876fc0ffec1a0bb1b24d25b198e commit d4697a6b56168876fc0ffec1a0bb1b24d25b198e Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2021-03-18 20:25:47 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2021-03-18 20:32:20 +0000 vtnet: fix TSO for TCP/IPv6 The decision whether a TCP packet is sent over IPv4 or IPv6 was based on ethertype, which works correctly. In D27926 the criteria was changed to checking if the CSUM_IP_TSO flag is set in the csum-flags and then considering it to be TCP/IPv4. However, the TCP stack sets the flag to CSUM_TSO for IPv4 and IPv6, where CSUM_TSO is defined as CSUM_IP_TSO|CSUM_IP6_TSO. Therefore TCP/IPv6 packets gets mis-classified as TCP/IPv4, which breaks TSO for TCP/IPv6. This patch bases the check again on the ethertype. This fix will be MFC instantly as discussed with re(gjb). MFC after: instantly PR: 254366 Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D29331 sys/dev/virtio/network/if_vtnet.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6064ea8172f078c54954bc6e8865625feb7979fe commit 6064ea8172f078c54954bc6e8865625feb7979fe Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2021-03-18 20:25:47 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2021-03-18 20:33:32 +0000 vtnet: fix TSO for TCP/IPv6 The decision whether a TCP packet is sent over IPv4 or IPv6 was based on ethertype, which works correctly. In D27926 the criteria was changed to checking if the CSUM_IP_TSO flag is set in the csum-flags and then considering it to be TCP/IPv4. However, the TCP stack sets the flag to CSUM_TSO for IPv4 and IPv6, where CSUM_TSO is defined as CSUM_IP_TSO|CSUM_IP6_TSO. Therefore TCP/IPv6 packets gets mis-classified as TCP/IPv4, which breaks TSO for TCP/IPv6. This patch bases the check again on the ethertype. This fix is instantly MFCed. Approved by: re(gjb) PR: 254366 Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D29331 (cherry picked from commit d4697a6b56168876fc0ffec1a0bb1b24d25b198e) sys/dev/virtio/network/if_vtnet.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
A commit in branch releng/13.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=11660fa28fd39a644cb7d30a4378cf4751b89f15 commit 11660fa28fd39a644cb7d30a4378cf4751b89f15 Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2021-03-18 20:25:47 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2021-03-18 20:35:49 +0000 vtnet: fix TSO for TCP/IPv6 The decision whether a TCP packet is sent over IPv4 or IPv6 was based on ethertype, which works correctly. In D27926 the criteria was changed to checking if the CSUM_IP_TSO flag is set in the csum-flags and then considering it to be TCP/IPv4. However, the TCP stack sets the flag to CSUM_TSO for IPv4 and IPv6, where CSUM_TSO is defined as CSUM_IP_TSO|CSUM_IP6_TSO. Therefore TCP/IPv6 packets gets mis-classified as TCP/IPv4, which breaks TSO for TCP/IPv6. This patch bases the check again on the ethertype. This fix is instantly MFCed. Approved by: re(gjb) PR: 254366 Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D29331 (cherry picked from commit d4697a6b56168876fc0ffec1a0bb1b24d25b198e) (cherry picked from commit 6064ea8172f078c54954bc6e8865625feb7979fe) sys/dev/virtio/network/if_vtnet.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
This will be in the upcoming RC3.