Bug 254366
| Summary: | Severe IPv6 TCP transfer issues on 13.0-RC2 with virtio network adapter | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Michael Tratz <michael> |
| Component: | kern | Assignee: | Michael Tuexen <tuexen> |
| Status: | Closed FIXED | ||
| Severity: | Affects Some People | CC: | afedorov, andrey, bryanv, net, networker-bugs, tuexen |
| Priority: | --- | Flags: | tuexen:
mfc-stable13+
|
| Version: | 13.0-STABLE | ||
| Hardware: | amd64 | ||
| OS: | Any | ||
| Attachments: | |||
|
Description
Michael Tratz
2021-03-17 18:07:23 UTC
Created attachment 223366 [details]
iperf3 transfer speed ipv4 on FreeBSD 13.0-RC2
Created attachment 223367 [details]
iperf3 ipv6 with ifconfig vtnet0 -tso6 FreeBSD 13.0-RC2
Created attachment 223368 [details]
ifconfig defaults on FreeBSD 13.0-RC2 virtio
Created attachment 223369 [details]
dmesg
Also it doesn't matter if the machine configuration is q35 or i440fx.
>After building a few kernels it is the following commit causing the issue: 475a60aec7e if_vtnet: Misc Tx path cleanup
Did you just remove this commit, or rolled back the branch to the commit before 475a60aec7e?
I tried to reproduce this bug with bhyve - no success.
FreeBSD-13-RC1 <-> FreeBSD-13-RC1 - 10-20 Gbit/s.
FreeBSD-13-Rc1 <-> Ubuntu - 10-20 Gbit/s.
What is the output of sysctl dev.vtnet after you did a transfer showing the problem? I only removed that single 475a60aec7e commit. I haven't tried it with bhyve. Maybe the issue only shows up if the hypervisor runs on a Linux system? I got a copy of Ubuntu Desktop 20.04.2.0 (and updated) and setup FreeBSD 13 using virt-manager. Same thing happens with a copy of the latest Proxmox setting up FreeBSD 13. I'll run another test and get the sysctl dev.vtnet output. I'll get two outputs one with the regular RC2 kernel and the other with the 475a60aec7e commit removed. guess the same issue was reported on freebsd-net@ by Blake Hartshorn. The .pcap files provided (privately) indicated that all segments sent larger than the MTU were dropped by the host OS. So it is a TSO issue... I guess so too. That's why also ifconfig vtnet0 -tso6 brings back acceptable speeds. But without the TSO the vm-to-vm throughput on the same host is not anywhere near IPv4 or what it was on FreeBSD 12 for IPv6. (In reply to Michael Tratz from comment #9) I'm not arguing that there is a bug. I'm just trying to figure out what is wrong with the patch. Also trying to reproduce this locally... Created attachment 223398 [details]
sysctl dev.vtnet (slow)
No worries Michael! Note: vtnet1 is the interface I'm using for testing. So look at that output. The VM was freshly started for the test. The iperf3 was run for 30 seconds.
Created attachment 223399 [details]
Fast iperf3 without 475a60aec7e
Here is the same output running the kernel without the 475a60aec7e commit.
OK, I know what the problem is. I can reproduce it locally. Thanks for the information. Will post a patch soon. Created attachment 223401 [details]
Proposed fix
(In reply to Michael Tratz from comment #12) Can you try the Proposed fix? Thank you so much Michael for the quick fix. Everything is back to normal. IPv6 is speedy again. (In reply to Michael Tratz from comment #16) Thanks for reporting back! The path is under review: https://reviews.freebsd.org/D29331 A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d4697a6b56168876fc0ffec1a0bb1b24d25b198e commit d4697a6b56168876fc0ffec1a0bb1b24d25b198e Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2021-03-18 20:25:47 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2021-03-18 20:32:20 +0000 vtnet: fix TSO for TCP/IPv6 The decision whether a TCP packet is sent over IPv4 or IPv6 was based on ethertype, which works correctly. In D27926 the criteria was changed to checking if the CSUM_IP_TSO flag is set in the csum-flags and then considering it to be TCP/IPv4. However, the TCP stack sets the flag to CSUM_TSO for IPv4 and IPv6, where CSUM_TSO is defined as CSUM_IP_TSO|CSUM_IP6_TSO. Therefore TCP/IPv6 packets gets mis-classified as TCP/IPv4, which breaks TSO for TCP/IPv6. This patch bases the check again on the ethertype. This fix will be MFC instantly as discussed with re(gjb). MFC after: instantly PR: 254366 Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D29331 sys/dev/virtio/network/if_vtnet.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6064ea8172f078c54954bc6e8865625feb7979fe commit 6064ea8172f078c54954bc6e8865625feb7979fe Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2021-03-18 20:25:47 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2021-03-18 20:33:32 +0000 vtnet: fix TSO for TCP/IPv6 The decision whether a TCP packet is sent over IPv4 or IPv6 was based on ethertype, which works correctly. In D27926 the criteria was changed to checking if the CSUM_IP_TSO flag is set in the csum-flags and then considering it to be TCP/IPv4. However, the TCP stack sets the flag to CSUM_TSO for IPv4 and IPv6, where CSUM_TSO is defined as CSUM_IP_TSO|CSUM_IP6_TSO. Therefore TCP/IPv6 packets gets mis-classified as TCP/IPv4, which breaks TSO for TCP/IPv6. This patch bases the check again on the ethertype. This fix is instantly MFCed. Approved by: re(gjb) PR: 254366 Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D29331 (cherry picked from commit d4697a6b56168876fc0ffec1a0bb1b24d25b198e) sys/dev/virtio/network/if_vtnet.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) A commit in branch releng/13.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=11660fa28fd39a644cb7d30a4378cf4751b89f15 commit 11660fa28fd39a644cb7d30a4378cf4751b89f15 Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2021-03-18 20:25:47 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2021-03-18 20:35:49 +0000 vtnet: fix TSO for TCP/IPv6 The decision whether a TCP packet is sent over IPv4 or IPv6 was based on ethertype, which works correctly. In D27926 the criteria was changed to checking if the CSUM_IP_TSO flag is set in the csum-flags and then considering it to be TCP/IPv4. However, the TCP stack sets the flag to CSUM_TSO for IPv4 and IPv6, where CSUM_TSO is defined as CSUM_IP_TSO|CSUM_IP6_TSO. Therefore TCP/IPv6 packets gets mis-classified as TCP/IPv4, which breaks TSO for TCP/IPv6. This patch bases the check again on the ethertype. This fix is instantly MFCed. Approved by: re(gjb) PR: 254366 Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D29331 (cherry picked from commit d4697a6b56168876fc0ffec1a0bb1b24d25b198e) (cherry picked from commit 6064ea8172f078c54954bc6e8865625feb7979fe) sys/dev/virtio/network/if_vtnet.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) This will be in the upcoming RC3. |
