Hello, I am experiencing a severe network performance regression on KVM-based environments (like GCE and Proxmox) using vtnet (VirtIO) network interfaces. Symptoms: Network throughput (e.g., during an scp transfer) collapses to extremely low speeds (approx. 3-4 KB/s). The guest VM is not overloaded (CPU remains idle). Root Cause (Specific Commit): This regression was introduced by commit: a98b5f3ff34e7ff851f6127a64ebd88818c8d004 If I revert this single commit, network performance returns to normal, even with TSO enabled. Workaround (Confirms TSO issue): The issue is directly related to TCP Segmentation Offload (TSO). Disabling TSO manually on the interface also resolves the performance problem: ifconfig vtnet0 -tso Steps to Reproduce: Run a 14-STABLE kernel containing commit a98b5f3ff34e. Host the VM on a KVM hypervisor. Attempt to transfer a file from the VM using scp from another host. Observe throughput drop to near-zero.
(In reply to lg from comment #0) A couple of clarification questions: * For reproducing the issue you copy a file from the VM to some other host? * Is the other host another VM running in parallel to the FreeBSD VM? * Is the other host a physical host? * Are you running any sort of NAT/Firewall on the FreeBSD VM? * Could you provide the output of ifconfig vtnet0? * Could you provide the output of sysctl hw.vtnet * Could you provide the output of sysctl dev.vtnet.0 * Could you run tcpdump -i vtnet0 -w out.pcap and share the .pcap file (a couple of seconds should be enough).
Created attachment 265139 [details] iperf3 tests on 13,14,15 I am running kvm on an older Xeon with 10-Gigabit X540-AT2 NICs. Its not the fastest box, but here are some baselines attached from STABLE 13,14,15 guests running iperf3 192.168.122.1 is the hypervisor .13 = stable 13 .14 = stable 14 .15 = stable 15 stable 15 is the king out of the box at least. If I disable tso on RELENG_15 it performs much like the others. The 3 VMs are all on the same KVM box with no other VMs running at the time.
(In reply to mike from comment #2) Can you answer any of the questions I provided?
Created attachment 265143 [details] vtnet0 pcap during SCP test Hello Please find bellow the requested information * For reproducing the issue you copy a file from the VM to some other host? For the test you can copy a file from the VM to your PC. * Is the other host another VM running in parallel to the FreeBSD VM? Yes but their is plenty of resources available on the host. * Is the other host a physical host? Physical this one does not matter, I have the issue with a VM on the same host or using my PC as a client. * Are you running any sort of NAT/Firewall on the FreeBSD VM? Not in the FreeBSD VM and not on the path between the FreeBSD VM and the "client" * Could you provide the output of ifconfig vtnet0? # ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec07b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> ether 92:93:c8:6e:95:77 inet 12.0.251.19 netmask 0xffff0000 broadcast 12.0.255.255 inet6 fe80::9093:c8ff:fe6e:9577%vtnet0 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> * Could you provide the output of sysctl hw.vtnet # sysctl hw.vtnet hw.vtnet.lro_mbufq_depth: 0 hw.vtnet.lro_entry_count: 128 hw.vtnet.rx_process_limit: 1024 hw.vtnet.tso_maxlen: 65535 hw.vtnet.mq_max_pairs: 32 hw.vtnet.mq_disable: 0 hw.vtnet.lro_disable: 1 hw.vtnet.tso_disable: 0 hw.vtnet.fixup_needs_csum: 0 hw.vtnet.csum_disable: 0 * Could you provide the output of sysctl dev.vtnet.0 dev.vtnet.0.txq0.rescheduled: 0 dev.vtnet.0.txq0.tso: 0 dev.vtnet.0.txq0.csum: 0 dev.vtnet.0.txq0.omcasts: 2 dev.vtnet.0.txq0.obytes: 258 dev.vtnet.0.txq0.opackets: 3 dev.vtnet.0.rxq0.rescheduled: 0 dev.vtnet.0.rxq0.host_lro: 0 dev.vtnet.0.rxq0.csum_failed: 0 dev.vtnet.0.rxq0.csum: 0 dev.vtnet.0.rxq0.ierrors: 0 dev.vtnet.0.rxq0.iqdrops: 0 dev.vtnet.0.rxq0.ibytes: 82768 dev.vtnet.0.rxq0.ipackets: 948 dev.vtnet.0.tx_task_rescheduled: 0 dev.vtnet.0.tx_tso_offloaded: 0 dev.vtnet.0.tx_csum_offloaded: 0 dev.vtnet.0.tx_defrag_failed: 0 dev.vtnet.0.tx_defragged: 0 dev.vtnet.0.tx_tso_without_csum: 0 dev.vtnet.0.tx_tso_not_tcp: 0 dev.vtnet.0.tx_csum_proto_mismatch: 0 dev.vtnet.0.tx_csum_unknown_ethtype: 0 dev.vtnet.0.rx_task_rescheduled: 0 dev.vtnet.0.rx_csum_offloaded: 0 dev.vtnet.0.rx_csum_failed: 0 dev.vtnet.0.rx_csum_inaccessible_ipproto: 0 dev.vtnet.0.rx_csum_bad_offset: 0 dev.vtnet.0.rx_csum_bad_ipproto: 0 dev.vtnet.0.rx_csum_bad_ethtype: 0 dev.vtnet.0.rx_mergeable_failed: 0 dev.vtnet.0.rx_enq_replacement_failed: 0 dev.vtnet.0.rx_frame_too_large: 0 dev.vtnet.0.mbuf_alloc_failed: 0 dev.vtnet.0.act_vq_pairs: 1 dev.vtnet.0.req_vq_pairs: 1 dev.vtnet.0.max_vq_pairs: 1 dev.vtnet.0.%iommu: dev.vtnet.0.%parent: virtio_pci2 dev.vtnet.0.%pnpinfo: vendor=0x00001af4 device=0x1000 subvendor=0x1af4 device_type=0x00000001 dev.vtnet.0.%location: dev.vtnet.0.%driver: vtnet dev.vtnet.0.%desc: VirtIO Networking Adapter * Details of the step to generate the pcap : - I have create a 1G file "truncate -s 1G /tmp/1G.data" - start the pcap filtering on the IP of the vtnet0 - From my other "scp user@12.0.251.19:/tmp/1G.data /dev/null" - Few seconds later I stopped the scp and the pcap. In wireshark I can see "[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]" And the "dev.vtnet.0.tx_tso_without_csum" is increasing during the test.
> And the "dev.vtnet.0.tx_tso_without_csum" is increasing during the test. That is the crucial point. This error condition leads to packet drops. I also observed that TXCSUM is not enabled on vtnet0. Is that intentional? Do you have review D52765 in the tree? Just to double check: In the VM you don't use any additional network feature like jails with epairs, NAT or any sort of IP forwarding or any tunnel interface over vtnet?
(In reply to Michael Tuexen from comment #5) No nothing additional features used, no nat, no jail no ip forwarding, no tunnel The -txcsum was not intentional here but it is in my rc.conf, it some legacy default config I used. For some interface I had to disabled the txcsum and it stayed. And indeed when I re-enabled it on the vtnet it is working fine. Yes I have the review D52765 I rebuilt the my kernel from stable/14 the 28 October 14.3-STABLE FreeBSD 14.3-STABLE #442 stable/14-n272729-a93e1b731ae4-dirty: Tue Oct 28 01:25:44 UTC 2025 When I discovered the issues I check the latest modification and rebuild the kernel without some commit. First I revert review D52684 but I still had the issues, then I revert the review D52765 and I get back my normal behaviour. Sorry I just see that I made a mistake in the title. I wanted to put D52765 If you need I am available to do some more test
Created attachment 265151 [details] test results and sysctl output etc Yes, all in the attachment. I re-ran the tests again with all the questions in the attachments. Each guest (13,14,15) is running stable from today All tests are on a Ubuntu 22 box with 2 E5-2690 v4 @ 2.60GHz and 128G of RAM. No other guests on the KVM server. Tests are all between the hypervisor hosting the guests and the guests. Each VM has 16G of RAM and 12 CPUs. Chipset is Q35. I included output from an Ubuntu guest as reference. FreeBSD does better in one direction than Linux. If you want pcaps as well, let me know. Or if other tests are needed.
(In reply to lg from comment #6) I am a bit confused right now. Are you saying that everything is fine with a default setup? But if you disable TXCSUM in the FreeBSD VM, there is a problem. Is that the situation?
(In reply to Michael Tuexen from comment #8) Yes in my rc.conf originaly I had this ifconfig_vtnet0="up media autoselect mtu 1500 -txcsum" ifconfig_vtnet0_alias0="inet 12.0.251.19 netmask 255.255.0.0" With that setup I have the issue, but if I use change my config to: ifconfig_vtnet0="up" ifconfig_vtnet0_alias0="inet 12.0.251.19 netmask 255.255.0.0" It is working fine. So for me the issue happen only when TXCSUM is disabled and TSO4 enabled To resume if TXCSUM is enabled and TSO4 enabled => no issue TXCSUM is enabled and TSO4 disable => no issue TXCSUM is disabled and TSO4 enabled => not working ( cf the pcap ) TXCSUM is disabled and TSO4 disable => no issue
(In reply to lg from comment #9) Thank you very much for the clarification. I will look into this and improve the way configuration works. I will need to do some testing which I can only do next week, since I am traveling this week and don't have access to the system I use for testing this stuff. Thanks for reporting the issue and providing the information needed to nail this down. Once I have a patch, I will add it to this bug report such that you can test it.
It would be great if you could test review D290773.
(In reply to Michael Tuexen from comment #11) Correction: it would be great if you could test review D53629.
(In reply to Michael Tuexen from comment #12) Of course no problem
(In reply to Michael Tuexen from comment #12) I have tested this, and unfortunately, I still encounter the issue when txcsum is disabled. From what I understand of your modification (please correct me if I'm wrong), you intend to enable tso4 only if txcsum is enabled, correct? For the test using the same SCP as before, I started with the default configuration: ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> TXCSUM and TSO are both enabled, and it is working. Then I disable txcsum ifconfig vtnet0 -txcsum ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec07b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> TXCSUM is disabled but TSO4 is still enabled, and it is not working all packet are dropped Next I disable the tso4 ifconfig vtnet0 -tso4 ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec06b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> TXCSUM and TSO are both disabled, and it is working. Then I try to re-enabled tso4 ifconfig vtnet0 tso4 ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec06b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> As you can see, TSO4 remained disabled. I believe this is the expected behavior. Then enabling TXCSUM ifconfig vtnet0 txcsum ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec06bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> Here it is working fine And finally enabling TSO4 ifconfig vtnet0 tso4 ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> Now TSO4 is enabling correctly.
Thanks for testing. I change the patch the last minute and did not change the sequence of checks and actions. I am still traveling, so right now I can't test locally. Please retest, I updated the patch in review D53629.
(In reply to Michael Tuexen from comment #15) Thx ! This one is much better. Now when we disable the TXCSUM the TSO4 is also disabled automatically. # ifconfig vtnet0 -txcsum # ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec06b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> And everything is working fine :)
(In reply to lg from comment #16) Great, thanks a lot for testing.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4c50ac68166caf7e08c5a9984d63fa91490fa50d commit 4c50ac68166caf7e08c5a9984d63fa91490fa50d Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2025-11-10 15:34:53 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2025-11-10 15:34:53 +0000 vtnet: fix enabling/disabling tso Transmit segment offloading depends on transmit checksum offloading. Enforce that constraint. This also fixes a bug, since if_hwassist bits are from the CSUM_ space, not from the IFCAP_ space. PR: 290773 Reviewed by: Timo Völker Tested by: lg@efficientip.com MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D53629 sys/dev/virtio/network/if_vtnet.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=52df18d91b83038d870c595799a47d983d7da645 commit 52df18d91b83038d870c595799a47d983d7da645 Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2025-11-10 15:34:53 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2025-11-12 08:39:01 +0000 vtnet: fix enabling/disabling tso Transmit segment offloading depends on transmit checksum offloading. Enforce that constraint. This also fixes a bug, since if_hwassist bits are from the CSUM_ space, not from the IFCAP_ space. PR: 290773 Reviewed by: Timo Völker Tested by: lg@efficientip.com Differential Revision: https://reviews.freebsd.org/D53629 (cherry picked from commit 4c50ac68166caf7e08c5a9984d63fa91490fa50d) sys/dev/virtio/network/if_vtnet.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0fb0ba51d8bc1e6673e073c1c5a02922f997f6b8 commit 0fb0ba51d8bc1e6673e073c1c5a02922f997f6b8 Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2025-11-10 15:34:53 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2025-11-12 08:40:44 +0000 vtnet: fix enabling/disabling tso Transmit segment offloading depends on transmit checksum offloading. Enforce that constraint. This also fixes a bug, since if_hwassist bits are from the CSUM_ space, not from the IFCAP_ space. PR: 290773 Reviewed by: Timo Völker Tested by: lg@efficientip.com Differential Revision: https://reviews.freebsd.org/D53629 (cherry picked from commit 4c50ac68166caf7e08c5a9984d63fa91490fa50d) sys/dev/virtio/network/if_vtnet.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
A commit in branch releng/15.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c50312b774e1f26e90d743ad7e02072738923a03 commit c50312b774e1f26e90d743ad7e02072738923a03 Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2025-11-10 15:34:53 +0000 Commit: Colin Percival <cperciva@FreeBSD.org> CommitDate: 2025-11-12 18:46:09 +0000 vtnet: fix enabling/disabling tso Transmit segment offloading depends on transmit checksum offloading. Enforce that constraint. This also fixes a bug, since if_hwassist bits are from the CSUM_ space, not from the IFCAP_ space. Approved by: re (cperciva) PR: 290773 Reviewed by: Timo Völker Tested by: lg@efficientip.com Differential Revision: https://reviews.freebsd.org/D53629 (cherry picked from commit 4c50ac68166caf7e08c5a9984d63fa91490fa50d) (cherry picked from commit 0fb0ba51d8bc1e6673e073c1c5a02922f997f6b8) sys/dev/virtio/network/if_vtnet.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)