I've got two free virtual machines in Oracle Cloud to run a VPN service there. I've noticed that the network speed was ok with direct connection, but was extreamly slow, when traffic crosses vtnet and VPN interfaces (tried both IPsec and OpenVPN). Adding hw.vtnet.csum_disable=1 fixed the problem. Seems like it is for a while in the wild, and still not fixed. https://serverfault.com/questions/817664/slow-network-fixed-adding-hw-vtnet-csum-disable-1-what-are-the-posible-side-effe
Currently freebsd-version shows 13.2-RELEASE-p9
Also the driver provides some hw.vtnet.fixup_needs_csum that is not documented in man page.
I think I have related experience with csum vtnet bug. On one VM (in KVM) with 2 vtnet NICs IPsec VPN the packets are not going thru until I disabled csum on "LAN" NIC vtnet1: ifconfig_vtnet0="inet AA.BB.CC.DD netmask 255.255.255.128" ifconfig_vtnet1="inet WW.XX.YY.ZZ/24 -rxcsum -txcsum" This machine works like an IPsec gateway, vtnet0 is facing the internet, vtnet1 connects private network with clients connecting thru IPsec. Without -rxcsum -txcsum on vtnet1 the client machines cannot connect to anything in the outside world. First discovered with FreeBSD 12.3 (installed in the VM), later upgraded to 13.2.
(In reply to crypt47 from comment #0) > but was extreamly slow, when traffic crosses vtnet and VPN interfaces May you please elaborate your setup that how the traffic crosses vtnet and VPN interfaces ?
Created attachment 248169 [details] port 80 dump
(In reply to Zhenlei Huang from comment #4) It is very simple. We have one machine "A" with outgoing interface (vtnet0) and some virtual (like ipsec0 or tun0). We also have another VM "B" in the same network and also vtnet0 interface which has nginx serving FreeBSD ISO file.:) If we directly request the ISO file from "A" works alight. If we request the file via VPN interfaces, we get a lot of retransmissions because packets began to be discarded. I'm attaching the dump file, captured on vtnet0.
If you view the attached 80.dump file with Wireshark and check "validate the TCP checksum" option, it will complain about server (10.0.0.53) packets. Can't say how much it's relevant, because it's not the server with VPN setup. It may be some Wireshark miscalculation.
As for the ipfw configuration, I don't want to post it in full, but I use ipfw_f add 10 reass all from any to any in // ASSEMBLING in the beginning. Some nat translation later. Nothing special. Also mss fix is used for IPsec, but it can't be relevant. Breaks with OpenVPN too.
Could you retest stable/14, stable/15, releng/15.0, or current and report if the issue persists with changing any vtnet specific variable?
No, I don't have capacity to test the changes right now.
As far as I remember the issue was reported on the internet (https://serverfault.com/questions/817664/slow-network-fixed-adding-hw-vtnet-csum-disable-1-what-are-the-posible-side-effe), so it has to be pretty easy to make a test environment.
(In reply to crypt47 from comment #10) Thanks for letting me know.
(In reply to crypt47 from comment #11) I don't use GCE and find the description not clear enough to reproduce it...
You don't have to use Google Cloud for that. I didn't. It's a general issue with KVM virtual machine and vtnet driver. You just have to run OpenVPN on it.
Something is wrong with the encapsulation.
> You just have to run OpenVPN on it. Or IPSec for that matter. As I've reported.
by the way, similar problem with if_wg tunnel, on 14.3p4, workarounded with ifconfig vtnet0 -tso4 -tso6 -lro -rxcsum
(In reply to vova from comment #17) Could you test with stable/14?
Interesting results: 14.3p4 ifconfig_vtnet0="DHCP -tso4 -tso6 -lro -rxcsum" Download: 1938.88 Mbps Upload: 789.48 Mbps # speedtest-go speedtest-go v1.7.10 (git-dev) @showwin ✓ ISP: 128.140.36.15 (Hetzner Online) [53.5774, 10.0785] ✓ Found 20 Public Servers ✓ Test Server: [69030] 467.36km Nuremberg (Germany) by Advin Services LLC ✓ Latency: 974.539µs Jitter: 342.617µs Min: 659.889µs Max: 1.746697ms ✓ Packet Loss Analyzer: Running in background (<= 30 Secs) ✓ Download: 1938.88 Mbps (Used: 2298.54MB) (Latency: 2ms Jitter: 1ms Min: 0ms Max: 5ms) ✓ Upload: 789.48 Mbps (Used: 983.55MB) (Latency: 4ms Jitter: 5ms Min: 0ms Max: 20ms) ✓ Packet Loss: N/A in vnet container (under pf nat): Speedtest by Ookla Server: Advin Services LLC - Nuremberg (id: 69030) ISP: Hetzner Online Idle Latency: 0.40 ms (jitter: 0.06ms, low: 0.36ms, high: 0.49ms) Download: 1504.38 Mbps (data used: 2.4 GB) 5.13 ms (jitter: 15.33ms, low: 0.36ms, high: 226.18ms) Upload: 1425.73 Mbps (data used: 1.9 GB) 2.02 ms (jitter: 5.17ms, low: 0.33ms, high: 379.86ms) Packet Loss: 0.0% 14.3p4 ifconfig_vtnet0="DHCP" dramatically worse: Download: 719.23 Mbps Upload: 485.25 Mbps # ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> speedtest-go speedtest-go v1.7.10 (git-dev) @showwin ✓ ISP: 128.140.36.15 (Hetzner Online) [53.5774, 10.0785] ✓ Found 20 Public Servers ✓ Test Server: [60469] 467.36km Nuremberg (Germany) by netcup GmbH ✓ Latency: 1.107351ms Jitter: 172.438µs Min: 800.457µs Max: 1.378362ms ✓ Packet Loss Analyzer: Running in background (<= 30 Secs) ✓ Download: 719.23 Mbps (Used: 770.40MB) (Latency: 1ms Jitter: 0ms Min: 0ms Max: 2ms) ✓ Upload: 485.25 Mbps (Used: 945.52MB) (Latency: 1ms Jitter: 0ms Min: 0ms Max: 4ms) ✓ Packet Loss: 0.48% (Sent: 415/Dup: 0/Max: 416) in vnet container (under pf nat): Speedtest by Ookla Server: Advin Services LLC - Nuremberg (id: 69030) ISP: Hetzner Online Idle Latency: 0.67 ms (jitter: 0.77ms, low: 0.34ms, high: 1.50ms) Download: 166.12 Mbps (data used: 243.8 MB) 0.77 ms (jitter: 0.51ms, low: 0.37ms, high: 5.65ms) Upload: 1595.95 Mbps (data used: 2.1 GB) 7.03 ms (jitter: 6.94ms, low: 0.70ms, high: 256.62ms) Packet Loss: 0.0% 14.3-STABLE stable/14-087fc6ae0a8c ifconfig_vtnet0="DHCP" Still not that good as before, but in vnet container - event better Download: 972.67 Mbps Upload: 523.19 Mbps # speedtest-go speedtest-go v1.7.10 (git-dev) @showwin ✓ ISP: 128.140.36.15 (Hetzner Online) [53.5774, 10.0785] ✓ Found 20 Public Servers ✓ Test Server: [60469] 467.36km Nuremberg (Germany) by netcup GmbH ✓ Latency: 905.265µs Jitter: 54.375µs Min: 808.469µs Max: 983.836µs ✓ Packet Loss Analyzer: Running in background (<= 30 Secs) ✓ Download: 972.67 Mbps (Used: 1283.25MB) (Latency: 1ms Jitter: 0ms Min: 0ms Max: 3ms) ✓ Upload: 523.19 Mbps (Used: 661.66MB) (Latency: 5ms Jitter: 6ms Min: 0ms Max: 24ms) ✓ Packet Loss: 0.00% (Sent: 283/Dup: 0/Max: 282) in vnet container (under pf nat): Speedtest by Ookla Server: Advin Services LLC - Nuremberg (id: 69030) ISP: Hetzner Online Idle Latency: 0.40 ms (jitter: 0.10ms, low: 0.32ms, high: 0.48ms) Download: 1813.96 Mbps (data used: 1.2 GB) 10.73 ms (jitter: 14.93ms, low: 1.37ms, high: 97.34ms) Upload: 1624.39 Mbps (data used: 2.8 GB) 3.16 ms (jitter: 8.67ms, low: 0.37ms, high: 325.18ms) Packet Loss: 0.0% and after returning to no offload: Download: 2052.35 Mbps Upload: 772.36 Mbps # ifconfig vtnet0 -tso4 -tso6 -lro -rxcsum # ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=cc00ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6,HWSTATS> # speedtest-go speedtest-go v1.7.10 (git-dev) @showwin ✓ ISP: 128.140.36.15 (Hetzner Online) [53.5774, 10.0785] ✓ Found 20 Public Servers ✓ Test Server: [69030] 467.36km Nuremberg (Germany) by Advin Services LLC ✓ Latency: 662.361µs Jitter: 113.466µs Min: 527.393µs Max: 883.733µs ✓ Packet Loss Analyzer: Running in background (<= 30 Secs) ✓ Download: 2052.35 Mbps (Used: 2593.14MB) (Latency: 3ms Jitter: 2ms Min: 0ms Max: 8ms) ✓ Upload: 772.36 Mbps (Used: 992.49MB) (Latency: 5ms Jitter: 3ms Min: 0ms Max: 12ms) ✓ Packet Loss: N/A in vnet container (under pf nat): Speedtest by Ookla Server: Advin Services LLC - Nuremberg (id: 69030) ISP: Hetzner Online Idle Latency: 0.43 ms (jitter: 0.03ms, low: 0.37ms, high: 0.44ms) Download: 2085.06 Mbps (data used: 1.1 GB) 12.66 ms (jitter: 21.69ms, low: 3.21ms, high: 237.03ms) Upload: 1584.51 Mbps (data used: 1.9 GB) 3.69 ms (jitter: 11.17ms, low: 0.41ms, high: 224.69ms) All measurements may be not that percise - virtual machine is chipest possible.
(In reply to vova from comment #19) That is really interesting. If I summarize it correctly, we have: vtnet: 14.3 stable/14 send w/o offloading 789 772 send w/ offloading 485 523 recv w/o offloading 1938 2052 recv w/ offloading 719 972 vnet: 14.3 stable/14 send w/ offloading 1595 1624 send w/o offloading 1425 1584 recv w/ offloading 166 1813 recv w/o offloading 1504 1085 (stable/14 of October 9th; Speed is in Mbps) What is your setup? I would like to be able to reproduce this locally? Are you just running the speedtests in a VM using vtnet? Or are other VMs doing forwarding involved? What is your setup with the vnet? Which interfaces are used in that case. Thanks for testing!
(In reply to Michael Tuexen from comment #20) About my setup: It is CX11 VM on hetzner (the cheapest one: 1 vCPU 2Gb RAM, 20 GB disk) all tests are with IPv4 (IPv6 is set up) VM basically does nothing, there are two if_wg links (I did not spot any significant difference running iperf3 over wireguard links) there are few vnet jails connected to bridge there is one jail with bind918 keeping few domains (not busy et all) also, in another jail telegram proxy with not much traffic also also there are frr10 running few BGP sessions over links just a little routes: # netstat -rn -f inet | wc -l 25 net/firewall - with pf jl0 -> epair connected to the same bridge # cat /etc/pf.conf # Interfaces ext_if = "vtnet0" # External interface int_ifs = "{ wg0, wg1, jl0 }" # Internal interfaces ns3 = "..." # Internal NAT address nz = "..." # Internal NAT address mtp = "..." # Internal NAT address bricks = "..." # Internal networks table <sshguard> persist # Define a persistent table for blocked IPs set skip on lo # Skip loopback traffic set loginterface $ext_if # Enable logging on the external interface scrub in all # Normalize all incoming traffic nat on $ext_if inet from $bricks to any -> ($ext_if) # NAT for internal networks nat on $ext_if inet from ($ext_if) to any -> ($ext_if) # NAT for packets originating from $ext_if rdr pass on $ext_if proto { tcp, udp } from any to ($ext_if) port 53 -> $ns3 rdr pass on $ext_if proto tcp from any to ($ext_if) port 2202 -> $nz port 22 rdr pass on $ext_if proto tcp from any to ($ext_if) port 8443 -> 172.22.15.5 port 8443 pass out quick on vtnet0 from $mtp to any keep state pass out quick on $ext_if from any to any keep state pass in on $int_ifs keep state pass in quick on $ext_if inet6 proto ipv6-icmp keep state pass in quick on $ext_if inet6 proto tcp to ($ext_if) port 22 keep state block in quick log from <sshguard> # Block and log traffic from blocklist block in log on vtnet0 all # Block and log all incoming traffic by default pass in on $ext_if proto tcp to ($ext_if) port 22 keep state # Allow SSH (TCP/22) pass in on $ext_if proto { tcp, udp } to $ns3 port 53 keep state # Allow DNS traffic pass in on $ext_if proto tcp to ($ext_if) port 2202 keep state # Allow custom SSH port (TCP/2202) pass in on $ext_if proto tcp to ($ext_if) port 8443 keep state # Allow MTP pass in on $ext_if inet proto icmp icmp-type echoreq keep state # Allow ICMP Echo Request (ping) antispoof quick for $ext_if # cat /etc/sysctl.conf vfs.zfs.min_auto_ashift=12 kern.init_shutdown_timeout="900" net.inet.ip.fw.one_pass=0 net.route.multipath=0 net.inet.ip.fw.dyn_udp_lifetime=30 kern.ipc.maxsockbuf=16777216
(In reply to vova from comment #19) one more thing to add - my vnet tests (from jail) were from jail to public speedtest server, so it passed whole chain: vnet-jail <-> epair <-> bridge <-> epair <-> pf/nat <-> vtnet
(In reply to vova from comment #21) Let us focus on the vtnet setup (will look at the jail later). You say that you have a vtnet interface and two wg interfaces. How is this part of the table: vtnet: 14.3 stable/14 send w/o offloading 789 772 send w/ offloading 485 523 recv w/o offloading 1938 2052 recv w/ offloading 719 972 generated? * Is the traffic directly going over vtnet? * Or is the traffic going over wg, which then gets over vtnet? * Does the traffic go through NAT? *
(In reply to Michael Tuexen from comment #23) directly over vtnet0 - public IPv4 set on vtnet0, default route over vtnet, speedtest just runs on host
(In reply to vova from comment #24) OK. Just running the speedtest in the VM, which uses a vtnet interface. The most simplest setup, right? Just double checking. This will be easy to reproduce for me.
(In reply to Michael Tuexen from comment #25) yep, only thing I can assume that may be disturbing - pf firewall.
(In reply to vova from comment #26) Do you also observe the problem without pf? If not, could you provide a minimal pf config showing the problem?
(In reply to Michael Tuexen from comment #27) I did not spot any issues that can be clearly attributed to pf. (my pf config above if need for reference)
(In reply to vova from comment #28) So you are basically saying, the it doesn't matter if you use pf or not. That is great. Let us focus on the vtnet scenario (not the jail) and on the receive side. There is a substantial difference triggered by the offload. I did the same test locally and got less bandwidth than you, about 260Mbps. But it was consistent on multiple runs and enabling or disabling the offload did not show an impact. The system was not CPU limited. So could you measure that a couple of times and see if disabling offload roughly doubles the receive rate? Could you also prove the output of sysctl dev.vtnet.0 and sysctl hw.vtnet?
(In reply to Michael Tuexen from comment #29) hm ... good question, I've created a new VM and did series of tests: CPX22 (2 vCPU 4GB RAM, old CX11 is not available any more) IPv4, Nuremberg just install 14.3, upgrade to 14.3p4 (with freebsd-update), install speedtest-go package, copy stable/14-087fc6ae0a8c kernel build outside as alternative summarized results: FreeBSD Offloading Try Download Upload Mbps Mbps 14.3p4 yes 1 1,413 736 14.3p4 yes 2 1,348 833 14.3p4 yes 3 1,377 901 14.3p4 yes 4 1,357 737 14.3p4 yes 5 1,352 571 14.3p4 yes 6 1,247 776 14.3p4 yes avg 1,349 759 CV% 4.11% 14.71% 14.3p4 no 1 1,574 681 14.3p4 no 2 1,598 702 14.3p4 no 3 1,237 602 14.3p4 no 4 1,537 778 14.3p4 no 5 1,541 745 14.3p4 no 6 1,568 623 14.3p4 no avg 1,509 688 CV% 8.96% 9.90% 14-087fc6ae0a8c yes 1 1,210 675 14-087fc6ae0a8c yes 2 1,332 707 14-087fc6ae0a8c yes 3 1,684 703 14-087fc6ae0a8c yes 4 1,320 460 14-087fc6ae0a8c yes 5 1,333 694 14-087fc6ae0a8c yes 6 1,279 825 14-087fc6ae0a8c yes avg 1,360 677 CV% 12.18% 17.58% 14-087fc6ae0a8c no 1 1,466 807 14-087fc6ae0a8c no 2 1,213 751 14-087fc6ae0a8c no 3 1,527 678 14-087fc6ae0a8c no 4 1,825 1,037 14-087fc6ae0a8c no 5 1,185 1,064 14-087fc6ae0a8c no 6 1,230 1,086 14-087fc6ae0a8c no avg 1,408 904 CV% 17.71% 19.81% What is seen: 1. diviations/variations is quite high, so, clearely does not worth to trust data after one measurement (my bad) 2. still incomparable with yours 260 Mbps (both up and down) - we masure something different? 3. turning off TSOs add ~10-12% of download consistently, but for upload -> sometimes improves, sometimes not (ifconfig vtnet0 -tso4 -tso6 -lro -rxcsum) 4. and these results still did not match what I have in my production VM (will re-test with multiple measurements) # sysctl dev.vtnet.0 dev.vtnet.0.txq0.rescheduled: 0 dev.vtnet.0.txq0.tso: 1388393 dev.vtnet.0.txq0.csum: 20810001 dev.vtnet.0.txq0.omcasts: 0 dev.vtnet.0.txq0.obytes: 18916415358 dev.vtnet.0.txq0.opackets: 20810007 dev.vtnet.0.rxq0.rescheduled: 0 dev.vtnet.0.rxq0.host_lro: 216036 dev.vtnet.0.rxq0.csum_failed: 0 dev.vtnet.0.rxq0.csum: 17119906 dev.vtnet.0.rxq0.ierrors: 0 dev.vtnet.0.rxq0.iqdrops: 0 dev.vtnet.0.rxq0.ibytes: 36642213337 dev.vtnet.0.rxq0.ipackets: 24902539 dev.vtnet.0.tx_task_rescheduled: 0 dev.vtnet.0.tx_tso_offloaded: 0 dev.vtnet.0.tx_csum_offloaded: 0 dev.vtnet.0.tx_defrag_failed: 0 dev.vtnet.0.tx_defragged: 0 dev.vtnet.0.tx_tso_without_csum: 0 dev.vtnet.0.tx_tso_not_tcp: 0 dev.vtnet.0.tx_csum_proto_mismatch: 0 dev.vtnet.0.tx_csum_unknown_ethtype: 0 dev.vtnet.0.rx_task_rescheduled: 0 dev.vtnet.0.rx_csum_offloaded: 0 dev.vtnet.0.rx_csum_failed: 0 dev.vtnet.0.rx_csum_bad_proto: 0 dev.vtnet.0.rx_csum_bad_offset: 0 dev.vtnet.0.rx_csum_bad_ipproto: 0 dev.vtnet.0.rx_csum_bad_ethtype: 0 dev.vtnet.0.rx_mergeable_failed: 0 dev.vtnet.0.rx_enq_replacement_failed: 0 dev.vtnet.0.rx_frame_too_large: 0 dev.vtnet.0.mbuf_alloc_failed: 0 dev.vtnet.0.act_vq_pairs: 1 dev.vtnet.0.req_vq_pairs: 1 dev.vtnet.0.max_vq_pairs: 1 dev.vtnet.0.%iommu: dev.vtnet.0.%parent: virtio_pci1 dev.vtnet.0.%pnpinfo: vendor=0x00001af4 device=0x1041 subvendor=0x1af4 device_type=0x00000001 dev.vtnet.0.%location: dev.vtnet.0.%driver: vtnet dev.vtnet.0.%desc: VirtIO Networking Adapter # sysctl hw.vtnet hw.vtnet.lro_mbufq_depth: 0 hw.vtnet.lro_entry_count: 128 hw.vtnet.rx_process_limit: 1024 hw.vtnet.tso_maxlen: 65535 hw.vtnet.mq_max_pairs: 32 hw.vtnet.mq_disable: 0 hw.vtnet.lro_disable: 1 hw.vtnet.tso_disable: 0 hw.vtnet.fixup_needs_csum: 0 hw.vtnet.csum_disable: 0
(In reply to vova from comment #30) A dumb question, but are these driver fixes actually in 14.3p# ? I thought they were only in RELENG_14 ?
(In reply to mike from comment #31) > A dumb question, but are these driver fixes actually in 14.3p# ? I thought they were > only in RELENG_14 ? Not a dumb question, they are indeed in main and stable/14, and not in 14.3. The first releases to include them will be 14.4 and 15.0.
(In reply to Ed Maste from comment #32) For the report here, note comment #30: > copy stable/14-087fc6ae0a8c kernel build outside
(In reply to mike from comment #31) You are right, the fixes are only in stable/14, stable/15, releng/15.0, and main. So comparing 14.3 and stable/14 is a check for "with the fixes" and "without the fixes".
(In reply to vova from comment #30) Thanks for doing these measurements! Some comments: * Your system and my system are completely different. So I do not expect the same numbers. I was just comparing the numbers between turning offload on and off. For my system, this does not make much of a difference. And I am not expecting that, because my system is not CPU limited, and all the offload stuff is about reducing CPU cycles. So it behaves expected. * Is the sysctl output from 14.3 or stable/14? hw.vtnet.lro_disable == 1 is the default now in stable/14, which disables HW LRO (not software LRO). But dev.vtnet.0.rxq0.host_lro == 216036 indicates that 216036 HW LRO was performed. That seems strange. * The numbers seems to indicate that on stable/14 the receive rate drops by 30% when you disable tso4, tso6, lro, and rxcsum. That is suspicious. I would assume that tso6 is irrelevant assuming that you use IPv4. Can you double check that? Also try disabling lro but keep rxcsum. disabling rxcsum should automatically disable lro. And I would not expect tso to have an impact on receiving data.
(In reply to Michael Tuexen from comment #35) > Is the sysctl output from 14.3 or stable/14? stable/14 new measurements on original VM (cx11): (reboot for each re-configuration) summary: FreeBSD Offloading Download (CV%) delta % Upload (CV%) delta % 14.3p4 yes 879 (24.21%) 456 (24.21%) 14.3p4 no 902 (13.36%) +2.59% 472 (13.36%) +3.60% 14-087fc6ae0a8c yes 877 (18.19%) -2.73% 476 (18.19%) +0.95% 14-087fc6ae0a8c no 877 (24.66%) -0.02% 404 (24.66%) -15.13% In general, I would assess results as - these cheap VMs do not provide any stable results to do such kind of measures, spikes are quite significant (shared hosting?). Also, I do not see much correlation on top of FreeBSD version or checksum offloading. Also, somehow, download is ~ twice faster than upload, no clue why ... Raw data (far-non-average results marked): FreeBSD Offloading Try Download Upload 14.3p4 yes 1 987 484 14.3p4 yes 2 963 532 14.3p4 yes 3 969 491 14.3p4 yes 4 828 405 14.3p4 yes 5 898 339 14.3p4 yes 6 984 490 14.3p4 yes 7 973 517 14.3p4 yes 8 922 499 14.3p4 yes 9 290 103 <<<<< 14.3p4 yes 10 974 695 14.3p4 yes avg 879 456 CV% 22.00% 33.20% 14.3p4 no 1 949 302 14.3p4 no 2 571 436 14.3p4 no 3 895 451 14.3p4 no 4 941 487 14.3p4 no 5 990 490 14.3p4 no 6 900 561 14.3p4 no 7 900 492 14.3p4 no 8 978 496 14.3p4 no 9 938 494 14.3p4 no 10 954 509 14.3p4 no avg 902 472 CV% 12.40% 13.40% 14-087fc6ae0a8c yes 1 936 538 14-087fc6ae0a8c yes 2 974 484 14-087fc6ae0a8c yes 3 752 441 14-087fc6ae0a8c yes 4 827 426 14-087fc6ae0a8c yes 5 965 493 14-087fc6ae0a8c yes 6 988 495 14-087fc6ae0a8c yes 7 981 524 14-087fc6ae0a8c yes 8 872 557 14-087fc6ae0a8c yes 9 991 500 14-087fc6ae0a8c yes 10 484 306 14-087fc6ae0a8c yes avg 877 476 CV% 18.19% 15.13% 14-087fc6ae0a8c no 1 942 485 14-087fc6ae0a8c no 2 956 489 14-087fc6ae0a8c no 3 970 517 14-087fc6ae0a8c no 4 868 493 14-087fc6ae0a8c no 5 965 467 14-087fc6ae0a8c no 6 988 528 14-087fc6ae0a8c no 7 277 98 <<<< 14-087fc6ae0a8c no 8 963 384 14-087fc6ae0a8c no 9 848 490 14-087fc6ae0a8c no 10 992 94 <<<< 14-087fc6ae0a8c no avg 877 404 CV% 24.66% 41.31% Command used to get statistics: for i in `seq 1 10`; do speedtest-go --server 60469 | awk '($2 == "Download:") { printf("%s\t", $3); } ($2 == "Upload:") { printf("%s\n", $3); }' done more info from sysctls on stable/14-087fc6ae0a8c: ifconfig_vtnet0="DHCP -tso4 -tso6 -lro -rxcsum" [root@mb0 /usr/home/vova]# sysctl dev.vtnet.0 dev.vtnet.0.txq0.rescheduled: 0 dev.vtnet.0.txq0.tso: 0 dev.vtnet.0.txq0.csum: 10705229 dev.vtnet.0.txq0.omcasts: 7 dev.vtnet.0.txq0.obytes: 7014701105 dev.vtnet.0.txq0.opackets: 10705766 dev.vtnet.0.rxq0.rescheduled: 0 dev.vtnet.0.rxq0.host_lro: 0 dev.vtnet.0.rxq0.csum_failed: 0 dev.vtnet.0.rxq0.csum: 2265936 dev.vtnet.0.rxq0.ierrors: 0 dev.vtnet.0.rxq0.iqdrops: 0 dev.vtnet.0.rxq0.ibytes: 14080902273 dev.vtnet.0.rxq0.ipackets: 11187201 dev.vtnet.0.tx_task_rescheduled: 0 dev.vtnet.0.tx_tso_offloaded: 0 dev.vtnet.0.tx_csum_offloaded: 10705229 dev.vtnet.0.tx_defrag_failed: 0 dev.vtnet.0.tx_defragged: 0 dev.vtnet.0.tx_tso_without_csum: 0 dev.vtnet.0.tx_tso_not_tcp: 0 dev.vtnet.0.tx_csum_proto_mismatch: 0 dev.vtnet.0.tx_csum_unknown_ethtype: 0 dev.vtnet.0.rx_task_rescheduled: 0 dev.vtnet.0.rx_csum_offloaded: 2265936 dev.vtnet.0.rx_csum_failed: 0 dev.vtnet.0.rx_csum_inaccessible_ipproto: 0 dev.vtnet.0.rx_csum_bad_offset: 0 dev.vtnet.0.rx_csum_bad_ipproto: 0 dev.vtnet.0.rx_csum_bad_ethtype: 0 dev.vtnet.0.rx_mergeable_failed: 0 dev.vtnet.0.rx_enq_replacement_failed: 0 dev.vtnet.0.rx_frame_too_large: 0 dev.vtnet.0.mbuf_alloc_failed: 0 dev.vtnet.0.act_vq_pairs: 1 dev.vtnet.0.req_vq_pairs: 1 dev.vtnet.0.max_vq_pairs: 1 dev.vtnet.0.%iommu: dev.vtnet.0.%parent: virtio_pci0 dev.vtnet.0.%pnpinfo: vendor=0x00001af4 device=0x1000 subvendor=0x1af4 device_type=0x00000001 dev.vtnet.0.%location: dev.vtnet.0.%driver: vtnet dev.vtnet.0.%desc: VirtIO Networking Adapter # sysctl hw.vtnet hw.vtnet.lro_mbufq_depth: 0 hw.vtnet.lro_entry_count: 128 hw.vtnet.rx_process_limit: 1024 hw.vtnet.tso_maxlen: 65535 hw.vtnet.mq_max_pairs: 32 hw.vtnet.mq_disable: 0 hw.vtnet.lro_disable: 1 hw.vtnet.tso_disable: 0 hw.vtnet.fixup_needs_csum: 0 stable/14-087fc6ae0a8c: ifconfig_vtnet0="DHCP" # ifconfig vtnet0 vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS> # sysctl dev.vtnet.0 dev.vtnet.0.txq0.rescheduled: 0 dev.vtnet.0.txq0.tso: 311575 dev.vtnet.0.txq0.csum: 11032984 dev.vtnet.0.txq0.omcasts: 7 dev.vtnet.0.txq0.obytes: 8466101606 dev.vtnet.0.txq0.opackets: 11033797 dev.vtnet.0.rxq0.rescheduled: 0 dev.vtnet.0.rxq0.host_lro: 0 dev.vtnet.0.rxq0.csum_failed: 0 dev.vtnet.0.rxq0.csum: 12244247 dev.vtnet.0.rxq0.ierrors: 0 dev.vtnet.0.rxq0.iqdrops: 0 dev.vtnet.0.rxq0.ibytes: 15515499836 dev.vtnet.0.rxq0.ipackets: 12244260 dev.vtnet.0.tx_task_rescheduled: 0 dev.vtnet.0.tx_tso_offloaded: 311575 dev.vtnet.0.tx_csum_offloaded: 11032984 dev.vtnet.0.tx_defrag_failed: 0 dev.vtnet.0.tx_defragged: 0 dev.vtnet.0.tx_tso_without_csum: 0 dev.vtnet.0.tx_tso_not_tcp: 0 dev.vtnet.0.tx_csum_proto_mismatch: 0 dev.vtnet.0.tx_csum_unknown_ethtype: 0 dev.vtnet.0.rx_task_rescheduled: 0 dev.vtnet.0.rx_csum_offloaded: 12244247 dev.vtnet.0.rx_csum_failed: 0 dev.vtnet.0.rx_csum_inaccessible_ipproto: 0 dev.vtnet.0.rx_csum_bad_offset: 0 dev.vtnet.0.rx_csum_bad_ipproto: 0 dev.vtnet.0.rx_csum_bad_ethtype: 0 dev.vtnet.0.rx_mergeable_failed: 0 dev.vtnet.0.rx_enq_replacement_failed: 0 dev.vtnet.0.rx_frame_too_large: 0 dev.vtnet.0.mbuf_alloc_failed: 0 dev.vtnet.0.act_vq_pairs: 1 dev.vtnet.0.req_vq_pairs: 1 dev.vtnet.0.max_vq_pairs: 1 dev.vtnet.0.%iommu: dev.vtnet.0.%parent: virtio_pci0 dev.vtnet.0.%pnpinfo: vendor=0x00001af4 device=0x1000 subvendor=0x1af4 device_type=0x00000001 dev.vtnet.0.%location: dev.vtnet.0.%driver: vtnet dev.vtnet.0.%desc: VirtIO Networking Adapter # sysctl hw.vtnet hw.vtnet.lro_mbufq_depth: 0 hw.vtnet.lro_entry_count: 128 hw.vtnet.rx_process_limit: 1024 hw.vtnet.tso_maxlen: 65535 hw.vtnet.mq_max_pairs: 32 hw.vtnet.mq_disable: 0 hw.vtnet.lro_disable: 1 hw.vtnet.tso_disable: 0 hw.vtnet.fixup_needs_csum: 0 hw.vtnet.csum_disable: 0
(In reply to vova from comment #36) First of all: Thank you very much for these measurements! This is very appreciated. Please note that upload and download bandwidth can be different, I would not assume that this is due to the interface. What I conclude from your measurements is that I do not see any indications anymore for checksum errors. I am not saying that the vtnet driver is perfect, just that I have no indication that the problem originally reported still persists. So I would be inclined to close this bug. What do you think? I you still have an issue needing to be investigated, I would suggest to open a new bug report and track it there. But the checksum issue seems to be resolved.
(In reply to Michael Tuexen from comment #37) No objections to close the bug from my side. Cannot detect any significant difference any more. The only notice here - I am not an original issue reporter.
(In reply to vova from comment #38) The original reporter indicated that testing is not possible due to time constraints. So I am closing this now.