Bug 259249 - vtnet: Checksum offloading enabled reduces pf NAT performance (~100mbps -> ~200kbps)
Summary: vtnet: Checksum offloading enabled reduces pf NAT performance (~100mbps -> ~2...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-net (Nobody)
URL:
Keywords: needs-qa, performance
Depends on:
Blocks:
 
Reported: 2021-10-18 16:33 UTC by emz
Modified: 2026-01-05 20:00 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description emz 2021-10-18 16:33:18 UTC
Router: FreeBSD 13
Network behind NAT: some Linux hosts

Router:
---------------------------------------
[root@balancer1:/etc]# ifconfig | grep -A1 vtnet | grep options        options=4c079b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>        options=4c079b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>


Host behind NAT, several curl launches:
---------------------------------------
[root@back1 server]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 9469M    0  527k    0     0   150k      0 17:55:26  0:00:03 17:55:23  150k^C
[root@back1 server]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 9469M    0  671k    0     0   188k      0 14:15:29  0:00:03 14:15:26  188k^C
[root@back1 server]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 9469M    0 1599k    0     0   198k      0 13:34:49  0:00:08 13:34:41  214k^C
[root@back1 server]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 9469M    0 1647k    0     0   225k      0 11:56:44  0:00:07 11:56:37  218k^C
[root@back1 server]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 9469M    0 1247k    0     0   134k      0 20:05:17  0:00:09 20:05:08  147k^C
[root@back1 server]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 9469M    0  383k    0     0   185k      0 14:31:44  0:00:02 14:31:42  185k^C

Router:
----------------------------------------------
[root@balancer1:/etc]# ifconfig vtnet0 -rxcsum
[root@balancer1:/etc]# ifconfig vtnet1 -rxcsum
[root@balancer1:/etc]# ifconfig vtnet1 -txcsum
[root@balancer1:/etc]# ifconfig vtnet0 -txcsum

Same host behind NAT:
----------------------------------------------
[root@back1 etc]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 9469M    3  309M    0     0  89.6M      0  0:01:45  0:00:03  0:01:42 89.6M^C
[root@back1 etc]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 9469M    0 76.9M    0     0  85.2M      0  0:01:51 --:--:--  0:01:51 85.2M^C
[root@back1 etc]# curl --output /dev/null https://mirror.yandex.ru/centos/8.4.2105/isos/x86_64/CentOS-8.4.2105-x86_64-dvd1.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  1 9469M    1 99.3M    0     0   123M      0  0:01:16 --:--:--  0:01:16  123M^C

Guess it's self-explanatory.
Comment 1 Aleksandr Fedorov freebsd_committer freebsd_triage 2021-10-21 08:23:31 UTC
Are you sure this is a regression? I do not know of a single version of FreeBSD that works without turning off [rx|tx]csum and LRO.
Comment 2 emz 2021-10-25 15:17:57 UTC
Well... not sure at this moment. May be not a regression, but ...is there a bug for this ?

BTW:
Workaround: it's enough to only turn off the rxcsum [and LRO ?]. With txcsum also turned off the ssh session became unresponsive (though receiving works just fine).
Comment 3 Michael Tuexen freebsd_committer freebsd_triage 2025-10-22 13:31:57 UTC
Can you retest with stable/14, stable/15, releng/15 or current?
Comment 4 vova 2025-10-22 14:55:59 UTC
As fast as as I have testing stand (see #276760), I've re-measured in NATed jail 

and here clear degradation of download for 14.3p4 with offloading if compare with 
14-stable or no offloading 

~26 Mbps vs ~800 Mbps normally


FreeBSD	Offloading	Download (CV%)	delta %		Upload (CV%)    delta %
14.3p4		yes	26 	19.96%			 445 	19.96%	
14.3p4		no	813 	23.93%	3079%		 528 	23.93%	19%
14-087fc6ae0a8c	yes	489 	58.95%	-40%		 431 	58.95%	-18%
14-087fc6ae0a8c	no	762 	33.78%	56%		 570 	33.78%	32%


raw data:
FreeBSD	Offloading	Try	Download	Upload
				
14.3p4		yes	1	 21 	 54 
14.3p4		yes	2	 18 	 769 
14.3p4		yes	3	 20 	 315 
14.3p4		yes	4	 28 	 540 
14.3p4		yes	5	 33 	 495 
14.3p4		yes	6	 27 	 356 
14.3p4		yes	7	 32 	 421 
14.3p4		yes	8	 22 	 325 
14.3p4		yes	9	 28 	 451 
14.3p4		yes	10	 27 	 728 
14.3p4		yes	avg	 26 	 445 
		CV%	19.96%	46.73%
				
14.3p4		no	1	 932 	 686 
14.3p4		no	2	 749 	 433 
14.3p4		no	3	 854 	 441 
14.3p4		no	4	 885 	 624 
14.3p4		no	5	 946 	 760 
14.3p4		no	6	 284 	 211 
14.3p4		no	7	 916 	 754 
14.3p4		no	8	 863 	 462 
14.3p4		no	9	 883 	 416 
14.3p4		no	10	 816 	 496 
14.3p4		no	avg	 813 	 528 
		CV%	23.93%	33.02%
				
14-087fc6ae0a8c	yes	1	 87 	 510 
14-087fc6ae0a8c	yes	2	 537 	 763 
14-087fc6ae0a8c	yes	3	 884 	 576 
14-087fc6ae0a8c	yes	4	 714 	 414 
14-087fc6ae0a8c	yes	5	 379 	 392 
14-087fc6ae0a8c	yes	6	 724 	 441 
14-087fc6ae0a8c	yes	7	 259 	 563 
14-087fc6ae0a8c	yes	8	 758 	 474 
14-087fc6ae0a8c	yes	9	 60 	 48 
14-087fc6ae0a8c	yes	10	 484 	 128 
14-087fc6ae0a8c	yes	avg	 489 	 431 
		CV%	58.95%	48.74%
				
14-087fc6ae0a8c	no	1	 964 	 541 
14-087fc6ae0a8c	no	2	 943 	 410 
14-087fc6ae0a8c	no	3	 980 	 478 
14-087fc6ae0a8c	no	4	 981 	 761 
14-087fc6ae0a8c	no	5	 861 	 757 
14-087fc6ae0a8c	no	6	 874 	 537 
14-087fc6ae0a8c	no	7	 765 	 471 
14-087fc6ae0a8c	no	8	 596 	 510 
14-087fc6ae0a8c	no	9	 327 	 511 
14-087fc6ae0a8c	no	10	 328 	 728 
14-087fc6ae0a8c	no	avg	 762 	 570 
		CV%	33.78%	22.59%
Comment 5 Michael Tuexen freebsd_committer freebsd_triage 2025-10-22 17:02:41 UTC
(In reply to vova from comment #4)
Could you guide me (possibly via private email) to configure your setup on one of my local machines? I would like to be able to do such measurements (upload/download) in a controlled environment to see what effects are related to vtnet and its interaction with nat on the one side and on changes in the network on the other side.
Comment 6 vova 2025-10-23 07:00:39 UTC
(In reply to Michael Tuexen from comment #5)

I've sent details in private email
Comment 7 vova 2025-10-23 07:49:07 UTC
(In reply to Michael Tuexen from comment #5)

VM Setup instructions roughly:

install FreeBSD 14.3 from ISO image, ZFS-based (amd64 machine - CX23 on hetzner)
freebsd-update fetch && freebsd-update install 

pkg update
pkg install speedtest-go cbsd

/usr/local/cbsd/sudoexec/initenv /usr/local/cbsd/share/initenv.conf default_vs=1 workdir=/jails 

# CBSD will create basic pf nat rules and bridge interface - cbsdbr0 with 10.0.0.1/24

cbsd jcreate jname=speedtest vnet=1 interface=cbsdbr0 sysrc=defaultrouter=10.0.0.1 astart=1
cbsd jstart speedtest
cbsd jlogin speedtest 

jail# pkg install speedtest-go
jail# speedtest-go --server 60469