Bug 243869

Summary:	ESXi v6.0 with VMNET 3 driver sluggish performance
Product:	Base System	Reporter:	Lefteris Tsintjelis <lefty>
Component:	kern	Assignee:	freebsd-net (Nobody) <net>
Status:	Closed DUPLICATE
Severity:	Affects Many People	CC:	avg, info, mp, pkelsey, sm
Priority:	---	Keywords:	performance, regression
Version:	12.1-RELEASE
Hardware:	amd64
OS:	Any

Description Lefteris Tsintjelis 2020-02-04 12:42:43 UTC

A bug/issue problem is introduced in FreeBSD 12.1 and vmxnet 3 network driver. After upgrading to FreeBSD 12.1 from 12.0 under ESXi v6.0 and using vmxnet 3, network performance becomes very sluggish. Problem is solved by either switching VM to use E1000 network driver or upgrading to a later ESXi version. Please fix the issue as ESXi upgrading is not always possible. Driver was working very well in 12.0. Thank you

Comment 1 Sergey Manucharian 2020-02-06 18:37:00 UTC

Same issue with a VPS at CloudAtCost.
Enabling TSO on vmx0 helps. No thorough tests yet, but at least "net/py-speedtest-cli" shows the same numbers as with FreeBSD 12.0.

Comment 2 Sergey Manucharian 2020-02-06 19:22:13 UTC

(In reply to Sergey Manucharian from comment #1)
Sorry, DISABLING TSO helps (I guess, I cannot edit the first comment).

Comment 3 Mark Peek freebsd_committer

2020-02-06 21:42:32 UTC

There have been various updates to ESXi 6.0. It would be good to know an exact build number to narrow down the exact ESXi build.

https://kb.vmware.com/s/article/2143832

Also, mentioning TSO and speedtest, it would be good to know more about the testing methodology used (between VM's, through a specific physical ethernet card, hardware features of that card) and the testing tools/methodology used.

Comment 4 info 2020-02-06 21:58:44 UTC

I experience the same thing on esxi 5.5 (ESXi550-201809001). I assumed it was the same issue as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242070

Comment 5 Lefteris Tsintjelis 2020-02-06 22:09:45 UTC

ESXi 6.0.0
Build 5050593 (HP Build) running on a HP ProLiant DL380 G7
Build 7967664 (Dell Build) running on Dells R610 and R710

All of the above VMs had the issue. -tso seems like the best workaround so far for me here too.

Testing actually is really simple. Just by using a simple "tail -n 1000 /var/log/messages" through a ssh connection is enough. Connections stop from responding all the time.

Comment 6 Mark Peek freebsd_committer

2020-02-06 22:25:43 UTC

Following the above link through to this bug:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236999

Do you have or have you tried applying this MFC fix?
https://svnweb.freebsd.org/base?view=revision&revision=356932

Comment 7 Mark Peek freebsd_committer

2020-02-07 22:07:21 UTC

I installed ESXi 6.0 build 7967664 and was able to reproduce the sluggish behavior. Applying the above MFC patch from bug 236999 resolved the issue for me.

Comment 8 Lefteris Tsintjelis 2020-02-08 15:35:03 UTC

This seems to resolve the issue here too in a couple of VMs so far

https://reviews.freebsd.org/D22967

Comment 9 Mark Peek freebsd_committer

2020-02-08 15:54:06 UTC


*** This bug has been marked as a duplicate of bug 236999 ***