|Summary:||vmx driver stops sending network packets and resets connections (TCP) but allows ICMP|
|Product:||Base System||Reporter:||Shirkdog <mshirk>|
|Component:||kern||Assignee:||Vincenzo Maffione <vmaffione>|
|Severity:||Affects Many People||CC:||ameliamikie100, arrowd, dpetrov67, emaste, freebsd.bugs.459, info, kgc, lefty, markj, michele, mrpippy, ncrogers, oz42, peter.sopko+freebsd.org, pkelsey, sascha, shurd, silvio.wanka, sjddota065, vmaffione|
Description Shirkdog 2019-04-04 01:52:28 UTC
Tested on 13-CURRENT, with a FreeBSD VM running under ESXi (not sure the versions, I actually do not have console access at this point). I did an upgrade from 12-ALPHA4 something to 13-CURRENT and after this was completed, users were unable to download files over 1MB from this server. I verified that I could SSH into this system and navigate the simple website that was serving files, but whenever I tried to download a 40MB file, the transfer would stop, the connection would reset. I verified the same thing was happening with SFTP, navigate to the file, everything worked, but start the transfer, connection reset. I enabled ICMP to be able to ping this system continuously, and noticed nothing was dropping out, and verified nothing else was going on (that I am aware of) with any type of security device, packet shaping, etc. But with ICMP working fine while TCP connections were reset, I went to look at NIC offloading features, and all of them were turned on. Running the following to disable all of it with the vmx driver, and all of the problems went away, file transfers started to work: ifconfig vmx0 -rxcsum -txcsum -tso4 -tso6 -lro -rxcsum6 -txcsum6 -vlanhwcsum -vlanhwtso I tried flipping each one off and on trying to see it was one particular option, and it appears TSO4/TSO6 is where the problems resides vmx0@pci0:3:0:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00 vendor = 'VMware' device = 'VMXNET3 Ethernet Controller' class = network subclass = ethernet  Hypervisor: Origin = "VMwareVMware"  vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at device 0.0 on pci3  vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at device 0.0 on pci3  vmx0: Using 512 tx descriptors and 256 rx descriptors  vmx0: Using 2 rx queues 2 tx queues  vmx0: failed to allocate 3 MSI-X vectors, err: 6 - using MSI  vmx0: Using an MSI interrupt  vmx0: Ethernet address: 00:50:56:8f:25:15  vmx0: netmap queues/slots: TX 1/512, RX 1/512
Comment 1 Patrick Kelsey 2019-04-04 17:40:58 UTC
I tried to reproduce using the latest snapshot on ESXi 6.7, but so far I have been unable to. I was able to repeatedly scp a 100 MiB file from the machine under test without issue (~line rate at 1G each time). You said all NIC offloads were enabled. lro isn't enabled by default, which suggests there may be other NIC-related settings that are not at defaults. If so, please share them. Details from the system I used: Client version: 1.25.0 Client build number: 7872652 ESXi version: 6.7.0 ESXi build number: 8169922 FreeBSD vmx-debug 13.0-CURRENT FreeBSD 13.0-CURRENT r345863 GENERIC amd64 vmx0: link state changed to UP vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci4 vmx0: Using 512 tx descriptors and 256 rx descriptors vmx0: Using 6 rx queues 6 tx queues vmx0: failed to allocate 7 MSI-X vectors, err: 6 - using MSI vmx0: Using an MSI interrupt vmx0: Ethernet address: 00:53:00:00:00:00 vmx0: netmap queues/slots: TX 1/512, RX 1/512 vmx0: link state changed to UP vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:53:00:00:00:00 inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Comment 2 Shirkdog 2019-04-04 17:59:41 UTC
I will try to get the specifics. I was unable to access the console where this VM reside, but my guess is an older version of ESXi.
Comment 3 Shirkdog 2019-04-09 17:47:19 UTC
Information I have. Protected by VMware HA, managed by vCenter Server. Compatibility: ESXi 6.0 update 3 host (VM version 11)
Comment 4 Sascha Klauder 2019-11-22 12:46:59 UTC
I can confirm this on ESXi 6.0 and FreeBSD 12.1-RELEASE-p1. Network traffic capture shows a lot of TCP retransmissions and iperf3 measures a mere ~25 kbit/s bandwith performance. Turning off TSO restores expected performance.
Comment 5 Dmitry Petrov 2019-12-26 14:20:38 UTC
Same problem on multiple hosts with VMware 6.5 with FreeBSD 12-stable guests. It usually manifests itself after VMotion.
Comment 6 Brendan Shanks 2019-12-27 19:52:19 UTC
I also hit this problem with VMware ESXi 6.0u3 after upgrading from 12.0 to 12.1-RELEASE-p1. No vCenter or vMotion going on, just a standalone ESXi install. Disabling TSO fixed the problem for me ('ifconfig vmx0 -tso'). I can test any fixes if it'll help.
Comment 7 Patrick Kelsey 2019-12-28 04:52:38 UTC
Repeating here what I just said on -net (basically, I think there is at least this bug in the TSO path of the current vmxnet3 driver): I am not able to test this at the moment, nor likely in the very near future, but I did have a few minutes to do some code reading and now believe that the following is part of the problem, if not the entire problem. Using r353803 as a reference, I believe line 1323 in sys/dev/vmware/vmxnet3/if_vmx.c (in vmxnet3_isc_txd_encap()) should be: sop->hlen = hdrlen + ipi->ipi_tcp_hlen; instead of the current: sop->hlen = hdrlen; This can be seen by going back to r333813 and examining the CSUM_TSO case of vmxnet3_txq_offload_ctx(). The final increment of *start in that case is what was literally lost in translation when converting the driver to iflib.
Comment 8 commit-hook 2020-01-13 21:27:18 UTC
A commit references this bug: Author: vmaffione Date: Mon Jan 13 21:26:17 UTC 2020 New revision: 356703 URL: https://svnweb.freebsd.org/changeset/base/356703 Log: vmx: fix initialization of TSO related descriptor fields Fix a mistake introduced by r343291, which ported the vmx(4) driver to iflib. In case of TSO, the hlen field of the (first) tx descriptor must be initialized to the cumulative length of Ethernet, IP and TCP headers. The length of the TCP header was missing. PR: 236999 Reported by: pkelsey Reviewed by: avg MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D22967 Changes: head/sys/dev/vmware/vmxnet3/if_vmx.c
Comment 9 SW@FL 2020-01-14 14:41:16 UTC
We was also running in this problem last year with upgrade test (to 12.0) and because I don't know this bug report up to now, I have done some testing last week. First I have upgrade a VM to 11.3 last patch level. There are also the vmx driver are removed from open-vm-tools and included into the kernel. A simple test for the wrong behavior is e.g. connect via SSH to the VM and call find /usr/share/doc -name "*.html" | xargs cat Under 11.3-RELEASE-p5 all works well. In the next step VM was upgraded to 12.1-RELEASE-p1 and now the problem appears. The output of the command stops some time and then some minutes and at the end and SSH timeout appears. I have compared the ifconfig output of both FreeBSD releases: # diff -w vmx1.vmkernel-11.3 vmx1.vmkernel-12.1 2c2 < options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> --- > options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> 4d3 < hwaddr 00:50:56:80:94:72 6d4 < nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> 8a7 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> There are two additional capabilities shown under 12: JUMBO_MTU and VLAN_HWTSO. For the second exist an ifconfig parameter to disable it (-vlanhwtso) but this does not solve the problem. For the first I can't find any possibility to disable it and also no documentation (I know jumbo frames and what happens if one site uses this frames and the switches are not configured for such MTUs). An other posting in the internet discusses problems with a pfsense VM with passthru network card and the new kernel vmx driver. They must enable TSO and LRO to solve there performance problems. So I have tried to disable TSO and voila the problem disappears. But I wounder if this coding bug was found just yesterday why the problem does not appear in 11.3?
Comment 10 commit-hook 2020-01-20 22:16:07 UTC
A commit references this bug: Author: vmaffione Date: Mon Jan 20 22:15:33 UTC 2020 New revision: 356932 URL: https://svnweb.freebsd.org/changeset/base/356932 Log: MFC r356703 vmx: fix initialization of TSO related descriptor fields Fix a mistake introduced by r343291, which ported the vmx(4) driver to iflib. In case of TSO, the hlen field of the (first) tx descriptor must be initialized to the cumulative length of Ethernet, IP and TCP headers. The length of the TCP header was missing. PR: 236999 Reported by: pkelsey Reviewed by: avg Differential Revision: https://reviews.freebsd.org/D22967 Changes: _U stable/12/ stable/12/sys/dev/vmware/vmxnet3/if_vmx.c
Comment 11 Mark Peek 2020-02-08 15:54:06 UTC
*** Bug 243869 has been marked as a duplicate of this bug. ***
Comment 12 Kubilay Kocak 2020-02-21 04:40:16 UTC
^Triage: - Track earliest (reported) version affected - Track stable merge - Assign to committer resolving
Comment 13 kgc 2020-03-25 00:47:24 UTC
I confirm it's resolved the issue on my vmware cluster as well. It would be great if this could get into 12.1-RELEASE-p4. Anything I can do to help?
Comment 14 Eugene Grosbein 2020-04-19 17:20:40 UTC
*** Bug 242070 has been marked as a duplicate of this bug. ***
Comment 15 info 2020-05-19 16:29:27 UTC
Will this fix be released as a patch to 12.1-RELEASE, or will it come with 12.2-RELEASE? Thanks
Comment 16 Gleb Popov 2020-06-30 13:24:51 UTC
Just bumped into this on 2 production servers. What needs to be done to get it as a patch for 12.1?
Comment 17 Mark Johnston 2020-07-20 18:19:38 UTC
We submitted an erratum notice request to secteam@, so this should be fixed in the next round of patches for 12.1 (no ETA yet).
Comment 18 michele 2020-08-05 18:27:33 UTC
(In reply to Mark Johnston from comment #17) Looks like "the next round of patches for 12.1" has just happened. Including the vmx patch.
Comment 19 Mark Johnston 2020-08-05 18:30:33 UTC
(In reply to michele from comment #18) Indeed, thanks. I will resolve the bug then. There are some other iflib/vmx patches recently merged to stable/12 that may be candidates for future ENs.
Comment 20 dote 2021-01-15 16:38:30 UTC
MARKED AS SPAM
Comment 21 Apkarc 2021-03-07 15:06:12 UTC
MARKED AS SPAM