Bug 236999 - vmx driver stops sending network packets and resets connections (TCP) but allows ICMP
Summary: vmx driver stops sending network packets and resets connections (TCP) but all...
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: Vincenzo Maffione
URL:
Keywords: regression
: 243869 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-04-04 01:52 UTC by Shirkdog
Modified: 2020-03-25 00:47 UTC (History)
11 users (show)

See Also:
koobs: mfc-stable12+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Shirkdog 2019-04-04 01:52:28 UTC
Tested on 13-CURRENT, with a FreeBSD VM running under ESXi (not sure the versions, I actually do not have console access at this point).

I did an upgrade from 12-ALPHA4 something to 13-CURRENT and after this was completed, users were unable to download files over 1MB from this server. I verified that I could SSH into this system and navigate the simple website that was serving files, but whenever I tried to download a 40MB file, the transfer would stop, the connection would reset.

I verified the same thing was happening with SFTP, navigate to the file, everything worked, but start the transfer, connection reset.

I enabled ICMP to be able to ping this system continuously, and noticed nothing was dropping out, and verified nothing else was going on (that I am aware of) with any type of security device, packet shaping, etc.

But with ICMP working fine while TCP connections were reset, I went to look at NIC offloading features, and all of them were turned on.

Running the following to disable all of it with the vmx driver, and all of the problems went away, file transfers started to work:

ifconfig vmx0 -rxcsum -txcsum -tso4 -tso6 -lro -rxcsum6 -txcsum6 -vlanhwcsum -vlanhwtso

I tried flipping each one off and on trying to see it was one particular option, and it appears TSO4/TSO6 is where the problems resides



vmx0@pci0:3:0:0:        class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00
    vendor     = 'VMware'
    device     = 'VMXNET3 Ethernet Controller'
    class      = network
    subclass   = ethernet

[1] Hypervisor: Origin = "VMwareVMware"
[1] vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at device 0.0 on pci3

[1] vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at device 0.0 on pci3
[1] vmx0: Using 512 tx descriptors and 256 rx descriptors
[1] vmx0: Using 2 rx queues 2 tx queues
[1] vmx0: failed to allocate 3 MSI-X vectors, err: 6 - using MSI
[1] vmx0: Using an MSI interrupt
[1] vmx0: Ethernet address: 00:50:56:8f:25:15
[1] vmx0: netmap queues/slots: TX 1/512, RX 1/512
Comment 1 Patrick Kelsey freebsd_committer 2019-04-04 17:40:58 UTC
I tried to reproduce using the latest snapshot on ESXi 6.7, but so far I have been unable to.  I was able to repeatedly scp a 100 MiB file from the machine under test without issue (~line rate at 1G each time).

You said all NIC offloads were enabled.  lro isn't enabled by default, which suggests there may be other NIC-related settings that are not at defaults.  If so, please share them.

Details from the system I used:

Client version:      1.25.0
Client build number: 7872652
ESXi version:        6.7.0
ESXi build number:   8169922

FreeBSD vmx-debug 13.0-CURRENT FreeBSD 13.0-CURRENT r345863 GENERIC  amd64

vmx0: link state changed to UP
vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci4
vmx0: Using 512 tx descriptors and 256 rx descriptors
vmx0: Using 6 rx queues 6 tx queues
vmx0: failed to allocate 7 MSI-X vectors, err: 6 - using MSI
vmx0: Using an MSI interrupt
vmx0: Ethernet address: 00:53:00:00:00:00
vmx0: netmap queues/slots: TX 1/512, RX 1/512
vmx0: link state changed to UP


vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500      options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:53:00:00:00:00
        inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Comment 2 Shirkdog 2019-04-04 17:59:41 UTC
I will try to get the specifics. I was unable to access the console where this VM reside, but my guess is an older version of ESXi.
Comment 3 Shirkdog 2019-04-09 17:47:19 UTC
Information I have. 

Protected by VMware HA, managed by vCenter Server.
Compatibility: ESXi 6.0 update 3 host (VM version 11)
Comment 4 Sascha Klauder 2019-11-22 12:46:59 UTC
I can confirm this on ESXi 6.0 and FreeBSD 12.1-RELEASE-p1.  Network traffic capture shows a lot of TCP retransmissions and iperf3 measures a mere ~25 kbit/s bandwith performance.  Turning off TSO restores expected performance.
Comment 5 dp 2019-12-26 14:20:38 UTC
Same problem on multiple hosts with VMware 6.5 with FreeBSD 12-stable guests.

It usually manifests itself after VMotion.
Comment 6 Brendan Shanks 2019-12-27 19:52:19 UTC
I also hit this problem with VMware ESXi 6.0u3 after upgrading from 12.0 to 12.1-RELEASE-p1. No vCenter or vMotion going on, just a standalone ESXi install. Disabling TSO fixed the problem for me ('ifconfig vmx0 -tso'). I can test any fixes if it'll help.
Comment 7 Patrick Kelsey freebsd_committer 2019-12-28 04:52:38 UTC
Repeating here what I just said on -net (basically, I think there is at least this bug in the TSO path of the current vmxnet3 driver):

I am not able to test this at the moment, nor likely in the very near future, but I did have a few minutes to do some code reading and now believe that the following is part of the problem, if not the entire problem.  Using r353803 as a reference, I believe line 1323 in sys/dev/vmware/vmxnet3/if_vmx.c (in vmxnet3_isc_txd_encap()) should be:

sop->hlen = hdrlen + ipi->ipi_tcp_hlen;

instead of the current:

sop->hlen = hdrlen;

This can be seen by going back to r333813 and examining the CSUM_TSO case of vmxnet3_txq_offload_ctx().  The final increment of *start in that case is what was literally lost in translation when converting the driver to iflib.
Comment 8 commit-hook freebsd_committer 2020-01-13 21:27:18 UTC
A commit references this bug:

Author: vmaffione
Date: Mon Jan 13 21:26:17 UTC 2020
New revision: 356703
URL: https://svnweb.freebsd.org/changeset/base/356703

Log:
  vmx: fix initialization of TSO related descriptor fields

  Fix a mistake introduced by r343291, which ported the vmx(4)
  driver to iflib.
  In case of TSO, the hlen field of the (first) tx descriptor must
  be initialized to the cumulative length of Ethernet, IP and TCP
  headers. The length of the TCP header was missing.

  PR:		236999
  Reported by:	pkelsey
  Reviewed by:	avg
  MFC after:	1 week
  Differential Revision:	https://reviews.freebsd.org/D22967

Changes:
  head/sys/dev/vmware/vmxnet3/if_vmx.c
Comment 9 SW@FL 2020-01-14 14:41:16 UTC
We was also running in this problem last year with upgrade test (to 12.0) and because I don't know this bug report up to now, I have done some testing last week.

First I have upgrade a VM to 11.3 last patch level. There are also the vmx driver are removed from open-vm-tools and included into the kernel. A simple test for the wrong behavior is e.g. connect via SSH to the VM and call

  find /usr/share/doc -name "*.html" | xargs cat

Under 11.3-RELEASE-p5 all works well. In the next step VM was upgraded to 12.1-RELEASE-p1 and now the problem appears. The output of the command stops some time and then some minutes and at the end and SSH timeout appears.

I have compared the ifconfig output of both FreeBSD releases:

 # diff -w vmx1.vmkernel-11.3 vmx1.vmkernel-12.1
2c2
<       options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
---
>       options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
4d3
<       hwaddr 00:50:56:80:94:72
6d4
<       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
8a7
>       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

There are two additional capabilities shown under 12: JUMBO_MTU and VLAN_HWTSO. For the second exist an ifconfig parameter to disable it (-vlanhwtso) but this does not solve the problem. For the first I can't find any possibility to disable it and also no documentation (I know jumbo frames and what happens if one site uses this frames and the switches are not configured for such MTUs). An other posting in the internet discusses problems with a pfsense VM with passthru network card and the new kernel vmx driver. They must enable TSO and LRO to solve there performance problems. So I have tried to disable TSO and voila the problem disappears.
But I wounder if this coding bug was found just yesterday why the problem does not appear in 11.3?
Comment 10 commit-hook freebsd_committer 2020-01-20 22:16:07 UTC
A commit references this bug:

Author: vmaffione
Date: Mon Jan 20 22:15:33 UTC 2020
New revision: 356932
URL: https://svnweb.freebsd.org/changeset/base/356932

Log:
  MFC r356703

  vmx: fix initialization of TSO related descriptor fields

  Fix a mistake introduced by r343291, which ported the vmx(4)
  driver to iflib.
  In case of TSO, the hlen field of the (first) tx descriptor must
  be initialized to the cumulative length of Ethernet, IP and TCP
  headers. The length of the TCP header was missing.

  PR:             236999
  Reported by:    pkelsey
  Reviewed by:    avg
  Differential Revision:  https://reviews.freebsd.org/D22967

Changes:
_U  stable/12/
  stable/12/sys/dev/vmware/vmxnet3/if_vmx.c
Comment 11 Mark Peek freebsd_committer 2020-02-08 15:54:06 UTC
*** Bug 243869 has been marked as a duplicate of this bug. ***
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2020-02-21 04:40:16 UTC
^Triage: 

- Track earliest (reported) version affected
- Track stable merge
- Assign to committer resolving
Comment 13 kgc 2020-03-25 00:47:24 UTC
I confirm it's resolved the issue on my vmware cluster as well.  It would be great if this could get into 12.1-RELEASE-p4. Anything I can do to help?