Bug 236999
| Summary: | vmx driver stops sending network packets and resets connections (TCP) but allows ICMP | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Shirkdog <mshirk> |
| Component: | kern | Assignee: | Vincenzo Maffione <vmaffione> |
| Status: | Closed FIXED | ||
| Severity: | Affects Many People | CC: | ameliamikie100, arrowd, dpetrov67, emaste, freebsd.bugs.459, info, kgc, lefty, markj, michele, mrpippy, ncrogers, oz42, peter.sopko+freebsd.org, pkelsey, sascha, shurd, silvio.wanka, sjddota065, vmaffione |
| Priority: | --- | Keywords: | regression |
| Version: | 12.0-RELEASE | Flags: | koobs:
mfc-stable12+
|
| Hardware: | amd64 | ||
| OS: | Any | ||
| See Also: |
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243392 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242070 |
||
I tried to reproduce using the latest snapshot on ESXi 6.7, but so far I have been unable to. I was able to repeatedly scp a 100 MiB file from the machine under test without issue (~line rate at 1G each time).
You said all NIC offloads were enabled. lro isn't enabled by default, which suggests there may be other NIC-related settings that are not at defaults. If so, please share them.
Details from the system I used:
Client version: 1.25.0
Client build number: 7872652
ESXi version: 6.7.0
ESXi build number: 8169922
FreeBSD vmx-debug 13.0-CURRENT FreeBSD 13.0-CURRENT r345863 GENERIC amd64
vmx0: link state changed to UP
vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci4
vmx0: Using 512 tx descriptors and 256 rx descriptors
vmx0: Using 6 rx queues 6 tx queues
vmx0: failed to allocate 7 MSI-X vectors, err: 6 - using MSI
vmx0: Using an MSI interrupt
vmx0: Ethernet address: 00:53:00:00:00:00
vmx0: netmap queues/slots: TX 1/512, RX 1/512
vmx0: link state changed to UP
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:53:00:00:00:00
inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
I will try to get the specifics. I was unable to access the console where this VM reside, but my guess is an older version of ESXi. Information I have. Protected by VMware HA, managed by vCenter Server. Compatibility: ESXi 6.0 update 3 host (VM version 11) I can confirm this on ESXi 6.0 and FreeBSD 12.1-RELEASE-p1. Network traffic capture shows a lot of TCP retransmissions and iperf3 measures a mere ~25 kbit/s bandwith performance. Turning off TSO restores expected performance. Same problem on multiple hosts with VMware 6.5 with FreeBSD 12-stable guests. It usually manifests itself after VMotion. I also hit this problem with VMware ESXi 6.0u3 after upgrading from 12.0 to 12.1-RELEASE-p1. No vCenter or vMotion going on, just a standalone ESXi install. Disabling TSO fixed the problem for me ('ifconfig vmx0 -tso'). I can test any fixes if it'll help.
Repeating here what I just said on -net (basically, I think there is at least this bug in the TSO path of the current vmxnet3 driver): I am not able to test this at the moment, nor likely in the very near future, but I did have a few minutes to do some code reading and now believe that the following is part of the problem, if not the entire problem. Using r353803 as a reference, I believe line 1323 in sys/dev/vmware/vmxnet3/if_vmx.c (in vmxnet3_isc_txd_encap()) should be: sop->hlen = hdrlen + ipi->ipi_tcp_hlen; instead of the current: sop->hlen = hdrlen; This can be seen by going back to r333813 and examining the CSUM_TSO case of vmxnet3_txq_offload_ctx(). The final increment of *start in that case is what was literally lost in translation when converting the driver to iflib. A commit references this bug: Author: vmaffione Date: Mon Jan 13 21:26:17 UTC 2020 New revision: 356703 URL: https://svnweb.freebsd.org/changeset/base/356703 Log: vmx: fix initialization of TSO related descriptor fields Fix a mistake introduced by r343291, which ported the vmx(4) driver to iflib. In case of TSO, the hlen field of the (first) tx descriptor must be initialized to the cumulative length of Ethernet, IP and TCP headers. The length of the TCP header was missing. PR: 236999 Reported by: pkelsey Reviewed by: avg MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D22967 Changes: head/sys/dev/vmware/vmxnet3/if_vmx.c We was also running in this problem last year with upgrade test (to 12.0) and because I don't know this bug report up to now, I have done some testing last week. First I have upgrade a VM to 11.3 last patch level. There are also the vmx driver are removed from open-vm-tools and included into the kernel. A simple test for the wrong behavior is e.g. connect via SSH to the VM and call find /usr/share/doc -name "*.html" | xargs cat Under 11.3-RELEASE-p5 all works well. In the next step VM was upgraded to 12.1-RELEASE-p1 and now the problem appears. The output of the command stops some time and then some minutes and at the end and SSH timeout appears. I have compared the ifconfig output of both FreeBSD releases: # diff -w vmx1.vmkernel-11.3 vmx1.vmkernel-12.1 2c2 < options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> --- > options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> 4d3 < hwaddr 00:50:56:80:94:72 6d4 < nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> 8a7 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> There are two additional capabilities shown under 12: JUMBO_MTU and VLAN_HWTSO. For the second exist an ifconfig parameter to disable it (-vlanhwtso) but this does not solve the problem. For the first I can't find any possibility to disable it and also no documentation (I know jumbo frames and what happens if one site uses this frames and the switches are not configured for such MTUs). An other posting in the internet discusses problems with a pfsense VM with passthru network card and the new kernel vmx driver. They must enable TSO and LRO to solve there performance problems. So I have tried to disable TSO and voila the problem disappears. But I wounder if this coding bug was found just yesterday why the problem does not appear in 11.3? A commit references this bug: Author: vmaffione Date: Mon Jan 20 22:15:33 UTC 2020 New revision: 356932 URL: https://svnweb.freebsd.org/changeset/base/356932 Log: MFC r356703 vmx: fix initialization of TSO related descriptor fields Fix a mistake introduced by r343291, which ported the vmx(4) driver to iflib. In case of TSO, the hlen field of the (first) tx descriptor must be initialized to the cumulative length of Ethernet, IP and TCP headers. The length of the TCP header was missing. PR: 236999 Reported by: pkelsey Reviewed by: avg Differential Revision: https://reviews.freebsd.org/D22967 Changes: _U stable/12/ stable/12/sys/dev/vmware/vmxnet3/if_vmx.c *** Bug 243869 has been marked as a duplicate of this bug. *** ^Triage: - Track earliest (reported) version affected - Track stable merge - Assign to committer resolving I confirm it's resolved the issue on my vmware cluster as well. It would be great if this could get into 12.1-RELEASE-p4. Anything I can do to help? *** Bug 242070 has been marked as a duplicate of this bug. *** Will this fix be released as a patch to 12.1-RELEASE, or will it come with 12.2-RELEASE? Thanks Just bumped into this on 2 production servers. What needs to be done to get it as a patch for 12.1? We submitted an erratum notice request to secteam@, so this should be fixed in the next round of patches for 12.1 (no ETA yet). (In reply to Mark Johnston from comment #17) Looks like "the next round of patches for 12.1" has just happened. Including the vmx patch. (In reply to michele from comment #18) Indeed, thanks. I will resolve the bug then. There are some other iflib/vmx patches recently merged to stable/12 that may be candidates for future ENs. MARKED AS SPAM MARKED AS SPAM |
Tested on 13-CURRENT, with a FreeBSD VM running under ESXi (not sure the versions, I actually do not have console access at this point). I did an upgrade from 12-ALPHA4 something to 13-CURRENT and after this was completed, users were unable to download files over 1MB from this server. I verified that I could SSH into this system and navigate the simple website that was serving files, but whenever I tried to download a 40MB file, the transfer would stop, the connection would reset. I verified the same thing was happening with SFTP, navigate to the file, everything worked, but start the transfer, connection reset. I enabled ICMP to be able to ping this system continuously, and noticed nothing was dropping out, and verified nothing else was going on (that I am aware of) with any type of security device, packet shaping, etc. But with ICMP working fine while TCP connections were reset, I went to look at NIC offloading features, and all of them were turned on. Running the following to disable all of it with the vmx driver, and all of the problems went away, file transfers started to work: ifconfig vmx0 -rxcsum -txcsum -tso4 -tso6 -lro -rxcsum6 -txcsum6 -vlanhwcsum -vlanhwtso I tried flipping each one off and on trying to see it was one particular option, and it appears TSO4/TSO6 is where the problems resides vmx0@pci0:3:0:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00 vendor = 'VMware' device = 'VMXNET3 Ethernet Controller' class = network subclass = ethernet [1] Hypervisor: Origin = "VMwareVMware" [1] vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at device 0.0 on pci3 [1] vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at device 0.0 on pci3 [1] vmx0: Using 512 tx descriptors and 256 rx descriptors [1] vmx0: Using 2 rx queues 2 tx queues [1] vmx0: failed to allocate 3 MSI-X vectors, err: 6 - using MSI [1] vmx0: Using an MSI interrupt [1] vmx0: Ethernet address: 00:50:56:8f:25:15 [1] vmx0: netmap queues/slots: TX 1/512, RX 1/512