Bug 198406 - More than 4 esxi vmxnet3 interfaces causes vlans attached to vmxnet interfaces to stop working
Summary: More than 4 esxi vmxnet3 interfaces causes vlans attached to vmxnet interface...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.1-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: FreeBSD bugs mailing list
Depends on:
Reported: 2015-03-08 00:29 UTC by Nick Hilliard
Modified: 2018-02-16 18:38 UTC (History)
8 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Nick Hilliard 2015-03-08 00:29:30 UTC
- vmware esxi 5.5.0 build 1892794
- freebsd 10.1-RELEASE amd64 VMs with vmxnet3 virtual ethernet interfaces
- vlans defined on vmx1 to vmx3

If a freebsd VM has 4 or fewer vmxnet3 interfaces, all vlans attached to any of the vmx interfaces will work fine, as expected.

If the freebsd VM has 5 more vmxnet3 interfaces, all vlans to all of the vmx interfaces will stop working.  tcpdump shows that no traffic is received, and that any traffic transmitted is not received by the destination host.  If the VM is shut down and the number of vmxnet3 interfaces is reduced to 4 or fewer, all the vlan interfaces start working again.

This is repeatable.
Comment 1 Nick Hilliard 2015-03-08 00:30:45 UTC
oops, typo.  that should read:

"If the freebsd VM has 5 or more vmxnet3 interfaces..."
Comment 2 timothyyl 2015-05-06 21:05:04 UTC
I believe I am seeing something very similar. In my case, everything works fine until I add a 5th interface, at which point nothing on that interface will work. The other interfaces continue to pass traffic, however. This essentially means that pfsense in VMware is useless if you plan on using one interface per port group and letting VMware do the tagging, since many places have greater than 5 VLANs in use. Any suggestions?
Comment 3 Nick Hilliard 2015-05-07 12:18:38 UTC
you can work around the problem by having 4 or fewer vmxnet3 interfaces, and then using vlan interfaces on freebsd.
Comment 4 timothyyl 2015-05-07 12:20:45 UTC
(In reply to nick from comment #3)

Thanks. I've switched to e1000 interfaces and the problem disappeared.
Comment 5 Nick Hilliard 2015-05-07 12:26:54 UTC
i wish you well with that :-)  if you're running anything later than esxi build 799733, the em driver will periodically hang with watchdog timeout kernel errors.  This was catastrophic in freebsd 8/9 but has improved in freebsd 10.1 to the point that it now only causes a small amount of packet loss.  ymmv.
Comment 6 timothyyl 2015-05-07 12:39:31 UTC
(In reply to nick from comment #5)

Awesome. Thanks for the heads up. Hopefully this bug will get some attention soon, as I would think that using port groups with FreeBSD/pfsense would be fairly common.
Comment 7 Andrew 2017-05-16 20:06:03 UTC
Just reproduced this bug on ESXi 6.5 with FreeBSD 11.0-RELEASE-p9 with four vmxnet3 interfaces 

Obviously problem goes from wrong FreeBSD vmx interface numbering in compare to VM ethernet interface order.

In my case mapping was follows:
ethernet0 -> vmx1
ethernet1 -> vmx2
ethernet2 -> vmx3
ethernet3 -> vmx0
I think something goes wrong on PCI probing/mapping stage. 
I've tried (not all options) to change pci bus numbers for ethernet interfaces in vmx config but without success.
Comment 8 Bryan Venteicher freebsd_committer 2017-06-24 17:34:31 UTC
There is no limit in the vmx(4) driver; it sounds like interrupts are broken when the fifth interface is added.

If the driver negotiates multiqueue (only if MSIX is available) perhaps we are exhausting some MSIX limit but an error is not bubbled up. You could try setting the `hw.vmx.mq_disable` tunable to disable multiqueue.

Unfortunately, I don't have access to an VMware ESXi environment to do any investigation.
Comment 9 Shawn 2017-09-29 06:58:47 UTC
for those of you who are still running into this problem:

The issue seems to be on the VMWare side with the ethernet pciSlotNumber (which is generated only on booting the VM with new vmxnet3 interfaces)

The interfaces, as they are presented to the VM, are out of order. 
You will see this since the MAC addresses do not correspond between VMWare and the VM's ifconfig.

the pci slot mapping is explained here


we worked around this problem my editing the VM.vmx file and re-arranging the slotnumber values

Lab environment: ESXi 5.5.0 2068190, FreeBSD 10.1, FreeBSD 10.2 (sorry haven't tested FreeBSD 11 as yet)

After adding all your interfaces to the VM (>4), boot up the VM then shut it down.

SSH to the VMWare Host (you could theoretically download the vmx file from the datastore, edit it and upload it again but we didn't do it that way)

cd /vmfs/volumes/DATASTORE/VM

vi into VM.vmx and remove all lines with ethernetX.pciSlotNumber

at the end of the file input this mapping (for as many interfaces as you need)

ethernet0.pciSlotNumber = "160"
ethernet1.pciSlotNumber = "1184"
ethernet2.pciSlotNumber = "192"
ethernet3.pciSlotNumber = "1216"
ethernet4.pciSlotNumber = "224"
ethernet5.pciSlotNumber = "1248"
ethernet6.pciSlotNumber = "256"

save file and exit

boot up your VM.

All interfaces should match up with the correct MAC addresses now. You can confirm with ifconfig and correlating it with the VM's settings in VMWare

If you are adding more than 8 interfaces the mapping should look like this

PciSlot# --------------------Interfaces ----------Hex values for Pcislot#
160 1184 2208	===>	vmx0	vmx1	vmx2	===>	A0	4A0	8A0
192 1216 2240 ===>	vmx3	vmx4	vmx5	===>	C0	4C0	8C0
224 1248 2272 ===>	vmx6	vmx7	vmx8	===>	E0	4E0	8E0
256 1280 2304	===>	vmx9	vmx10	vmx11	===>	100	400	900