Bug 255086

Summary: 13.0-RELEASE installer reboots when trying to DHCP vmx0 interface
Product: Base System Reporter: Markus Wild <freebsd-bugs>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Only Me CC: avg, dpetrov67, info
Priority: ---    
Version: 13.0-STABLE   
Hardware: amd64   
OS: Any   

Description Markus Wild 2021-04-15 09:40:49 UTC
How to reproduce:

Trying to install new 13.0-RELEASE vm on vmware 5.5 using disc ISO:
- VM is configured for 16GB, 4 cores, 2 vmxnet3 interfaces
- installer starts from ISO:
  - configure auto-ZFS
  - change swap to 16GB
  - select stripe on the single disk (da0)
  - confirm and let the installer set up the disk
  - when the network installer starts and asks whether to configure vmx0:
    - configure ipv4
    - configure using DHCP
    => VM reboots, no panic, no additional information. Console window goes
       black and the VM eventually reboots. I don't see any error in vmware
       either

When I try the same process using 2 E1000 interfaces, it works fine.
Comment 1 Markus Wild 2021-04-15 09:58:36 UTC
Just tried to add a vmxnet3 interface to the new VM that was configured
with 2 E1000 interfaces. As soon as I do

ifconfig vmx0 up

the whole VM resets, no panic string. I started the VM with verbose flag
on, no change.
Comment 2 Andriy Gapon freebsd_committer freebsd_triage 2021-04-15 10:48:51 UTC
It looks like you are hitting a bug in vmware.
I recall having a similar issue with 5.5 U1.
Try upgrading to the latest patch version or a newer version altogether.
Comment 3 Andriy Gapon freebsd_committer freebsd_triage 2021-04-15 10:49:59 UTC
You may want to check if dev.vmx.0.iflib.disable_msix=0 works as a workaround.
Comment 4 Markus Wild 2021-04-15 11:22:45 UTC
(In reply to Andriy Gapon from comment #3)
No, this doesn't work (same crash), and I'm on the last 5.5 version. 
Can't upgrade due to license issues.

I've just found: https://reviews.freebsd.org/D22819

If I revert this and boot with the release-kernel minus this change,
vmx works fine. How can I achieve the same result without patching the
kernel?
Comment 5 Andriy Gapon freebsd_committer freebsd_triage 2021-09-08 19:58:08 UTC
Another option is to disable MSI-X for the network interface in the VMWare configuration. There is (possibly undocumented) configuration option ethernet%d.intrMode. It can be set to 2 to enable MSI interrupts or to 1 to enable legacy interrupts only.

In any case, a VMWare crash is a VMWare bug.
It should not crash no matter what a guest does.
Comment 6 info 2021-10-04 21:39:47 UTC
I was hit by this. I also can't upgrade from esxi 5.5. Just wanted to say that setting ethernet0.intrMode to 2 has stopped my vms from crashing when dhclient runs.

I don't know what the performance impact is, but I'm hoping it's less than switching to E1000 interfaces would be.
Comment 7 Andriy Gapon freebsd_committer freebsd_triage 2021-10-28 17:15:07 UTC
For posterity, in comment #3 I suggested incorrect value for the tunable resulting in a double negation.
The suggestion should be to set dev.vmx.0.iflib.disable_msix=1 in /boot/loader.conf.
Comment 8 Andriy Gapon freebsd_committer freebsd_triage 2021-10-28 17:17:17 UTC
Also interesting that pre-iflib vmxnet3 driver does not seem to hit this problem even when MSI-X is enabled and used (and the blacklist is disabled).