Bug 238095

Summary: Current console input freezes in hyperv gen 2 VM after randon amount of time
Product: Base System Reporter: scorpionmage
Component: miscAssignee: freebsd-virtualization (Nobody) <virtualization>
Status: New ---    
Severity: Affects Some People CC: hselasky, info, markus.egger, me, michael.adm, sdolha
Priority: ---    
Version: CURRENT   
Hardware: amd64   
OS: Any   

Description scorpionmage 2019-05-24 05:20:33 UTC
Installation of FreeBSD-current on a HyperV Gen 2 VM, the release/version is r347896.

After booting up properly, the console input freezes after a random amount of time. This can be if a user is logged in or not. It can be from 5 minutes after being turned on to 3 hours when this happens, it is random.

This freeze also happens if X is started - X starts with xmonad as the mouse does not work in Gen 2 VMs. There is no screensaver installed.

Console output is not affected, currently a "buildworld && buildkernel" is running and it is outputting new lines. But no input is possible.

Remote login using SSH is not affected, it is still possible to login remotely.

Logging in remotely, dmesg does not report anything.
Comment 1 Dries Michiels freebsd_committer freebsd_triage 2019-05-25 09:56:35 UTC
I have noticed this too in my hyper-V FreeBSD-head vm.
Comment 2 Markus Egger 2019-06-27 11:27:03 UTC
Me, too.

- Windows Server 2016 Hypervisor w/ current updates
- FreeBSD 12.0p6 x64 in Gen2 VM
Comment 3 Sorin Dolha 2019-07-30 18:06:48 UTC
Me too, same diagnostics, in Hyper-V on Windows 10 (build 1803), with FreeBSD image downloaded and installed from scratch, without any configurations.
Comment 4 Juan Ramón Molina Menor 2019-10-07 15:28:30 UTC
I’m also seeing this in Windows 10 with 12.1-BETA3. Three times just today. Nothing suspicious is logged.
Comment 5 Michael 2019-11-24 18:30:44 UTC
Me, too.

- Windows Server 2019 Hypervisor w/ current updates
- FreeBSD 13.0 CURRENT x64 GENERIC in Gen2 VM
Comment 6 Michael 2019-11-29 13:48:51 UTC
Maybe this bug is related to this bug? :

If you try on the hypervisor (in powershell)

for gen1 vm:
PS > Disable-VMConsoleSupport -VMName "test-FreeBSD-gen1"
Disable-VMConsoleSupport : Failed to remove device "Microsoft:Hyper-V:Synthetic Mouse": Invalid operation (0x80041016).

for gen2 vm:
PS > Disable-VMConsoleSupport -VMName "test-FreeBSD-gen2"
Disable-VMConsoleSupport : Failed to remove device "Synthetic Mouse": General failure. (0x80041001).
Comment 7 Hans Petter Selasky freebsd_committer freebsd_triage 2019-12-01 20:10:32 UTC
How many CPU's are there?

--HPS
Comment 8 Michael 2019-12-02 07:11:41 UTC
The CPU is usually eight, but the same behavior with four and twelve cpu's
Comment 9 Michael 2019-12-03 05:29:42 UTC
(In reply to Hans Petter Selasky from comment #7)
Yes, it seems the number of CPU's makes sense in this bug. All VM's with CPU's more than one lost keyboard input, one VM with one CPU did not lose keyboard input in same hipervisor during the day already (maybe this is a coincidence, or is this really a problem but not a solution)
Comment 10 Michael 2019-12-03 07:39:42 UTC
Maybe this also relates to this bug:
when using the clipboard when copying one character in the hypervisor and Ctrl-V in the VM, we have two identical messages in the console at the same time (verbose logs is on - boot_verbose="YES" -> /boot/loader.conf)

hvkbd0: Unsupported unicode
hvkbd0: Unsupported unicode
Comment 11 Juan Ramón Molina Menor 2019-12-05 21:56:44 UTC
(In reply to Michael from comment #9)
I’m using 2 CPUs and regularly seeing this. I’ll test tomorrow with only 1 CPU.
Comment 12 Juan Ramón Molina Menor 2019-12-06 15:10:42 UTC
(In reply to Juan Ramón Molina Menor from comment #11)
I can reproduce the issue by using 2 or 4 CPUs, but not with only one. I have not found yet a reliable way to trigger it…
Comment 13 Hans Petter Selasky freebsd_committer freebsd_triage 2019-12-06 15:21:57 UTC
Last time I was debugging something in this area, I found that the UART registers, which appear emulated, have some races. For example if I was reading the UART registers right before posting a doorbell on the mlx4en driver, the hypervisor would reboot the image :-(

+       if (mlx4_crash != 0) {
+               while (1) {
+                       inb(0x3f8+5);
+               }
+       }
+       __raw_writel((__force u32)val,
+            &priv->mfunc.comm->slave_write);

I was not able to get to the bottom of this due to lack of interest from the hypervisor developers.

This race happens only with N number of CPUs.

You may play around with inb() in the network driver code and see if you can provoke a similar action.

--HPS
Comment 14 Michael 2019-12-06 16:26:09 UTC
These console input freezes occur even without a single network interface in general
Comment 15 Michael 2020-01-06 13:17:58 UTC
Faced an interesting moment. I install from the iso-image from 2019-12-05.
FreeBSD-13.0-CURRENT-amd64-20191205-r355406-disc1.iso
When loading an already working system at the time of the countdown of 10 seconds of the boot selection option, press Enter and get a completely freezing system. If we do not touch anything (a countdown of 10 seconds occurs), the system boots and works.
Made a complete system update.
make buildworld && make buildkernel KERNCONF=GENERIC
make installkernel KERNCONF=GENERIC
make installworld
mergemaster -Ui
reboot
...the same - at the time of the countdown of 10 seconds of the boot selection option, press Enter and get a completely freezing system. If we do not touch anything (a countdown of 10 seconds occurs), the system boots and works.
ok.
mount -t msdosfs /dev/da0p1 /mnt
cp /boot/loader.efi /mnt/EFI/freebsd
reboot
at the time of the countdown of 10 seconds of the boot selection option, press Enter and system boots normal, as it shuld.
Could these spontaneous keyboard disappearances in running multiprocessor virtual machines be due to this?
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240734