Bug 195819 - TSC calibration can hang FreeBSD in Hyper-V Generation 2 VM
Summary: TSC calibration can hang FreeBSD in Hyper-V Generation 2 VM
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.1-STABLE
Hardware: amd64 Any
: Normal Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords: patch, uefi
Depends on:
Blocks: 203349
  Show dependency treegraph
 
Reported: 2014-12-09 07:35 UTC by andy zhang
Modified: 2020-11-06 01:10 UTC (History)
13 users (show)

See Also:


Attachments
efifb driver issue in hyperv 2012R2 (11.83 KB, image/png)
2014-12-09 07:35 UTC, andy zhang
no flags Details
Verbose mode in boot (48.28 KB, image/png)
2015-07-15 20:36 UTC, Przemysław Mika
no flags Details
FreeBSD 11.0-CURRENT r286893 (48.73 KB, image/png)
2015-08-24 08:46 UTC, Przemysław Mika
no flags Details
Also hung at Calibrating TSC clock ... (48.28 KB, image/png)
2015-09-02 04:15 UTC, Chris Lee
no flags Details
ACPI APIC Table hang (52.29 KB, image/png)
2015-09-09 23:06 UTC, Chris Lee
no flags Details
the installation welcome screen (8.51 KB, image/png)
2015-11-25 06:40 UTC, Dexuan Cui
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description andy zhang 2014-12-09 07:35:25 UTC
Created attachment 150384 [details]
efifb driver issue in hyperv 2012R2

Test Environment: 
      HyperV 2012R2 or VMWare 10.0 workstation
installation ISO: ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-uefi-dvd1.iso

From FreeBSD10.1, it claims that it would support UEFI.
In HyperV 2012R2, I tried to Create a Generation 2 VM (secure boot disabled), and then install from the above IOS. In the boot up, it would hang on the following:
    VT: running with driver "efifb"

After code check, there is something wrong with efifb driver in FreeBSD 10.1. I tried same steps in VMWare 10.0, it has same issue.

Note:
If you try to repro this issue in VMWare 10.0, you should follow this link to enable UEFI:
https://community.landesk.com/support/docs/DOC-27736
Comment 1 Kylie 2014-12-15 01:32:22 UTC
Andy checked that below fix is in 10.1 release. 
r268227 | emaste | 2014-07-03 10:53:28 -0700 (Thu, 03 Jul 2014) | 8 lines

Display efi framebuffer dimensions on boot
Comment 2 Przemysław Mika 2015-07-15 20:36:32 UTC
Created attachment 158812 [details]
Verbose mode in boot

boot in verbose mode gets "freez loop" at
"Calibrating TSC clock"
Comment 3 Marcel Moolenaar freebsd_committer freebsd_triage 2015-08-12 16:05:00 UTC
This is believed to be fixed for everybody in FreeBSD -current as of today (revision 286667). Please test at your earliest convenience.
Comment 4 Przemysław Mika 2015-08-24 08:46:44 UTC
Created attachment 160298 [details]
FreeBSD 11.0-CURRENT r286893

On Hyper-V 2012 R2 with FreeBSD 11.0-CURRENT r286893 still gets the same freeze loop.
Comment 5 Marcel Moolenaar freebsd_committer freebsd_triage 2015-08-24 14:34:30 UTC
(In reply to Przemysław Mika from comment #4)

Tou want to create a different bug for that. Your console obviously works, so that menas this bug is still resolved. Your hang seems to be related to calibrating the TSC and chances are slim that that will get resolved without a bug for it.

Thanks,
Comment 6 Przemysław Mika 2015-08-24 20:03:30 UTC
(In reply to Marcel Moolenaar from comment #5)
In my opinion Andy didn't have problem with display efi framebuffer
(I guess that he didn't type boot -v).
My png files only shows what's going wrong when kenrel boots in verbose mode.
If it's really needed to create new bug I will do this.

Thanks for Your hard job.
Comment 7 Marcel Moolenaar freebsd_committer freebsd_triage 2015-08-24 20:13:36 UTC
This bug is not the same as the EFI frame buffer problems that were fixed. Re-opening so that the issue can be looked into.
Comment 8 Chris Lee 2015-09-02 04:15:01 UTC
Created attachment 160616 [details]
Also hung at Calibrating TSC clock ...

Encountering the same issue with 10.2-RELEASE UEFI bootonly ISO in Hyper-V generation 2 vm.  The machine with the Hyper-V role installed is running Windows 10 Professional.  VM is configured with Dynamic Memory enabled and 4GB startup RAM with Minimum 512MB, and 4 virtual CPUs.  All options are selected for Integration Services.

Please let me know if I can help with diagnosis/resolution.
Comment 9 Chris Lee 2015-09-04 05:05:16 UTC
(In reply to Chris Lee from comment #8)
Also tried without Dynamic Memory enabled, and with Integration Services disabled.  Same effect.
Comment 10 Przemysław Mika 2015-09-05 06:23:16 UTC
(In reply to Chris Lee from comment #9)
Hi Chris,

it's no matter which settings you will set in VM and which version of FreeBSD will be used.
 
Andy Zhang also wrote he tried to run on VMWare 10.0 and it has same issue.

it's not working with UEFI:
- FreeBSD 10.1, 
- FreeBSD 10.2, 
- FreeBSD 11-CURRENT
at:
- Hyper-V 2012 r2
- VMWare 10.0
   
Effect is allways the same "Calibrationg TSC clock"

I hope some one from Development Team will soon look closer to this issue.
Comment 11 Mark Trettin 2015-09-05 14:36:10 UTC
I believe what is happening is the TSC calibration is dropping to the i8254 PIT to calibrate the TSC and I don't think the Generation 2 Hyper-V has that.
Comment 12 Gerard Seibert 2015-09-05 16:32:39 UTC
(In reply to Mark Trettin from comment #11)

You might be able to get some useful information from this URL:

https://technet.microsoft.com/en-us/library/dn848318.aspx
Comment 13 Mark Trettin 2015-09-08 19:05:49 UTC
I was able to get further set machdep.disable_tsc_calibration=1
Comment 14 Chris Lee 2015-09-09 23:06:14 UTC
Created attachment 160883 [details]
ACPI APIC Table hang

I also get farther with
set machdep.disable_tsc_calibration=1
boot -v
but it still hangs.

Perhaps this bug should be retitled as it appears to have nothing to do with the efifb driver?
Comment 15 Chris Lee 2015-09-09 23:08:50 UTC
To be fair, FreeBSD is not officially supported in a Hyper-V Generation 2 virtual machine.  I would, however, like to help change that.
Comment 16 Dexuan Cui 2015-11-25 03:39:49 UTC
(In reply to Mark Trettin from comment #11)
> I believe what is happening is the TSC calibration is dropping to the i8254 PIT to
> calibrate the TSC and I don't think the Generation 2 Hyper-V has that.

Hi Mark, you're right. In Hyper-V generation-2 Linux VM, the ACPI timer is used to calibrate TSC.

With "set machdep.disable_tsc_calibration=1", the bootup can go further but the VM still hangs later -- I guess this may be a bad side effect of disabling TSC calibration and I guess the TSC has to be calibrated here, e.g., by the ACPI timer?
Comment 17 Ed Maste freebsd_committer freebsd_triage 2015-11-25 04:27:09 UTC
> With "set machdep.disable_tsc_calibration=1", the bootup can go further but the
> VM still hangs later -- I guess this may be a bad side effect of disabling TSC
> calibration and I guess the TSC has to be calibrated here, e.g., by the ACPI
> timer?

When machdep.disable_tsc_calibration=1 is set the TSC frequency should be determined from the CPU identification instead.

Are you able to try a verbose boot with machdep.disable_tsc_calibration=1 set, and attach the boot log?
Comment 18 Dexuan Cui 2015-11-25 06:37:51 UTC
(In reply to Ed Maste from comment #17)
> When machdep.disable_tsc_calibration=1 is set the TSC frequency should be
> determined from the CPU identification instead.
OK, got it.

> Are you able to try a verbose boot with machdep.disable_tsc_calibration=1 set, and
> attach the boot log
Chris Lee has attached the log/screen in Comment 14 when the kernel hangs. I got the same log. It looks the kernel is in some busy loop forever.
NB: I guess Chris uses SMP VM.

If I use only 1 vCPU, FreeBSD 10.2 VM's kernel can boot fine and it finally stops at the Welcome screen for the installation (I'll attach the screen) -- the (virtual) keyboard doesn't work, so we can't go further...
This is because Generation-2 Hyper-V VM doesn't have the legacy 8042 keyboard, so a driver for the Hyper-V synthetic keyboard device is required inside the FreeBSD VM (BTW, Linux already has the driver) -- a new TODO. :-)
Comment 19 Dexuan Cui 2015-11-25 06:40:29 UTC
Created attachment 163496 [details]
the installation welcome screen
Comment 20 Dexuan Cui 2015-12-16 13:06:55 UTC
FYI: in a Generation-1 Hyper-V FreeBSD 10.2 VM running on Hyper-V 2012 R2, the tsc calibration by PIT seems broken:

dmesg shows
"
Calibrating TSC clock ... TSC clock: 1264483083 Hz
CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz (1264.48-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfe982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x200<ERMS>
  XSAVE Features=0x1<XSAVEOPT>
"

We can see the 2.50GHz CPU is detected as 1.26G... It looks i8254_delay() may not work properly somehow.

I tried to put "machdep.tsc_skip_calibration=1" in /boot/loader.conf, but it looks this doesn't take effect -- anyone knows which loader config file I should put the setting into?
Comment 21 Ed Maste freebsd_committer freebsd_triage 2015-12-26 23:36:12 UTC
(In reply to Przemysław Mika from comment #10)

11-CURRENT and stable/10 currently do work with vmware / UEFI.
Comment 22 Dexuan Cui 2015-12-28 01:44:27 UTC
(In reply to Ed Maste from comment #21)

It looks the i8254 PIT counter emulated by Hyper-V is not reliable and I made a workaround patch here:
https://github.com/dcui/freebsd/commit/07acbd8a3771adba9ac0fca807ec629ff05a807c
Comment 23 Mark Linimon freebsd_committer freebsd_triage 2016-05-20 07:40:20 UTC
Canonicalize assignment.
Comment 24 Dexuan Cui 2016-08-11 02:56:41 UTC
(In reply to Dexuan Cui from comment #22)
The TSC calibration issue was fixed in Apr:
https://github.com/freebsd/freebsd/commit/cff47489671a6ec6470f706f530df99c158511b0 and the fix was merged into 10.3, stable/11 and 12.-CURRENT.

but unluckily today I found the I couldn't boot UEFI VM (i.e. Hyper-V Generation-2 VM) with the 10.3 uefi iso  and 11-beta1  iso. :-(

The symptom is the same (it looks the kernel crashes at the very early point where the uefi fb driver fails to load?). I'll report a new bug.
Comment 25 Dexuan Cui 2016-08-11 03:28:57 UTC
(In reply to Dexuan Cui from comment #24)
I reported a new bug:
Bug 211746 - [Hyper-V] UEFI VM can't boot from the iso installation disk 

It looks the uefi loaders in 10.3  and 11-alpha/beta fail to load the kernel.
Comment 26 Li-Wen Hsu freebsd_committer freebsd_triage 2020-11-03 17:07:45 UTC
Is this still happening in 12.x and, if it's possible to test with snapshot builds, 13.0-CURRENT?
Comment 27 Dexuan Cui 2020-11-03 23:48:48 UTC
(In reply to Li-Wen Hsu from comment #26)
I think we can close this bug, because I believe this bug was fixed in 2016 (please refer to Comment 24). This bug should not reproduce with 10.3+, 11.x, 12.x or 13.x and newer.
Comment 28 Li-Wen Hsu freebsd_committer freebsd_triage 2020-11-05 18:35:04 UTC
(In reply to Dexuan Cui from comment #27)
Thanks for the explanation. IIUC, the both problems in this and bug 211746 have been fixed.  It just doesn't get final tests.  Is anyone able to confirm that 12.2-RELEASE and -CURRENT can boot in the Hyper-V Generation-2 VM?
Comment 29 Dexuan Cui 2020-11-05 20:19:48 UTC
(In reply to Li-Wen Hsu from comment #28)

Yes, I believe both the problems are fixed.

I just created some Generation-2 32-CPU VMs on 2 different Hyper-V hosts (Build 17763, and 20195(internal build)) with these images and all of the VMs booted up successfully:

http://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/12.2-STABLE/amd64/20201105/FreeBSD-12.2-STABLE-amd64-20201105-r367336.vhd.xz

http://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/13.0-CURRENT/amd64/20201029/FreeBSD-13.0-CURRENT-amd64-20201029-b9403d7aae8.vhd.xz
Comment 30 Li-Wen Hsu freebsd_committer freebsd_triage 2020-11-06 01:10:18 UTC
(In reply to Dexuan Cui from comment #29)
Thanks very much for confirming the fix!