Bug 203682

Summary: Slow boot under VMware Fusion w/ UEFI firmware and SMP -- until APs are launched
Product: Base System Reporter: Marcel Moolenaar <marcel>
Component: kernAssignee: freebsd-virtualization mailing list <virtualization>
Status: Closed FIXED    
Severity: Affects Some People CC: allanjude, brnrd, chris, emaste, gonzo, jhb, s.temerkhanov, work+freebsd
Priority: ---    
Version: CURRENT   
Hardware: amd64   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214518

Description Marcel Moolenaar freebsd_committer 2015-10-10 17:37:29 UTC
Host:
- Mac OS 10.10
- Mac OS 10.11
Hypervisor:
- VMware Fusion 7 Pro
- VMware Fusion 8 Pro
VM configuration:
- UEFI firmware
- UEFI console (std 1024x768)
- Multiple CPUs (minimal 2)
- RAM (don't care AFAICT)

Description:
As soon as FreeBSD takes the APs out of INIT and has them spinning in a tight loop and until until the APs are launched, the UEFI frame buffer is *very* slow. It takes 1 or 2 seconds to scroll 1 line of characters (in graphics mode).

After the APs are launched, frame buffer access reverts to normal.

From the code (init_secondary_tail() in sys/x86/x86/mp_x86.c) it looks like the APs, while waiting to be unleashed, are not completely configured yet. In particular (and this comes from a gut feeling) the PAT MR is configured until the end (see pmap_init_pat() in sys/amd64/amd64/pmap.c). I wonder if the APs, while not being configured completely, cause the H/W (VMware Fusion in this case) to behave erratically.
Comment 1 Bernard Spil freebsd_committer 2016-07-09 10:30:30 UTC
Any updates on this?

Just ran into this with 11-CURRENTJULY2016 TrueOS Desktop image (linux-drm-4.6) on VMWare Workstation 12.1.1
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2016-07-13 22:37:27 UTC
Fix assignment.
Comment 3 Christian Ullrich 2016-08-16 14:21:18 UTC
Me too, with 11-PRERELEASE as of right now on VMware ESXi 6. It wouldn't be so bad if the thing didn't have to initialize a cool three dozen PCI buses ...
Comment 4 John Baldwin freebsd_committer freebsd_triage 2016-11-09 17:10:29 UTC
As a workaround, try adding 'options EARLY_AP_STARTUP' to your kernel config as it will start the APs sooner and should be fine on amd64.  (I plan to enable it by default in 12 "soon".
Comment 5 John Baldwin freebsd_committer freebsd_triage 2016-11-09 17:25:37 UTC
Note that the root issue is probably just that the spin on 'aps_ready' is expensive to simulate on hypervisors.  The vcpu is just going to run continually until it gets preempted by the hypervisor if it runs on a real CPU.
Comment 6 Marcel Moolenaar freebsd_committer 2016-11-09 17:32:19 UTC
(In reply to John Baldwin from comment #4)

I can confirm that this solves/addresses the problem sufficiently.

A 4 core VM with the EFI frame buffer boots what appears to be normally.
Comment 7 Marcel Moolenaar freebsd_committer 2016-11-09 17:35:06 UTC
(In reply to John Baldwin from comment #5)

Very possible. Maybe this theory be validated by using the monitor and mwait instructions on recent CPUs?
Comment 8 John Baldwin freebsd_committer freebsd_triage 2016-11-09 18:12:36 UTC
The problem with monitor/mwait is that it has to be emulated by the hypervisor (you get a VM-exit for it), but that's actually hard to do because the hypervisor can't tell when the relevant line is going to be updated (I'm pretty sure that bhyve just returns immediately from mwait for example).  We can however put the APs to sleep while spinning on aps_ready via hlt and use an IPI to resume them.  This should work with most if not all hypervisors.  That said, I do plan to enable EARLY_AP_STARTUP in 12 anyway (perhaps I'll do it at the vendor summit later this week).
Comment 9 commit-hook freebsd_committer 2016-12-16 21:11:30 UTC
A commit references this bug:

Author: jhb
Date: Fri Dec 16 21:10:38 UTC 2016
New revision: 310177
URL: https://svnweb.freebsd.org/changeset/base/310177

Log:
  Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default.

  PR:		199321, 203682
  MFC after:	2 months
  Sponsored by:	Netflix

Changes:
  head/sys/amd64/conf/GENERIC
  head/sys/amd64/conf/MINIMAL
  head/sys/i386/conf/GENERIC
Comment 10 Sergey Temerkhanov 2017-03-11 09:43:03 UTC
Just a note: this behavior is also present in FreeBSD-10.3-RELEASE-amd64-*.iso but FreeBSD-10.3-RELEASE-amd64-uefi-*.iso images are free from this
Comment 11 commit-hook freebsd_committer 2017-05-24 00:01:41 UTC
A commit references this bug:

Author: jhb
Date: Wed May 24 00:00:56 UTC 2017
New revision: 318763
URL: https://svnweb.freebsd.org/changeset/base/318763

Log:
  MFC 310177: Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default.

  PR:		199321, 203682
  Discussed with:	re (kib)
  Relnotes:	yes

Changes:
_U  stable/11/
  stable/11/sys/amd64/conf/GENERIC
  stable/11/sys/amd64/conf/MINIMAL
  stable/11/sys/i386/conf/GENERIC
Comment 12 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:43:04 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 13 Oleksandr Tymoshenko freebsd_committer freebsd_triage 2019-01-21 18:00:14 UTC
There is a commit referencing this PR, but it's still not closed and has been inactive for some time. Closing the PR as fixed but feel free to re-open it if the issue hasn't been completely resolved.

Thanks
Comment 14 vvv 2019-01-21 18:04:26 UTC
Yes, it's fixed in 11.2 and 12.0.