Bug 203682 - Slow boot under VMware Fusion w/ UEFI firmware and SMP -- until APs are launched
Summary: Slow boot under VMware Fusion w/ UEFI firmware and SMP -- until APs are launched
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
Depends on:
Reported: 2015-10-10 17:37 UTC by Marcel Moolenaar
Modified: 2019-01-21 18:04 UTC (History)
8 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Marcel Moolenaar freebsd_committer 2015-10-10 17:37:29 UTC
- Mac OS 10.10
- Mac OS 10.11
- VMware Fusion 7 Pro
- VMware Fusion 8 Pro
VM configuration:
- UEFI firmware
- UEFI console (std 1024x768)
- Multiple CPUs (minimal 2)
- RAM (don't care AFAICT)

As soon as FreeBSD takes the APs out of INIT and has them spinning in a tight loop and until until the APs are launched, the UEFI frame buffer is *very* slow. It takes 1 or 2 seconds to scroll 1 line of characters (in graphics mode).

After the APs are launched, frame buffer access reverts to normal.

From the code (init_secondary_tail() in sys/x86/x86/mp_x86.c) it looks like the APs, while waiting to be unleashed, are not completely configured yet. In particular (and this comes from a gut feeling) the PAT MR is configured until the end (see pmap_init_pat() in sys/amd64/amd64/pmap.c). I wonder if the APs, while not being configured completely, cause the H/W (VMware Fusion in this case) to behave erratically.
Comment 1 Bernard Spil freebsd_committer 2016-07-09 10:30:30 UTC
Any updates on this?

Just ran into this with 11-CURRENTJULY2016 TrueOS Desktop image (linux-drm-4.6) on VMWare Workstation 12.1.1
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2016-07-13 22:37:27 UTC
Fix assignment.
Comment 3 Christian Ullrich 2016-08-16 14:21:18 UTC
Me too, with 11-PRERELEASE as of right now on VMware ESXi 6. It wouldn't be so bad if the thing didn't have to initialize a cool three dozen PCI buses ...
Comment 4 John Baldwin freebsd_committer freebsd_triage 2016-11-09 17:10:29 UTC
As a workaround, try adding 'options EARLY_AP_STARTUP' to your kernel config as it will start the APs sooner and should be fine on amd64.  (I plan to enable it by default in 12 "soon".
Comment 5 John Baldwin freebsd_committer freebsd_triage 2016-11-09 17:25:37 UTC
Note that the root issue is probably just that the spin on 'aps_ready' is expensive to simulate on hypervisors.  The vcpu is just going to run continually until it gets preempted by the hypervisor if it runs on a real CPU.
Comment 6 Marcel Moolenaar freebsd_committer 2016-11-09 17:32:19 UTC
(In reply to John Baldwin from comment #4)

I can confirm that this solves/addresses the problem sufficiently.

A 4 core VM with the EFI frame buffer boots what appears to be normally.
Comment 7 Marcel Moolenaar freebsd_committer 2016-11-09 17:35:06 UTC
(In reply to John Baldwin from comment #5)

Very possible. Maybe this theory be validated by using the monitor and mwait instructions on recent CPUs?
Comment 8 John Baldwin freebsd_committer freebsd_triage 2016-11-09 18:12:36 UTC
The problem with monitor/mwait is that it has to be emulated by the hypervisor (you get a VM-exit for it), but that's actually hard to do because the hypervisor can't tell when the relevant line is going to be updated (I'm pretty sure that bhyve just returns immediately from mwait for example).  We can however put the APs to sleep while spinning on aps_ready via hlt and use an IPI to resume them.  This should work with most if not all hypervisors.  That said, I do plan to enable EARLY_AP_STARTUP in 12 anyway (perhaps I'll do it at the vendor summit later this week).
Comment 9 commit-hook freebsd_committer 2016-12-16 21:11:30 UTC
A commit references this bug:

Author: jhb
Date: Fri Dec 16 21:10:38 UTC 2016
New revision: 310177
URL: https://svnweb.freebsd.org/changeset/base/310177

  Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default.

  PR:		199321, 203682
  MFC after:	2 months
  Sponsored by:	Netflix

Comment 10 Sergey Temerkhanov 2017-03-11 09:43:03 UTC
Just a note: this behavior is also present in FreeBSD-10.3-RELEASE-amd64-*.iso but FreeBSD-10.3-RELEASE-amd64-uefi-*.iso images are free from this
Comment 11 commit-hook freebsd_committer 2017-05-24 00:01:41 UTC
A commit references this bug:

Author: jhb
Date: Wed May 24 00:00:56 UTC 2017
New revision: 318763
URL: https://svnweb.freebsd.org/changeset/base/318763

  MFC 310177: Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default.

  PR:		199321, 203682
  Discussed with:	re (kib)
  Relnotes:	yes

_U  stable/11/
Comment 12 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:43:04 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
- Untouched since 2018-01-01.
- Affects Base System OR Documentation


Reset to open status.

I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 13 Oleksandr Tymoshenko freebsd_committer freebsd_triage 2019-01-21 18:00:14 UTC
There is a commit referencing this PR, but it's still not closed and has been inactive for some time. Closing the PR as fixed but feel free to re-open it if the issue hasn't been completely resolved.

Comment 14 vvv 2019-01-21 18:04:26 UTC
Yes, it's fixed in 11.2 and 12.0.