Bug 203682

Summary:	Slow boot under VMware Fusion w/ UEFI firmware and SMP -- until APs are launched
Product:	Base System	Reporter:	Marcel Moolenaar <marcel>
Component:	kern	Assignee:	freebsd-virtualization (Nobody) <virtualization>
Status:	Closed FIXED
Severity:	Affects Some People	CC:	allanjude, brnrd, chris, emaste, gonzo, jhb, s.temerkhanov, work+freebsd
Priority:	---
Version:	CURRENT
Hardware:	amd64
OS:	Any
See Also:	https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214518

Description Marcel Moolenaar freebsd_committer

2015-10-10 17:37:29 UTC

Host:
- Mac OS 10.10
- Mac OS 10.11
Hypervisor:
- VMware Fusion 7 Pro
- VMware Fusion 8 Pro
VM configuration:
- UEFI firmware
- UEFI console (std 1024x768)
- Multiple CPUs (minimal 2)
- RAM (don't care AFAICT)

Description:
As soon as FreeBSD takes the APs out of INIT and has them spinning in a tight loop and until until the APs are launched, the UEFI frame buffer is *very* slow. It takes 1 or 2 seconds to scroll 1 line of characters (in graphics mode).

After the APs are launched, frame buffer access reverts to normal.

From the code (init_secondary_tail() in sys/x86/x86/mp_x86.c) it looks like the APs, while waiting to be unleashed, are not completely configured yet. In particular (and this comes from a gut feeling) the PAT MR is configured until the end (see pmap_init_pat() in sys/amd64/amd64/pmap.c). I wonder if the APs, while not being configured completely, cause the H/W (VMware Fusion in this case) to behave erratically.

Comment 1 Bernard Spil freebsd_committer

2016-07-09 10:30:30 UTC

Any updates on this?

Just ran into this with 11-CURRENTJULY2016 TrueOS Desktop image (linux-drm-4.6) on VMWare Workstation 12.1.1

Comment 2 Mark Linimon freebsd_committer

2016-07-13 22:37:27 UTC

Fix assignment.

Comment 3 Christian Ullrich 2016-08-16 14:21:18 UTC

Me too, with 11-PRERELEASE as of right now on VMware ESXi 6. It wouldn't be so bad if the thing didn't have to initialize a cool three dozen PCI buses ...

Comment 4 John Baldwin freebsd_committer

2016-11-09 17:10:29 UTC

As a workaround, try adding 'options EARLY_AP_STARTUP' to your kernel config as it will start the APs sooner and should be fine on amd64.  (I plan to enable it by default in 12 "soon".

Comment 5 John Baldwin freebsd_committer

2016-11-09 17:25:37 UTC

Note that the root issue is probably just that the spin on 'aps_ready' is expensive to simulate on hypervisors.  The vcpu is just going to run continually until it gets preempted by the hypervisor if it runs on a real CPU.

Comment 6 Marcel Moolenaar freebsd_committer

2016-11-09 17:32:19 UTC

(In reply to John Baldwin from comment #4)

I can confirm that this solves/addresses the problem sufficiently.

A 4 core VM with the EFI frame buffer boots what appears to be normally.

Comment 7 Marcel Moolenaar freebsd_committer

2016-11-09 17:35:06 UTC

(In reply to John Baldwin from comment #5)

Very possible. Maybe this theory be validated by using the monitor and mwait instructions on recent CPUs?

Comment 8 John Baldwin freebsd_committer

2016-11-09 18:12:36 UTC

The problem with monitor/mwait is that it has to be emulated by the hypervisor (you get a VM-exit for it), but that's actually hard to do because the hypervisor can't tell when the relevant line is going to be updated (I'm pretty sure that bhyve just returns immediately from mwait for example).  We can however put the APs to sleep while spinning on aps_ready via hlt and use an IPI to resume them.  This should work with most if not all hypervisors.  That said, I do plan to enable EARLY_AP_STARTUP in 12 anyway (perhaps I'll do it at the vendor summit later this week).

Comment 9 commit-hook freebsd_committer

2016-12-16 21:11:30 UTC

A commit references this bug:

Author: jhb
Date: Fri Dec 16 21:10:38 UTC 2016
New revision: 310177
URL: https://svnweb.freebsd.org/changeset/base/310177

Log:
  Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default.

  PR:		199321, 203682
  MFC after:	2 months
  Sponsored by:	Netflix

Changes:
  head/sys/amd64/conf/GENERIC
  head/sys/amd64/conf/MINIMAL
  head/sys/i386/conf/GENERIC

Comment 10 Sergey Temerkhanov 2017-03-11 09:43:03 UTC

Just a note: this behavior is also present in FreeBSD-10.3-RELEASE-amd64-*.iso but FreeBSD-10.3-RELEASE-amd64-uefi-*.iso images are free from this

Comment 11 commit-hook freebsd_committer

2017-05-24 00:01:41 UTC

A commit references this bug:

Author: jhb
Date: Wed May 24 00:00:56 UTC 2017
New revision: 318763
URL: https://svnweb.freebsd.org/changeset/base/318763

Log:
  MFC 310177: Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default.

  PR:		199321, 203682
  Discussed with:	re (kib)
  Relnotes:	yes

Changes:
_U  stable/11/
  stable/11/sys/amd64/conf/GENERIC
  stable/11/sys/amd64/conf/MINIMAL
  stable/11/sys/i386/conf/GENERIC

Comment 12 Eitan Adler freebsd_committer

2018-05-28 19:43:04 UTC

batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.

Comment 13 Oleksandr Tymoshenko freebsd_committer

2019-01-21 18:00:14 UTC

There is a commit referencing this PR, but it's still not closed and has been inactive for some time. Closing the PR as fixed but feel free to re-open it if the issue hasn't been completely resolved.

Thanks

Comment 14 vvv 2019-01-21 18:04:26 UTC

Yes, it's fixed in 11.2 and 12.0.