|Summary:||Slow boot under VMware Fusion w/ UEFI firmware and SMP -- until APs are launched|
|Product:||Base System||Reporter:||Marcel Moolenaar <marcel>|
|Component:||kern||Assignee:||freebsd-virtualization mailing list <virtualization>|
|Severity:||Affects Some People||CC:||allanjude, brnrd, chris, emaste, gonzo, jhb, s.temerkhanov, work+freebsd|
Description Marcel Moolenaar 2015-10-10 17:37:29 UTC
Host: - Mac OS 10.10 - Mac OS 10.11 Hypervisor: - VMware Fusion 7 Pro - VMware Fusion 8 Pro VM configuration: - UEFI firmware - UEFI console (std 1024x768) - Multiple CPUs (minimal 2) - RAM (don't care AFAICT) Description: As soon as FreeBSD takes the APs out of INIT and has them spinning in a tight loop and until until the APs are launched, the UEFI frame buffer is *very* slow. It takes 1 or 2 seconds to scroll 1 line of characters (in graphics mode). After the APs are launched, frame buffer access reverts to normal. From the code (init_secondary_tail() in sys/x86/x86/mp_x86.c) it looks like the APs, while waiting to be unleashed, are not completely configured yet. In particular (and this comes from a gut feeling) the PAT MR is configured until the end (see pmap_init_pat() in sys/amd64/amd64/pmap.c). I wonder if the APs, while not being configured completely, cause the H/W (VMware Fusion in this case) to behave erratically.
Comment 1 Bernard Spil 2016-07-09 10:30:30 UTC
Any updates on this? Just ran into this with 11-CURRENTJULY2016 TrueOS Desktop image (linux-drm-4.6) on VMWare Workstation 12.1.1
Comment 2 Mark Linimon 2016-07-13 22:37:27 UTC
Comment 3 Christian Ullrich 2016-08-16 14:21:18 UTC
Me too, with 11-PRERELEASE as of right now on VMware ESXi 6. It wouldn't be so bad if the thing didn't have to initialize a cool three dozen PCI buses ...
Comment 4 John Baldwin 2016-11-09 17:10:29 UTC
As a workaround, try adding 'options EARLY_AP_STARTUP' to your kernel config as it will start the APs sooner and should be fine on amd64. (I plan to enable it by default in 12 "soon".
Comment 5 John Baldwin 2016-11-09 17:25:37 UTC
Note that the root issue is probably just that the spin on 'aps_ready' is expensive to simulate on hypervisors. The vcpu is just going to run continually until it gets preempted by the hypervisor if it runs on a real CPU.
Comment 6 Marcel Moolenaar 2016-11-09 17:32:19 UTC
(In reply to John Baldwin from comment #4) I can confirm that this solves/addresses the problem sufficiently. A 4 core VM with the EFI frame buffer boots what appears to be normally.
Comment 7 Marcel Moolenaar 2016-11-09 17:35:06 UTC
(In reply to John Baldwin from comment #5) Very possible. Maybe this theory be validated by using the monitor and mwait instructions on recent CPUs?
Comment 8 John Baldwin 2016-11-09 18:12:36 UTC
The problem with monitor/mwait is that it has to be emulated by the hypervisor (you get a VM-exit for it), but that's actually hard to do because the hypervisor can't tell when the relevant line is going to be updated (I'm pretty sure that bhyve just returns immediately from mwait for example). We can however put the APs to sleep while spinning on aps_ready via hlt and use an IPI to resume them. This should work with most if not all hypervisors. That said, I do plan to enable EARLY_AP_STARTUP in 12 anyway (perhaps I'll do it at the vendor summit later this week).
Comment 9 commit-hook 2016-12-16 21:11:30 UTC
A commit references this bug: Author: jhb Date: Fri Dec 16 21:10:38 UTC 2016 New revision: 310177 URL: https://svnweb.freebsd.org/changeset/base/310177 Log: Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default. PR: 199321, 203682 MFC after: 2 months Sponsored by: Netflix Changes: head/sys/amd64/conf/GENERIC head/sys/amd64/conf/MINIMAL head/sys/i386/conf/GENERIC
Comment 10 Sergey Temerkhanov 2017-03-11 09:43:03 UTC
Just a note: this behavior is also present in FreeBSD-10.3-RELEASE-amd64-*.iso but FreeBSD-10.3-RELEASE-amd64-uefi-*.iso images are free from this
Comment 11 commit-hook 2017-05-24 00:01:41 UTC
A commit references this bug: Author: jhb Date: Wed May 24 00:00:56 UTC 2017 New revision: 318763 URL: https://svnweb.freebsd.org/changeset/base/318763 Log: MFC 310177: Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default. PR: 199321, 203682 Discussed with: re (kib) Relnotes: yes Changes: _U stable/11/ stable/11/sys/amd64/conf/GENERIC stable/11/sys/amd64/conf/MINIMAL stable/11/sys/i386/conf/GENERIC
Comment 12 Eitan Adler 2018-05-28 19:43:04 UTC
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 13 Oleksandr Tymoshenko 2019-01-21 18:00:14 UTC
There is a commit referencing this PR, but it's still not closed and has been inactive for some time. Closing the PR as fixed but feel free to re-open it if the issue hasn't been completely resolved. Thanks
Comment 14 vvv 2019-01-21 18:04:26 UTC
Yes, it's fixed in 11.2 and 12.0.