Host: - Mac OS 10.10 - Mac OS 10.11 Hypervisor: - VMware Fusion 7 Pro - VMware Fusion 8 Pro VM configuration: - UEFI firmware - UEFI console (std 1024x768) - Multiple CPUs (minimal 2) - RAM (don't care AFAICT) Description: As soon as FreeBSD takes the APs out of INIT and has them spinning in a tight loop and until until the APs are launched, the UEFI frame buffer is *very* slow. It takes 1 or 2 seconds to scroll 1 line of characters (in graphics mode). After the APs are launched, frame buffer access reverts to normal. From the code (init_secondary_tail() in sys/x86/x86/mp_x86.c) it looks like the APs, while waiting to be unleashed, are not completely configured yet. In particular (and this comes from a gut feeling) the PAT MR is configured until the end (see pmap_init_pat() in sys/amd64/amd64/pmap.c). I wonder if the APs, while not being configured completely, cause the H/W (VMware Fusion in this case) to behave erratically.
Any updates on this? Just ran into this with 11-CURRENTJULY2016 TrueOS Desktop image (linux-drm-4.6) on VMWare Workstation 12.1.1
Fix assignment.
Me too, with 11-PRERELEASE as of right now on VMware ESXi 6. It wouldn't be so bad if the thing didn't have to initialize a cool three dozen PCI buses ...
As a workaround, try adding 'options EARLY_AP_STARTUP' to your kernel config as it will start the APs sooner and should be fine on amd64. (I plan to enable it by default in 12 "soon".
Note that the root issue is probably just that the spin on 'aps_ready' is expensive to simulate on hypervisors. The vcpu is just going to run continually until it gets preempted by the hypervisor if it runs on a real CPU.
(In reply to John Baldwin from comment #4) I can confirm that this solves/addresses the problem sufficiently. A 4 core VM with the EFI frame buffer boots what appears to be normally.
(In reply to John Baldwin from comment #5) Very possible. Maybe this theory be validated by using the monitor and mwait instructions on recent CPUs?
The problem with monitor/mwait is that it has to be emulated by the hypervisor (you get a VM-exit for it), but that's actually hard to do because the hypervisor can't tell when the relevant line is going to be updated (I'm pretty sure that bhyve just returns immediately from mwait for example). We can however put the APs to sleep while spinning on aps_ready via hlt and use an IPI to resume them. This should work with most if not all hypervisors. That said, I do plan to enable EARLY_AP_STARTUP in 12 anyway (perhaps I'll do it at the vendor summit later this week).
A commit references this bug: Author: jhb Date: Fri Dec 16 21:10:38 UTC 2016 New revision: 310177 URL: https://svnweb.freebsd.org/changeset/base/310177 Log: Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default. PR: 199321, 203682 MFC after: 2 months Sponsored by: Netflix Changes: head/sys/amd64/conf/GENERIC head/sys/amd64/conf/MINIMAL head/sys/i386/conf/GENERIC
Just a note: this behavior is also present in FreeBSD-10.3-RELEASE-amd64-*.iso but FreeBSD-10.3-RELEASE-amd64-uefi-*.iso images are free from this
A commit references this bug: Author: jhb Date: Wed May 24 00:00:56 UTC 2017 New revision: 318763 URL: https://svnweb.freebsd.org/changeset/base/318763 Log: MFC 310177: Enable EARLY_AP_STARTUP on amd64 and i386 kernels by default. PR: 199321, 203682 Discussed with: re (kib) Relnotes: yes Changes: _U stable/11/ stable/11/sys/amd64/conf/GENERIC stable/11/sys/amd64/conf/MINIMAL stable/11/sys/i386/conf/GENERIC
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
There is a commit referencing this PR, but it's still not closed and has been inactive for some time. Closing the PR as fixed but feel free to re-open it if the issue hasn't been completely resolved. Thanks
Yes, it's fixed in 11.2 and 12.0.