Bug 213333 - FreeBSD 11-RELEASE fails to boot under KVM/Qemu
Summary: FreeBSD 11-RELEASE fails to boot under KVM/Qemu
Status: Closed DUPLICATE of bug 213155
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-virtualization (Nobody)
Depends on:
Reported: 2016-10-09 10:29 UTC by Lex van Roon
Modified: 2018-03-12 11:29 UTC (History)
14 users (show)

See Also:

Screenshot of failed boot of 11-RELEASE/amd64 on Debian/KVM (30.28 KB, image/png)
2016-10-09 10:29 UTC, Lex van Roon
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Lex van Roon 2016-10-09 10:29:27 UTC
Created attachment 175567 [details]
Screenshot of failed boot of 11-RELEASE/amd64 on Debian/KVM

- Debian Jessie 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux
- Qemu-kvm 1:2.1+dfsg-12+deb8u6

- 11-RELEASE amd64 bootonly iso

While booting a fresh virtual machine using 11-RELEASE/amd64, the kernel hangs right after it's loaded (see attached screenshot). This happens both with ACPI enabled and with ACPI disabled. I tried booting 10.3-RELEASE, which runs perfectly. I also operate multiple 10.[23] VM's on the same hypervisor, which all work.

Do note, the spinner spins about 2~3 x before it freezes.
Comment 1 Andrew Keling 2016-10-30 19:51:16 UTC
I am also seeing this issue with KVM on OpenSUSE 42.1. I have also tried the same troubleshooting steps. 

My system information.

Repository: openSUSE-Leap-42.1-Update
Name: qemu-kvm
Version: 2.3.1-19.3
Arch: x86_64
Vendor: openSUSE

sharp@TwoRivers:~> uname -a
Linux TwoRivers 4.1.34-33-default #1 SMP PREEMPT Thu Oct 20 08:03:29 UTC 2016 (fe18aba) x86_64 x86_64 x86_64 GNU/Linux

sharp@TwoRivers:~> cat /proc/cpuinfo                                                                                                                                                                                    [192/948]
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 1
model name      : AMD FX(tm)-8120 Eight-Core Processor
stepping        : 2
microcode       : 0x6000629
cpu MHz         : 1400.000
cache size      : 2048 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf
 pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core perfctr_nb ar
at cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold vmmcall
bugs            : fxsave_leak sysret_ss_attrs
bogomips        : 6241.87
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb
Comment 2 Andrew Keling 2016-10-30 20:01:16 UTC
Found a work around. Seems to be an issue with copying the the host CPU topology and with the Opteron_G4 CPU arch too.

Switching to Opteron_G3 or selecting Hypervisor_Default seems to mitigate this issue.
Comment 3 Andrew Keling 2016-10-30 20:04:55 UTC
Should note that this corrected my issue upgrading my servers from 10.3 to 11. I was also seeing issue with the install cd locking up too.
Comment 4 Lex van Roon 2016-11-21 21:19:35 UTC
I can confirm that by switching the hypervisor to Opteron_G3 I've got a working FreeBSD 11.
Comment 5 Cameron 2017-01-01 05:11:27 UTC
I can confirm the EXACT same issue here. My server has dual Opteron 6348 CPU's. FreeBSD 10.3 and earlier all work correctly.

My other system with a Xeon D-1541 has no such problem.
Comment 6 Cameron 2017-01-01 05:22:24 UTC
To clarify my comment, using Opteron_G5 or Opteron_G4 causes boot to hang with FreeBSD 11.0-p6. Using Opteron_G3 works.
Comment 7 Miguel Castellanos 2017-01-13 00:08:57 UTC
I confirm the same problem with Opteron 6272 Processors, under Centos 7.2/KVM Hypervisor.
Comment 8 Cameron 2017-01-13 00:11:16 UTC
Can anyone try booting FreeBSD 11.0 directly on the hardware? Unfortunately, I'm unable to. This may not be KVM related.

I also tried 11-STABLE around a week back, the issue is still present there.
Comment 9 Luciano Mannucci 2017-01-13 11:41:16 UTC
(In reply to Cameron from comment #8)
I tried on Intel I3 Vaio and on an dual Intel(R) Xeon(R) CPU E5410  @ 2.33GHz
and it doesn't boot.
I guess it may be a bios issue linked to UEFI beeing advertized but
not properly implemented (Just a guess). 10.3-RELEASE boots flawlessly
on the same hardware.

Hope this helps,

Comment 10 Miguel Castellanos 2017-01-13 13:06:21 UTC
I tried directly on the hardware as requested.

FreeBSD-11.RELEASE booted correctly under the physical hardware, but fails to boot as guest on a qemu virtual machine under KVM (Centos 7.2) on the same hardware.

I tested on a SuperMicro server with dual AMD Opteron 6272 processors, with a non UEFI BIOS.

If you need more details please let me know.


Comment 11 Cameron 2017-01-13 22:23:22 UTC
I have a non-UEFI BIOS with a Supermicro H8DG6-F motherboard and dual 6348 CPU's. This issue doesn't appear to be UEFI related.
Comment 12 Lex van Roon 2017-01-15 13:42:34 UTC
I am also experiencing these issues with a non-UEFI based system. Motherboard: Asus M5A99X EVO, CPU: AMD FX-8120
Comment 13 tsuroerusu 2017-09-13 11:27:53 UTC
I can also attest to this problem that everybody else is talking about. I was just trying to install FreeBSD 11.1-RELEASE in a VM using KVM on CentOS 7 when I ran into this when the installer refused to boot. The problem occurs both with "Copy host CPU configuration" as well as the "Opteron_G4" and "Opteron_G5" options. Opteron_G3 had it booting without an issue.
Comment 14 uewsq 2017-09-24 20:49:08 UTC
This is affecting me too. I tried to identify the CPU feature flags that might be preventing boot on "Opteron_G4", by iteratively attempting boot with each single non-G3 feature flag disabled in KVM. Unfortunately none of those attempts worked, so it seems this must be due to a some combination of multiple CPU feature flags.
Comment 15 Abi 2017-10-16 15:03:27 UTC
Affecting me too. Started since pfsense 2.4 uses FreeBSD 11
Comment 16 mainland 2017-12-05 14:25:51 UTC
I have the same issue with both 10.4 and 11.1 kernels---I cannot upgrade from 10.3. Setting hw.use_xsave=0 in loader.conf does not help.
Comment 17 Cameron 2017-12-05 20:21:05 UTC
I've managed to work around this. 

There's 2 issues here:
1. Rather than using an Opteron_* model in kvm for the CPU, etc. Use kvm64. Others may work as well. I believe this alone will Just Work if you haven't been building from source.

2. Did you build everything/anything from source? You probably don't want to set 
'CPUTYPE?=native', etc in /etc/make.conf (or perhaps you can find something "safe" to set it to?). 

If you are building from source AND had CPUTYPE set to something like Opteron_*, you'll probably need to rebuild your *current* working version before you can attempt to switch the CPU model to 'kvm64'. An Opteron optimized build won't even boot if you set the CPU model to 'kvm64' IIRC.

First, unset CPUTYPE if you have it set and then rebuild _everything_. Rebuild world, kernel, and ports. Once you've confirmed that everything still works (you can reboot, your daemons still come up, etc), you should be able to switch your virtual CPU model to kvm64.

Once your system runs with with kvm64 set, you can build the newer version of FreeBSD that you want to move to. To be safe, try the newer kernel first (as you should in general anyway) to confirm that it's working without surprises.

Seeing as how I found that much more than the kernel and the base system were affected, this could very well be a compiler regression for the Opteron target.

Anyway, hope this helps!
Comment 18 Mateusz Piotrowski freebsd_committer 2018-02-16 02:01:46 UTC
Same on FreeBSD 11.1 amd64. Switching to kvm64 or Opteron_G3 "fixes" this issue for me.
Comment 19 mainland 2018-03-10 00:32:23 UTC
I've attached a patch that fixes this regression in bug #213155.
Comment 20 Andriy Gapon freebsd_committer 2018-03-12 11:29:17 UTC

*** This bug has been marked as a duplicate of bug 213155 ***