Bug 291768 - bhyve won't boot anything on 15.0 - AMD EPYC 9124
Summary: bhyve won't boot anything on 15.0 - AMD EPYC 9124
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 15.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords: regression
: 293147 (view as bug list)
Depends on:
Blocks:
 
Reported: 2025-12-18 11:07 UTC by Marko Cupać
Modified: 2026-09-22 17:08 UTC (History)
20 users (show)

See Also:


Attachments
linux inside bhyve kernel panic (15.63 KB, image/png)
2026-01-20 16:43 UTC, deimos
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marko Cupać 2025-12-18 11:07:56 UTC
Hi,

I can't boot existing or install new VMs on 15-RELEASE-p1 on two identical servers, Supermicro with AMD EPYC 9124 CPUs. Works fine on 14.3-RELEASE-p6 on same hardware.

On another hardware, HP Z Workstation with 13th Gen Intel Core i7-13700K, I can boot and install VMs on both 15-RELEASE-p1 and 14.3-RELEASE-p6.

I use boot environments so I can switch between 14.3 and 15.0 easily.

All my VMs are FreeBSD 14.3 and 15.0, uefi mode, nvme disks:

loader="uefi"
cpu=2
memory=2G
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="UID"
network0_mac="MAC"

VM starts, stays running, but gives nothing on console. Turning on graphics / VNC serves black screen.

Changing loader to bhyvectl shows FreeBSD booting on console but hangs at:

/boot/kernel/kernel text=0x180078 text=0xd97948 text=0x43a46b data=0x180+0xe80 data=0x187590+0x478a70 0x8+0x18e940+0x8+0x1b2b03/
Loading configured modules...
can't find '/boot/entropy'
can't find '/etc/hostid'

Thank you in advance,
Comment 1 Michael Osipov freebsd_committer freebsd_triage 2025-12-18 16:31:58 UTC
Can you bisect to the first broken change?
Comment 2 Marko Cupać 2025-12-18 17:15:04 UTC
(In reply to Michael Osipov from comment #1)

Hi,

I could, but I'm gonna need some instruction on how to do it.

I guess I will need to sync sources from different points in time, build kernel and world from them and attempting to start VMs until I get to the point where they doesn't start anymore. But where do I start? In which increments do I sync newer versions?
Comment 3 Michael Osipov freebsd_committer freebsd_triage 2025-12-18 17:24:24 UTC
(In reply to Marko Cupać from comment #2)

Your understanding is absolutely correct. You can do this with "git bisect", it will perform a binary search on between the commits for you, you just need to do the testing.

Read on "git bisect": https://www.metaltoad.com/blog/beginners-guide-git-bisect-process-elimination

And you start from here:
$ git bisect start release/15.0.0 release/14.3.0-p6
Binäre Suche: eine Merge-Basis muss geprüft werden
[712806fc4b5470eb7d9ce537e3cdf3b386455d86] vfs: retried++ -> retried = true for the boolean
osipovmi@deblndw011x:~/var/Projekte/freebsd/src ((712806fc4b54...)|BISECTING)

It offers me 712806fc4b5470eb7d9ce537e3cdf3b386455d86 as a start between 14.3-p6 and 15.0
Comment 4 Marko Cupać 2025-12-18 18:47:47 UTC
(In reply to Michael Osipov from comment #3)

Thanks for the instruction.

I'm stuck for the time being. Here's what I did:

bectl create BISECT (from 14.3-RELEASE-p6)
bectl mount BISECT
chroot </tmp/BISECTdir>
git clone https://git.FreeBSD.org/src.git /usr/src
cd /usr/src
git bisect start
git bisect bad release/15.0.0
git bisect good release/14.3.0-p6
(got same result as you)
make -j 48 buildworld
make -j 48 buildkernel
make installkernel
(exit chroot)
bectl activate -t BISECT
shutdown -r now

At this point server did restart and is pingable but not accessible over ssh. I will have physical access to it on Monday.

In the meantime if there's something I should've done differently please let me know.
Comment 5 Michael Osipov freebsd_committer freebsd_triage 2025-12-18 19:01:14 UTC
(In reply to Marko Cupać from comment #4)

Do you have lights out management (here OpenBMC) configured?
Comment 6 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2025-12-18 21:52:43 UTC
This won't work.  You can only meaningfully bisect between two commits if one is a direct ancestor of the other, and 14.3 is not an ancestor of 15.0.  Their common ancestor, commit 712806fc4b54, is the last commit before stable/14 was branched in August 2023, so you just tried to boot a 15.0-RELEASE world on a 28-month-old 14.0-CURRENT kernel.
Comment 7 Michael Osipov freebsd_committer freebsd_triage 2025-12-18 22:04:04 UTC
(In reply to Dag-Erling Smørgrav from comment #6)

What would be the advise for users to bisect in general. How to compare two major versions to find the cause? Find the correct commits in main where releases were created?
Comment 8 Marko Cupać 2025-12-18 22:31:24 UTC
(In reply to Dag-Erling Smørgrav from comment #6)

Indeed, I'd say I am definitely not at level required for pinpointing problematic commit. I hoped I'll be asked to perhaps send some verbose logs, set some config variable, or apply some patches to release/15.0.0 source and rebuild.

I understand this is probably specific to my particular hardware and I can't expect more knowledgeable people who would know how to fix this to have it for testing. I'd be glad to help with any testing but I'm gonna need detailed instruction.

I could stay on 14.X until its EoL in November of 2028 and keep my fingers crossed this gets resolved until then. But I guess it is in FreeBSD project's interest to resolve any regressions introduced in 15.X as soon as possible.

Best regards,
Comment 9 Bakul Shah 2025-12-18 23:23:56 UTC
If you are using root on zfs & have enough free space, one suggestion fir git bisect based debugging is to use a separate boot environ for each test build & install.

AFAIK git bisect can work between any two end-points (one good, one bad). They can be on different branches. See for example:

https://nathanchance.dev/posts/working-with-git-bisect/
Comment 10 Michael Osipov freebsd_committer freebsd_triage 2025-12-19 08:00:10 UTC
Marko, can you at least test latest 14-STABLE?
Comment 11 Marko Cupać 2025-12-22 12:59:09 UTC
Hi,

I'm back in the office, hopefully progressing with bisection.

I have successfully started VM on 15.X, from commit 73465bb47bb37e6efdc61827e260a070dda0a0dc.

Tomorrow I will have complete bisect log as well as information about situation with 14-STABLE.

Best regards,
Comment 12 Marko Cupać 2025-12-23 13:45:20 UTC
Hi,

It appears that this one broke it:
https://cgit.freebsd.org/src/commit/?h=releng/15.0&id=9114e15a86949f7d168bf09dbe78a340d5444e80

The one before it works:
https://cgit.freebsd.org/src/commit/?h=releng/15.0&id=32873ecb20aebc5117a849dc9f2a11bd9b364a6b

14-STABLE works.

Setting `vm.pmap.la57=0` in /boot/loader.conf on 15.0-RELEASE-p1 host results in bhyve VMs being able to boot.

Best regards,
Comment 13 Ivan Rozhuk 2025-12-23 14:48:38 UTC
Same for me on AMD 5950x, 2700 and probably others.
Comment 14 Michael Osipov freebsd_committer freebsd_triage 2025-12-23 15:44:59 UTC
(In reply to Marko Cupać from comment #12)

Thank you Marko for the investigation. Let's add the committer to the issue.
Comment 15 Robert Ricci 2025-12-23 16:10:22 UTC
I can confirm that this workaround also fixes the behavior that I reported on the virtualization mailing list:

    https://lists.freebsd.org/archives/freebsd-virtualization/2025-December/003363.html

... so that report can be considered a duplicate of this issue.

Very nice job tracking this down, thank you!
Comment 16 Ivan Rozhuk 2025-12-23 23:12:43 UTC
Small update.

My issue was triggered by cached BHYVE_UEFI.fd from 2022.05: it breaks load all VMs, no matter to vm.pmap.la57 value on virtualization host:
================================
UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (BHYVE, 0x00010000)
map: No mapping found.
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell> 
================================
(VM serial port output)
Comment 17 deimos 2026-01-20 16:43:36 UTC
Created attachment 267333 [details]
linux inside bhyve kernel panic
Comment 18 deimos 2026-01-20 17:23:22 UTC
Hello,

I also have an AMD EPYC 9132 with nearly the same issues. After setting vm.pmap.la57=0 in /boot/loader.conf, I'm able to boot and run some operating systems.

FreeBSD, OpenBSD, and debian 13 trixie seem to work fine inside bhyve. Installing Fedora Workstation 43 and Arch Linux from 2025-11-01 ISO works fine for the installation (based on the 6.17-xx kernel version). Once the system in the VM updates to 6.18.xx kernel version, the kernel panics. Alpine linux 3.2 won't even boot because it uses the 6.18 kernel by default. I've attached a screenshot of the alpine kernel panic when first starting. Both 6.17 and 6.18 report RDSEED2 broken, but only 6.18 seems to fail to boot.

As a test, I'm' able to run everything without issue on a Ryzen 9 system (same FreeBSD 15/bhyve/vm configuration with linux-zvol). 

I have not yet had time to attach gdb to the failing bhyve VMs to figure out what's happening on the EPYC system.
Comment 19 deimos 2026-01-20 18:50:08 UTC
(In reply to deimos from comment #18)

I confirm the RDSEED32 issue is a CPU bug in the Zen5 architecture[0]. Bhyve is correctly passing the CPU information through to the VM. It's the linux kernel that is unable to handle it correctly inside the VM[1]. 

[0] https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7055.html
[1] https://www.linuxjournal.com/content/amd-confirms-zen-5-rng-flaw-when-random-isnt-random-enough
Comment 20 D. Sun 2026-02-07 23:21:45 UTC
(In reply to Ivan Rozhuk from comment #16)

I have the same problem currently, setting vm.pmap.la57=0 in /boot/loader.conf and rebooting did not make any difference.  In my case, the CPU is a 9950X.
Comment 21 Tom 2026-02-13 00:44:21 UTC
*** Bug 293147 has been marked as a duplicate of this bug. ***
Comment 22 Tom 2026-02-13 00:58:22 UTC
I had the same issue with the same processor and the workaround to set vm.pmap.la57=0 worked for me.
Comment 23 Tom 2026-02-13 01:23:31 UTC
One oddity I've noticed that might be related: A bunch of CPU feature flags are missing inside the guest including LA57. I've reproduced this with both a FreeBSD guest and a Linux guest.

The flags that are present on the FreeBSD host but not the FreeBSD guest: INVPCID,PQM,PQE,ADX,AVX512IFMA,CLFLUSHOPT,CLWB,AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,LA57,RDPID

I compiled a test C program that uses AVX512IFMA instructions, and ran it inside the guest virtual machine. It ran successfully, proving the instructions work. It's just the flags are missing.

I stumbled upon this because it breaks compiling some Qt software under Linux with -march=znver4 despite the CPU being a Zen 4 CPU.

FreeBSD Host:
```
$ cat /var/run/dmesg.boot| grep -i "structured extended"
  Structured Extended Features=0xf1bf97a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,AVX512CD,SHA,AVX512BW,AVX512VL>
  Structured Extended Features2=0x415fde<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,LA57,RDPID>
```

FreeBSD Guest:
```
$ cat /var/run/dmesg.boot| grep -i "structured extended"
  Structured Extended Features=0xf01703a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,AVX512F,AVX512DQ,RDSEED,SMAP,AVX512CD,SHA,AVX512BW,AVX512VL>
  Structured Extended Features2=0x600<VAES,VPCLMULQDQ>
```
Comment 24 erwin.freebsd-bugzilla 2026-03-10 18:34:24 UTC
FYI: vm.pmap.la57=0 helped with my 7965WX, I can boot Windows now. With the default it hangs even before the uefi is loaded.

I don't have more than 256TB ram so disabling is not yet a dealbreaker. Disabling by default might be a good idea for now.
Comment 25 Konstantin Belousov freebsd_committer freebsd_triage 2026-03-21 01:12:27 UTC
There must be some log by kernel, by bhyve process itself, or might be
something in vm that should provide a clue what is going on.
Comment 26 ShengYi Hung freebsd_committer freebsd_triage 2026-03-22 02:38:28 UTC
(In reply to Tom from comment #23)

I think avx512 is a separate problem. But could you please try the patch here?

https://reviews.freebsd.org/D56021
Comment 27 ShengYi Hung freebsd_committer freebsd_triage 2026-03-22 02:39:47 UTC
(In reply to Tom from comment #23)

Also, is it possible for me to access such machine? I mean the machine with LA57 feature.
Comment 28 Tom 2026-03-22 15:06:24 UTC
(In reply to ShengYi Hung from comment #26)

Thank you for the patch! I'll try that out.

(In reply to ShengYi Hung from comment #27)

Yep. I'll reach out to your e-mail to discuss the details.
Comment 29 ShengYi Hung freebsd_committer freebsd_triage 2026-04-08 15:12:58 UTC
(In reply to deimos from comment #19)
For RDSEED, could you please try https://reviews.freebsd.org/D56312 and its parent revision?
Comment 30 D. Sun 2026-04-11 23:59:41 UTC
Turns out that setting vm.pmap.la57=0 didn't work because I also had locked edk2-bhyve-g202202_10 at some point in the past.  Oops!  Updating to edk2-bhyve-g202508 resolved this problem for me.
Comment 31 deimos 2026-04-14 19:31:23 UTC
(In reply to ShengYi Hung from comment #29)

Unfortunately, the EPYC server here is now a production system so I can't apply custom patches. I'll see if I can get a hold of another EPYC and test it.
Comment 32 Jeff Griffith 2026-04-16 03:36:46 UTC
I am also affected by this bug. CPU is AMD EPYC 8024P. The suggested workaround of setting vm.pmap.la57=0 in /boot/loader.conf worked for me.
Comment 33 Jeff Griffith 2026-04-21 03:05:30 UTC
(In reply to ShengYi Hung from comment #29)

I tried this against the src from releng/15.0 and didn't have any luck (issue persists without 'vm.pmap.la57=0' in /boot/loader.conf. Do I need to try the patches against a different branch?
Comment 34 Kyle Evans freebsd_committer freebsd_triage 2026-04-21 05:26:40 UTC
I feel obligated to ask out of ignorance; given the coincidence with EPYC models, is there a chance that LA57 makes vmm more likely to exercise INVLPGB[0] which surfaces as... this?

[0] https://cgit.freebsd.org/src/commit/?id=1b8e5c02f5c07521129e06ff8ab7c660238fd75c
Comment 35 ShengYi Hung freebsd_committer freebsd_triage 2026-04-29 16:13:28 UTC
(In reply to deimos from comment #31)
Emm, I cannot reproduce on my 9950x workstation. The alpine linux with 6.18.3 boot sucessfully.
Comment 36 Joel Bodenmann freebsd_committer freebsd_triage 2026-05-13 17:48:01 UTC
vm.pmap.la57=0 allowed me to boot Windows guests again (CPU is threadripper 7960X).
Comment 37 deimos 2026-06-19 18:11:35 UTC
(In reply to ShengYi Hung from comment #35)

Hello. The 9950 seems to be a Ryzen model number. Is this correct?

If so, I expect this to be the case because the bug only appears on EPYC 9000 series processors.

My Ryzen CPUs run current linux kernels in bhyve in 15.0 just fine. 

I have not yet tried the 15.1 release nor the linux 7.x kernels in bhyve.
Comment 38 ShengYi Hung freebsd_committer freebsd_triage 2026-06-30 21:16:40 UTC
https://reviews.freebsd.org/D57978

Please check this.
Comment 39 Pavel Radvan 2026-07-18 15:01:31 UTC
Hi.
I could confirm that setting vm.pmap.la57=0 in boot/loader.conf on HP server with EPYC CPU helped to have bhyve working.
I have there installed FreeBSD 15.1 Stable and it runs from half of April to today without any crash.
CPU is:
hw.machine: amd64
hw.model: AMD EPYC 9115 16-Core Processor
hw.ncpu: 32
hw.machine_arch: amd64

Server is HP Proliant DL325g11 with latest firmware (from April 2026)
I have ther 2 VMs now with Windows Server 2025 Standard and it is running nicely and is stable.

Next step will be to switch from stable to release.
Comment 40 ShengYi Hung freebsd_committer freebsd_triage 2026-07-19 13:41:46 UTC
(In reply to Pavel Radvan from comment #39)
Hi, I am wondering if you can help me test with my patch as I don't have epyc 9xxx series machine. It helps me get rid of the la57 quirk in here.
Comment 41 Pavel Radvan 2026-07-19 18:07:44 UTC
(In reply to ShengYi Hung from comment #40)
Hi.
I am not much using Linux. So I do not know.
I have several servers with Epyc, but are used in my customers networks.
I could test on one of them, because it is school which has now holidays and there is also planned maintenance and changes in configurations.
So there is good time to do some tests.
Where is possible to get that patch ?
Pavel
Comment 42 ShengYi Hung freebsd_committer freebsd_triage 2026-07-20 13:43:50 UTC
(In reply to Pavel Radvan from comment #41)

Hi, the patch should fix not only on Linux but also on FreeBSD guest that relies your host to set la57=0 quirk. The patch is here: https://reviews.freebsd.org/D57978
Comment 43 Pavel Radvan 2026-08-14 20:40:13 UTC
(In reply to ShengYi Hung from comment #42)
Hi.
I have looked on patch today - i did not have time sooner.
I understand I have to patch basic system or it could be tested in VM on bhyve ? I am using BVCP which is now doing bhyve management.
Since I have server in some preparation I could test, but I simply do not much want to destroy whole system.
Could you recommend some guide or steps to apple patch to have some info for you which could help you ?
Pavel
Comment 44 commit-hook freebsd_committer freebsd_triage 2026-09-11 13:02:27 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ef26f64c11220ea303de16d088c585f9d2908b0f

commit ef26f64c11220ea303de16d088c585f9d2908b0f
Author:     ShengYi Hung <aokblast@FreeBSD.org>
AuthorDate: 2026-09-11 08:22:39 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-09-11 13:01:44 +0000

    amd64/pmap: Return la57 in amd nested page

    AMD doesn't encode page depth in eptp. As a result, the page level is
    decided by the host la57 value. Without this, it uses 4 level page and
    therefore cause machine enable la57 have garbage page translation.

    PR:             291768
    Reviewed by:    kib
    Tested by:      Antranig Vartanian <antranigv@freebsd.am>
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D57978

 sys/amd64/amd64/pmap.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
Comment 45 commit-hook freebsd_committer freebsd_triage 2026-09-22 14:50:10 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=9d9063e123daa9dab485f01d9087b5c7b984f6e1

commit 9d9063e123daa9dab485f01d9087b5c7b984f6e1
Author:     ShengYi Hung <aokblast@FreeBSD.org>
AuthorDate: 2026-09-11 08:22:39 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-09-22 13:01:16 +0000

    amd64/pmap: Return la57 in amd nested page

    AMD doesn't encode page depth in eptp. As a result, the page level is
    decided by the host la57 value. Without this, it uses 4 level page and
    therefore cause machine enable la57 have garbage page translation.

    PR:             291768
    Reviewed by:    kib
    Tested by:      Antranig Vartanian <antranigv@freebsd.am>
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D57978

    (cherry picked from commit ef26f64c11220ea303de16d088c585f9d2908b0f)

 sys/amd64/amd64/pmap.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
Comment 46 commit-hook freebsd_committer freebsd_triage 2026-09-22 17:08:29 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ccaeaa843055442fd73020f6efce2a23fdfc0e84

commit ccaeaa843055442fd73020f6efce2a23fdfc0e84
Author:     ShengYi Hung <aokblast@FreeBSD.org>
AuthorDate: 2026-09-11 08:22:39 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-09-22 17:06:52 +0000

    amd64/pmap: Return la57 in amd nested page

    AMD doesn't encode page depth in eptp. As a result, the page level is
    decided by the host la57 value. Without this, it uses 4 level page and
    therefore cause machine enable la57 have garbage page translation.

    PR:             291768
    Reviewed by:    kib
    Tested by:      Antranig Vartanian <antranigv@freebsd.am>
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D57978

    (cherry picked from commit ef26f64c11220ea303de16d088c585f9d2908b0f)

 sys/amd64/amd64/pmap.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)