Hi, I can't boot existing or install new VMs on 15-RELEASE-p1 on two identical servers, Supermicro with AMD EPYC 9124 CPUs. Works fine on 14.3-RELEASE-p6 on same hardware. On another hardware, HP Z Workstation with 13th Gen Intel Core i7-13700K, I can boot and install VMs on both 15-RELEASE-p1 and 14.3-RELEASE-p6. I use boot environments so I can switch between 14.3 and 15.0 easily. All my VMs are FreeBSD 14.3 and 15.0, uefi mode, nvme disks: loader="uefi" cpu=2 memory=2G network0_type="virtio-net" network0_switch="public" disk0_type="nvme" disk0_name="disk0.img" uuid="UID" network0_mac="MAC" VM starts, stays running, but gives nothing on console. Turning on graphics / VNC serves black screen. Changing loader to bhyvectl shows FreeBSD booting on console but hangs at: /boot/kernel/kernel text=0x180078 text=0xd97948 text=0x43a46b data=0x180+0xe80 data=0x187590+0x478a70 0x8+0x18e940+0x8+0x1b2b03/ Loading configured modules... can't find '/boot/entropy' can't find '/etc/hostid' Thank you in advance,
Can you bisect to the first broken change?
(In reply to Michael Osipov from comment #1) Hi, I could, but I'm gonna need some instruction on how to do it. I guess I will need to sync sources from different points in time, build kernel and world from them and attempting to start VMs until I get to the point where they doesn't start anymore. But where do I start? In which increments do I sync newer versions?
(In reply to Marko Cupać from comment #2) Your understanding is absolutely correct. You can do this with "git bisect", it will perform a binary search on between the commits for you, you just need to do the testing. Read on "git bisect": https://www.metaltoad.com/blog/beginners-guide-git-bisect-process-elimination And you start from here: $ git bisect start release/15.0.0 release/14.3.0-p6 Binäre Suche: eine Merge-Basis muss geprüft werden [712806fc4b5470eb7d9ce537e3cdf3b386455d86] vfs: retried++ -> retried = true for the boolean osipovmi@deblndw011x:~/var/Projekte/freebsd/src ((712806fc4b54...)|BISECTING) It offers me 712806fc4b5470eb7d9ce537e3cdf3b386455d86 as a start between 14.3-p6 and 15.0
(In reply to Michael Osipov from comment #3) Thanks for the instruction. I'm stuck for the time being. Here's what I did: bectl create BISECT (from 14.3-RELEASE-p6) bectl mount BISECT chroot </tmp/BISECTdir> git clone https://git.FreeBSD.org/src.git /usr/src cd /usr/src git bisect start git bisect bad release/15.0.0 git bisect good release/14.3.0-p6 (got same result as you) make -j 48 buildworld make -j 48 buildkernel make installkernel (exit chroot) bectl activate -t BISECT shutdown -r now At this point server did restart and is pingable but not accessible over ssh. I will have physical access to it on Monday. In the meantime if there's something I should've done differently please let me know.
(In reply to Marko Cupać from comment #4) Do you have lights out management (here OpenBMC) configured?
This won't work. You can only meaningfully bisect between two commits if one is a direct ancestor of the other, and 14.3 is not an ancestor of 15.0. Their common ancestor, commit 712806fc4b54, is the last commit before stable/14 was branched in August 2023, so you just tried to boot a 15.0-RELEASE world on a 28-month-old 14.0-CURRENT kernel.
(In reply to Dag-Erling Smørgrav from comment #6) What would be the advise for users to bisect in general. How to compare two major versions to find the cause? Find the correct commits in main where releases were created?
(In reply to Dag-Erling Smørgrav from comment #6) Indeed, I'd say I am definitely not at level required for pinpointing problematic commit. I hoped I'll be asked to perhaps send some verbose logs, set some config variable, or apply some patches to release/15.0.0 source and rebuild. I understand this is probably specific to my particular hardware and I can't expect more knowledgeable people who would know how to fix this to have it for testing. I'd be glad to help with any testing but I'm gonna need detailed instruction. I could stay on 14.X until its EoL in November of 2028 and keep my fingers crossed this gets resolved until then. But I guess it is in FreeBSD project's interest to resolve any regressions introduced in 15.X as soon as possible. Best regards,
If you are using root on zfs & have enough free space, one suggestion fir git bisect based debugging is to use a separate boot environ for each test build & install. AFAIK git bisect can work between any two end-points (one good, one bad). They can be on different branches. See for example: https://nathanchance.dev/posts/working-with-git-bisect/
Marko, can you at least test latest 14-STABLE?
Hi, I'm back in the office, hopefully progressing with bisection. I have successfully started VM on 15.X, from commit 73465bb47bb37e6efdc61827e260a070dda0a0dc. Tomorrow I will have complete bisect log as well as information about situation with 14-STABLE. Best regards,
Hi, It appears that this one broke it: https://cgit.freebsd.org/src/commit/?h=releng/15.0&id=9114e15a86949f7d168bf09dbe78a340d5444e80 The one before it works: https://cgit.freebsd.org/src/commit/?h=releng/15.0&id=32873ecb20aebc5117a849dc9f2a11bd9b364a6b 14-STABLE works. Setting `vm.pmap.la57=0` in /boot/loader.conf on 15.0-RELEASE-p1 host results in bhyve VMs being able to boot. Best regards,
Same for me on AMD 5950x, 2700 and probably others.
(In reply to Marko Cupać from comment #12) Thank you Marko for the investigation. Let's add the committer to the issue.
I can confirm that this workaround also fixes the behavior that I reported on the virtualization mailing list: https://lists.freebsd.org/archives/freebsd-virtualization/2025-December/003363.html ... so that report can be considered a duplicate of this issue. Very nice job tracking this down, thank you!
Small update. My issue was triggered by cached BHYVE_UEFI.fd from 2022.05: it breaks load all VMs, no matter to vm.pmap.la57 value on virtualization host: ================================ UEFI Interactive Shell v2.2 EDK II UEFI v2.70 (BHYVE, 0x00010000) map: No mapping found. Press ESC in 1 seconds to skip startup.nsh or any other key to continue. Shell> ================================ (VM serial port output)
Created attachment 267333 [details] linux inside bhyve kernel panic
Hello, I also have an AMD EPYC 9132 with nearly the same issues. After setting vm.pmap.la57=0 in /boot/loader.conf, I'm able to boot and run some operating systems. FreeBSD, OpenBSD, and debian 13 trixie seem to work fine inside bhyve. Installing Fedora Workstation 43 and Arch Linux from 2025-11-01 ISO works fine for the installation (based on the 6.17-xx kernel version). Once the system in the VM updates to 6.18.xx kernel version, the kernel panics. Alpine linux 3.2 won't even boot because it uses the 6.18 kernel by default. I've attached a screenshot of the alpine kernel panic when first starting. Both 6.17 and 6.18 report RDSEED2 broken, but only 6.18 seems to fail to boot. As a test, I'm' able to run everything without issue on a Ryzen 9 system (same FreeBSD 15/bhyve/vm configuration with linux-zvol). I have not yet had time to attach gdb to the failing bhyve VMs to figure out what's happening on the EPYC system.
(In reply to deimos from comment #18) I confirm the RDSEED32 issue is a CPU bug in the Zen5 architecture[0]. Bhyve is correctly passing the CPU information through to the VM. It's the linux kernel that is unable to handle it correctly inside the VM[1]. [0] https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7055.html [1] https://www.linuxjournal.com/content/amd-confirms-zen-5-rng-flaw-when-random-isnt-random-enough
(In reply to Ivan Rozhuk from comment #16) I have the same problem currently, setting vm.pmap.la57=0 in /boot/loader.conf and rebooting did not make any difference. In my case, the CPU is a 9950X.
*** Bug 293147 has been marked as a duplicate of this bug. ***
I had the same issue with the same processor and the workaround to set vm.pmap.la57=0 worked for me.
One oddity I've noticed that might be related: A bunch of CPU feature flags are missing inside the guest including LA57. I've reproduced this with both a FreeBSD guest and a Linux guest. The flags that are present on the FreeBSD host but not the FreeBSD guest: INVPCID,PQM,PQE,ADX,AVX512IFMA,CLFLUSHOPT,CLWB,AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,LA57,RDPID I compiled a test C program that uses AVX512IFMA instructions, and ran it inside the guest virtual machine. It ran successfully, proving the instructions work. It's just the flags are missing. I stumbled upon this because it breaks compiling some Qt software under Linux with -march=znver4 despite the CPU being a Zen 4 CPU. FreeBSD Host: ``` $ cat /var/run/dmesg.boot| grep -i "structured extended" Structured Extended Features=0xf1bf97a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,AVX512CD,SHA,AVX512BW,AVX512VL> Structured Extended Features2=0x415fde<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,LA57,RDPID> ``` FreeBSD Guest: ``` $ cat /var/run/dmesg.boot| grep -i "structured extended" Structured Extended Features=0xf01703a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,AVX512F,AVX512DQ,RDSEED,SMAP,AVX512CD,SHA,AVX512BW,AVX512VL> Structured Extended Features2=0x600<VAES,VPCLMULQDQ> ```
FYI: vm.pmap.la57=0 helped with my 7965WX, I can boot Windows now. With the default it hangs even before the uefi is loaded. I don't have more than 256TB ram so disabling is not yet a dealbreaker. Disabling by default might be a good idea for now.