Created attachment 161834 [details] bhyve exception log NetBSD 7.0 causes bhyve to exit with an exception, but only on AMD processors. Intel works fine. uname -a: FreeBSD olgeni 10.2-STABLE FreeBSD 10.2-STABLE #5 r288925: Tue Oct 6 20:48:48 CEST 2015 root@olgeni:/usr/obj/usr/src/sys/KERNEL amd64 Processor: CPU: AMD Phenom(tm) II X4 965 Processor (3411.31-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x100f43 Family=0x10 Model=0x4 Stepping=3 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x802009<SSE3,MON,CX16,POPCNT> AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!> AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT> SVM: NP,NRIP,NAsids=64 TSC: P-state invariant How to repeat - download NetBSD-7.0-amd64.iso, then try this: ====================================================================== truncate -s8G netbsd.img cat > netbsd.map << EOF (hd0) netbsd.img (cd0) NetBSD-7.0-amd64.iso EOF cat > netbsd.grub << EOF knetbsd -h -r cd0a /netbsd boot EOF sudo grub-bhyve -r cd0 -m netbsd.map -M 1024 netbsd < netbsd.grub sudo bhyve -c 1 -m 1024M -H -P -A -W -w \ -l com1,stdio \ -s 0:0,hostbridge \ -s 1:0,lpc \ -s 2:0,virtio-net,tap0 \ -s 3,ahci-cd,NetBSD-7.0-amd64.iso \ -s 4,virtio-blk,netbsd.img netbsd ====================================================================== You'll get something similar to this: root on cd0a dumps on cd0b root file system type: cd9660 warning: no /dev/console vm exit[0] reason SVM rip 0xffffffff80229f5f inst_length 2 exitcode 0x7b exitinfo1 0x3f8021c exitinfo2 0xffffffff80229f61
Canonicalize assignment.
Tested again. 11.0-RC2 with NetBSD 7.0.1: cd0 at atapibus0 drive 0: <BHYVE SATA DVD ROM, BHYVE-445C-CB8B-8A6, 001> cdrom removable pad0: outputs: 44100Hz, 16-bit, stereo audio0 at pad0: half duplex, playback, capture boot device: cd0 cd0(ahcisata0:0:0): Check Condition on CDB: 0x46 00 00 00 00 00 00 00 08 00 SENSE KEY: Illegal Request ASC/ASCQ: Invalid Command Operation Code root on cd0a dumps on cd0b root file system type: cd9660 warning: no /dev/console Abort trap From the vm-bhyve log: Sep 12 10:20:16: initialising Sep 12 10:20:16: [loader: grub] Sep 12 10:20:16: [uefi: no] Sep 12 10:20:16: [cpu: 1] Sep 12 10:20:16: [memory: 512M] Sep 12 10:20:16: [hostbridge: standard] Sep 12 10:20:16: [com ports: com1] Sep 12 10:20:16: [uuid: 9a8e5695-78c1-11e6-a88c-bcaec55be5e5] Sep 12 10:20:16: [utctime: no] Sep 12 10:20:16: [debug mode: no] Sep 12 10:20:16: [primary disk: disk0.img] Sep 12 10:20:16: [primary disk dev: file] Sep 12 10:20:16: generated static mac 58:9c:fc:0b:b5:ee (based on 'netbsd:0:1473668416:0') Sep 12 10:20:16: initialising network device tap0 Sep 12 10:20:16: adding tap0 -> bridge1 (public) Sep 12 10:20:16: booting Sep 12 10:20:16: create file /storage/vm-bhyve/netbsd/device.map Sep 12 10:20:16: -> (hd0) /storage/vm-bhyve/netbsd/disk0.img Sep 12 10:20:16: -> (cd0) /storage/vm-bhyve/.iso/NetBSD-7.0.1-amd64.iso Sep 12 10:20:16: create file /storage/vm-bhyve/netbsd/grub.cfg Sep 12 10:20:16: -> timeout=3 Sep 12 10:20:16: -> menuentry 'netbsd (bhyve install)' { Sep 12 10:20:16: -> root=cd0 Sep 12 10:20:16: -> knetbsd -h -r cd0a /netbsd Sep 12 10:20:16: -> } Sep 12 10:20:16: /usr/local/sbin/grub-bhyve -m /storage/vm-bhyve/netbsd/device.map -M 512M -r host -d /storage/vm-bhyve/netbsd netbsd Sep 12 10:20:20: [bhyve options: -c 1 -m 512M -AHP -U 9a8e5695-78c1-11e6-a88c-bcaec55be5e5] Sep 12 10:20:20: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/storage/vm-bhyve/netbsd/disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:b5:ee] Sep 12 10:20:20: [bhyve console: -l com1,stdio] Sep 12 10:20:20: [bhyve iso device: -s 3:0,ahci-cd,/storage/vm-bhyve/.iso/NetBSD-7.0.1-amd64.iso] Sep 12 10:20:20: starting bhyve (run 1) Sep 12 10:20:27: bhyve exited with status 134 Sep 12 10:20:27: destroying network device tap0 Sep 12 10:20:27: stopped
I can confirm this with identical symptoms, on different AMD hardware, with more recent versions: uname -a FreeBSD bhyvehost 11.0-RELEASE-p8 FreeBSD 11.0-RELEASE-p8 #0: Wed Feb 22 06:12:04 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 dmesg [...] CPU: AMD E-350 Processor (1596.63-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x500f10 Family=0x14 Model=0x1 Stepping=0 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x802209<SSE3,MON,SSSE3,CX16,POPCNT> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x35ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,IBS,SKINIT,WDT> [...] Linux installations and live CDs are working fine. Steps: zfs create -s -V 5G bhyveguests/netbsd ==> install.map <== (hd0) /dev/zvol/bhyveguests/netbsd (cd0) /scratch/NetBSD-7.1/amd64/boot-com.iso ==> install.sh <== grub-bhyve -m install.map -r cd0 -M 512 netbsdvm << EOF knetbsd -h -r cd0a (cd0)/netbsd boot EOF bhyve -c 1 -m 512M -H -P -A \ -l com1,stdio \ -s 0:0,hostbridge \ -s 1:0,lpc -s 2:0,virtio-net,tap0 \ -s 3,ahci-cd,/scratch/NetBSD-7.1/amd64/boot-com.iso \ -s 4,virtio-blk,/dev/zvol/bhyveguests/netbsd \ netbsdvm ./install.sh dies with: [...] cd0 at atapibus0 drive 0: <BHYVE SATA DVD ROM, BHYVE-5932-F7DC-D40, 001> cdrom removable pad0: outputs: 44100Hz, 16-bit, stereo audio0 at pad0: half duplex, playback, capture boot device: cd0 cd0(ahcisata0:0:0): Check Condition on CDB: 0x46 00 00 00 00 00 00 00 08 00 SENSE KEY: Illegal Request ASC/ASCQ: Invalid Command Operation Code root on cd0a dumps on cd0b root file system type: cd9660 warning: no /dev/console vm exit[0] reason SVM rip 0xffffffff8022a72f inst_length 2 exitcode 0x7b exitinfo1 0x3f8021c exitinfo2 0xffffffff8022a731
I was able to get an install/boot working fine on a r317273 CURRENT Ryzen system with 1 and 4 vCPUs using the NetBSD 7.1 boot-com ISO.
Quick update - did not notice anything like this in a long time.
Closing per reporter's no longer experiencing the issue.
Can someone reopen this bug? I saw this issue, too, after 9+ years. Host: FreeBSD/amd64 14.2-RELEASE-p2 (GENERIC) FUJITSU PRIMERGY MX130S2 CPU: AMD Sempron(tm) 145 Processor (2800.14-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x100f63 Family=0x10 Model=0x6 Stepping=3 Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> Features2=0x802009<SSE3,MON,CX16,POPCNT> AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!> AMD Features2=0x837fd<LAHF,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,NodeId> SVM: NP,NRIP,NAsids=64 TSC: P-state invariant dmesg: https://github.com/n12i/bsd/blob/main/dmesg.boot-mx130s2.txt Setup: # pkg install sysutils/vm-bhyve # pkg install sysutils/grub2-bhyve $ grep '^vm' /etc/rc.conf.local vm_enable="YES" vm_dir="zfs:zroot/bhyve" Results: # vm create netbsd # vm install -f netbsd NetBSD-10.1-amd64.iso ... [ 2.6641566] swwdog0: software watchdog initialized [ 2.6641566] boot device: cd0 [ 2.6641566] cd0(ahcisata0:0:0): Check Condition on CDB: 0x46 00 00 00 00 00 00 00 08 00 [ 2.6727995] SENSE KEY: Illegal Request [ 2.6727995] ASC/ASCQ: Invalid Command Operation Code [ 2.6727995] root on cd0a dumps on cd0b [ 2.6727995] cd0(ahcisata0:0:0): Check Condition on CDB: 0x46 00 00 00 00 00 00 00 08 00 [ 2.6827311] SENSE KEY: Illegal Request [ 2.6827311] ASC/ASCQ: Invalid Command Operation Code [ 2.6827311] root file system type: cd9660 [ 2.6827311] kern.module.path=/stand/amd64/10.1/modules [ 2.7624147] entropy: best effort Abort trap
Reopening as per request
Booting and installing NetBSD on an AMD Ryzen 7 7840U worked fine. Looking at the error being returned: exitcode 0x7b shows it was an IO exit exitinfo1 can be decoded from the AMD Programmers Manual Volume 2 Chapter 15.10.2 0x3f8021c shows REP OUTS to port 0x3f8 (COM1 serial port) exitinfo2 is the address of the instruction following the OUTS The IO is handled in svm_handle_io() https://github.com/freebsd/freebsd-src/blob/main/sys/amd64/vmm/amd/svm.c#L785 There is a check for the SVM DecodeAssist feature here: https://github.com/freebsd/freebsd-src/blob/main/sys/amd64/vmm/amd/svm.c#L808 The system information listed in this bug report do not have DecodeAssist listed. The 7840U I used shows these SVM features: SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 By stubbing that call to decode_assist() to return 0, I was able to get the same error to occur on my hardware. Looking back in the history, the check svm_handlt_io() was put in via this commit and commit message: https://github.com/freebsd/freebsd-src/commit/bbadcde418c29fe312a1d9d39b768735f285efbd "Don't attempt to handle INS/OUTS VM-exits unless the DecodeAssist capabilityis available. The effective segment field in EXITINFO1 is not valid withoutthis capability." You could try modifying the code to not do the decode_assist() check and see if it works on your system. Or try turning on graphics for the VM to see if it will boot using a graphics console.
I modified sys/amd64/vmm/amd/svm.c, but got same "Abort trap". --- a/sys/amd64/vmm/amd/svm.c +++ b/sys/amd64/vmm/amd/svm.c @@ -142,8 +142,8 @@ flush_by_asid(void) static __inline int decode_assist(void) { - - return (svm_feature & AMD_CPUID_SVM_DECODE_ASSIST); + return 1; + /* return (svm_feature & AMD_CPUID_SVM_DECODE_ASSIST); */ } static void The command lines: # make -C /usr/src/sys/modules/vmm obj depend all install # kldunload vmm # kldload vmm # vm install -f netbsd-10 NetBSD-10.99.12-amd64-dvd.iso # cat netbsd-10/bhyve.log vm exit[0] reason SVM rip 0xffffffff80234d9f inst_length 2 exitcode 0x7b exitinfo1 0x3f8021c exitinfo2 0xffffffff80234da1 I'll try graphics="yes" later. Thanks.
(In reply to NAKAJI Hiroyuki from comment #10) I would not expect that code change to give the exact same error. Given your build steps, vmm.ko was likely installed into /boot/modules. The kldload would likely look in from /boot/kernel first (look at sysctl kern.module_path) loading the existing one. You should kldload it with a fully qualified path to make sure you are loading the new one.
(In reply to Mark Peek from comment #11) Exactly, I had wrong vmm.ko loaded. Thanks for finding my mistake. $ kldstat -v | grep vmm 26 1 0xffffffff83400000 33e438 vmm.ko (/boot/kernel/vmm.ko) I reloaded vmm.ko and got the message of NetBSD's installer!! $ sudo kldunload vmm $ sudo kldload /boot/modules/vmm.ko $ kldstat -v | grep vmm 26 1 0xffffffff83400000 33c0a8 vmm.ko (/boot/modules/vmm.ko) $ sudo vm install -f netbsd-10 NetBSD-10.99.22-amd64.iso ... [ 3.1743834] entropy: best effort Created tmpfs /dev (1835008 byte, 3552 inodes) You are using a serial console, we do not know your terminal emulation. Please select one, typical values are: vt100 ansi xterm Terminal type (just hit ENTER for 'vt220'): My code change was so ad-hoc, and I'm not sure about the better way.
A "me too" on this. I'm using a quite old Lenovo Ideapad Z575 and vm-bhyve. The SVM error is slightly different, it looks like EDK2 is trying to access 0x511 (Firmware port?). I used a different workaround: update bhyve(8) to log and ignore SVM errors if environment variable "_BHYVE_AMD_A8_ENABLE_" is set. ==== head /var/run/dmesg.boot ===== Copyright (c) 1992-2025 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 15.0-CURRENT #4 main-n276261-297473069610-dirty: Mon Apr 7 12:16:25 EDT 2025 root@dell7390:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) VT(vga): resolution 640x480 CPU: AMD A8-3500M APU with Radeon(tm) HD Graphics (1497.33-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x300f10 Family=0x12 Model=0x1 Stepping=0 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x802009<SSE3,MON,CX16,POPCNT> AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!> AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT> SVM: NP,NRIP,NAsids=64 TSC: P-state invariant, performance statistics ===== bhyve.log ===== vm exit[0] reason SVM rip 0x000000007fa6e480 inst_length 2 exitcode 0x7b exitinfo1 0x511021d exitinfo2 0x7fa6e482 ==== usr.sbin/bhyve/amd64/vmexit.c patch ==== diff --git a/usr.sbin/bhyve/amd64/vmexit.c b/usr.sbin/bhyve/amd64/vmexit.c index 944f5de34645..94dde19eb931 100644 --- a/usr.sbin/bhyve/amd64/vmexit.c +++ b/usr.sbin/bhyve/amd64/vmexit.c @@ -288,6 +288,10 @@ vmexit_svm(struct vmctx *ctx __unused, struct vcpu *vcpu, struct vm_run *vmrun) EPRINTLN("\texitcode\t%#lx", vme->u.svm.exitcode); EPRINTLN("\texitinfo1\t%#lx", vme->u.svm.exitinfo1); EPRINTLN("\texitinfo2\t%#lx", vme->u.svm.exitinfo2); + if (getenv("_BHYVE_AMD_A8_ENABLE_")) { /* workaround for Lenovo Ideapad Z575 */ + EPRINTLN("\tignoring\t_BHYVE_AMD_A8_ENABLE_ defined"); + return (VMEXIT_CONTINUE); + } return (VMEXIT_ABORT); }
(In reply to Keith White from comment #13) I'd be curious to know if the workaround in #10 would work for you as well. Did you happen to try that change at all?
No, I had not tried #10. I still wanted to see any errors and to be able to turn any workaround on/off. Thanks for the suggestion, though! I tried a modified version of #10 that provides a hw.vmm.svm.disable_decode_assist sysctl that optionally disables the decode_assist() check. This too works for me. A long-winded way of saying "yes" #10 also works for me. diff --git a/sys/amd64/vmm/amd/svm.c b/sys/amd64/vmm/amd/svm.c index 6c16daaa47c2..63f559037687 100644 --- a/sys/amd64/vmm/amd/svm.c +++ b/sys/amd64/vmm/amd/svm.c @@ -122,6 +122,11 @@ static uint32_t nasid; SYSCTL_UINT(_hw_vmm_svm, OID_AUTO, num_asids, CTLFLAG_RDTUN, &nasid, 0, "Number of ASIDs supported by this processor"); +/* AMD A8 has no decode_assist, allow disabling the check */ +static int disable_decode_assist; +SYSCTL_INT(_hw_vmm_svm, OID_AUTO, disable_decode_assist, CTLFLAG_RWTUN, + &disable_decode_assist, 0, "Set to 1 to disable the decode_assist check"); + /* Current ASID generation for each host cpu */ static struct asid asid[MAXCPU]; @@ -146,7 +151,7 @@ static __inline int decode_assist(void) { - return (svm_feature & AMD_CPUID_SVM_DECODE_ASSIST); + return ((svm_feature & AMD_CPUID_SVM_DECODE_ASSIST) || disable_decode_assist); } static void
(In reply to Keith White from comment #15) This change worked for me. Thanks!
Thank you for the confirmations and the sysctl patch. I'll see about getting it out for a code review before committing it.