Bug 258776

Summary: 13.0-RELEASE Installation media kernel panic under Qemu/KVM
Product: Base System Reporter: Yann Droneaud <yann>
Component: kernAssignee: freebsd-emulation (Nobody) <emulation>
Status: Closed Not A Bug    
Severity: Affects Only Me CC: cynix
Priority: ---    
Version: 13.0-RELEASE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
kernel.log
none
kernel.log (verbose)
none
libvirt guest description
none
kernel log as guest with Epyc as CPU model
none
kernel log, booted on host computer (baremetal) instead of being virtualized none

Description Yann Droneaud 2021-09-29 10:23:47 UTC
Created attachment 228242 [details]
kernel.log

Booting amd64/x86_64 from 13.0-RELEASE .iso under libvirtd/QEMU/KVM/Linux always trigger the following panic:

  Fatal trap 12: page fault while in kernel mode
  cpuid = 0; apic id = 00
  fault virtual address	= 0xfffffe004adb50c0
  fault code		= supervisor write data, page not present
  instruction pointer	= 0x20:0xffffffff81086d0d
  stack pointer	        = 0x28:0xfffffe004adb3850
  frame pointer	        = 0x28:0xfffffe004adb3850
  code segment		= base rx0, limit 0xfffff, type 0x1b
  			= DPL 0, pres 1, long 1, def32 0, gran 1
  processor eflags	= interrupt enabled, resume, IOPL = 0
  current process		= 615 (syslogd)
  trap number		= 12
  panic: page fault
  cpuid = 0
  time = 1632906951
  KDB: stack backtrace:
  #0 0xffffffff80c57345 at kdb_backtrace+0x65
  #1 0xffffffff80c09d21 at vpanic+0x181
  #2 0xffffffff80c09b93 at panic+0x43
  #3 0xffffffff8108b187 at trap_fatal+0x387
  #4 0xffffffff8108b1df at trap_pfault+0x4f
  #5 0xffffffff8108a83d at trap+0x27d
  #6 0xffffffff810617a8 at calltrap+0x8
  #7 0xffffffff81064e71 at fpugetregs+0x171
  #8 0xffffffff810686ec at sendsig+0x16c
  #9 0xffffffff80c0f527 at postsig+0x1a7
  #10 0xffffffff80c6dfa7 at ast+0x307
  #11 0xffffffff810641b9 at doreti_ast+0x1f

Ubuntu 21.04 is used as host operating system with:
  - libvirt-daemon-driver-qemu/hirsute-updates,now 7.0.0-2ubuntu2.1
  - qemu-system-x86/hirsute-updates,hirsute-security,now 1:5.2+dfsg-9ubuntu3.1 amd64
  - linux-image-generic/hirsute-updates,hirsute-security 5.11.0.37.39 amd64
Comment 1 Yann Droneaud 2021-09-29 10:24:15 UTC
Created attachment 228243 [details]
kernel.log (verbose)
Comment 2 Yann Droneaud 2021-09-29 10:26:00 UTC
Created attachment 228244 [details]
libvirt guest description
Comment 3 Yann Droneaud 2021-09-29 10:28:06 UTC
I tried QEMU's Q53 and i440FX chipset configuration with same results.
Comment 4 Yann Droneaud 2021-09-29 10:43:12 UTC
The issue can be worked around by setting a different CPU model from the host, host being an AMD Ryzen 9 5950x: specifing "EPYC" instead allows to boot to the installer.
Comment 5 Yann Droneaud 2021-09-29 11:33:40 UTC
(In reply to Yann Droneaud from comment #4)

Switching from "host" CPU model to EPYC CPU model as guest CPU, the kernel messages are modified this way:
  -CPU: AMD EPYC-Milan Processor (3393.63-MHz K8-class CPU)
  -  Origin="AuthenticAMD"  Id=0xa00f11  Family=0x19  Model=0x1  Stepping=1
  +CPU: AMD EPYC Processor (3393.62-MHz K8-class CPU)
  +  Origin="AuthenticAMD"  Id=0x800f12  Family=0x17  Model=0x1  Stepping=2
     Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  -  Features2=0xfff83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  +  Features2=0xfef83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
     AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  -  AMD Features2=0xc003f7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,Topology,PCXC>
  -  Structured Extended Features=0x219c07ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
  -  Structured Extended Features2=0x40060c<UMIP,PKU,VAES,VPCLMULQDQ,RDPID>
  -  Structured Extended Features3=0xac000010<FSRM,IBPB,STIBP,ARCH_CAP,SSBD>
  -  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  -  IA32_ARCH_CAPS=0x69<RDCL_NO,SKIP_L1DFL_VME,MDS_NO>
  -  AMD Extended Feature Extensions ID EBX=0x300d205<CLZERO,XSaveErPtr,WBNOINVD,IBPB,IBRS,STIBP,SSBD,VIRT_SSBD>
  +  AMD Features2=0x4003f5<LAHF,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,Topology>
  +  Structured Extended Features=0x209c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA>
  +  XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE>
     SVM: NP,NRIP,NAsids=16
Comment 6 Yann Droneaud 2021-09-29 12:54:03 UTC
Created attachment 228248 [details]
kernel log as guest with Epyc as CPU model

(In reply to Yann Droneaud from comment #5)
Comment 7 Yann Droneaud 2021-09-29 12:55:15 UTC
Created attachment 228249 [details]
kernel log, booted on host computer (baremetal) instead of being virtualized
Comment 8 Yann Droneaud 2021-09-29 15:58:57 UTC
Bypassing libvirtd, but reusing some of the command line parameter it uses, I can reproduce the issue with qemu:

  qemu-system-x86_64 -m 1024 \
    -machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
    -cpu EPYC-Milan,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,vaes=on,vpclmulqdq=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,pcid=off,svme-addr-chk=off \
    -cdrom  FreeBSD-13.0-RELEASE-amd64-disc1.iso 

I will try to trim the options to the one that trigger the panic.
Comment 9 Yann Droneaud 2021-09-29 16:14:52 UTC
(In reply to Yann Droneaud from comment #8)

Boot: 
  qemu-system-x86_64 -m 1024 -machine pc-q35-5.2,accel=kvm -cpu EPYC      -cdrom FreeBSD-13.0-RELEASE-amd64-disc1.iso 
  qemu-system-x86_64 -m 1024 -machine pc-q35-5.2,accel=kvm -cpu EPYC-v1   -cdrom FreeBSD-13.0-RELEASE-amd64-disc1.iso 
  qemu-system-x86_64 -m 1024 -machine pc-q35-5.2,accel=kvm -cpu EPYC-v2   -cdrom FreeBSD-13.0-RELEASE-amd64-disc1.iso 
  qemu-system-x86_64 -m 1024 -machine pc-q35-5.2,accel=kvm -cpu EPYC-v3   -cdrom FreeBSD-13.0-RELEASE-amd64-disc1.iso
  qemu-system-x86_64 -m 1024 -machine pc-q35-5.2,accel=kvm -cpu EPYC-IBPB -cdrom FreeBSD-13.0-RELEASE-amd64-disc1.iso

Panic: 
  qemu-system-x86_64 -m 1024 -machine pc-q35-5.2,accel=kvm -cpu EPYC-Milan    -cdrom FreeBSD-13.0-RELEASE-amd64-disc1.iso
  qemu-system-x86_64 -m 1024 -machine pc-q35-5.2,accel=kvm -cpu EPYC-Milan-v1 -cdrom FreeBSD-13.0-RELEASE-amd64-disc1.iso
Comment 10 Yann Droneaud 2021-09-30 08:13:09 UTC
Problem also happen with 12.2 GENERIC kernel
Comment 11 Yann Droneaud 2021-10-07 09:28:07 UTC
When testing against upstream Qemu, I've found v6.0.0 reproduces the problem, and v6.1.0 doesn't !

So maybe it's not a problem in FreeBSD implementation after all.

Doing some bisecting, I've identified a commit in Qemu that seems to fix my issue:

  commit fea4500841024195ec701713e05b92ebf667f192 (HEAD)
  Author: David Edmondson <david.edmondson@oracle.com>
  Date:   Mon Jul 5 11:46:31 2021 +0100

      target/i386: Populate x86_ext_save_areas offsets using cpuid where possible
    
      Rather than relying on the X86XSaveArea structure definition,
      determine the offset of XSAVE state areas using CPUID leaf 0xd where
      possible (KVM and HVF).
    
      Signed-off-by: David Edmondson <david.edmondson@oracle.com>
      Message-Id: <20210705104632.2902400-8-david.edmondson@oracle.com>
      Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

I can't tell anymore if there's a bug in FreeBSD kernel, so I'm closing the bug.
Comment 12 cynix 2021-12-18 10:02:04 UTC
(In reply to Yann Droneaud from comment #11)

If you can't update QEMU or change CPU type on the host, another possible workaround is to add hw.use_xsave=0 to /boot/loader.conf.