Bug 170388

Summary: 8-STABLE amd64 past r233799 is unable to boot in certain KVM environments [regression]
Product: Base System Reporter: Jimmy Olgeni <olgeni>
Component: amd64Assignee: Konstantin Belousov <kib>
Status: Closed FIXED    
Severity: Affects Only Me CC: jkim
Priority: Normal    
Version: 8.3-STABLE   
Hardware: Any   
OS: Any   

Description Jimmy Olgeni freebsd_committer freebsd_triage 2012-08-05 10:40:02 UTC
FreeBSD 8-STABLE is unable to boot in some KVM environments after r233799.

------------------------------------------------------------------------
r233799 | jkim | 2012-04-02 20:27:06 +0200 (Mon, 02 Apr 2012) | 4 lines

MFC:    r233702

Work around Erratum 721 for AMD Family 10h and 12h processors.

------------------------------------------------------------------------

The following panic message is shown immediately after the kernel starts:

===
kernel trap 9 with interrupts disabled

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer    = 0x20:0xffffffff808fd8dc
stack pointer          = 0x28:0xffffffff80cd5440
frame pointer          = 0x28:0xffffffff80cd5470
code segment           = base rx0,  limit 0xfffff, type 0x1b
                       = DPL 0, pres 1,  long 1, def32 0, gran 1
processor eflags       = resume,  IOPL = 0
current process        = 0 ()
trap number            = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff8064e18e at ??+0
#1 0xffffffff8061b247 at ??+0
#2 0xffffffff80912ca0 at ??+0
#3 0xffffffff80913235 at ??+0
#4 0xffffffff808faad4 at ??+0
#5 0xffffffff80904cc2 at ??+0
#6 0xffffffff801a0244 at ??+0
===

When booting from the latest 8-STABLE branch the booting process
just halts after the first panic line.

The same problem can be observed in 9-STABLE, but I did not actually
bisect all the way to the relevant commit yet. However, r233702
seems a reasonable guess.

Once the system is booted by reverting r233799, the following
processor information can be read:

    CPU: Six-Core AMD Opteron(tm) Processor 2427 (2211.49-MHz K8-class CPU)
      Origin = "AuthenticAMD"  Id = 0x100f80  Family = 10  Model = 8  Stepping = 0
      Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
      Features2=0x80802001<SSE3,CX16,POPCNT,HV>
      AMD Features=0xe6500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM,3DNow!+,3DNow!>
      AMD Features2=0x1f7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch>
      TSC: P-state invariant

Also, using x86info:

    x86info v1.30.  Dave Jones 2001-2011
    Feedback to <davej@redhat.com>.

    Extended Family: 1 Extended Model: 0 Family: 15 Model: 8 Stepping: 0
    CPU Model (x86info's best guess): Phenom/Athlon/Sempron/Turion (II)/Opteron (HY-D0)
    Processor name string (BIOS programmed): Six-Core AMD Opteron(tm) Processor 2427

    Monitor/Mwait: min/max line size 0/0, ecx bit 0 support, enumeration extension
    SVM: revision 1, 16 ASIDs, np, NRIPSave
    Address Size: 48 bits virtual, 40 bits physical
    running at an estimated 2.35GHz

Unfortunately I have no detailed information about the QEMU
configuration, but I set up a dedicated VM that I could use to test
patches.
Comment 1 Kostik Belousov 2012-08-05 14:55:18 UTC
Try this. Comment should give enough explanation what happens there, my
guess anyway.

diff --git a/sys/amd64/amd64/initcpu.c b/sys/amd64/amd64/initcpu.c
index 3890551..dbeaec6 100644
--- a/sys/amd64/amd64/initcpu.c
+++ b/sys/amd64/amd64/initcpu.c
@@ -91,11 +91,17 @@ init_amd(void)
 	 *
 	 * http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_Gd.pdf
 	 * http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_Gd.pdf
+	 *
+	 * Hypervisors do not provide access to the errata MSR,
+	 * causing #GP exception on attempt to apply the errata.  The
+	 * MSR write shall be done on host and persist globally
+	 * anyway, so do not try to do it when under virtualization.
 	 */
 	switch (CPUID_TO_FAMILY(cpu_id)) {
 	case 0x10:
 	case 0x12:
-		wrmsr(0xc0011029, rdmsr(0xc0011029) | 1);
+		if ((cpu_feature2 & CPUID2_HV) == 0)
+			wrmsr(0xc0011029, rdmsr(0xc0011029) | 1);
 		break;
 	}
 }
Comment 2 Jimmy Olgeni freebsd_committer freebsd_triage 2012-08-05 19:05:34 UTC
On Sun, 5 Aug 2012, Konstantin Belousov wrote:

> Try this. Comment should give enough explanation what happens there, my
> guess anyway.

Works great: I just booted into 8-STABLE.

I'll try with 9-STABLE too - rebuilding right now.

Thanks!
Comment 3 Jimmy Olgeni freebsd_committer freebsd_triage 2012-08-06 00:27:48 UTC
On Sun, 5 Aug 2012, Konstantin Belousov wrote:

> Try this. Comment should give enough explanation what happens there, my
> guess anyway.

Works great on 9-STABLE too (tested on r239079).

Looks like a *great* candidate for 9.1.

-- 
jimmy
Comment 4 Mark Linimon freebsd_committer freebsd_triage 2012-08-06 10:20:26 UTC
Responsible Changed
From-To: freebsd-amd64->kib

kib has a patch and submitter notes it fixes the problem.
Comment 5 dfilter service freebsd_committer freebsd_triage 2012-08-07 09:36:20 UTC
Author: kib
Date: Tue Aug  7 08:36:10 2012
New Revision: 239125
URL: http://svn.freebsd.org/changeset/base/239125

Log:
  Do not apply errata 721 workaround when under hypervisor, since
  typical hypervisor does not implement access to the required MSR,
  causing #GP on boot.
  
  Reported and tested by:	olgeni
  PR:	amd64/170388
  MFC after:	3 days

Modified:
  head/sys/amd64/amd64/initcpu.c

Modified: head/sys/amd64/amd64/initcpu.c
==============================================================================
--- head/sys/amd64/amd64/initcpu.c	Tue Aug  7 07:52:25 2012	(r239124)
+++ head/sys/amd64/amd64/initcpu.c	Tue Aug  7 08:36:10 2012	(r239125)
@@ -91,11 +91,17 @@ init_amd(void)
 	 *
 	 * http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_Gd.pdf
 	 * http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_Gd.pdf
+	 *
+	 * Hypervisors do not provide access to the errata MSR,
+	 * causing #GP exception on attempt to apply the errata.  The
+	 * MSR write shall be done on host and persist globally
+	 * anyway, so do not try to do it when under virtualization.
 	 */
 	switch (CPUID_TO_FAMILY(cpu_id)) {
 	case 0x10:
 	case 0x12:
-		wrmsr(0xc0011029, rdmsr(0xc0011029) | 1);
+		if ((cpu_feature2 & CPUID2_HV) == 0)
+			wrmsr(0xc0011029, rdmsr(0xc0011029) | 1);
 		break;
 	}
 }
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Comment 6 Jimmy Olgeni freebsd_committer freebsd_triage 2012-08-19 13:04:08 UTC
Hello,

Just a quick reminder about the MFC of r239125 :)

Thanks!

-- 
jimmy
Comment 7 dfilter service freebsd_committer freebsd_triage 2012-09-08 17:48:06 UTC
Author: kib
Date: Sat Sep  8 16:47:52 2012
New Revision: 240240
URL: http://svn.freebsd.org/changeset/base/240240

Log:
  MFC r239125:
  Do not apply errata 721 workaround when under hypervisor, since
  typical hypervisor does not implement access to the required MSR,
  causing #GP on boot.
  
  PR:	amd64/170388
  Approved by:	re (kensmith)

Modified:
  releng/9.1/sys/amd64/amd64/initcpu.c
Directory Properties:
  releng/9.1/sys/   (props changed)

Modified: releng/9.1/sys/amd64/amd64/initcpu.c
==============================================================================
--- releng/9.1/sys/amd64/amd64/initcpu.c	Sat Sep  8 16:45:48 2012	(r240239)
+++ releng/9.1/sys/amd64/amd64/initcpu.c	Sat Sep  8 16:47:52 2012	(r240240)
@@ -91,11 +91,17 @@ init_amd(void)
 	 *
 	 * http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_Gd.pdf
 	 * http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_Gd.pdf
+	 *
+	 * Hypervisors do not provide access to the errata MSR,
+	 * causing #GP exception on attempt to apply the errata.  The
+	 * MSR write shall be done on host and persist globally
+	 * anyway, so do not try to do it when under virtualization.
 	 */
 	switch (CPUID_TO_FAMILY(cpu_id)) {
 	case 0x10:
 	case 0x12:
-		wrmsr(0xc0011029, rdmsr(0xc0011029) | 1);
+		if ((cpu_feature2 & CPUID2_HV) == 0)
+			wrmsr(0xc0011029, rdmsr(0xc0011029) | 1);
 		break;
 	}
 }
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Comment 8 Konstantin Belousov freebsd_committer freebsd_triage 2012-09-08 17:52:27 UTC
State Changed
From-To: open->closed

Merged.