Bug 228768 - EARLY_AP_STARTUP causes panic on amd machines
Summary: EARLY_AP_STARTUP causes panic on amd machines
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Only Me
Assignee: John Baldwin
URL:
Keywords: patch
: 225450 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-06-05 17:33 UTC by Roger Hammerstein
Modified: 2018-11-26 17:46 UTC (History)
5 users (show)

See Also:
jhb: mfc-stable12+
jhb: mfc-stable11+

Attachments
lock_delay.patch (1.54 KB, patch)
2018-10-30 17:10 UTC, John Baldwin 		no flags Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Roger Hammerstein 2018-06-05 17:33:11 UTC 
update from August of last year:
https://marc.info/?l=freebsd-stable&m=150221767415633&w=2

The EARLY_AP_STARTUP kernel option causes kernel panics
on SUN X4200 AMD machines.   Commenting it out of GENERIC and
new kernel allows boot.  This also happens on a non-Sun 2x dual core 
AMD machine.

This continues with 11.2-RC1.

How can i help fix this? 

I did see the two threads on -current from 2016:
https://marc.info/?l=freebsd-current&m=146339417805599&w=2
https://marc.info/?l=freebsd-current&m=148009805024056&w=2


setting hint.hpet.0.per_cpu=0 doesn't help.  
Removing EARLY_AP_STARTUP still works.

adding these did not help get more info.   Are there others to try?

   options         KTR
   options         KTR_COMPILE=KTR_PROC
   options                 KTR_MASK=KTR_PROC
   options         KTR_VERBOSE=1



Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-PRERELEASE #10 r334650M: Tue Jun  5 15:50:30 UTC 2018
    root@xxx:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(vga): resolution 640x480
CPU: Dual-Core AMD Opteron(tm) Processor 2218 (2593.16-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x40f12  Family=0xf  Model=0x41  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT
>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>
  SVM: NAsids=64
real memory  = 4563402752 (4352 MB)
avail memory = 4104101888 (3913 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <SUN    X4200 M2>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 2 core(s)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 64/32 (20171214/tbfadt-748)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe1Block: 128/64 (20171214/tbfadt-748)
ioapic1: Changing APIC ID to 16
ioapic2: Changing APIC ID to 17
ioapic3 <Version 1.1> irqs 24-47 on motherboard
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 48-54 on motherboard
ioapic2 <Version 1.1> irqs 56-62 on motherboard
cpu0 sleepq_broadcast(0xfffff80003998500, 0)
cpu0 sleepq_broadcast(0xfffff80003998000, 0)
cpu0 sleepq_broadcast(0xfffff80003997a00, 0)
SMP: AP CPU ^M
k^M

kernel trap 12 with interrupts disabled
kernel trap 12 with interrupts disabled


Fatal trap -2116731224: UNKNOWN while in kernel\xff\xff^Akernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x2
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff81d53eb1
stack pointer           = 0x28:0xfffffe00f6dd83b0
frame pointer           = 0x28:0xffffffff81d53ec0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 11 (idle: cpu1)
trap number             = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff80ad5b67 at kdb_backtrace+0x67
#1 0xffffffff80a96827 at vpanic+0x177
#2 0xffffffff80a966a3 at panic+0x43
#3 0xffffffff80ec625f at trap_fatal+0x35f
#4 0xffffffff80ec62b9 at trap_pfault+0x49
#5 0xffffffff80ec5a87 at trap+0x2c7
#6 0xffffffff80ea7a7c at calltrap+0x8
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x202
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xfffff8000398a647
stack pointer           = 0x28:0xfffffe00f6dd8010
frame pointer           = 0x28:0x1
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 11 (idle: cpu1)
trap number             = 12
panic: page fault
cpuid = 1
Uptime: 1s
Rebooting...



--------------------------------------------------------------------


Table 'APIC' at 0xdfff0390
APIC: Found table at 0xdfff0390
APIC: Using the MADT enumerator.
MADT: Found CPU APIC ID 0 ACPI ID 1: enabled
SMP: Added CPU 0 (AP)
MADT: Found CPU APIC ID 1 ACPI ID 2: enabled
SMP: Added CPU 1 (AP)
MADT: Found CPU APIC ID 2 ACPI ID 3: enabled
SMP: Added CPU 2 (AP)
MADT: Found CPU APIC ID 3 ACPI ID 4: enabled
SMP: Added CPU 3 (AP)
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-PRERELEASE #10 r334650M: Tue Jun  5 15:50:30 UTC 2018
    root@xxx:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
Table 'FACP' at 0xdfff0290
Table 'APIC' at 0xdfff0390
Table 'SPCR' at 0xdfff0420
Table 'SLIT' at 0xdfff0470
Table 'OEMB' at 0xdfffe040
Table 'HPET' at 0xdfff6410
Table 'IPET' at 0xdfff6450
Table 'SRAT' at 0xdfff6490
SRAT: Found table at 0xdfff6490
SRAT: Found CPU APIC ID 0 domain 0: enabled
SRAT: Found CPU APIC ID 1 domain 0: enabled
SRAT: Found memory domain 0 addr 0x0 len 0xa0000: enabled
SRAT: Found memory domain 0 addr 0x100000 len 0x7ff00000: enabled
SRAT: Found CPU APIC ID 2 domain 1: enabled
SRAT: Found CPU APIC ID 3 domain 1: enabled
SRAT: Found memory domain 1 addr 0x80000000 len 0x60000000: enabled
SRAT: Found memory domain 1 addr 0x100000000 len 0x20000000: enabled
Table 'FACP' at 0xdfff0290
Table 'APIC' at 0xdfff0390
Table 'SPCR' at 0xdfff0420
Table 'SLIT' at 0xdfff0470
SLIT: Found table at 0xdfff0470
SLIT.Localities: 2
0: 10 10
1: 10 10
PPIM 0: PA=0xa0000, VA=0xffffffff82210000, size=0x10000, mode=0
VT(vga): resolution 640x480
Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff82197000.
Preloaded boot_entropy_cache "/boot/entropy" at 0xffffffff8219ff38.
Calibrating TSC clock ... TSC clock: 2593156593 Hz
CPU: Dual-Core AMD Opteron(tm) Processor 2218 (2593.16-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x40f12  Family=0xf  Model=0x41  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT
>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>
  SVM: Features=0x0
Revision=1, ASIDs=64
L1 2MB data TLB: 8 entries, fully associative
L1 2MB instruction TLB: 8 entries, fully associative
L1 4KB data TLB: 32 entries, fully associative
L1 4KB instruction TLB: 32 entries, fully associative
L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
L2 2MB unified TLB: 0 entries, disabled/not present
L2 4KB data TLB: 512 entries, 4-way associative
L2 4KB instruction TLB: 512 entries, 4-way associative
L2 unified cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative
real memory  = 4563402752 (4352 MB)
Physical memory chunk(s):
0x0000000000010000 - 0x0000000000098fff, 561152 bytes (137 pages)
0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages)
0x0000000002200000 - 0x00000000d8a44fff, 3598995456 bytes (878661 pages)
0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048 pages)
avail memory = 4104101888 (3913 MB)
Event timer "LAPIC" quality 100
LAPIC: ipi_wait() us multiplier 72 (r 3600257 tsc 2593156593)
ACPI APIC Table: <SUN    X4200 M2>
Package ID shift: 1
L2 cache ID shift: 0
L1 cache ID shift: 0
Core ID shift: 0
INTR: Adding local APIC 1 as a target
INTR: Adding local APIC 2 as a target
INTR: Adding local APIC 3 as a target
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 2 core(s)
Package HW ID = 0 (0)
        Core HW ID = 0 (0)
                CPU0 (BSP): APIC ID: 0 (0)
        Core HW ID = 1 (0x1)
                CPU1 (AP): APIC ID: 1 (0x1)
Package HW ID = 1 (0x1)
        Core HW ID = 2 (0x2)
                CPU2 (AP): APIC ID: 2 (0x2)
        Core HW ID = 3 (0x3)
                CPU3 (AP): APIC ID: 3 (0x3)
APIC: CPU 0 has ACPI ID 1
APIC: CPU 1 has ACPI ID 2
APIC: CPU 2 has ACPI ID 3
APIC: CPU 3 has ACPI ID 4
SRAT: CPU 0 has memory domain 0
SRAT: CPU 1 has memory domain 0
SRAT: CPU 2 has memory domain 1
SRAT: CPU 3 has memory domain 1
x86bios:  IVT 0x000000-0x0004ff at 0xfffff80000000000
x86bios: SSEG 0x098000-0x098fff at 0xfffffe00f6df4000
x86bios: EBDA 0x09d000-0x09ffff at 0xfffff8000009d000
x86bios:  ROM 0x0a0000-0x0fefff at 0xfffff800000a0000
Pentium Pro MTRR support enabled
ULE: setup cpu 0
ULE: setup cpu 1
ULE: setup cpu 2
ULE: setup cpu 3
ACPI: RSDP 0x00000000000F9470 000024 (v02 SUN   )
ACPI: XSDT 0x00000000DFFF0100 00006C (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: FACP 0x00000000DFFF0290 0000F4 (v03 SUN    X4200 M2 00000039 MSFT 00000097)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 64/32 (20171214/tbfadt-748)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe1Block: 128/64 (20171214/tbfadt-748)
ACPI: DSDT 0x00000000DFFF04A0 005F64 (v01 SUN    X4200 M1 00000039 INTL 20050624)
ACPI: FACS 0x00000000DFFFE000 000040
ACPI: APIC 0x00000000DFFF0390 000086 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: SPCR 0x00000000DFFF0420 000050 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: SLIT 0x00000000DFFF0470 000030 (v01 SUN    OEMSLIT  00000039 MSFT 00000097)
ACPI: OEMB 0x00000000DFFFE040 000063 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: HPET 0x00000000DFFF6410 000038 (v01 SUN    OEMHPET0 00000039 MSFT 00000097)
ACPI: IPET 0x00000000DFFF6450 000038 (v01 SUN    OEMHPET1 00000039 MSFT 00000097)
ACPI: SRAT 0x00000000DFFF6490 000110 (v01 AMD    HAMMER   00000001 AMD  00000001)
ACPI: SSDT 0x00000000DFFF65A0 0004F0 (v01 A M I  POWERNOW 00000001 AMD  00000001)
MADT: Found IO APIC ID 15, Interrupt 0 at 0xfec00000
ioapic0: ver 0x11 maxredir 0x17
ioapic0: Routing external 8259A's -> intpin 0
MADT: Found IO APIC ID 16, Interrupt 48 at 0xfeafd000
ioapic1: Changing APIC ID to 16
ioapic1: WARNING: intbase 48 != expected base r24
ioapic1: ver 0x11 maxredir 0x06
MADT: Found IO APIC ID 17, Interrupt 56 at 0xfeafc000
ioapic2: Changing APIC ID to 17
ioapic2: WARNING: intbase 56 != expected base r55
ioapic2: ver 0x11 maxredir 0x06
MADT: Found IO APIC ID 14, Interrupt 24 at 0xfeaff000
ioapic3: WARNING: intbase 24 != expected base r63
ioapic3: ver 0x11 maxredir 0x17
MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
ioapic3 <Version 1.1> irqs 24-47 on motherboard
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 48-54 on motherboard
ioapic2 <Version 1.1> irqs 56-62 on motherboard
cpu0 BSP:
     ID: 0x00000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
cpu0 sleepq_broadcast(0xfffff80003998500, 0)
cpu0 sleepq_broadcast(0xfffff80003998000, 0)
cpu0 sleepq_broadcast(0xfffff80003997a00, 0)
SMP: AP CPU ^M^M^M
^M^M^M^M^M^M^M

^Mkernel trap 12 with interrupts disabled^M
kernel trap 12 with interrupts disab\xff\xff\xff\xff\xff\xff\xff\xff\xa8>\xd5\x81\xff\xff\xff\xff\xff\xff\xff\xff\x90|\xdd\
xf6kernel trap 12 with interrupts disabled


Fatal trap -153257424: UNKNOWN ^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M
^M^M^M^M^M^M^M^Mk^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M^M
Comment 1 Roger Hammerstein 2018-10-29 02:51:42 UTC 
This issue with SMP and EARLY_AP_STARTUP continues with 12.0-BETA2 r339840.

Here is the latest console output as it double-faults and reboots

How can I help ?  What other information could I give? 


Table 'FACP' at 0xdfff0290
Table 'APIC' at 0xdfff0390
Table 'SPCR' at 0xdfff0420
Table 'SLIT' at 0xdfff0470
SLIT: Found table at 0xdfff0470
SLIT.Localities: 2
0: 10 10
1: 10 10
PPIM 0: PA=0xa0000, VA=0xffffffff82810000, size=0x10000, mode=0
PPIM 1: PA=0x11ffe7000, VA=0xffffffff82820000, size=0x1000, mode=0x6
pmap: large map 8 PML4 slots (4096 Gb)
VT(vga): resolution 640x480
Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff82638000.
Preloaded boot_entropy_cache "/boot/entropy" at 0xffffffff826411f8.
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
Calibrating TSC clock ... TSC clock: 2593156801 Hz
CPU: Dual-Core AMD Opteron(tm) Processor 2218 (2593.16-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x40f12  Family=0xf  Model=0x41  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>
  SVM: Features=0x0
Revision=1, ASIDs=64
L1 2MB data TLB: 8 entries, fully associative
L1 2MB instruction TLB: 8 entries, fully associative
L1 4KB data TLB: 32 entries, fully associative
L1 4KB instruction TLB: 32 entries, fully associative
L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
L2 2MB unified TLB: 0 entries, disabled/not present
L2 4KB data TLB: 512 entries, 4-way associative
L2 4KB instruction TLB: 512 entries, 4-way associative
L2 unified cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative
real memory  = 4563402752 (4352 MB)
Physical memory chunk(s):
0x0000000000010000 - 0x000000000009bfff, 573440 bytes (140 pages)
0x0000000000103000 - 0x00000000001fffff, 1036288 bytes (253 pages)
0x0000000002800000 - 0x00000000d8a67fff, 3592847360 bytes (877160 pages)
0x0000000100000000 - 0x000000011ffe5fff, 536764416 bytes (131046 pages)
avail memory = 4112728064 (3922 MB)
MADT: Found CPU APIC ID 0 ACPI ID 1: enabled
SMP: Added CPU 0 (AP)
MADT: Found CPU APIC ID 1 ACPI ID 2: enabled
SMP: Added CPU 1 (AP)
MADT: Found CPU APIC ID 2 ACPI ID 3: enabled
SMP: Added CPU 2 (AP)
MADT: Found CPU APIC ID 3 ACPI ID 4: enabled
SMP: Added CPU 3 (AP)
Event timer "LAPIC" quality 100
LAPIC: ipi_wait() us multiplier 72 (r 3600094 tsc 2593156801)
ACPI APIC Table: <SUN    X4200 M2>
Package ID shift: 1
L2 cache ID shift: 0
L1 cache ID shift: 0
Core ID shift: 0
INTR: Adding local APIC 1 as a target
INTR: Adding local APIC 2 as a target
INTR: Adding local APIC 3 as a target
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 2 core(s)
Package HW ID = 0
        Core HW ID = 0
                CPU0 (BSP): APIC ID: 0
        Core HW ID = 1
                CPU1 (AP): APIC ID: 1
Package HW ID = 1
        Core HW ID = 2
                CPU2 (AP): APIC ID: 2
        Core HW ID = 3
                CPU3 (AP): APIC ID: 3
APIC: CPU 0 has ACPI ID 1
APIC: CPU 1 has ACPI ID 2
APIC: CPU 2 has ACPI ID 3
APIC: CPU 3 has ACPI ID 4
Pentium Pro MTRR support enabled
x86bios:  IVT 0x000000-0x0004ff at 0xfffff80000000000
x86bios: SSEG 0x098000-0x098fff at 0xfffffe0000614000
x86bios: EBDA 0x09d000-0x09ffff at 0xfffff8000009d000
x86bios:  ROM 0x0a0000-0x0fefff at 0xfffff800000a0000
SRAT: CPU 0 has memory domain 0
SRAT: CPU 1 has memory domain 0
SRAT: CPU 2 has memory domain 1
SRAT: CPU 3 has memory domain 1
random: read 3840 bytes from preloaded cache
random: unblocking device.
arc4random: read 32 bytes from preloaded cache
VIMAGE (virtualized network stack) enabled
ULE: setup cpu 0
ULE: setup cpu 1
ULE: setup cpu 2
ULE: setup cpu 3
ACPI: RSDP 0x00000000000F9470 000024 (v02 SUN   )
ACPI: XSDT 0x00000000DFFF0100 00006C (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: FACP 0x00000000DFFF0290 0000F4 (v03 SUN    X4200 M2 00000039 MSFT 00000097)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 64/32 (20181003/tbfadt-748)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe1Block: 128/64 (20181003/tbfadt-748)
ACPI: DSDT 0x00000000DFFF04A0 005F64 (v01 SUN    X4200 M1 00000039 INTL 20050624)
ACPI: FACS 0x00000000DFFFE000 000040
ACPI: APIC 0x00000000DFFF0390 000086 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: SPCR 0x00000000DFFF0420 000050 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: SLIT 0x00000000DFFF0470 000030 (v01 SUN    OEMSLIT  00000039 MSFT 00000097)
ACPI: OEMB 0x00000000DFFFE040 000063 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: HPET 0x00000000DFFF6410 000038 (v01 SUN    OEMHPET0 00000039 MSFT 00000097)
ACPI: IPET 0x00000000DFFF6450 000038 (v01 SUN    OEMHPET1 00000039 MSFT 00000097)
ACPI: SRAT 0x00000000DFFF6490 000110 (v01 AMD    HAMMER   00000001 AMD  00000001)
ACPI: SSDT 0x00000000DFFF65A0 0004F0 (v01 A M I  POWERNOW 00000001 AMD  00000001)
MADT: Found IO APIC ID 15, Interrupt 0 at 0xfec00000
ioapic0: ver 0x11 maxredir 0x17
ioapic0: Routing external 8259A's -> intpin 0
MADT: Found IO APIC ID 16, Interrupt 48 at 0xfeafd000
ioapic1: Changing APIC ID to 16
ioapic1: WARNING: intbase 48 != expected base r24
ioapic1: ver 0x11 maxredir 0x06
MADT: Found IO APIC ID 17, Interrupt 56 at 0xfeafc000
ioapic2: Changing APIC ID to 17
ioapic2: WARNING: intbase 56 != expected base r55
ioapic2: ver 0x11 maxredir 0x06
MADT: Found IO APIC ID 14, Interrupt 24 at 0xfeaff000
ioapic3: WARNING: intbase 24 != expected base r63
ioapic3: ver 0x11 maxredir 0x17
MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
ioapic3 <Version 1.1> irqs 24-47 on motherboard
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 48-54 on motherboard
ioapic2 <Version 1.1> irqs 56-62 on motherboard
lapic: Divisor 2, Frequency 99737098 Hz
cpu0 BSP:
    ID: 0x00000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU k#3er nLaelu ntrackph eder1!2 n
t2 s hc pi untr3ap  AteP1r:rup
     dwi i tsah bin led
l 0t aDni:c:r 0rupx0m30_fault0t 0_h0odlisd:a faul Vtble oRn n:ofdau
e    x
8ry,
     005Faa0tddalr: 0x 0tra10 fLDffp f1f2fffR81:fc : 0x00p07000a
0ge 0fcpu0au00 DltidF = 2R
fff i mxef w=h ffifl1fe
i  nl
     K iDBnt: s0:tk ack baerc0kxn0t00rae10cl7 e0:0
0int1dpe:
 010pumide: 1 0s=
5x  0A0; utomat4ipicc  ir00edo oTtP  i=R:  n 0011
e  0fas0ecu0ol0ndst v -i000 r prtSVeRsus: a al k 0eyx ao00n0dd0 1frthfeess      co= nsol0
  to x2  abtimort4
er
e : f0ax000100euf thelrm: t0x000 1000c0 err: 0xo00d0000f0 pmc: 0x000104Re0
                b oot iAngM.D .=ex.

[reboots]




--------
a second boot's panic, showing more interleaved-like text

cpu0 BSP:
     ID: 0x00000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU
erkkkernkkkkkkk
  neFla ttarla p 12 dwitho uibntlerreupt sf daiusablledt

r
i
 Fpata l 0txrafp f12:f pfagfe ffauflt fw8h0ib17ee in1 keernel8 mod e
rcspuid p= 1 ; 0axpfififc


 efradulri t0vi xfffvisrffotura r8e0al080d in05 4as0d00d
rrt8 resuct0sxi5o r     =9  0xn0f,x 4ff0p
ata, potfufe8l1tc fcc7478toiodne                vr=1 0siolup ervis0xao2r t0iread od
a1    nr
c 1ige 0  rno1t 2prenss e0txrntuf
 instrfutcfitonfi foen0po 0p0io0i60n0nterter    080=    =   0r013xx 22000x:f0f:xfff0fxffff8f0ff0f8ff0f0ff8514fff008c00b77e6
4r178bc
4
 s s0x1ta ckr tapcko1inte 5rp     0x ff ffo f800  0i31d  en58=te0r  0rxf2la8gs: 0         x01x f0 ff  f0f4e6  00=0
 cs 002xa028xc9:c02
  f s=m0e  ffpfofiesn0s t00x028 2dse ra 07  0x2e8  e s  000x2 8
      f 0 xr2a80 gsx 2m80:0x2x8
feffff posbiase fe0n g0sba00ste2 0ea0xfc9cr      0
cfode  f ff s fefg mef n8t2              = 0=b0ase f20 0x00 0k,g lsxib2am8s:i0te x 0ffx0fxfffffffffff, ffftf8y1ffpe ffc8020000xf2190b


                cod
in                 e=  DcPpLuis 0e,dgm  preens= t       13,      =l;o  ngabp ica1s ,ie  d0x de0f,3=2 0 l 0,i g3mra
   t1 p0xa
pfnffriffco: dcoeubsles foa rul etfyplate
=3ug id0 =sx1b
        r       e
 LI 0,O          tisu   meme= = DP, 1
 b     KPDB: L  spres= 1,t  aclkong 0
f32acckt uraced:
e  U 0p,ti gmrer:an 1s
 Aut om1
Lpratto piroccc erebesoostso     i      nr 15  sef=ela congsd1  s= 1-  press  (iare dlek:s eumcpeyu ,o InO )P
ons=he ap0c n
   courlmree ntbot  praeborrotc
es      -s->     =P res1s2 a    = 11
p ke( onan tihec:  pageifacuonsdoll
ec:to rp ebicopot,u 21)tim
 =
   -tr
y:    KDap noum sbwesttcrck b    o      f=c ktt a1e2e
  stem
       nnic:ime: 1s
agReb ooting.f..
Comment 2 John Baldwin freebsd_committer freebsd_triage 2018-10-29 19:45:25 UTC 
Hmm, I thought had sent a mail about this earlier, but I guess not.  Are you able to put DDB in your kernel config and get a db> prompt at all?  If so, a stack trace would be really helpful.  If not, one thing to try first that might help is to get down from 4 CPUs to 2 CPUs.  You can set these hints at the loader prompt (or in loader.conf) to disable CPUs 2 and 3:

hint.lapic.2.disabled=1
hint.lapic.3.disabled=1
Comment 3 Roger Hammerstein 2018-10-30 13:15:19 UTC 
(In reply to John Baldwin from comment #2)

With DDB and WITNESS,

MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
ioapic3 <Version 1.1> irqs 24-47 on motherboard
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 48-54 on motherboard
ioapic2 <Version 1.1> irqs 56-62 on motherboard
lapic: Divisor 2, Frequency 99737090 Hz
cpu0 BSP:
     ID: 0x00000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU
FtKernel page fault with the following non-sleepable locks held:
exclusive spin mutex clk (clk) r = 0 (0xffffffff81e70d78) locked @ /usr/src/sys/x86/isa/clock.c:209
exclusive spin mutex ap boot (ap boot) r = 0 (0xffffffff81fc4788) locked @ /usr/src/sys/x86/x86/mp_x86.c:1013
stack backtrace:


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0xfffff80080054000
fault code              = supervisor read instruction, protection violation
instruction pointer     = 0x20:0xfffff80080054000
stack pointer           = 0x28:0xfffffe0002a8f8b0
frame pointer           = 0x28:0x0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 11 (idle: cpu1)
[ thread pid 11 tid 100004 ]
Stopped at      0xfffff80080054000
db> trace
Tracing pid 11 tid 100004 td 0xfffff80080054580
??() at 0xfffff80080054000
db>




Any other commands to run?



db> show locks
exclusive spin mutex clk (clk) r = 0 (0xffffffff81e70d78) locked @ /usr/src/sys/x86/isa/clock.c:209
exclusive spin mutex ap boot (ap boot) r = 0 (0xffffffff81fc4788) locked @ /usr/src/sys/x86/x86/mp_x86.c:1013
db>


db> show alllocks
Process 0 (kernel) thread 0xffffffff81e8a3e0 (100000)
exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff818020c0) locked @ /usr/src/sys/kern/kern_module.c:116
db>


db> bt
Tracing pid 11 tid 100004 td 0xfffff80080054580
??() at 0xfffff80080054000
db>

db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
   12     0     0     0  WL      (threaded)                  [intr]
100008                   I                                   [swi6: task queue]
100009                   I                                   [swi6: Giant taskq]
100011                   I                                   [swi5: fast taskq]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
100005                   CanRun                              [idle: cpu2]
100006                   CanRun                              [idle: cpu3]
    1     0     0     0  ?L                                  [kernel]
   10     0     0     0  RL                                  [audit]
    0     0     0     0  RLs     (threaded)                  [kernel]
100000                   Run     CPU 0                       [swapper]
100007                   RunQ                                [aiod_kick taskq]
100010                   RunQ                                [thread taskq]
100012                   RunQ                                [config_0]
100013                   RunQ                                [kqueue_ctx taskq]
db>


db> show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe0081100ec0
curthread    = 0xfffff80080054580: pid 11 tid 100004 "idle: cpu1"
curpcb       = 0
fpcurthread  = none
idlethread   = 0xfffff80080054580: tid 100004 "idle: cpu1"
curpmap      = 0xffffffff81e8aaa8
tssp         = 0xffffffff81fb5e88
commontssp   = 0xffffffff81fb5e88
rsp0         = 0x0
gs32p        = 0xffffffff81fbcac0
ldt          = 0xffffffff81fbcb00
tss          = 0xffffffff81fbcaf0
curvnet      = 0
spin locks held:
exclusive spin mutex clk (clk) r = 0 (0xffffffff81e70d78) locked @ /usr/src/sys/x86/isa/clock.c:209
exclusive spin mutex ap boot (ap boot) r = 0 (0xffffffff81fc4788) locked @ /usr/src/sys/x86/x86/mp_x86.c:1013
db>


db> show reg
cs                        0x20
ds                        0x28  ll+0x7
es                        0x28  ll+0x7
fs                        0x28  ll+0x7
gs                        0x28  ll+0x7
ss                        0x28  ll+0x7
rax         0xfffff80080054580
rcx                      0x5a8  ll+0x587
rdx         0xffffffff811b4939  do_execve.fexecv_proc_title+0x98a5
rbx                          0
rsp         0xfffffe0002a8f8b0
rbp                          0
rsi                        0x8
rdi         0xffffffff81d93998  w_mtx
r8          0xffffffff811bd5a6
r9                        0x6c  ll+0x4b
r10         0xfffff800dfd79600
r11                          0
r12         0xffffffff81dd7718
r13                          0
r14         0xffffffff8125cb1b
r15         0xfffff800dfd78b80
rip         0xfffff80080054000
rflags                 0x10286
0xfffff80080054000
db>


db> alltrace

Tracing command intr pid 12 tid 100008 td 0xfffff80080053580
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100009 td 0xfffff80080053000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100011 td 0xfffff800031f1000
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 100003 td 0xfffff800031df000
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 100004 td 0xfffff80080054580
??() at 18446735279764357120


Tracing command idle pid 11 tid 100005 td 0xfffff80080054000
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 100006 td 0xfffff800031de580
fork_trampoline() at fork_trampoline

Tracing command kernel pid 1 tid 100002 td 0xfffff800031df580
fork_trampoline() at fork_trampoline

Tracing command audit pid 10 tid 100001 td 0xfffff80080055000
fork_trampoline() at fork_trampoline

Tracing command kernel pid 0 tid 100000 td 0xffffffff81e8a3e0 (CPU 0)
??() at 0
KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+43/frame 0xfffffe0002a8f040
kdb_reenter() at kdb_reenter+47/frame 0xfffffe0002a8f050
trap() at trap+92/frame 0xfffffe0002a8f160
calltrap() at calltrap+8/frame 0xfffffe0002a8f160
--- trap 12, rip = 18446744071578560736, rsp = 18446741874730922544, rbp = 18446741874730922688 ---
db_read_bytes() at db_read_bytes+128/frame 0xfffffe0002a8f2c0
db_get_value() at db_get_value+51/frame 0xfffffe0002a8f300
db_backtrace() at db_backtrace+567/frame 0xfffffe0002a8f390
db_trace_thread() at db_trace_thread+61/frame 0xfffffe0002a8f3b0
_db_stack_trace_all() at _db_stack_trace_all+244/frame 0xfffffe0002a8f450
db_command() at db_command+633/frame 0xfffffe0002a8f520
db_command_loop() at db_command_loop+100/frame 0xfffffe0002a8f530
db_trap() at db_trap+239/frame 0xfffffe0002a8f5c0
kdb_trap() at kdb_trap+451/frame 0xfffffe0002a8f630
trap_fatal() at trap_fatal+689/frame 0xfffffe0002a8f680
trap_pfault() at trap_pfault+98/frame 0xfffffe0002a8f6d0
trap() at trap+670/frame 0xfffffe0002a8f7e0
calltrap() at calltrap+8/frame 0xfffffe0002a8f7e0
--- trap 12, rip = 18446735279764357120, rsp = 18446741874730924208, rbp = 0 ---
??() at 18446735279764357120
*** error reading from address 8 ***
KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+43/frame 0xfffffe0002a8f2b0
kdb_reenter() at kdb_reenter+47/frame 0xfffffe0002a8f2c0
db_get_value() at db_get_value+77/frame 0xfffffe0002a8f300
db_backtrace() at db_backtrace+567/frame 0xfffffe0002a8f390
db_trace_thread() at db_trace_thread+61/frame 0xfffffe0002a8f3b0
_db_stack_trace_all() at _db_stack_trace_all+244/frame 0xfffffe0002a8f450
db_command() at db_command+633/frame 0xfffffe0002a8f520
db_command_loop() at db_command_loop+100/frame 0xfffffe0002a8f530
db_trap() at db_trap+239/frame 0xfffffe0002a8f5c0
kdb_trap() at kdb_trap+451/frame 0xfffffe0002a8f630
trap_fatal() at trap_fatal+689/frame 0xfffffe0002a8f680
trap_pfault() at trap_pfault+98/frame 0xfffffe0002a8f6d0
trap() at trap+670/frame 0xfffffe0002a8f7e0
calltrap() at calltrap+8/frame 0xfffffe0002a8f7e0
--- trap 12, rip = 18446735279764357120, rsp = 18446741874730924208, rbp = 0 ---
??() at 18446735279764357120
db>

db> show lapic
lapic ID = 1
version  = 1.0
max LVT  = 5
SVR      = ff (enabled)
TPR      = 00
In-service Interrupts:
TMR Interrupts:
IRR Interrupts:
db>





with the cpus 2 and 3 disabled:
----------------------------------------------------
ioapic3 <Version 1.1> irqs 24-47 on motherboard
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 48-54 on motherboard
ioapic2 <Version 1.1> irqs 56-62 on motherboard
lapic: Divisor 2, Frequency 99736985 Hz
cpu0 BSP:
     ID: 0x00000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU #1 Launched!
cpu1 AP:
     ID: 0x01000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
[hangs here]
----------------------------------



with just DDB and w/o WITNESS:
----------------------------------

defini8254_delay() at) a     i8254_delay+0x1e/f_lock_indefinite_check() at  _mtx_lock_indefinite_check+0x21/frame 0xfffffe0
002986680
   _mtx_lock_spin_cookie()   at0_mtx_lock_spin_cookie+0xd4/frame 0xfffffe00029866f0
 at getit()     atkg_spin_cookie+0xd4/frame   0xfffffe0000606190
   getit()    atfgfe0002986710
rami8254_delay() at          0xfffffe00006061b0
/frame   0xfffffe0002986730
 at _mi8254_delay+0x1e/frame 0xfffffe00006061d0
  at_mtx_lock_indefinite_check()       atk_mtx_lock_indefinite_check+0x21/fram                             0x_fffffe0002986
740
in_cookie()ck_s_mtx_lock_spin_cookie+0xd4/frame 0xfffffe0000606250
     getit() atmgetit+0x91/frame   0xfffffe0000606270
 0xfffffe0054_delay()  at
 i8ge254_delay+0x1e/frame 0xfffffe0000606290
   _mtx_lgeock_indefinite_check() at  _mtx_lock_indefinite_ch eck+0x21/frame 0xfffffe00006062a0
  0xfffffe00029867d0
pin_cookie()              i8254_delay+0x1e/frame 0xfffffe00029867f0
 at _mtx_lock_spin_cookie+0xd4/frame           0x_mtx_lock_indefinite_check+
21/frgetit() at         0xfffffe0002986800
 0xfffffe0000606330
_spi8254_delay() at  i8254_delay+0x1e/                   atf_mtx_lock_spin_cookie+0xd4/frame 0xfffffde0002986870
eckg() ati_mtx_lock_indefinite_check+0x21/frame                               at
                                                                                geti_mtx_lock_spin_cookie() at  _mtx_lock_s
pin_cookie+0xd4/frame i0xfffffe00006063d0
 at getit()     ateglay+it+0x91/frame 0xfffffe00006063f0
   i8254_delay() atfi8254_delay+0x1e/_mtx_lock_indefinite_check() at                    0xfffffe0000606410
fra_mme   0xfffffe0002efinite_check() at_mtx_lock_spin_cookie()            atck+0x2_lock_spin_cookie+0xd4/frame 0xfffffe000
2986930
     0xfffffe0()    at6g420
it+0x91/frame 0xfffffe0002986950
ie()    at4_delay()                           atmie  0xfffffe0000606490
e/fgetit() at           0xfffffe0002986970
 0x_mtx_lock_indefinite_checki8254_delay() at   i8254_delay+0x1e/frame 0xfffffe00006064d0
   _mtx_mtx_lock_indefinite_check+0x21/frame 0xfff     at002986980
_in_mdefinite_check+0x21/frame 0xfffffe0000            at _mtx_lock_spin_cookie+0xd4/frame 0xfffffe000298         at _mtx_l
ock_spin_cookie+0xd4/frame 0xfffffe0000606550
   getit(getit+0x91/frame 0xfffffe0002986a10
   ati8254_dt+0x91/frame 0xfffffe0000606570
   i8254_delai8254_delay+0x1e/frame 0xfffffe0002986a30
  a_mitx_lock_indefinite_check() afra_mtx_lock_indefinite_check+0x21/frame 0xfffffe0002986a40
  0xfffffe0000606590
spi_n_cookie()      ate_mtx_lock_spin_cookie+0xd4/frame 0x                  at8_mtx_lock_indefinite_c             atame      0xfffffe00006065a0
0xf_mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0xd4/frame 0xfffffe0000606610
  atgietit()     atlgay+0x1e/frame                      0x
                                                          fffi8ffe0002986af0
at _mtx_lock_indefinite_check()      atx_ffffe0000606650
defin_mtx_lock_indefinite_check() at     0xfffffe0002986b00
fin_mtx_lock_spin_cookie()      atf_mtx_lock_sp0606660
kie_mtx_lock_spin_cookie() at  _mtx_lock_spin_cookie+0xd4/frame 0xfffffe00006066d0
    0xfffffe0002986b70
   getitge()        atfgrame 0xfffffe00006066f0
0xfi8254_delay()    at
                      i8254_delay+0x1e/frame 0xfffffe0000606710
at _mtx_lockdelay+0x1e/frame    0xfffffe0002986bb0
  at__mtx_lock_indefinite_check+0x21/frame 0xfffffe0000606720
   _atx_mtx_lock_indefinite_check+0x21/frame 0xfffffe0002986bc0
  a_m_mtx_lock_sn_cookie()          atm_mtx_lock_spin_cookie+0xd4/frame 0xffff                     getit()     at0g000606790


and it eventually reboots itselfs without going to a db> prompt
------------------------------------------------------------
Comment 4 John Baldwin freebsd_committer freebsd_triage 2018-10-30 17:09:52 UTC 
Oh, I think I have seen this once before.  The issue is that the mtx_lock_spin() code tries to use DELAY(1) and DELAY() is trying to use the "clock_lock" in getit().  In your case, 'tsc_is_invariant' isn't true so DELAY() in sys/x86/x86/delay.c isn't using the TSC.

I think the error is probably that we shouldn't be using DELAY(1) in our mutex code as it is too high level and/or we should special case DELAY(1) (vs DELAY(n)).  Older versions of FreeBSD would map DELAY(1) to just 'inb(0x84)' without talking to the 8254 at all.  It looks like i8254_delay() already does that when inside of KDB.

Also, checking for tsc_is_invariant in DELAY is probably overly-correct.  We aren't going to change P-states while we are spinning since we are actively spinning on the CPU.  The worst that might happen is that if the CPU were running at a lower clock, we might wait too long.  Given how DELAY is used, that's probably better than falling back to a slow timecounter with locks, etc.
Comment 5 John Baldwin freebsd_committer freebsd_triage 2018-10-30 17:10:55 UTC 
Created attachment 198774 [details]
lock_delay.patch
Comment 6 John Baldwin freebsd_committer freebsd_triage 2018-10-30 17:15:09 UTC 
FYI, I'd also probably go for changing delay_tc()'s special case of the very-common TSC case to just use the simpler loop without worrying about overflow, etc.  That said, I think the idea of having cpu_lock_delay() is probably worth fixing even if other changes to DELAY() would effectively fix this anyway.
Comment 7 Roger Hammerstein 2018-10-30 18:34:08 UTC 
(In reply to John Baldwin from comment #5)


this patch worked, but i needed two include statements.

patch  < ./lock.patch
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- sys/kern/kern_mutex.c      (revision 339923)
|+++ sys/kern/kern_mutex.c      (working copy)
--------------------------
Patching file sys/kern/kern_mutex.c using Plan A...
Hunk #1 succeeded at 1206.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- sys/x86/include/x86_var.h  (revision 339923)
|+++ sys/x86/include/x86_var.h  (working copy)
--------------------------
Patching file sys/x86/include/x86_var.h using Plan A...
Hunk #1 succeeded at 119.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- sys/x86/x86/delay.c        (revision 339923)
|+++ sys/x86/x86/delay.c        (working copy)
--------------------------
Patching file sys/x86/x86/delay.c using Plan A...
Hunk #1 succeeded at 110.
done




--- kern_mutex.o ---
/usr/src/sys/kern/kern_mutex.c:1209:3: error: implicit declaration of function 'cpu_lock_delay' is invalid in C99 [-Werror,
-Wimplicit-function-declaration]
                cpu_lock_delay();
                ^
/usr/src/sys/kern/kern_mutex.c:1209:3: note: did you mean 'lock_delay'?
/usr/src/sys/sys/lock.h:252:6: note: 'lock_delay' declared here
void    lock_delay(struct lock_delay_arg *);
        ^
/usr/src/sys/kern/kern_mutex.c:1209:3: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
                cpu_lock_delay();
                ^
--- kern_physio.o ---



added
#include <x86/include/x86_var.h>

to sys/kern/kern_mutex.c


and then

--- delay.o ---
cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O -pipe  -g -nostdinc  -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD  -MF.depend.delay.o -MTdelay.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -Werror  /usr/src/sys/x86/x86/delay.c
/usr/src/sys/x86/x86/delay.c:115:1: error: no previous prototype for function 'cpu_lock_delay' [-Werror,-Wmissing-prototypes]
cpu_lock_delay(void)
^
1 error generated.
*** [delay.o] Error code 1



added
#include <x86/include/x86_var.h>
to
/usr/src/sys/x86/x86/delay.c




newest boot:

INTR: Adding local APIC 1 as a target
INTR: Adding local APIC 2 as a target
INTR: Adding local APIC 3 as a target
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 2 core(s)
Package HW ID = 0
        Core HW ID = 0
                CPU0 (BSP): APIC ID: 0
        Core HW ID = 1
                CPU1 (AP): APIC ID: 1
Package HW ID = 1
        Core HW ID = 2
                CPU2 (AP): APIC ID: 2
        Core HW ID = 3
                CPU3 (AP): APIC ID: 3
APIC: CPU 0 has ACPI ID 1
APIC: CPU 1 has ACPI ID 2
APIC: CPU 2 has ACPI ID 3
APIC: CPU 3 has ACPI ID 4
x86bios:  IVT 0x000000-0x0004ff at 0xfffff80000000000
x86bios: SSEG 0x098000-0x098fff at 0xfffffe0000736000
x86bios: EBDA 0x09d000-0x09ffff at 0xfffff8000009d000
x86bios:  ROM 0x0a0000-0x0fefff at 0xfffff800000a0000
SRAT: CPU 0 has memory domain 0
SRAT: CPU 1 has memory domain 0
SRAT: CPU 2 has memory domain 1
SRAT: CPU 3 has memory domain 1
Pentium Pro MTRR support enabled
random: read 3840 bytes from preloaded cache
random: unblocking device.
arc4random: read 32 bytes from preloaded cache
VIMAGE (virtualized network stack) enabled
ULE: setup cpu 0
ULE: setup cpu 1
ULE: setup cpu 2
ULE: setup cpu 3
ACPI: RSDP 0x00000000000F9470 000024 (v02 SUN   )
ACPI: XSDT 0x00000000DFFF0100 00006C (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: FACP 0x00000000DFFF0290 0000F4 (v03 SUN    X4200 M2 00000039 MSFT 00000097)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 64/32 (20181003/tbfadt-748)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe1Block: 128/64 (20181003/tbfadt-748)
ACPI: DSDT 0x00000000DFFF04A0 005F64 (v01 SUN    X4200 M1 00000039 INTL 20050624)
ACPI: FACS 0x00000000DFFFE000 000040
ACPI: APIC 0x00000000DFFF0390 000086 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: SPCR 0x00000000DFFF0420 000050 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: SLIT 0x00000000DFFF0470 000030 (v01 SUN    OEMSLIT  00000039 MSFT 00000097)
ACPI: OEMB 0x00000000DFFFE040 000063 (v01 SUN    X4200 M2 00000039 MSFT 00000097)
ACPI: HPET 0x00000000DFFF6410 000038 (v01 SUN    OEMHPET0 00000039 MSFT 00000097)
ACPI: IPET 0x00000000DFFF6450 000038 (v01 SUN    OEMHPET1 00000039 MSFT 00000097)
ACPI: SRAT 0x00000000DFFF6490 000110 (v01 AMD    HAMMER   00000001 AMD  00000001)
ACPI: SSDT 0x00000000DFFF65A0 0004F0 (v01 A M I  POWERNOW 00000001 AMD  00000001)
MADT: Found IO APIC ID 15, Interrupt 0 at 0xfec00000
ioapic0: ver 0x11 maxredir 0x17
ioapic0: Routing external 8259A's -> intpin 0
MADT: Found IO APIC ID 16, Interrupt 48 at 0xfeafd000
ioapic1: Changing APIC ID to 16
ioapic1: WARNING: intbase 48 != expected base r24
ioapic1: ver 0x11 maxredir 0x06
MADT: Found IO APIC ID 17, Interrupt 56 at 0xfeafc000
ioapic2: Changing APIC ID to 17
ioapic2: WARNING: intbase 56 != expected base r55
ioapic2: ver 0x11 maxredir 0x06
MADT: Found IO APIC ID 14, Interrupt 24 at 0xfeaff000
ioapic3: WARNING: intbase 24 != expected base r63
ioapic3: ver 0x11 maxredir 0x17
MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
ioapic3 <Version 1.1> irqs 24-47 on motherboard
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 48-54 on motherboard
ioapic2 <Version 1.1> irqs 56-62 on motherboard
lapic: Divisor 2, Frequency 99736935 Hz
cpu0 BSP:
     ID: 0x00000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU #1 Launched!
cpu1 AP:
     ID: 0x01000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU #2 Launched!
cpu2 AP:
     ID: 0x02000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU #3 Launched!
cpu3 AP:
     ID: 0x03000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
TSC timecounter discards lower 1 bit(s)
Timecounter "TSC-low" frequency 1296582722 Hz quality -100
random: entropy device external interface
wlan: <802.11 Link Layer>
snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
EFI systbl not available
[ath_hal] loaded
nfslock: pseudo-device
crypto: <crypto core>
tcp_log: tcp_log device
module_register_init: MOD_LOAD (vesa, 0xffffffff81066bb0, 0) error 19
io: <I/O>
kbd: new array size 4
kbd1 at kbdmux0
mem: <memory>
netmap: loaded module
null: <full device, null device, zero device>
hpt27xx: RocketRAID 27xx controller driver v1.2.8
hptnr: R750/DC7280 controller driver v1.1.5
hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
crypto: assign cryptosoft0 driver id 0, flags 0x6000000
crypto: cryptosoft0 registers alg 1 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 2 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 3 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 4 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 5 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 16 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 6 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 7 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 32 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 18 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 19 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 20 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 8 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 15 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 9 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 10 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 13 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 14 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 34 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 35 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 36 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 37 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 11 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 22 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 23 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 25 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 24 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 26 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 27 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 28 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 21 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 17 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 29 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 30 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 31 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 38 flags 0 maxoplen 0
acpi0: <SUN X4200 M2> on motherboard
ACPI: 2 ACPI AML tables successfully acquired and loaded
ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
acpi0: Power Button (fixed)
acpi0: wakeup code va 0xfffffe00007f5000 pa 0x99000
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
cpu0: Processor \134_PR_.P001 (ACPI ID 1) -> APIC ID 0
cpu0: <ACPI CPU> on acpi0
cpu0: switching to generic Cx mode
cpu1: Processor \134_PR_.P002 (ACPI ID 2) -> APIC ID 1
cpu1: <ACPI CPU> on acpi0
cpu2: Processor \134_PR_.P003 (ACPI ID 3) -> APIC ID 2
cpu2: <ACPI CPU> on acpi0
cpu3: Processor \134_PR_.P004 (ACPI ID 4) -> APIC ID 3
cpu3: <ACPI CPU> on acpi0
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
attimer0: Can't map interrupt.
Table 'FACP' at 0xdfff0290
FACP: Found table at 0xdfff0290
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 1 vector 48
ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 49


 12.0-BETA2 FreeBSD 12.0-BETA2 #3 r339840M
Comment 8 John Baldwin freebsd_committer freebsd_triage 2018-10-30 19:06:29 UTC 
Ok, I will work on getting a cleaned up version of this ready and merged for 11 and 12.  First I will add cpu_lock_delay() and then I will follow up with a second change to add a real version of it for x86.
Comment 9 John Baldwin freebsd_committer freebsd_triage 2018-11-01 21:12:02 UTC 
I still need to do some more testing, but you can grab an updated patch from https://github.com/freebsd/freebsd/compare/master...bsdjhb:cpu_lock_delay.diff

This is what I'm planning to commit as a series of 3 separate patches.
Comment 10 Roger Hammerstein 2018-11-04 14:28:29 UTC 
(In reply to John Baldwin from comment #9)

yes, that patch also works.

I updated to a slightly newer version and applied that patch:
12.0-BETA3 FreeBSD 12.0-BETA3 #4 r340076M


MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
ioapic3 <Version 1.1> irqs 24-47 on motherboard
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 48-54 on motherboard
ioapic2 <Version 1.1> irqs 56-62 on motherboard
lapic: Divisor 2, Frequency 99737060 Hz
cpu0 BSP:
     ID: 0x00000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU #1 Launched!
cpu1 AP:
     ID: 0x01000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU #3 Launched!
cpu3 AP:
     ID: 0x03000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
SMP: AP CPU #2 Launched!
cpu2 AP:
     ID: 0x02000000   VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
   AMD ext features: 0x00010003
   AMD elvt0: 0x00010000
TSC timecounter discards lower 1 bit(s)
Timecounter "TSC-low" frequency 1296579466 Hz quality -100
Comment 11 commit-hook freebsd_committer freebsd_triage 2018-11-05 22:54:56 UTC 
A commit references this bug:

Author: jhb
Date: Mon Nov  5 22:54:04 UTC 2018
New revision: 340170
URL: https://svnweb.freebsd.org/changeset/base/340170

Log:
  Add a custom implementation of cpu_lock_delay() for x86.

  Avoid using DELAY() since it can try to use spin locks on CPUs without
  a P-state invariant TSC.  For cpu_lock_delay(), always use the TSC if
  it exists (even if it is not P-state invariant) to delay for a
  microsecond.  If the TSC does not exist, read from I/O port 0x84 to
  delay instead.

  PR:		228768
  Reported by:	Roger Hammerstein <cheeky.m@live.com>
  Reviewed by:	kib
  MFC after:	3 days
  Differential Revision:	https://reviews.freebsd.org/D17851

Changes:
  head/sys/amd64/include/cpu.h
  head/sys/i386/include/cpu.h
  head/sys/x86/x86/delay.c
Comment 12 commit-hook freebsd_committer freebsd_triage 2018-11-08 22:40:38 UTC 
A commit references this bug:

Author: jhb
Date: Thu Nov  8 22:39:40 UTC 2018
New revision: 340269
URL: https://svnweb.freebsd.org/changeset/base/340269

Log:
  MFC 340164,340168,340170: Add custom cpu_lock_delay() for x86.

  340164:
  Add a KPI for the delay while spinning on a spin lock.

  Replace a call to DELAY(1) with a new cpu_lock_delay() KPI.  Currently
  cpu_lock_delay() is defined to DELAY(1) on all platforms.  However,
  platforms with a DELAY() implementation that uses spin locks should
  implement a custom cpu_lock_delay() doesn't use locks.

  340168:
  Add a delay_tsc() static function for when DELAY() uses the TSC.

  This uses slightly simpler logic than the existing code by using the
  full 64-bit counter and thus not having to worry about counter
  overflow.

  340170:
  Add a custom implementation of cpu_lock_delay() for x86.

  Avoid using DELAY() since it can try to use spin locks on CPUs without
  a P-state invariant TSC.  For cpu_lock_delay(), always use the TSC if
  it exists (even if it is not P-state invariant) to delay for a
  microsecond.  If the TSC does not exist, read from I/O port 0x84 to
  delay instead.

  PR:		228768
  Approved by:	re (gjb)

Changes:
_U  stable/12/
  stable/12/sys/amd64/include/cpu.h
  stable/12/sys/arm/include/cpu.h
  stable/12/sys/arm64/include/cpu.h
  stable/12/sys/i386/include/cpu.h
  stable/12/sys/kern/kern_mutex.c
  stable/12/sys/mips/include/cpu.h
  stable/12/sys/powerpc/include/cpu.h
  stable/12/sys/riscv/include/cpu.h
  stable/12/sys/sparc64/include/cpu.h
  stable/12/sys/x86/x86/delay.c
Comment 13 commit-hook freebsd_committer freebsd_triage 2018-11-08 22:43:44 UTC 
A commit references this bug:

Author: jhb
Date: Thu Nov  8 22:42:58 UTC 2018
New revision: 340270
URL: https://svnweb.freebsd.org/changeset/base/340270

Log:
  MFC 340164,340168,340170: Add custom cpu_lock_delay() for x86.

  340164:
  Add a KPI for the delay while spinning on a spin lock.

  Replace a call to DELAY(1) with a new cpu_lock_delay() KPI.  Currently
  cpu_lock_delay() is defined to DELAY(1) on all platforms.  However,
  platforms with a DELAY() implementation that uses spin locks should
  implement a custom cpu_lock_delay() doesn't use locks.

  340168:
  Add a delay_tsc() static function for when DELAY() uses the TSC.

  This uses slightly simpler logic than the existing code by using the
  full 64-bit counter and thus not having to worry about counter
  overflow.

  340170:
  Add a custom implementation of cpu_lock_delay() for x86.

  Avoid using DELAY() since it can try to use spin locks on CPUs without
  a P-state invariant TSC.  For cpu_lock_delay(), always use the TSC if
  it exists (even if it is not P-state invariant) to delay for a
  microsecond.  If the TSC does not exist, read from I/O port 0x84 to
  delay instead.

  PR:		228768

Changes:
_U  stable/11/
  stable/11/sys/amd64/include/cpu.h
  stable/11/sys/arm/include/cpu.h
  stable/11/sys/arm64/include/cpu.h
  stable/11/sys/i386/include/cpu.h
  stable/11/sys/kern/kern_mutex.c
  stable/11/sys/mips/include/cpu.h
  stable/11/sys/powerpc/include/cpu.h
  stable/11/sys/riscv/include/cpu.h
  stable/11/sys/sparc64/include/cpu.h
  stable/11/sys/x86/x86/delay.c
Comment 14 John Baldwin freebsd_committer freebsd_triage 2018-11-26 17:46:28 UTC 
*** Bug 225450 has been marked as a duplicate of this bug. ***