Bug 233534

Summary: Panic on booting 12 on Jetway JNF9HG
Product: Base System Reporter: Christian Ullrich <chris>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed Not A Bug    
Severity: Affects Some People CC: abob, emaste, janm, roel
Priority: --- Keywords: regression
Version: 12.0-STABLE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
Verbose boot on stable/11 none

Description Christian Ullrich 2018-11-26 15:44:51 UTC 
Created attachment 199572 [details]
Verbose boot on stable/11

I have a reproducible panic booting 12-RC2 as well as stable/12, 2cf4a7e0d8 
from 2018-11-23, on a Jetway JNF9HG board, Celeron N2930 CPU, booting with 
UEFI. The same box has no problems with stable/11 18f83cbbc9 from 2018-11-22.

This is booting FreeBSD-12.0-RC2-amd64-mini-memstick.img:

 >> FreeBSD EFI boot block
    Loader path: /boot/loader.efi

    Initializing modules: ZFS UFS
    Load Path: \EFI\BOOT\BOOTX64.EFI
    Load Device: 
PciRoot(0x0)/Pci(0x14,0x0)/USB(0x1,0x0)/USB(0x3,0x0)/HD(1,MBR,0x90909090,0x1,0x640)
    BootCurrent: 0009
    BootOrder: 0007 0008 0005 0006 0003 0004 0009[*] 0001 0002
    Probing 11 block devices.....*....+..+ done
     ZFS found the following pools: zroot
     UFS found 1 partition
Consoles: EFI console
FreeBSD/amd64 EFI loader, Revision 1.1

    Command line arguments: loader.efi
    EFI version: 2.31
    EFI Firmware: American Megatrends (rev 5.09)
    Console: efi (0x20000000)
    Load Path: HD(2,MBR,0x90909090,0x641,0xB7990)
    Load Device: 
PciRoot(0x0)/Pci(0x14,0x0)/USB(0x1,0x0)/USB(0x3,0x0)/HD(2,MBR,0x90909090,0x641,0xB7990)
    BootCurrent: 0009
    BootOrder: 0007 0008 0005 0006 0003 0004 0009[*] 0001 0002
    BootInfo Path: 
PciRoot(0x0)/Pci(0x14,0x0)/USB(0x1,0x0)/USB(0x3,0x0)/HD(1,MBR,0x90909090,0x1,0x640)
    BootInfo Path: 
VenHw(2D6447EF-3BC9-41A0-AC19-4D51D01B4CE6,4300380036003000300030003800380036003100340044004500460036003000360041003100350041003100420035000000)
Ignoring Boot0009: No Media Path
Trying ESP: 
PciRoot(0x0)/Pci(0x14,0x0)/USB(0x1,0x0)/USB(0x3,0x0)/HD(2,MBR,0x90909090,0x641,0xB7990)
Setting currdev to disk0p2:
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
-
Loading kernel...
/boot/kernel/kernel text=0x1677738 data=0x1cd288+0x768b40 
syms=[0x8+0x174cd8+0x8+0x19224a]
Loading configured modules...
can't find '/boot/entropy'
Start @ 0xffffffff80341000 ...
EFI framebuffer information:
addr, size     0xc0000000, 0x1d4c00
dimensions     800 x 600
stride         800
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
---<<BOOT>>---
Table 'FACP' at 0xb95fa1b8
Table 'APIC' at 0xb95fa2c8
APIC: Found table at 0xb95fa2c8
APIC: Using the MADT enumerator.
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RC2 r340839 GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on 
LLVM 6.0.1)
Table 'FACP' at 0xb95fa1b8
Table 'APIC' at 0xb95fa2c8
Table 'FPDT' at 0xb95fa350
Table 'MCFG' at 0xb95fa398
Table 'LPIT' at 0xb95fa3d8
Table 'HPET' at 0xb95fa4e0
Table 'SSDT' at 0xb95fa518
Table 'SSDT' at 0xb95fac80
Table 'SSDT' at 0xb95faf10
Table 'UEFI' at 0xb95fb090
ACPI: No SRAT table found
PPIM 0: PA=0xc0000000, VA=0xffffffff82810000, size=0x1d5000, mode=0x1
pmap: large map 8 PML4 slots (4096 Gb)
VT(efifb): resolution 800x600
Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff8263e000.
Table 'FACP' at 0xb95fa1b8
FACP: Found table at 0xb95fa1b8
Calibrating TSC clock ... TSC clock: 1833378866 Hz
CPU: Intel(R) Celeron(R) CPU  N2930  @ 1.83GHz (1833.38-MHz K8-class CPU)
   Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
   AMD Features2=0x101<LAHF,Prefetch>
   Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
   VT-x: (disabled in BIOS) Basic Features=0xda0400<SMM,INS/OUTS,TRUE>
         Pin-Based Controls=0x7f<ExtINT,NMI,VNMI,PreTmr>
         Primary Processor 
Controls=0xfff9fffe<INTWIN,TSCOff,HLT,INVLPG,MWAIT,RDPMC,RDTSC,CR3-LD,CR3-ST,CR8-LD,CR8-ST,TPR,NMIWIN,MOV-DR,IO,IOmap,MTF,MSRmap,MONITOR,PAUSE>
         Secondary Processor 
Controls=0x28ef<APIC,EPT,DT,RDTSCP,VPID,WBINVD,UG,RDRAND,VMFUNC>
         Exit Controls=0xda0400<PAT-LD,EFER-SV,PTMR-SV>
         Entry Controls=0xda0400
         EPT Features=0x6114141<XO,PW4,UC,WB,2M,INVEPT,single,all>
         VPID Features=0xf01<INVVPID,individual,single,all,single-globals>
   TSC: P-state invariant, performance statistics
DTLB: 4k pages, fully associative, 32 entries
Data TLB: 4 KB Pages, 4-way set associative, 128 entries
Instruction TLB: 4 KByte pages, fully associative, 48 entries
DTLB: 4 KByte/2 MByte pages, 4-way associative, 16 entries
L2 cache: 1024 kbytes, 16-way associative, 64 bytes/line
real memory  = 8589934592 (8192 MB)
Physical memory chunk(s):
0x0000000000010000 - 0x000000000003efff, 192512 bytes (47 pages)
0x0000000000040000 - 0x000000000009efff, 389120 bytes (95 pages)
0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages)
0x0000000002800000 - 0x000000001fffffff, 494927872 bytes (120832 pages)
0x0000000020100000 - 0x00000000b95bbfff, 2571878400 bytes (627900 pages)
0x00000000b9b86000 - 0x00000000b9b86fff, 4096 bytes (1 pages)
0x00000000b9bc9000 - 0x00000000b9d37fff, 1503232 bytes (367 pages)
0x00000000b9ffd000 - 0x00000000b9ffffff, 12288 bytes (3 pages)
0x0000000100000000 - 0x0000000231aaefff, 5128253440 bytes (1252015 pages)
avail memory = 8164179968 (7785 MB)
intel stolen mem: base rxbb000000 size 64 MB
MADT: Found CPU APIC ID 0 ACPI ID 1: enabled
SMP: Added CPU 0 (AP)
MADT: Found CPU APIC ID 2 ACPI ID 2: enabled
SMP: Added CPU 2 (AP)
MADT: Found CPU APIC ID 4 ACPI ID 3: enabled
SMP: Added CPU 4 (AP)
MADT: Found CPU APIC ID 6 ACPI ID 4: enabled
SMP: Added CPU 6 (AP)
Event timer "LAPIC" quality 600
LAPIC: ipi_wait() us multiplier 34 (r 5300196 tsc 1833378866)
ACPI APIC Table: <_ _>
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
Package ID shift: 4
L2 cache ID shift: 2
L1 cache ID shift: 1
Core ID shift: 1
INTR: Adding local APIC 2 as a target
INTR: Adding local APIC 4 as a target
INTR: Adding local APIC 6 as a target
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
Package HW ID = 0
         Core HW ID = 0
                 CPU0 (BSP): APIC ID: 0
         Core HW ID = 1
                 CPU1 (AP): APIC ID: 2
         Core HW ID = 2
                 CPU2 (AP): APIC ID: 4
         Core HW ID = 3
                 CPU3 (AP): APIC ID: 6
APIC: CPU 0 has ACPI ID 1
APIC: CPU 1 has ACPI ID 2
APIC: CPU 2 has ACPI ID 3
APIC: CPU 3 has ACPI ID 4
Pentium Pro MTRR support enabled
x86bios:  IVT 0x000000-0x0004ff at 0xfffff80000000000
x86bios: SSEG 0x03e000-0x03efff at 0xfffffe000248c000
x86bios: EBDA 0x09d000-0x09ffff at 0xfffff8000009d000
x86bios:  ROM 0x0a0000-0x0fefff at 0xfffff800000a0000
arc4random: no preloaded entropy cache
VIMAGE (virtualized network stack) enabled
ULE: setup cpu 0
ULE: setup cpu 1
ULE: setup cpu 2
ULE: setup cpu 3
ACPI: RSDP 0x00000000B95EF000 000024 (v02 _     )
ACPI: XSDT 0x00000000B95EF078 000074 (v01 _      _        01072009 AMI 
00010013)
ACPI: FACP 0x00000000B95FA1B8 00010C (v05 _      _        01072009 AMI 
00010013)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 
128/32 (20181003/tbfadt-748)
ACPI: DSDT 0x00000000B95EF180 00B038 (v02 _      _        01072009 INTL 
20120913)
ACPI: FACS 0x00000000B9760F80 000040
ACPI: APIC 0x00000000B95FA2C8 000084 (v03 _      _        01072009 AMI 
00010013)
ACPI: FPDT 0x00000000B95FA350 000044 (v01 _      _        01072009 AMI 
00010013)
ACPI: MCFG 0x00000000B95FA398 00003C (v01 _      _        01072009 MSFT 
00000097)
ACPI: LPIT 0x00000000B95FA3D8 000104 (v01 _      _        00000003 VLV2 
0100000D)
ACPI: HPET 0x00000000B95FA4E0 000038 (v01 _      _        01072009 AMI. 
00000005)
ACPI: SSDT 0x00000000B95FA518 000763 (v01 PmRef  CpuPm    00003000 INTL 
20061109)
ACPI: SSDT 0x00000000B95FAC80 000290 (v01 PmRef  Cpu0Tst  00003000 INTL 
20061109)
ACPI: SSDT 0x00000000B95FAF10 00017A (v01 PmRef  ApTst    00003000 INTL 
20061109)
ACPI: UEFI 0x00000000B95FB090 000042 (v01 _      _        00000000 
00000000)
MADT: Found IO APIC ID 1, Interrupt 0 at 0xfec00000
ioapic0: ver 0x20 maxredir 0x56
ioapic0: Routing external 8259A's -> intpin 0
lapic0: Routing NMI -> LINT1
lapic0: LINT1 trigger: edge
lapic0: LINT1 polarity: high
lapic2: Routing NMI -> LINT1
lapic2: LINT1 trigger: edge
lapic2: LINT1 polarity: high
lapic4: Routing NMI -> LINT1
lapic4: LINT1 trigger: edge
lapic4: LINT1 polarity: high
lapic6: Routing NMI -> LINT1
lapic6: LINT1 trigger: edge
lapic6: LINT1 polarity: high
MADT: Interrupt override: source 0, irq 2
ioapic0: Routing IRQ 0 -> intpin 2
MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
ioapic0 <Version 2.0> irqs 0-86 on motherboard
lapic: Divisor 2, Frequency 41667702 Hz
lapic: deadline tsc mode, Frequency 1833378866 Hz
cpu0 BSP:
      ID: 0x00000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
SMP: AP CPU #2 Launched!
cpu2 AP:
      ID: 0x04000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
SMP: AP CPU #3 Launched!
cpu3 AP:
      ID: 0x06000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
SMP: AP CPU #1 Launched!
cpu1 AP:
      ID: 0x02000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
SMP: passed TSC synchronization test
Timecounter "TSC" frequency 1833378866 Hz quality 1000
snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5 
feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
wlan: <802.11 Link Layer>
random: entropy device external interface
EFI Runtime entry 30 mapping attributes unsupported
EFI Runtime entry 31 mapping attributes unsupported
EFI Runtime entry 32 mapping attributes unsupported
kbd: new array size 4
kbd1 at kbdmux0
mem: <memory>
netmap: loaded module
null: <full device, null device, zero device>
nfslock: pseudo-device
[ath_hal] loaded
crypto: <crypto core>
module_register_init: MOD_LOAD (vesa, 0xffffffff810f8750, 0) error 19
io: <I/O>
tcp_log: tcp_log device
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
hptnr: R750/DC7280 controller driver v1.1.5
hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2
hpt27xx: RocketRAID 27xx controller driver v1.2.8
nexus0
kernel trap 12 with interrupts disabled
kernel trap 12 with interrupts disabled
EFI rt_gettime call faulted, error 14
efirtc0: cannot read EFI realtime clock, error 14
cryptosoft0: <software crypto> on motherboard
crypto: assign cryptosoft0 driver id 0, flags 0x6000000
crypto: cryptosoft0 registers alg 1 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 2 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 3 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 4 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 5 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 16 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 6 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 7 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 32 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 18 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 19 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 20 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 8 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 15 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 9 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 10 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 13 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 14 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 34 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 35 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 36 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 37 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 11 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 22 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 23 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 25 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 24 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 26 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 27 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 28 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 21 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 17 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 29 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 30 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 31 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 38 flags 0 maxoplen 0
acpi0: <_> on motherboard
ACPI: 4 ACPI AML tables successfully acquired and loaded
PCIe: Memory Mapped configuration base @ 0xe0000000
panic: smp_targeted_tlb_shootdown: interrupts disabled
cpuid = 3
time = 1
KDB: stack backtrace:
#0 0xffffffff80be74a7 at kdb_backtrace+0x67
#1 0xffffffff80b9b093 at vpanic+0x1a3
#2 0xffffffff80b9aee3 at panic+0x43
#3 0xffffffff811eda2f at smp_targeted_tlb_shootdown+0x40f
#4 0xffffffff811ed60d at smp_masked_invltlb+0x3d
#5 0xffffffff8105d5c5 at pmap_invalidate_range+0x1b5
#6 0xffffffff8106a429 at pmap_change_attr_locked+0x859
#7 0xffffffff81069804 at pmap_mapdev_internal+0x424
#8 0xffffffff81075ed0 at pcie_cfgregopen+0x60
#9 0xffffffff80451f10 at acpi_attach+0x390
#10 0xffffffff80bd6efc at device_attach+0x3ec
#11 0xffffffff80bd81dc at bus_generic_attach+0x5c
#12 0xffffffff80bd6efc at device_attach+0x3ec
#13 0xffffffff80bd88b8 at bus_generic_new_pass+0x118
#14 0xffffffff80bda577 at root_bus_configure+0x77
#15 0xffffffff811dbce9 at configure+0x9
#16 0xffffffff80b31a78 at mi_startup+0x118
#17 0xffffffff8034102c at btext+0x2c
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort


A successful boot log from stable/11 is attached for comparison, and because it is to big to post here.
Comment 1 Christian Ullrich 2018-12-06 13:04:22 UTC 
Data point: Same behavior with 12-RC3.
Comment 2 Christian Ullrich 2018-12-13 15:05:18 UTC 
Ugh. Found it. The hardware needs "efi.rt.disabled" set, that's all.

I just spent two days bisecting around, and at some point I became aware of this tunable, except in that commit it was still called "efi.rt_disabled", with an underscore. I then obviously missed when it was renamed in r336924, and that invalidated the bisect results.

Anyway, with the variable set, and correctly spelled, releng/12.0 boots perfectly fine.
Comment 3 Ed Maste freebsd_committer freebsd_triage 2018-12-17 17:55:29 UTC 
I'm glad you've found a workaround, although we should really fix the underlying issue (at least adding a quirk to disable it automatically).
Comment 4 Christian Ullrich 2018-12-17 20:22:41 UTC 
I thought, since the option is documented in UPDATING, that it was meant to be this way. Should I open a new bug for creating this quirk specifically for my hardware?
Comment 5 abob 2019-02-21 17:51:18 UTC 
I confirm exactly the same problem.
Motherboard with J1900 CPU. FreeBSD-12.0-RELEASE-amd64.

Try on 2019-Feb-21 FreeBSD-12.0-STABLE-amd64 and FreeBSD-13.0-CURRENT-amd64.
With same result.

Workaround work for me:

add to /boot/loader.conf
efi.rt.disabled=1
Comment 6 roel 2019-05-23 10:45:34 UTC 
The proposed workaround did not work for me. Boot still freezes at the same point, kernel never loads. I've rolled back to 11.2-RELEASE.

Other bugs with seem to be describing exactly the same issue:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221376

and

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235017
Comment 7 Christian Ullrich 2019-05-23 11:10:11 UTC 
Neither your issue nor either of the two you linked look very similar to this one. Here, the kernel loads and starts just fine, and then it crashes while initializing the hardware. In particular, your kernel apparently just freezes, i.e. stops. Mine panics, producing a backtrace.

Do you have a backtrace that passes through any of the functions mine does (in frame #3 and below)?
Comment 8 Jan Martin Mikkelsen 2019-06-04 15:37:07 UTC 
For completeness, the underlying problem is fixed in r348539.