Bug 275086

Summary: Old i386 boot: CPU0: local APIC error 0x4 – Cannot boot any longer
Product: Base System Reporter: freebsd-bug-report.e3f7c7f2e19a <freebsd-bug-report.e3f7c7f2e19a>
Component: kernAssignee: Mark Johnston <markj>
Status: In Progress ---    
Severity: Affects Many People CC: ben, freebsd-bug-report.e3f7c7f2e19a, imp, markj
Priority: ---    
Version: 14.0-RELEASE   
Hardware: i386   
OS: Any   
Attachments:
Description Flags
Error messages on FreeBSD memstick boot. TOP: 14.0-RC2 and 13.2. BOTTOM: 13.1
none
fix none

Description freebsd-bug-report.e3f7c7f2e19a@inrobotico.com 2023-11-14 18:44:51 UTC 
Created attachment 246315 [details]
Error messages on FreeBSD memstick boot. TOP: 14.0-RC2 and 13.2. BOTTOM: 13.1

Hey,

Cannot boot FreeBSD-14.0-RC2-i386-mini-memstick.img on an old AMD i386 LOL

CPU0: local APIC error 0x4
CPU0: local APIC error 0x4
CPU0: local APIC error 0x4

I got this shame error with both 14.0-RC2 and 13.2 (see attachment image TOP pane).

Also on 13.1 cannot boot (see attachment image BOTTOM pane).

Seems until now the 12.4 is the last one working. (Time to go back to bloated Debian)

Hope this helps.

Thanks!

== some system info ==

FreeBSD 12.4-RELEASE r372781 GENERIC i386
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): resolution 640x480
CPU: AMD Sempron(tm) 3000+ (1999.39-MHz 686-class CPU)
  Origin="AuthenticAMD"  Id=0x6a0  Family=0x6  Model=0xa  Stepping=0
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
  AMD Features=0xc0480800<SYSCALL,MP,MMX+,3DNow!+,3DNow!>
real memory  = 1073741824 (1024 MB)
avail memory = 1016045568 (968 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <AMIINT VIA_K7  >
Comment 1 Warner Losh freebsd_committer freebsd_triage 2023-11-14 21:48:45 UTC 
Likely due to a change in a ACPICA import.
Honestly: you'd likely need to find which one by bisecting the road from stable/12 branch point to stable/13, then figure out what caused the breakage in that... then figure out a fix... and ideally get it upstreamed...
This sort of thing is impossible to remotely debug and/or debug without strong motivation to make it work.
Comment 2 freebsd-bug-report.e3f7c7f2e19a@inrobotico.com 2023-11-15 22:13:42 UTC 
My bad. 

As per documentation somewhere, added hint.apic.0.disabled="1" to the /boot/loader.conf, and this issue was solved.

However, /boot/kernel/zfs.ko et co. were not upgraded, when skipping major releases to 13.0: https://forums.freebsd.org/threads/freebsd-13-1-upgrade-issue.85190/post-628769
Comment 3 ben 2025-01-01 11:44:29 UTC 
Created attachment 256313 [details]
fix

The offending commit appears to be 279cd05b7e4d.

0x4 in the ECR is apparently:

> Bit 2: Send Accept Error.
> Set when the local APIC detects that a message it sent was not accepted by any > APIC on the APIC bus. Used only on P6 family and Pentium processors.

I guess pre-P4 systems don't like IPIs fired off into oblivion.  Attached patch adds a check for multiple CPUs to match the one in smp_targeted_tlb_shootdown(), which seems to be the only other caller.  Tested on single-CPU 440BX and multiprocessor i840.
Comment 4 commit-hook freebsd_committer freebsd_triage 2025-01-07 18:14:53 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=254a2b767f9a39f1541e0a07a70bbe269e86ad70

commit 254a2b767f9a39f1541e0a07a70bbe269e86ad70
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2025-01-07 17:58:58 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2025-01-07 18:13:54 +0000

    x86: Short-circuit ipi_all_but_self() on UP systems

    Apparently this is required on old intel hw, see
    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275086#c3

    PR:             275086
    Reviewed by:    mav, kib
    Fixes:          279cd05b7e4d ("Use APIC_IPI_DEST_OTHERS for bitmapped IPIs too.")
    MFC after:      1 week
    Diagnosed by:   Ben Wilber <ben@desync.com>
    Differential Revision:  https://reviews.freebsd.org/D48361

 sys/x86/x86/mp_x86.c | 3 +++
 1 file changed, 3 insertions(+)
Comment 5 Mark Johnston freebsd_committer freebsd_triage 2025-01-07 18:15:31 UTC 
(In reply to ben from comment #3)
Thanks for the analysis.  I committed a modified version of the patch.
Comment 6 Mark Johnston freebsd_committer freebsd_triage 2025-01-07 18:15:56 UTC 
Reopening since it's a legitimate bug.