Bug 250755

Summary: 12.2-RELEASE i386 GENERIC kernel fails to load
Product: Base System Reporter: p5B2EA84B3
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Only Me CC: arichardson, dim, emaste, gjb, kib
Priority: ---    
Version: 12.2-RELEASE   
Hardware: i386   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257149
Attachments:
Description Flags
screenshot failing to load 12-RELEASE i386 GENERIC kernel none

Description p5B2EA84B3 2020-10-31 14:45:52 UTC
After trying to upgrade an 586 CPU system from 12.1-RELEASE to 12.2-RELEASE the deployment via installkernel and installworld failed. 

Having read https://www.freebsd.org/releases/12.2R/relnotes.html I understood that before FreeBSD-13.0 this system would not be affected and it is still the same like in https://www.freebsd.org/releases/12.1R/relnotes.html

While other i366 systems could be upgraded without any problems from the same buildworld/buildkernel run I got stuck on finding any changes/errors on the build/deploy procedure here.

Finally I extracted the RELEASE-12.2 GENERIC kernel
kernel.txz	d578b60e98b62f3f412068655f365a816e4c8fe9733ea45a9c6d14cb42ff64d4	899	kernel	"Kernel (MANDATORY)"	on

on the system to exclude any potential local errors here.
Rebooting the 586 system resulted in a BTX halted. See screenshot attached.
Comment 1 p5B2EA84B3 2020-10-31 14:47:48 UTC
Created attachment 219238 [details]
screenshot failing to load 12-RELEASE i386 GENERIC kernel
Comment 2 Konstantin Belousov freebsd_committer freebsd_triage 2020-10-31 16:02:32 UTC
It seems that 12.2 compiler on i386 targets 686 by default.  Your faulting
instruction is cmovne.

We proclaimed that 13.0 will switch to 686 by default, but it seems that the change
leaked into 12.2, perhaps with clang 10.0.1 import.  I just verified that cc from
12.2-RELEASE/i386 does generate cmov* instructions with default -march.
Comment 3 p5B2EA84B3 2020-10-31 23:28:22 UTC
Can a makebuildworld/kernel be compiled with -march=i586 ?
If that can be done, how should I try this?
Sorry limited knowledge here on this matter ;)
Comment 4 Dimitry Andric freebsd_committer freebsd_triage 2020-11-01 12:04:09 UTC
The bump to i686 was originally committed to head in base r353936, after base r352030 bumped it to i586 at first. On 2020-01-07, so already 11 months ago, this was all merged back to stable/12 in base r356460, as part of the clang 9.0.0 merge. Similarly, on 2020-05-05 for stable/11, in base r360658.

In any case, I have tried buiding stable/12 world with base r353936 and base r352030 reverted, so effectively going back to i486 as default:

Index: contrib/llvm-project/clang/lib/Driver/ToolChains/Arch/X86.cpp
===================================================================
--- contrib/llvm-project/clang/lib/Driver/ToolChains/Arch/X86.cpp       (revision 367228)
+++ contrib/llvm-project/clang/lib/Driver/ToolChains/Arch/X86.cpp       (working copy)
@@ -94,7 +94,7 @@

   switch (Triple.getOS()) {
   case llvm::Triple::FreeBSD:
-    return "i686";
+    //return "i686"; FALLTHROUGH
   case llvm::Triple::NetBSD:
   case llvm::Triple::OpenBSD:
     return "i486";

However, this leads to a -Werror problem again:

--- atomic.o ---
/usr/src/contrib/llvm-project/compiler-rt/lib/builtins/atomic.c:178:3: error: large atomic operation may incur significant performance penalty [-Werror,-Watomic-alignment]
  LOCK_FREE_CASES();
  ^
/usr/src/contrib/llvm-project/compiler-rt/lib/builtins/atomic.c:160:9: note: expanded from macro 'LOCK_FREE_CASES'
        LOCK_FREE_ACTION(uint64_t);                                            \
        ^
/usr/src/contrib/llvm-project/compiler-rt/lib/builtins/atomic.c:176:21: note: expanded from macro 'LOCK_FREE_ACTION'
  *((type *)dest) = __c11_atomic_load((_Atomic(type) *)src, model);            \
                    ^
1 error generated.
*** [atomic.o] Error code 1

Originally we have -Wno-atomic-alignment in libcompiler_rt's Makefile, but we removed it base r364782, because:
> After base r364753, there should be no need to suppress -Watomic-alignment
> warnings anymore for compiler-rt's atomic.c. This occurred because the
> IS_LOCK_FREE_8 macro was not correctly defined to 0 for mips, and this
> caused the compiler to emit a runtime call to __atomic_is_lock_free(),
> and that triggers the warning.

However, the changes in base r364753 are not enough for this particular warning.  Alex, you also worked on these warnings upstream, right? I think we could probably add __i386__ to the following line in contrib/llvm-project/compiler-rt/lib/builtins/atomic.c to work around it (optionally with a check if the target CPU is really < i586):

/// 32 bit MIPS and PowerPC don't support 8-byte lock_free atomics
#if defined(__mips__) || (!defined(__powerpc64__) && defined(__powerpc__))
Comment 5 p5B2EA84B3 2020-11-01 12:11:06 UTC
Meantime I tried this:

make buildworld CPUTYPE=i586
and
make buildkernel CPUTYPE=i586
both completed.

But 
make installkernel CPUTYPE=i586

failed with 
*** Signal 4

Any hints please?
Comment 6 Konstantin Belousov freebsd_committer freebsd_triage 2020-11-01 12:44:31 UTC
(In reply to Dimitry Andric from comment #4)
I was just surprised that this change went into 12.x, at least without loud
announcement.  More, I suggest to take a look at
https://www.freebsd.org/releases/12.2R/relnotes.html
11.1. Default CPUTYPE Change
Comment 7 p5B2EA84B3 2022-02-12 12:06:14 UTC
What is the last RELEASE or STABLE revision usable for 586 hardware?
I still need to run FreeBSD on that machines and just replacing hardware is not an option.