Bug 257132

Summary: clang 12.0.1 fails to cross-compile kern_mbuf - amd64 host, i386 target
Product: Base System Reporter: imbutler <imb>
Component: binAssignee: Warner Losh <imp>
Status: Closed FIXED    
Severity: Affects Only Me CC: dim, imb, imp, misho
Priority: --- Keywords: regression
Version: CURRENTFlags: koobs: mfc-stable13+
koobs: mfc-stable12+
Hardware: amd64   
OS: Any   
See Also: https://bugs.llvm.org/show_bug.cgi?id=51066
Attachments:
Description Flags
preprocessed text (gzipped) none

Description imbutler 2021-07-12 14:41:44 UTC
When cross-building for i386 target on amd64, clang 12.0.1 trips over an assertion:

Assertion failed: ((EltVT.getSizeInBits() >= 64 || (uint64_t)((int64_t)Val >> EltVT.getSizeInBits()) + 1 < 2) && "getConstant with a uint64_t value that doesn't fit in the type!"), function getConstant, file /usr/local/release-builds/i386/usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp, line 1337.
PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: cc -target i386-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/i386.i386/tmp -B/usr/obj/usr/src/i386.i386/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -MD -MF.depend.kern_mbuf.o -MTkern_mbuf.o -fdebug-prefix-map=./machine=/usr/src/sys/i386/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mno-mmx -mno-sse -msoft-float -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error=tautological-compare -Wno-error=empty-body -Wno-error=parentheses-equality -Wno-error=unused-function -Wno-error=pointer-sign -Wno-error=shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -mno-aes -mno-avx -std=iso9899:1999 -Werror /usr/src/sys/kern/kern_mbuf.c
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'Function Pass Manager' on module '/usr/src/sys/kern/kern_mbuf.c'.
4.      Running pass 'X86 DAG->DAG Instruction Selection' on function '@m_free'
#0 0x00000000047256d1 PrintStackTrace /usr/local/release-builds/i386/usr/src/contrib/llvm-project/llvm/lib/Support/Unix/Signals.inc:565:13
#1 0x00000000047238a5 RunSignalHandlers /usr/local/release-builds/i386/usr/src/contrib/llvm-project/llvm/lib/Support/Signals.cpp:72:18
#2 0x00000000046bd49e HandleCrash /usr/local/release-builds/i386/usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:75:5
#3 0x00000000046bd623 CrashRecoverySignalHandler /usr/local/release-builds/i386/usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:0:51
#4 0x0000000805d03eb0 handle_signal /usr/local/release-builds/i386/usr/src/lib/libthr/thread/thr_sig.c:0:3
cc: error: clang frontend command failed with exit code 134 (use -v to see invocation)
FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-rc2-0-ge7dac564cd0e)
Target: i386-unknown-freebsd14.0
Thread model: posix
InstalledDir: /usr/bin
cc: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
cc: note: diagnostic msg: /tmp/kern_mbuf-e8afab.c
cc: note: diagnostic msg: /tmp/kern_mbuf-e8afab.sh
cc: note: diagnostic msg: 

preprocessed source (as above) attached
shell script (as above) contains:

# Crash reproducer for FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-rc2-0-ge7dac564cd0e)
# Driver args: "--target=i386-unknown-freebsd14.0" "--sysroot=/usr/obj/usr/src/i386.i386/tmp" "-B" "/usr/obj/usr/src/i386.i386/tmp/usr/bin" "-c" "-O2" "-pipe" "-fno-strict-aliasing" "-nostdinc" "-I" "." "-I" "/usr/src/sys" "-I" "/usr/src/sys/contrib/ck/include" "-I" "/usr/src/sys/contrib/libfdt" "-D" "_KERNEL" "-D" "HAVE_KERNEL_OPTION_HEADERS" "-include" "opt_global.h" "-fno-common" "-MD" "-MF" ".depend.kern_mbuf.o" "-MT" "kern_mbuf.o" "-fdebug-prefix-map=./machine=/usr/src/sys/i386/include" "-fdebug-prefix-map=./x86=/usr/src/sys/x86/include" "-mno-mmx" "-mno-sse" "-msoft-float" "-ffreestanding" "-fwrapv" "-fstack-protector" "-Wall" "-Wredundant-decls" "-Wnested-externs" "-Wstrict-prototypes" "-Wmissing-prototypes" "-Wpointer-arith" "-Wcast-qual" "-Wundef" "-Wno-pointer-sign" "-D" "__printf__=__freebsd_kprintf__" "-Wmissing-include-dirs" "-fdiagnostics-show-option" "-Wno-unknown-pragmas" "-Wno-error=tautological-compare" "-Wno-error=empty-body" "-Wno-error=parentheses-equality" "-Wno-error=unused-function" "-Wno-error=pointer-sign" "-Wno-error=shift-negative-value" "-Wno-address-of-packed-member" "-Wno-format-zero-length" "-mno-aes" "-mno-avx" "-std=iso9899:1999" "-Werror" "/usr/src/sys/kern/kern_mbuf.c"
# Original command:  "/usr/bin/cc" "-cc1" "-triple" "i386-unknown-freebsd14.0" "-emit-obj" "--mrelax-relocations" "-disable-free" "-main-file-name" "kern_mbuf.c" "-mrelocation-model" "static" "-mframe-pointer=all" "-relaxed-aliasing" "-fno-rounding-math" "-mconstructor-aliases" "-ffreestanding" "-target-cpu" "i686" "-target-feature" "-mmx" "-target-feature" "-sse" "-target-feature" "-aes" "-target-feature" "-avx" "-no-implicit-float" "-tune-cpu" "generic" "-fno-split-dwarf-inlining" "-debugger-tuning=gdb" "-nostdsysteminc" "-nobuiltininc" "-resource-dir" "/usr/lib/clang/12.0.1" "-dependency-file" ".depend.kern_mbuf.o" "-MT" "kern_mbuf.o" "-sys-header-deps" "-include" "opt_global.h" "-I" "." "-I" "/usr/src/sys" "-I" "/usr/src/sys/contrib/ck/include" "-I" "/usr/src/sys/contrib/libfdt" "-D" "_KERNEL" "-D" "HAVE_KERNEL_OPTION_HEADERS" "-D" "__printf__=__freebsd_kprintf__" "-isysroot" "/usr/obj/usr/src/i386.i386/tmp" "-O2" "-Wall" "-Wredundant-decls" "-Wnested-externs" "-Wstrict-prototypes" "-Wmissing-prototypes" "-Wpointer-arith" "-Wcast-qual" "-Wundef" "-Wno-pointer-sign" "-Wmissing-include-dirs" "-Wno-unknown-pragmas" "-Wno-error=tautological-compare" "-Wno-error=empty-body" "-Wno-error=parentheses-equality" "-Wno-error=unused-function" "-Wno-error=pointer-sign" "-Wno-error=shift-negative-value" "-Wno-address-of-packed-member" "-Wno-format-zero-length" "-Werror" "-std=iso9899:1999" "-fdebug-compilation-dir" "/usr/obj/usr/src/i386.i386/sys/SARAH" "-fdebug-prefix-map=./machine=/usr/src/sys/i386/include" "-fdebug-prefix-map=./x86=/usr/src/sys/x86/include" "-ferror-limit" "19" "-fwrapv" "-stack-protector" "1" "-fgnuc-version=4.2.1" "-vectorize-loops" "-vectorize-slp" "-faddrsig" "-o" "kern_mbuf.o" "-x" "c" "/usr/src/sys/kern/kern_mbuf.c"
 "/usr/bin/cc" "-cc1" "-triple" "i386-unknown-freebsd14.0" "-emit-obj" "--mrelax-relocations" "-disable-free" "-main-file-name" "kern_mbuf.c" "-mrelocation-model" "static" "-mframe-pointer=all" "-relaxed-aliasing" "-fno-rounding-math" "-mconstructor-aliases" "-ffreestanding" "-target-cpu" "i686" "-target-feature" "-mmx" "-target-feature" "-sse" "-target-feature" "-aes" "-target-feature" "-avx" "-no-implicit-float" "-tune-cpu" "generic" "-fno-split-dwarf-inlining" "-debugger-tuning=gdb" "-nostdsysteminc" "-nobuiltininc" "-sys-header-deps" "-D" "_KERNEL" "-D" "HAVE_KERNEL_OPTION_HEADERS" "-D" "__printf__=__freebsd_kprintf__" "-O2" "-Wall" "-Wredundant-decls" "-Wnested-externs" "-Wstrict-prototypes" "-Wmissing-prototypes" "-Wpointer-arith" "-Wcast-qual" "-Wundef" "-Wno-pointer-sign" "-Wmissing-include-dirs" "-Wno-unknown-pragmas" "-Wno-error=tautological-compare" "-Wno-error=empty-body" "-Wno-error=parentheses-equality" "-Wno-error=unused-function" "-Wno-error=pointer-sign" "-Wno-error=shift-negative-value" "-Wno-address-of-packed-member" "-Wno-format-zero-length" "-Werror" "-std=iso9899:1999" "-fdebug-prefix-map=./machine=/usr/src/sys/i386/include" "-fdebug-prefix-map=./x86=/usr/src/sys/x86/include" "-ferror-limit" "19" "-fwrapv" "-stack-protector" "1" "-fgnuc-version=4.2.1" "-vectorize-loops" "-vectorize-slp" "-faddrsig" "-x" "c" "kern_mbuf-e8afab.c"
Comment 1 imbutler 2021-07-12 14:43:39 UTC
Created attachment 226396 [details]
preprocessed text (gzipped)
Comment 2 Dimitry Andric freebsd_committer freebsd_triage 2021-07-12 18:02:19 UTC
This appears to be caused by some form of static assertion, but I haven't been able to pinpoint the line(s) in the kern_mbuf.c file. I also wonder why it doesn't seem to happen "natively" on i386?

In any case, I have minimized the test case and submitted it upstream as:
https://bugs.llvm.org/show_bug.cgi?id=51066
Comment 3 Dimitry Andric freebsd_committer freebsd_triage 2021-07-12 18:36:20 UTC
It turns out that this is due some code generated in $OBJDIR/offset.inc, during an amd64 to i386 cross-build:

# 33 "/usr/src/sys/sys/kpilite.h"
# 1 "./offset.inc" 1
#ifndef _OFFSET_INC_
#define _OFFSET_INC_
#if 0 /* disabled by -frewrite-includes */
#if !defined(GENOFFSET) && (!defined(KLD_MODULE) || defined(KLD_TIED))
#endif
#endif /* disabled by -frewrite-includes */
#if 1 /* evaluated by -frewrite-includes */
# 4 "./offset.inc"
struct thread_lite {
        u_char  pad_td_priority[0x27e - 0];
        u_char  td_priority;
        u_char  pad_td_critnest[0x30c - (0x27e + sizeof(u_char))];
        u_int   td_critnest;
        u_char  pad_td_owepreempt[0xb8 - (0x30c + sizeof(u_int))];
        u_char  td_owepreempt;
        u_char  pad_td_pinned[0xd8 - (0xb8 + sizeof(u_char))];
        int     td_pinned;
};
#endif

E.g. this defines tpad_td_owepreempt as a negative size:

27: error: array is too large (18446744073709551016 elements)
        u_char  pad_td_owepreempt[0xb8 - (0x30c + sizeof(u_int))];
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For some reason, generating this offset.inc file on a native i386 build results in a very different struct:

struct thread_lite {
        u_char  pad_td_owepreempt[0xb8 - 0];
        u_char  td_owepreempt;
        u_char  pad_td_pinned[0xd8 - (0xb8 + sizeof(u_char))];
        int     td_pinned;
        u_char  pad_td_priority[0x27e - (0xd8 + sizeof(int))];
        u_char  td_priority;
        u_char  pad_td_critnest[0x30c - (0x27e + sizeof(u_char))];
        u_int   td_critnest;
};
Comment 4 imbutler 2021-07-14 15:03:50 UTC
It's not only cross-compilation that fails. In an i386 VM and building the GENERIC kernel with a pentium3 target, I get a similar failure ..

--- subr_prng.o ---
Assertion failed: ((EltVT.getSizeInBits() >= 64 || (uint64_t)((int64_t)Val >> EltVT.getSizeInBits()) + 1 < 2) && "getConstant with a uint64_t value that doesn't fit in the type!"), function getConstant, file /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp, line 1337.
PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: cc -target i386-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/i386.i386/tmp -B/usr/obj/usr/src/i386.i386/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -march=pentium3 -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -MD -MF.depend.subr_prng.o -MTsubr_prng.o -fdebug-prefix-map=./machine=/usr/src/sys/i386/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mno-mmx -mno-sse -msoft-float -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error=tautological-compare -Wno-error=empty-body -Wno-error=parentheses-equality -Wno-error=unused-function -Wno-error=pointer-sign -Wno-error=shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -mno-aes -mno-avx -std=iso9899:1999 -Werror /usr/src/sys/kern/subr_prng.c
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'Function Pass Manager' on module '/usr/src/sys/kern/subr_prng.c'.
4.      Running pass 'X86 DAG->DAG Instruction Selection' on function '@prng32'
#0 0x030f49c4 (/usr/bin/cc+0x30f49c4)
#1 0x030f4e30 (/usr/bin/cc+0x30f4e30)
#2 0x030f2a8e (/usr/bin/cc+0x30f2a8e)
#3 0x03090424 (/usr/bin/cc+0x3090424)
#4 0x030905c4 (/usr/bin/cc+0x30905c4)
#5 0x24073399 (/lib/libthr.so.3+0x18399)
cc: error: clang frontend command failed with exit code 134 (use -v to see invocation)
FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-rc2-0-ge7dac564cd0e)
Target: i386-unknown-freebsd14.0
Thread model: posix
InstalledDir: /usr/bin
cc: note: diagnostic msg: 
********************
Comment 5 imbutler 2021-07-14 15:50:28 UTC
Executing the script to build offset.inc on a native pentium3 host also now yields the problematic output .. something changed with this sometime after July 5th when I last built this natively ..

imb@sarah:/sys/kern> sh ./genoffset.sh /usr/obj/usr/src/i386.i386/sys/SARAH/genoffset.o
#ifndef _OFFSET_INC_
#define _OFFSET_INC_
#if !defined(GENOFFSET) && (!defined(KLD_MODULE) || defined(KLD_TIED))
struct thread_lite {
        u_char  pad_td_owepreempt[0xb8 - 0];
        u_char  td_owepreempt;
        u_char  pad_td_pinned[0xd8 - (0xb8 + sizeof(u_char))];
        int     td_pinned;
        u_char  pad_td_priority[0x27e - (0xd8 + sizeof(int))];
        u_char  td_priority;
        u_char  pad_td_critnest[0x30c - (0x27e + sizeof(u_char))];
        u_int   td_critnest;
};
#endif
#endif
Comment 6 imbutler 2021-07-14 17:15:14 UTC
Executing the script to build offset.inc on a native pentium3 host also now yields the problematic output .. something changed with this sometime after July 5th when I last built this natively ..

imb@sarah:/sys/kern> sh ./genoffset.sh /usr/obj/usr/src/i386.i386/sys/SARAH/genoffset.o
#ifndef _OFFSET_INC_
#define _OFFSET_INC_
#if !defined(GENOFFSET) && (!defined(KLD_MODULE) || defined(KLD_TIED))
struct thread_lite {
        u_char  pad_td_owepreempt[0xb8 - 0];
        u_char  td_owepreempt;
        u_char  pad_td_pinned[0xd8 - (0xb8 + sizeof(u_char))];
        int     td_pinned;
        u_char  pad_td_priority[0x27e - (0xd8 + sizeof(int))];
        u_char  td_priority;
        u_char  pad_td_critnest[0x30c - (0x27e + sizeof(u_char))];
        u_int   td_critnest;
};
#endif
#endif

(In reply to imbutler from comment #5)

Ignore this -- I misread the two outputs
Comment 7 Warner Losh freebsd_committer freebsd_triage 2021-07-16 04:58:16 UTC
The difference in the offset.inc file was caused by awk changes in behavior that I've reverted.

The clang bug is still there, but we won't trigger it anymore. You'll need a new awk as described in UPDATING for cross build kernels to work.

I'm not sure if dim@ wants to keep this open or not for the clang assert...
Comment 8 Dimitry Andric freebsd_committer freebsd_triage 2021-07-16 06:53:59 UTC
(In reply to Warner Losh from comment #7)
No, if this mystery is now solved, let's not keep waiting for the upstream bug to be fixed. This might take a long time :)
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2021-07-16 07:22:09 UTC
(In reply to Warner Losh from comment #7)

If you're aware of them, could you ref the commit links here (original, revert and merges?) Feel free to self-assign and close when done (unless someone else resolved :))
Comment 10 Warner Losh freebsd_committer freebsd_triage 2021-07-16 16:42:27 UTC
My commit of the one true awk 20210215 version in f39dd6a9784467f0db5886012b3f4b13899be6b8 (Jul 7 23:00 UTC) broke things. The fix (which reverted just the 0x filtering bit of the commit upstream that caused this trouble) is d4d252c49976de33d0a2926df733744d0b8d95fa (Jul 15 23:00 UTC). In -current. I MFC'd these changes after 3 days, so stable/12 and stable/13 were also affected, though for only a few days.

stable/13:
666abb0888d277e82c6468851e015798e9a7629f Jul 10 18:00 UTC to 3e8044635219ecc467165d0fd020df03ec02ff2c Jul 15 23:00 UTC

stable/12:
6edf5082bab71cf923efff9f18e38efe5b83b0ec Jul 10 18:00 UTC to 40a925385fa6b7c1a177880e36aa0fc278043e49 Jul 15 23:00 UTC
Comment 11 Dimitry Andric freebsd_committer freebsd_triage 2021-07-26 06:53:14 UTC
*** Bug 257407 has been marked as a duplicate of this bug. ***