Bug 232048

Summary: CURRENT: clang crashes during buildworld
Product: Base System Reporter: Koop Mast <kwm>
Component: binAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed Not A Bug    
Severity: Affects Only Me CC: marklmi26-fbsd
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   

Description Koop Mast freebsd_committer freebsd_triage 2018-10-07 21:09:36 UTC
I'm trying to upgrade a amd64 machine from 12.0-ALPHA7 r338893 to svn r339223, I think it happens when it tries to build the llvm stuff for the second time.

I don't have a /etc/src.conf, my /etc/make.conf only has "MALLOC_PRODUCTION=yes".

--- CodeGen/CodeGenPGO.o ---
c++  -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin  -O2 -pipe -I/usr/obj/usr/src/amd64.amd64/lib/clang/libclang -I/usr/obj/usr/src/amd64.amd64/lib/clang/libllvm -I/usr/src/contrib/llvm/tools/clang/lib/Basic -I/usr/src/contrib/llvm/tools/clang/lib/Driver -I/usr/src/contrib/llvm/tools/clang/include -DCLANG_ENABLE_ARCMT -DCLANG_ENABLE_STATIC_ANALYZER -I/usr/src/lib/clang/include -I/usr/src/contrib/llvm/include -DLLVM_BUILD_GLOBAL_ISEL -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DLLVM_DEFAULT_TARGET_TRIPLE=\"x86_64-unknown-freebsd12.0\" -DLLVM_HOST_TRIPLE=\"x86_64-unknown-freebsd12.0\" -DDEFAULT_SYSROOT=\"\" -DLLVM_TARGET_ENABLE_AARCH64 -DLLVM_TARGET_ENABLE_ARM -DLLVM_TARGET_ENABLE_MIPS -DLLVM_TARGET_ENABLE_POWERPC -DLLVM_TARGET_ENABLE_SPARC -DLLVM_TARGET_ENABLE_X86 -DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser -DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter -DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler -DLLVM_NATIVE_TARGET=LLVMInitializeX86Target -DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo -DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC -ffunction-sections -fdata-sections -gline-tables-only -MD -MF.depend.CodeGen_CodeGenPGO.o -MTCodeGen/CodeGenPGO.o -fstack-protector-strong -Qunused-arguments  -std=c++11 -fno-exceptions -fno-rtti -gline-tables-only -stdlib=libc++ -Wno-c++11-extensions  -c /usr/src/contrib/llvm/tools/clang/lib/CodeGen/CodeGenPGO.cpp -o CodeGen/CodeGenPGO.o
--- CodeGen/CGStmtOpenMP.o ---
c++: error: unable to execute command: Killed
c++: error: clang frontend command failed due to signal (use -v to see invocation)
--- CodeGen/CodeGenTBAA.o ---
c++  -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin  -O2 -pipe -I/usr/obj/usr/src/amd64.amd64/lib/clang/libclang -I/usr/obj/usr/src/amd64.amd64/lib/clang/libllvm -I/usr/src/contrib/llvm/tools/clang/lib/Basic -I/usr/src/contrib/llvm/tools/clang/lib/Driver -I/usr/src/contrib/llvm/tools/clang/include -DCLANG_ENABLE_ARCMT -DCLANG_ENABLE_STATIC_ANALYZER -I/usr/src/lib/clang/include -I/usr/src/contrib/llvm/include -DLLVM_BUILD_GLOBAL_ISEL -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DLLVM_DEFAULT_TARGET_TRIPLE=\"x86_64-unknown-freebsd12.0\" -DLLVM_HOST_TRIPLE=\"x86_64-unknown-freebsd12.0\" -DDEFAULT_SYSROOT=\"\" -DLLVM_TARGET_ENABLE_AARCH64 -DLLVM_TARGET_ENABLE_ARM -DLLVM_TARGET_ENABLE_MIPS -DLLVM_TARGET_ENABLE_POWERPC -DLLVM_TARGET_ENABLE_SPARC -DLLVM_TARGET_ENABLE_X86 -DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser -DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter -DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler -DLLVM_NATIVE_TARGET=LLVMInitializeX86Target -DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo -DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC -ffunction-sections -fdata-sections -gline-tables-only -MD -MF.depend.CodeGen_CodeGenTBAA.o -MTCodeGen/CodeGenTBAA.o -fstack-protector-strong -Qunused-arguments  -std=c++11 -fno-exceptions -fno-rtti -gline-tables-only -stdlib=libc++ -Wno-c++11-extensions  -c /usr/src/contrib/llvm/tools/clang/lib/CodeGen/CodeGenTBAA.cpp -o CodeGen/CodeGenTBAA.o
--- CodeGen/CGStmtOpenMP.o ---
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
Target: x86_64-unknown-freebsd12.0
Thread model: posix
InstalledDir: /usr/obj/usr/src/amd64.amd64/tmp/usr/bin
c++: note: diagnostic msg: PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
c++: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
c++: note: diagnostic msg: /tmp/CGStmtOpenMP-6b8420.cpp
c++: note: diagnostic msg: /tmp/CGStmtOpenMP-6b8420.sh
c++: note: diagnostic msg: 

********************
*** [CodeGen/CGStmtOpenMP.o] Error code 254

make[6]: stopped in /usr/src/lib/clang/libclang
1 error

make[6]: stopped in /usr/src/lib/clang/libclang
*** [all_subdir_lib/clang/libclang] Error code 2
Comment 1 Koop Mast freebsd_committer freebsd_triage 2018-10-07 21:14:25 UTC
The mentioned files in the crash are too large for bugzilla so you can get them here:

https://people.freebsd.org/~kwm/CGStmtOpenMP-6b8420.tar.xz

SHA512 (CGStmtOpenMP-6b8420.tar.xz) = e3ed18a91d74da957b5230b3c4e527e52c7012af9ef6c2ae263fe102ea4afba9f581d2b9f616e8005d865a01314193663cd2379148bc4040863ea58d15cfe037
Comment 2 Mark Millard 2018-10-07 22:41:17 UTC
A common reason for:

c++: error: unable to execute command: Killed

is being low on free RAM for too long (too
many tries to get more free RAM).

The console would have reported a message (that in
part may be a misnomer), something like:

. . . kernel: pid 1010 (stress), uid 0, was killed: out of swap space

You can likely believe the "out of swap space" if you also
got messages like:

. . . kernel: swap_pager_getswapspace(32): failed

but otherwise you may have lots of swap space but still
get "was killed" messages. In part this is because
FreeBSD does not swap out processes that stay runnable,
so runnable processes that keep lots of RAM in the
"active" status can lead to sustained periods of low
free RAM (and, so, probably paging).

Changing vm.pageout_oom_seq from its default figure of
12 to something larger makes FreeBSD tolerate the low
free RAM longer (more tries to free RAM). Folks have
use figures like:

sysctl vm.pageout_oom_seq=120
sysctl vm.pageout_oom_seq=1024

on small single board computers with 4 cores (1 hardware
thread per core) to allow buildworld buildkernel to complete
for -j4 with 1 GiByte or 2 GiByte of RAM.

I/O/storage latency for paging and swapping can contribute
to needing a larger vm.pageout_oom_seq .

Another thing that might help such builds use less RAM at times
is to use:

LDFLAGS.lld+= -Wl,--no-threads

Using a smaller N for -jN is another example (for 1<N).


You were not explicit about properties such as the -jN or
the amount of RAM or other such. So these notes are not
informed by such contextual properties and may not apply.

If they do apply, there are other bugzilla reports that this
is a duplicate of. It has been an on-going issue for some
time but is not obvious from what FreeBSD reports when it
does the process kills. Mark Johnston provided the information
tying vm.pageout_oom_seq values to controlling the behavior.
Comment 3 Koop Mast freebsd_committer freebsd_triage 2018-10-10 20:37:53 UTC
This was indeed a OOM issue, another process was hogging all the memory. I should have check dmesg first to see if there was something about clang getting killed because of no more available memory.