Comment 1 Brooks Davis 2020-12-17 00:10:30 UTC

Is the crash reproducible (e.g. if you run /tmp/check_utility_compat-2e4627.sh when the system isn't compiling other things)?

I see a stray:

Apr 14 15:20:35 bvgm kernel: pid 2224 (named), uid 53, was killed: out of swap space

in your pasted output which suggests you're running low on memory. This is how I'd expect clang to report being killed by the OOM killer.

When I run run /tmp/check_utility_compat-2e4627.sh no crash happens.
16Gb RAM installed on host and 8Gb of swap. X server is not used. Text mode. Do you think this is OOM?

Comment 3 Brooks Davis 2020-12-17 06:12:22 UTC

Yeah, sounds like OOM given that you're low enough on memory that a relatively small process like named is getting killed. What -j flag were you using to build?

/etc/src.conf  config is very very old and "# Apr 14 15:20:35 bvgm kernel: pid 2224 (named), uid 53, was killed: out of swap space"  this record was added when i used named as DNS server. Hardware already changed 2 times.

I use -j1 because when I try to use more then one thread to compile I had the same crashes in sudden places.

Ok. I'll add maximum available RAM 32Gb and try to compile world again.

Actually I guess it's too much for clang > 16Gb RAM to compile without X

(In reply to Brooks Davis from comment #1)

In case it is not clear: The "was killed" notice is in a comment in /etc/src.conf and dates back to April. It is not part of the console log for the failing compile.

(Comment 4 confused me until I went back and looked.)

(In reply to igor.polovykh from comment #7)

If you can repeat the failure, do you get any interesting
console log messages ( /var/log/messages content ) during
the build? Does top show low free RAM? Swap nearly all in
use?

If you do get one or more "was killed: out of swap space"
messages: Be warned that the "out of swap space" part can
be a misnomer. Only if you also see message(s) with text
of the form "swap_pager_getswapspace(...): failed" is the
swap space part of the notice the actual cause as far as I
know. Other causes include:

Sustained low free RAM (via stays-runnable processes).
A sufficiently delayed pageout.
The swap blk uma zone was exhausted.
The swap pctrie uma zone was exhausted.

There are tunables for some of that that make some
of those not trip as soon ( /etc/sys.conf content ):

#
# Delay when persistent low free RAM leads to
# Out Of Memory killing of processes. The
# delay is a count of kernel-attempts to gain
# free RAM (so not time units).
vm.pageout_oom_seq=120

(The default is 12 as far as I know.)

NOTE: stable/12 -r351776 got the support for the following:
(I've not checked the match to releases.)

#
# For plunty of swap/paging space (will not
# run out), avoid pageout delays leading to
# Out Of Memory killing of processes:
vm.pfault_oom_attempts=-1


That last has the alternative structure (replace
???'s with positive integers):

#
# For possibly insufficient swap/paging space
# (might run out), increase the pageout delay
# that leads to Out Of Memory killing of
# processes:
#vm.pfault_oom_attempts= ???
#vm.pfault_oom_wait= ???
# (The multiplication of the two values is the
# total but there are other potential tradoffs
# in the factors multiplied for the same total.)

For reference:

# sysctl -d vm.pfault_oom_wait
vm.pfault_oom_wait: Number of seconds to wait for free pages before retrying the page fault handler

# sysctl -d vm.pfault_oom_attempts
vm.pfault_oom_attempts: Number of page allocation attempts in page fault handler before it triggers OOM handling

# sysctl -d vm.pageout_oom_seq
vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM

added 16Gb RAM to server. Try to compile world with origin/stable/12 branch

always get error

clang  -O2 -pipe -fno-common   -D_XOPEN_SOURCE_EXTENDED -DENABLE_WIDEC -I. -I/usr/obj/usr/src/amd64.amd64/lib/ncurses/ncursesw -I/usr/src/lib/ncurses/ncursesw -I/usr/src/lib/ncurses/ncurses -I/usr/src/contrib/$
--- lib/libufs__L ---
/usr/src/sys/ufs/ffs/ffs_subr.c:331:24: error: use of undeclared identifier 'CK_SUPPORTED'
                fs->fs_metackhash &= CK_SUPPORTED;
                                     ^
1 error generated.
*** [ffs_subr.o] Error code 1

make[4]: stopped in /usr/src/lib/libufs
1 error

make[4]: stopped in /usr/src/lib/libufs
--- lib/ncurses/ncurses__L ---
--- cddl/lib/libzfs_core__L ---
--- lib/ncurses/ncursesw__L ---


There is no "releng*" labels on https://github.com/freebsd/freebsd repository

i got sources from svn repository branch 'releng/12.2'
world compiled successfully with -j4 and 32Gb RAM.

Conclusion: clang utilises many RAM to build complex projects.
Mark, You was right that this was OOM crash. I even did not expect that 16Gb RAM will not be enough to build world.

Thank you for sysctl settings that tied with oom. I'll keep in mind it in future.

(In reply to igor.polovykh from comment #11)

I do not know if you have ZFS or other sources of possibly significant
memory use, depending on ZFS tuning. But I on occasion do -j4
buildworld's on machines with 16 GiByte, 12 GiByte, 8 GiByte, 4 GiByte,
and even 2 GiByte --but with the types of tuning that I reported. This
is over a variety of archtiectures: powerpc64 (12 GiByte these days),
armv7 (2 GiByte), aarch64 (16/8/4 GiByte), each with 4 cores. (My
amd64 context has lots more RAM and hardware threads --and so is not
comparable.)

All these are UFS based contexts, all SSD based (many using USB3 SSDs),
SSDs that seem to perform nicely for the purpose. I also avoid tmpfs or
other such to avoid the memory use competing with the compiler and
linker instances.

I also have used ( in /etc/make.conf ):

LDFLAGS.lld+= -Wl,--no-threads

to limit the llvm linker's multi-threading (for its memory use
consequences). So linking may take longer.

I have done a "13" buildworld on a 1 GiByte aarch64, but not with -j4 .
If I remember right it was -j2 . This is the only context to use
significant swap/paging space as I remember, significant enough that
-j3 likely would not have worked. (I normally keep the swap space sized
somewhat below where booting would start to complain about it being too
much without other tuning.)

Another point is that these machine are not running X11 or other such:
serial console and console-like ssh use. Strongly biased to the resources
being available to the builds overall.

(In reply to Mark Millard from comment #12)

I misremembered: I've looked up my most modern notes in:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227609

and the aarch64 1 GiByte RAM buildworld was -j4 and used
a maxmium of around 1927Mi of swap space. (So RAM+swap_used
was about 3 GiByte.)

It was armv7 with 1 GiByte that appeared that it would
have been problematical, mostly because it does not
allow as much swap space for the same amount of RAM
(while avoiding the boot messsages about the swap space
size).

(In reply to igor.polovykh from comment #11)

It looks like no equivalent of:

https://svnweb.freebsd.org/base?view=revision&revision=367101

was MFC'd to stable/12 . I wonder if such might help.

There is/was a related report:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241848

(It has notes about some build experiments I did before
head -r367101 , reporting RAM and swap use and such.)

(In reply to igor.polovykh from comment #11)
the same crash with rebooting system while world compilation with 32Gb RAM occurred last night. 
There are no any messages in console log messages ( /var/log/messages content ) about it. Just silent restart of system.

I'll try to set 
LDFLAGS.lld+= -Wl,--no-threads
and then try oom tunables in sysctl

New strange behaviour with LDFLAGS.lld+= -Wl,--no-threads in make.conf.

clang++  -O2 -pipe -fno-common -I/usr/obj/usr/src/amd64.amd64/lib/clang/libllvm -I/usr/src/contrib/llvm-project/llvm/lib/Target/AArch64 -I/usr/src/contrib/llvm-project/llvm/lib/Target/ARM -I/usr/src/contrib/llvm-project/llvm/lib/Target/$
Stack dump:
0.      Program arguments: clang++ -O2 -pipe -fno-common -I/usr/obj/usr/src/amd64.amd64/lib/clang/libllvm -I/usr/src/contrib/llvm-project/llvm/lib/Target/AArch64 -I/usr/src/contrib/llvm-project/llvm/lib/Target/ARM -I/usr/src/contrib/llv$
1.      <eof> parser at end of file
2.      Per-module optimization passes
3.      Running pass 'CallGraph Pass Manager' on module '/usr/src/contrib/llvm-project/llvm/lib/Transforms/IPO/Attributor.cpp'.
#0 0x00000000038320be PrintStackTrace /usr/src/contrib/llvm-project/llvm/lib/Support/Unix/Signals.inc:564:13
#1 0x0000000003830335 RunSignalHandlers /usr/src/contrib/llvm-project/llvm/lib/Support/Signals.cpp:69:18
#2 0x0000000003834341 HandleCrash /usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:77:5
#3 0x0000000003834341 CrashRecoverySignalHandler /usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:382:51
#4 0x000000000470a100 handle_signal /usr/src/lib/libthr/thread/thr_sig.c:0:3
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Target: x86_64-unknown-freebsd12.2
Thread model: posix
InstalledDir: /usr/bin
clang++: note: diagnostic msg: PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
clang++: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang++: note: diagnostic msg: /tmp/Attributor-1e1087.cpp
clang++: note: diagnostic msg: /tmp/Attributor-1e1087.sh
clang++: note: diagnostic msg:

********************
*** Error code 254

Stop.
make[6]: stopped in /usr/src/lib/clang/libllvm
*** Error code 1
*** Error code 1
*** Error code 1
*** Error code 1
*** Error code 1
*** Error code 1

I've found cause of clang crashes and rebooting while compilation. It just was overheating of CPU when temperature of all cores raised above 70°C. Short time compilations didn't give such behaviour. Sorry. It's my fault but I think this case will be useful for someone with similar situation.

*** This bug has been marked as a duplicate of bug 250388 ***