Bug 230454

Summary: Compiling world fails on /usr/src/contrib/llvm/tools/clang/lib/CodeGen/CGCXXABI.cpp
Product: Base System Reporter: tschweikle
Component: kernAssignee: Dimitry Andric <dim>
Status: In Progress ---    
Severity: Affects Only Me CC: dim, imp, marklmi26-fbsd, rgrimes, toolchain, tschweikle
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227609
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230402
Attachments:
Description Flags
Invocation script none

Description tschweikle 2018-08-08 11:54:26 UTC
c++: error: unable to execute command: Killed
c++: error: clang frontend command failed due to signal (use -v to see invocation)
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
Target: x86_64-unknown-freebsd12.0
Thread model: posix
InstalledDir: /usr/bin
c++: note: diagnostic msg: PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
c++: note: diagnostic msg:
Comment 1 tschweikle 2018-08-08 11:57:29 UTC
Created attachment 196006 [details]
Invocation script
Comment 2 tschweikle 2018-08-08 12:12:43 UTC
Second attachment was not accepted because of size (has 14MB, max. accepted 1MB).
Comment 3 Dimitry Andric freebsd_committer 2018-08-11 06:30:39 UTC
Most likely, you are running out of memory.  Try lowering your make -j level, or adding some more swap (or memory :).
Comment 4 Rodney W. Grimes freebsd_committer 2018-08-11 14:00:18 UTC
(In reply to Dimitry Andric from comment #3)
From other issues we have seen on aarch64 and small memory amd64 VM's adding swap is rarely the solution to this, you either have to add memory or reduce -j level.
Comment 5 Warner Losh freebsd_committer 2018-08-11 14:15:12 UTC
(In reply to Rodney W. Grimes from comment #4)

You need to add non-flash swap typically to make any headway, although reducing -j is more effective. Spinning disks suffer much less from these issues, but are slower. Also, on rpi3 there appears to be a hanging bug in the USB stack which can sometimes be triggered. The arm64 issues stem from crappy nand and bad drivers causing undue (multi-second) delays. But the original poster is running on amd64 where that's not an issue because most people aren't putting swap on crappy SD cards, crappy USB thumb drives and our SATA drivers there are much, much more mature and resilient.
Comment 6 Mark Millard 2018-08-11 16:43:46 UTC
(In reply to Warner Losh from comment #5)

While Bob P.'s reports were with large gstat reporting large
ms/w and/or ms/r, Trev had made reports of getting the problem
without gstat indicating multi-second delays (averages for some
time intervals) and having lots of swap still available, starting
at:

https://lists.freebsd.org/pipermail/freebsd-arm/2018-June/018151.html

(But Trev only provided short segments of the records in
messages instead of posting the whole logs somewhere. This
limits confirmation.)



Note to avoid misinterpretation: I am in general agreement with
the following as appropriate experiments . . .

"You need to add non-flash swap"
and
"Spinning disks suffer much less from these issues"

although I would explicitly add that there are SSD-based USB
drives and at least some of these might be about as good as
using a direct SATA SSD environment with good SSDs. (Correct
me if I'm wrong for some reason.)

Notes on testing:

Having records from, say, periodic gstat runs, is a good cross
check in all cases. Having the information-reporting patches
from Mark Johnston are appropriate for testing as well.

http://www.zefox.net/~fbsd/rpi3/swaptests/tools/ has various
things that have been used in testing. But batchqueue.patch
did not pan out and likely should be avoided. Similarly for
config_options. The readme is what talks about:

sysctl vm.pageout_oom_seq=120

The rest are patches for reporting of additional information
during operation. These are likely more important than the
gstat ms/w and ms/r records.

The context for the patches is recent head, not 11.x .
Comment 7 Mark Millard 2018-08-12 17:50:32 UTC
(In reply to Mark Millard from comment #6)

[This is extracted from another context that
involved the Pine64+ 2GB.]

As of updating to -r337400 the Pine64+ 2GB no
longer will boot from the e.MMC on the microsd
adapter card. (I switched to tracking fully
modern dts use, u-boot, etc.)

So I tried a build via a USB SSD as the root
file system and swap partition. As reported in:

https://lists.freebsd.org/pipermail/freebsd-arm/2018-August/018605.html

it failed with an Out Of Memory kill. (More detail there.
I did not adjust vm.pageout_oom_seq .)

This should have avoided I/O latency problems being
involved. (That message is part of a long on-going
thread tied to OOM kills, most of the reports involving
large I/O latencies being involved.)

Very little swap was observed to be in use. This was
a context with Mark Johnston's reporting patches in
place, but they only reported the kill itself, not
waiting for I/O or such.

I can not change the Affects Only Me status.

Dimitry Andric is possibly the wrong assignee:
not the compiler's problem if the system did an
Out Of Memory kill of a process.

Other bugzilla's are: 227609 230402, not just this
230454.
Comment 8 Mark Millard 2020-03-11 04:45:45 UTC
I'll note that folks have been reporting problems
with 1 GiByte machines for -j1 buildworld's based
on the compilation of gmock-matchers_test.cc .
An example is:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241848

It might be better to make such the base so
the information is not as old as here: declare
older submittals as duplicates of the modern
information example?

(llvm materials have grown as well.)

Head has also added an paging/swapping I/O
related OOM criteria (that was temporarily
broken a littel while back). I've not used
a 1 GiBYTe context but I use two settings,
one of which is for avoiding I/O related
kills happening as easilly:

#
# Delay when persistent low free RAM leads to
# Out Of Memory killing of processes:
vm.pageout_oom_seq=120
#
# For plunty of swap/paging space (will not
# run out), avoid pageout delays leading to
# Out Of Memory killing of processes:
vm.pfault_oom_attempts=-1

I do this on 2 GiByte+ contexts to better
avoid OOM kills. vm.pfault_oom_attempts is
head-only last I knew but vm.pageout_oom_seq
is much older.

Some folks have used much larger
vm.pageout_oom_seq values.
Comment 9 Mark Millard 2020-03-31 18:48:24 UTC
(In reply to Mark Millard from comment #8)

I've tried a 1 GiByte aarch64 context for a -j4
buildworld buildkernel . . .

I had a RPi3 that was based on head -r358966 do a
build world buildkernel of the same version, from
scratch, -j4 style. The RPi3 is a 1 GiByte RAM
context. I had 3072 GiBytes for the swap partition.
That ,and the ufs file system, were on a USB SSD,
not the microsd card.

The build completed without any /var/log/message or
console output during the build. My modified version
of top reported (details copied from a ssh window) . . .

For Mem: 738512Ki MaxObsActive, 190608Ki MaxObsWired, 906372Ki MaxObs(Act+Wir)
For Swap:  1927Mi MaxObsUsed

(top was started before the build. "MaxObs" is short
for "Maximum Observed".)

The build took a few minutes under 31 hrs.

The build used (my PINE64 media are also set up
to boot the RPi3, explaining some naming):

vm.pageout_oom_seq=120
vm.pfault_oom_attempts=-1
vfs.root.mountfrom="ufs:/dev/gpt/PINE642Groot"
dumpdev="/dev/gpt/PINE642Gswp2"
/dev/gpt/PINE642Groot           /               ufs rw,noatime          1 1
/dev/gpt/PINE642Gswap           none            swap sw                 0 0

(So this avoided the microsd card for ufs and
swap/page space.)

Overall, it looks like having more than 2 GiBytes
of swap partition is appropriate for -j4 : 1927
MiByte is not much less than 2048 MiByte.

But, with appropriate configuration anyway, the
RPi3 can do buildworld buildkernel for head 13,
even -j4 style.

This was aarch64. armv7 style with 1 GiByte RAM
does not allow as much swap/page space without
complaining at boot. It does not appear that
such a -j4 build would be appropriate for armv7.
But I've not investigated what would fit.