Bug 250388

Summary: unable to make buildworld releng/12.1. clang always crashes
Product: Base System Reporter: igor.polovykh
Component: binAssignee: freebsd-toolchain (Nobody) <toolchain>
Status: Open ---    
Severity: Affects Some People CC: arichardson, lwhsu, marklmi26-fbsd
Priority: --- Flags: koobs: maintainer-feedback? (arichardson)
koobs: mfc-stable12?
Version: 12.1-STABLE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
build log with crash scripts none

Description igor.polovykh 2020-10-16 06:53:53 UTC
I can't update system for 3 months yet. Clang always crashes in different stages.

I've already tried to remove /usr/src + /usr/obj completely and get clean sources from remote repository. Nothing helps.


[ root@ ] # uname -a
FreeBSD 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 55f3b6123c2(releng/12.1) amd64


[ root@ ] # cat /etc/make.conf

SVN_UPDATE=yes
SVN=/usr/local/bin/svn
WITH_PKGNG=yes

WITHOUT_X11=yes:wq
OPTIONS_UNSET="X11 GTK2"

DEFAULT_VERSIONS= ruby=2.7 python=3.8 python2=2.7 python3=3.8 perl5=5.30 ssl=libressl pgsql=11 mysql=10.2 node=10 php=7.4
OPTIONS_UNSET=X11

security_p5-GSSAPI_SET     = GSSAPI_MIT
OPTIONS_UNSET+= GSSAPI_BASE
OPTIONS_SET+=   GSSAPI_MIT

MAKE_SHELL?=sh

CC=clang
CXX=clang++
CPP=clang-cpp

KERNCONFDIR=/root
KERNCONF=BVGM



[ root@ ] # cat /etc/src.conf
#HOSTAPD_CFLAGS+=-DEAP_SERVER -DEAP_GTC -DEAP_AKA -DEAP_SIM -DEAP_GPSK
#
# EAP_SERVER option is not available on FREEBSD_9_RELEASE (error in kernel compilation)
#
HOSTAPD_CFLAGS+=-DEAP_GTC -DEAP_AKA -DEAP_SIM -DEAP_GPSK
HOSTAPD_CFLAGS+=-DEAP_PAX -DEAP_SAKE
WITHOUT_SENDMAIL=yes
LOADER_ZFS_SUPPORT=yes

# Apr 14 15:20:35 bvgm kernel: pid 2224 (named), uid 53, was killed: out of swap space
MALLOC_PRODUCTION=yes


WITHOUT_IPX=yes
WITHOUT_IPFILTER=yes
WITHOUT_I4B=yes
WITHOUT_BLUETOOTH=yes
Comment 1 igor.polovykh 2020-10-16 06:58:49 UTC
Created attachment 218786 [details]
build log with crash scripts
Comment 2 Mark Millard 2020-10-29 08:26:35 UTC
(In reply to igor.polovykh from comment #0)

> . . . killed: out of swap space

If your console messasges do not include
messages about "swap_pager_getswapspace(...): failed",
then it is unlikely that being out of swap space
is the actual issue even when it reports: "was killed:
out of swap space" messages. For such contexts, making
the swap area bigger does not help.

In other words, "was killed: out of swap space"
is frequently a misnomer and not to be believed
for "why" the kill happened or what should be
done about it --without other evidence also being
present anyway.

Do you have "swap_pager_getswapspace(...): failed"
notices? The answer would likely guide what sorts
of things would be appropriate to avoid the "killed:
out of swap space" events.

FYI: Other causes of the kill events include . . .

Sustained low free RAM (via stays-runnable processes).
A sufficiently delayed pageout.
The swap blk uma zone was exhausted.
The swap pctrie uma zone was exhausted.

(stays-runnable processes are not swapped out
[kernel stacks are not swapped out] but do actively
compete for RAM via paging activity. In such a
context, free RAM can stay low.)
Comment 3 Dimitry Andric freebsd_committer 2020-10-29 19:29:56 UTC
Your system is running out of memory compiling part of googletest.

Suggestions:
* Add more RAM or swap
* Lower optimization level
* Wait for base r367101 to be backported to stable/12, or apply it manually
Comment 4 igor.polovykh 2020-11-01 06:47:25 UTC
I have no X server installed. Console based system.
16Gb RAM. 8Gb swap.
RAM is more then enough installed for text based configuration. While compilation RAM usage is not grow to 70%.

I upgrade system to 12.2-RC3 FreeBSD 12.2-RC3 manually with one thread (-j1) and world was built successfully.

My auto buildworld uses 4 threads (-j4) to compile world. I loaded core dump to dbg and looked through shortly where and why crash happens. I guess it connected with multithread compilation.
Comment 5 igor.polovykh 2020-11-01 19:12:58 UTC
FreeBSD 12.2-RC3 compiled successfully with multithread make options (-j4)
This is a bug in releng/12.1 sources

config files are the same for releng/12.1 and for releng/12.2
Comment 6 igor.polovykh 2020-11-01 19:14:32 UTC
FreeBSD 12.2-RC3 compiled successfully with multithread make options (-j4)
This is a bug in releng/12.1 sources

config files are the same for releng/12.1 and for releng/12.2
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2020-11-02 05:20:15 UTC
^Triage: Request feedback from committer of base r367101 about a stable/12 merge
Comment 8 Alex Richardson freebsd_committer 2020-11-02 12:02:11 UTC
I believe that backporting this change is very low risk. However it depends on a previous commits being merged too. I do not currently have a setup to do this backporting, but if someone else would like to backport 366850 + 367101 (and possibly 366851) please go ahead.