Bug 250388 - unable to make buildworld releng/12.1. clang always crashes
Summary: unable to make buildworld releng/12.1. clang always crashes
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.1-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-toolchain (Nobody)
URL:
Keywords:
: 247440 251886 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-10-16 06:53 UTC by igor.polovykh
Modified: 2021-02-26 08:35 UTC (History)
4 users (show)

See Also:
koobs: maintainer-feedback? (arichardson)
koobs: mfc-stable12?


Attachments
build log with crash scripts (798.48 KB, application/octet-stream)
2020-10-16 06:58 UTC, igor.polovykh
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description igor.polovykh 2020-10-16 06:53:53 UTC
I can't update system for 3 months yet. Clang always crashes in different stages.

I've already tried to remove /usr/src + /usr/obj completely and get clean sources from remote repository. Nothing helps.


[ root@ ] # uname -a
FreeBSD 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 55f3b6123c2(releng/12.1) amd64


[ root@ ] # cat /etc/make.conf

SVN_UPDATE=yes
SVN=/usr/local/bin/svn
WITH_PKGNG=yes

WITHOUT_X11=yes:wq
OPTIONS_UNSET="X11 GTK2"

DEFAULT_VERSIONS= ruby=2.7 python=3.8 python2=2.7 python3=3.8 perl5=5.30 ssl=libressl pgsql=11 mysql=10.2 node=10 php=7.4
OPTIONS_UNSET=X11

security_p5-GSSAPI_SET     = GSSAPI_MIT
OPTIONS_UNSET+= GSSAPI_BASE
OPTIONS_SET+=   GSSAPI_MIT

MAKE_SHELL?=sh

CC=clang
CXX=clang++
CPP=clang-cpp

KERNCONFDIR=/root
KERNCONF=BVGM



[ root@ ] # cat /etc/src.conf
#HOSTAPD_CFLAGS+=-DEAP_SERVER -DEAP_GTC -DEAP_AKA -DEAP_SIM -DEAP_GPSK
#
# EAP_SERVER option is not available on FREEBSD_9_RELEASE (error in kernel compilation)
#
HOSTAPD_CFLAGS+=-DEAP_GTC -DEAP_AKA -DEAP_SIM -DEAP_GPSK
HOSTAPD_CFLAGS+=-DEAP_PAX -DEAP_SAKE
WITHOUT_SENDMAIL=yes
LOADER_ZFS_SUPPORT=yes

# Apr 14 15:20:35 bvgm kernel: pid 2224 (named), uid 53, was killed: out of swap space
MALLOC_PRODUCTION=yes


WITHOUT_IPX=yes
WITHOUT_IPFILTER=yes
WITHOUT_I4B=yes
WITHOUT_BLUETOOTH=yes
Comment 1 igor.polovykh 2020-10-16 06:58:49 UTC
Created attachment 218786 [details]
build log with crash scripts
Comment 2 Mark Millard 2020-10-29 08:26:35 UTC
(In reply to igor.polovykh from comment #0)

> . . . killed: out of swap space

If your console messasges do not include
messages about "swap_pager_getswapspace(...): failed",
then it is unlikely that being out of swap space
is the actual issue even when it reports: "was killed:
out of swap space" messages. For such contexts, making
the swap area bigger does not help.

In other words, "was killed: out of swap space"
is frequently a misnomer and not to be believed
for "why" the kill happened or what should be
done about it --without other evidence also being
present anyway.

Do you have "swap_pager_getswapspace(...): failed"
notices? The answer would likely guide what sorts
of things would be appropriate to avoid the "killed:
out of swap space" events.

FYI: Other causes of the kill events include . . .

Sustained low free RAM (via stays-runnable processes).
A sufficiently delayed pageout.
The swap blk uma zone was exhausted.
The swap pctrie uma zone was exhausted.

(stays-runnable processes are not swapped out
[kernel stacks are not swapped out] but do actively
compete for RAM via paging activity. In such a
context, free RAM can stay low.)
Comment 3 Dimitry Andric freebsd_committer freebsd_triage 2020-10-29 19:29:56 UTC
Your system is running out of memory compiling part of googletest.

Suggestions:
* Add more RAM or swap
* Lower optimization level
* Wait for base r367101 to be backported to stable/12, or apply it manually
Comment 4 igor.polovykh 2020-11-01 06:47:25 UTC
I have no X server installed. Console based system.
16Gb RAM. 8Gb swap.
RAM is more then enough installed for text based configuration. While compilation RAM usage is not grow to 70%.

I upgrade system to 12.2-RC3 FreeBSD 12.2-RC3 manually with one thread (-j1) and world was built successfully.

My auto buildworld uses 4 threads (-j4) to compile world. I loaded core dump to dbg and looked through shortly where and why crash happens. I guess it connected with multithread compilation.
Comment 5 igor.polovykh 2020-11-01 19:12:58 UTC
FreeBSD 12.2-RC3 compiled successfully with multithread make options (-j4)
This is a bug in releng/12.1 sources

config files are the same for releng/12.1 and for releng/12.2
Comment 6 igor.polovykh 2020-11-01 19:14:32 UTC
FreeBSD 12.2-RC3 compiled successfully with multithread make options (-j4)
This is a bug in releng/12.1 sources

config files are the same for releng/12.1 and for releng/12.2
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2020-11-02 05:20:15 UTC
^Triage: Request feedback from committer of base r367101 about a stable/12 merge
Comment 8 Alex Richardson freebsd_committer freebsd_triage 2020-11-02 12:02:11 UTC
I believe that backporting this change is very low risk. However it depends on a previous commits being merged too. I do not currently have a setup to do this backporting, but if someone else would like to backport 366850 + 367101 (and possibly 366851) please go ahead.
Comment 9 igor.polovykh 2021-02-26 06:13:30 UTC
*** Bug 251886 has been marked as a duplicate of this bug. ***
Comment 10 igor.polovykh 2021-02-26 06:18:25 UTC
*** Bug 247440 has been marked as a duplicate of this bug. ***
Comment 11 igor.polovykh 2021-02-26 06:19:14 UTC
I've found cause of clang crashes and rebooting while compilation. It just was overheating of CPU when temperature of all cores raised above 70°C. Short time compilations didn't give such behaviour. Sorry. It's my fault but I think this case will be useful for someone with similar situation. Perhaps it's possible to make special exception when clang crashes by similar cause and write appropriate message to log.
Comment 12 Dimitry Andric freebsd_committer freebsd_triage 2021-02-26 08:18:18 UTC
Thanks Igor, time to replace that hardware! :)
Comment 13 igor.polovykh 2021-02-26 08:35:53 UTC
No. Changing hardware is too expensive. Just replace thermal grease on CPU. :)