Created attachment 222742 [details]
/tmp/PassBuilder-2f2b98.sh

PassBuilder-2f2b98.sh

Created attachment 222743 [details]
truncated /tmp/PassBuilder.cpp

Comments removed and finally truncated using command "head". Zipped. File was 10 MB, zipped 1,9 MB. File size limit 1000 kB is too low.

Unfortuantely, "swapinfo" after the process-kill does not report
the swap usage from just before the process-kill.

The message with "was killed: out of swap space" is commonly a
misnomer: other things can cause the kills but the same message
is output for such. If you had actually run out of swap space
there should be one or more messages that would have text
something like:

swap_pager_getswapspace(??): failed

The 4 causes of kills that I know of are:

Sustained low free RAM (via stays-runnable processes).
A sufficiently delayed pageout.
The swap blk uma zone was exhausted.
The swap pctrie uma zone was exhausted.

Your "indefinite wait buffer" messages suggest the 2nd
may be involved in addition to the 1st of those 4.

There are tunables for those 2 that make them not
trip as soon for the 1st or ever for the 2nd
( /etc/sys.conf content or sysctl assignments ):

#
# Delay when persistent low free RAM leads to
# Out Of Memory killing of processes. The
# delay is a count of kernel-attempts to gain
# free RAM (so not time units).
vm.pageout_oom_seq=120

(The default is 12 as far as I know.)

#
# For plunty of swap/paging space (will not
# run out), avoid pageout delays leading to
# Out Of Memory killing of processes:
vm.pfault_oom_attempts=-1

I expect that 12.2-RELEASE has both of those
available but I've not been using 12.x .

Comment 4 Daniel Engberg 2021-02-24 06:14:16 UTC

Hi Jouni,

Unforunately 1Gb of RAM is way too little to build world (FreeBSD) using 4 workers. LLVM/Clang will at some stages use around ~800Mbyte per job which will push about 3x of the available amount of RAM to swap. This doesn't necessarily mean that it'll run out of memory but if you have a fairly slow storage device (SD card, USB memory and even a mechanical HDD to some extent) you'll get so slow access that it'll spend minutes trying shuffle data back and forth between main memory and swap which eventually will lead to the kernel killing processes to restore performance and lower memory pressure.

Now, if you want to run buildworld on your device which I wouldn't recommend with only 1Gb of RAM you can at best use 2 jobs (-j2 but I would recommend -j1) but be aware of excess swapping at some stages and it's going to be very slow. Unless you have a beefier device at hand I'd recommend cross-compiling but I'm not aware of any easy way upgrading a system going that route.

Additionally if you're using the official image(s) you'll need to modify /etc/fstab and increase and/or remove the limit of tmpfs otherwise make installworld will fail.

(In reply to daniel.engberg.lists from comment #4)

I agree in general that for the context -j2 or -j1 would
likely be better, especially for trying to use microsd
cards for the file system and swap/paging space.

But, I've tried a 1 GiByte aarch64 FreeBSD context for
a -j4 buildworld buildkernel in the past and had it
work (because of appropriate configuration). . .

I had a RPi3 that was based on head -r358966 do a
build world buildkernel of the same version, from
scratch, -j4 style. The RPi3 is a 1 GiByte RAM
context. I had 3072 GiBytes for the swap partition.
That and the ufs file system, were on a USB SSD,
not the microsd card. (No ZFS involved. No memory
based /tmp involved: generally avoiding such
competition for memory vs. the compiler and linker
and the like.)

The build completed without any /var/log/message or
console output during the build. My modified version
of top reported (details copied from a ssh window) . . .

For Mem: 738512Ki MaxObsActive, 190608Ki MaxObsWired, 906372Ki MaxObs(Act+Wir)
For Swap:  1927Mi MaxObsUsed

(top was started before the build. "MaxObs" is short
for "Maximum Observed".)

The build took a few minutes under 31 hrs.

The build used (my PINE64 media are also set up
to boot the RPi3, explaining some naming):

vm.pageout_oom_seq=120
vm.pfault_oom_attempts=-1
vfs.root.mountfrom="ufs:/dev/gpt/PINE642Groot"
dumpdev="/dev/gpt/PINE642Gswp2"
/dev/gpt/PINE642Groot           /               ufs rw,noatime          1 1
/dev/gpt/PINE642Gswap           none            swap sw                 0 0

(So this avoided the microsd card for ufs and
for swap/page space.)

Overall, it looks like having more than 2 GiBytes
of swap partition is appropriate for -j4 : 1927
MiByte is not much less than 2048 MiByte.

But, with appropriate configuration anyway, the
RPi3 can do buildworld buildkernel for head 13,
even -j4 style.

That was aarch64 FreeBSD. armv7 FreeBSD style with
1 GiByte RAM does not allow as much swap/page space
without complaining at boot. It does not appear that
such a -j4 build would be appropriate under armv7
FreeBSD. (As I remember, I've done a -j2 build on
armv7 that worked well but I'd have to look up the
details.)

Thank you for the really specific explanation. It was nice to read it. Another similar bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214299 . The processor was specified as two cores aarch64, URL: http://espressobin.net/tech-spec/ .

The -j1 option stopped as in attachment 1 [details]. The swap space was increased to 11 GB total before the run. The -j2 right after this run stopped similarly as the -j4. The first run took multiple days (a full 3-4 days) and the USB console was helpful. I'm reading the settings in the comments today. The version was 12.2-RELEASE (at svn revision 369333) and a binary distribution exists. It could be easier to install it and update it later. Only the kernel would be compiled. It is possible to use a full day to read more about the build.

The -j1 stopped in error code 137 at 'all_subdir_lib'.

Attachment 1 [details], build log
----- clip -----
echo gmock-matchers_test.full: /usr/obj/usr/src/arm64.aarch64/tmp/usr/lib/libc++.a >> .depend.gmock-matchers_test
c++  -target aarch64-unknown-freebsd12.2 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin  -O2 -pipe -fno-common -g -MD -MF.depend.gmock-matchers_test.gmock-matchers_test.o -MTgmock-matchers_test.o -fstack-protector-strong -Wsystem-headers -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wredundant-decls -Wmissing-variable-declarations -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Qunused-arguments  -I/usr/obj/usr/src/arm64.aarch64/tmp/usr/include/private  -I/usr/src/contrib/googletest/googlemock/include -I/usr/src/contrib/googletest/googlemock -I/usr/src/contrib/googletest/googletest/include -I/usr/src/contrib/googletest/googletest -I/usr/obj/usr/src/arm64.aarch64/tmp/usr/include/private -DGTEST_HAS_POSIX_RE=1 -DGTEST_HAS_PTHREAD=1 -DGTEST_HAS_STREAM_REDIRECTION=1 -frtti -Wno-deprecated-copy -Wno-signed-unsigned-wchar -DGTEST_HAS_POSIX_RE=1 -DGTEST_HAS_PTHREAD=1 -DGTEST_HAS_STREAM_REDIRECTION=1 -frtti -Wno-deprecated-copy -Wno-signed-unsigned-wchar -std=c++11 -Wno-deprecated-declarations   -Wno-c++11-extensions  -c /usr/src/contrib/googletest/googlemock/test/gmock-matchers_test.cc -o gmock-matchers_test.o
/usr/src/contrib/googletest/googlemock/test/gmock-matchers_test.cc:1028:8: warning: private field 'c_' is not used [-Wunused-private-field]
  char c_;
       ^
Killed
*** [all_subdir_lib] Error code 137

make[2]: stopped in /usr/src
1 error

make[2]: stopped in /usr/src
*** [everything] Error code 2

make[1]: stopped in /usr/src
1 error

make[1]: stopped in /usr/src
*** [buildworld] Error code 2

make: stopped in /usr/src

make: stopped in /usr/src

SD-card was HD-quality (almost the fastest).

(In reply to Jouni Laakso from comment #6)

I'll note that there are tradeoffs where one can have
too much swap: there is kernel memory use involved in
keeping track of the paging/swapping and it completes
with other kernel memory use.

If you got a boot message of the form:

warning: total configured swap (??? pages) exceeds maximum recommended amount (??? pages).

then you may have had such problems. There are also
warnings in the documentation about causing tradeoffs
if kern.maxswzone is increased (less space for other
kernel information). (See "man 8 loader" and its
kern.maxswzone description, although the "eight times"
material does not seem to be correct for arm64. Also
the wording presumes decreasing kern.maxswzone in
order to make more room for other things. Making less
room has to be inferred for increases.)

On arm64, an 11 GiByte swap space would need a fair
amount of RAM to avoid getting the warning.

(In reply to Jouni Laakso from comment #6)

See:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241848

about issues with compiling gmock-matchers_test.cc . It
turns out that clang with -O2 uses a huge amount of
memory and time for that file (but does not for -O use).

Things were changed to use -O in HEAD -r367101 ( main
433f33d285eee7c ) but it was not MFC'd to stable/12 .

If you locally make a similar change, you likely can get
gmock-matchers_test.cc to build.