Bug 227609 - Compiling world from sources fails: out of swap space
Summary: Compiling world from sources fails: out of swap space
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
: 231123 (view as bug list)
Depends on:
Reported: 2018-04-18 13:57 UTC by tschweikle
Modified: 2021-02-12 22:02 UTC (History)
4 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description tschweikle 2018-04-18 13:57:15 UTC

Comment 1 tschweikle 2018-04-18 14:08:44 UTC
checking out sources, then configuring, then compiling tools, then world.
Compiling wold fails with out of swap space. In some rare cases compiling world succeeds, but then compiling kernel fails. I never got both: world an kernel.

Swap is 8GiByte, RAM is 4GiByte.

FreeBSD fbsd12.bfs.de 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r330841: Wed Mar 14 09:29:53 CET 2018     root@fbsd12.bfs.de:/usr/obj/usr/src/amd64.amd64/sys/FBSD12  amd64

Last world and kernel compile succeeded was Wed Mar 14 09:29:53 CET 2018
svn release r330841

Later tries ended all with out of swap space:
Root mount waiting for: usbus1 usbus0
ugen0.2: <VMware VMware Virtual USB Mouse> at usbus0
ugen0.3: <vendor 0x0e0f VMware Virtual USB Hub> at usbus0
uhub2 on uhub1
uhub2: <VMware Virtual USB Hub> on usbus0
Root mount waiting for: usbus1 usbus0
uhub0: 6 ports with 6 removable, self powered
uhub2: 7 ports with 7 removable, self powered
VMware memory control driver initialized
intsmb0: <Intel PIIX4 SMBUS Interface> port 0x1040-0x104f at device 7.3 on pci0
intsmb0: intr SMI disabled revision 0
smbus0: <System Management Bus> on intsmb0
uhid0 on uhub1
uhid0: <VMware> on usbus0
uhid1 on uhub1
uhid1: <VMware> on usbus0
pid 88103 (cc), uid 0, was killed: out of swap space
pid 85453 (make), uid 0, was killed: out of swap space
pid 2650 (ld.lld), uid 0, was killed: out of swap space

source tree is on svn release:
Comment 2 Mark Millard 2018-04-19 01:49:18 UTC
Given the exchanges on the lists about such, you are likely to be asked about the
configuration: Is ZFS involved? Just UFS?

Most of the reports are tied to ZFS-involved contexts. There have been some where ZFS was not involved but UFS was. (-r331879 may have largely taken care of no-ZFS contexts.)

There may be more context/environment information requested as well.

The reported problems are not limited to buildworld buildkernel sorts of activity,
but definitely include examples of such activity.
Comment 3 tschweikle 2018-04-20 11:09:44 UTC
Two systems, one zfs, one ufs, same problem

- 4GiB RAM, 8GiB Swap

# zfs get all zroot
NAME   PROPERTY              VALUE                  SOURCE
zroot  type                  filesystem             -
zroot  creation              Fri Oct 10 16:32 2014  -
zroot  used                  14.8G                  -
zroot  available             25.7G                  -
zroot  referenced            919M                   -
zroot  compressratio         1.15x                  -
zroot  mounted               yes                    -
zroot  quota                 none                   default
zroot  reservation           none                   default
zroot  recordsize            128K                   default
zroot  mountpoint            legacy                 local
zroot  sharenfs              off                    default
zroot  checksum              fletcher4              local
zroot  compression           off                    default
zroot  atime                 on                     default
zroot  devices               on                     default
zroot  exec                  on                     default
zroot  setuid                on                     default
zroot  readonly              off                    default
zroot  jailed                off                    default
zroot  snapdir               hidden                 default
zroot  aclmode               discard                default
zroot  aclinherit            restricted             default
zroot  canmount              on                     default
zroot  xattr                 off                    temporary
zroot  copies                1                      default
zroot  version               5                      -
zroot  utf8only              off                    -
zroot  normalization         none                   -
zroot  casesensitivity       sensitive              -
zroot  vscan                 off                    default
zroot  nbmand                off                    default
zroot  sharesmb              off                    default
zroot  refquota              none                   default
zroot  refreservation        none                   default
zroot  primarycache          all                    default
zroot  secondarycache        all                    default
zroot  usedbysnapshots       0                      -
zroot  usedbydataset         919M                   -
zroot  usedbychildren        13.9G                  -
zroot  usedbyrefreservation  0                      -
zroot  logbias               latency                default
zroot  dedup                 off                    default
zroot  mlslabel                                     -
zroot  sync                  standard               default
zroot  refcompressratio      1.00x                  -
zroot  written               919M                   -
zroot  logicalused           14.7G                  -
zroot  logicalreferenced     905M                   -
zroot  volmode               default                default
zroot  filesystem_limit      none                   default
zroot  snapshot_limit        none                   default
zroot  filesystem_count      none                   default
zroot  snapshot_count        none                   default
zroot  redundant_metadata    all                    default

#zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
zroot                      14.8G  25.7G   919M  legacy
zroot/home                  192K  25.7G    96K  /home
zroot/home/sct-muc           96K  25.7G    96K  /home/sct-muc
zroot/tmp                   188M  25.7G   188M  /tmp
zroot/usr                  10.1G  25.7G  3.88G  /usr
zroot/usr/ports            3.59G  25.7G  2.76G  /usr/ports
zroot/usr/ports/distfiles   853M  25.7G   853M  /usr/ports/distfiles
zroot/usr/ports/packages     96K  25.7G    96K  /usr/ports/packages
zroot/usr/src              2.66G  25.7G  2.66G  /usr/src
zroot/var                  3.52G  25.7G  3.09G  /var
zroot/var/crash             194M  25.7G   194M  /var/crash
zroot/var/db                241M  25.7G   196M  /var/db
zroot/var/db/pkg           45.0M  25.7G  45.0M  /var/db/pkg
zroot/var/empty              96K  25.7G    96K  /var/empty
zroot/var/log              8.85M  25.7G  8.85M  /var/log
zroot/var/mail              120K  25.7G   120K  /var/mail
zroot/var/run               184K  25.7G   184K  /var/run
zroot/var/tmp               172K  25.7G   172K  /var/tmp

#zpool status
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub canceled on Fri Apr 20 12:54:12 2018

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          da0p3     ONLINE       0     0     0

errors: No known data errors

# zpool list
zroot  41.8G  14.8G  27.0G         -    53%    35%  1.00x  ONLINE  -

# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/gpt/swap0    8388608        0  8388608     0%

- 2GiB RAM, 8GiB Swap
Comment 4 Mark Millard 2018-04-21 00:59:10 UTC
(In reply to tschweikle from comment #3)

It is interesting that you have examples of both contexts.

For the UFS context, if you can somehow get to -r331879 or later (or at least test with such a version), you might get past the problem for that context.
Comment 5 Mark Millard 2018-04-29 20:41:04 UTC
(In reply to tschweikle from comment #1)

Some characteristics of the environment
not mentioned are:

How many processors/cores/threads?
What sort of value for -j<?> for the buildworld/buildkernel.

For reference (but only a partial result so far):

I currently have a smaller aarch64 context building world
and kernel (4 core Pine64+ 2GiB, -j4, UFS, Swap 3584M).
Before this I updated to -r333079. The -r333079 was a jump
from materials from around the beginning of the year and was
done via a cross build. I then used poudriere-devel and pkg
to update ports (1 builder but allowing all 4 cores to be
used in a builder). It survived the somewhat over 29 hr
build of the 101 involved ports, 14.5 hr or so being for
devel/llvm60 . So far it is 10 hr or so into the -j4
buildworld and shows no problems yet, but it will be some
time before it is done if it finishes okay. I'll report.

(I use a 128 GiB emmc on an adapter instead of using a
normal usdcard.)
Comment 6 Mark Millard 2018-04-30 13:06:06 UTC
(In reply to Mark Millard from comment #5)

The Pine64+ 2GiB finished the buildworld buildkernel in
a little under 19.5 hr.
Comment 7 Mark Millard 2018-08-12 17:24:47 UTC
(In reply to Mark Millard from comment #6)

As of updating to -r337400 the Pine64+ 2GB no
longer will boot from the e.MMC on the microsd
adapter card. (I switched to tracking fully
modern dts use, u-boot, etc.)

So I tried a build via a USB SSD as the root
file system and swap partition. As reported in:


it failed with an OOM kill.

This should have avoided I/O latency problems being
involved. (That message is part of a long on-going
thread tied to OOM kills, most of the reports involving
large I/O latencies being involved.)

I can not change the "Affects Only Me status.
Comment 8 Mark Millard 2018-08-12 17:52:17 UTC
(In reply to Mark Millard from comment #7)

Other bugzilla's are: 230402 230454.
Comment 9 Mark Linimon freebsd_committer freebsd_triage 2018-08-13 01:24:07 UTC
(In reply to Mark Millard from comment #7)

I've changed the "affects only me" status but honestly twiddling that knob doesn't really do anything.
Comment 10 matthias.freitag 2018-09-03 20:24:15 UTC
*** Bug 231123 has been marked as a duplicate of this bug. ***
Comment 11 Mark Millard 2020-04-01 19:05:52 UTC
(In reply to Mark Millard from comment #7)

[I've been updating old notes in this area with
recent results from a 1 GiByte RAM context: an
RPi3. It can do a head -j4 buildworld buildkernel
when appropriately configured. Material copied to
below from another place I updated.]

I've tried a 1 GiByte aarch64 context for a -j4
buildworld buildkernel . . .

I had a RPi3 that was based on head -r358966 do a
build world buildkernel of the same version, from
scratch, -j4 style. The RPi3 is a 1 GiByte RAM
context. I had 3072 GiBytes for the swap partition.
That ,and the ufs file system, were on a USB SSD,
not the microsd card.

The build completed without any /var/log/message or
console output during the build. My modified version
of top reported (details copied from a ssh window) . . .

For Mem: 738512Ki MaxObsActive, 190608Ki MaxObsWired, 906372Ki MaxObs(Act+Wir)
For Swap:  1927Mi MaxObsUsed

(top was started before the build. "MaxObs" is short
for "Maximum Observed".)

The build took a few minutes under 31 hrs.

The build used (my PINE64 media are also set up
to boot the RPi3, explaining some naming):

/dev/gpt/PINE642Groot           /               ufs rw,noatime          1 1
/dev/gpt/PINE642Gswap           none            swap sw                 0 0

(So this avoided the microsd card for ufs and
swap/page space.)

Overall, it looks like having more than 2 GiBytes
of swap partition is appropriate for -j4 : 1927
MiByte is not much less than 2048 MiByte.

But, with appropriate configuration anyway, the
RPi3 can do buildworld buildkernel for head 13,
even -j4 style.

This was aarch64. armv7 style with 1 GiByte RAM
does not allow as much swap/page space without
complaining at boot. It does not appear that
such a -j4 build would be appropriate for armv7.
But I've not investigated what would fit.
Comment 12 Vladislav V. Prodan 2021-02-12 15:32:41 UTC
# uname -a
FreeBSD vb-12.2.0.domain.com 12.2-STABLE FreeBSD 12.2-STABLE r369250 GENERIC  amd64

# uname -KU
1202505 1202505

RAM  - 2GB
SWAP - 500MB

These options didn't help: 

We have an unfilled SWAP 
Swap: 500M Total, 106M Used, 394M Free, 21% Inuse

Constant warnings to the console and after a while one of the processes responsible for buildworld is killed:

swap_pager_getswapspace(32): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(5): failed
swap_pager_getswapspace(6): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(2): failed
swap_pager_getswapspace(32): failed
pid 73372 (c++), jid 0, uid 0, was killed: out of swap space
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(28): failed
swap_pager_getswapspace(32): failed
Comment 13 Mark Millard 2021-02-12 20:23:23 UTC
(In reply to Vladislav V. Prodan from comment #12)

You do not indicate if your buildworld buildkernel
was using -j1 or some other -jN figure. The larger
the N the more memory space required.

> Swap: 500M Total, 106M Used, 394M Free, 21% Inuse

This can change far more rapidly than you can get
updates to be displayed.

Also, if the requested swap space is larger than the
free space, the used figure will not go to zero
temporarily but instead will stay unchanged.

But it does help to have very frequent output to get
an idea if one is getting close to running out. But
the display has to have started before the problem

Technically there can be allocation failures for
internal tracking reasons instead of just the swap
space on media.

Having only around 400M free is not much margin for
the swap space.

> RAM  - 2GB
> SWAP - 500MB

Having RAM+SWAP be only 2.5GB has be a problem for
various vintages of FreeBSD, even for -j1 in
some cases. But I do not have specifics for
stable/12 -r369250 .

> swap_pager_getswapspace(32): failed

This type of message means that you really did
reach the swap space limit (or an internal tracking
limit). Material was kept in RAM that it was
trying to write out to swap.

As far as I can tell, you need either a smaller -jN
figure or for the RAM+SWAP to be larger (or some
combination of such).

Also, are you using a swap partition(s)? A swap file?
I strongly suggest only using partition(s) for swap.
See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206048
for why.
Comment 14 Vladislav V. Prodan 2021-02-12 20:48:34 UTC
I set 1 CPU in this VM and am trying to make world and a custom kernel in one thread.

P.S. I have always used swap as a disk partition. 

# swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/gpt/swap-ada0    512000     209M     291M    42%

root@vb-12:~ # top -PSTH -b 10
last pid: 92497;  load averages:  1.01,  1.07,  1.03  up 0+04:15:08    22:46:09
475 threads:   4 running, 457 sleeping, 14 waiting
CPU: 84.3% user,  0.0% nice,  6.7% system,  0.3% interrupt,  8.8% idle
Mem: 145M Active, 51M Inact, 12M Laundry, 1395M Wired, 352M Free
ARC: 1013M Total, 318M MFU, 542M MRU, 2080K Anon, 13M Header, 139M Other
     744M Compressed, 2030M Uncompressed, 2.73:1 Ratio
Swap: 500M Total, 209M Used, 290M Free, 41% Inuse

100537 root         78    0   200M   148M RUN      0:03  23.00% c++
100003 root        155 ki31     0B    16K RUN     21:52   0.00% idle
100203 root        -12    -     0B  5088K -        2:06   0.00% zpool-zroot{zio_write_issue}
100037 root        -96    -     0B   224K WAIT     0:34   0.00% intr{irq21: pcm0 ahci0}
100020 root        -76    -     0B   400K -        0:28   0.00% kernel{if_io_tqg_0}
100078 root        -16    -     0B    16K -        0:21   0.00% rand_harvestq
100000 root        -16    -     0B   400K swapin   0:20   0.00% kernel{swapper}
100022 root        -76    -     0B   400K -        0:15   0.00% kernel{if_config_tqg_0}
100492 root         20    0   194M  8812K RUN      0:13   0.00% tmux
100044 root        -92    -     0B   224K WAIT     0:12   0.00% intr{irq19: em0:irq0+}
Comment 15 Mark Millard 2021-02-12 21:32:46 UTC
(In reply to Vladislav V. Prodan from comment #14)

I did not think to ask about ZFS use or its ARC
memory use.

My figures and experience are for UFS contexts that use
less memory.

So, under default tuning of ZFS, my judgments are probably
significant underestimates of the memory use.

I'm afraid I've no clue how to tune with ZFS involved. I
can only suggest trying to use less memory and/or having
even more swap.

The combination:

Mem: 145M Active, 51M Inact, 12M Laundry, 1395M Wired, 352M Free
ARC: 1013M Total
100537 root         78    0   200M   148M RUN      0:03  23.00% c++
(so 148M resident of 200M size)

Looks messy for the activity. Just the 1395M Wired leaves you only
something like (based on RAM+SWAP) 2560M - 1395M == 1165M space
for more and there can be compiles or link sa hake more space than
Comment 16 Mark Millard 2021-02-12 21:34:55 UTC
(In reply to Mark Millard from comment #15)

Sorry: "compiles or link sa hake more space" should
be "compiles or links that take more space".
Comment 17 Vladislav V. Prodan 2021-02-12 21:48:26 UTC
Thank you for reminding me, now I will edit the ZFS memory consumption settings for such a small amount of RAM after the installation script.
Comment 18 Mark Millard 2021-02-12 22:02:22 UTC
(In reply to Vladislav V. Prodan from comment #17)

It might be worth leaving notes here about what
combination of adjustments for ZFS proved
sufficient for your example context if you find
a combination that does work.

Finding information about settings for such
contexts seems difficult.