Bug 217010 - FreeBSD-head-i386-build build is failing because xz ran out of memory
Summary: FreeBSD-head-i386-build build is failing because xz ran out of memory
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Xin LI
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-02-11 20:37 UTC by Enji Cooper
Modified: 2018-08-12 14:34 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Enji Cooper freebsd_committer 2017-02-11 20:37:56 UTC
From https://ci.freebsd.org/job/FreeBSD-head-i386-build/347/console :

19:50:56 cd //usr/src/release/dist/base;  tar cvf - --exclude usr/lib/debug . |  xz -T 0 > //usr/src/release/dist/base.txz
19:50:56 xz: (stdin): Cannot allocate memory
19:50:56 a .
19:50:56 a ./bin
19:50:56 a ./.profile
19:50:56 a ./.cshrc
19:50:56 a ./sbin
19:50:56 a ./lib
19:50:56 a ./tmp
19:50:56 a ./boot
19:50:56 a ./proc
19:50:56 a ./dev
19:50:56 a ./rescue
19:50:56 a ./media
19:50:56 a ./mnt
19:50:56 a ./libexec
19:50:56 a ./etc
19:50:56 a ./var
19:50:56 a ./usrtar: ./usr/: Write error
19:50:56 *** Error code 1
19:50:56 
19:50:56 Stop.
19:50:56 make[2]: stopped in /usr/src
19:50:56 *** Error code 1
19:50:56 
19:50:56 Stop.
19:50:56 make[1]: stopped in /usr/src
19:50:56 *** Error code 1
19:50:56 
19:50:56 Stop.
Comment 1 Enji Cooper freebsd_committer 2017-02-11 20:38:29 UTC
delphij: I'm CCing you because this might be an issue with contrib/xz .
Comment 2 Xin LI freebsd_committer 2017-02-12 17:37:44 UTC
(In reply to Ngie Cooper from comment #1)
The default memory usage per thread is, according to manual page of xz(1), 94MiB:

                     -6            8 MiB            6            94 MiB            9 MiB


So with -T 0, the maximum usage would be greater than 2GiB on a system with 12 CPUs for a sufficiently large file.

Can you try e.g. changing the threads to a limited number, e.g. cap at 8 for 32-bit systems and see if that would mitigate this?

If that works, I think we need to rethink the -T 0 strategy...
Comment 3 Li-Wen Hsu freebsd_committer 2017-02-12 20:07:17 UTC
(In reply to Xin LI from comment #2)
I've tried putting "XZ_CMD=xz -T 8" in make.conf and that works.
Comment 4 Li-Wen Hsu freebsd_committer 2017-02-16 05:08:51 UTC
Should we fix this in release/ and/or xz itself to prevent it exhausting all memory?
Comment 5 Li-Wen Hsu freebsd_committer 2017-02-18 02:56:37 UTC
After discussing with delphij@, we should fix this in xz(1)
Comment 6 Li-Wen Hsu freebsd_committer 2017-02-18 02:57:36 UTC
Xin, could you help to check this?