Bug 217010

Summary: FreeBSD-head-i386-build build is failing because xz ran out of memory
Product: Base System Reporter: Enji Cooper <ngie>
Component: miscAssignee: Xin LI <delphij>
Status: Closed FIXED    
Severity: Affects Some People CC: delphij, lwhsu
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   

Description Enji Cooper freebsd_committer freebsd_triage 2017-02-11 20:37:56 UTC 
From https://ci.freebsd.org/job/FreeBSD-head-i386-build/347/console :

19:50:56 cd //usr/src/release/dist/base;  tar cvf - --exclude usr/lib/debug . |  xz -T 0 > //usr/src/release/dist/base.txz
19:50:56 xz: (stdin): Cannot allocate memory
19:50:56 a .
19:50:56 a ./bin
19:50:56 a ./.profile
19:50:56 a ./.cshrc
19:50:56 a ./sbin
19:50:56 a ./lib
19:50:56 a ./tmp
19:50:56 a ./boot
19:50:56 a ./proc
19:50:56 a ./dev
19:50:56 a ./rescue
19:50:56 a ./media
19:50:56 a ./mnt
19:50:56 a ./libexec
19:50:56 a ./etc
19:50:56 a ./var
19:50:56 a ./usrtar: ./usr/: Write error
19:50:56 *** Error code 1
19:50:56 
19:50:56 Stop.
19:50:56 make[2]: stopped in /usr/src
19:50:56 *** Error code 1
19:50:56 
19:50:56 Stop.
19:50:56 make[1]: stopped in /usr/src
19:50:56 *** Error code 1
19:50:56 
19:50:56 Stop.
Comment 1 Enji Cooper freebsd_committer freebsd_triage 2017-02-11 20:38:29 UTC 
delphij: I'm CCing you because this might be an issue with contrib/xz .
Comment 2 Xin LI freebsd_committer freebsd_triage 2017-02-12 17:37:44 UTC 
(In reply to Ngie Cooper from comment #1)
The default memory usage per thread is, according to manual page of xz(1), 94MiB:

                     -6            8 MiB            6            94 MiB            9 MiB


So with -T 0, the maximum usage would be greater than 2GiB on a system with 12 CPUs for a sufficiently large file.

Can you try e.g. changing the threads to a limited number, e.g. cap at 8 for 32-bit systems and see if that would mitigate this?

If that works, I think we need to rethink the -T 0 strategy...
Comment 3 Li-Wen Hsu freebsd_committer freebsd_triage 2017-02-12 20:07:17 UTC 
(In reply to Xin LI from comment #2)
I've tried putting "XZ_CMD=xz -T 8" in make.conf and that works.
Comment 4 Li-Wen Hsu freebsd_committer freebsd_triage 2017-02-16 05:08:51 UTC 
Should we fix this in release/ and/or xz itself to prevent it exhausting all memory?
Comment 5 Li-Wen Hsu freebsd_committer freebsd_triage 2017-02-18 02:56:37 UTC 
After discussing with delphij@, we should fix this in xz(1)
Comment 6 Li-Wen Hsu freebsd_committer freebsd_triage 2017-02-18 02:57:36 UTC 
Xin, could you help to check this?
Comment 7 Xin LI freebsd_committer freebsd_triage 2021-05-02 16:42:54 UTC 
This was fixed in 866fe991ac4014aafb031f2651f52444e2fdf3c5 (and upstream fixed it differently in f99e4a2d11f051c280ce97f0feb06463a9e8c23d).