Bug 233543

Summary: bsdtar: enable xz threads support
Product: Base System Reporter: Stefan Ehmann <shoesoft>
Component: binAssignee: Martin Matuska <mm>
Status: In Progress ---    
Severity: Affects Only Me CC: alexandre.martins, cem, flo, freebsd, pi
Priority: --- Keywords: feature, patch, performance
Version: 12.0-STABLEFlags: koobs: mfc-stable11?
koobs: mfc-stable12?
Hardware: Any   
OS: Any   
Attachments:
Description Flags
patch enabling xz threads support
none
patch for ports-mgmt/pkg to fix pkg-static linking none

Description Stefan Ehmann 2018-11-26 18:55:24 UTC
Created attachment 199578 [details]
patch enabling xz threads support

Currently, bsdtar is built without XZ multi-threading support.

Can be tested with this command:
tar -Jcf /dev/null --options xz:threads=4 $HOME

After setting HAVE_LZMA_STREAM_ENCODER_MT, it works as expected, see attached patch.

If this feature is enabled, the xz:threads option should also be mentioned in the man page.

Either add a reference to -T option in xz(1) or duplicate the info from there:
> Specify the number of worker threads to use. Setting threads to
> a special value 0 makes xz use as many threads as there are CPU
> cores on the system.
Comment 1 Conrad Meyer freebsd_committer 2018-11-26 19:08:03 UTC
+1

Alternatively, a workaround is to just use 'tar -f - | xz -T0'.
Comment 2 commit-hook freebsd_committer 2018-11-26 21:46:03 UTC
A commit references this bug:

Author: mm
Date: Mon Nov 26 21:45:27 UTC 2018
New revision: 340997
URL: https://svnweb.freebsd.org/changeset/base/340997

Log:
  libarchive configuration changes
  - move HAVE_BZLIB_H, HAVE_LIBLZMA and HAVE_LZMA_H to config_freebsd.h
  - activate support for multi-threaded lzma encoding [1]

  PR:		233543 [1]
  Reported by:	cem
  MFC after:	1 week

Changes:
  head/lib/libarchive/Makefile
  head/lib/libarchive/config_freebsd.h
Comment 3 Conrad Meyer freebsd_committer 2018-11-26 23:32:36 UTC
(In reply to commit-hook from comment #2)
I didn't report it :-).
Comment 4 Martin Matuska freebsd_committer 2018-11-27 09:25:44 UTC
Sorry, oversaw that.
Comment 5 Conrad Meyer freebsd_committer 2018-12-20 16:10:05 UTC
*** Bug 234217 has been marked as a duplicate of this bug. ***
Comment 6 Florian Smeets freebsd_committer 2018-12-22 05:03:37 UTC
Mention that the patch that was committed in r340997 was reverted in r341453:

Author: sbruno
Date: Tue Dec  4 03:23:14 2018
New Revision: 341453
URL: https://svnweb.freebsd.org/changeset/base/341453

Log:
  Revert r340997 at the request of multiple users.
  - breaks ports-mgmt/pkg build for mips64, powerpc64 and i386 for some users.
  
  --- pkg-static ---
  /usr/lib/liblzma.a(stream_encoder_mt.o): In function `mythread_cond_init':
  /usr/local/poudriere/jails/ppc64/usr/src/contrib/xz/src/common/mythread.h:230:
  undefined reference to `pthread_condattr_init'
  
  Reported by:	jhibbits zeising
Comment 7 Stefan Ehmann 2018-12-22 11:34:32 UTC
I can reproduce this on i386, see below.

The problem is that -llzma occurs before -lpthread on the linker command for pkg-static.
If -lpthread is put at the end of the command, it links OK.
So far, I haven't been able to fix this is in the build system, probably some libtool magic required.

/bin/sh ../libtool  --tag=CC    --mode=link cc  -O2 -pipe  -Wno-error -fstack-protector -fno-strict-aliasing  -Wall -Wno-unused-function -D_BSD_SOURCE -DINET6=1  -all-static -fstack-protector   -Wl,--enable-new-dtags -o pkg-static  pkg-add.o pkg-alias.o  pkg-annotate.o pkg-audit.o  pkg-autoremove.o pkg-backup.o  pkg-check.o pkg-clean.o pkg-config.o  pkg-convert.o pkg-create.o  pkg-delete.o pkg-event.o pkg-fetch.o  pkg-globals.o pkg-info.o pkg-install.o  pkg-lock.o pkg-main.o pkg-plugins.o  pkg-query.o pkg-register.o pkg-repo.o  pkg-rquery.o pkg-search.o pkg-set.o  pkg-shell.o pkg-shlib.o pkg-ssh.o  pkg-stats.o pkg-update.o  pkg-updating.o pkg-upgrade.o  pkg-utils.o pkg-version.o pkg-which.o  ../libpkg/libpkg_static.la  ../compat/libbsd_compat.la -ljail  -larchive -lz -lutil -lbz2 -llzma -lssl  -lpthread -lcrypto -lm  -lelf  -ljail -larchive -lz -lbz2 -llzma
libtool: link: cc -O2 -pipe -Wno-error -fstack-protector -fno-strict-aliasing -Wall -Wno-unused-function -D_BSD_SOURCE -DINET6=1 -static -fstack-protector -Wl,--enable-new-dtags -o pkg-static pkg-add.o pkg-alias.o pkg-annotate.o pkg-audit.o pkg-autoremove.o pkg-backup.o pkg-check.o pkg-clean.o pkg-config.o pkg-convert.o pkg-create.o pkg-delete.o pkg-event.o pkg-fetch.o pkg-globals.o pkg-info.o pkg-install.o pkg-lock.o pkg-main.o pkg-plugins.o pkg-query.o pkg-register.o pkg-repo.o pkg-rquery.o pkg-search.o pkg-set.o pkg-shell.o pkg-shlib.o pkg-ssh.o pkg-stats.o pkg-update.o pkg-updating.o pkg-upgrade.o pkg-utils.o pkg-version.o pkg-which.o  ../libpkg/.libs/libpkg_static.a ../compat/.libs/libbsd_compat.a -lutil -lssl -lpthread -lcrypto -lm -lelf -ljail -larchive -lz -lbz2 -llzma
Comment 8 Stefan Ehmann 2019-02-10 22:00:56 UTC
Created attachment 201909 [details]
patch for ports-mgmt/pkg to fix pkg-static linking

Attached a patch for ports-mgmt/pkg that fixes pkg-static build on i386 for me (with threads support enabled).

I don't think that patching Makefile.in is the cleanest way to fix this. But it's already patched so it doesn't make things much worse.

Maybe it can be fixed directly in lib/libarchive but my efforts failed.