Created attachment 199578 [details] patch enabling xz threads support Currently, bsdtar is built without XZ multi-threading support. Can be tested with this command: tar -Jcf /dev/null --options xz:threads=4 $HOME After setting HAVE_LZMA_STREAM_ENCODER_MT, it works as expected, see attached patch. If this feature is enabled, the xz:threads option should also be mentioned in the man page. Either add a reference to -T option in xz(1) or duplicate the info from there: > Specify the number of worker threads to use. Setting threads to > a special value 0 makes xz use as many threads as there are CPU > cores on the system.
+1 Alternatively, a workaround is to just use 'tar -f - | xz -T0'.
A commit references this bug: Author: mm Date: Mon Nov 26 21:45:27 UTC 2018 New revision: 340997 URL: https://svnweb.freebsd.org/changeset/base/340997 Log: libarchive configuration changes - move HAVE_BZLIB_H, HAVE_LIBLZMA and HAVE_LZMA_H to config_freebsd.h - activate support for multi-threaded lzma encoding [1] PR: 233543 [1] Reported by: cem MFC after: 1 week Changes: head/lib/libarchive/Makefile head/lib/libarchive/config_freebsd.h
(In reply to commit-hook from comment #2) I didn't report it :-).
Sorry, oversaw that.
*** Bug 234217 has been marked as a duplicate of this bug. ***
Mention that the patch that was committed in r340997 was reverted in r341453: Author: sbruno Date: Tue Dec 4 03:23:14 2018 New Revision: 341453 URL: https://svnweb.freebsd.org/changeset/base/341453 Log: Revert r340997 at the request of multiple users. - breaks ports-mgmt/pkg build for mips64, powerpc64 and i386 for some users. --- pkg-static --- /usr/lib/liblzma.a(stream_encoder_mt.o): In function `mythread_cond_init': /usr/local/poudriere/jails/ppc64/usr/src/contrib/xz/src/common/mythread.h:230: undefined reference to `pthread_condattr_init' Reported by: jhibbits zeising
I can reproduce this on i386, see below. The problem is that -llzma occurs before -lpthread on the linker command for pkg-static. If -lpthread is put at the end of the command, it links OK. So far, I haven't been able to fix this is in the build system, probably some libtool magic required. /bin/sh ../libtool --tag=CC --mode=link cc -O2 -pipe -Wno-error -fstack-protector -fno-strict-aliasing -Wall -Wno-unused-function -D_BSD_SOURCE -DINET6=1 -all-static -fstack-protector -Wl,--enable-new-dtags -o pkg-static pkg-add.o pkg-alias.o pkg-annotate.o pkg-audit.o pkg-autoremove.o pkg-backup.o pkg-check.o pkg-clean.o pkg-config.o pkg-convert.o pkg-create.o pkg-delete.o pkg-event.o pkg-fetch.o pkg-globals.o pkg-info.o pkg-install.o pkg-lock.o pkg-main.o pkg-plugins.o pkg-query.o pkg-register.o pkg-repo.o pkg-rquery.o pkg-search.o pkg-set.o pkg-shell.o pkg-shlib.o pkg-ssh.o pkg-stats.o pkg-update.o pkg-updating.o pkg-upgrade.o pkg-utils.o pkg-version.o pkg-which.o ../libpkg/libpkg_static.la ../compat/libbsd_compat.la -ljail -larchive -lz -lutil -lbz2 -llzma -lssl -lpthread -lcrypto -lm -lelf -ljail -larchive -lz -lbz2 -llzma libtool: link: cc -O2 -pipe -Wno-error -fstack-protector -fno-strict-aliasing -Wall -Wno-unused-function -D_BSD_SOURCE -DINET6=1 -static -fstack-protector -Wl,--enable-new-dtags -o pkg-static pkg-add.o pkg-alias.o pkg-annotate.o pkg-audit.o pkg-autoremove.o pkg-backup.o pkg-check.o pkg-clean.o pkg-config.o pkg-convert.o pkg-create.o pkg-delete.o pkg-event.o pkg-fetch.o pkg-globals.o pkg-info.o pkg-install.o pkg-lock.o pkg-main.o pkg-plugins.o pkg-query.o pkg-register.o pkg-repo.o pkg-rquery.o pkg-search.o pkg-set.o pkg-shell.o pkg-shlib.o pkg-ssh.o pkg-stats.o pkg-update.o pkg-updating.o pkg-upgrade.o pkg-utils.o pkg-version.o pkg-which.o ../libpkg/.libs/libpkg_static.a ../compat/.libs/libbsd_compat.a -lutil -lssl -lpthread -lcrypto -lm -lelf -ljail -larchive -lz -lbz2 -llzma
Created attachment 201909 [details] patch for ports-mgmt/pkg to fix pkg-static linking Attached a patch for ports-mgmt/pkg that fixes pkg-static build on i386 for me (with threads support enabled). I don't think that patching Makefile.in is the cleanest way to fix this. But it's already patched so it doesn't make things much worse. Maybe it can be fixed directly in lib/libarchive but my efforts failed.
I'd like to see this committed.
Created attachment 228222 [details] bsdtar(1) patch Document xz:threads option.
Created attachment 228274 [details] Re-enable multi-threading support for CURRENT (335c4f8edb3a) I believe it is safe to attempt this patch again. At least, it didn't break pkg on i386 for me. FYI, I also created patches for pkg to add multi-threading option. https://github.com/juikim/pkg/commit/67c9465ace456769bb2a41b72ade07c5c46953d6.patch This patch applies cleanly on ports/pkg (1.17.2) and ports/pkg-devel (1.17.99.5).
(In reply to Jung-uk Kim from comment #11) Sorry, there was a formatting bug in the pkg_create(3). Try this patch instead: https://github.com/juikim/pkg/commit/4e57147a0516b287b7c7898ac5a5615e36b536c8.patch
For better maintainability, it would be great to submit any code changes to contrib/libarchive as a PR to upstream.
(In reply to Martin Matuska from comment #13) I agree but the only upstreamable libarchive patch is bsdtar(1) in comment #10.
What about config_freebsd.h? That's bundled upstream as well.
(In reply to Martin Matuska from comment #15) Is it really a static file? That's ugly. :-( Anyway, I'll submit a pull request, then.
(In reply to Jung-uk Kim from comment #16) Please see pull request #1664. https://github.com/libarchive/libarchive/pull/1664
(In reply to Felix Johnson from comment #10) Can you please submit your patch to upstream? https://github.com/libarchive/libarchive Thanks!
Created attachment 231620 [details] Re-enable multi-threading support for CURRENT (3ec0dc367bff)
(In reply to Jung-uk Kim from comment #18) I created a pull request myself. https://github.com/libarchive/libarchive/pull/1665
Created attachment 233058 [details] Re-enable multi-threading support for CURRENT after 7f815d4f128f Rebased for 7f815d4f128f
(In reply to Jung-uk Kim from comment #20) ^Triage: note that the pull request was accepted some time ago.
^Triage: this PR is now OBE since the upstream pull request was granted.