Created attachment 214705 [details] error log $ poudriere jail -cj 121aarch64 -x -a arm64.aarch64 -v 12.1-RELEASE $ poudriere testport -j 121aarch64 textproc/py-sphinx [...] ===> Extracting for py37-sphinx-3.0.2,1 => SHA256 Checksum OK for python/Sphinx-3.0.2.tar.gz. tar: Pathname can't be converted from UTF-8 to current locale. tar: Error exit delayed from previous errors. *** Error code 1
(In reply to Jan Beich from comment #0) Hi. Sphinx was updated to 3.0.3. Could you check if this is still happening? Issue here when creating the env... [00:02:19] Error: Failed to 'make native-xtools'
(In reply to Danilo G. Baio from comment #1) > Sphinx was updated to 3.0.3. > Could you check if this is still happening? I can still reproduce: ===> Extracting for py37-sphinx-3.0.3,1 => SHA256 Checksum OK for python/Sphinx-3.0.3.tar.gz. tar: Pathname can't be converted from UTF-8 to current locale. tar: Error exit delayed from previous errors. *** Error code 1 > Issue here when creating the env... > [00:02:19] Error: Failed to 'make native-xtools' Try building from source instead e.g., $ poudriere jail -cj 121aarch64 -x -a arm64.aarch64 -v releng/12.1 -m svn+https Dropping -x (native-xtools) is not recommended as it'd make port-building extremely slow.
On our ref machines (ref[11|12|13]-aarch64.freebsd.org), that doesn't happen. I can simulate that on poudriere as you suggested (QEMU env). But see how it's odd: root@12-aarch64-default:~ # locale LANG= LC_CTYPE="C" LC_COLLATE="C" LC_TIME="C" LC_NUMERIC="C" LC_MONETARY="C" LC_MESSAGES="C" LC_ALL= root@12-aarch64-default:/tmp # tar -zxf /portdistfiles/python/Sphinx-3.0.3.tar.gz tar: Pathname can't be converted from UTF-8 to current locale. tar: Error exit delayed from previous errors. root@12-aarch64-default:/tmp # echo $? 1 root@12-aarch64-default:/tmp # setenv LANG en_US.UTF-8 root@12-aarch64-default:/tmp # setenv LC_ALL en_US.UTF-8 root@12-aarch64-default:/tmp # setenv MM_CHARSET UTF-8 root@12-aarch64-default:/tmp # locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_ALL=en_US.UTF-8 root@12-aarch64-default:/tmp # tar -zxf /portdistfiles/python/Sphinx-3.0.3.tar.gz root@12-aarch64-default:/tmp # echo $? 0 There are some workarounds here. But I think we should link this with a PR for bsdtar.
Created attachment 214764 [details] py-sphinx-3.0.3-QEMU_EMULATING.patch A possible workaround, using gtar.
FWIW- There's some semi-known issue with statically linked tar from the base system and UTF-8 filenames under poudriere; this has reproduced in the past with one of the lang/go ports as well, but I haven't had time to dig in yet. It's certainly a problem with tar rather than poudriere or sphinx, though.
(In reply to Kyle Evans from comment #5) Hi Kyle. Thanks for the heads up.
I've opened another PR to track this on base. (bug #246670) Jan, could you validate this patch on your arm* jails?
Tijl, why bug 215882 limits USE_LOCALE to configure and build phases? This port needs to set UTF-8 locale during extract phase. (In reply to Danilo G. Baio from comment #7) > Jan, could you validate this patch on your arm* jails? It does help. A better fix would be to set USE_LOCALE=en_US.UTF-8.
QEMU_EMULATING builds that don't use native-xtools are probably unaffected because /usr/bin/tar there is dynamically linked, not a hardlink to /nxb-bin/usr/bin/tar.
(In reply to Jan Beich from comment #8) > Tijl, why bug 215882 limits USE_LOCALE to configure and build phases? This port needs to set UTF-8 locale during extract phase. The logic behind that was that there's ports tree code and upstream code. That commit forces the C locale for the ports tree. USE_LOCALE can then be used to set a different locale for the upstream code. I agree it would make sense to extract files using the upstream locale, but I find it strange that tar worries about encoding. It should just treat file names as binary data like the file system does. How else can it handle packing and extracting /home with users using different locales? The error with static tar is probably because it cannot load modules from /usr/lib/i18n.
(In reply to Tijl Coosemans from comment #10) > I find it strange that tar worries about encoding. It should just treat file names as binary data like the file system does. How else can it handle packing and extracting /home with users using different locales? Apparently one can package a /home like that using tar cf home.tar --option hdrcharset=BINARY /home. Without that option tar converts all file names from the current locale to UTF-8. If "EXTRACT_CMD=${SETENV} LC_ALL=en_US.UTF-8 ${TAR}" in the port Makefile works I would use that for now. If more ports require this we can do something in bsd.port.mk. When FreeBSD 11 goes EOL we can also use C.UTF-8 as the ports tree locale. Dynamically linked tar may not error but it will try to convert the UTF-8 characters to US-ASCII and replace any character that fails with a question mark so whatever fix you choose you shouldn't condition it on QEMU_EMULATING.
I see the same error today on mips.mips64 using native-x-tools and poudriere-devel.
Created attachment 214844 [details] py-sphinx-3.0.3-QEMU_EMULATING-v02.patch
(In reply to Tijl Coosemans from comment #11) Thank you. If you all can test I'll appreciate.
Hi, With the patch, it will build in the directory on the (amd64 environment) server but it will not build in the poudriere jail (mips64.mips) (same error as before).
(In reply to tech-lists from comment #15) Try removing the .ifdef.
Created attachment 214873 [details] py-sphinx-tar_utf8.patch
(In reply to Danilo G. Baio from comment #17) I tested it on arm64.aarch64 (12.1). Could you confirm if it works on 11.X/CURRENT as well?
(In reply to Tijl Coosemans from comment #16) Theree's no .ifdef. in the Makefile
(In reply to Danilo G. Baio from comment #17) Problem persists on mips64.mips (native-x-tools) with this patch as well.
Comment on attachment 214873 [details] py-sphinx-tar_utf8.patch Works fine with the patch here: 11.3 aarch64/armv6, 12.1 aarch64/armv7/armv6, 13.0 armv7, all using native-xtools.
(In reply to tech-lists from comment #20) I can confirm the build still fails with the revised patch on mips64 using the following steps on 12.1-RELEASE-p5 amd64: # poudriere jail -c -j 121mips64 -b -m src=/usr/src -v releng/12.1 -a mips.mips64 -x -J 12 # poudriere testport -j 121mips64 textproc/py-sphinx ===> Extracting for py37-sphinx-3.0.3,1 => SHA256 Checksum OK for python/Sphinx-3.0.3.tar.gz. tar: Failed to set default locale tar: Pathname can't be converted from UTF-8 to current locale. tar: Error exit delayed from previous errors. *** Error code 1
(In reply to David Z. from comment #22) "Failed to set default locale" is something else. Doesn't /usr/share/locale/en_US.UTF-8 exist on mips64?
(In reply to Tijl Coosemans from comment #23) $ ls /poudriere/jails/121mips64/usr/share/locale/en_US.UTF-8/ LC_COLLATE LC_MESSAGES LC_NUMERIC LC_CTYPE LC_MONETARY LC_TIME
(In reply to David Z. from comment #24) Hmm, the only thing I can come up with is that mips is perhaps big-endian while the native tar is little-endian and that files under /usr/share/locale are endian dependent. Can you check if any of the files under /poudriere/jails/121mips64/usr/share/locale/en_US.UTF-8/ are different from the native files from the same FreeBSD version? Maybe create a hexdump of them using hd(1) and then diff(1) the hexdumps. See if there are any byte swaps.
Created attachment 214910 [details] py-sphinx-tar_utf8-mips.patch
lang/go has this same extract issue, and they use this EXTRACT_CMD: EXTRACT_CMD= ${SETENV} LC_ALL=en_US.UTF-8 ${TAR} but it has `ONLY_FOR_ARCHS= aarch64 amd64 armv6 armv7 i386`. I added in this patch (attachment 214910 [details]) the option to extract with cpio on mips*, could be gtar as well. I'm not so comfortable with this, CCing portmgr and linimon to help reviewing please. lang/go and now textproc/py-sphinx, probably we will face this issue again soon. In sphinx, the file that is causing this is an image file, used by tests, which is not even necessary for building. Sometimes I feel that is easier to remove that from the archive and host it somewhere.
(In reply to Danilo G. Baio from comment #27) You can't build lang/go with qemu-user-static
(In reply to Mikael Urankar from comment #28) Right, but you can't get past the extract phase to see that it's otherwise broken because of this. I hit this myself when I went to go look at it last time.
The files in /usr/share/locale are indeed big-endian on mips. That means "make native-xtools" in base should build/install its own /nxb-bin/usr/share/locale and compile libraries and programs with _PATH_LOCALE (in paths.h) set to "/nxb-bin/usr/share/locale". Until that is fixed in all supported FreeBSD versions ports will have to use "EXTRACT_CMD= ${SETENV} LC_ALL=en_US.UTF-8 /usr/bin/tar". cpio is also part of libarchive like tar, and suffers from the same problem.
(In reply to Tijl Coosemans from comment #30) Thank you again for the info. did you mean `/usr/bin/bsdtar` ? "EXTRACT_CMD= ${SETENV} LC_ALL=en_US.UTF-8 /usr/bin/bsdtar" ... # poudriere testport -j 12-mips64 -p default -o textproc/py-sphinx -I # jexec 12-mips64-default-n env -i TERM=$TERM /usr/bin/login -fp root root@12-mips64-default:/tmp # /usr/bin/tar -xf /portdistfiles/python/Sphinx-3.0.3.tar.gz tar: Pathname can't be converted from UTF-8 to current locale. tar: Error exit delayed from previous errors. root@12-mips64-default:/tmp # /usr/bin/bsdtar -xf /portdistfiles/python/Sphinx-3.0.3.tar.gz root@12-mips64-default:/tmp # echo $? 0 root@12-mips64-default:/tmp # file /usr/bin/tar /usr/bin/tar: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 12.1, FreeBSD-style, stripped root@12-mips64-default:/tmp # file /usr/bin/bsdtar /usr/bin/bsdtar: ELF 64-bit MSB executable, MIPS, MIPS-III version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, for FreeBSD 12.1, stripped root@12-mips64-default:/tmp # locale LANG= LC_CTYPE="C" LC_COLLATE="C" LC_TIME="C" LC_NUMERIC="C" LC_MONETARY="C" LC_MESSAGES="C" LC_ALL= side note, cpio can also extract sphinx on mips 12.1 mips.mips64 using native-xtools without LC_ALL=en_US.UTF-8.
(In reply to Danilo G. Baio from comment #31) Here /usr/bin/tar is a symbolic link to bsdtar. If poudriere replaces that with the one from /nxb-bin then yes, you'll have to use /usr/bin/bsdtar. Doesn't cpio produce the wrong file name for that image then?
(In reply to Tijl Coosemans from comment #32) Yes, but in this case this file can be ignored, unless you want to run `make test` Not defending the use of cpio, let's forget that. "EXTRACT_CMD= ${SETENV} LC_ALL=en_US.UTF-8 /usr/bin/bsdtar" This seems to cover all scenarios, thank you =) I've tested on: 11.3, 12.1, CURRENT (amd64, i386) 12.1 arm64.aarch64 using native-xtools 12.1 mips.mips64 using native-xtools jbeich@, could you run your tests again, please?
Created attachment 214961 [details] py-sphinx-bsdtar-utf8.patch
(In reply to Danilo G. Baio from comment #34) Can confirm this patch fixes the issue on mips64.mips, many thanks
Sorry, scratch that previous comment. It's still failing now but in a different place: [00:02:09] =========================================================================== [00:02:09] ====> Running Q/A tests (stage-qa) [00:02:12] ====> Checking for pkg-plist issues (check-plist) [00:02:12] ===> Parsing plist [00:02:13] ===> Checking for items in STAGEDIR missing from pkg-plist [00:02:13] ===> Checking for items in pkg-plist which are not in STAGEDIR [00:02:14] ===> No pkg-plist issues found (check-plist) [00:02:14] =>> Checking for staging violations... done [00:02:14] =>> Error: Filesystem touched during stage (files must install to ${STAGEDIR}): [00:02:14] usr/local/lib/python3.7/__pycache__/_sysconfigdata_m_freebsd12_.cpython-37.pyc: [00:02:14] size (18620, 18572) [00:02:14] Saving textproc/py-sphinx | py37-sphinx-3.0.3,1 wrkdir [00:02:29] Saved textproc/py-sphinx | py37-sphinx-3.0.3,1 wrkdir to: /usr/local/poudriere/data/wrkdirs/12R-mips64-x-default/default/py37-sphinx-3.0.3,1.txz [00:02:29] build of textproc/py-sphinx | py37-sphinx-3.0.3,1 ended at Thu May 28 14:42:38 BST 2020 [00:02:29] build time: 00:02:09 [00:02:29] !!! build failure encountered !!! [00:02:29] Error: Build failed in phase: stage_fs_violation [00:02:29] Logs: /usr/local/poudriere/data/logs/bulk/12R-mips64-x-default/2020-05-28_14h40m09s [00:02:29] WWW: https://REDACTED].zyxst.net//build.html?mastername=12R-mips64-x-default&build=2020-05-28_14h40m09s [00:02:29] Cleaning up [00:02:29] Unmounting file systems
A commit references this bug: Author: antoine Date: Thu May 28 13:52:34 UTC 2020 New revision: 536770 URL: https://svnweb.freebsd.org/changeset/ports/536770 Log: Recompile _sysconfigdata.py after reinplacing it PR: 246618 With hat: portmgr Changes: head/lang/python37/Makefile head/lang/python38/Makefile
Path: /usr/local/poudriere/ports/default Working Copy Root Path: /usr/local/poudriere/ports/default URL: https://svn.freebsd.org/ports/head Relative URL: ^/head Repository Root: https://svn.freebsd.org/ports Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5 Revision: 536770 Node Kind: directory Schedule: normal Last Changed Author: antoine Last Changed Rev: 536770 Last Changed Date: 2020-05-28 14:52:06 +0100 (Thu, 28 May 2020) # poudriere testport -j 12R-mips64-x -J64 textproc/py-sphinx [00:02:17] =>> Error: Filesystem touched during stage (files must install to ${STAGEDIR}): [00:02:17] usr/local/lib/python3.7/__pycache__/_sysconfigdata_m_freebsd12_.cpython-37.pyc: [00:02:17] size (18620, 18572) [00:02:17] Saving textproc/py-sphinx | py37-sphinx-3.0.3,1 wrkdir [00:02:32] Saved textproc/py-sphinx | py37-sphinx-3.0.3,1 wrkdir to: /usr/local/poudriere/data/wrkdirs/12R-mips64-x-default/default/py37-sphinx-3.0.3,1.txz [00:02:32] build of textproc/py-sphinx | py37-sphinx-3.0.3,1 ended at Thu May 28 15:28:13 BST 2020 [00:02:32] build time: 00:02:11 [00:02:32] !!! build failure encountered !!! [00:02:32] Error: Build failed in phase: stage_fs_violation [00:02:32] Logs: /usr/local/poudriere/data/logs/bulk/12R-mips64-x-default/2020-05-28_15h25m41s Would you like me to post the entire logfile?
(In reply to tech-lists from comment #38) WFM on amd64 cross-building mips64 with native-xtools after forcing a rebuild of python3.7 with the update in r536770.
Please note that these _sysconfigdata patches to 3.7 and 3.8 are incomplete; they omit the -OO case. The complete fix for all versions currently in tree has been languishing in my bug #242896 for six months now.
(In reply to David Z. from comment #39) I'm going to try rebuilding all the ports in mips64.mips vm and report back
(In reply to David Z. from comment #39) ok this builds on mips64.mips with the third patch posted after first deleting all packages. Many thanks,
A commit references this bug: Author: dbaio Date: Sat May 30 12:27:38 UTC 2020 New revision: 537077 URL: https://svnweb.freebsd.org/changeset/ports/537077 Log: textproc/py-sphinx: Fix build (extract) with static bsdtar(1) ===> Extracting for py37-sphinx-3.0.3,1 => SHA256 Checksum OK for python/Sphinx-3.0.3.tar.gz. tar: Pathname can't be converted from UTF-8 to current locale. tar: Error exit delayed from previous errors. *** Error code 1 Issue found at least on arm64.aarch64 and mips.mips64 builds using native-x-tools/poudriere. As /usr/bin/tar is replaced by the binary in /nxb-bin/, pointing EXTRACT_CMD to /usr/bin/bsdtar instead. root@12-mips64-default:/tmp # file /usr/bin/tar /usr/bin/tar: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 12.1, FreeBSD-style, stripped root@12-mips64-default:/tmp # file /usr/bin/bsdtar /usr/bin/bsdtar: ELF 64-bit MSB executable, MIPS, MIPS-III version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, for FreeBSD 12.1, stripped This patch bypass the issue here (all scenarios we have tested), but the problem still exists and it's being tracked in bug 246670. Please, see more details in bug 246618. Thanks to tijl, jbeich, kevans and all people who helped in testing. PR: 246618, 246670 Submitted by: tijl Reported by: jbeich Changes: head/textproc/py-sphinx/Makefile
Thank you everyone.
A commit references this bug: Author: dbaio Date: Mon Jun 15 11:22:39 UTC 2020 New revision: 538872 URL: https://svnweb.freebsd.org/changeset/ports/538872 Log: MFH: r536770 r536776 r538670 Recompile _sysconfigdata.py after reinplacing it PR: 246618 With hat: portmgr Fix build with various python ABI With hat: portmgr lang/python37: Fix security issues The patches for CVE-2019-18348 and CVE-2020-8492 are in the 3.7 branch and will be present on the next release. Patch for applying CVE-2020-8492 fix here in the ports tree was reported and submitted by Dani <i.dani@outlook.com>. PR: 246808 X-MFH-with: 536770, 536776 Security: ca595a25-91d8-11ea-b470-080027846a02 (CVE-2019-18348) Security: a27b0bb6-84fc-11ea-b5b4-641c67a117d8 (CVE-2020-8492) Approved by: ports-secteam (joneum) Changes: _U branches/2020Q2/ branches/2020Q2/lang/python37/Makefile branches/2020Q2/lang/python37/distinfo branches/2020Q2/lang/python38/Makefile