Bug 242896 - lang/python*: Fail to package in poudriere (testport) when cross-building
Summary: lang/python*: Fail to package in poudriere (testport) when cross-building
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-python (Nobody)
Keywords: needs-patch, needs-qa
Depends on:
Reported: 2019-12-26 18:44 UTC by Christian Ullrich
Modified: 2020-05-29 06:57 UTC (History)
4 users (show)

See Also:
koobs: merge-quarterly?

Patch. (5.89 KB, patch)
2019-12-27 14:11 UTC, Christian Ullrich
no flags Details | Diff
Patch. (4.75 KB, patch)
2020-05-29 06:57 UTC, Christian Ullrich
chris: maintainer-approval?
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Ullrich 2019-12-26 18:44:39 UTC
[This is difficult to explain, and I'm not sure I've entirely understood it myself. Bear with me, please.]

"poudriere testport" (as well as bulk -t) of Python ports (not the lang/python*, but the py-* ports) fails during stage when using a cross-build jail with native-xtools (/nxb-bin):

=>> Error: Filesystem touched during stage (files must install to ${STAGEDIR}):
        size (18800, 18752)
Bug 208282 introduced a fix for incorrect contents of _sysconfigdata_etc_.py in the cross-build situation by SUB_FILESing away the /nxb-bin prefix, which has apparently worked well since, and *does* work (albeit by accident, see below) with "poudriere bulk".

It looks like there was an oversight in this fix, in that the .py file is rewritten *after* it has been byte-compiled. Hence, the .pyc files still contain the /nxb-bin prefix in the various variables:

# tar -xvf /usr/local/poudriere/data/packages/aarch64-default-raspi3/All/python37-3.7.6.txz &> /dev/null
# fgrep nxb usr/local/lib/python3.7/_sysconfigdata_m_freebsd12_.py
# strings usr/local/lib/python3.7/__pycache__/_sysconfigdata_m_freebsd12_.cpython-37.pyc | fgrep nxb-bin/usr/bin/cc | head -n 1
-L. -lpython3.7mzO/nxb-bin/usr/bin/cc -pthread -shared -L/usr/local/lib  -fstack-protector-strongZ

This inconsistency does not affect poudriere bulk because poudriere only looks for staging violations in testport, and in bulk with -t. However, the .pyc is updated during the build once something imports sysconfig, leading to the staging violation.

I have no clue how to correctly fix this. Perhaps by recompiling _sysconfigdata_etc_.py after removing the prefixes, so that a) the files are consistent and b) the .pyc files' mtime is updated avoiding the automatic recompilation?
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-27 02:25:05 UTC
Thank you for the report Christian,

^Triage: Request feedback from committer (sbruno) who resolved bug 208282 

The block in question added in ports r447129 probably needs to be done earlier (than post-install)
Comment 2 Christian Ullrich 2019-12-27 14:11:51 UTC
Created attachment 210252 [details]

This patch fixes this bug and two related ones.

Firstly, it recompiles the .pyc files after modifying the .py file. This applies to lang/python3[578] because lang/python36 already has (most of) it.

Secondly, it adds the missing recompilation with -OO to lang/python36.

Thirdly, it generates correct file names involving ${ABIFLAGS}. There are several places in post-install where file names are assembled that include ABI flags. Previously, these were fixed as "m" (PYMALLOC enabled) for 3.[67] and empty for 3.8 (where the "m" flag has been removed upstream). (This only fixes the clearly somewhat rare case of cross-building 3.[67] with native-xtools and without PYMALLOC.)

Tested lang/python3[5-8] using poudriere bulk -t cross-building to aarch64, with and without PYMALLOC. Once the packages were successfully built, I manually checked that the .pycs are newer than the .py and that neither contains the /nxb-bin prefix.

Tested devel/py-zope.interface@py3[5-8] using poudriere bulk -t cross-building to aarch64, with and without PYMALLOC, using BUILD_ALL_PYTHON_FLAVORS. This port includes a binary extension, for added ABIFLAGS enjoyment. (I noticed that this .so is linked to the respective libpython....so with 3.[567], but not with 3.8. It can be imported and calling random functions on it does not crash, so I assume this is by design.)
Comment 3 Sean Bruno freebsd_committer 2019-12-29 21:19:45 UTC
I don't think I'll be able to look into this so I'm hoping someone else can pick this up.

Someone with a better understanding of the machine that is the python build system would be desirable.
Comment 4 Christian Ullrich 2020-05-29 06:57:56 UTC
Created attachment 215000 [details]

Rebased patch after r536770.