Bug 242896
| Summary: | lang/python*: Fail to package in poudriere (testport) when cross-building | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Christian Ullrich <chris> | ||||||||
| Component: | Individual Port(s) | Assignee: | freebsd-python (Nobody) <python> | ||||||||
| Status: | Open --- | ||||||||||
| Severity: | Affects Some People | CC: | bdrewery, bugmeister, koobs, mat, pi, python | ||||||||
| Priority: | --- | Keywords: | needs-qa | ||||||||
| Version: | Latest | Flags: | koobs:
merge-quarterly?
|
||||||||
| Hardware: | Any | ||||||||||
| OS: | Any | ||||||||||
| See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208282 | ||||||||||
| Attachments: |
|
||||||||||
Thank you for the report Christian, ^Triage: Request feedback from committer (sbruno) who resolved bug 208282 The block in question added in ports r447129 probably needs to be done earlier (than post-install) Created attachment 210252 [details]
Patch.
This patch fixes this bug and two related ones.
Firstly, it recompiles the .pyc files after modifying the .py file. This applies to lang/python3[578] because lang/python36 already has (most of) it.
Secondly, it adds the missing recompilation with -OO to lang/python36.
Thirdly, it generates correct file names involving ${ABIFLAGS}. There are several places in post-install where file names are assembled that include ABI flags. Previously, these were fixed as "m" (PYMALLOC enabled) for 3.[67] and empty for 3.8 (where the "m" flag has been removed upstream). (This only fixes the clearly somewhat rare case of cross-building 3.[67] with native-xtools and without PYMALLOC.)
Tested lang/python3[5-8] using poudriere bulk -t cross-building to aarch64, with and without PYMALLOC. Once the packages were successfully built, I manually checked that the .pycs are newer than the .py and that neither contains the /nxb-bin prefix.
Tested devel/py-zope.interface@py3[5-8] using poudriere bulk -t cross-building to aarch64, with and without PYMALLOC, using BUILD_ALL_PYTHON_FLAVORS. This port includes a binary extension, for added ABIFLAGS enjoyment. (I noticed that this .so is linked to the respective libpython....so with 3.[567], but not with 3.8. It can be imported and calling random functions on it does not crash, so I assume this is by design.)
I don't think I'll be able to look into this so I'm hoping someone else can pick this up. Someone with a better understanding of the machine that is the python build system would be desirable. Created attachment 215000 [details]
Patch.
Rebased patch after r536770.
Created attachment 258449 [details]
patch to lang/python*
^Triage: remove obsolete versions from patch.
^Triage: to submitter: I'm sorry that this PR did not get looked at in a reasonable period of time. Several of the patches pertained to python versions that are now obsolete. As well, new python versions have been added in the meantime (39, 310, 311). Is it still worth attempting to work on this bug report? (In reply to Mark Linimon from comment #6) > Is it still worth attempting to work on this bug report? To be honest, I don't know. I can say that the secondary symptom fixed by (what remains of) my patch, i.e. the /nxb-bin paths in the .opt-2.pyc file, is still there in 3.8 and 3.11 (so probably in the two in between as well). The primary symptom, staging violations, does not happen anymore, but I don't know why. Some trivial tests to see if it causes something not to build or run have not turned up any problems either. My local overlay does not have anything like this patch in it, and certainly has not in years; I would have noticed any actual problem affecting my own use of Python on aarch64. |
[This is difficult to explain, and I'm not sure I've entirely understood it myself. Bear with me, please.] "poudriere testport" (as well as bulk -t) of Python ports (not the lang/python*, but the py-* ports) fails during stage when using a cross-build jail with native-xtools (/nxb-bin): =>> Error: Filesystem touched during stage (files must install to ${STAGEDIR}): usr/local/lib/python3.7/__pycache__/_sysconfigdata_m_freebsd12_.cpython-37.pyc: size (18800, 18752) Bug 208282 introduced a fix for incorrect contents of _sysconfigdata_etc_.py in the cross-build situation by SUB_FILESing away the /nxb-bin prefix, which has apparently worked well since, and *does* work (albeit by accident, see below) with "poudriere bulk". It looks like there was an oversight in this fix, in that the .py file is rewritten *after* it has been byte-compiled. Hence, the .pyc files still contain the /nxb-bin prefix in the various variables: # tar -xvf /usr/local/poudriere/data/packages/aarch64-default-raspi3/All/python37-3.7.6.txz &> /dev/null # fgrep nxb usr/local/lib/python3.7/_sysconfigdata_m_freebsd12_.py # strings usr/local/lib/python3.7/__pycache__/_sysconfigdata_m_freebsd12_.cpython-37.pyc | fgrep nxb-bin/usr/bin/cc | head -n 1 -L. -lpython3.7mzO/nxb-bin/usr/bin/cc -pthread -shared -L/usr/local/lib -fstack-protector-strongZ This inconsistency does not affect poudriere bulk because poudriere only looks for staging violations in testport, and in bulk with -t. However, the .pyc is updated during the build once something imports sysconfig, leading to the staging violation. I have no clue how to correctly fix this. Perhaps by recompiling _sysconfigdata_etc_.py after removing the prefixes, so that a) the files are consistent and b) the .pyc files' mtime is updated avoiding the automatic recompilation?