Created attachment 168607 [details] python _sysconfigdata.py Python saves it's build environment variable in ${PREFIX}/lib/pythonX.X/_sysconfigdata.py This is used to provide module which needs to compile C/C++ code the compiler, cflags etc ... Since the official packages are build via qemu-user-static with a non-standard path (/nxb-bin/) wrong variables are saved and package cannot be build on the arm board. I based my patch on bug #205600. Tested with python2.7, 3.3 and 3.4 Note that this affect also mips and possibly others.
There is a typo in your patch: REINPLACE_CMDNPLACE_CMD should be REINPLACE_CMD
Created attachment 170050 [details] python _sysconfigdata.py
@Emmanuel, To what extent is this issue a bug that should also be fixed upstream?
This is usually not really a bug, it expects to be able to use the compiler it was built with afterwards.
Yeah this seems right to me.
Yeah it's not a bug in python. I've missed a Makefile that is also used (I must have patch it locally and simply forgot about it), I'll update the diff as soon as my poudriere is up and running.
Created attachment 171923 [details] python arm fix This patch would do.
I'm a little confused. If it's not a bug in Python and expected behaviour, why are changes being made in lang/python*? Is a ".if ${CC} == /nxb-bin/usr/bin/cc" block also required for every other language port that expects/does/embeds the same? Why doesn't the embedded compiler location/executable exist?
When cross-builing python with qemu-static the compiler is /nxb-bin/usr/bin/cc and python save this value for compiling module after. /nxb-bin/ doesn't exists on arm image (or mips or whatever). If I was crazy enough to build python from an arm board I wouldn't have the problem. As for other ports it's possible, I know that perl needs it and iirc the changes have been commited.
(In reply to Emmanuel Vadot from comment #9) Could this be done in python.mk or at a higher level than in individual ports?
I don't know if it's easily doable on a higher level, I don't know the port infrastructure. I've put it here because it's simple and since there is already stuff done in every python port Makefile I guessed that it was okay. All I want is to be able to install python module that compile C file (that's not really uncommon).
ping guys ?
*** Bug 219658 has been marked as a duplicate of this bug. ***
This is an artifact of the way we build arm packages using qemu-user-static. It is a bug for the resulting package to reference /nxb-bin/usr/bin/cc, since that path will not exist on a system on which the package is installed.
Some examples of users running into this issue: http://www.bsdforen.de/threads/python-modul-f%C3%BCr-gpio.32163/ https://strog.org/2017/02/19/installing-x84-bbs-on-raspberry-pi-zero-running-freebsd-11 https://github.com/yzgyyang/guide/blob/master/BeagleBone_FreeBSD_SPI_Setup.md
I'm pretty sure that my patch is outdated now but it should be easy to update it for latest python version in ports.
(In reply to Ed Maste from comment #14) Do you know of a bugzilla issue this one can be blocked by to at least document (if not resolve) the root cause bug in qemu-user-static? If not can one be created? CC maintainer. Would prefer to keep 'problem' issues and 'cause/solution' issues separate, particularly in cases where the root cause and problem are in different domains (in this case different ports), and looked after by different people or teams.
To be explicit, if my reading of comment 14 is correct and the root cause is within qemu-user-static and not in lang/python*, then its preferable to fix it there over a workaround in each python port.
> Do you know of a bugzilla issue this one can be blocked by to at least > document (if not resolve) the root cause bug in qemu-user-static? There's no bug in qemu-user-static. The bug is in the way we build the package. In brief: - The arm builds use qemu-user-static to execute individual ARM binaries on an amd64 host - Because this can be slow, the pkg build replaces some ARM binaries with native amd64 binaries that act exactly as the native ARM binaries they replace - /nxb-bin/usr/bin/cc is a version of Clang that runs on an amd64 host and defaults to producing arm output - /nxb-bin/usr/bin/cc exists nowhere but the FreeBSD arm package build infrastructure
It seems that poudriere concept of make.nxb.conf is suboptimal for cross-compile. I think that we must remove make.nxb.conf and either of an expand HLINK_FILES also to files previously used in make.nxb.conf (or better, to all files existing in /nxb-bin subtree) or use overlayfs (if it's stable) and mount /nxb-bin over jail's root Imho, both variants are easy enough and looks like real solution for me. There are many related problems with this, some configure scripts ignores env CC, LD variables for host based runtime tests (so it uses native(emulated) compiler), other ports uses clang++ (probably all py-qt4-* ports). Michal
(In reply to Michal Meloun from comment #20) I *think* the correct answer is that ports/Mk things should be taught to deal with the cross build environment of QEMU user. There is no bug in python's build or run time. There is no bug in qemu-user-static. The "bug" is that the build CC is not the runtime CC. Python cannot be the only port that is going to fail this way. The attached patch inflicts maintenance requirements on the *maintainer* of freebsd's python port. I don't think that's what we want here. It *does* work however. :-)
(In reply to Sean Bruno from comment #21) After being reminded by mat@ that we already have precedent for this type of fix in the tree (lang/perl5.26), I think this is a better solution than anything else we have at our disposal: ------------------------------------------------------------------------ r412502 | mat | 2016-04-04 03:48:54 -0600 (Mon, 04 Apr 2016) | 6 lines Fix manual build of Perl modules when Perl has been cross compiled. PR: 205600 Reported by: Mark Millard Sponsored by: Absolight ------------------------------------------------------------------------
A commit references this bug: Author: sbruno Date: Wed Aug 2 17:58:04 UTC 2017 New revision: 447129 URL: https://svnweb.freebsd.org/changeset/ports/447129 Log: Add a code block for the qemu-user enabled cross build environment. When using this environment in poudriere, CC is not set to the default of /usr/bin/cc and a cross-compile toolchain is used. We need to hand edit this so that the run time configuration for python matches what the FreeBSD base system provides. PR: 208282 Submitted by: manu Approved by: portmgr (mat) Changes: head/lang/python27/Makefile head/lang/python33/Makefile head/lang/python34/Makefile head/lang/python35/Makefile
A commit references this bug: Author: sbruno Date: Thu Aug 3 00:56:08 UTC 2017 New revision: 447189 URL: https://svnweb.freebsd.org/changeset/ports/447189 Log: Add a code block for the qemu-user enabled cross build environment. When using this environment in poudriere, CC is not set to the default of /usr/bin/cc and a cross-compile toolchain is used. We need to hand edit this so that the run time configuration for python matches what the FreeBSD base system provides. PR: 208282 Submitted by: manu Approved by: portmgr (mat) Changes: head/lang/python36/Makefile
I'd like to see a new issue created to (root cause) resolve this wherever it needs to be done, so that in the long-term the per-port specific blocks can ultimately be removed. Assign to committer resolving, retaining maintainer in CC
Whoops, it was Sean who done the deed
(In reply to Kubilay Kocak from comment #25) I'll ponder what can be done to the ports system to alleviate the maintainer burden and add a ticket when I come up with something.
Re-open: this commit broke python33, python34 and python35 on arm and mips Failure logs: http://beefy8.nyi.freebsd.org/data/head-armv6-default/p447380_s322073/logs/errors/python33-3.3.6_11.log http://beefy8.nyi.freebsd.org/data/head-armv6-default/p447380_s322073/logs/errors/python34-3.4.6_5.log http://beefy8.nyi.freebsd.org/data/head-armv6-default/p447380_s322073/logs/errors/python35-3.5.3_5.log
(In reply to Antoine Brodin from comment #28) Ugh ... testing now.
Author: sbruno Date: Tue Aug 8 01:39:37 UTC 2017 New revision: 447530 URL: https://svnweb.freebsd.org/changeset/ports/447530 Log: Pointyhat to me. A stray '.' somehow made it past my testing. Do *not* bump portrevision as this only affects the packaging/stage of these ports on mips/armv6 or other cross compiled targets. PR: 221202 Reported by: antoine Changes: head/lang/python33/Makefile head/lang/python34/Makefile head/lang/python35/Makefile
A commit references this bug: Author: sunpoet Date: Sun Aug 13 16:35:04 UTC 2017 New revision: 447898 URL: https://svnweb.freebsd.org/changeset/ports/447898 Log: MFH: r447129 r447530 Add a code block for the qemu-user enabled cross build environment. When using this environment in poudriere, CC is not set to the default of /usr/bin/cc and a cross-compile toolchain is used. We need to hand edit this so that the run time configuration for python matches what the FreeBSD base system provides. PR: 208282 Submitted by: manu Approved by: portmgr (mat) Pointyhat to me. A stray '.' somehow made it past my testing. Do *not* bump portrevision as this only affects the packaging/stage of these ports on mips/armv6 or other cross compiled targets. PR: 221202 Reported by: antoine Approved by: ports-secteam (zi) Changes: _U branches/2017Q3/ branches/2017Q3/lang/python27/Makefile branches/2017Q3/lang/python33/Makefile branches/2017Q3/lang/python34/Makefile branches/2017Q3/lang/python35/Makefile