I believe this is the second time I've hit a python failure with indications that the port is not jobs safe. MAKE_JOBS_UNSAFE=yes should be set on this port. It will cause over 10,000 ports to skip in a bulk build if it breaks. =========================================================================== =======================<phase: build >============================ ===> Building for python27-2.7.9_1 --- Include/graminit.h --- --- python-gdb.py --- --- Parser/acceler.o --- --- Parser/grammar1.o --- --- Parser/listnode.o --- --- Include/graminit.h --- /usr/bin/make Parser/pgen --- python-gdb.py --- install -m 0644 ./Tools/gdb/libpython.py python-gdb.py --- Parser/grammar1.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c --- Parser/acceler.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c --- Parser/node.o --- --- Parser/listnode.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c --- Parser/node.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/node.o Parser/node.c --- Include/graminit.h --- --- Parser/tokenizer_pgen.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c --- Parser/printgrammar.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c --- Parser/pgenmain.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c --- Parser/parser.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/parser.o Parser/parser.c --- Parser/parsetok.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/parsetok.o Parser/parsetok.c --- Include/graminit.h --- --- Parser/bitset.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/bitset.o Parser/bitset.c --- Parser/metagrammar.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/metagrammar.o Parser/metagrammar.c --- Parser/metagrammar.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/metagrammar.o Parser/metagrammar.c --- Include/graminit.h --- --- Parser/firstsets.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/firstsets.o Parser/firstsets.c --- Parser/grammar.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/grammar.o Parser/grammar.c --- Parser/grammar.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/grammar.o Parser/grammar.c --- Parser/pgen.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/pgen.o Parser/pgen.c --- Include/graminit.h --- --- Objects/obmalloc.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Objects/obmalloc.o Objects/obmalloc.c --- Python/mysnprintf.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c --- Python/pyctype.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Python/pyctype.o Python/pyctype.c --- Parser/myreadline.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/myreadline.o Parser/myreadline.c --- Parser/tokenizer.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/tokenizer.o Parser/tokenizer.c --- Objects/abstract.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Objects/abstract.o Objects/abstract.c --- Objects/boolobject.o --- cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Objects/boolobject.o Objects/boolobject.c --- Include/graminit.h --- --- Parser/pgen --- cc -DNDEBUG -lpthread -Wl,-rpath=/usr/lib:/usr/local/lib -L/usr/local/lib Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -ldl -L/usr/local/lib -lintl -lutil -o Parser/pgen Parser/pgen.o: file not recognized: File truncated *** [Parser/pgen] Error code 1 make[2]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9 1 error make[2]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9 *** [Include/graminit.h] Error code 2 make[1]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9 1 error make[1]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9 ===> Compilation failed unexpectedly. Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer. *** Error code 1
Is the issue only intermittently reproducible, or reproducible on demand such as with high -jN? Upstream Python bug #22359 contains a patch which addresses what appears to be the same issue. Can you apply the patch and let us know how it goes? If this is the cause, then all branches < 3.5 will be affected See also: https://mail.python.org/pipermail/python-dev/2014-September/136255.html
It's hard to reproduce. I don't think the key is high -jN, but rather some -jN when the machine is under heavy load. This is only the second time I've seen it. So yes I can apply the patch, but I don't think that will really "prove" anything. It would probably build fine on the next attempt even if I don't apply the patch.
okay, the upstream patch seems to be to trunk and I can't see that it applies easily to python 2.7. I attempted the python27 build again, it passed as expected. It's going to be difficult to reproduce reliably. I think the machine has to be under high load.
How does it link at all with "-ldl" (last command in your output)? Just wonder.
DragonFly had a stub libdl.so. It doesn't do anything. We got tired of patching all the S/W to remove -ldl flags...
s/had a stub/has a stub/
I hit this again on another bulk build. I highly recommend that all affect python ports be set with MAKE_JOBS_UNSAFE=yes until a patch can be provided. It's clear there is an issue, I think.
John, just to clarify, you reproduced the issue with the upstream patch or without? If without, I can try to provide a back-ported patch if it would be helpful to you. In the short-term, and until we figure this out (root-cause and fix), I don't mind approving the MAKE_JOBS_UNSAFE=yes change once above question is clarified
I could not make a patch. The difference between trunk and version 2.7 is extreme. Thus, I did not add the patch. My only point is that I am hitting this regularly now and it broke 9500 ports from the last build, so the impact of not setting this jobs unsafe is severe -- one of the most severe in the tree. I think all affected python should be set to -j1 asap. P.S. I don't know what changed with python to make this happen now. I don't recall this ever happening before. It must be a recent addition to all branches.
Also, I need to iterate: Simply applying the patch will not prove anything. I can not reproduce the breakage at will. I would have to run with it for months to be confident it was working.
*** Bug 201974 has been marked as a duplicate of this bug. ***
A commit references this bug: Author: koobs Date: Thu Jul 30 03:31:58 UTC 2015 New revision: 393217 URL: https://svnweb.freebsd.org/changeset/ports/393217 Log: lang/python{27,32,33,34}: Mark MAKE_JOBS_UNSAFE Parser/pgen code intermittently and non-deterministically fails at build time causing errors including, among others: * Parser/pgen.o: file not recognized: File truncated * pgenmain.c:(.text+0x244): undefined reference to `_Py_pgen' This is apparently due to incorrect uses of recursive make [1] which was fixed in the upstream Python 'default' (3.5) branch [2]. This change marks all Python port versions as MAKE_JOBS_UNSANFE until the the original changeset [1] and the resulting regression in cross-builds [3], whos fix is still pending, can be backported. [1] https://bugs.python.org/issue22359 [2] https://hg.python.org/cpython/rev/c2a53aa27cad [3] https://bugs.python.org/issue22625 PR: 200622 Reported by: marino MFH: 2015Q3 Changes: head/lang/python27/Makefile head/lang/python32/Makefile head/lang/python33/Makefile head/lang/python34/Makefile
A commit references this bug: Author: koobs Date: Wed Aug 5 13:48:26 UTC 2015 New revision: 393590 URL: https://svnweb.freebsd.org/changeset/ports/393590 Log: MFH: r393217 lang/python{27,32,33,34}: Mark MAKE_JOBS_UNSAFE Parser/pgen code intermittently and non-deterministically fails at build time causing errors including, among others: * Parser/pgen.o: file not recognized: File truncated * pgenmain.c:(.text+0x244): undefined reference to `_Py_pgen' This is apparently due to incorrect uses of recursive make [1] which was fixed in the upstream Python 'default' (3.5) branch [2]. This change marks all Python port versions as MAKE_JOBS_UNSANFE until the the original changeset [1] and the resulting regression in cross-builds [3], whos fix is still pending, can be backported. [1] https://bugs.python.org/issue22359 [2] https://hg.python.org/cpython/rev/c2a53aa27cad [3] https://bugs.python.org/issue22625 PR: 200622 Approved by: portmgr (feld) Changes: _U branches/2015Q3/ branches/2015Q3/lang/python27/Makefile branches/2015Q3/lang/python32/Makefile branches/2015Q3/lang/python33/Makefile branches/2015Q3/lang/python34/Makefile
Committed & MFH'd to 2015Q3. Appropriate breadcrumbs and upstream links have been referenced in the Makefile for future backporting of the root-cause fix.
Purportedly fixed in 3.5+, maybe we can revisit this for python35-37? https://bugs.python.org/issue22359#msg264035
(looks like python35/Makefile just inherited it from 34 -- ditto 36 and 37.)
(In reply to Conrad Meyer from comment #15) I agree, try to build few times on 24+ core system and if it doesn't fail, I would enable concurrency and see what happens :)
(In reply to Dmitry Sivachenko from comment #17) Well, it works fine on a 16 core VM, if that helps :-).
Also seems totally fine on 32-ish core Threadripper (16 x 2 HT). Python3.7 build seems to max out at about 8-10 compilers at once anyway -- lots of mostly trivial quick-building C files and only a few that take time. Time savings on that Threadripper: Before: 142.70 real 122.50 user 11.85 sys After: $ time make -C lang/python37 99.05 real 157.87 user 13.41 sys Index: lang/python37/Makefile =================================================================== --- lang/python37/Makefile (revision 481792) +++ lang/python37/Makefile (working copy) @@ -37,7 +37,6 @@ TEST_ARGS= TESTOPTS=-j${MAKE_JOBS_NUMBER} MAKE_ARGS+= INSTALL_SHARED="${INSTALL_LIB}" # Strip shared library -MAKE_JOBS_UNSAFE= yes # Parser/pgen build bug. See Issue: 200622, 201974 SUB_FILES= pkg-message SUB_LIST= PYTHON_SUFFIX=${PYTHON_SUFFIX}
(In reply to Conrad Meyer from comment #15) I've created a new bug 232308 for removing MAKE_JOBS_UNSAFE
(In reply to Kubilay Kocak from comment #20) Perfect, thanks!