Summary: | lang/python27: Not jobs safe (All Python < 3.5) | ||
---|---|---|---|
Product: | Ports & Packages | Reporter: | John Marino <marino> |
Component: | Individual Port(s) | Assignee: | Kubilay Kocak <koobs> |
Status: | Closed FIXED | ||
Severity: | Affects Only Me | CC: | cem, demon, koobs, python, slvr32 |
Priority: | --- | Keywords: | patch |
Version: | Latest | Flags: | koobs:
maintainer-feedback+
koobs: merge-quarterly+ |
Hardware: | Any | ||
OS: | Any | ||
URL: | https://bugs.python.org/issue22359 | ||
See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232308 |
Description
John Marino
2015-06-03 21:10:43 UTC
Is the issue only intermittently reproducible, or reproducible on demand such as with high -jN? Upstream Python bug #22359 contains a patch which addresses what appears to be the same issue. Can you apply the patch and let us know how it goes? If this is the cause, then all branches < 3.5 will be affected See also: https://mail.python.org/pipermail/python-dev/2014-September/136255.html It's hard to reproduce. I don't think the key is high -jN, but rather some -jN when the machine is under heavy load. This is only the second time I've seen it. So yes I can apply the patch, but I don't think that will really "prove" anything. It would probably build fine on the next attempt even if I don't apply the patch. okay, the upstream patch seems to be to trunk and I can't see that it applies easily to python 2.7. I attempted the python27 build again, it passed as expected. It's going to be difficult to reproduce reliably. I think the machine has to be under high load. How does it link at all with "-ldl" (last command in your output)? Just wonder. DragonFly had a stub libdl.so. It doesn't do anything. We got tired of patching all the S/W to remove -ldl flags... s/had a stub/has a stub/ I hit this again on another bulk build. I highly recommend that all affect python ports be set with MAKE_JOBS_UNSAFE=yes until a patch can be provided. It's clear there is an issue, I think. John, just to clarify, you reproduced the issue with the upstream patch or without? If without, I can try to provide a back-ported patch if it would be helpful to you. In the short-term, and until we figure this out (root-cause and fix), I don't mind approving the MAKE_JOBS_UNSAFE=yes change once above question is clarified I could not make a patch. The difference between trunk and version 2.7 is extreme. Thus, I did not add the patch. My only point is that I am hitting this regularly now and it broke 9500 ports from the last build, so the impact of not setting this jobs unsafe is severe -- one of the most severe in the tree. I think all affected python should be set to -j1 asap. P.S. I don't know what changed with python to make this happen now. I don't recall this ever happening before. It must be a recent addition to all branches. Also, I need to iterate: Simply applying the patch will not prove anything. I can not reproduce the breakage at will. I would have to run with it for months to be confident it was working. *** Bug 201974 has been marked as a duplicate of this bug. *** A commit references this bug: Author: koobs Date: Thu Jul 30 03:31:58 UTC 2015 New revision: 393217 URL: https://svnweb.freebsd.org/changeset/ports/393217 Log: lang/python{27,32,33,34}: Mark MAKE_JOBS_UNSAFE Parser/pgen code intermittently and non-deterministically fails at build time causing errors including, among others: * Parser/pgen.o: file not recognized: File truncated * pgenmain.c:(.text+0x244): undefined reference to `_Py_pgen' This is apparently due to incorrect uses of recursive make [1] which was fixed in the upstream Python 'default' (3.5) branch [2]. This change marks all Python port versions as MAKE_JOBS_UNSANFE until the the original changeset [1] and the resulting regression in cross-builds [3], whos fix is still pending, can be backported. [1] https://bugs.python.org/issue22359 [2] https://hg.python.org/cpython/rev/c2a53aa27cad [3] https://bugs.python.org/issue22625 PR: 200622 Reported by: marino MFH: 2015Q3 Changes: head/lang/python27/Makefile head/lang/python32/Makefile head/lang/python33/Makefile head/lang/python34/Makefile A commit references this bug: Author: koobs Date: Wed Aug 5 13:48:26 UTC 2015 New revision: 393590 URL: https://svnweb.freebsd.org/changeset/ports/393590 Log: MFH: r393217 lang/python{27,32,33,34}: Mark MAKE_JOBS_UNSAFE Parser/pgen code intermittently and non-deterministically fails at build time causing errors including, among others: * Parser/pgen.o: file not recognized: File truncated * pgenmain.c:(.text+0x244): undefined reference to `_Py_pgen' This is apparently due to incorrect uses of recursive make [1] which was fixed in the upstream Python 'default' (3.5) branch [2]. This change marks all Python port versions as MAKE_JOBS_UNSANFE until the the original changeset [1] and the resulting regression in cross-builds [3], whos fix is still pending, can be backported. [1] https://bugs.python.org/issue22359 [2] https://hg.python.org/cpython/rev/c2a53aa27cad [3] https://bugs.python.org/issue22625 PR: 200622 Approved by: portmgr (feld) Changes: _U branches/2015Q3/ branches/2015Q3/lang/python27/Makefile branches/2015Q3/lang/python32/Makefile branches/2015Q3/lang/python33/Makefile branches/2015Q3/lang/python34/Makefile Committed & MFH'd to 2015Q3. Appropriate breadcrumbs and upstream links have been referenced in the Makefile for future backporting of the root-cause fix. Purportedly fixed in 3.5+, maybe we can revisit this for python35-37? https://bugs.python.org/issue22359#msg264035 (looks like python35/Makefile just inherited it from 34 -- ditto 36 and 37.) (In reply to Conrad Meyer from comment #15) I agree, try to build few times on 24+ core system and if it doesn't fail, I would enable concurrency and see what happens :) (In reply to Dmitry Sivachenko from comment #17) Well, it works fine on a 16 core VM, if that helps :-). Also seems totally fine on 32-ish core Threadripper (16 x 2 HT). Python3.7 build seems to max out at about 8-10 compilers at once anyway -- lots of mostly trivial quick-building C files and only a few that take time. Time savings on that Threadripper: Before: 142.70 real 122.50 user 11.85 sys After: $ time make -C lang/python37 99.05 real 157.87 user 13.41 sys Index: lang/python37/Makefile =================================================================== --- lang/python37/Makefile (revision 481792) +++ lang/python37/Makefile (working copy) @@ -37,7 +37,6 @@ TEST_ARGS= TESTOPTS=-j${MAKE_JOBS_NUMBER} MAKE_ARGS+= INSTALL_SHARED="${INSTALL_LIB}" # Strip shared library -MAKE_JOBS_UNSAFE= yes # Parser/pgen build bug. See Issue: 200622, 201974 SUB_FILES= pkg-message SUB_LIST= PYTHON_SUFFIX=${PYTHON_SUFFIX} (In reply to Conrad Meyer from comment #15) I've created a new bug 232308 for removing MAKE_JOBS_UNSAFE (In reply to Kubilay Kocak from comment #20) Perfect, thanks! |