Bug 200622 - lang/python27: Not jobs safe (All Python < 3.5)
Summary: lang/python27: Not jobs safe (All Python < 3.5)
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Kubilay Kocak
URL: https://bugs.python.org/issue22359
Keywords: patch
: 201974 (view as bug list)
Depends on:
Blocks:
 
Reported: 2015-06-03 21:10 UTC by John Marino
Modified: 2018-10-16 15:31 UTC (History)
5 users (show)

See Also:
koobs: maintainer-feedback+
koobs: merge-quarterly+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description John Marino freebsd_committer 2015-06-03 21:10:43 UTC
I believe this is the second time I've hit a python failure with indications that the port is not jobs safe.

MAKE_JOBS_UNSAFE=yes should be set on this port.  It will cause over 10,000 ports to skip in a bulk build if it breaks.

===========================================================================
=======================<phase: build          >============================
===>  Building for python27-2.7.9_1
--- Include/graminit.h ---
--- python-gdb.py ---
--- Parser/acceler.o ---
--- Parser/grammar1.o ---
--- Parser/listnode.o ---
--- Include/graminit.h ---
/usr/bin/make Parser/pgen
--- python-gdb.py ---
install  -m 0644 ./Tools/gdb/libpython.py python-gdb.py
--- Parser/grammar1.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c
--- Parser/acceler.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
--- Parser/node.o ---
--- Parser/listnode.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c
--- Parser/node.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/node.o Parser/node.c
--- Include/graminit.h ---
--- Parser/tokenizer_pgen.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c
--- Parser/printgrammar.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c
--- Parser/pgenmain.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c
--- Parser/parser.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/parser.o Parser/parser.c
--- Parser/parsetok.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/parsetok.o Parser/parsetok.c
--- Include/graminit.h ---
--- Parser/bitset.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/bitset.o Parser/bitset.c
--- Parser/metagrammar.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/metagrammar.o Parser/metagrammar.c
--- Parser/metagrammar.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/metagrammar.o Parser/metagrammar.c
--- Include/graminit.h ---
--- Parser/firstsets.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/firstsets.o Parser/firstsets.c
--- Parser/grammar.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/grammar.o Parser/grammar.c
--- Parser/grammar.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/grammar.o Parser/grammar.c
--- Parser/pgen.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/pgen.o Parser/pgen.c
--- Include/graminit.h ---
--- Objects/obmalloc.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Objects/obmalloc.o Objects/obmalloc.c
--- Python/mysnprintf.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c
--- Python/pyctype.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Python/pyctype.o Python/pyctype.c
--- Parser/myreadline.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/myreadline.o Parser/myreadline.c
--- Parser/tokenizer.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Parser/tokenizer.o Parser/tokenizer.c
--- Objects/abstract.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Objects/abstract.o Objects/abstract.c
--- Objects/boolobject.o ---
cc -c -fno-strict-aliasing -pipe -O2 -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Objects/boolobject.o Objects/boolobject.c
--- Include/graminit.h ---
--- Parser/pgen ---
cc -DNDEBUG -lpthread  -Wl,-rpath=/usr/lib:/usr/local/lib -L/usr/local/lib Parser/acceler.o  Parser/grammar1.o  Parser/listnode.o  Parser/node.o  Parser/parser.o  Parser/parsetok.o  Parser/bitset.o  Parser/metagrammar.o  Parser/firstsets.o  Parser/grammar.o  Parser/pgen.o Objects/obmalloc.o  Python/mysnprintf.o  Python/pyctype.o  Parser/tokenizer_pgen.o  Parser/printgrammar.o  Parser/pgenmain.o -ldl -L/usr/local/lib -lintl -lutil -o Parser/pgen
Parser/pgen.o: file not recognized: File truncated
*** [Parser/pgen] Error code 1

make[2]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9
1 error

make[2]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9
*** [Include/graminit.h] Error code 2

make[1]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9
1 error

make[1]: stopped in /wrkdirs/lang/python27/work/Python-2.7.9
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2015-06-04 05:11:27 UTC
Is the issue only intermittently reproducible, or reproducible on demand such as with high -jN?

Upstream Python bug #22359 contains a patch which addresses what appears to be the same issue. Can you apply the patch and let us know how it goes?

If this is the cause, then all branches < 3.5 will be affected

See also: https://mail.python.org/pipermail/python-dev/2014-September/136255.html
Comment 2 John Marino freebsd_committer 2015-06-04 05:49:17 UTC
It's hard to reproduce.   I don't think the key is high -jN, but rather some -jN when the machine is under heavy load.

This is only the second time I've seen it.

So yes I can apply the patch, but I don't think that will really "prove" anything.  It would probably build fine on the next attempt even if I don't apply the patch.
Comment 3 John Marino freebsd_committer 2015-06-04 12:47:55 UTC
okay, the upstream patch seems to be to trunk and I can't see that it applies easily to python 2.7.

I attempted the python27 build again, it passed as expected.  It's going to be difficult to reproduce reliably.  I think the machine has to be under high load.
Comment 4 Dmitry Sivachenko freebsd_committer 2015-06-04 13:59:22 UTC
How does it link at all with "-ldl" (last command in your output)?
Just wonder.
Comment 5 John Marino freebsd_committer 2015-06-04 14:00:51 UTC
DragonFly had a stub libdl.so.  It doesn't do anything.  We got tired of patching all the S/W to remove -ldl flags...
Comment 6 John Marino freebsd_committer 2015-06-04 14:01:51 UTC
s/had a stub/has a stub/
Comment 7 John Marino freebsd_committer 2015-06-06 22:04:19 UTC
I hit this again on another bulk build.  I highly recommend that all affect python ports be set with MAKE_JOBS_UNSAFE=yes until a patch can be provided.  It's clear there is an issue, I think.
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2015-06-07 03:07:53 UTC
John, just to clarify, you reproduced the issue with the upstream patch or without?

If without, I can try to provide a back-ported patch if it would be helpful to you.

In the short-term, and until we figure this out (root-cause and fix), I don't mind approving the MAKE_JOBS_UNSAFE=yes change once above question is clarified
Comment 9 John Marino freebsd_committer 2015-06-07 06:11:22 UTC
I could not make a patch.  The difference between trunk and version 2.7 is extreme.

Thus, I did not add the patch.

My only point is that I am hitting this regularly now and it broke 9500 ports from the last build, so the impact of not setting this jobs unsafe is severe -- one of the most severe in the tree.

I think all affected python should be set to -j1 asap.



P.S.  I don't know what changed with python to make this happen now.  I don't recall this ever happening before.  It must be a recent addition to all branches.
Comment 10 John Marino freebsd_committer 2015-06-07 06:22:40 UTC
Also, I need to iterate:

Simply applying the patch will not prove anything.   I can not reproduce the breakage at will.  I would have to run with it for months to be confident it was working.
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2015-07-30 02:47:53 UTC
*** Bug 201974 has been marked as a duplicate of this bug. ***
Comment 12 commit-hook freebsd_committer 2015-07-30 03:32:58 UTC
A commit references this bug:

Author: koobs
Date: Thu Jul 30 03:31:58 UTC 2015
New revision: 393217
URL: https://svnweb.freebsd.org/changeset/ports/393217

Log:
  lang/python{27,32,33,34}: Mark MAKE_JOBS_UNSAFE

  Parser/pgen code intermittently and non-deterministically fails
  at build time causing errors including, among others:

   * Parser/pgen.o: file not recognized: File truncated
   * pgenmain.c:(.text+0x244): undefined reference to `_Py_pgen'

  This is apparently due to incorrect uses of recursive make [1] which
  was fixed in the upstream Python 'default' (3.5) branch [2].

  This change marks all Python port versions as MAKE_JOBS_UNSANFE until
  the the original changeset [1] and the resulting regression in
  cross-builds [3], whos fix is still pending, can be backported.

  [1] https://bugs.python.org/issue22359
  [2] https://hg.python.org/cpython/rev/c2a53aa27cad
  [3] https://bugs.python.org/issue22625

  PR:		200622
  Reported by:	marino
  MFH:		2015Q3

Changes:
  head/lang/python27/Makefile
  head/lang/python32/Makefile
  head/lang/python33/Makefile
  head/lang/python34/Makefile
Comment 13 commit-hook freebsd_committer 2015-08-05 13:48:32 UTC
A commit references this bug:

Author: koobs
Date: Wed Aug  5 13:48:26 UTC 2015
New revision: 393590
URL: https://svnweb.freebsd.org/changeset/ports/393590

Log:
  MFH: r393217

  lang/python{27,32,33,34}: Mark MAKE_JOBS_UNSAFE

  Parser/pgen code intermittently and non-deterministically fails
  at build time causing errors including, among others:

   * Parser/pgen.o: file not recognized: File truncated
   * pgenmain.c:(.text+0x244): undefined reference to `_Py_pgen'

  This is apparently due to incorrect uses of recursive make [1] which
  was fixed in the upstream Python 'default' (3.5) branch [2].

  This change marks all Python port versions as MAKE_JOBS_UNSANFE until
  the the original changeset [1] and the resulting regression in
  cross-builds [3], whos fix is still pending, can be backported.

  [1] https://bugs.python.org/issue22359
  [2] https://hg.python.org/cpython/rev/c2a53aa27cad
  [3] https://bugs.python.org/issue22625

  PR:		200622

  Approved by:	portmgr (feld)

Changes:
_U  branches/2015Q3/
  branches/2015Q3/lang/python27/Makefile
  branches/2015Q3/lang/python32/Makefile
  branches/2015Q3/lang/python33/Makefile
  branches/2015Q3/lang/python34/Makefile
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2015-08-05 13:49:48 UTC
Committed & MFH'd to 2015Q3. Appropriate breadcrumbs and upstream links have been referenced in the Makefile for future backporting of the root-cause fix.
Comment 15 Conrad Meyer freebsd_committer 2018-10-11 03:04:45 UTC
Purportedly fixed in 3.5+, maybe we can revisit this for python35-37?

https://bugs.python.org/issue22359#msg264035
Comment 16 Conrad Meyer freebsd_committer 2018-10-11 03:06:03 UTC
(looks like python35/Makefile just inherited it from 34 -- ditto 36 and 37.)
Comment 17 Dmitry Sivachenko freebsd_committer 2018-10-11 08:46:34 UTC
(In reply to Conrad Meyer from comment #15)


I agree, try to build few times on 24+ core system and if it doesn't fail, I would enable concurrency and see what happens :)
Comment 18 Conrad Meyer freebsd_committer 2018-10-11 09:00:21 UTC
(In reply to Dmitry Sivachenko from comment #17)
Well, it works fine on a 16 core VM, if that helps :-).
Comment 19 Conrad Meyer freebsd_committer 2018-10-11 09:27:36 UTC
Also seems totally fine on 32-ish core Threadripper (16 x 2 HT).  Python3.7 build seems to max out at about 8-10 compilers at once anyway -- lots of mostly trivial quick-building C files and only a few that take time.

Time savings on that Threadripper:

Before:
142.70 real       122.50 user        11.85 sys

After:
$ time make -C lang/python37
99.05 real       157.87 user        13.41 sys

Index: lang/python37/Makefile
===================================================================
--- lang/python37/Makefile      (revision 481792)
+++ lang/python37/Makefile      (working copy)
@@ -37,7 +37,6 @@
 TEST_ARGS=             TESTOPTS=-j${MAKE_JOBS_NUMBER}

 MAKE_ARGS+=            INSTALL_SHARED="${INSTALL_LIB}"                         # Strip shared library
-MAKE_JOBS_UNSAFE=      yes                                                     # Parser/pgen build bug. See Issue: 200622, 201974

 SUB_FILES=             pkg-message
 SUB_LIST=              PYTHON_SUFFIX=${PYTHON_SUFFIX}
Comment 20 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-16 06:48:22 UTC
(In reply to Conrad Meyer from comment #15)

I've created a new bug 232308 for removing MAKE_JOBS_UNSAFE
Comment 21 Conrad Meyer freebsd_committer 2018-10-16 15:31:59 UTC
(In reply to Kubilay Kocak from comment #20)
Perfect, thanks!