Bug 252057 - lang/python38: 3.8.6_1 -> 3.8.7 strip issues
Summary: lang/python38: 3.8.6_1 -> 3.8.7 strip issues
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Wen Heping
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2020-12-22 23:19 UTC by John Kennedy
Modified: 2021-01-24 23:24 UTC (History)
7 users (show)

See Also:
koobs: maintainer-feedback+


Attachments
DRAFT-PR252057-fix-hardcoded-paths-for-python38.patch (83.12 KB, patch)
2020-12-23 13:03 UTC, Kai Knoblich
koobs: maintainer-approval+
Details | Diff
Mk/Uses/python.mk: Handle python dynamic library naming (2.44 KB, patch)
2020-12-23 13:45 UTC, Dima Panov
no flags Details | Diff
DRAFT-PR252057-fix-hardcoded-paths-for-python38-v2.patch (79.44 KB, patch)
2020-12-23 14:16 UTC, Kai Knoblich
kai: maintainer-approval+
Details | Diff
Mk/Uses/python.mk: Handle python dynamic library naming v2 (2.45 KB, patch)
2020-12-23 14:46 UTC, Dima Panov
no flags Details | Diff
Fix-hardcoded-paths-for-python38-v2.patch (75.38 KB, patch)
2020-12-24 11:18 UTC, Kai Knoblich
kai: maintainer-approval+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description John Kennedy 2020-12-22 23:19:27 UTC
After the upgrade, my local poudriere build failed 7 python packages and skipped another 96.

Failed ports: textproc/py-markupsafe:stage devel/py-cffi:stage textproc/py-pystemmer:stage textproc/py-libxml2:stage graphics/py-cairo:stage databases/py-sqlite3:stage devel/gobject-introspection:package

The first 3 I looked at all seemed to have similar issues with strip:

[py38-markupsafe]
running install_scripts
writing list of installed files to '/wrkdirs/usr/ports/textproc/py-markupsafe/work-py38/.PLIST.pymodtmp'
/usr/bin/strip /wrkdirs/usr/ports/textproc/py-markupsafe/work-py38/stage/usr/local/lib/python3.8/site-packages/markupsafe/_speedups.so
strip: open /wrkdirs/usr/ports/textproc/py-markupsafe/work-py38/stage/usr/local/lib/python3.8/site-packages/markupsafe/_speedups.so failed: No such file or directory
*** Error code 1

[py38-cffi]
running install_scripts
writing list of installed files to '/wrkdirs/usr/ports/devel/py-cffi/work-py38/.PLIST.pymodtmp'
/usr/bin/strip /wrkdirs/usr/ports/devel/py-cffi/work-py38/stage/usr/local/lib/python3.8/site-packages/_cffi_backend.so
strip: open /wrkdirs/usr/ports/devel/py-cffi/work-py38/stage/usr/local/lib/python3.8/site-packages/_cffi_backend.so failed: No such file or directory
*** Error code 1

[py38-pystemmer]
running install_scripts
writing list of installed files to '/wrkdirs/usr/ports/textproc/py-pystemmer/work-py38/.PLIST.pymodtmp'
/usr/bin/strip /wrkdirs/usr/ports/textproc/py-pystemmer/work-py38/stage/usr/local/lib/python3.8/site-packages/Stemmer.so
strip: open /wrkdirs/usr/ports/textproc/py-pystemmer/work-py38/stage/usr/local/lib/python3.8/site-packages/Stemmer.so failed: No such file or directory
*** Error code 1
Comment 1 John Kennedy 2020-12-23 03:16:28 UTC
This does appear to be just a python38 package problem.  If I just reverse the 3.8.7 patch on top of the revision=558955 ports tree, the other packages seem to be building.
Comment 2 Dima Panov freebsd_committer 2020-12-23 05:39:15 UTC
Holy cow!

This is a future of all python branches. We need to rework our py build system

* bpo-42604: Now all platforms use a value for the “EXT_SUFFIX” build variable derived from SOABI (for instance in freeBSD, “EXT_SUFFIX” is now “.cpython-310d.so” instead of “.so”). Previosuly only Linux, Mac and VxWorks were using a value for “EXT_SUFFIX” that included “SOABI”.
Comment 3 Dima Panov freebsd_committer 2020-12-23 05:49:00 UTC
(In reply to Dima Panov from comment #2)
See https://bugs.python.org/issue42604

py3.8 already get it with release, py3.9 will get it with 3.9.2, may be 3.6/3.7 will get it too
Comment 4 Kai Knoblich freebsd_committer 2020-12-23 09:26:46 UTC
The most Python ports that fails at the moment with Python 3.8 have a "post-patch" target which contain a fixed path to the shared library.

For example the databases/py-sqlite3 port has following line:

> post-install:
>         ${STRIP_CMD} ${STAGEDIR}${PYTHON_LIBDIR}/lib-dynload/_sqlite3.so

Changing it to

> post-install:
>         ${STRIP_CMD} ${STAGEDIR}${PYTHON_LIBDIR}/lib-dynload/*.so

remedies the issue.
Comment 5 Dima Panov freebsd_committer 2020-12-23 09:32:28 UTC
Now we have only two ways to implement.

Way #1 (bad): patch py38 to revert build scheme from .${SOABI}${SHLIB_SUFFIX} to .${SHLIB_SUFFIX} (old way, none of python consumers needs to ajusted)

Way #2 (Right Way™): patch all python3[679] to apply new build scheme (till it will be released oficially), patch Uses/python.mk to subst EXT_SUFFIX for plist AND fix all ports which build shared python library in portstree due to hardcoded paths and names in build/strip procedures and plists
Comment 6 Dima Panov freebsd_committer 2020-12-23 10:19:25 UTC
(In reply to Kai Knoblich from comment #4)
I prefer to not cut the whole library name but use named wildcard

${STRIP_CMD} ${STAGEDIR}${PYTHON_LIBDIR}/lib-dynload/_sqlite3*.so
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2020-12-23 11:11:15 UTC
(In reply to Dima Panov from comment #5)

We should use lang/python38 to first 'prototype' what might end up being back-ported (but possibly wont be) to 37,36.

We can move whatever common bits are needed longer term to python.mk in a separate changeset. Right now 38 is broken and I'm +1 on the minimal necessary step to get them working without a revert

At this point, Id actually be fixing what I consider to be a root cause, which is individual ports plists being coupled to the python implementation by hardcoding they're STRIP lines

@Kai are there any ports failing that *aren't* due to hardcoded <filename.so> lines?
Comment 8 Dima Panov freebsd_committer 2020-12-23 11:27:30 UTC
(In reply to Kubilay Kocak from comment #7)
Yep, revert is noop here because new python releases of all branchel will switch to this scheme unconditionally


> @Kai are there any ports failing that *aren't* due to hardcoded <filename.so> lines?

As I see in my poudriere, answer is 'no', all fails caused by strip/plist issues

and if for pure py-ports with autoplist we simply can fix by use named wildcard, ports like gobject-introspection or py-grpcio needs to use some hacks depends on py version.
And it is an argument for my opinion, posted earliesr -- all python3* ports should import the new naming scheme and all consumers will be adjusted without any extra .ifdefs
Comment 9 Dima Panov freebsd_committer 2020-12-23 11:56:34 UTC
Folks, I propose this solution: py_version depended variable PYTHON_EXT_SUFFIX which can be used in reinplace formulas and plist substitution. 

https://people.freebsd.org/~fluffy/-patches/python_ext.diff

It allows to keep unpatched python3[679] and handle shlib suffixes at consumers fields.

Tested, works for me with all py flavors

We need to correct many consumers ports but this tast will be anyway faced with future python releases so we can handle it before increment default python version :)
Comment 10 Kai Knoblich freebsd_committer 2020-12-23 13:03:46 UTC
Created attachment 220839 [details]
DRAFT-PR252057-fix-hardcoded-paths-for-python38.patch

Attached is a WIP patch that deals with the hardcoded path of most ports. It keeps the library name as suggested in comment #6.

I left out the devel/py-grpcio port and all the py-qt5-* ports as they don't seem to be affected.

Please note: Some ports could still be missing, because I've only looked for py-* ports make use of ${STRIP_CMD} and the whole patch still needs QA via bulk-runs.

Together with fluffy@'s patch the whole Python 3.8 issue could probably be straightened out relatively quickly.
Comment 11 Wen Heping freebsd_committer 2020-12-23 13:10:33 UTC
(In reply to Dima Panov from comment #9)
I would like use this patch to resolv the problem.

wen
Comment 12 Dima Panov freebsd_committer 2020-12-23 13:18:54 UTC
(In reply to Kai Knoblich from comment #10)
Please cut out emulators/ucon64 update from this patch

I've catches some other ports which affected library names too. They easy be fixed after python.mk patch is landed.
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2020-12-23 13:39:06 UTC
I think both changes should land (separately).

1) the hardcoded/coupled plist entries need to be fixed (kais patch, which resolves the present regression)

2) ports not using autoplist need a variable (which changes across versions, dima's patch)
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2020-12-23 13:41:02 UTC
Comment on attachment 220839 [details]
DRAFT-PR252057-fix-hardcoded-paths-for-python38.patch

Approved by: koobs (python)

Minus emulators/ucon64 PORTVERSION update chunk

@Dima can you attach your patch here too please
Comment 15 Dima Panov freebsd_committer 2020-12-23 13:45:06 UTC
Created attachment 220841 [details]
Mk/Uses/python.mk: Handle python dynamic library naming

patch aslo devel/gobject-introspection and devel/py-grpcio as examples to apply new library naming scheme
Comment 16 Kai Knoblich freebsd_committer 2020-12-23 14:16:19 UTC
Created attachment 220843 [details]
DRAFT-PR252057-fix-hardcoded-paths-for-python38-v2.patch

(In reply to Dima Panov from comment #12)

Oops, good catch! That one crept in and is removed from the updated patch.

Setting maintainer-approval flag accordingly as given by koobs@ in comment #14.

I'll soon do the appropriate QA steps and if everything is ok, commit the patch for the hardcoded paths accordingly.
Comment 17 Dima Panov freebsd_committer 2020-12-23 14:46:31 UTC
Created attachment 220844 [details]
Mk/Uses/python.mk: Handle python dynamic library naming v2

Sligtly modified version -- exclude '.so' from variable to allow to use it with not only library names, pkgconfig too, for example.
With py!= 38 variable will be empty

Catched on devel/talloc
Comment 18 Kai Knoblich freebsd_committer 2020-12-24 11:18:16 UTC
Created attachment 220888 [details]
Fix-hardcoded-paths-for-python38-v2.patch

Here's a cleaned-up patch that leaves out a couple of ports as listed below:

These ports use pkg-plists and should therefore be fixed once the patch for python.mk is in:

- archivers/py-lz4
- converters/py-bsdconv
- devel/py-jep
- graphics/py-cairo
- net/py-libdnet
- science/py-pymol
- security/py-gpgme

Following ports were also omitted because they are only for Python 2.7 and will be removed soon:

- databases/py-MySqlDb (Python 2.7 only, legacy)
- devel/py-subprocess32 (Python 2.7 only, legacy)
Comment 19 Dima Panov freebsd_committer 2020-12-24 11:37:43 UTC
(In reply to Kai Knoblich from comment #18)
Sure, I have plist patches prepared for those ports

Waiting for approval to patch Mk/Uses/python.mk to start committing fixes :)
Comment 20 Wen Heping freebsd_committer 2020-12-24 12:47:12 UTC
(In reply to Dima Panov from comment #19)
Please go ahead !

wen
Comment 21 commit-hook freebsd_committer 2020-12-24 13:14:42 UTC
A commit references this bug:

Author: fluffy
Date: Thu Dec 24 13:14:02 UTC 2020
New revision: 559071
URL: https://svnweb.freebsd.org/changeset/ports/559071

Log:
  Mk/Uses/python.mk: introduce PYTHON_EXT_SUFFIX variable

  Since Python-3.8.7 it uses unified with other platforns shared libs naming scheme:
  instead of plain .so suffix now it refers to used cpython: .cpython-${PYTHON_SUFFIX}.so

  Future Python releases will use this scheme too, it will be hooked via version check

  PR:		252057
  Approved by:	python (wen)

Changes:
  head/Mk/Uses/python.mk
Comment 22 Dima Panov freebsd_committer 2020-12-24 13:41:45 UTC
(In reply to Kai Knoblich from comment #18)
Mk variable landed, ports with plist changes are touched

@Kai, commit your part asap :)
Comment 23 commit-hook freebsd_committer 2020-12-24 13:46:46 UTC
A commit references this bug:

Author: kai
Date: Thu Dec 24 13:46:23 UTC 2020
New revision: 559094
URL: https://svnweb.freebsd.org/changeset/ports/559094

Log:
  Relax hardcoded paths to fix build with Python 3.8.7

  Since r558913 Python 3.8 incorporates BPO-42604 [1] which changed the
  shared libs naming scheme.  This means "EXT_SUFFIX" is now derived from
  SOABI and yields with Python 3.8 to ".cpython-38.so" instead of ".so".

  The affected ports strip the libaries in the "post-install" target via
  hardcoded path(s) and the build fails at the end because the new extension
  is not expected at this place.

  Remedy the issue by adding wildcards to these paths.  This should also
  prepare the ports for future Python releases, which will use the new shared
  libs naming scheme.

  [1] https://bugs.python.org/issue42604

  PR:		252057
  Reported by:	John Kennedy
  Reviewed by:	fluffy, koobs
  Approved by:	koobs (python)

Changes:
  head/archivers/py-brotli/Makefile
  head/archivers/py-lzma/Makefile
  head/archivers/py-python-lhafile/Makefile
  head/archivers/py-zstandard/Makefile
  head/archivers/py-zstd/Makefile
  head/astro/py-ephem/Makefile
  head/audio/py-aubio/Makefile
  head/audio/py-pyaudio/Makefile
  head/audio/py-pyliblo/Makefile
  head/audio/py-pymad/Makefile
  head/cad/py-gdspy/Makefile
  head/comms/py-hidapi/Makefile
  head/converters/py-rencode/Makefile
  head/converters/py-zfec/Makefile
  head/databases/py-PyGreSQL/Makefile
  head/databases/py-apsw/Makefile
  head/databases/py-asyncpg/Makefile
  head/databases/py-bsddb3/Makefile
  head/databases/py-cassandra-driver/Makefile
  head/databases/py-leveldb/Makefile
  head/databases/py-lmdb/Makefile
  head/databases/py-peewee/Makefile
  head/databases/py-pglast/Makefile
  head/databases/py-psycopg2/Makefile
  head/databases/py-pylibmc/Makefile
  head/databases/py-pymssql/Makefile
  head/databases/py-pyodbc/Makefile
  head/databases/py-sqlite3/Makefile
  head/databases/py-sqlrelay/Makefile
  head/devel/py-bullet3/Makefile
  head/devel/py-cbor/Makefile
  head/devel/py-cffi/Makefile
  head/devel/py-coverage/Makefile
  head/devel/py-crcmod/Makefile
  head/devel/py-cymem/Makefile
  head/devel/py-editdistance/Makefile
  head/devel/py-fam/Makefile
  head/devel/py-fastcache/Makefile
  head/devel/py-fastnumbers/Makefile
  head/devel/py-freebsd/Makefile
  head/devel/py-fusefs/Makefile
  head/devel/py-greenlet/Makefile
  head/devel/py-grpcio-tools/Makefile
  head/devel/py-ice37/Makefile
  head/devel/py-immutables/Makefile
  head/devel/py-intbitset/Makefile
  head/devel/py-lazy-object-proxy/Makefile
  head/devel/py-libversion/Makefile
  head/devel/py-libvirt/Makefile
  head/devel/py-llfuse/Makefile
  head/devel/py-llvmlite/Makefile
  head/devel/py-msgpack/Makefile
  head/devel/py-murmurhash/Makefile
  head/devel/py-opendht/Makefile
  head/devel/py-pendulum/Makefile
  head/devel/py-posix_ipc/Makefile
  head/devel/py-py-ubjson/Makefile
  head/devel/py-pyjq/Makefile
  head/devel/py-pyrsistent/Makefile
  head/devel/py-python-Levenshtein/Makefile
  head/devel/py-python-pcre/Makefile
  head/devel/py-ruamel.yaml.clib/Makefile
  head/devel/py-setproctitle/Makefile
  head/devel/py-simplejson/Makefile
  head/devel/py-urwid/Makefile
  head/devel/py-uvloop/Makefile
  head/devel/py-websockets/Makefile
  head/devel/py-wrapt/Makefile
  head/devel/py-xxhash/Makefile
  head/devel/py-yaml/Makefile
  head/devel/py-yappi/Makefile
  head/devel/py-zope.interface/Makefile
  head/dns/py-pycares/Makefile
  head/dns/py-pydnstable/Makefile
  head/dns/py-pywdns/Makefile
  head/finance/py-ta-lib/Makefile
  head/ftp/py-pycurl/Makefile
  head/graphics/py-ming/Makefile
  head/graphics/py-pygraphviz/Makefile
  head/graphics/py-zbar-py/Makefile
  head/mail/py-milter/Makefile
  head/mail/py-notmuch2/Makefile
  head/math/py-PySCIPOpt/Makefile
  head/math/py-cdecimal/Makefile
  head/math/py-cryptominisat/Makefile
  head/math/py-cvxpy/Makefile
  head/math/py-cyipopt/Makefile
  head/math/py-fastcluster/Makefile
  head/math/py-gmpy2/Makefile
  head/math/py-graphillion/Makefile
  head/math/py-hdmedians/Makefile
  head/math/py-levmar/Makefile
  head/math/py-libpoly/Makefile
  head/math/py-numexpr/Makefile
  head/math/py-numpy-stl/Makefile
  head/math/py-pdal/Makefile
  head/math/py-piranha/Makefile
  head/math/py-pycosat/Makefile
  head/math/py-pyhull/Makefile
  head/math/py-pyipopt/Makefile
  head/math/py-pynleq2/Makefile
  head/math/py-python-fcl/Makefile
  head/math/py-rpy2/Makefile
  head/math/py-symcxx/Makefile
  head/math/py-triangle/Makefile
  head/misc/py-fast-histogram/Makefile
  head/misc/py-onnx/Makefile
  head/misc/py-python-geohash/Makefile
  head/multimedia/py-cec/Makefile
  head/net/py-confluent-kafka/Makefile
  head/net/py-ldap/Makefile
  head/net/py-ldap0/Makefile
  head/net/py-libfte/Makefile
  head/net/py-maxminddb/Makefile
  head/net/py-miniupnpc/Makefile
  head/net/py-mpi4py/Makefile
  head/net/py-netifaces/Makefile
  head/net/py-pcapy/Makefile
  head/net/py-pyenet/Makefile
  head/net/py-pypcap/Makefile
  head/net/py-pysendfile/Makefile
  head/net/py-radix/Makefile
  head/net-im/py-toxcore-c/Makefile
  head/net-p2p/py-libtorrent-rasterbar/Makefile
  head/news/py-sabyenc3/Makefile
  head/print/py-psautohint/Makefile
  head/print/py-pycups/Makefile
  head/print/py-pycups1/Makefile
  head/science/py-gpaw/Makefile
  head/science/py-netCDF4/Makefile
  head/science/py-scikit-sparse/Makefile
  head/science/py-segyio/Makefile
  head/science/py-spglib/Makefile
  head/security/py-angr/Makefile
  head/security/py-coincurve/Makefile
  head/security/py-kerberos/Makefile
  head/security/py-m2crypto/Makefile
  head/security/py-pyscard/Makefile
  head/security/py-pysha3/Makefile
  head/security/py-python-axolotl-curve25519/Makefile
  head/security/py-pyvex/Makefile
  head/security/py-scrypt/Makefile
  head/security/py-sequoia/Makefile
  head/security/py-xmlsec/Makefile
  head/sysutils/py-psutil/Makefile
  head/sysutils/py-rdiff-backup/Makefile
  head/sysutils/py-scandir/Makefile
  head/sysutils/py-uptime/Makefile
  head/textproc/py-jsonslicer/Makefile
  head/textproc/py-libxml2/Makefile
  head/textproc/py-markupsafe/Makefile
  head/textproc/py-mwparserfromhell/Makefile
  head/textproc/py-patiencediff/Makefile
  head/textproc/py-pikepdf/Makefile
  head/textproc/py-pyahocorasick/Makefile
  head/textproc/py-pystemmer/Makefile
  head/textproc/py-regex/Makefile
  head/www/py-biscuits/Makefile
  head/www/py-html5-parser/Makefile
  head/www/py-tornado/Makefile
  head/www/py-tornado4/Makefile
  head/www/py-tornado6/Makefile
  head/www/py-wsaccel/Makefile
  head/www/py-yarl/Makefile
  head/x11-fonts/py-opentype-sanitizer/Makefile
  head/x11-toolkits/py-fltk/Makefile
Comment 24 Dima Panov freebsd_committer 2020-12-24 16:56:00 UTC
Landed!
Comment 25 Kubilay Kocak freebsd_committer freebsd_triage 2020-12-25 02:37:04 UTC
<3
Comment 26 commit-hook freebsd_committer 2020-12-31 12:05:52 UTC
A commit references this bug:

Author: kai
Date: Thu Dec 31 12:04:53 UTC 2020
New revision: 559723
URL: https://svnweb.freebsd.org/changeset/ports/559723

Log:
  math/py-matplotlib: Fix build with Python 3.8.7

  * Since r558913 the port fails to package.  Adjust pkg-plist with new
    PYTHON_EXT_SUFFIX variable introduced to python.mk in r559071 to remedy
    the issue.

  PR:		252057 (related)
  Submitted by:	portmgr (blanket: build fix)

Changes:
  head/math/py-matplotlib/pkg-plist
Comment 27 commit-hook freebsd_committer 2021-01-24 23:23:28 UTC
A commit references this bug:

Author: fluffy
Date: Sun Jan 24 23:22:54 UTC 2021
New revision: 562517
URL: https://svnweb.freebsd.org/changeset/ports/562517

Log:
  astro/astrometry: fix packaging with py38 (PYTHON_EXT_SUFFIX)

  PR:		252057 (related)
  Approved by:	portmgr (blanket: build fix)
  MFH:		2021Q1

Changes:
  head/astro/astrometry/pkg-plist
Comment 28 commit-hook freebsd_committer 2021-01-24 23:24:30 UTC
A commit references this bug:

Author: fluffy
Date: Sun Jan 24 23:24:07 UTC 2021
New revision: 562518
URL: https://svnweb.freebsd.org/changeset/ports/562518

Log:
  MFH: r562517

  astro/astrometry: fix packaging with py38 (PYTHON_EXT_SUFFIX)

  PR:		252057 (related)
  Approved by:	portmgr (blanket: build fix)

Changes:
_U  branches/2021Q1/
  branches/2021Q1/astro/astrometry/pkg-plist