Bug 252057
| Summary: | lang/python38: 3.8.6_1 -> 3.8.7 strip issues | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | John Kennedy <warlock> | ||||||||||||
| Component: | Individual Port(s) | Assignee: | Wen Heping <wen> | ||||||||||||
| Status: | Closed FIXED | ||||||||||||||
| Severity: | Affects Many People | CC: | ari, chris, fluffy, kai, michael.glaus, pi, python | ||||||||||||
| Priority: | --- | Keywords: | regression | ||||||||||||
| Version: | Latest | Flags: | koobs:
maintainer-feedback+
|
||||||||||||
| Hardware: | Any | ||||||||||||||
| OS: | Any | ||||||||||||||
| Attachments: |
|
||||||||||||||
|
Description
John Kennedy
2020-12-22 23:19:27 UTC
This does appear to be just a python38 package problem. If I just reverse the 3.8.7 patch on top of the revision=558955 ports tree, the other packages seem to be building. Holy cow! This is a future of all python branches. We need to rework our py build system * bpo-42604: Now all platforms use a value for the “EXT_SUFFIX” build variable derived from SOABI (for instance in freeBSD, “EXT_SUFFIX” is now “.cpython-310d.so” instead of “.so”). Previosuly only Linux, Mac and VxWorks were using a value for “EXT_SUFFIX” that included “SOABI”. (In reply to Dima Panov from comment #2) See https://bugs.python.org/issue42604 py3.8 already get it with release, py3.9 will get it with 3.9.2, may be 3.6/3.7 will get it too The most Python ports that fails at the moment with Python 3.8 have a "post-patch" target which contain a fixed path to the shared library. For example the databases/py-sqlite3 port has following line: > post-install: > ${STRIP_CMD} ${STAGEDIR}${PYTHON_LIBDIR}/lib-dynload/_sqlite3.so Changing it to > post-install: > ${STRIP_CMD} ${STAGEDIR}${PYTHON_LIBDIR}/lib-dynload/*.so remedies the issue. Now we have only two ways to implement.
Way #1 (bad): patch py38 to revert build scheme from .${SOABI}${SHLIB_SUFFIX} to .${SHLIB_SUFFIX} (old way, none of python consumers needs to ajusted)
Way #2 (Right Way™): patch all python3[679] to apply new build scheme (till it will be released oficially), patch Uses/python.mk to subst EXT_SUFFIX for plist AND fix all ports which build shared python library in portstree due to hardcoded paths and names in build/strip procedures and plists
(In reply to Kai Knoblich from comment #4) I prefer to not cut the whole library name but use named wildcard ${STRIP_CMD} ${STAGEDIR}${PYTHON_LIBDIR}/lib-dynload/_sqlite3*.so (In reply to Dima Panov from comment #5) We should use lang/python38 to first 'prototype' what might end up being back-ported (but possibly wont be) to 37,36. We can move whatever common bits are needed longer term to python.mk in a separate changeset. Right now 38 is broken and I'm +1 on the minimal necessary step to get them working without a revert At this point, Id actually be fixing what I consider to be a root cause, which is individual ports plists being coupled to the python implementation by hardcoding they're STRIP lines @Kai are there any ports failing that *aren't* due to hardcoded <filename.so> lines? (In reply to Kubilay Kocak from comment #7) Yep, revert is noop here because new python releases of all branchel will switch to this scheme unconditionally > @Kai are there any ports failing that *aren't* due to hardcoded <filename.so> lines? As I see in my poudriere, answer is 'no', all fails caused by strip/plist issues and if for pure py-ports with autoplist we simply can fix by use named wildcard, ports like gobject-introspection or py-grpcio needs to use some hacks depends on py version. And it is an argument for my opinion, posted earliesr -- all python3* ports should import the new naming scheme and all consumers will be adjusted without any extra .ifdefs Folks, I propose this solution: py_version depended variable PYTHON_EXT_SUFFIX which can be used in reinplace formulas and plist substitution. https://people.freebsd.org/~fluffy/-patches/python_ext.diff It allows to keep unpatched python3[679] and handle shlib suffixes at consumers fields. Tested, works for me with all py flavors We need to correct many consumers ports but this tast will be anyway faced with future python releases so we can handle it before increment default python version :) Created attachment 220839 [details] DRAFT-PR252057-fix-hardcoded-paths-for-python38.patch Attached is a WIP patch that deals with the hardcoded path of most ports. It keeps the library name as suggested in comment #6. I left out the devel/py-grpcio port and all the py-qt5-* ports as they don't seem to be affected. Please note: Some ports could still be missing, because I've only looked for py-* ports make use of ${STRIP_CMD} and the whole patch still needs QA via bulk-runs. Together with fluffy@'s patch the whole Python 3.8 issue could probably be straightened out relatively quickly. (In reply to Dima Panov from comment #9) I would like use this patch to resolv the problem. wen (In reply to Kai Knoblich from comment #10) Please cut out emulators/ucon64 update from this patch I've catches some other ports which affected library names too. They easy be fixed after python.mk patch is landed. I think both changes should land (separately). 1) the hardcoded/coupled plist entries need to be fixed (kais patch, which resolves the present regression) 2) ports not using autoplist need a variable (which changes across versions, dima's patch) Comment on attachment 220839 [details]
DRAFT-PR252057-fix-hardcoded-paths-for-python38.patch
Approved by: koobs (python)
Minus emulators/ucon64 PORTVERSION update chunk
@Dima can you attach your patch here too please
Created attachment 220841 [details]
Mk/Uses/python.mk: Handle python dynamic library naming
patch aslo devel/gobject-introspection and devel/py-grpcio as examples to apply new library naming scheme
Created attachment 220843 [details] DRAFT-PR252057-fix-hardcoded-paths-for-python38-v2.patch (In reply to Dima Panov from comment #12) Oops, good catch! That one crept in and is removed from the updated patch. Setting maintainer-approval flag accordingly as given by koobs@ in comment #14. I'll soon do the appropriate QA steps and if everything is ok, commit the patch for the hardcoded paths accordingly. Created attachment 220844 [details]
Mk/Uses/python.mk: Handle python dynamic library naming v2
Sligtly modified version -- exclude '.so' from variable to allow to use it with not only library names, pkgconfig too, for example.
With py!= 38 variable will be empty
Catched on devel/talloc
Created attachment 220888 [details]
Fix-hardcoded-paths-for-python38-v2.patch
Here's a cleaned-up patch that leaves out a couple of ports as listed below:
These ports use pkg-plists and should therefore be fixed once the patch for python.mk is in:
- archivers/py-lz4
- converters/py-bsdconv
- devel/py-jep
- graphics/py-cairo
- net/py-libdnet
- science/py-pymol
- security/py-gpgme
Following ports were also omitted because they are only for Python 2.7 and will be removed soon:
- databases/py-MySqlDb (Python 2.7 only, legacy)
- devel/py-subprocess32 (Python 2.7 only, legacy)
(In reply to Kai Knoblich from comment #18) Sure, I have plist patches prepared for those ports Waiting for approval to patch Mk/Uses/python.mk to start committing fixes :) (In reply to Dima Panov from comment #19) Please go ahead ! wen A commit references this bug: Author: fluffy Date: Thu Dec 24 13:14:02 UTC 2020 New revision: 559071 URL: https://svnweb.freebsd.org/changeset/ports/559071 Log: Mk/Uses/python.mk: introduce PYTHON_EXT_SUFFIX variable Since Python-3.8.7 it uses unified with other platforns shared libs naming scheme: instead of plain .so suffix now it refers to used cpython: .cpython-${PYTHON_SUFFIX}.so Future Python releases will use this scheme too, it will be hooked via version check PR: 252057 Approved by: python (wen) Changes: head/Mk/Uses/python.mk (In reply to Kai Knoblich from comment #18) Mk variable landed, ports with plist changes are touched @Kai, commit your part asap :) A commit references this bug: Author: kai Date: Thu Dec 24 13:46:23 UTC 2020 New revision: 559094 URL: https://svnweb.freebsd.org/changeset/ports/559094 Log: Relax hardcoded paths to fix build with Python 3.8.7 Since r558913 Python 3.8 incorporates BPO-42604 [1] which changed the shared libs naming scheme. This means "EXT_SUFFIX" is now derived from SOABI and yields with Python 3.8 to ".cpython-38.so" instead of ".so". The affected ports strip the libaries in the "post-install" target via hardcoded path(s) and the build fails at the end because the new extension is not expected at this place. Remedy the issue by adding wildcards to these paths. This should also prepare the ports for future Python releases, which will use the new shared libs naming scheme. [1] https://bugs.python.org/issue42604 PR: 252057 Reported by: John Kennedy Reviewed by: fluffy, koobs Approved by: koobs (python) Changes: head/archivers/py-brotli/Makefile head/archivers/py-lzma/Makefile head/archivers/py-python-lhafile/Makefile head/archivers/py-zstandard/Makefile head/archivers/py-zstd/Makefile head/astro/py-ephem/Makefile head/audio/py-aubio/Makefile head/audio/py-pyaudio/Makefile head/audio/py-pyliblo/Makefile head/audio/py-pymad/Makefile head/cad/py-gdspy/Makefile head/comms/py-hidapi/Makefile head/converters/py-rencode/Makefile head/converters/py-zfec/Makefile head/databases/py-PyGreSQL/Makefile head/databases/py-apsw/Makefile head/databases/py-asyncpg/Makefile head/databases/py-bsddb3/Makefile head/databases/py-cassandra-driver/Makefile head/databases/py-leveldb/Makefile head/databases/py-lmdb/Makefile head/databases/py-peewee/Makefile head/databases/py-pglast/Makefile head/databases/py-psycopg2/Makefile head/databases/py-pylibmc/Makefile head/databases/py-pymssql/Makefile head/databases/py-pyodbc/Makefile head/databases/py-sqlite3/Makefile head/databases/py-sqlrelay/Makefile head/devel/py-bullet3/Makefile head/devel/py-cbor/Makefile head/devel/py-cffi/Makefile head/devel/py-coverage/Makefile head/devel/py-crcmod/Makefile head/devel/py-cymem/Makefile head/devel/py-editdistance/Makefile head/devel/py-fam/Makefile head/devel/py-fastcache/Makefile head/devel/py-fastnumbers/Makefile head/devel/py-freebsd/Makefile head/devel/py-fusefs/Makefile head/devel/py-greenlet/Makefile head/devel/py-grpcio-tools/Makefile head/devel/py-ice37/Makefile head/devel/py-immutables/Makefile head/devel/py-intbitset/Makefile head/devel/py-lazy-object-proxy/Makefile head/devel/py-libversion/Makefile head/devel/py-libvirt/Makefile head/devel/py-llfuse/Makefile head/devel/py-llvmlite/Makefile head/devel/py-msgpack/Makefile head/devel/py-murmurhash/Makefile head/devel/py-opendht/Makefile head/devel/py-pendulum/Makefile head/devel/py-posix_ipc/Makefile head/devel/py-py-ubjson/Makefile head/devel/py-pyjq/Makefile head/devel/py-pyrsistent/Makefile head/devel/py-python-Levenshtein/Makefile head/devel/py-python-pcre/Makefile head/devel/py-ruamel.yaml.clib/Makefile head/devel/py-setproctitle/Makefile head/devel/py-simplejson/Makefile head/devel/py-urwid/Makefile head/devel/py-uvloop/Makefile head/devel/py-websockets/Makefile head/devel/py-wrapt/Makefile head/devel/py-xxhash/Makefile head/devel/py-yaml/Makefile head/devel/py-yappi/Makefile head/devel/py-zope.interface/Makefile head/dns/py-pycares/Makefile head/dns/py-pydnstable/Makefile head/dns/py-pywdns/Makefile head/finance/py-ta-lib/Makefile head/ftp/py-pycurl/Makefile head/graphics/py-ming/Makefile head/graphics/py-pygraphviz/Makefile head/graphics/py-zbar-py/Makefile head/mail/py-milter/Makefile head/mail/py-notmuch2/Makefile head/math/py-PySCIPOpt/Makefile head/math/py-cdecimal/Makefile head/math/py-cryptominisat/Makefile head/math/py-cvxpy/Makefile head/math/py-cyipopt/Makefile head/math/py-fastcluster/Makefile head/math/py-gmpy2/Makefile head/math/py-graphillion/Makefile head/math/py-hdmedians/Makefile head/math/py-levmar/Makefile head/math/py-libpoly/Makefile head/math/py-numexpr/Makefile head/math/py-numpy-stl/Makefile head/math/py-pdal/Makefile head/math/py-piranha/Makefile head/math/py-pycosat/Makefile head/math/py-pyhull/Makefile head/math/py-pyipopt/Makefile head/math/py-pynleq2/Makefile head/math/py-python-fcl/Makefile head/math/py-rpy2/Makefile head/math/py-symcxx/Makefile head/math/py-triangle/Makefile head/misc/py-fast-histogram/Makefile head/misc/py-onnx/Makefile head/misc/py-python-geohash/Makefile head/multimedia/py-cec/Makefile head/net/py-confluent-kafka/Makefile head/net/py-ldap/Makefile head/net/py-ldap0/Makefile head/net/py-libfte/Makefile head/net/py-maxminddb/Makefile head/net/py-miniupnpc/Makefile head/net/py-mpi4py/Makefile head/net/py-netifaces/Makefile head/net/py-pcapy/Makefile head/net/py-pyenet/Makefile head/net/py-pypcap/Makefile head/net/py-pysendfile/Makefile head/net/py-radix/Makefile head/net-im/py-toxcore-c/Makefile head/net-p2p/py-libtorrent-rasterbar/Makefile head/news/py-sabyenc3/Makefile head/print/py-psautohint/Makefile head/print/py-pycups/Makefile head/print/py-pycups1/Makefile head/science/py-gpaw/Makefile head/science/py-netCDF4/Makefile head/science/py-scikit-sparse/Makefile head/science/py-segyio/Makefile head/science/py-spglib/Makefile head/security/py-angr/Makefile head/security/py-coincurve/Makefile head/security/py-kerberos/Makefile head/security/py-m2crypto/Makefile head/security/py-pyscard/Makefile head/security/py-pysha3/Makefile head/security/py-python-axolotl-curve25519/Makefile head/security/py-pyvex/Makefile head/security/py-scrypt/Makefile head/security/py-sequoia/Makefile head/security/py-xmlsec/Makefile head/sysutils/py-psutil/Makefile head/sysutils/py-rdiff-backup/Makefile head/sysutils/py-scandir/Makefile head/sysutils/py-uptime/Makefile head/textproc/py-jsonslicer/Makefile head/textproc/py-libxml2/Makefile head/textproc/py-markupsafe/Makefile head/textproc/py-mwparserfromhell/Makefile head/textproc/py-patiencediff/Makefile head/textproc/py-pikepdf/Makefile head/textproc/py-pyahocorasick/Makefile head/textproc/py-pystemmer/Makefile head/textproc/py-regex/Makefile head/www/py-biscuits/Makefile head/www/py-html5-parser/Makefile head/www/py-tornado/Makefile head/www/py-tornado4/Makefile head/www/py-tornado6/Makefile head/www/py-wsaccel/Makefile head/www/py-yarl/Makefile head/x11-fonts/py-opentype-sanitizer/Makefile head/x11-toolkits/py-fltk/Makefile Landed! <3 A commit references this bug: Author: kai Date: Thu Dec 31 12:04:53 UTC 2020 New revision: 559723 URL: https://svnweb.freebsd.org/changeset/ports/559723 Log: math/py-matplotlib: Fix build with Python 3.8.7 * Since r558913 the port fails to package. Adjust pkg-plist with new PYTHON_EXT_SUFFIX variable introduced to python.mk in r559071 to remedy the issue. PR: 252057 (related) Submitted by: portmgr (blanket: build fix) Changes: head/math/py-matplotlib/pkg-plist A commit references this bug: Author: fluffy Date: Sun Jan 24 23:22:54 UTC 2021 New revision: 562517 URL: https://svnweb.freebsd.org/changeset/ports/562517 Log: astro/astrometry: fix packaging with py38 (PYTHON_EXT_SUFFIX) PR: 252057 (related) Approved by: portmgr (blanket: build fix) MFH: 2021Q1 Changes: head/astro/astrometry/pkg-plist A commit references this bug: Author: fluffy Date: Sun Jan 24 23:24:07 UTC 2021 New revision: 562518 URL: https://svnweb.freebsd.org/changeset/ports/562518 Log: MFH: r562517 astro/astrometry: fix packaging with py38 (PYTHON_EXT_SUFFIX) PR: 252057 (related) Approved by: portmgr (blanket: build fix) Changes: _U branches/2021Q1/ branches/2021Q1/astro/astrometry/pkg-plist |
