I've decided to give pep517 support a try by converting one of my ports (devel/py-pytest-asyncio) which no longer provides setup.py. I've run into an installation problem, as the wheel name (pytest_asyncio) appears to be different from the module/port name (which is pytest-asyncio, as mentioned in setup.cfg), while the framework blindly uses PORTNAME. This happens: ``` ===> Building for py39-pytest-asyncio-0.20.3 ... Successfully built pytest_asyncio-0.20.3-py3-none-any.whl ... ===> Generating temporary packing list ... FileNotFoundError: [Errno 2] No such file or directory: '/work/usr/ports/devel/py-pytest-asyncio/work-py39/pytest-asyncio-0.20.3/dist/pytest-asyncio-0.20.3-*.whl' ``` I'm not sure where the name mangling happens, but our framework [does](https://github.com/freebsd/freebsd-ports/blob/a61f6ab3b67be72b81ab5b0099f08fd1f95e605e/Mk/Uses/python.mk#L655) name mangling for egg names, so I assume a similar thing is required here. See attached a patch that converts py-pytest-asyncio to pep517 and a patch that fixes the framework for it. The latter is likely incomplete (may need to use wheel name in more places and may need similar handling for the version).
Created attachment 239408 [details] pytest-asyncio patch
Created attachment 239409 [details] python.mk patch
Comment on attachment 239409 [details] python.mk patch Allowing to override the wheel filename is a direct violation of the binary distribution format (wheel) specification, originally PEP-427, so that will not be supported. https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-name-convention However, per both the wheel specification and similar to PEP-503, distribution (PORTNAME) names are to be normalised from "[-_.]+" to "_" and lowercased. Good start however.
*** Bug 268892 has been marked as a duplicate of this bug. ***
I'm now working to update mail/py-spf-engine to 3.0.2 and faced same problem. On PyPI it is registered as 'spf-engine'. https://pypi.org/project/spf-engine/ But wheel name seems to use 'spf_engine'. In fact, if I check https://pypi.org/project/spf-engine/#files, file name of source distribution is 'spf-engine-3.0.2.tar.gz' but that of build distribution is 'spf_engine-3.0.2-py3-none-any.whl'.
See also bug 268892. Do we want to discuss PORTNAME definition issues here in this bug (such as the conflicting guidelines described in that bug)? If not, I will re-open 268892 as a separate (but somewhat related) bug.
(In reply to Yasuhiro Kimura from comment #5) audio/py-gtts: Update to 2.3.0, PR 268902 --- ===> Building for py39-gtts-2.3.0 Successfully built gTTS-2.3.0-py3-none-any.whl ===> Staging for py39-gtts-2.3.0 FileNotFoundError: [Errno 2] No such file or directory: '/wrkdirs/usr/ports/audio/py-gtts/work-py39/gTTS-2.3.0/dist/gtts-2.3.0-*.whl' PYPI name is: gTTS 2.3.0 gTTS-2.3.0.tar.gz gTTS-2.3.0-py3-none-any.whl --- In this case gtts vs gTTS seams to be the problem. Is there a way to patch it on port just as a workaround solution?
(In reply to Nuno Teixeira from comment #7) (...) Simple solution is to rename PORTNAME from gtts to gTTS to build ok.
Even though the wheel specification discourages it, since uppercase letters were supported in an earlier specification version, they will continue to be supported here. Thus PORTNAMEs will need to match the exact casing of upstream.
(In reply to Charlie Li from comment #9) Is this a reason to start renaming ports or should we wait until a better option comes up?
(In reply to Charlie Li from comment #9) The PORTNAME guidance on the wiki conflicts for cases like gTTS (MUST be lowercase and MUST match upstream) - see bug 268892. Should the 'lowercase' guidance just be removed on the wiki for python ports that are in PyPi (in favor of 'exact match' only)? Note the case for directory names can still be 'lower'.
(In reply to Charlie Li from comment #3) I checked PyPI for a few minutes and could easily find some projects that the wheel filename doesn't match the convention. https://pypi.org/project/odoo12-addons-oca-maintenance/#files https://pypi.org/project/windy-weather-crawler/#files https://pypi.org/project/utils-nuuuwan/#files https://pypi.org/project/mobio-media-sdk/#files https://pypi.org/project/pyagrum-nightly/#files https://pypi.org/project/mkdocs-latest-version-tag-plugin/#files https://pypi.org/project/streammachine-api-definitions/#files I guess there are a lot of projects that violate PEP 427, and probably it's impossible to make them correct violation. So I think we need to handle such cases by our side.
(In reply to Yasuhiro Kimura from comment #12) These ones are compliant, with dashes/hyphens normalised to underscores for the built distributions (wheels).
(In reply to John Hein from comment #11) Looking at the revision history, looks like an oversight when clarifying differing cases on whether the package is registered on PyPI or not. For packages with uppercase letters like gTTS, PyPI does register them as such, so PORTNAME casing has to match here, which is consistent with our policy. We do have to update our policy in light of PEP-517 and the addition/changeover of MASTER_SITES=PYPI, however. (In reply to Nuno Teixeira from comment #10) Yes, PORTNAMEs have to match casing, but the directory names can stay.
(In reply to Charlie Li from comment #13) > These ones are compliant, with dashes/hyphens normalised to underscores for the built distributions (wheels). Then it seems 'USE_PYTHON=pep517' doesn't consider such normalization. As I wrote in comment #5, wheel filename of 'spf-engine' is 'spf_engine-3.0.2-py3-none-any.whl' as a result of normalization. But at staging phase framework looks for 'spf-engine-3.0.2-*.whl' as following. yasu@rolling-vm-freebsd2[1022]% make stage ===> License APACHE20 GPLv2+ accepted by the user ===> py311-spf-engine-3.0.2 depends on file: /usr/local/sbin/pkg - found ===> Fetching all distfiles required by py311-spf-engine-3.0.2 for building ===> Extracting for py311-spf-engine-3.0.2 => SHA256 Checksum OK for spf-engine-3.0.2.tar.gz. ===> Patching for py311-spf-engine-3.0.2 ===> Applying FreeBSD patches for py311-spf-engine-3.0.2 from /usr/ports/mail/py-spf-engine/files ===> py311-spf-engine-3.0.2 depends on package: py311-flit-core>=3.8<4 - found ===> py311-spf-engine-3.0.2 depends on file: /usr/local/bin/python3.11 - found ===> py311-spf-engine-3.0.2 depends on package: py311-build>0 - found ===> py311-spf-engine-3.0.2 depends on package: py311-installer>0 - found ===> Configuring for py311-spf-engine-3.0.2 ===> Building for py311-spf-engine-3.0.2 * Getting build dependencies for wheel... * Building wheel... Successfully built spf_engine-3.0.2-py3-none-any.whl ===> Staging for py311-spf-engine-3.0.2 ===> py311-spf-engine-3.0.2 depends on package: py311-authres>0 - found ===> py311-spf-engine-3.0.2 depends on package: py311-milter>0 - found ===> py311-spf-engine-3.0.2 depends on package: py311-pyspf>0 - found ===> py311-spf-engine-3.0.2 depends on file: /usr/local/bin/python3.11 - found ===> Generating temporary packing list ===> Creating groups. ===> Creating users Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/usr/local/lib/python3.11/site-packages/installer/__main__.py", line 85, in <module> _main(sys.argv[1:], "python -m installer") File "/usr/local/lib/python3.11/site-packages/installer/__main__.py", line 73, in _main with WheelFile.open(args.wheel) as source: File "/usr/local/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/installer/sources.py", line 122, in open with zipfile.ZipFile(path) as f: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/zipfile.py", line 1281, in __init__ self.fp = io.open(file, filemode) ^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/usr0/freebsd/ports/work/usr/ports/mail/py-spf-engine/work-py311/spf-engine-3.0.2/dist/spf-engine-3.0.2-*.whl' *** Error code 1 Stop. make: stopped in /usr/ports/mail/py-spf-engine yasu@rolling-vm-freebsd2[1023]%
Created attachment 239433 [details] python.mk: normalise wheel filename arguments [apply using git-am(1)] This preserves PORTNAMEs matching exactly those in PyPI or package metadata, but correctly normalising the resultant wheel filename.
Comment on attachment 239433 [details] python.mk: normalise wheel filename arguments [0] https://packaging.python.org/en/latest/specifications/binary-distribution-format/
(In reply to Charlie Li from comment #16) I applied attachment 239433 [details] and staging failed with different error. yasu@rolling-vm-freebsd2[1034]% make stage ===> License APACHE20 GPLv2+ accepted by the user ===> py311-spf-engine-3.0.2 depends on file: /usr/local/sbin/pkg - found ===> Fetching all distfiles required by py311-spf-engine-3.0.2 for building ===> Extracting for py311-spf-engine-3.0.2 => SHA256 Checksum OK for spf-engine-3.0.2.tar.gz. ===> Patching for py311-spf-engine-3.0.2 ===> Applying FreeBSD patches for py311-spf-engine-3.0.2 from /usr/ports/mail/py-spf-engine/files ===> py311-spf-engine-3.0.2 depends on package: py311-flit-core>=3.8<4 - found ===> py311-spf-engine-3.0.2 depends on file: /usr/local/bin/python3.11 - found ===> py311-spf-engine-3.0.2 depends on package: py311-build>0 - found ===> py311-spf-engine-3.0.2 depends on package: py311-installer>0 - found ===> Configuring for py311-spf-engine-3.0.2 ===> Building for py311-spf-engine-3.0.2 * Getting build dependencies for wheel... * Building wheel... Successfully built spf_engine-3.0.2-py3-none-any.whl ===> Staging for py311-spf-engine-3.0.2 ===> py311-spf-engine-3.0.2 depends on package: py311-authres>0 - found ===> py311-spf-engine-3.0.2 depends on package: py311-milter>0 - found ===> py311-spf-engine-3.0.2 depends on package: py311-pyspf>0 - found ===> py311-spf-engine-3.0.2 depends on file: /usr/local/bin/python3.11 - found ===> Generating temporary packing list ===> Creating groups. ===> Creating users sed: /usr0/freebsd/ports/work/usr/ports/mail/py-spf-engine/work-py311/stage/usr/local/lib/python3.11/site-packages/spf-engine-3.0.2.dist-info/RECORD: No such file or directory *** Error code 1 Stop. make: stopped in /usr/ports/mail/py-spf-engine yasu@rolling-vm-freebsd2[1035]%
Created attachment 239434 [details] python.mk: normalise wheel filename arguments v2 Also normalise directory names inside wheel, particularly for RECORD processing
(In reply to Charlie Li from comment #19) With attachment 239434 [details] `make stage` finished successfully. But now `make check-plist` fails as following. ====> Checking for pkg-plist issues (check-plist) ===> Parsing plist ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: etc/python-policyd-spf/policyd-spf.conf.commented Error: Orphaned: share/man/man1/policyd-spf.1.gz Error: Orphaned: share/man/man5/policyd-spf.conf.5.gz Error: Orphaned: share/man/man5/policyd-spf.peruser.5.gz Error: Orphaned: share/man/man8/pyspf-milter.8.gz ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: %%PYTHON_LIBDIR%%/site-..etc/python-policyd-spf/policyd-spf.conf.commented Error: Missing: %%PYTHON_LIBDIR%%/site-..share/man/man1/policyd-spf.1 Error: Missing: %%PYTHON_LIBDIR%%/site-..share/man/man5/policyd-spf.conf.5 Error: Missing: %%PYTHON_LIBDIR%%/site-..share/man/man5/policyd-spf.peruser.5 Error: Missing: %%PYTHON_LIBDIR%%/site-..share/man/man8/pyspf-milter.8 ===> Error: Plist issues found. *** Error code 1 Stop. make: stopped in /usr/ports/mail/py-spf-engine So please let me confirm one point. Should 'USE_PYTHON=autoplist pep517' work fine?
(In reply to Yasuhiro Kimura from comment #20) 'USE_PYTHON=autoplist pep517' works as intended, but currently does not handle data_files installation as setuptools deprecated this practice [0]. spf-engine uses flit, which still supports this [1]. We'll have to workshop this separately, since this bug is about normalising wheel filenames. [0] https://github.com/pypa/setuptools/discussions/2648 [1] https://flit.pypa.io/en/latest/pyproject_toml.html#external-data-section
(In reply to Charlie Li from comment #21) Thanks for explanation. Then what is the best practice at the moment? Should I stop using 'USE_PYTHON=autoplist' and add pkg-plist?
(In reply to Charlie Li from comment #19) Your patch v2 seems to work as expected for me, at least for ports like devel/py-pyls-black and textproc/py-whatthepatch :)
(In reply to Yasuhiro Kimura from comment #22) Anyway, autoplist is never a good solution: it is not a big work to maintain a fixed pkg-plist, and a fixed plist is often useful.
(In reply to Thierry Thomas from comment #24) autoplist is staying and will continue to be preferred in the Python ports policy when a RECORD (or a method to derive it) exists, as it is part of the Python packaging standard. RECORD *is* the fixed plist, there is no need to reinvent the wheel (pun intended).
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=952e0dba497e276c99d0ffb78cecb77a65349832 commit 952e0dba497e276c99d0ffb78cecb77a65349832 Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2023-01-13 04:12:01 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2023-01-14 01:31:41 +0000 python.mk: normalise wheel filename arguments in PEP517_INSTALL_CMD The living binary distribution format specification derived from PEP-427 [0] prescribes that: In distribution names, any run of -_. characters (HYPHEN-MINUS, LOW LINE and FULL STOP) should be replaced with _ (LOW LINE), and uppercase characters should be replaced with corresponding lowercase ones. This is equivalent to PEP 503 normalisation followed by replacing - with _. Tools consuming wheels must be prepared to accept . (FULL STOP) and uppercase letters, however, as these were allowed by an earlier version of this specification. This fixes staging for packages built under PEP-517 with dashes (HYPHEN-MINUS) in their names. [0] https://packaging.python.org/en/latest/specifications/binary-distribution-format/ Reported by: amdmi3 Tested by: yasu, rhurlin PR: 268893 With hat: python Approved by: mentors (implicit) Mk/Uses/python.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Normalisation fix committed. Please continue the discussion about data_files handling in autoplist at review D38050.