Bug 259981

Summary: devel/py-setuptools_scm: Fix incorrect *_DEPENDS, causing devel/py-pyparsing (and others) Fails to configure: The 'packaging>=20.0' distribution was not found
Product: Ports & Packages Reporter: Ivan Rozhuk <rozhuk.im>
Component: Individual Port(s)Assignee: Kai Knoblich <kai>
Status: Closed FIXED    
Severity: Affects Many People CC: djohnson, edgeman, f.bergknecht, freebsd, kai, o.hushchenkov, ohartmann, python, rhurlin, rozhuk.im, se, sunpoet
Priority: --- Keywords: regression
Version: LatestFlags: kai: maintainer-feedback+
kai: maintainer-feedback+
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259912
Attachments:
Description Flags
py-setuptools_scm-workaround-for-py-packaging-v1.patch kai: maintainer-approval+

Description Ivan Rozhuk 2021-11-22 11:02:55 UTC
===> Fetching all distfiles required by py38-pyparsing-3.0.6 for building
===>  Extracting for py38-pyparsing-3.0.6
=> SHA256 Checksum OK for pyparsing-3.0.6.tar.gz.
===>  Patching for py38-pyparsing-3.0.6
===>   py38-pyparsing-3.0.6 depends on package: py38-setuptools>0 - found
===>   py38-pyparsing-3.0.6 depends on file: /usr/local/bin/python3.8 - found
===>   py38-pyparsing-3.0.6 depends on file: /usr/local/bin/ccache - found
===>  Configuring for py38-pyparsing-3.0.6
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 23, in <module>
    setup(  # Distribution meta-data
  File "/usr/local/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.8/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 455, in __init__
    _Distribution.__init__(self, {
  File "/usr/local/lib/python3.8/distutils/dist.py", line 292, in __init__
    self.finalize_options()
  File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 800, in finalize_options
    for ep in sorted(eps, key=by_order):
  File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 799, in <lambda>
    eps = map(lambda e: e.load(), pkg_resources.iter_entry_points(group))
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in load
    self.require(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2472, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'packaging>=20.0' distribution was not found and is required by the application
*** Error code 1
Comment 1 Matthew Kern 2021-11-22 15:23:32 UTC
Trying to install py-packaging with portmaster will fail, since py-pyparsing is listed as a runtime requirement for py-packaging (so portmaster tries to run the py-pyparsing build before the install step from py-packaging).

I was able to proceed by running 'make install clean' within /usr/ports/devel/py-packaging, then within /usr/ports/devel/py-pyparsing, then continuing with the rest of the portmaster step I was completing.
Comment 2 Ivan Rozhuk 2021-11-24 09:48:08 UTC
Probably this is not py-pyparsing error.

Same error was with other python ports until I run "pkg autoremove", it remove:
	py38-Babel: 2.9.1
	py38-Jinja2: 3.0.1
	py38-beaker: 1.11.0
	py38-cython: 0.29.24
	py38-docutils: 0.17.1,1
	py38-html5lib: 1.0.1
	py38-lxml: 4.6.4
	py38-mako: 1.0.14_1
	py38-markupsafe: 2.0.1
	py38-ply: 3.11
	py38-pyaes: 1.6.1
	py38-pyparsing: 3.0.4
	py38-pytest-runner: 5.3.1
	py38-setuptools_scm: 6.3.2
	py38-tomli: 1.2.2
	py38-webencodings: 0.5.1
	py38-wheel: 0.36.2

Somehow py38-packaging was not installed, but py38-setuptools_scm requires this.
Removing py38-setuptools_scm - fix issue for me.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2021-11-25 06:03:39 UTC
Upstream declares:

  ./work-py39/setuptools_scm-6.3.2/pyproject.toml:    "packaging>=20.0"

But devel/pysetuptools_scm declares packaging as a RUN_DEPENDS, but should be a BUILD_DEPENDS.

Workaround until resolved: install devel/py-packaging manually
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2021-11-25 06:05:11 UTC
(In reply to Kubilay Kocak from comment #3)

Clarifying, upstream declares:

[build-system]
requires=..., "packaging>=20.0",...
Comment 5 Kai Knoblich freebsd_committer 2021-11-25 17:59:16 UTC
The problem occurs in environments where devel/py-setuptools_scm is installed and devel/py-packaging is also updated together with other Python ports during an upgrade run via portmaster.

The old version of devel/py-packaging will be removed at some point and the new version will be installed too late which will then leads to the issue described in comment #2.

This is because devel/py-packaging is required for run time [1] by devel/py-setuptools_scm and the latter one is very interlocked with devel/py-setuptools.  It's executed together almost every time a Python port is built and will fail if a dependency is missing during run time.

The issue can be reproduced via:

- Install/Build devel/py-pyparsing 3.0.4
- Install/Build devel/py-packaging 21.2_1
- Install/Build devel/py-setuptools_scm
- Apply patches to update devel/py-packaging to 21.3 and devel/py-pyparsing to 3.0.6
- Try to upgrade the ports with portmaster

The whole problem will most likely recur the next time devel/py-packaging is updated.  Probably a long-term solution would be if portmaster treats ports like devel/py-setuptools* and their dependencies with higher priority over other Ports during an upgrade run.

(In reply to Kubilay Kocak from comment #3)

I'll add devel/py-packaging to BUILD_DEPENDS as it is in "pyproject.toml" and you have already mentioned it although it's not the solution to the problem reported with devel/py-pyparsing as described above.

--
[1] https://github.com/pypa/setuptools_scm/blob/v6.3.2/setup.cfg#L30
Comment 6 Stefan Eßer freebsd_committer 2021-12-04 22:05:11 UTC
I can confirm that py-setuptools_scm installed prevents building of other Python ports, if py-packaging or py-pyparsing are not available.

But I do not see any issue when updating these ports with portmaster.

I have tested all permutations of port names in the following command:

# portmaster devel/py-pyparsing devel/py-packaging devel/py-setuptools_scm

And in all cases the re-installation succeeded (as would have a port upgrade):

===>>> The following actions were performed:
	Re-installation of py38-pyparsing-3.0.6
	Re-installation of py38-packaging-21.3
	Re-installation of py38-setuptools_scm-6.3.2

The dependency checks performed by portmaster are somewhat different from those performed by a plain make, since portmaster plans all updates before starting the first one.

But the updates are then performed in the same way as by a plain "make all deinstall install" in the port directory.

Portmaster considers RUN_DEPENDS to actually mean that, i.e. a dependency that has to be provided to run the program built by the port. This is different from a plain "make", since that performs builds and installations of RUN_TIME dependencies before starting the "stage" phase.

But this difference is irrelevant for this PR.


A solution to this issue might be to treat a missing py-pyparsing or py-packaging in the same way as if py-setuptools_scm was not installed.

The huge majority of Python ports does not depend on py-setuptools_scm - specifically py-packaging and py-pyparsing do not depend on it. (Of more than 7000 devel/py-* ports less than 70 directly depend on py-setuptools_scm.)

But if py-setuptools_scm *is* installed, a missing py-pyparsing oder py-packages does suddenly become a configure dependency, since py-setuptools_scm is (unnecessarily) invoked, but fails due to the missing run-time dependency.

Thus, a missing py-pyparsing or py-packaging should not cause py-setuptools_scm to fail in such a way, that setup.py is stopped. For ports that do not directly depend on py-setuptools_scm, such a failure should be ignored.
Comment 7 Stefan Eßer freebsd_committer 2021-12-04 22:33:46 UTC
*** Bug 259993 has been marked as a duplicate of this bug. ***
Comment 8 Kai Knoblich freebsd_committer 2021-12-06 11:05:48 UTC
Created attachment 229940 [details]
py-setuptools_scm-workaround-for-py-packaging-v1.patch

Hm, devel/py-setuptools_scm has a fallback (via "try ... except") in [1] when devel/py-packaging is absent.  It's also the only location where devel/py-packaging is (optionally) required at the moment.

However, this code is not executed because devel/py-setuptools, which checks the dependencies upon invocation and terminates as soon as these are not met, which leads then to the errors in comment #0.

Attached is a workaround which should remedy the issue by patching out devel/py-packaging from "setup.cfg" but keep it in RUN_DEPENDS.

It also adds devel/py-packaging to BUILD_DEPENDS as defined in pyproject.toml although it's not strictly required to build the port.

I'll leave this patch here for a few days for general testing and feedback.

QA:
~~~
- poudriere -> OK (12.2-RELEASE amd64 against all consumers)
- portmaster -> OK (using the scenario outlined in comment #5)
- "make test" -> OK

--
[1] https://github.com/pypa/setuptools_scm/blob/v6.3.2/src/setuptools_scm/_version_cls.py
Comment 9 Koichiro Iwao freebsd_committer 2021-12-09 14:55:29 UTC
*** Bug 259993 has been marked as a duplicate of this bug. ***
Comment 10 Stefan Eßer freebsd_committer 2021-12-09 16:39:25 UTC
(In reply to Kai Knoblich from comment #8)

I have tested the patch with multiple pre-conditions and build tools, and it fixes all issues caused by either or both of devel/py-pyparsing and devel/py-packaging missing when devel/py-setuptools_scm is installed.

Please commit this patch to fix the issues reported in this PR and in PR 259993.
Comment 11 commit-hook freebsd_committer 2021-12-09 18:58:37 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=ad7f115cddd949d6ffd41bd9889328c06f8906e4

commit ad7f115cddd949d6ffd41bd9889328c06f8906e4
Author:     Kai Knoblich <kai@FreeBSD.org>
AuthorDate: 2021-12-09 18:49:49 +0000
Commit:     Kai Knoblich <kai@FreeBSD.org>
CommitDate: 2021-12-09 18:49:49 +0000

    devel/py-setuptools_scm: Turn py-packaging into a soft dependency

    * Since the 6.3.0 release, devel/py-packaging has been defined in the
      "setup.cfg" as an unconditional dependency.  However this leads to
      problems if py-packaging is uninstalled but py-setuptools_scm
      persists, then the missing port can no longer be built. [1]

      Although py-setuptools_scm has a fallback to devel/py-setuptools when
      py-packaging is absent this fallback code is not executed.  This is
      because py-setuptools, which checks the dependencies of
      py-setuptools_scm upon invocation and terminates as soon as these are
      not met.

      Fix this issue by converting py-packaging into an optional dependency
      for py-setuptools_scm so that the fallback code is really used if
      py-packaging is absent.  But keep the latter one in the RUN_DEPENDS in
      order to continue to reflect the actual dependencies.

    * Also add py-packaging to BUILD_DEPENDS as defined in "pyproject.toml"
      although it's not strictly required to build the port. [2]

    * Bump PORTREVISION due package change.

    PR:             259981
    Reported by:    many [1], koobs [2]
    Tested by:      se, myself

 devel/py-setuptools_scm/Makefile                    |  4 +++-
 devel/py-setuptools_scm/files/patch-setup.cfg (new) | 19 +++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)
Comment 12 Kai Knoblich freebsd_committer 2021-12-09 19:18:21 UTC
I'll leave this bug open as reference for a few more days.
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-09 22:40:31 UTC
Beautiful. Python dynamic, conditional, at run-time or try/except dependency checks and code path changes are notorious for creating these situations, which are of the class 'non-deterministic dependency sets' and resolving them at the root cause upstream is absolutely the correct resolution.
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-26 01:13:27 UTC
Good to close Kai, or remaining things to resolve?
Comment 15 Kai Knoblich freebsd_committer 2021-12-27 21:00:13 UTC
(In reply to Kubilay Kocak from comment #14)

Thanks for the info, Kubilay. It's good to close so I'm closing this now.