Ports of python modules which come from python itself (databases/py-sqlite3, databases/py-gdbm, x11-toolkits/py-tkinter) use custom setup.py files. These use distutils which are deprecated and removed in upcoming python 3.12. Switch these to setuptools to allow building with python3. Buildtested with all supported python versions.
setup.py itself is also deprecated. If we are to continue having these separated out from the base Python distribution, please use pyproject.toml instead.
pyproject.toml is not yet supported by our Uses/python.mk (I've had to add `setup.py` with `from setuptools import setup; setup()` for a couple of ports which have really dropped it) so migrating off it is a large task and out of scope of this pr. I'm also not quite sure how we can switch to pyproject.toml for at least py-sqlite, as setup.py there contains some logic dependent on python version.
I'm going to commit that support within the week, pending "test suite" results (see https://wiki.freebsd.org/Python/PEP-517 and bug 255722). This is very much in scope here because the former method is no longer compliant with modern Python packaging standards.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=7446740638e7d833fde0962e71ef440a6af3c749 commit 7446740638e7d833fde0962e71ef440a6af3c749 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-12-15 15:07:26 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-12-15 15:14:33 +0000 */py-{gdbm,sqlite3,tkinter}: switch from distutils to setuptools For python modules which come from python itself, switch from distutils (no longer present in python 3.12) to setuptools, fixing these mudules for python 3.12. PR: 268283 Approved by: vishwin (python@) databases/py-gdbm/files/setup3.py | 12 +++++++----- databases/py-sqlite3/files/setup3.py | 13 +++++++------ x11-toolkits/py-tkinter/files/setup3.py | 12 ++++++------ 3 files changed, 20 insertions(+), 17 deletions(-)
This was not approved, please revert. maintainer-feedback was to prevent the timeout only.
> This was not approved It was. > maintainer-feedback was to prevent the timeout only. Sorry, there's no way for me to interpret explicit approval as non-approval. > please revert. On which basis? This fixes setup.py for python 3.12 and is required for its addition. It does not change change the behavior for other versions. As mentioned, it does not intersect with your work - you're free to replace setup.py support with whatever regardless of whether setup.py's for these 3 ports use distutils or setuptools.
'Set (+) when you provide feedback. Avoids "maintainer timeout"' is all maintainer-feedback means here. The feedback in previous comments was not approval to commit. Python 3.12 has not even reached release candidate yet. The base distribution will continue to successfully build without these, as we have experienced for so many versions now. For the separated components alluded to here, things are going to break or hold other items up, and they will get worked on, but do it properly. setup.py is deprecated along with distutils getting removed, so continuing to use the former is not acceptable, even if it does not error out.
> The feedback in previous comments was not approval to commit Oh, my bad then. Bugzilla's, to be precise. > Python 3.12 has not even reached release candidate yet This is not relevant to adding it to the tree, the same way how previous prereleases were added (and helped a lot to prepare the tree for the new python version). > setup.py is deprecated along with distutils getting removed, so continuing to use the former is not acceptable, even if it does not error out. This is nonsence, as "continuing to use" setup.py's for these ports is the only thing which keeps them working. As long as setup.py's are in place, these should be maintained and kept functioning, that's what I do. There's no alternative to them. Please provide a valid reason for it to be rolled back, such as it breaks something or it blocks someone's work. For now it looks like you're trying to block both my and Wen's, who did similar python3.12 port in parallel.
setup.py has been deprecated since setuptools 58.3.0 (https://github.com/pypa/setuptools/commit/fc5c3083908170e1cbfd1dec91f87a3e2707e3fa), so if anyone is that anal about using setup.py (on Python 3), go ahead and add a devel/py-setuptools58 to BUILD_DEPENDS. distutils will not get retroactively removed in future point releases of existing Python major releases, so that method continues to work there. The alternative for this case is to construct pyproject.toml, which, surprise, also works for every supported Python 3 production release! Python 3.12 not even reaching release candidate or in our tree is very relevant, not least *precisely* due to distutils removal amongst other items. distutils removal from the base distribution is not merely a technical change, but a structural change on how our corner of the Python packaging ecosystem moves forward, and it starts here. The valid reason to revert is the lack of explicit approval, periodt.
(In reply to Charlie Li from comment #9) > setup.py has been deprecated since setuptools 58.3.0 (https://github.com/pypa/setuptools/commit/fc5c3083908170e1cbfd1dec91f87a3e2707e3fa) ... distutils will not get retroactively removed in future point releases of existing Python major releases, so that method continues to work there Well this is in fact the valid reason for revert. The only one mentioned in the tread so far.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=62723bbe60705c70cf2dceb23a22a73bdd4d4ae3 commit 62723bbe60705c70cf2dceb23a22a73bdd4d4ae3 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-12-15 17:29:27 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-12-15 17:31:40 +0000 Revert "*/py-{gdbm,sqlite3,tkinter}: switch from distutils to setuptools" This reverts commit 7446740638e7d833fde0962e71ef440a6af3c749. The change is not valid as setuptools have deprecated setup.py, while distutils will be supported in existing point releases. Python 3.12 support will require different solution. PR: 268283 databases/py-gdbm/files/setup3.py | 12 +++++------- databases/py-sqlite3/files/setup3.py | 13 ++++++------- x11-toolkits/py-tkinter/files/setup3.py | 12 ++++++------ 3 files changed, 17 insertions(+), 20 deletions(-)
I think this is the first unblocked issue to get to Python 3.12. What would we need to do to resolve this?
Created attachment 249898 [details] Proposed PEP517 switch (apply via git am) Hi. I've created pyproject.toml for sqlite module that works for python3 (python27 is left on distutils). I'm not using py-installer because it cannot install anything outside site-packages dir. poudriere checks passed for py39, py311, and py27. BTW, homeassistant dropped 3.11 support and now requires 3.12.
(In reply to Oleg Sidorkin from comment #13) thanks for port, do you mind to have a port for HA?
(In reply to vova from comment #14) It's a bit offtopic here, but long story short: HA has ton of dependencies, and often exact versions of dependencies are required. And its plugins want another ton of dependencies. HA tries to install them via pip install. E.g. HA wants numpy==1.26.0 and cannot build it correctly on FreeBSD via pip. I have to patch its requirements to >=1.26.0 for now for it to accept the version from ports. So it is not really ports-friendly. In my opinion it's better to install HA into venv inside a jail, provide some small number of packages that HA cannot install correctly via pip install and let HA to do the rest.
Yep, do exactly the same, install HA into jail with minum ports and all other python dependancies installed by HA itself with PIP. so before it was required 1. find out what dependancies to be pre-installed (libraries and py- portrs like py-sqlite3) 2. arrange all port installation magic by hands (rc.d script, proper home dir, etc) and now one more problem appeared - 3. build your own python312 so, port which will not require much in system sounds reasonable compromise for me ... to simplify jail creation and HA upgrade procedure ...
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=dfb9729196516470a10113c454ada9ba09ecbe96 commit dfb9729196516470a10113c454ada9ba09ecbe96 Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2025-03-08 06:24:43 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2025-03-08 06:24:43 +0000 databases/py-sqlite3: remove Python 2 support No remaining consumers in the tree. This is to prepare for conversion to USE_PYTHON=pep517, as none of such tooling works with Python 2. PR: 268283 databases/py-sqlite3/Makefile | 8 ++--- databases/py-sqlite3/files/setup.py | 16 ++++++--- databases/py-sqlite3/files/setup3.py (gone) | 56 ----------------------------- 3 files changed, 14 insertions(+), 66 deletions(-)
Created attachment 258428 [details] Proposed PEP517 switch (apply via git am) v.2 Updated patch
Comment on attachment 258428 [details] Proposed PEP517 switch (apply via git am) v.2 No need for unzip or cython3 (this does not even remotely use either). do-install also duplicates the default intended behaviour for USE_PYTHON=pep517. I have a much simpler version of this that has been dogfooded for quite a while now.
Currently x11-toolkits/py-tkinter@py27 has one remaining consumer, games/renpy. Their new branch, 8, supports Python 3. Two ways of going about this, whichever is easier: build the tkinter extension as originally intended as part of lang/python27, or update games/renpy. So far not too much luck with the former (actually haven't been able to package lang/python27 in a while for unrelated reasons), and there is an open PR with patch for the latter.
(In reply to Charlie Li from comment #20) Have figured out how to re-include _tkinter into the lang/python27 build to keep things moving here. Updating games/renpy is much more involved than what is apparent.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=7e633595cf9df9a557a7d3b421b73e74ab0c057f commit 7e633595cf9df9a557a7d3b421b73e74ab0c057f Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2025-03-10 00:02:09 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2025-03-10 19:28:13 +0000 lang/python27: re-add _tkinter into the distribution x11-toolkits/py-tkinter is otherwise built separately, ie disabled in the main Python distribution build, due to circular (build) dependencies with Python itself in lang/tk*. Since this Python version is never default anymore, having been EOLed for years among other things, it is safe to re-add _tkinter as part of the main distribution build. Also update pkg-message to reflect extension module ports unavailability. This is to benefit the only remaining consumer, games/renpy, as x11-toolkits/py-tkinter transitions to Python 3+ only. The relevant OPTION has been removed as having _tkinter always present makes the option a no-op. PR: 268283 games/renpy/Makefile | 8 +++----- lang/python27/Makefile | 4 ++-- lang/python27/files/patch-setup.py | 18 +++++++++--------- lang/python27/pkg-message | 9 ++------- lang/python27/pkg-plist | 1 + 5 files changed, 17 insertions(+), 23 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=62b50f7bd59a6da561894816e2ec0b3352b9208f commit 62b50f7bd59a6da561894816e2ec0b3352b9208f Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2025-03-10 19:23:54 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2025-03-10 19:28:14 +0000 x11-toolkits/py-tkinter: remove Python 2 support No remaining consumers in the tree after re-including _tkinter into lang/python27 itself. This is to prepare for conversion to USE_PYTHON=pep517, as none of such tooling works with Python 2. PR: 268283 x11-toolkits/py-tkinter/Makefile | 7 ++--- x11-toolkits/py-tkinter/files/setup.py | 9 +++--- x11-toolkits/py-tkinter/files/setup3.py (gone) | 40 -------------------------- 3 files changed, 6 insertions(+), 50 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=b4529dc74457f6c0db0692aee8976b89a52aae5d commit b4529dc74457f6c0db0692aee8976b89a52aae5d Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2025-03-12 02:34:18 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2025-03-12 02:35:51 +0000 x11-toolkits/py-tkinter: convert to USE_PYTHON=pep517 Note USES=tk limited to 8.6; 9.0 support is not expected until Python 3.14 PR: 268283 x11-toolkits/py-tkinter/Makefile | 36 ++++++++++++------- .../py-tkinter/files/pyproject.toml.in (new) | 11 ++++++ x11-toolkits/py-tkinter/files/setup.py (gone) | 40 ---------------------- x11-toolkits/py-tkinter/files/setup.py.in (new) | 24 +++++++++++++ 4 files changed, 59 insertions(+), 52 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=7a5f12ca6a82f80b6846aa727807cf697d1f5345 commit 7a5f12ca6a82f80b6846aa727807cf697d1f5345 Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2025-03-12 02:24:07 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2025-03-12 02:31:08 +0000 databases/py-gdbm: convert to USE_PYTHON=pep517 PR: 268283 databases/py-gdbm/Makefile | 27 ++++++++++++++++++------- databases/py-gdbm/files/pyproject.toml.in (new) | 11 ++++++++++ databases/py-gdbm/files/setup.py | 21 ++++--------------- 3 files changed, 35 insertions(+), 24 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=b7e75ada7e388c089d7655d78077f81d75cdb070 commit b7e75ada7e388c089d7655d78077f81d75cdb070 Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2025-03-12 02:32:06 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2025-03-12 02:32:06 +0000 databases/py-sqlite3: convert to USE_PYTHON=pep517 PR: 268283 databases/py-sqlite3/Makefile | 26 +++++++++++++--------- databases/py-sqlite3/files/pyproject.toml.in (new) | 11 +++++++++ databases/py-sqlite3/files/setup.py | 22 ++++-------------- 3 files changed, 30 insertions(+), 29 deletions(-)