Bug 251019 - [NEW PORT] lang/tauthon: Backwards-compatible fork of Python 2.7 interpreter with Python 3.x features
Summary: [NEW PORT] lang/tauthon: Backwards-compatible fork of Python 2.7 interpreter ...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords: feature, needs-qa
Depends on:
Blocks: 249337
  Show dependency treegraph
 
Reported: 2020-11-10 15:42 UTC by Olivier Certner
Modified: 2021-06-02 17:22 UTC (History)
6 users (show)

See Also:


Attachments
Patch file for new port (284.75 KB, patch)
2020-11-10 15:42 UTC, Olivier Certner
no flags Details | Diff
build QA log from a 12.1 fresh FreeBSD install (19.74 KB, text/plain)
2020-12-11 04:05 UTC, Chris Hutchinson
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Olivier Certner 2020-11-10 15:42:38 UTC
Created attachment 219523 [details]
Patch file for new port

Small description (see pkg-desc in diff):

"""
Tauthon is a backward-compatible fork of Python's 2.7.18 interpreter with new syntax, builtins, and libraries backported from Python 3.x.

Python code and C-extensions targeting Python 2.7 or below are expected to run unmodified on Tauthon and produce the same output. But with Tauthon, that code can now use some of the new features from Python 3.x.
"""

Tauthon can be used with old code bases that will take time to migrate (or won't be migrated at all). My use case is the build infrastructure for New Moon, which I'll submit next as a new port.

This port is modeled after lang/python27, with irrelevant bits (AFAIK) removed, and specific ones added. As lang/python*, core modules with big dependencies are omitted, and should be re-enabled in other ports (I did not do it since I don't have any immediate use for them).

No changes were made to Uses/python.mk, so currently no existing py-* port can use this version automatically. bapt@ is against such a change, but it might be useful for folks that want to install some packages under Tauthon (I currently don't have any use for this), although I guess that 'pip' could be used instead.

I intend to maintain it, but I'd be happy to defer to python@ (and participate there from time to time) if you prefer.
Comment 1 Chris Hutchinson 2020-11-12 23:18:47 UTC
(In reply to Olivier Certner from comment #0)
A great contribution, Olivier. Thank you!
I hope it is imported into the ports tree quickly. :-)

--Chris
Comment 2 Daniel Engberg 2020-11-14 01:06:07 UTC
Not sure if this is a good idea since Python 2.7 is going to get axed soon as it's EOL upstream. Looking at commit rate this is seems likely to face the same fate (abandonware). 
https://github.com/naftaliharris/tauthon/commits/master
Comment 3 Olivier Certner 2020-11-15 18:27:55 UTC
(In reply to daniel.engberg.lists from comment #2)

The commit rate for a project like Tauthon, whose aim is to continue maintaining Python 2.7 (bugs, security, and the occasional backport of 3.x features), which itself has been in maintenance mode in the past years, is hardly a relevant metric. Clearly, it will remain low compared to ones in expansion. And it actually *should*, without a change in the project goal.

If you dig on GitHub a bit more, you'll see that there was a change in maintainership about 2 years ago, which went smoothly, and several guys very much interested in making sure it continues (e.g., one runs his startup infrastructure on it).

I'm pretty sure there are lots of ports in the tree for projects who haven't seen many commits in *years*. This is hardly a problem as long as they still build and are used/maintained by some.
Comment 4 Matthias Andree freebsd_committer 2020-12-02 16:53:54 UTC
It may save those ports that upstream maintainers have put on life support (i. e. only security/critical fixes) but where the upstream maintenance moved elsewhere.

Marking it a blocker on #249337.
Comment 5 Chris Hutchinson 2020-12-11 04:05:34 UTC
Created attachment 220447 [details]
build QA log from a 12.1 fresh FreeBSD install

Hello,
I have a need for this for one of my ports that
while perfectly functional as intended. Was depreciated
due to use of py27. So I applied the patch attached to
this pr on a fresh 12.1 amd64 install, with a svn up of
ports/head to test this. Everything built/tested/worked
as intended (see log attached). I would very much
appreciate this port be committed.

Thank you for all your time, and consideration.

--Chris
Comment 6 commit-hook freebsd_committer 2020-12-11 08:45:43 UTC
A commit references this bug:

Author: mandree
Date: Fri Dec 11 08:44:48 UTC 2020
New revision: 557705
URL: https://svnweb.freebsd.org/changeset/ports/557705

Log:
  [NEW] lang/tauthon: Backwards-compatible fork of Python 2.7 interpreter with Python 3.x features

  ------------------------------------------------------------------------------
  Note this is not fully connected yet, and needs further changes to integrate
  with flavors, addition to Mk/Uses/python.mk (or some equivalent USES=tauthon)
  that may need discussion first, for instance, on the package name prefix.

  This is committed to serve as a common test bed and for experimentation.
  ------------------------------------------------------------------------------

  Maintainer:	olce.freebsd.ports@certner.fr (Olivier Certner)

  Tauthon is a backward-compatible fork of Python's 2.7.18 interpreter with new
  syntax, builtins, and libraries backported from Python 3.x.

  Python code and C-extensions targeting Python 2.7 or below are expected to run
  unmodified on Tauthon and produce the same output. But with Tauthon, that code
  can now use some of the new features from Python 3.x.

  This is not an official Python release. Because of its new features, while
  preserving 2.7 compatibility, Tauthon reports 2.8 as its major/minor versions.

  Non-exhaustive list of new features:
  - Function Annotations
  - Keyword-Only Arguments
  - "async" and "await" Syntax
  - Argument-less "super"
  - New Metaclass Syntax
  - "nonlocal"
  - "yield from" Syntax
  - "concurrent.futures" Module
  - UTF-8 as the default source encoding

  WWW: https://github.com/naftaliharris/tauthon

  (Committed with minor changes over submission, particularly a few reorderings
  in Makefile to appease portlint -CA.)

  PR:		251019
  Submitted by:	Olivier Certner (maintainer)

Changes:
  head/lang/Makefile
  head/lang/tauthon/
  head/lang/tauthon/Makefile
  head/lang/tauthon/distinfo
  head/lang/tauthon/files/
  head/lang/tauthon/files/patch-Doc__library__fcntl.rst
  head/lang/tauthon/files/patch-Lib_distutils_command_build__scripts.py
  head/lang/tauthon/files/patch-Lib_distutils_command_install__lib.py
  head/lang/tauthon/files/patch-Lib_distutils_tests_test__install__lib.py
  head/lang/tauthon/files/patch-Makefile.pre.in
  head/lang/tauthon/files/patch-Misc_python-config.in
  head/lang/tauthon/files/patch-Modules__fcntlmodule.c
  head/lang/tauthon/files/patch-Modules_posixmodule.c
  head/lang/tauthon/files/patch-pr192365
  head/lang/tauthon/files/patch-setup.py
  head/lang/tauthon/files/patch-setup_metadata.py
  head/lang/tauthon/pkg-descr
  head/lang/tauthon/pkg-message
  head/lang/tauthon/pkg-plist
Comment 7 Matthias Andree freebsd_committer 2020-12-11 08:46:51 UTC
I have committed this in r557705, but we need proposals for:

- what's a sensible package name prefix (py28- or ta28- or tt28-
  or something else?)

- can we integrate it with Mk/Uses/python.mk, or should we use a separate
  USES=tauthon and Mk/Uses/tauthon.mk on the assumption that this is essentially
  a dead end for ports put on life support (i. e. no Python 3.x migration) by
  their respective upstream maintainers.
Comment 8 Olivier Certner 2020-12-11 13:15:55 UTC
The port alone is useful, so thanks for importing it.

As for your questions:

- For the package prefix, I don't have strong opinions. "ta28-" seems a priori appealing because:
1. It makes it clearer from the package name that the package does not run on standard Python.
2. This would be closer to what Pip should do, if it supports Tauthon specifically at some point. For now, the Pip tag on Tauthon is "tauthon28".
Of course that would mean that there's no more a clear connection to Python (the language) in the package name. So it depends on what we find mostly relevant.

- I did not start trying to adapt Uses/python.mk at first, because I didn't need it. I've glanced several times at it and it seems that a lot of the tricks in there are equally applicable to Tauthon. But indeed, it may be better to just duplicate it to Uses/tauthon.mk and adapt the copy. If we do not copy, then Uses/python.mk will either need to accept 28 or be adapted to support custom revisions (such as ta28). Plus, as time goes by, changes may become increasingly Python 3 specific, forcing to test if the version is not 28 (or ta28).

What do you think?
Comment 9 Matthias Andree freebsd_committer 2020-12-11 13:31:18 UTC
I don't have strong opinions either way but am slightly inclined to keep tauthon.mk separate from Python. 

My thinking is the majority of important packages should have switched to Python 3.x now, and for these, we don't need tauthon.for those, and we should not care.

For packages where the upstream maintainers made a clear decision not to port to Python 3.x (Mailman 2.x for instance, where 3.x is a rewrite-from-scratch with all lost features and other idiosyncrasies that entails), tauthon is an option, and there a switch to a USES=tauthon would not really hurt.

I am not familiar with Python porting to FreeBSD, so perhaps adding python@ to Cc: brings some people in that can help with making the decision on my questions above in comment #7, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251019#c7
Comment 10 Chris Hutchinson 2020-12-11 16:23:34 UTC
A big thanks for all the work that's gone into doing this.
I won't profess to being a Python guru. But IMHO currently
it seems initially providing:
USES=tauthon
Then later, should it become a popular option, and
continue development. A tua28/ta28/tt28 approach would
make more sense.

Thanks again!

--Chris
Comment 11 Olivier Certner 2020-12-12 15:56:12 UTC
(In reply to Chris Hutchinson from comment #10)

I was thinking the same, but this doesn't seem trivial (see second part below).

Unless, your port already has its own specific build and install targets that do not rely on specifics of Uses/python.mk (e.g., distutils), in which case you can already use:
BUILD_DEPENDS+=tauthon:lang/tauthon
RUN_DEPENDS+=tauthon:lang/tauthon

Which port are you talking about?


(In reply to Matthias Andree from comment #9)

If we take 'mail/mailman' as an example, it depends on 'dns/py-dnspython', which itself depends on 'security/py-pycryptodome'. Each one passes PY_FLAVOR to its dependency. 'dns/py-dnspython' uses the 'concurrent' and 'cython' python.mk's options, and both dependencies use "distutils" and "autoplist".

Which means that, to convert 'mail/mailman' to use Tauthon, we have at least the following options:
1. We indeed plug Tauthon into Uses/python.mk, adding particular flavors and the ability to specify a non-official (not named python*) ports as the interpreter.
2. We instead duplicate the dependencies so that they use Tauthon, and we copy and adapt the wheel (python.mk) to at least support the options mentioned above.

I looked into python.mk and, unless I'm missing something, 1. doesn't seem to be that difficult, but then there are several things to decide upon (such as, e.g., whether Tauthon would be used if 2.8 was the version specified or some other new variable/option should indicate that).

Option 2. seems to go against the very concept of flavors, leading to ports duplication, as well as code duplication (tauthon.mk).

So I came up with a third possibility: To conditionalize the USES=python or USES=tauthon in dependencies depending on the passed flavor (whether it starts with 'py' or, let's say, 'ta'). If this is possible, at least we would not need to duplicate ports even if we indeed duplicate python.mk into tauthon.mk.

As time goes by I expect that most Python ports will abandon compatibility with Python 2.7, because they will introduce the use of Python 3 syntax and features. Maybe these ports would still run under Tauthon with the appropriate '__future__' imports, but I'm just speculating at this point. And I do not know if the features backported into Tauthon cover most of uses cases or not (maybe they do now, but who knows how long this will stay true). So I guess that, overtime, ports that we need to run under Tauthon will have to be duplicated. Then, dodging the bullet now may be the better option, if there are not too many of them.
Comment 12 Chris Hutchinson 2020-12-14 19:45:59 UTC
(In reply to Olivier Certner from comment #11)
> I was thinking the same, but this doesn't seem trivial (see second part below).
> 
> Unless, your port already has its own specific build and install targets that do not rely on specifics of Uses/python.mk (e.g., distutils), in which case you can already use:
> BUILD_DEPENDS+=tauthon:lang/tauthon
> RUN_DEPENDS+=tauthon:lang/tauthon
> 
> Which port are you talking about?
Thank you for the reply, Oliver.
devel/tailor:
(Tailor is a tool to migrate changesets between Aegis, ArX, Bazaar, Bazaar-NG, CVS, Codeville, Darcs, Git, Mercurial, Monotone, Perforce, Subversion, and Tla repositories.)
I thought it might be helpful for many, given the impending FreeBSD
switch from SVN to GIT.
But given it's use (depends) on version 2.7. It was deleted from the
ports tree back in September. Between $work && trying to tool up for
the GIT change. I haven't found enough time to convert this to py3+ :-(
In case it interests anyone; the source is located:
https://bsdforge.com/projects/source/devel/tailor-0.9.35.tar.gz
Unless you (or anyone) comes up with a better suggestion. I'm just
going to install Tauthon and build/run depends against it, and see
what happens. Then share the results here.

Thanks again!

--Chris
Comment 13 Chris Hutchinson 2020-12-14 20:39:49 UTC
Well, not surprisingly, it failed.
Tauthon built/installed as a BUILD/RUN DEPENDS
but not having:

USES=           tauthon:2.8

and

USE_TAUTHON=     autoplist distutils

available. The port failed to build.
So. Failing those. It's going to be a bit more
complicated than a simple drop-in.

Not that it wouldn't be fairly obvious; I just
thought I'd mention it.

--Chris
Comment 14 Olivier Certner 2021-06-02 17:22:27 UTC
Port has been in tree for some months. It will never be connected to Mk/Uses/python.mk. So closing.