Bug 242463 - devel/mercurial: Update to 5.2.2
Summary: devel/mercurial: Update to 5.2.2
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-python mailing list
URL:
Keywords: buildisok, needs-qa
Depends on:
Blocks:
 
Reported: 2019-12-05 19:36 UTC by Olivier Duchateau
Modified: 2020-01-19 14:43 UTC (History)
5 users (show)

See Also:
bugzilla: maintainer-feedback? (python)


Attachments
Patch to update devel/mercurial (4.00 KB, patch)
2019-12-05 19:36 UTC, Olivier Duchateau
no flags Details | Diff
[patch] update mercurial to 5.2.1 including FREEBSD option (authormapsuffix) (5.32 KB, patch)
2019-12-27 20:22 UTC, John Hein
no flags Details | Diff
Python 3 support (66.60 KB, patch)
2019-12-30 12:24 UTC, Peter Wullinger
no flags Details | Diff
Python 3 support + FreeBSD authormapsuffix map + plist fixes (69.84 KB, patch)
2019-12-31 17:00 UTC, Peter Wullinger
no flags Details | Diff
Python 3 support + FreeBSD authormapsuffix map + plist fixes (70.16 KB, patch)
2020-01-02 16:50 UTC, Peter Wullinger
no flags Details | Diff
Python 3 support + FreeBSD authormapsuffix map + plist fixes (70.96 KB, patch)
2020-01-02 17:09 UTC, Peter Wullinger
no flags Details | Diff
v5.2.2, Python 3 support, FreeBSD authormapsuffix map, plist fixes (71.21 KB, patch)
2020-01-07 17:23 UTC, Peter Wullinger
no flags Details | Diff
v5.2.2, Python 3 support, keep python2 default, FreeBSD authormapsuffix map, plist fixes (70.83 KB, patch)
2020-01-07 20:53 UTC, Peter Wullinger
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Olivier Duchateau 2019-12-05 19:36:31 UTC
Created attachment 209740 [details]
Patch to update devel/mercurial

Update to 5.2.1
Comment 1 Automation User 2019-12-20 18:30:21 UTC
Build info is available at https://gitlab.com/swills/freebsd-ports/pipelines/104494374
Comment 2 John Hein 2019-12-27 20:22:56 UTC
Created attachment 210264 [details]
[patch] update mercurial to 5.2.1 including FREEBSD option (authormapsuffix)

Here's an update to the 5.2.1 patch that gets the FREEBSD option working again.  The only difference between this and Olivier's patch is the change to files/extra-patch-authormapsuffix

QA:
  poudriere - 11/amd64 ok (options: default + FREEBSD - i.e., all options on)
Comment 3 Peter Wullinger 2019-12-30 12:24:38 UTC
Created attachment 210328 [details]
Python 3 support

Add (flavored) Python 3 support.

- replace USES=python:2.7 by USES=python. This enables flavoured builds. Both Python 2 and Python 3 versions build successfully.
- add USE_PYTHON=autoplist. The current fixed plist breaks because of Python 3 compiled file locations (__pycache__)
- remove plist entries that get autogenerated by autoplist

CAVEATS:
The mercurial port does not have a flavor prefix, so the py3, py2 packages don't get a different name. 

This may have to to change, since we might need both versions for the time being. I'm not sure what the implications of a rename to py-mercurial would be.

The Python 3 version seems to work as expected (modify, commit, clone, etc.). I have not done any more complex testing.
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-31 01:52:36 UTC
(In reply to Peter Wullinger from comment #3)

Thanks Peter

 - Please add PYTHON_PKGNAMEPREFIX to the port, as package names must be unique
 - Please add USE_PYTHON=concurrent and confirm that all files created within shared/common locations (not specific to Python version) are made unique (they should be suffixed with PYTHON_VER). You can use make makeplist to check this. 

Port should additionally be QA tested with poudriere to ensure that the pkg-plist (manual or autoplist'd) doesn't have orphaned or missing entries after the above changes
Comment 5 Peter Wullinger 2019-12-31 11:31:27 UTC
(In reply to Kubilay Kocak from comment #4)

Thanks for the pointers.

The main remaining culprit seems to be mercurial/__modulepolicy__.py, which gets not included in the autoplist. I need to include it without having separate plists.
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-31 11:58:39 UTC
(In reply to Peter Wullinger from comment #5)

Yeh, I believe I was looking at this a number of weeks back.

module policy is treated specially, in setup.py [1]:

class hgbuildpy(build_py):
...
    def run(self):
...
        write_if_changed(os.path.join(basepath, '__modulepolicy__.py'), content)


[1] https://www.mercurial-scm.org/repo/hg-committed/file/tip/setup.py#l644

Technically, its a packaging bug, as all created files (especially manual ones) need to be added to the list of output files, so they are included in setup.py install --record output, which autoplist uses

This is a common issue with packages that do/execute custom things, see

https://github.com/skorokithakis/tbvaccine/blob/master/setup.py#L14

https://stackoverflow.com/questions/25686900/pip-install-misses-some-generated-files-when-writing-installed-files-txt
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-31 12:00:30 UTC
(In reply to Kubilay Kocak from comment #6)

Web searches for: "self.outputs" setup.py returns lots of results
Comment 8 Peter Wullinger 2019-12-31 17:00:59 UTC
Created attachment 210359 [details]
Python 3 support + FreeBSD authormapsuffix map + plist fixes

Updates patch for Python 3 support

- enable USE_PYTHON= concurrent
- bring back omitted authormapsuffix patch
- add PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX}
- add setup.py patch for keeping track of __modulepolicy__.py

This makes the port into mercurial@py37 and mercurial@py27.
Given that we are 6 hours till Python 2 EOL, I would consider

py27_PKGNAMESUFFIX=

instead. This makes the Python 3 flavor the default and the Python 2 flavor use the suffix:

mercurial-py27-5.2.1.txz
mercurial-5.2.1.txz
 
Checks done:
- pkg-plist for py27 and py37 look okay
- parallel install of both packages (py37 flavor has non-prefixed files)

local poudriere builds (testport)
12.1/amd64: @py27
12.1/amd64: @py37
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-01 02:30:32 UTC
(In reply to Peter Wullinger from comment #8)

Great patch Peter, thank you

One thing left is to check the pkg-plist (make makeplist) for any leftover files not processed by USE_PYTHON=concurrent to ensure there's none that are still installed in common/shared (non Python-version-specific) locations.

From the updated patch it looks like (at least) the manual pages will still conflict, and need to be processed with concurrent as well:

man/man1/hg.1.gz
man/man5/hgignore.5.gz
man/man5/hgignore.5.gz
man/man5/hgrc.5.gz
man/man5/hgrc.5.gz

You should be able to do this by extending UNIQUE_SUFFIX_FILES (or UNIQUE_FIND_SUFFIX_FILES, or UNIQUE_SUFFIX_TYPES (See Mk/Uses/uniquefiles.mk examples and how Mk/Uses/python.mk uses uniquefiles)

We also need to check that any and all python scripts that are installed use the specific Python-version executable the port was built with in their shebang lines. 

Any that aren't handled by setuptools automatic shebang processing (via setup.py:console_scripts/scripts methods) need to be processed with USES=shebangfix
Comment 10 John Hein 2020-01-02 16:19:18 UTC
(In reply to Peter Wullinger from comment #8)
The authormapsuffix addition should use b'' string markers to match how mercurial is marking strings for python 2/3 compatibility.  See attachment 210264 [details] (from comment 2).
Comment 11 Peter Wullinger 2020-01-02 16:47:02 UTC
(In reply to Kubilay Kocak from comment #9)

Looks fine by me

I get

    man/man1/hg-2.7.1.gz
    man/man5/hgignore-2.7.5.gz
    man5/hgrc-2.7.5.gz

vs.

    man/man1/hg-3.7.1.gz
    man/man1/hg.1.gz
    man/man5/hgignore-3.7.5.gz
    man/man5/hgignore.5.gz
    man/man5/hgrc-3.7.5.gz
    man/man5/hgrc.5.gz

Additionally, common/shared go to

   share/doc/py27-mercurial/
   share/licenses/mercurial-py27-5.2.1/
   share/py27-mercurial/

vs

   share/doc/py37-mercurial/
   share/licenses/mercurial-py37-5.2.1/
   share/py37-mercurial/
Comment 12 Peter Wullinger 2020-01-02 16:50:00 UTC
Created attachment 210396 [details]
Python 3 support + FreeBSD authormapsuffix map + plist fixes

Same as 210359 but with proper bytestring literals for authormapsuffix.

Thanks to John Hein for pointing that out.
Comment 13 Peter Wullinger 2020-01-02 17:09:08 UTC
Created attachment 210397 [details]
Python 3 support + FreeBSD authormapsuffix map + plist fixes

Now we shebang fixes for contrib/*

Note that this does not yet properly fix the env-shebang from www/hgweb.cgi (www/hgweb.fcgi is rewritten).
Comment 14 Olivier Duchateau 2020-01-07 06:02:34 UTC
A new release (5.2.2) is available. As it is bugfix release (5.2.x) support of Python3 is still experimental.

Default Python in 5.2.x serie is 2.7.
Comment 15 Peter Wullinger 2020-01-07 17:23:01 UTC
Created attachment 210508 [details]
v5.2.2, Python 3 support, FreeBSD authormapsuffix map, plist fixes

(In reply to Olivier Duchateau from comment #14)

I read this as that you would like to keep USES = python:2.7 without PKGNAMESUFFIX for now. Is that correct?

Looks like the update to 5.2.2 is trivial, port-wise (see patch)

poudrier testport results:
  py27 amd64/12.1: ok
  py37 amd64/12.1: ok
Comment 16 Olivier Duchateau 2020-01-07 18:07:22 UTC
(In reply to Peter Wullinger from comment #15)

Mercurial 5.2.2 is minor release, so support of python3 is still experimental (and there is no announce). You can provide an option (if users want to try), but by default it is still python 2.7.

So the line:

> USES= python:2.7

is still valid.
Comment 17 Peter Wullinger 2020-01-07 20:53:40 UTC
Created attachment 210516 [details]
v5.2.2, Python 3 support, keep python2 default, FreeBSD authormapsuffix map, plist fixes

(In reply to Olivier Duchateau from comment #16)

So be it then:

- keep USES=python:2.7,
- allow switching flavors (via Makefile) edit.

That should make it possible to easily switch to Python 3 once experimental state is officially dropped.

Mercurial's Python 3 wiki page claims that 5.2 is the first version with official Python 3 support (https://www.mercurial-scm.org/wiki/Python3). The release notes do not clarify the state of that support, though.
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-08 03:41:03 UTC
From the mercurial wiki [1]:

Mercurial 5.2 is the first release that officially has support for Python 3. 

Supported Python 3 versions are 3.5, 3.6, and 3.7. 

Python 3.8 mostly works, but there are a few known incompatibilities

It is the project policy for Mercurial and its core extensions to be compatible with Python 3. Over 99% of tests pass with Python 3 and test regressions are treated seriously. 

But also, from version 5.x overview [2]

If you package Mercurial or distribute it to users, we do not recommend making Python 3 the default at this time.

We should probably make a mercurial python 3 port/package, or custom flavour the mercurial port making Python 2.x the default, but allowing Python 3 builds.

[1] https://www.mercurial-scm.org/wiki/Python3
[2] https://www.mercurial-scm.org/wiki/Release5.0
Comment 19 Alexander Ushakov 2020-01-19 14:43:28 UTC
Will mercurial port be updated to version 5.2 soon? Or it'll be delayed until solving python version selecting issue?