Bug 242463 - devel/mercurial: Update to 5.4
Summary: devel/mercurial: Update to 5.4
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-python (Nobody)
URL:
Keywords: buildisok
Depends on:
Blocks:
 
Reported: 2019-12-05 19:36 UTC by Olivier Duchateau
Modified: 2020-06-29 03:25 UTC (History)
11 users (show)

See Also:
bugzilla: maintainer-feedback? (python)
antoine: exp-run-


Attachments
Patch to update devel/mercurial (4.00 KB, patch)
2019-12-05 19:36 UTC, Olivier Duchateau
no flags Details | Diff
[patch] update mercurial to 5.2.1 including FREEBSD option (authormapsuffix) (5.32 KB, patch)
2019-12-27 20:22 UTC, John Hein
no flags Details | Diff
Python 3 support (66.60 KB, patch)
2019-12-30 12:24 UTC, Peter Wullinger
no flags Details | Diff
Python 3 support + FreeBSD authormapsuffix map + plist fixes (69.84 KB, patch)
2019-12-31 17:00 UTC, Peter Wullinger
no flags Details | Diff
Python 3 support + FreeBSD authormapsuffix map + plist fixes (70.16 KB, patch)
2020-01-02 16:50 UTC, Peter Wullinger
no flags Details | Diff
Python 3 support + FreeBSD authormapsuffix map + plist fixes (70.96 KB, patch)
2020-01-02 17:09 UTC, Peter Wullinger
no flags Details | Diff
v5.2.2, Python 3 support, FreeBSD authormapsuffix map, plist fixes (71.21 KB, patch)
2020-01-07 17:23 UTC, Peter Wullinger
no flags Details | Diff
v5.2.2, Python 3 support, keep python2 default, FreeBSD authormapsuffix map, plist fixes (70.83 KB, patch)
2020-01-07 20:53 UTC, Peter Wullinger
no flags Details | Diff
mercurial update to 5.3.1 (69.27 KB, patch)
2020-03-07 18:55 UTC, Sergey A. Osokin
no flags Details | Diff
mercurial update to 5.3.1 (69.27 KB, patch)
2020-04-01 18:50 UTC, Sergey A. Osokin
no flags Details | Diff
mercurial update to 5.3.2 (70.20 KB, patch)
2020-04-06 22:38 UTC, Sergey A. Osokin
no flags Details | Diff
mercurial update to 5.4 (70.19 KB, patch)
2020-05-13 01:34 UTC, Sergey A. Osokin
no flags Details | Diff
mercurial update to 5.4 (70.32 KB, patch)
2020-05-17 23:00 UTC, Sergey A. Osokin
no flags Details | Diff
update to mercurial 5.4.1, bring bang setup.py autoplist fix, add extension compat message (71.04 KB, patch)
2020-06-06 15:38 UTC, Peter Wullinger
no flags Details | Diff
proposed patch (175.08 KB, patch)
2020-06-17 05:35 UTC, Steve Wills
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Olivier Duchateau 2019-12-05 19:36:31 UTC
Created attachment 209740 [details]
Patch to update devel/mercurial

Update to 5.2.1
Comment 1 Automation User 2019-12-20 18:30:21 UTC
Build info is available at https://gitlab.com/swills/freebsd-ports/pipelines/104494374
Comment 2 John Hein 2019-12-27 20:22:56 UTC
Created attachment 210264 [details]
[patch] update mercurial to 5.2.1 including FREEBSD option (authormapsuffix)

Here's an update to the 5.2.1 patch that gets the FREEBSD option working again.  The only difference between this and Olivier's patch is the change to files/extra-patch-authormapsuffix

QA:
  poudriere - 11/amd64 ok (options: default + FREEBSD - i.e., all options on)
Comment 3 Peter Wullinger 2019-12-30 12:24:38 UTC
Created attachment 210328 [details]
Python 3 support

Add (flavored) Python 3 support.

- replace USES=python:2.7 by USES=python. This enables flavoured builds. Both Python 2 and Python 3 versions build successfully.
- add USE_PYTHON=autoplist. The current fixed plist breaks because of Python 3 compiled file locations (__pycache__)
- remove plist entries that get autogenerated by autoplist

CAVEATS:
The mercurial port does not have a flavor prefix, so the py3, py2 packages don't get a different name. 

This may have to to change, since we might need both versions for the time being. I'm not sure what the implications of a rename to py-mercurial would be.

The Python 3 version seems to work as expected (modify, commit, clone, etc.). I have not done any more complex testing.
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-31 01:52:36 UTC
(In reply to Peter Wullinger from comment #3)

Thanks Peter

 - Please add PYTHON_PKGNAMEPREFIX to the port, as package names must be unique
 - Please add USE_PYTHON=concurrent and confirm that all files created within shared/common locations (not specific to Python version) are made unique (they should be suffixed with PYTHON_VER). You can use make makeplist to check this. 

Port should additionally be QA tested with poudriere to ensure that the pkg-plist (manual or autoplist'd) doesn't have orphaned or missing entries after the above changes
Comment 5 Peter Wullinger 2019-12-31 11:31:27 UTC
(In reply to Kubilay Kocak from comment #4)

Thanks for the pointers.

The main remaining culprit seems to be mercurial/__modulepolicy__.py, which gets not included in the autoplist. I need to include it without having separate plists.
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-31 11:58:39 UTC
(In reply to Peter Wullinger from comment #5)

Yeh, I believe I was looking at this a number of weeks back.

module policy is treated specially, in setup.py [1]:

class hgbuildpy(build_py):
...
    def run(self):
...
        write_if_changed(os.path.join(basepath, '__modulepolicy__.py'), content)


[1] https://www.mercurial-scm.org/repo/hg-committed/file/tip/setup.py#l644

Technically, its a packaging bug, as all created files (especially manual ones) need to be added to the list of output files, so they are included in setup.py install --record output, which autoplist uses

This is a common issue with packages that do/execute custom things, see

https://github.com/skorokithakis/tbvaccine/blob/master/setup.py#L14

https://stackoverflow.com/questions/25686900/pip-install-misses-some-generated-files-when-writing-installed-files-txt
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-31 12:00:30 UTC
(In reply to Kubilay Kocak from comment #6)

Web searches for: "self.outputs" setup.py returns lots of results
Comment 8 Peter Wullinger 2019-12-31 17:00:59 UTC
Created attachment 210359 [details]
Python 3 support + FreeBSD authormapsuffix map + plist fixes

Updates patch for Python 3 support

- enable USE_PYTHON= concurrent
- bring back omitted authormapsuffix patch
- add PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX}
- add setup.py patch for keeping track of __modulepolicy__.py

This makes the port into mercurial@py37 and mercurial@py27.
Given that we are 6 hours till Python 2 EOL, I would consider

py27_PKGNAMESUFFIX=

instead. This makes the Python 3 flavor the default and the Python 2 flavor use the suffix:

mercurial-py27-5.2.1.txz
mercurial-5.2.1.txz
 
Checks done:
- pkg-plist for py27 and py37 look okay
- parallel install of both packages (py37 flavor has non-prefixed files)

local poudriere builds (testport)
12.1/amd64: @py27
12.1/amd64: @py37
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-01 02:30:32 UTC
(In reply to Peter Wullinger from comment #8)

Great patch Peter, thank you

One thing left is to check the pkg-plist (make makeplist) for any leftover files not processed by USE_PYTHON=concurrent to ensure there's none that are still installed in common/shared (non Python-version-specific) locations.

From the updated patch it looks like (at least) the manual pages will still conflict, and need to be processed with concurrent as well:

man/man1/hg.1.gz
man/man5/hgignore.5.gz
man/man5/hgignore.5.gz
man/man5/hgrc.5.gz
man/man5/hgrc.5.gz

You should be able to do this by extending UNIQUE_SUFFIX_FILES (or UNIQUE_FIND_SUFFIX_FILES, or UNIQUE_SUFFIX_TYPES (See Mk/Uses/uniquefiles.mk examples and how Mk/Uses/python.mk uses uniquefiles)

We also need to check that any and all python scripts that are installed use the specific Python-version executable the port was built with in their shebang lines. 

Any that aren't handled by setuptools automatic shebang processing (via setup.py:console_scripts/scripts methods) need to be processed with USES=shebangfix
Comment 10 John Hein 2020-01-02 16:19:18 UTC
(In reply to Peter Wullinger from comment #8)
The authormapsuffix addition should use b'' string markers to match how mercurial is marking strings for python 2/3 compatibility.  See attachment 210264 [details] (from comment 2).
Comment 11 Peter Wullinger 2020-01-02 16:47:02 UTC
(In reply to Kubilay Kocak from comment #9)

Looks fine by me

I get

    man/man1/hg-2.7.1.gz
    man/man5/hgignore-2.7.5.gz
    man5/hgrc-2.7.5.gz

vs.

    man/man1/hg-3.7.1.gz
    man/man1/hg.1.gz
    man/man5/hgignore-3.7.5.gz
    man/man5/hgignore.5.gz
    man/man5/hgrc-3.7.5.gz
    man/man5/hgrc.5.gz

Additionally, common/shared go to

   share/doc/py27-mercurial/
   share/licenses/mercurial-py27-5.2.1/
   share/py27-mercurial/

vs

   share/doc/py37-mercurial/
   share/licenses/mercurial-py37-5.2.1/
   share/py37-mercurial/
Comment 12 Peter Wullinger 2020-01-02 16:50:00 UTC
Created attachment 210396 [details]
Python 3 support + FreeBSD authormapsuffix map + plist fixes

Same as 210359 but with proper bytestring literals for authormapsuffix.

Thanks to John Hein for pointing that out.
Comment 13 Peter Wullinger 2020-01-02 17:09:08 UTC
Created attachment 210397 [details]
Python 3 support + FreeBSD authormapsuffix map + plist fixes

Now we shebang fixes for contrib/*

Note that this does not yet properly fix the env-shebang from www/hgweb.cgi (www/hgweb.fcgi is rewritten).
Comment 14 Olivier Duchateau 2020-01-07 06:02:34 UTC
A new release (5.2.2) is available. As it is bugfix release (5.2.x) support of Python3 is still experimental.

Default Python in 5.2.x serie is 2.7.
Comment 15 Peter Wullinger 2020-01-07 17:23:01 UTC
Created attachment 210508 [details]
v5.2.2, Python 3 support, FreeBSD authormapsuffix map, plist fixes

(In reply to Olivier Duchateau from comment #14)

I read this as that you would like to keep USES = python:2.7 without PKGNAMESUFFIX for now. Is that correct?

Looks like the update to 5.2.2 is trivial, port-wise (see patch)

poudrier testport results:
  py27 amd64/12.1: ok
  py37 amd64/12.1: ok
Comment 16 Olivier Duchateau 2020-01-07 18:07:22 UTC
(In reply to Peter Wullinger from comment #15)

Mercurial 5.2.2 is minor release, so support of python3 is still experimental (and there is no announce). You can provide an option (if users want to try), but by default it is still python 2.7.

So the line:

> USES= python:2.7

is still valid.
Comment 17 Peter Wullinger 2020-01-07 20:53:40 UTC
Created attachment 210516 [details]
v5.2.2, Python 3 support, keep python2 default, FreeBSD authormapsuffix map, plist fixes

(In reply to Olivier Duchateau from comment #16)

So be it then:

- keep USES=python:2.7,
- allow switching flavors (via Makefile) edit.

That should make it possible to easily switch to Python 3 once experimental state is officially dropped.

Mercurial's Python 3 wiki page claims that 5.2 is the first version with official Python 3 support (https://www.mercurial-scm.org/wiki/Python3). The release notes do not clarify the state of that support, though.
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-08 03:41:03 UTC
From the mercurial wiki [1]:

Mercurial 5.2 is the first release that officially has support for Python 3. 

Supported Python 3 versions are 3.5, 3.6, and 3.7. 

Python 3.8 mostly works, but there are a few known incompatibilities

It is the project policy for Mercurial and its core extensions to be compatible with Python 3. Over 99% of tests pass with Python 3 and test regressions are treated seriously. 

But also, from version 5.x overview [2]

If you package Mercurial or distribute it to users, we do not recommend making Python 3 the default at this time.

We should probably make a mercurial python 3 port/package, or custom flavour the mercurial port making Python 2.x the default, but allowing Python 3 builds.

[1] https://www.mercurial-scm.org/wiki/Python3
[2] https://www.mercurial-scm.org/wiki/Release5.0
Comment 19 Alexander Ushakov 2020-01-19 14:43:28 UTC
Will mercurial port be updated to version 5.2 soon? Or it'll be delayed until solving python version selecting issue?
Comment 20 Sergey A. Osokin freebsd_committer 2020-03-07 18:55:12 UTC
Created attachment 212224 [details]
mercurial update to 5.3.1

Hi,

here is the update to the latest mercurial release 5.3.1.
Comment 21 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-09 03:18:12 UTC
(In reply to Sergey A. Osokin from comment #20)

Thanks Sergey. Could you update your patch to keep python:2.7 in there, so we can solve the addition/allowing Python 3 issue separately, or in a subsequent commit, as it *may* be better that that first lands as a separate Python 3 version of the port, unless Someone (TM) can provide clarity/verification of the impact on users with regard to extensions.

We may also be able to conditionally display a pkg-message (and maybe create an UPDATING entry) warning users that extensions need to be Python version compatible with the version of Python that the port is built with.
Comment 22 Sergey A. Osokin freebsd_committer 2020-04-01 18:50:57 UTC
Created attachment 212948 [details]
mercurial update to 5.3.1

Hi,

here's the recent patch with:
* python 2.7
* whitespace fix
Comment 23 Sergey A. Osokin freebsd_committer 2020-04-06 22:38:07 UTC
Created attachment 213140 [details]
mercurial update to 5.3.2

Update devel/mercurial to 5.3.2 and fix FreeBSD-specific patch.
Comment 24 Kurt Jaeger freebsd_committer 2020-04-18 17:29:34 UTC
testbuild@work
Comment 25 Kurt Jaeger freebsd_committer 2020-04-18 18:06:35 UTC
Adding 

%%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.py
%%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyc
%%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyo

to pkg-plist and it builds fine on 13a, 13i, 12a, 11.3a.
Comment 26 Kurt Jaeger freebsd_committer 2020-04-20 05:31:18 UTC
osa@ (submitter of final version) requests an exp-run for this mercurial update.
Comment 27 Antoine Brodin freebsd_committer 2020-04-20 05:50:24 UTC
USE_PYTHON=concurrent  does not make sense for a port that has USES=python:2.7
Comment 28 Antoine Brodin freebsd_committer 2020-04-20 06:51:04 UTC
This needs more review/qa than an exp-run
Comment 29 Sergey A. Osokin freebsd_committer 2020-04-28 21:32:26 UTC
Hello,

could you please provide an update.

Thanks.
Comment 30 Sergey A. Osokin freebsd_committer 2020-05-13 01:34:53 UTC
Created attachment 214439 [details]
mercurial update to 5.4
Comment 31 Kurt Jaeger freebsd_committer 2020-05-17 18:59:22 UTC
Building in poudriere brings up this error:

Error: Package names are not unique with flavors:
py37: mercurial-5.4
py27: mercurial-5.4
Error: maybe use <flavor>_PKGNAMEPREFIX/SUFFIX.
Comment 32 Sunpoet Po-Chuan Hsieh freebsd_committer 2020-05-17 19:15:00 UTC
(In reply to Sergey A. Osokin from comment #30)

If you relax USES=python, please add "PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX}" to avoid pkgname conflict.
Comment 33 Sergey A. Osokin freebsd_committer 2020-05-17 23:00:13 UTC
Created attachment 214593 [details]
mercurial update to 5.4

Add "PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX}" to avoid pkgname conflict.
Comment 34 Sergey A. Osokin freebsd_committer 2020-05-17 23:00:55 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #32)

Thanks for the update, the patch has been updated.
Comment 35 Kurt Jaeger freebsd_committer 2020-05-18 13:06:19 UTC
(In reply to Antoine Brodin from comment #28)
So, what else needs to be checked ? (5.4 testbuilds done, looks fine)
Comment 36 Sunpoet Po-Chuan Hsieh freebsd_committer 2020-05-18 18:34:24 UTC
IMHO, next step is to update its dependent ports.

deskutils/zim
devel/git-cinnabar
devel/git-remote-hg
devel/hg-git
devel/hgreviewboard
devel/hgsvn
devel/ocaml-opam
devel/py-hglib
devel/py-setuptools_scm
devel/tortoisehg
ports-mgmt/portshaker
www/py-kallithea
www/trac-mercurial

Since it's flavored now, "hg:devel/mercurial" has to be changed.
You may need to use "hg${PYTHON_PKGNAMESUFFIX}:devel/mercurial@${PY_FLAVOR}" or
"mercurial${PYTHON_PKGNAMESUFFIX}>=0:devel/mercurial@${PY_FLAVOR}" instead.

The latter would be better if you need to specify the minimal version required.
Comment 37 Sunpoet Po-Chuan Hsieh freebsd_committer 2020-05-18 18:42:21 UTC
(In reply to Sergey A. Osokin from comment #34)

I noticed that the PLIST order has changed but the original order (LANG=C) is correct. Uppercase comes before lowercase. DATADIR (expands to share/mercurial) comes after man.
Comment 38 Charlie Li 2020-05-18 19:35:52 UTC
(In reply to Kurt Jaeger from comment #25)
.pyc and .pyo only work for Python 2. In order to package properly for Python 3, conditionals are needed:

%%PYTHON2%%%%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyc
%%PYTHON2%%%%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyo
%%PYTHON3%%%%PYTHON_SITELIBDIR%%/mercurial/__pycache__/__modulepolicy__.cpython-%%PYTHON_SUFFIX%%.opt-1.pyc
%%PYTHON3%%%%PYTHON_SITELIBDIR%%/mercurial/__pycache__/__modulepolicy__.cpython-%%PYTHON_SUFFIX%%.pyc

Kind of annoying since distutils/autoplist doesn't pick up __modulepolicy__.py since it is auto-generated.

Also, hgweb.cgi needs the shebangfix treatment.
Comment 39 Peter Wullinger 2020-05-19 14:29:47 UTC
(In reply to Charlie Li from comment #38)
> Kind of annoying since distutils/autoplist doesn't pick up __modulepolicy__.py since it is auto-generated.

Somewhere around the patch to 5.3.1 my changes for setup.py to include __modulepolicy__.py and derivations in the list of reported output files got dropped:

def get_outputs(self, include_bytecode=1):
    from itertools import chain
    import importlib
    extra_modules = [os.path.join(self.build_lib, 'mercurial', '__modulepolicy__.py')]
    extra_outputs = list(extra_modules)
    if include_bytecode:
        for extra_module in extra_modules:
            if self.compile:
                extra_outputs.append(
                    importlib.util.cache_from_source(filename, optimization='')
                )
                if self.optimize > 0:
                    extra_outputs.append(
                        importlib.util.cache_from_source(filename, optimization=self.optimize)
                    )
    return chain(
        build_py.get_outputs(self),
        extra_outputs
    )

I consider this rather crude, but the proper fix should probably happen upstream, anyway.

I'm not sure why

SHEBANG_FILES=	contrib/*

does not apply shebangfix to hgweb.cgi (but does properly update hgweb.fcgi).
Comment 40 Peter Wullinger 2020-06-06 15:38:45 UTC
Created attachment 215296 [details]
update to mercurial 5.4.1, bring bang setup.py autoplist fix, add extension compat message

Next attempt to bring my last patch up to date

- update to mercurial 5.4.1
- bring back setup.py get_outputs() workaround
- drop USES=python:2.7
- use PKGNAMESUFFIX=
- add SHEBANG_REGEX= so that hgweb.cgi gets updated properly
- add install message wrt. extension compat
Comment 41 Peter Wullinger 2020-06-07 08:52:40 UTC
(In reply to Peter Wullinger from comment #40)

build ok for 12.1-AMD64, py27 and py37

> - add SHEBANG_REGEX= so that hgweb.cgi gets updated properly

Still does not happen. I'm at a loss.
Comment 42 Jan Beich freebsd_committer 2020-06-15 04:12:39 UTC
Comment on attachment 215296 [details]
update to mercurial 5.4.1, bring bang setup.py autoplist fix, add extension compat message

> +PKGNAMESUFFIX=	${PYTHON_PKGNAMESUFFIX}

Maybe keep unsuffixed version e.g., 
- ${PYTHON_PKGNAMESUFFIX:N-py27} for Python 2 (like in ports r538844)
- ${PYTHON_PKGNAMESUFFIX:N-py${PYTHON3_DEFAULT:S/.//}} for Python 3
Comment 43 Steve Wills freebsd_committer 2020-06-17 05:35:49 UTC
Created attachment 215646 [details]
proposed patch

(In reply to Jan Beich from comment #42)

Here's my proposal