Created attachment 209740 [details] Patch to update devel/mercurial Update to 5.2.1
Build info is available at https://gitlab.com/swills/freebsd-ports/pipelines/104494374
Created attachment 210264 [details] [patch] update mercurial to 5.2.1 including FREEBSD option (authormapsuffix) Here's an update to the 5.2.1 patch that gets the FREEBSD option working again. The only difference between this and Olivier's patch is the change to files/extra-patch-authormapsuffix QA: poudriere - 11/amd64 ok (options: default + FREEBSD - i.e., all options on)
Created attachment 210328 [details] Python 3 support Add (flavored) Python 3 support. - replace USES=python:2.7 by USES=python. This enables flavoured builds. Both Python 2 and Python 3 versions build successfully. - add USE_PYTHON=autoplist. The current fixed plist breaks because of Python 3 compiled file locations (__pycache__) - remove plist entries that get autogenerated by autoplist CAVEATS: The mercurial port does not have a flavor prefix, so the py3, py2 packages don't get a different name. This may have to to change, since we might need both versions for the time being. I'm not sure what the implications of a rename to py-mercurial would be. The Python 3 version seems to work as expected (modify, commit, clone, etc.). I have not done any more complex testing.
(In reply to Peter Wullinger from comment #3) Thanks Peter - Please add PYTHON_PKGNAMEPREFIX to the port, as package names must be unique - Please add USE_PYTHON=concurrent and confirm that all files created within shared/common locations (not specific to Python version) are made unique (they should be suffixed with PYTHON_VER). You can use make makeplist to check this. Port should additionally be QA tested with poudriere to ensure that the pkg-plist (manual or autoplist'd) doesn't have orphaned or missing entries after the above changes
(In reply to Kubilay Kocak from comment #4) Thanks for the pointers. The main remaining culprit seems to be mercurial/__modulepolicy__.py, which gets not included in the autoplist. I need to include it without having separate plists.
(In reply to Peter Wullinger from comment #5) Yeh, I believe I was looking at this a number of weeks back. module policy is treated specially, in setup.py [1]: class hgbuildpy(build_py): ... def run(self): ... write_if_changed(os.path.join(basepath, '__modulepolicy__.py'), content) [1] https://www.mercurial-scm.org/repo/hg-committed/file/tip/setup.py#l644 Technically, its a packaging bug, as all created files (especially manual ones) need to be added to the list of output files, so they are included in setup.py install --record output, which autoplist uses This is a common issue with packages that do/execute custom things, see https://github.com/skorokithakis/tbvaccine/blob/master/setup.py#L14 https://stackoverflow.com/questions/25686900/pip-install-misses-some-generated-files-when-writing-installed-files-txt
(In reply to Kubilay Kocak from comment #6) Web searches for: "self.outputs" setup.py returns lots of results
Created attachment 210359 [details] Python 3 support + FreeBSD authormapsuffix map + plist fixes Updates patch for Python 3 support - enable USE_PYTHON= concurrent - bring back omitted authormapsuffix patch - add PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX} - add setup.py patch for keeping track of __modulepolicy__.py This makes the port into mercurial@py37 and mercurial@py27. Given that we are 6 hours till Python 2 EOL, I would consider py27_PKGNAMESUFFIX= instead. This makes the Python 3 flavor the default and the Python 2 flavor use the suffix: mercurial-py27-5.2.1.txz mercurial-5.2.1.txz Checks done: - pkg-plist for py27 and py37 look okay - parallel install of both packages (py37 flavor has non-prefixed files) local poudriere builds (testport) 12.1/amd64: @py27 12.1/amd64: @py37
(In reply to Peter Wullinger from comment #8) Great patch Peter, thank you One thing left is to check the pkg-plist (make makeplist) for any leftover files not processed by USE_PYTHON=concurrent to ensure there's none that are still installed in common/shared (non Python-version-specific) locations. From the updated patch it looks like (at least) the manual pages will still conflict, and need to be processed with concurrent as well: man/man1/hg.1.gz man/man5/hgignore.5.gz man/man5/hgignore.5.gz man/man5/hgrc.5.gz man/man5/hgrc.5.gz You should be able to do this by extending UNIQUE_SUFFIX_FILES (or UNIQUE_FIND_SUFFIX_FILES, or UNIQUE_SUFFIX_TYPES (See Mk/Uses/uniquefiles.mk examples and how Mk/Uses/python.mk uses uniquefiles) We also need to check that any and all python scripts that are installed use the specific Python-version executable the port was built with in their shebang lines. Any that aren't handled by setuptools automatic shebang processing (via setup.py:console_scripts/scripts methods) need to be processed with USES=shebangfix
(In reply to Peter Wullinger from comment #8) The authormapsuffix addition should use b'' string markers to match how mercurial is marking strings for python 2/3 compatibility. See attachment 210264 [details] (from comment 2).
(In reply to Kubilay Kocak from comment #9) Looks fine by me I get man/man1/hg-2.7.1.gz man/man5/hgignore-2.7.5.gz man5/hgrc-2.7.5.gz vs. man/man1/hg-3.7.1.gz man/man1/hg.1.gz man/man5/hgignore-3.7.5.gz man/man5/hgignore.5.gz man/man5/hgrc-3.7.5.gz man/man5/hgrc.5.gz Additionally, common/shared go to share/doc/py27-mercurial/ share/licenses/mercurial-py27-5.2.1/ share/py27-mercurial/ vs share/doc/py37-mercurial/ share/licenses/mercurial-py37-5.2.1/ share/py37-mercurial/
Created attachment 210396 [details] Python 3 support + FreeBSD authormapsuffix map + plist fixes Same as 210359 but with proper bytestring literals for authormapsuffix. Thanks to John Hein for pointing that out.
Created attachment 210397 [details] Python 3 support + FreeBSD authormapsuffix map + plist fixes Now we shebang fixes for contrib/* Note that this does not yet properly fix the env-shebang from www/hgweb.cgi (www/hgweb.fcgi is rewritten).
A new release (5.2.2) is available. As it is bugfix release (5.2.x) support of Python3 is still experimental. Default Python in 5.2.x serie is 2.7.
Created attachment 210508 [details] v5.2.2, Python 3 support, FreeBSD authormapsuffix map, plist fixes (In reply to Olivier Duchateau from comment #14) I read this as that you would like to keep USES = python:2.7 without PKGNAMESUFFIX for now. Is that correct? Looks like the update to 5.2.2 is trivial, port-wise (see patch) poudrier testport results: py27 amd64/12.1: ok py37 amd64/12.1: ok
(In reply to Peter Wullinger from comment #15) Mercurial 5.2.2 is minor release, so support of python3 is still experimental (and there is no announce). You can provide an option (if users want to try), but by default it is still python 2.7. So the line: > USES= python:2.7 is still valid.
Created attachment 210516 [details] v5.2.2, Python 3 support, keep python2 default, FreeBSD authormapsuffix map, plist fixes (In reply to Olivier Duchateau from comment #16) So be it then: - keep USES=python:2.7, - allow switching flavors (via Makefile) edit. That should make it possible to easily switch to Python 3 once experimental state is officially dropped. Mercurial's Python 3 wiki page claims that 5.2 is the first version with official Python 3 support (https://www.mercurial-scm.org/wiki/Python3). The release notes do not clarify the state of that support, though.
From the mercurial wiki [1]: Mercurial 5.2 is the first release that officially has support for Python 3. Supported Python 3 versions are 3.5, 3.6, and 3.7. Python 3.8 mostly works, but there are a few known incompatibilities It is the project policy for Mercurial and its core extensions to be compatible with Python 3. Over 99% of tests pass with Python 3 and test regressions are treated seriously. But also, from version 5.x overview [2] If you package Mercurial or distribute it to users, we do not recommend making Python 3 the default at this time. We should probably make a mercurial python 3 port/package, or custom flavour the mercurial port making Python 2.x the default, but allowing Python 3 builds. [1] https://www.mercurial-scm.org/wiki/Python3 [2] https://www.mercurial-scm.org/wiki/Release5.0
Will mercurial port be updated to version 5.2 soon? Or it'll be delayed until solving python version selecting issue?
Created attachment 212224 [details] mercurial update to 5.3.1 Hi, here is the update to the latest mercurial release 5.3.1.
(In reply to Sergey A. Osokin from comment #20) Thanks Sergey. Could you update your patch to keep python:2.7 in there, so we can solve the addition/allowing Python 3 issue separately, or in a subsequent commit, as it *may* be better that that first lands as a separate Python 3 version of the port, unless Someone (TM) can provide clarity/verification of the impact on users with regard to extensions. We may also be able to conditionally display a pkg-message (and maybe create an UPDATING entry) warning users that extensions need to be Python version compatible with the version of Python that the port is built with.
Created attachment 212948 [details] mercurial update to 5.3.1 Hi, here's the recent patch with: * python 2.7 * whitespace fix
Created attachment 213140 [details] mercurial update to 5.3.2 Update devel/mercurial to 5.3.2 and fix FreeBSD-specific patch.
testbuild@work
Adding %%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.py %%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyc %%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyo to pkg-plist and it builds fine on 13a, 13i, 12a, 11.3a.
osa@ (submitter of final version) requests an exp-run for this mercurial update.
USE_PYTHON=concurrent does not make sense for a port that has USES=python:2.7
This needs more review/qa than an exp-run
Hello, could you please provide an update. Thanks.
Created attachment 214439 [details] mercurial update to 5.4
Building in poudriere brings up this error: Error: Package names are not unique with flavors: py37: mercurial-5.4 py27: mercurial-5.4 Error: maybe use <flavor>_PKGNAMEPREFIX/SUFFIX.
(In reply to Sergey A. Osokin from comment #30) If you relax USES=python, please add "PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX}" to avoid pkgname conflict.
Created attachment 214593 [details] mercurial update to 5.4 Add "PKGNAMESUFFIX=${PYTHON_PKGNAMESUFFIX}" to avoid pkgname conflict.
(In reply to Sunpoet Po-Chuan Hsieh from comment #32) Thanks for the update, the patch has been updated.
(In reply to Antoine Brodin from comment #28) So, what else needs to be checked ? (5.4 testbuilds done, looks fine)
IMHO, next step is to update its dependent ports. deskutils/zim devel/git-cinnabar devel/git-remote-hg devel/hg-git devel/hgreviewboard devel/hgsvn devel/ocaml-opam devel/py-hglib devel/py-setuptools_scm devel/tortoisehg ports-mgmt/portshaker www/py-kallithea www/trac-mercurial Since it's flavored now, "hg:devel/mercurial" has to be changed. You may need to use "hg${PYTHON_PKGNAMESUFFIX}:devel/mercurial@${PY_FLAVOR}" or "mercurial${PYTHON_PKGNAMESUFFIX}>=0:devel/mercurial@${PY_FLAVOR}" instead. The latter would be better if you need to specify the minimal version required.
(In reply to Sergey A. Osokin from comment #34) I noticed that the PLIST order has changed but the original order (LANG=C) is correct. Uppercase comes before lowercase. DATADIR (expands to share/mercurial) comes after man.
(In reply to Kurt Jaeger from comment #25) .pyc and .pyo only work for Python 2. In order to package properly for Python 3, conditionals are needed: %%PYTHON2%%%%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyc %%PYTHON2%%%%PYTHON_SITELIBDIR%%/mercurial/__modulepolicy__.pyo %%PYTHON3%%%%PYTHON_SITELIBDIR%%/mercurial/__pycache__/__modulepolicy__.cpython-%%PYTHON_SUFFIX%%.opt-1.pyc %%PYTHON3%%%%PYTHON_SITELIBDIR%%/mercurial/__pycache__/__modulepolicy__.cpython-%%PYTHON_SUFFIX%%.pyc Kind of annoying since distutils/autoplist doesn't pick up __modulepolicy__.py since it is auto-generated. Also, hgweb.cgi needs the shebangfix treatment.
(In reply to Charlie Li from comment #38) > Kind of annoying since distutils/autoplist doesn't pick up __modulepolicy__.py since it is auto-generated. Somewhere around the patch to 5.3.1 my changes for setup.py to include __modulepolicy__.py and derivations in the list of reported output files got dropped: def get_outputs(self, include_bytecode=1): from itertools import chain import importlib extra_modules = [os.path.join(self.build_lib, 'mercurial', '__modulepolicy__.py')] extra_outputs = list(extra_modules) if include_bytecode: for extra_module in extra_modules: if self.compile: extra_outputs.append( importlib.util.cache_from_source(filename, optimization='') ) if self.optimize > 0: extra_outputs.append( importlib.util.cache_from_source(filename, optimization=self.optimize) ) return chain( build_py.get_outputs(self), extra_outputs ) I consider this rather crude, but the proper fix should probably happen upstream, anyway. I'm not sure why SHEBANG_FILES= contrib/* does not apply shebangfix to hgweb.cgi (but does properly update hgweb.fcgi).
Created attachment 215296 [details] update to mercurial 5.4.1, bring bang setup.py autoplist fix, add extension compat message Next attempt to bring my last patch up to date - update to mercurial 5.4.1 - bring back setup.py get_outputs() workaround - drop USES=python:2.7 - use PKGNAMESUFFIX= - add SHEBANG_REGEX= so that hgweb.cgi gets updated properly - add install message wrt. extension compat
(In reply to Peter Wullinger from comment #40) build ok for 12.1-AMD64, py27 and py37 > - add SHEBANG_REGEX= so that hgweb.cgi gets updated properly Still does not happen. I'm at a loss.
Comment on attachment 215296 [details] update to mercurial 5.4.1, bring bang setup.py autoplist fix, add extension compat message > +PKGNAMESUFFIX= ${PYTHON_PKGNAMESUFFIX} Maybe keep unsuffixed version e.g., - ${PYTHON_PKGNAMESUFFIX:N-py27} for Python 2 (like in ports r538844) - ${PYTHON_PKGNAMESUFFIX:N-py${PYTHON3_DEFAULT:S/.//}} for Python 3
Created attachment 215646 [details] proposed patch (In reply to Jan Beich from comment #42) Here's my proposal
Created attachment 217082 [details] update to mercurial 5.4.2, setup.py autoplist fix, extension compat message shebang fix update to mercurial 5.4.2 - keep setup.py shebang fix. Can I get an opinion on this? It saves a lot of plist-entries at the cost of a hacky setup.py patch - add py27_PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX} this makes the Python 3 version the default, non-prefixed version - finally(?) fix the hgweb shebang issue Thanks to Steve Wills for the fix
I was updating devel/tortoisehg and bumped into mercurial being too old. What's blocking this?
(In reply to Gleb Popov from comment #45) See comment 36, lots of testing of dependent ports
Hum, using ${PYTHON_PKGNAMEPREFIX}mercurial>=0:devel/mercurial@${PY_FLAVOR} as dependency line doesn't work, because the package is named "mercurial" for py37 and "py27-mercurial" for py27. Am I missing something or it should be fixed on the mercurial port side? P.S. Dependening on "hg" executable has the same problem.
Created attachment 217641 [details] patch I've created the PY_MERCURIAL variable in Uses/python.mk and used it in dependency lines for mentioned ports. I suspect it is wrong to add PYTHON_PKGNAMEPREFIX for py27 flavor only. Instead, it should be added to every non-default flavor. Another thing to consider is hg executable suffix. Some application might call "hg" program and will fail in 2.7 case.
Created attachment 217677 [details] patch Fixed plist, package naming depending on flavor, NLS option patch and remove python:concurrent. This patch passes poudriere bulk -t for all mentioned ports. OK to commit?
A commit references this bug: Author: arrowd Date: Sun Sep 6 15:09:30 UTC 2020 New revision: 547798 URL: https://svnweb.freebsd.org/changeset/ports/547798 Log: devel/mercurial: Update to 5.5 and flavorize. Adjust dependency lines and bump PORTREVISION's for ports that depend on mercurial. PR: 242463 Submitted by: Peter Wullinger <freebsd@dhke.de> (original patch) Changes: head/Mk/Uses/python.mk head/deskutils/zim/Makefile head/devel/git-cinnabar/Makefile head/devel/git-remote-hg/Makefile head/devel/hg-git/Makefile head/devel/hgreviewboard/Makefile head/devel/hgsvn/Makefile head/devel/mercurial/Makefile head/devel/mercurial/distinfo head/devel/mercurial/files/extra-patch-authormapsuffix head/devel/mercurial/files/extra-patch-setup.py head/devel/mercurial/files/pkg-message.in head/devel/mercurial/pkg-plist head/devel/ocaml-opam/Makefile head/devel/py-hglib/Makefile head/devel/py-setuptools_scm/Makefile head/devel/tortoisehg/Makefile head/devel/tortoisehg/distinfo head/devel/tortoisehg/pkg-plist head/ports-mgmt/portshaker/Makefile head/www/trac-mercurial/Makefile
I've gone ahead and committed this. I'll leave this PR open for some time while waiting for pointy hats.
Created attachment 217824 [details] [patch] add back help for --authormapsuffix (In reply to Gleb Popov from comment #51) The help info for 'hg help --authormapsuffix' (if FREEBSD option is on) got dropped between attachment 214593 [details] and attachment 215296 [details]. I assume this was accidental - it was probably somewhat hard for users to track the thread of patches for this long lived PR. Patch attached.
Thanks to all for finally getting this committed, by the way.
Created attachment 217825 [details] [patch] add back help for --authormapsuffix (v2) (v2) better ordering of --authormapsuffix help text (matching previous patches that had this).
A commit references this bug: Author: arrowd Date: Thu Sep 10 05:06:53 UTC 2020 New revision: 548183 URL: https://svnweb.freebsd.org/changeset/ports/548183 Log: devel/mercurial: Fix the extra patch for FREEBSD option. PR: 242463 Submitted by: John Hein <jcfyecrayz@liamekaens.com> Changes: head/devel/mercurial/Makefile head/devel/mercurial/files/extra-patch-authormapsuffix
(In reply to John Hein from comment #54) Thanks for pointing this out, John. I guess, I'll close this now.