Created attachment 208562 [details] Changes to Mk/Uses/scons.mk for suppport of the :py3 argument. There are some attempts on flavoring devel/scons to support python3. These PRs did not get through. I don't know the details why. This is my attempt on creating a scons-py3 slave port and supporting it in Mk/Uses/scons.mk. It works for me. I can create the mongodb42 port with USES=scons:py3.
Created attachment 208563 [details] Changes to devel/scons to support a slave port.
Created attachment 208564 [details] Makefile for the new devel/scons-py3 port.
Though I understand the motivation for this port proposal, we'd prefer not to add py2/3 specific ports, because: - Python ports should always allow *any* Python version to be used for the build that the upstream supports. Any Python port unnecessarily restricting the declared version support is considered a bug. - All else being equal, there ought to be no work required for 'specifically' or explicitly flavouring any Python port, and instead leverage the automatic flavouring that is produced solely by declaring the versions that the software *supports*. Perhaps scons is differerent, but in that case, we should understand where and how, and seek to aim our effort toward addressing those questions rather than creating a potentially very short lived port that has limited utility, and comes with downsides. - It breaks with the conventions/mechanism that we have developed to allow arbitrary user-selection of their preferred Python version, while minimising conflicts between packages. We should resolve any barriers/issues with the main scons port not allowing, or not being currently appropriate for building with Python 3.x. Could you please add any relevant scons issue ID's to this bugs See Also field
@Sunpoet Let me know if/how I can help figure out the best way forward, available on IRC (freenode) any time to assist.
(In reply to Ronald Klop from comment #1) I think scons@py27 should be able to coexist with scons@py3x before python 2.7 EoL. In your proposal, it's scons and scons-py3.
Regarding using scons with python 3.x, my initial plan was to change its dependency from python 2.7 to python 3.x and patch dependent ports. But that's a long list and I failed to fix some ports. I think the better solution right now is as follows: - Add flavored devel/scons by relaxing USES=python - Change Mk/Uses/scons.mk to use flavored scons and add :py27 arg for scons@py27 - Let scons@py27 and scons@py3x coexist - Change all dependent ports which are not python3-ready to scons:py27
(In reply to Sunpoet Po-Chuan Hsieh from comment #6) Hi, would it be possible to make scons@py27 the default? So that existing ports need not be updated at once, but new ports can start using scons@py3x? To stay backwards compatible for now? I have never 'flavoured' a port, so I hope I don't ask the obvious.
Created attachment 208682 [details] Add flavor to scons port and Mk/Uses New patch which uses flavor to create a py27-scons and a py36-scons package. This works for me with the existing mongodb36 ports which uses py27-scons without any changes to the mongodb36 port. And my new mongodb42 port uses py36-scons because it declares "USES=python:3.5+,build scons". These py*-scons packages conflict with each other. But that shouldn't be to big of a problem. If it is I'm willing to look into the uniquefiles Uses and see if that helps. AFAIK this change keeps the ports tree backwards compatible. Please give feedback.
(In reply to Sunpoet Po-Chuan Hsieh from comment #6) This is the better option. The scons port should do nothing but declare the versions *it* supports, and scons consumers ought to declare specifically the version(s) they support, allowing them to move forward at their own pace, without 'imperative' selection by the framework. Notes: - It's worth considering having scons.mk support <version-specifier> in the exact same way that USES=python supports it, rather than it being an FLAVOR argument (scons:py27) This allows, scons consuming ports to declaratively specify/declare version support, rather than 'imperatively' 'choosing' a framework specific implementation details (flavor) This also enables / gives us the ability to factor out a generic <version-specifier> support across more areas of the framework, starting with Python / Scons, and improving it in the process ("!=X" support, "X,Y" support, etc), which has huge benefits for consistency and version derivation across our tree.
(In reply to Ronald Klop from comment #7) Consumers requiring a specific version should be update to correctly declare their version support, at the same time that the scons port is updated to declare correctly its supported versions (in this case: "relaxing python versions to allow 2/3") It is not worth the benefit of punting the problem of updating ("fixing") consumers down the road
(In reply to Ronald Klop from comment #8) What are the conflicting files? If similar to other python packages/software, only a small proportion of expected files in non-python-version-specific locations will conflict (binaries in LOCALBASE, man pages, docs) and in that case, they ought to be resolved such that they are concurrently installable.
Created attachment 208730 [details] Add flavor to scons port and Mk/Uses New version. - No more conflicts because of --standard-lib and uniquefiles:dirs. - No implicit flavor chosen. Explicit need to specify USES=scons:py36. - USES=scons defaults to scons:py27 to stay backwards compatible. - /usr/local/bin/scons -> scons-3.6 is only installed by scons:py36 (this was automaticly done by the python/uniquefiles framework). - Easy to add more py* versions. Is there something necessary to fix or improve on this patch?
Created attachment 208731 [details] Add flavor to scons port and Mk/Uses Minor fix: USES=scons:py27 was not working, only USES=scons. Fixed, now supports both.
(In reply to Kubilay Kocak from comment #9) I'm looking for feedback on the latest patches. Does this version already provide the features you are talking about or did I misunderstand something? In short: What is needed to make this ready for commit?
Python2 being EOLed in a couple of weeks, is it worth pushing this further more?
Add a comment with the reply I got in my exp-run request to portmgr@. On Wed, Dec 4, 2019 at 11:01 PM Ronald Klop <ronald-lists@klop.ws> wrote: > > Dear portmgr, > > Is it possible to run an exp-run for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241463 . > Scons is a build tool for python, so it has quite some dependencies. > > It is the first time I request an exp-run, so please tell if I miss something in my request. Hi, Before an exp-run, this needs to be reviewed / improved / approved by maintainer. Some problems I see: - USES=uniquefiles:dirs seems not needed - USE_PYTHON=flavors seems not needed - PYDISTUTILS_INSTALLARGS is overridden - I don't like the scons arguments, in my opinion it should be 2 or 3, not py27 or py36 Cheers, Antoine Below is the corrected devel/scons diff Index: devel/scons/Makefile =================================================================== --- devel/scons/Makefile (revision 519049) +++ devel/scons/Makefile (working copy) @@ -5,6 +5,7 @@ PORTVERSION= 3.1.1 CATEGORIES= devel python MASTER_SITES= SF +PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX} MAINTAINER= sunpoet@FreeBSD.org COMMENT= Build tool alternative to make @@ -14,8 +15,8 @@ RUN_DEPENDS= gm4:devel/m4 -USES= python:2.7 shebangfix -USE_PYTHON= autoplist distutils +USES= python shebangfix +USE_PYTHON= autoplist concurrent distutils NO_ARCH= yes @@ -27,4 +28,8 @@ @${REINPLACE_CMD} -e 's|distutils\(\.command\.install\)$$|setuptools\1|; s|distutils\(\.command\.install\.\)|setuptools\1|' ${WRKSRC}/setup.py @${FIND} ${WRKSRC} -name '*.orig' -delete -.include <bsd.port.mk> +.include <bsd.port.pre.mk> + +PYDISTUTILS_INSTALLARGS+= --standard-lib + +.include <bsd.port.post.mk>
(In reply to Baptiste Daroussin from comment #15) "the last major version 2.7 will be released in April 2020" http://pyfound.blogspot.com/2019/12/python-2-sunset.html Our python27 port does not have a deprecation/expiration set. https://www.freshports.org/lang/python27 https://www.freshports.org/lang/python2 According to FreshPorts about 3400 ports have a run time dependency on python27 and even more a build time dependency. So I don't see information about "Python2 being EOLed in a couple of weeks". Please point me to relevant information about this EOL plan of our python ports. Because as you point out it would be a waste of time to flavor scons if not necessary. In the meantime this only stops people from using scons with python3 and helping the transition.
Created attachment 210285 [details] Add flavor to scons port and Mk/Uses Applied the feedback from Antoine comment #16. - Options are now scons:2 or scons:3 (scons defaults to 2 for backwards compatibility) instead of scons:py27. Antione likes this more. - uniquefiles:dirs was not needed - flavors was not needed And also: - Use PYTHON2_DEFAULT instead of hardcoded version numbers. - Removed unused variable from scons.mk.
Created attachment 212879 [details] Udpate scons to 3.1.2 + remove hardcoded python version 2
(In reply to VVD from comment #19) I'm trying to build databases/mongodb36 with this scons-3.1.2. But it always tries to build scons with python3 while mongodb36 needs scons to run with python2. Did you look into this? Is there a solution for this situation?
(In reply to VVD from comment #19) Thanks VVD. I just updated my ports and attempted to rebuild our packages. Kept getting: ... install -m 0644 /usr/ports/devel/py-setuptools/files/easy-install.pth.dist /var/ports/usr/ports/devel/py-setuptools/work-py37/stage/usr/local/lib/python3.7/site-packages/ ===> Creating unique files: Move MAN files needing SUFFIX ===> Creating unique files: Move files needing SUFFIX Move: bin/easy_install --> bin/easy_install-3.7 Link: @bin/easy_install --> bin/easy_install-3.7 ====> Compressing man pages (compress-man) ===> Installing for py37-setuptools-44.0.0 ===> Checking if py37-setuptools is already installed ===> py37-setuptools-44.0.0 is already installed You may wish to ``make deinstall'' and install this port again by ``make reinstall'' to upgrade it properly. If you really wish to overwrite the old port of py37-setuptools without deleting it first, set the variable "FORCE_PKG_REGISTER" in your environment or the "make install" command line. *** Error code 1 Stop. make[1]: stopped in /usr/ports/devel/py-setuptools *** Error code 1 Stop. make: stopped in /usr/ports/devel/scons Applied your patch, and make -C /usr/ports/devel/scons clean package built an scons package on todays rebuilt 12.1S amd64 and i386. :) The portmaster thread was: devel/erlang-erlware_commons 55/363 >> devel/erlang-cf >> devel/rebar3 >> devel/git >> devel/p5-subversion >> devel/subversion >> www/serf >> devel/scons >> devel/py-setuptools@py27 (151/151) So just when you think you're done, I tried: make -C /usr/ports/devel/subversion clean package ... ===> Building for serf-1.3.9_4 scons: Reading SConscript files ... File "/var/ports/usr/ports/www/serf/work/serf-1.3.9/SConstruct", line 189 print 'Warning: Used unknown variables:', ', '.join(unknown.keys()) ^ SyntaxError: invalid syntax Hmmm, looks like a python issue, probably needs 2.7. Successfully building ports is one measure, getting the successfully built port to function is another. ;)
(In reply to dewayne from comment #21) Just rebuild subversion without any issue. Rebuilded several times git last weeks. How to found your build errors?
Ok, rebuild serf. Check Ronald Klop's patch than.
After some hours on this, I've found that something in my make.conf has corrupted the situation. From my earlier post, it looks like scons will build with python 3.7, but it doesn't appear to run. My apologies for distracting the discussion.
To all cc's on this issue. The difficulty is not to run scons with python 2 or 3. Since scons 3.x supports both python versions. See https://scons.org/tag/releases.html. The difficulty is that the SConstruct files of other ports are written in python 2 (like mongodb36) or 3 (like upcoming mongodb42). Our current ports framework has no way to create both versions of scons. We can't create ports using scons with python3 as long as there are ports needing scons with python2. The first line of /usr/local/bin/scons says "#!/usr/local/bin/python2.7". That is the only line which need to be changed to run with python3. But we currently can't do that. My solution would be to flavor the port and create a py27-scons and py37-scons. The pkg build infrastructure can than use the one it needs.
Created attachment 213623 [details] Update to 3.1.2 Just update it to 3.1.2 than.
(In reply to Ronald Klop from comment #25) Python 2 has already reached EoL and will be removed from ports tree after the end of this year. And when python 2 has gone all ports that depends on python2-scons get broken anyway. So right way we should go is * Switch devel/scons to use python 3. * If it is possible fix ports that require python2-scons so they can be build with python3-scons. * Otherwise mark them BROKEN and set EXPIRATION_DATE. And first step is to create patch that make devel/scons to use python 3, do exp-run and see what/how-many ports are not compatible with python3-scons.
I think it done already: https://svnweb.freebsd.org/changeset/ports/536494 https://svnweb.freebsd.org/changeset/ports/536495 https://svnweb.freebsd.org/changeset/ports/536497
https://svnweb.freebsd.org/ports?view=revision&revision=536496
flavored scons landed. Thanks for your patience.