Bug 241463

Summary: devel/scons: flavor scons for python3
Product: Ports & Packages Reporter: Ronald Klop <ronald-lists>
Component: Individual Port(s)Assignee: Sunpoet Po-Chuan Hsieh <sunpoet>
Status: Closed FIXED    
Severity: Affects Some People CC: dewayne, djohnson, freebsd, freebsd, koobs, pi, python, sunpoet, vvd, yaruta.arkadiy, yasu
Priority: --- Keywords: feature, needs-qa
Version: LatestFlags: koobs: maintainer-feedback? (sunpoet)
ronald-lists: exp-run?
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237122
Bug Depends on: 246391    
Bug Blocks: 242002    
Description Flags
Changes to Mk/Uses/scons.mk for suppport of the :py3 argument.
Changes to devel/scons to support a slave port.
Makefile for the new devel/scons-py3 port.
Add flavor to scons port and Mk/Uses
Add flavor to scons port and Mk/Uses
Add flavor to scons port and Mk/Uses
Add flavor to scons port and Mk/Uses
ronald-lists: maintainer-approval?
Udpate scons to 3.1.2 + remove hardcoded python version 2
Update to 3.1.2 vvd: maintainer-approval?

Description Ronald Klop 2019-10-24 12:22:32 UTC
Created attachment 208562 [details]
Changes to Mk/Uses/scons.mk for suppport of the :py3 argument.

There are some attempts on flavoring devel/scons to support python3. These PRs did not get through. I don't know the details why.

This is my attempt on creating a scons-py3 slave port and supporting it in Mk/Uses/scons.mk.
It works for me. I can create the mongodb42 port with USES=scons:py3.
Comment 1 Ronald Klop 2019-10-24 12:23:36 UTC
Created attachment 208563 [details]
Changes to devel/scons to support a slave port.
Comment 2 Ronald Klop 2019-10-24 12:24:30 UTC
Created attachment 208564 [details]
Makefile for the new devel/scons-py3 port.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2019-10-25 07:23:45 UTC
Though I understand the motivation for this port proposal, we'd prefer not to add py2/3 specific ports, because:

- Python ports should always allow *any* Python version to be used for the build that the upstream supports. Any Python port unnecessarily restricting the declared version support is considered a bug.

- All else being equal, there ought to be no work required for 'specifically' or explicitly flavouring any Python port, and instead leverage the automatic flavouring that is produced solely by declaring the versions that the software *supports*. Perhaps scons is differerent, but in that case, we should understand where and how, and seek to aim our effort toward addressing those questions rather than creating a potentially very short lived port that has limited utility, and comes with downsides.

- It breaks with the conventions/mechanism that we have developed to allow arbitrary user-selection of their preferred Python version, while minimising conflicts between packages.

We should resolve any barriers/issues with the main scons port not allowing, or not being currently appropriate for building with Python 3.x.

Could you please add any relevant scons issue ID's to this bugs See Also field
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2019-10-25 08:25:46 UTC
@Sunpoet Let me know if/how I can help figure out the best way forward, available on IRC (freenode) any time to assist.
Comment 5 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-10-29 15:41:31 UTC
(In reply to Ronald Klop from comment #1)

I think scons@py27 should be able to coexist with scons@py3x before python 2.7 EoL. In your proposal, it's scons and scons-py3.
Comment 6 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-10-29 15:45:07 UTC
Regarding using scons with python 3.x, my initial plan was to change its dependency from python 2.7 to python 3.x and patch dependent ports. But that's a long list and I failed to fix some ports.

I think the better solution right now is as follows:
- Add flavored devel/scons by relaxing USES=python
- Change Mk/Uses/scons.mk to use flavored scons and add :py27 arg for scons@py27
- Let scons@py27 and scons@py3x coexist
- Change all dependent ports which are not python3-ready to scons:py27
Comment 7 Ronald Klop 2019-10-29 17:35:03 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #6)
Hi, would it be possible to make scons@py27 the default? So that existing ports need not be updated at once, but new ports can start using scons@py3x?
To stay backwards compatible for now?
I have never 'flavoured' a port, so I hope I don't ask the obvious.
Comment 8 Ronald Klop 2019-10-29 22:56:07 UTC
Created attachment 208682 [details]
Add flavor to scons port and Mk/Uses

New patch which uses flavor to create a py27-scons and a py36-scons package.
This works for me with the existing mongodb36 ports which uses py27-scons without any changes to the mongodb36 port.
And my new mongodb42 port uses py36-scons because it declares "USES=python:3.5+,build scons".
These py*-scons packages conflict with each other. But that shouldn't be to big of a problem. If it is I'm willing to look into the uniquefiles Uses and see if that helps.
AFAIK this change keeps the ports tree backwards compatible. Please give feedback.
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2019-10-30 01:33:17 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #6)

This is the better option. The scons port should do nothing but declare the versions *it* supports, and scons consumers ought to declare specifically the version(s) they support, allowing them to move forward at their own pace, without 'imperative' selection by the framework.


- It's worth considering having scons.mk support <version-specifier> in the exact same way that USES=python supports it, rather than it being an FLAVOR argument (scons:py27)

This allows, scons consuming ports to declaratively specify/declare version support, rather than 'imperatively' 'choosing' a framework specific implementation details (flavor)

This also enables / gives us the ability to factor out a generic <version-specifier> support across more areas of the framework, starting with Python / Scons, and improving it in the process ("!=X" support, "X,Y" support, etc), which has huge benefits for consistency and version derivation across our tree.
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2019-10-30 01:35:51 UTC
(In reply to Ronald Klop from comment #7)

Consumers requiring a specific version should be update to correctly declare their version support, at the same time that the scons port is updated to declare correctly its supported versions (in this case: "relaxing python versions to allow 2/3")

It is not worth the benefit of punting the problem of updating ("fixing") consumers down the road
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2019-10-30 01:37:16 UTC
(In reply to Ronald Klop from comment #8)

What are the conflicting files? If similar to other python packages/software, only a small proportion of expected files in non-python-version-specific locations will conflict (binaries in LOCALBASE, man pages, docs) and in that case, they ought to be resolved such that they are concurrently installable.
Comment 12 Ronald Klop 2019-10-31 11:32:16 UTC
Created attachment 208730 [details]
Add flavor to scons port and Mk/Uses

New version.

- No more conflicts because of --standard-lib and uniquefiles:dirs.
- No implicit flavor chosen. Explicit need to specify USES=scons:py36.
- USES=scons defaults to scons:py27 to stay backwards compatible.
- /usr/local/bin/scons -> scons-3.6 is only installed by scons:py36 (this was automaticly done by the python/uniquefiles framework).
- Easy to add more py* versions.

Is there something necessary to fix or improve on this patch?
Comment 13 Ronald Klop 2019-10-31 11:39:21 UTC
Created attachment 208731 [details]
Add flavor to scons port and Mk/Uses

Minor fix: USES=scons:py27 was not working, only USES=scons. Fixed, now supports both.
Comment 14 Ronald Klop 2019-11-06 09:06:02 UTC
(In reply to Kubilay Kocak from comment #9)
I'm looking for feedback on the latest patches. Does this version already provide the features you are talking about or did I misunderstand something?

In short: What is needed to make this ready for commit?
Comment 15 Baptiste Daroussin freebsd_committer 2019-12-05 17:54:35 UTC
Python2 being EOLed in a couple of weeks, is it worth pushing this further more?
Comment 16 Ronald Klop 2019-12-09 21:20:41 UTC
Add a comment with the reply I got in my exp-run request to portmgr@.

On Wed, Dec 4, 2019 at 11:01 PM Ronald Klop <ronald-lists@klop.ws> wrote:
> Dear portmgr,
> Is it possible to run an exp-run for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241463 .
> Scons is a build tool for python, so it has quite some dependencies.
> It is the first time I request an exp-run, so please tell if I miss something in my request.
Before an exp-run,  this needs to be reviewed / improved / approved by
Some problems I see:
- USES=uniquefiles:dirs seems not needed
- USE_PYTHON=flavors seems not needed
- I don't like the scons arguments,  in my opinion it should be 2 or
3,  not py27 or py36
Below is the corrected devel/scons diff
Index: devel/scons/Makefile
--- devel/scons/Makefile        (revision 519049)
+++ devel/scons/Makefile        (working copy)
@@ -5,6 +5,7 @@
 CATEGORIES=    devel python
 MAINTAINER=    sunpoet@FreeBSD.org
 COMMENT=       Build tool alternative to make
@@ -14,8 +15,8 @@
 RUN_DEPENDS=   gm4:devel/m4
-USES=          python:2.7 shebangfix
-USE_PYTHON=    autoplist distutils
+USES=          python shebangfix
+USE_PYTHON=    autoplist concurrent distutils
 NO_ARCH=       yes
@@ -27,4 +28,8 @@
        @${REINPLACE_CMD} -e
s|distutils\(\.command\.install\.\)|setuptools\1|' ${WRKSRC}/setup.py
        @${FIND} ${WRKSRC} -name '*.orig' -delete
-.include <bsd.port.mk>
+.include <bsd.port.pre.mk>
+PYDISTUTILS_INSTALLARGS+=      --standard-lib
+.include <bsd.port.post.mk>
Comment 17 Ronald Klop 2019-12-28 14:19:28 UTC
(In reply to Baptiste Daroussin from comment #15)

"the last major version 2.7 will be released in April 2020"

Our python27 port does not have a deprecation/expiration set.
According to FreshPorts about 3400 ports have a run time dependency on python27 and even more a build time dependency.

So I don't see information about "Python2 being EOLed in a couple of weeks". Please point me to relevant information about this EOL plan of our python ports. Because as you point out it would be a waste of time to flavor scons if not necessary. In the meantime this only stops people from using scons with python3 and helping the transition.
Comment 18 Ronald Klop 2019-12-28 15:02:05 UTC
Created attachment 210285 [details]
Add flavor to scons port and Mk/Uses

Applied the feedback from Antoine comment #16.
- Options are now scons:2 or scons:3 (scons defaults to 2 for backwards compatibility) instead of scons:py27. Antione likes this more.
- uniquefiles:dirs was not needed
- flavors was not needed
And also:
- Use PYTHON2_DEFAULT instead of hardcoded version numbers.
- Removed unused variable from scons.mk.
Comment 19 VVD 2020-03-31 01:10:17 UTC
Created attachment 212879 [details]
Udpate scons to 3.1.2 + remove hardcoded python version 2
Comment 20 Ronald Klop 2020-04-01 20:51:39 UTC
(In reply to VVD from comment #19)
I'm trying to build databases/mongodb36 with this scons-3.1.2. But it always tries to build scons with python3 while mongodb36 needs scons to run with python2.
Did you look into this? Is there a solution for this situation?
Comment 21 dewayne 2020-04-16 14:25:11 UTC
(In reply to VVD from comment #19)
Thanks VVD. I just updated my ports and attempted to rebuild our packages. 
Kept getting:
install  -m 0644 /usr/ports/devel/py-setuptools/files/easy-install.pth.dist /var/ports/usr/ports/devel/py-setuptools/work-py37/stage/usr/local/lib/python3.7/site-packages/
===> Creating unique files: Move MAN files needing SUFFIX
===> Creating unique files: Move files needing SUFFIX
Move: bin/easy_install --> bin/easy_install-3.7
Link: @bin/easy_install --> bin/easy_install-3.7
====> Compressing man pages (compress-man)
===>  Installing for py37-setuptools-44.0.0
===>  Checking if py37-setuptools is already installed
===>   py37-setuptools-44.0.0 is already installed
      You may wish to ``make deinstall'' and install this port again
      by ``make reinstall'' to upgrade it properly.
      If you really wish to overwrite the old port of py37-setuptools
      without deleting it first, set the variable "FORCE_PKG_REGISTER"
      in your environment or the "make install" command line.
*** Error code 1

make[1]: stopped in /usr/ports/devel/py-setuptools
*** Error code 1

make: stopped in /usr/ports/devel/scons

Applied your patch, and 
 make -C /usr/ports/devel/scons clean package
built an scons package on todays rebuilt 12.1S amd64 and i386. :)

The portmaster thread was:
devel/erlang-erlware_commons 55/363 >> devel/erlang-cf >> devel/rebar3 >> devel/git >> devel/p5-subversion >> devel/subversion >> www/serf >> devel/scons >> devel/py-setuptools@py27 (151/151)

So just when you think you're done, I tried: make -C /usr/ports/devel/subversion  clean package
===>  Building for serf-1.3.9_4
scons: Reading SConscript files ...
  File "/var/ports/usr/ports/www/serf/work/serf-1.3.9/SConstruct", line 189

    print 'Warning: Used unknown variables:', ', '.join(unknown.keys())


SyntaxError: invalid syntax

Hmmm, looks like a python issue, probably needs 2.7.  Successfully building ports is one measure, getting the successfully built port to function is another. ;)
Comment 22 VVD 2020-04-16 14:37:05 UTC
(In reply to dewayne from comment #21)
Just rebuild subversion without any issue.
Rebuilded several times git last weeks.

How to found your build errors?
Comment 23 VVD 2020-04-16 14:47:42 UTC
Ok, rebuild serf.

Check Ronald Klop's patch than.
Comment 24 dewayne 2020-04-17 01:52:20 UTC
After some hours on this, I've found that something in my make.conf has corrupted the situation. From my earlier post, it looks like scons will build with python 3.7, but it doesn't appear to run.  My apologies for distracting the discussion.
Comment 25 Ronald Klop 2020-04-18 19:03:04 UTC
To all cc's on this issue.

The difficulty is not to run scons with python 2 or 3. Since scons 3.x supports both python versions. See https://scons.org/tag/releases.html.

The difficulty is that the SConstruct files of other ports are written in python 2 (like mongodb36) or 3 (like upcoming mongodb42). Our current ports framework has no way to create both versions of scons.

We can't create ports using scons with python3 as long as there are ports needing scons with python2.

The first line of /usr/local/bin/scons says "#!/usr/local/bin/python2.7". That is the only line which need to be changed to run with python3. But we currently can't do that.

My solution would be to flavor the port and create a py27-scons and py37-scons. The pkg build infrastructure can than use the one it needs.
Comment 26 VVD 2020-04-20 23:39:39 UTC
Created attachment 213623 [details]
Update to 3.1.2

Just update it to 3.1.2 than.
Comment 27 Yasuhiro KIMURA 2020-04-30 02:41:06 UTC
(In reply to Ronald Klop from comment #25)

Python 2 has already reached EoL and will be removed from ports tree after the end of this year. And when python 2 has gone all ports that depends on python2-scons get broken anyway. So right way we should go is

* Switch devel/scons to use python 3.
* If it is possible fix ports that require python2-scons so they can be build with python3-scons.
* Otherwise mark them BROKEN and set EXPIRATION_DATE.

And first step is to create patch that make devel/scons to use python 3, do exp-run and see what/how-many ports are not compatible with python3-scons.
Comment 30 Sunpoet Po-Chuan Hsieh freebsd_committer 2020-05-25 19:11:39 UTC
flavored scons landed. Thanks for your patience.