Bug 233770 - QA lang/python37 as default python3 version
Summary: QA lang/python37 as default python3 version
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Many People
Assignee: freebsd-python mailing list
URL:
Keywords: needs-qa
Depends on: 233298
Blocks:
  Show dependency treegraph
 
Reported: 2018-12-04 10:22 UTC by Ruslan Makhmatkhanov
Modified: 2018-12-13 06:19 UTC (History)
5 users (show)

See Also:
koobs: exp-run+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ruslan Makhmatkhanov freebsd_committer 2018-12-04 10:22:10 UTC
Hello, can we please do an exp-run of python3 ports with python3.7 as default? 
Python3.7 has many exciting features and performance improvements, and I'm using it as default for almost year already with pydata stack and gnome3. 

I'm aware of only two ports that fails to work with python3 - accessibility/accerciser and current version science/py-sckit-learn. I have local patches for both of them, but we need to know all the stoppers. 

Thanks.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2018-12-04 11:46:00 UTC
I'm all for moving as quickly to using the latest and greatest in the Python ecosystem.

However, we need to keep in mind that:

a) The vast majority of Python ports, currently use loose and implicit rather than precise and explicit version specifiers. That is, either; 

  1) Under-specify/declare their version support, with a major class example being "X.Y+", without an upper bound, or
  2) Don't specify version support at all (bare USES=python), used to imprecisely mean something like 'all versions' or 'python 2 and 3'.

b) Python package support for the latest (usually 1-2) and greatest versions is a fairly slow moving target. It takes time and manual interventions upstream, for those later versions to enter the CI ecosystem/configuration into a 'tested by default' status.

c) With the vast majority of Python ports sitting without test suites, an exp-run can only tell us a limited amount about potential issues, and in particularly, not at run-time, in order to give us a good answer to 'is it a good time to switch'.

An exp-run should be a first step, and not the only step/data point, to deciding to move forwards on later versions being a default, no matter how great it would be to do so.

We should consider within that decision other issues such as, but not limited to:

- Support periods (if they differ) between releases
- Time / number of releases since releases
- How to improve Python port version specification / explicitness to make it easier to move default versions without introducing unknown/untestable regressions (until after the fact).

The question I believe we actually need to pose and ultimately answer (outside of the scope of this exp-run/issue) is, what (else) do we need to do to make moving default versions forward a trivial/non-event.

A large part of that question is whether we take an "exclusive allow" vs "exclusive deny" approach to version support in Python ports. Right now we're in 'exclusive deny mode' where most ports are loosely or under specified (incorrectly), until bugs/issues are reported.

@Ruslan Maybe its worth a few of us (python) getting together on IRC to flesh out a plan for this. It'd be great to make it easier for these things to move forward faster in the future too. Think: python default = 3.x, which is a superset of this problem.
Comment 2 Ruslan Makhmatkhanov freebsd_committer 2018-12-04 12:20:39 UTC
Hi Kubilay!

Yes, you are right, this is just the first step in a plan of moving to python3 as default and we already did that in past without much problem - for example switching from 3.5 to 3.6 was quite smooth AFAIR. 

Then we can do more broad exp-run with python 3.7 as default to identify ports that are tied to python 2.x and mark them as such explicitly. This will allow them to work after switching the tree to 3.7 as default.

I can't say how slowly the transition to 3.x is going, but ~1100 ports installed in my system are python3 capable, except the exaile and deluge. And moreover, the current trend for upstream is to drop python 2.x support altogether like we saw it with gnome3, gajim, jupyter-notebook and scikit-learn to name of few. 

While this should work, I'm all for discussing how to simplify the future upgrades. We may chat at weekend if there are no objections.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2018-12-04 12:23:43 UTC
(In reply to Ruslan Makhmatkhanov from comment #2)

Sounds good :)
Comment 4 Antoine Brodin freebsd_committer 2018-12-12 20:22:52 UTC
New failures on 11.2 i386:

+ {"origin"=>"accessibility/accerciser", "phase"=>"package", "errortype"=>"???"}
+ {"origin"=>"archivers/py-attic", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"astro/py-astropy", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"biology/py-biom-format", "flavor"=>"py37", "phase"=>"build", "errortype"=>"compiler_error"}
+ {"origin"=>"databases/py-pymssql", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-datrie", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-efl", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-intbitset", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-jcc", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-llfuse", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-pymtbl", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-python-gitlab", "flavor"=>"py37", "phase"=>"build", "errortype"=>"???"}
+ {"origin"=>"devel/py-sarge", "flavor"=>"py37", "phase"=>"configure", "errortype"=>"???"}
+ {"origin"=>"devel/py-statgrab", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/pysvn", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"graphics/py-PyOpenGL-accelerate", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"graphics/py-pyproj", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"math/py-PyMetis", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"math/py-levmar", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"math/py-symcxx", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"net/libproxy-python", "flavor"=>"py37", "phase"=>"stage", "errortype"=>"???"}
+ {"origin"=>"net/py-pynmsg", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"net/py-pypcap", "flavor"=>"py37", "phase"=>"build", "errortype"=>"compiler_error"}
+ {"origin"=>"science/py-mlpy", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"science/py-scikit-learn", "flavor"=>"py37", "phase"=>"build", "errortype"=>"linker_error"}
+ {"origin"=>"security/morphis", "phase"=>"build", "errortype"=>"???"}
+ {"origin"=>"textproc/py-pyctpp2", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"www/py-http-parser", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"www/py-url", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}


New failure logs on 11.2 i386:

http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/accerciser-3.22.0.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-attic-0.16_1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-astropy-1.1.2_1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-biom-format-2.1.7.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-pymssql-2.1.3.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-datrie-0.7.1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-efl-1.20.0.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-intbitset-2.3.0_1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-jcc-3.0.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-llfuse-1.2_2.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-pymtbl-0.4.1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-python-gitlab-1.6.0.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-sarge-0.1.4.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-statgrab-0.7.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/pysvn-py37-1.9.3.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-PyOpenGL-accelerate-3.1.0_1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-pyproj-1.9.5.1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-PyMetis-2016.2.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-levmar-0.2.2.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-symcxx-0.1.9_1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-libproxy-python-0.4.15.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-pynmsg-0.4.0.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-pypcap-1.2.2.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-mlpy-3.5.0_6.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-scikit-learn-0.20.1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/morphis-0.8.18.6_1.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-pyctpp2-0.12.2_4.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-http-parser-0.8.3.log
http://package23.nyi.freebsd.org/data/112i386-default-PR233770-804-828/2018-12-11_12h24m58s/logs/errors/py37-url-0.4.2.log
Comment 5 commit-hook freebsd_committer 2018-12-12 21:57:55 UTC
A commit references this bug:

Author: rm
Date: Wed Dec 12 21:57:06 UTC 2018
New revision: 487327
URL: https://svnweb.freebsd.org/changeset/ports/487327

Log:
  accessibility/accerciser: fix packaging with python 3.7

  Add upstream patch to fix packaging/runtime with python 3.7

  PR:		233770
  With hat:	gnome

Changes:
  head/accessibility/accerciser/files/
  head/accessibility/accerciser/files/patch-src_lib_accerciser-accerciser.py
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2018-12-13 03:07:14 UTC
Will re-assign when if/when next exp-run required

Thank you Antoine!
Comment 7 Antoine Brodin freebsd_committer 2018-12-13 06:19:06 UTC
New failures on 11.2 amd64:

+ {"origin"=>"accessibility/accerciser", "phase"=>"package", "errortype"=>"???"}
+ {"origin"=>"archivers/py-attic", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"astro/py-astropy", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"biology/py-biom-format", "flavor"=>"py37", "phase"=>"build", "errortype"=>"compiler_error"}
+ {"origin"=>"databases/py-pymssql", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-datrie", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-efl", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-intbitset", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-jcc", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-llfuse", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-pymtbl", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/py-python-gitlab", "flavor"=>"py37", "phase"=>"build", "errortype"=>"???"}
+ {"origin"=>"devel/py-sarge", "flavor"=>"py37", "phase"=>"configure", "errortype"=>"???"}
+ {"origin"=>"devel/py-statgrab", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"devel/pysvn", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"graphics/py-PyOpenGL-accelerate", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"graphics/py-pyproj", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"math/py-PyMetis", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"math/py-levmar", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"math/py-symcxx", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"net/libproxy-python", "flavor"=>"py37", "phase"=>"stage", "errortype"=>"???"}
+ {"origin"=>"net/py-pynmsg", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"net/py-pypcap", "flavor"=>"py37", "phase"=>"build", "errortype"=>"compiler_error"}
+ {"origin"=>"science/py-mlpy", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"science/py-scikit-learn", "flavor"=>"py37", "phase"=>"build", "errortype"=>"linker_error"}
+ {"origin"=>"security/morphis", "phase"=>"build", "errortype"=>"???"}
+ {"origin"=>"textproc/py-pyctpp2", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"www/py-http-parser", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}
+ {"origin"=>"www/py-url", "flavor"=>"py37", "phase"=>"build", "errortype"=>"clang"}