Bug 204519 - Mk/bsd.default-versions.mk: Set Python 3.5 as the default 3.x version
Summary: Mk/bsd.default-versions.mk: Set Python 3.5 as the default 3.x version
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Ruslan Makhmatkhanov
URL:
Keywords: needs-patch, needs-qa
Depends on: 207752 207754 207761 207771 213342 213393
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-13 10:30 UTC by Gerard Seibert
Modified: 2016-10-14 19:53 UTC (History)
6 users (show)

See Also:
koobs: exp-run?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gerard Seibert 2015-11-13 10:30:24 UTC
I would like to request that python 3.5 be set as the default 3,X version. It was released on 09/13/2015 and is apparently quite stable.

Thank You :)
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-13 10:33:44 UTC
I'll take this (as I requested the issue be created)
Comment 2 Mathieu Arnold freebsd_committer 2015-11-13 11:06:56 UTC
it should be given to portmgr for exp-run, why take it back ?
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-13 11:09:20 UTC
I can request an exp-run whilst being 'current responsible' (assignee)

Having said that, if you want to create a patch and take it for an exp-run, by all means :)
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-13 11:10:23 UTC
Also note, it's still within the Ports Framework component, and the change technically doesn't require portmgr approval. But certainly an exp-run is warranted for this kind of change
Comment 5 Mathieu Arnold freebsd_committer 2015-11-13 11:12:19 UTC
(In reply to Kubilay Kocak from comment #3)
> I can request an exp-run whilst being 'current responsible' (assignee)
> 
> Having said that, if you want to create a patch and take it for an exp-run,
> by all means :)

When you want an exp-run, you assign the pr to portmgr, it'll get back to you when the exp-run is done.
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-13 11:18:54 UTC
OK, portmgr will handle this one.
Comment 7 Andrew Berg 2015-11-13 12:29:13 UTC
Making the default 3.5 will break many things because of limitations in Poudriere and/or ports. Currently, Poudriere cannot calculate dependencies of dependencies correctly for Python. Because of the isolation that Poudriere enforces, it can build dependencies for the *default* version of Python rather than what is needed to satisfy the dependency. Samba and Salt are good examples - go try to build packages for them using Poudriere with 3.x set as the default Python.

IIRC, Baptiste is trying to solve issues like this with subpackages, where a port can then depend on a specific subpackage which is built with specific options rather than a port, which can have any arbitrary options set.
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-13 14:02:00 UTC
@Andrew, is this a change (better/worse/same) in behaviour from 3.4 being the current 3.x default?
Comment 9 Andrew Berg 2015-11-13 14:14:12 UTC
(In reply to Kubilay Kocak from comment #8)
It's the same. A port wants the py27-foo package because it needs 2.7, but the py34-foo package was built instead because 3.4 was the default and foo supports both 2.x and 3.x. You'd just get py35-foo instead.
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-13 14:17:18 UTC
So that would leave us in the net same position, except with 3.5 as the default version (which is the upstream recommendation: use the latest point release of a major version branch).

Any poudriere/pkg/framework issues need to be resolved independently (im with you, this would be really nice) and does not block this issue .
Comment 11 Antoine Brodin freebsd_committer 2015-11-15 10:02:15 UTC
Take for exp-run.

Some ports are likely not ready because of the PYO removal (PEP 0488)

- https://reviews.freebsd.org/D4147

- lines 113-115 of Mk/bsd.kde4.mk
- x11/kdelibs4/files/patch-cmake_modules_PythonMacros.cmake

- accessibility/accerciser/pkg-plist
- accessibility/orca/pkg-plist
- accessibility/py3-atspi/pkg-plist
- audio/gnome-music/pkg-plist
- audio/rhythmbox/pkg-plist
- deskutils/alacarte/pkg-plist (this one installs versioned files in an unversioned lib/python directory)
- devel/gitg/pkg-plist
- devel/py-ice/pkg-plist
- devel/py3-gobject3
- dns/bundy/pkg-plist
- editors/gedit-plugins/pkg-plist
- graphics/eog-plugins/pkg-plist
- graphics/py-opencv/pkg-plist
- math/rpcalc/pkg-plist
- math/convertall/pkg-plist
- multimedia/py3-gstreamer1/pkg-plist
- sysutils/dvdvideo/pkg-plist
- sysutils/backupchecker/pkg-plist
- x11/xcb-proto/pkg-plist

I tried to run a few tests but the version of pytest in the tree doesn't work with python 3.5
Comment 12 Antoine Brodin freebsd_committer 2015-11-16 12:57:43 UTC
Exp-run results:

http://package22.nyi.freebsd.org/build.html?mastername=101i386-default-PR204519&build=2015-11-15_00h52m49s

4 new failures,  but it's only the tip of the iceberg,  see comment #11

+ {"origin"=>"audio/gnome-music", "pkgname"=>"gnome-music-3.16.2", "phase"=>"package", "errortype"=>"???"}
+ {"origin"=>"graphics/py3-cairo", "pkgname"=>"py35-cairo-1.10.0_3", "phase"=>"configure", "errortype"=>"configure_error"}
+ {"origin"=>"sysutils/backupchecker", "pkgname"=>"backupchecker-1.7", "phase"=>"package", "errortype"=>"???"}
+ {"origin"=>"sysutils/dvdvideo", "pkgname"=>"dvdvideo-20130117_2", "phase"=>"package", "errortype"=>"???"}

23 new ports skipped due to those 4 failures

Failure logs:

http://package22.nyi.freebsd.org/data/101i386-default-PR204519/2015-11-15_00h52m49s/logs/errors/gnome-music-3.16.2.log
http://package22.nyi.freebsd.org/data/101i386-default-PR204519/2015-11-15_00h52m49s/logs/errors/py35-cairo-1.10.0_3.log
http://package22.nyi.freebsd.org/data/101i386-default-PR204519/2015-11-15_00h52m49s/logs/errors/backupchecker-1.7.log
http://package22.nyi.freebsd.org/data/101i386-default-PR204519/2015-11-15_00h52m49s/logs/errors/dvdvideo-20130117_2.log

For the py3-cairo failure,  waflib/Build.py part of http://sources.debian.net/patches/patch/py3cairo/1.10.0%2Bdfsg-5/81_pickling-again.patch/ fixes it,  but I couldn't run the regression tests  (pytest in the tree has run time failures with python 3.5)
Comment 13 Ruslan Makhmatkhanov freebsd_committer 2015-11-19 15:27:28 UTC
(In reply to Antoine Brodin from comment #11)

- deskutils/alacarte/pkg-plist (this one installs versioned files in an unversioned lib/python directory)


alacarte is broken anyway. I got this why running it:

[rm@smsh-zfs ~]> alacarte
Traceback (most recent call last):
  File "/usr/local/bin/alacarte", line 21, in <module>
    from Alacarte.MainWindow import main
ImportError: No module named 'Alacarte'

I have a fix to make it build with python3 and to install it into correct directory layout, but it still fails to run:

[rm@smsh-zfs ~]> alacarte 
/usr/local/lib/python3.5/site-packages/Alacarte/MainWindow.py:43: Warning: invalid (NULL) pointer instance
  self.tree.add_from_file(os.path.join(config.pkgdatadir, 'alacarte.ui'))
/usr/local/lib/python3.5/site-packages/Alacarte/MainWindow.py:43: Warning: g_signal_connect_object: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
  self.tree.add_from_file(os.path.join(config.pkgdatadir, 'alacarte.ui'))
/usr/local/lib/python3.5/site-packages/Alacarte/MainWindow.py:43: Warning: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
  self.tree.add_from_file(os.path.join(config.pkgdatadir, 'alacarte.ui'))
Segmentation fault (core dumped)

So I believe you may just mark it uncoditionally broken and exclude from exp-run.
Comment 14 Ruslan Makhmatkhanov freebsd_committer 2015-11-19 16:28:04 UTC
Here is standalone chunk. Just run this in python3.5 interpreter:

>>> import gi
>>> gi.require_version('Gtk', '3.0')
>>> from gi.repository import Gtk
>>> 
>>> tree = Gtk.Builder()
>>> tree.add_from_file('/usr/local/share/alacarte/alacarte.ui')
__main__:1: Warning: invalid (NULL) pointer instance
__main__:1: Warning: g_signal_connect_object: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
__main__:1: Warning: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
Segmentation fault (core dumped)
Comment 15 commit-hook freebsd_committer 2016-03-06 22:10:08 UTC
A commit references this bug:

Author: rm
Date: Sun Mar  6 22:09:24 UTC 2016
New revision: 410491
URL: https://svnweb.freebsd.org/changeset/ports/410491

Log:
  multimedia/py3-gstreamer1: fix packaging with python 3.5

  PR:     204519
  With hat:   gnome

Changes:
  head/multimedia/py3-gstreamer1/Makefile
  head/multimedia/py3-gstreamer1/pkg-plist
Comment 16 commit-hook freebsd_committer 2016-03-06 22:13:11 UTC
A commit references this bug:

Author: rm
Date: Sun Mar  6 22:12:29 UTC 2016
New revision: 410493
URL: https://svnweb.freebsd.org/changeset/ports/410493

Log:
  audio/gnome-music: fix packaging with python 3.5

  PR:		204519
  With hat:   gnome

Changes:
  head/audio/gnome-music/Makefile
  head/audio/gnome-music/pkg-plist
Comment 17 Gerard Seibert 2016-03-07 11:40:01 UTC
I just noticed that Python 3.5.1 was released on 2015-12-07, which fixed a number of issues: <https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-1-final>
Comment 18 Ruslan Makhmatkhanov freebsd_committer 2016-03-07 11:42:01 UTC
(In reply to Gerard Seibert from comment #17)
Version 3.5.1 already in ports tree since 20 Dec 2015
Comment 19 Ruslan Makhmatkhanov freebsd_committer 2016-05-06 11:53:59 UTC
Antoine, would you please restart the exp-run with fresh tree? There were many ports fixed so far.
Comment 20 Antoine Brodin freebsd_committer 2016-05-06 12:21:15 UTC
Some things that probably still need fixes:


- lines 115-117 of Mk/bsd.kde4.mk
- x11/kdelibs4/files/patch-cmake_modules_PythonMacros.cmake

- accessibility/py3-speech-dispatcher
- audio/rhythmbox
- graphics/py-opencv
- math/rpcalc
- math/convertall
Comment 21 Antoine Brodin freebsd_committer 2016-05-06 12:33:23 UTC
Also graphics/blender has USES=python:3.4 , I don't know if it can be changed to USES=python:3.4+
Comment 22 Shane 2016-05-07 02:55:28 UTC
(In reply to Antoine Brodin from comment #21)

No, each blender release is fairly strongly tied to and only supported when built with a specific version of python, no effort is made to work with more than one python version.

There has been a new version released that isn't in ports yet, v2.77 that will use python 3.5
Comment 24 Antoine Brodin freebsd_committer 2016-05-08 09:38:38 UTC
With default python3 version changed and default python version changed there are those failures (py-opencv may have been broken before):

http://package23.nyi.freebsd.org/data/101i386-default-PR205419-py3/2016-05-08_05h53m28s/logs/errors/py35-opencv-2.4.9_3.log
http://package23.nyi.freebsd.org/data/101i386-default-PR205419-py3/2016-05-08_05h53m28s/logs/errors/convertall-0.6.0.log
http://package23.nyi.freebsd.org/data/101i386-default-PR205419-py3/2016-05-08_05h53m28s/logs/errors/rpcalc-0.7.0_1.log

I can't test python kde4 ports as some dependencies fail to build with python3 (sysutils/qzeitgeist for instance)
Comment 25 commit-hook freebsd_committer 2016-09-23 22:46:00 UTC
A commit references this bug:

Author: rm
Date: Fri Sep 23 22:45:03 UTC 2016
New revision: 422698
URL: https://svnweb.freebsd.org/changeset/ports/422698

Log:
  graphics/py-opencv: limit python version to 2.x

  Python module of OpenCV 2.4.9 is not compatible with Python 3.x
  - both on cmake infrastructure level and on module itself level,
  so just mark it as Python 2.x only and remove all the python3
  shims - they are don't make any difference anyway.
  We can patch that hardly to make it work, but it's better to just
  update to latest version that have python3 support out of the box.

  PR:		204519 (for tracking)

Changes:
  head/graphics/opencv/Makefile
  head/graphics/py-opencv/Makefile
  head/graphics/py-opencv/pkg-plist
Comment 26 commit-hook freebsd_committer 2016-10-01 15:37:15 UTC
A commit references this bug:

Author: jbeich
Date: Sat Oct  1 15:36:15 UTC 2016
New revision: 423072
URL: https://svnweb.freebsd.org/changeset/ports/423072

Log:
  graphics/py-opencv: drop python3 vestige after r422698

  PYTHON_REL is defined in bsd.port.pre.mk, so this unlikely to ever have
  worked unless defined via make.conf. Note, USES=python only supports
  overriding PYTHON_VERSION apart from DEFAULT_VERSIONS=python*.

    $ make -V PYTHON_VERSION PYTHON_REL=3500
    python2.7

    $ PYTHON_REL=3500 make -V PYTHON_REL
    2712

    $ make -V PYTHON_REL PYTHON_VERSION=python3.5
    3502

  PR:		204519 (for tracking)

Changes:
  head/graphics/opencv/Makefile
Comment 27 Ruslan Makhmatkhanov freebsd_committer 2016-10-02 17:52:30 UTC
Antoine, I reread what you said and it looks we doing something different: I mean sysutils/qzeitgeist stuff. qzeitgeist needs python for build and there is no particular python version set in Makefile, so it should build just fine with default python version (2.7). 

As far I understand, you also changed "python" version in DEFAULT_VERSIONS, so your poudriere make.conf looks like that:

DEFAULT_VERSIONS=python=3.5 python3=3.5

while it should looked like that

DEFAULT_VERSIONS=python=2.7 python2=2.7 python3=3.5

because now I only interested in ports, that define python:3 or python:3.3+ in USES. I don't count that every port that have USES=python can be built with 3.5 or with 3.x ever. We now only want to change default python3 (3.4 -> 3.5) version, not the python version at all. 

Would you please limit your port list for testing to only those that set:
USES=python3
USES=python3.3+
USES=python:3,build
USES=python:3,run
USES=python:3.3+,build
USES=python:3.3+,run

and let me know what's broken with 3.5 now? Thanks a lot!
Comment 29 Ruslan Makhmatkhanov freebsd_committer 2016-10-02 19:27:15 UTC
Yep, and math/convertall and math/rpcalc too. kde/cmake stuff seems to be fixed, but they are out of interest at the moment. So only four ports stop the 3.4 -> 3.5 switching?
Comment 30 Antoine Brodin freebsd_committer 2016-10-02 19:33:12 UTC
(In reply to Ruslan Makhmatkhanov from comment #29)
Probably, I will do the exp-run again when accessibility/py3-speech-dispatcher and audio/rhythmbox are fixed.
Comment 31 Ruslan Makhmatkhanov freebsd_committer 2016-10-12 14:48:13 UTC
(In reply to Antoine Brodin from comment #30)
Antoine, I made the patches for them (see the "Depends on" PR's) and awaiting for maintainer approval. But maybe you can start the exp-run by applying them manually to the test tree because it may took up to 2 weeks awaiting approval? Thanks
Comment 32 Mathieu Arnold freebsd_committer 2016-10-12 15:56:00 UTC
You don't need to wait to commit them, they are fixing build-time problems when building with python 3.5, and thus, are covered by the blanket approval:

https://www.freebsd.org/doc/en/books/porters-handbook/makefile-maintainer.html
Comment 33 Ruslan Makhmatkhanov freebsd_committer 2016-10-12 16:05:33 UTC
Ok, will do then. I also can't test math/convertall and math/rpcalc, because they depend upon llvm37, that's marked broken with py3.
Comment 34 Ruslan Makhmatkhanov freebsd_committer 2016-10-12 19:10:45 UTC
All set
Comment 35 commit-hook freebsd_committer 2016-10-14 19:50:44 UTC
A commit references this bug:

Author: antoine
Date: Fri Oct 14 19:50:00 UTC 2016
New revision: 423986
URL: https://svnweb.freebsd.org/changeset/ports/423986

Log:
  Change the default version of python3 from 3.4 to 3.5
  Thanks to Ruslan Makhmatkhanov for doing all the fixes

  PR:		204519
  With hat:	portmgr

Changes:
  head/Mk/Uses/python.mk
  head/Mk/bsd.default-versions.mk
  head/UPDATING