Bug 213636 - math/py-matplotlib: Fails to build with python3
Summary: math/py-matplotlib: Fails to build with python3
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Tobias Kortkamp
Keywords: needs-qa
: 214853 (view as bug list)
Depends on:
Reported: 2016-10-20 04:57 UTC by Loïc Bartoletti
Modified: 2017-12-06 19:47 UTC (History)
9 users (show)

See Also:
bugzilla: maintainer-feedback? (mainland)

svndiff Makefile working with py27 and py35 with Tk_Agg option (414 bytes, patch)
2016-12-31 18:24 UTC, c.brinkhaus
no flags Details | Diff
Patch to make the port work with both Python 2 and 3. (1.57 KB, patch)
2017-08-06 21:18 UTC, rsmith
no flags Details | Diff
Updated patch using option helpers and just using bsd.port.mk. (1.29 KB, patch)
2017-08-12 23:13 UTC, rsmith
no flags Details | Diff
Updated patch; pet portlint (1.33 KB, patch)
2017-08-14 05:13 UTC, rsmith
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Loïc Bartoletti freebsd_committer 2016-10-20 04:57:15 UTC
Build fails, when you enable python3 as python's default versions with default config for py-matplotlib:
     EXAMPLES=on: Build and/or install examples
     GTKAGGBACKEND=on: GTKAgg backend support
     GTKBACKEND=on: GTK backend support
     QT4AGGBACKEND=off: Qt4Agg backend support
     TKAGGBACKEND=on: TKAgg backend support
     WXAGGBACKEND=off: WXAgg backend support


                macosx: no  [Mac OS-X only]
                qt5agg: no  [PyQt5 not found]
                qt4agg: no  [skipping due to configuration]
               gtk3agg: no  [Requires pygobject to be installed.]
             gtk3cairo: no  [Requires cairocffi or pycairo to be installed.]
                gtkagg: no  [Requires pygtk]
                 tkagg: yes [installing, version not identified]
                 wxagg: no  [skipping due to configuration]
                   gtk: no  [Requires pygtk]
                   agg: yes [installing]
                 cairo: no  [cairocffi or pycairo not found]
             windowing: no  [Microsoft Windows only]

                dvipng: no
           ghostscript: no
                 latex: no
               pdftops: no

                  dlls: no  [skipping due to configuration]

                        * The following required packages can not be built:
                        * gtkagg, gtk

Builds success with GTKAGGBACKEND=off and GTKBACKEND=off
Comment 1 Kurt Jaeger freebsd_committer 2016-11-23 07:23:39 UTC
Port was set to python:2 with r426883 from PR#214600.

So a patch is needed to allow build with python3.
Comment 2 c.brinkhaus 2016-12-31 18:24:37 UTC
Created attachment 178422 [details]
svndiff Makefile working with py27 and py35 with Tk_Agg option

First thank you for the report. I have tried a modified Makefile as attached and only the reduced sets of options. The default options in make.conf are DOC, NLS and EXAMPLES unset. For one poudriere set I have additionally a py-make.conf as
DEFAULT_VERSIONS=python=3.5 python2=2.7 python3=3.5

With the setup poudriere testport is happy with the normal python2.7. A poudriere testport with python3.5 configured as default looks fine as well.

This is no solution to fix the default options with python3. But it is something which might be better than nothing and help the transition from python2 to python3 for some projects.
Comment 3 c.brinkhaus 2016-12-31 18:31:23 UTC
The report seems to be similar to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214853.
Comment 4 rsmith 2017-01-02 19:01:28 UTC
*** Bug 214853 has been marked as a duplicate of this bug. ***
Comment 5 rsmith 2017-01-02 19:09:01 UTC
I've tried adjusting USES based on the Python version used (see bug 214853), but I couldn't get that to work.

An option that should work would be to create a py3-matplotlib port that USES python:3+ and has the TKAGGBACKEND (on by default) option but not the GTKBACKEND or GTKAGGBACKEND options.
I'm not sure about the QT?AGGBACKEND and  WXAGGBACKEND; I've never used them.

Would that be an acceptable solution?
Comment 6 c.brinkhaus 2017-01-02 20:32:39 UTC
(In reply to rsmith from comment #5)
Dear rsmith,
first I like to thank you for your report which enabled me to run py-matplotlib on python3.5. About the way to proceed I am not sure. There are already some py**-tkinter ports, but I am sure that the aim is to have similar ports for python2.* and python3.*. It is not easy and likely exceeds my knowledge. May be things will change once the default python version has changed, which is not as simple as it sounds, too.
Kind regards,
Comment 7 Kurt Jaeger freebsd_committer 2017-07-30 13:00:05 UTC
version 2.0.2 was released in may and is supposed to support py3.
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-01 06:27:51 UTC
py3-ports are currently (and recently) only allowed with portmgr approval under the following conditions:

- Port must be named category/py3-foo
- Port must be a *slave* port of an existing category/py-foo port
- Port must be *both* Python 2/3 compatible (USES=python), not just 3.4+, etc

Accordingly, I'd suggest making this issue about updating to the version mentioned in comment 7

Producing a Python 3 version of the package in the official package repositories will be handled shortly with a special version of poudriere that automatically produces py3-* ports without them needing to exist in the tree.

Having said that, *if* the current version spec (python:2.7) is *incorrect*, in that the current (port) version *does* support 3.x, then attachment 178422 [details] should land, as the present version spec is actually incorrect (independent of whether packages can be produced from it by poudriere or not).

Please advise whether the above is the case or not.
Comment 9 rsmith 2017-08-03 18:03:56 UTC
(In reply to Kubilay Kocak from comment #8)

It depends on what you consider "work"?

If the Python 2.7 restriction is removed and *only* the TKAGGBACKEND option is used, then the port builds and works with Python 3.5 and 3.6.

Hovewer, the default configuration (which also has GTKBACKEND and GTKAGGBACKEND options enabled does not build).
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-05 05:06:10 UTC
(In reply to rsmith from comment #9)

The word 'work' was specifically and intentionally not used, to focus just on whether or not the USES=python:<version-spec> is incorrect as it relates to what matplotlib is supposed to support (per upstream), not whether other factors (default options or other issues/bugs) prevent that from being true at a certain point in time or in certain conditions (downstream).

Keep in mind that USES=python:<version-spec> is a declarative specifier (port supports this/these versions), not an imperative one (use this version)

Looking back, ports r426883 (update to 1.5.3) states:

- Additionally, 1.5.3 version doesn't build with python-3.X because
  'import gtk' which the build tries fails in python3. Therefore, python:2.

Based on this and on my reading of this issue (which may be incorrect or incomplete), this was and is an incorrect (at a minimum, imprecise) warrant and/or method for restricting builds to 2.7.

1) If the presence (or non-presence) or a build configuration (OPTION or OPTIONs in this case) prevents building with a particular version, then the restriction should be limited/scoped to that build option or build options, not globally.

2) If a python package ('upstream python package' is meant here, not 'freebsd package') supports Python 2 and 3 builds, then the default configuration (read: OPTIONS_DEFAULTS) of the port and it's dependencies should support building with 2/3 by default.

Accordingly, and at a minimum:

1) USES=python:<version-spec> should be relaxed and be made to correctly reflect that matplotlib supports both Python 2 & 3. 

2) The OPTIONS_DEFAULTS of this port should be modified to work with Python 2 and 3 builds (if that is indeed possible).

(1) and (2) alone are probably insufficient to ensure a good user experience, if certain combinations of OPTIONS must be enabled/disabled to produce a valid Python 3 build environment, so:

If certain options are mutually exclusive (in that some prevent Python 2 or 3 builds), then OPTIONS_RADIO's or conditional blocks should be added that check for valid/invalid options combinations and inform the user with appropriate BROKEN/IGNORE/other methods when incompatible options/configurations are selected.
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-05 05:11:59 UTC
To explicitly separate the issues here:

1) Does matplotlib 1.5.3 support (bugs and build environments aside) Python 3.x? If so, USES=python:<version-spec> should reflect that (upstream intended) version support as closely as possible without being incorrect.

2) Does this ports OPTIONS_DEFAULT (or its dependencies OPTIONS_DEFAULT), support building with both Python 2 and 3. If so, there's nothing to do. If not, OPTIONS_DEFAULTS should change to make that the case.

3) If (2) results in changes, what impacts does this have and what other changes may be required in other ports/packages. There could be conflicting dependencies from other ports and/or their dependencies, users could be forced to run multiple frontend frameworks, etc.

(1) can and should be resolved independently from (2) potentially (3)
Comment 12 rsmith 2017-08-06 13:40:02 UTC
(In reply to Kubilay Kocak from comment #11)

1) Yes.

2) No. As far as I know, OPTIONS_DEFAULT should only include TKAGGBACKEND for it to work with both Python 2 and 3, since GTKBACKEND and GTKAGGBACKEND do not work with Python 3.

3) Looking at Freshports for the active ports that require this port, few of them seem to *require* a specific GUI toolkit to be enabled when looking at their Makefiles. Only math/cadabra2 uses gtkmm30 and gdkpixbuf2, which *might* imply a need for GTKBACKEND or GTKAGGBACKEND. But this port is limited to Python 2.

Would it be allowed to set different OPTIONS_DEFAULT depending on the Python version? Set it conditionally in the main port and override it in a py3 slave port?
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-06 14:08:12 UTC
(In reply to rsmith from comment #12)

It's possible (and I believe there are existing cases in the tree), but that may or may not be the correct, right or best way to address the issue, say versus slave ports (port per backend or similar, or other methods.

Other options to consider:

- Scope/set USES=python:<version-spec> per OPTION rather than un-conditionally
- Scope/set OPTIONS_DEFINE based on python version (only adding them for relevant versions)

I haven't researched this to the point where I know the entire matrix of backends and their dependencies, so I can't comment too much on which of the above (or combination of the above) would best suit, but the port (and ports in general) should be as declarative as possible with regard to options, dependencies and their relationships.

On py3-* ports, they are temporary workarounds so they cant be considered as an ultimate/final solution to this "backend option mutual exclusion of python support" issue.
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-06 14:15:11 UTC
(In reply to Kubilay Kocak from comment #13)

And independently to all that, the current (global) version-spec is still incorrect, see comment 11 point (1)
Comment 15 rsmith 2017-08-06 21:18:41 UTC
Created attachment 185105 [details]
Patch to make the port work with both Python 2 and 3.

With regard to Koobs' comments, I'd like to propose this patch.

For both Python versions it sets the TKAgg backend as default.
For Python2 it adds the GTK and GTKAgg backends.

Furthermore it warns that the Python 3 port is broken with either GTKBACKEND ot GTKAGGBACKEND.
Comment 16 Richard Gallamore freebsd_committer 2017-08-12 09:17:50 UTC
Some things about this patch.

* Instead of using broken with an if, use option helpers. GTKBACKEND_BROKEN= This is broken.
* Remove the bsd.port.options.mk. This is not needed.
* I don't know why the .pre/post.mk files are being added, why is this?
Comment 17 rsmith 2017-08-12 23:10:19 UTC
(In reply to Richard Gallamore from comment #16)

The usage info in /usr/ports/Mk/bsd.port.options.mk led me to believe that using the three different includes was necessary. I'll update the patch.
Comment 18 rsmith 2017-08-12 23:13:01 UTC
Created attachment 185337 [details]
Updated patch using option helpers and just using bsd.port.mk.
Comment 19 Kurt Jaeger freebsd_committer 2017-08-13 09:02:21 UTC
Portlint is unhappy:

make: "/home/pi/m/math/py-matplotlib/Makefile" line 36: Malformed conditional (${PYTHON_MAJOR_VER} == 2)
make: "/home/pi/m/math/py-matplotlib/Makefile" line 72: Malformed conditional (${PYTHON_MAJOR_VER} == 3)
make: Fatal errors encountered -- cannot continue
FATAL ERROR: make(1) died with status 256 and returned '
make: stopped in /home/pi/m/math/py-matplotlib' at /usr/local/bin/portlint line 3471.
Comment 20 rsmith 2017-08-14 05:13:23 UTC
Created attachment 185389 [details]
Updated patch; pet portlint

Reworked the patch to fix portlint FATALs.

Used bsd.port.pre.mk/bsd.port.post.mk to get ".if ${PYTHON_MAJOR_VER} == 3" to work.

This means I had to drop the modification of OPTIONS_DEFAULT based on the Python version. Portlint complains if I try to expand the OPTIONS_DEFAULT within the bsd.port.pre.mk/bsd.port.post.mk block. It also complains if I put bsd.port.pre.mk before the options.
Comment 21 Tobias Kortkamp freebsd_committer 2017-12-05 10:54:57 UTC
(In reply to rsmith from comment #20)
I put a FLAVOR based version of the patch up for review here:
Comment 22 commit-hook freebsd_committer 2017-12-06 19:47:36 UTC
A commit references this bug:

Author: tobik
Date: Wed Dec  6 19:46:53 UTC 2017
New revision: 455676
URL: https://svnweb.freebsd.org/changeset/ports/455676

  math/py-matplotlib: Allow build for Python 3.x

  - x11-toolkits/{py-gtk2,py-wxPython28} do not support Python 3.x, so
    exclude the GTKBACKEND, GTKAGGBACKEND, and WXAGGBACKEND in that case.

  PR:		213636
  Reported by:	lbartoletti@tuxfamily.org
  Submitted by:	rsmith@xs4all.nl (based on)
  Reviewed by:	mat
  Approved by:	maintainer timeout
  Differential Revision:	https://reviews.freebsd.org/D13377