Bug 224024 - DEFAULT_VERSIONS for python and PYTHON_VERSION broken after r455210 (FLAVORS)
Summary: DEFAULT_VERSIONS for python and PYTHON_VERSION broken after r455210 (FLAVORS)
Status: Closed Works As Intended
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Port Management Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-12-01 18:43 UTC by John W. O'Brien
Modified: 2019-01-16 15:14 UTC (History)
7 users (show)

See Also:


Attachments
patch, python flavor uses DEFAULT_VERSIONS += python= (633 bytes, patch)
2017-12-03 11:38 UTC, yamayan
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description John W. O'Brien 2017-12-01 18:43:34 UTC
My goal

I want to build and manage packages in the spirit of PEP 394, so that I can maintain concurrent installs of ports for any combination of supported python versions, one of which may be for the default python version, one of which may be for the default python2 version, and one of which may be for the default python3 version. In the past, I have accomplished this by setting DEFAULT_VERSIONS in /usr/local/etc/poudriere.d/make.conf, and then setting PYTHON_VERSION in each of /usr/local/etc/poudriere.d/pythonXY-make.conf for XY in {27,34,35,36}. This yields four different repositories with packages that can, generally speaking, be installed concurrently.

Problem

This technique stopped working after FLAVORS were introduced in r455210.

To demonstrate

% cat /etc/make.conf
DEFAULT_VERSIONS= python=3.6 python2=2.7 python3=3.6
% cd /usr/ports/devel/py-pandas
% make -V FLAVOR
% make PYTHON_VERSION=python3.4 -V FLAVOR


Expected behavior

% make -V FLAVOR
py36
% make PYTHON_VERSION=python3.4 -V FLAVOR
py34


Observed behavior

% make -V FLAVOR
py27
% make PYTHON_VERSION=python3.4 -V FLAVOR
py27
Comment 1 Charlie Li 2017-12-01 19:27:18 UTC
In bsd.default-versions.mk, ${lang}_DEFAULT is deprecated, so that should be broken.

mat@ explained in his email to freebsd-ports@ announcing FLAVORS how to pass the arguments:
> For other people, to build the Python 3.6 version of, for example,
> databases/py-gdbm, you need to run:
>   
>  # make FLAVOR=py36 install

In conclusion, I believe the current behaviour you are experiencing is the correct behaviour.
Comment 2 Antoine Brodin freebsd_committer 2017-12-01 19:28:18 UTC
First,  devel/py-pandas doesn't exist

Secondly,  please use FLAVOR=,  stop abusing PYTHON_VERSION
Comment 3 John W. O'Brien 2017-12-01 20:17:02 UTC
(In reply to Charlie Li from comment #1)

I'm not trying to use ${lang}_DEFAULT, and I'm not claiming it's broken. Can you help me understand your point there?

As for "make FLAVOR=blah install", I did read the announcement and saw that statement. However, Uses/python.mk still claims that

# Variables, which can be set by a user:
#
# PYTHON_VERSION        - The chosen Python interpreter including the version,
#                         e.g. python2.7, python3.3, etc. This allows the user
#                         to override the currently chosen default version and
#                         to install the port for a specific Python version.
#                         It must not be set by a port.

How is that variable supposed to interact with FLAVOR?
Comment 4 John W. O'Brien 2017-12-01 20:22:51 UTC
(In reply to Antoine Brodin from comment #2)

First, oops, I meant math/py-pandas. I was experimenting with several different ports to make sure what I was seeing was widely reproducible, and ended up editing some parts of the bug description incompletely.

Second, are you saying that the way to accomplish my goal is by replacing

    PYTHON_VERSION=pythonXY

with

    FLAVOR=pyXY

in /usr/local/etc/poudriere.d/pythonXY-make.conf?

I don't understand why you consider what I'm doing abusive. It seems to be consistent with the documented behavior, and it has allowed me to achieve a goal that is important to me. If there is a better way to do this with FLAVORS, I am eager to learn what that is and to abandon my evil ways.
Comment 5 Antoine Brodin freebsd_committer 2017-12-01 20:24:54 UTC
(In reply to John W. O'Brien from comment #4)
If you use poudriere,  you can do:

poudriere bulk -j foo -p bar math/py-pandas@py36
Comment 6 John W. O'Brien 2017-12-01 20:32:03 UTC
(In reply to Antoine Brodin from comment #5)

Ah, I see. So I would no longer need to use multiple configuration sets except for cases when I actually needed different OPTIONs when building for different version of PYTHON.

OK, that definitely helps with my use case. It still seems like DEFAULT_VERSIONS and PYTHON_VERSION aren't behaving as expected though. There's probably still something I don't understand about how they are intended to work together with FLAVORS.
Comment 7 John W. O'Brien 2017-12-01 20:42:55 UTC
(In reply to John W. O'Brien from comment #6)

The command

    poudriere bulk -j 104amd64 -z drivel devel/py-attrs@py34

complains that

    Ignored: Unknown flavor 'py34', possible flavors: py27 py36 .

because devel/py-attrs sets

    USES=           python
    USE_PYTHON=     autoplist distutils

and

#       flavors         - Force creation of flavors for Python 2 and 3 default
#                         versions, where applicable.
[...]
#       allflavors      - Generate flavors for all possible versions and not
#                         simply the default ones.
#[...]
#       distutils       - Use distutils as do-configure, do-build and
#                         do-install targets. implies flavors.

I am still worse off than before because I want what "allflavors" does but I'm only getting what "flavors" does. Is there a way to control that from poudriere's end?

I realize this is no longer in scope for the original bug I reported. How would you like to continue this thread?
Comment 8 Antoine Brodin freebsd_committer 2017-12-01 20:45:48 UTC
(In reply to John W. O'Brien from comment #7)

Put in poudriere.d/make.conf : BUILD_ALL_PYTHON_FLAVORS=yes

Then:

poudriere bulk -j 104amd64 -z drivel devel/py-attrs@all
or
poudriere bulk -j 104amd64 -z drivel devel/py-attrs@py34
Comment 9 John W. O'Brien 2017-12-01 20:52:36 UTC
(In reply to Antoine Brodin from comment #8)

That was definitely the piece I hadn't put together. Thank you. Consider me reformed.

I will await additional information on the question of PYTHON_VERSION.
Comment 10 Antoine Brodin freebsd_committer 2017-12-01 20:54:29 UTC
(In reply to John W. O'Brien from comment #9)
I was never a fan of overriding PYTHON_VERSION,  it was a hack that worked sometimes and sometimes not.
Comment 11 John W. O'Brien 2017-12-01 21:03:23 UTC
(In reply to Antoine Brodin from comment #10)

With the introduction of FLAVORS, what are the remaining use cases for PYTHON_VERSION? Should it be deprecated?
Comment 12 yamayan 2017-12-03 11:38:51 UTC
Created attachment 188490 [details]
patch, python flavor uses DEFAULT_VERSIONS += python=
Comment 13 Antoine Brodin freebsd_committer 2017-12-03 12:27:25 UTC
There is a better patch at https://reviews.freebsd.org/D13326
Comment 14 yamayan 2017-12-03 13:09:28 UTC
(In reply to Antoine Brodin from comment #13)
I don't think D13326'patch is better, but it works fine.
Comment 15 Mathieu Arnold freebsd_committer 2017-12-04 09:20:00 UTC
(In reply to John W. O'Brien from comment #11)
> (In reply to Antoine Brodin from comment #10)
> 
> With the introduction of FLAVORS, what are the remaining use cases for
> PYTHON_VERSION? Should it be deprecated?

There are absolutely no case of setting PYTHON_VERSION.  To set the Python version you want, you have to use DEFAULT_VERSIONS.
Comment 16 David Naylor freebsd_committer 2018-01-17 06:37:20 UTC
I would like to add my 2c on PYTHON_VERSION: 

There are, I think, two situations where PYTHON_VERSION adds functionality that FLAVORS does not provide:
 - if a port supports multiple versions of python but does not support FLAVORS (for example end user programs[1]).  
 - if a port supports multiple versions of python but is only a consumer of python (be it for building or for support scripts at runtime)

In both these cases if someone wants to use a different version of python, without changing the default version of python, then PYTHON_VERSION is the recommended (and currently supported) way.  

I do, however, think FLAVORS and PYTHON_VERSION need to be enhanced, specifically:
 - setting PYTHON_VERSION then forces the default FLAVOR, and
 - setting PYTHON_VERSION outside the range of FLAVORS gets that version added to FLAVORS

Alternatively, we need to drop support for using using non-default versions of python (and correctly handle when a port does not support a default version).

[1] See the discussion around textproc/meld: https://lists.freebsd.org/pipermail/svn-ports-head/2018-January/165377.html
Comment 17 Caroline Webb 2019-01-16 12:48:25 UTC
MARKED AS SPAM
Comment 18 Mathieu Arnold freebsd_committer 2019-01-16 13:05:43 UTC
The PYTHON_VERSION variable is internal and should never be set directly.
Comment 19 John W. O'Brien 2019-01-16 14:36:19 UTC
(In reply to Mathieu Arnold from comment #18)

In that case the documentation is wrong and should be updated. If it is internal then it is not a variable "which can be set by a user." The full quote from Mk/Uses/python.mk in comment #3 still reflects the current state of that file.
Comment 20 commit-hook freebsd_committer 2019-01-16 15:14:30 UTC
A commit references this bug:

Author: mat
Date: Wed Jan 16 15:13:52 UTC 2019
New revision: 490485
URL: https://svnweb.freebsd.org/changeset/ports/490485

Log:
  PYTHON_VERSION cannot be changed, use DEFAULT_VERSIONS.

  PR:		224024

Changes:
  head/Mk/Uses/python.mk