Bug 237795 - Mk/Uses/python.mk: "needs Python 3.x at least, but 2.7 was specified" (eg: devel/gobject-introspection)
Summary: Mk/Uses/python.mk: "needs Python 3.x at least, but 2.7 was specified" (eg: de...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Kubilay Kocak
URL:
Keywords:
: 240155 240429 (view as bug list)
Depends on: 242614
Blocks:
  Show dependency treegraph
 
Reported: 2019-05-08 12:31 UTC by Marco Beishuizen
Modified: 2020-03-16 06:21 UTC (History)
19 users (show)

See Also:


Attachments
Mk/Uses/python.mk: remove broken conditional (641 bytes, patch)
2019-12-12 01:46 UTC, Chris Hutchinson
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Marco Beishuizen 2019-05-08 12:31:58 UTC
When trying to portupgrade www/firefox to 66.0.5, it tries to build the dependency devel/gobject-introspection. The default Python version is 3.6 on the system, but building gobject-introspection still fails:

...
===>  Patching for libnotify-0.7.7_1
===>   libnotify-0.7.7_1 depends on executable: gmake - found
===>   libnotify-0.7.7_1 depends on package: pkgconf>=1.3.0_1 - found
===>   libnotify-0.7.7_1 depends on executable: g-ir-scanner - not found
===>  gobject-introspection-1.56.1,1 needs Python 3.4 at least, but 2.7 was
specified.
*** Error code 1

Stop.
make[3]: stopped in /usr/ports/devel/gobject-introspection
*** Error code 1

Stop.
make[2]: stopped in /usr/ports/devel/libnotify
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/www/firefox
*** Error code 1
...
Comment 1 Marco Beishuizen 2019-05-08 18:05:50 UTC
Turns out that it's probably a portupgrade/flavors bug. I have a patch installed for portupgrade that should make portupgrade flavor-capable. But somehow this doesn't work well in case of gobject-introspection. Upgrading with portmaster went ok.

I'll close this one for now since it's not a gobject-introspection bug.
Comment 2 Marco Beishuizen 2019-05-10 10:37:49 UTC
Reopen this bug because it doesn't seem to be a portupgrade bug. Tried to "make install clean" math/gnumeric in ports, but stops at devel/gobject-introspection with the same error.
Comment 3 Mikhail Teterin freebsd_committer 2019-06-05 02:07:41 UTC
If portmgr actually built ports from source -- rather than relying on Podrierre to install all dependencies from pre-built packages -- they would've encountered this bug much earlier. It affects ports needing one version of Python, but depending on ports requiring another.

For example:

...
===>   gstreamer1-plugins-gl-1.14.4_2 depends on shared library: libgraphene-1.0.so - not found
===>  graphene-1.8.2_1 needs Python 3.4 at least, but 2.7 was specified.
*** Error code 1

Installing the dependencies (like graphene in my example) first -- directly from command line -- is the workaround...
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-05 09:18:29 UTC
(In reply to Marco Beishuizen from comment #0)

Can you please provide more information, including:

- Exact FreeBSD version (uname -a output)
- pkg version -v output (as an attachment)
- /etc/make.conf contents (as an attachment)
- Full build log, including portupgrade/portmaster command (as an attachment)

^Triage: Leave this as an individual port issue until (if) isolated to a framework issue.
Comment 5 Marco Beishuizen 2019-06-15 18:46:07 UTC
I need closing this one again because the bug seems to have disappeared.

I did notice that the bug doesn't appear when using portmaster. Portmaster is builing devel/bison before building gobject-introspection, and then the bug doesn't happen. Using portupgrade or doing a make install directly was triggering the bug because devel/bison wasn't build first for some reason.

Trying to reproduce the bug today failed: bison does build before gobject-introspection now, and the bug disappears.

My uname -a:
FreeBSD yokozuna 12.0-STABLE FreeBSD 12.0-STABLE r343805 YOKOZUNA  amd64

My make.conf:
CPUTYPE?=nocona
KERNCONF=YOKOZUNA
DISTDIR=/var/ports/distfiles
PACKAGES=/var/ports/packages
OPTIONS_UNSET=GNOME GCONF DCONF GSETTINGS CUPS AVAHI HAL ALSA PULSEAUDIO WAYLAND COLORD
OPTIONS_SET=LPR NVIDIA NVTHREADS SNDIO OSS ZTS
PKG_NOCOMPRESS=1
WITH_MPM=event
#DEVELOPER=yes                                                                        
#BATCH=yes                                                                            
#DISABLE_VULNERABILITIES=yes                                                          
#FORCE_PKG_REGISTER=yes                                                               
DEFAULT_VERSIONS+= mysql=8.0 pgsql=11 emacs=nox lua=5.3
MESA_LLVM_VER=80
.if ${.CURDIR:M*/editors/emacs*}
FLAVOR?=nox
.endif
Comment 6 Mikhail Teterin freebsd_committer 2019-06-16 10:04:45 UTC
Gentlemen, what's happening? The problem's been analyzed... When building a port (P0), that -- for whatever reason -- wants Python-N, the Python-version is propagated to all dependencies.

When one of the dependencies (Pk) is incompatible with Python-N, you get the  error...

In Marco's case, for example, P0 is math/gnumeric, which insists on Python-2.7 -- and Pk is devel/gobject-introspection, which wants 3.4+ or higher.

This is difficult to reproduce, because, once you have the dependency (such as gir) installed -- either through a direct "make install" or via a pre-built package -- the dependents will build just fine... The problem strikes only, when the dependency is built automatically -- with the requested Python-version passed from the dependent.

The bug is not with devel/gobject-introspection itself. This is a portmgr (or python@ ?) issue -- there needs to be a way to distinguish between the two cases:

 1. I'm going to use Python-N, and so must you, or else we'd be incompatible at run-time.
 2. I need to use Python-N, but I don't care, what you're using.
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-16 10:50:13 UTC
I have no idea how the situation ended up resolving itself, but being provided that information, or an opportunity to obtain it may have been enlightening.

As far as I can see, the gnumeric case is a case of wrong dependencies.

gnumeric cannot logically *only* support 2.7, and have a set of dependencies that *don't also* support 2.7.

Either gnumeric is incorrectly constraining itself to 2.7 when it supports 3.x, or gobject-introspection is incorrectly constrained to 3.4+, when it also supports 2.7, or some dependencies are spurious.

math/gnumeric (2.7)
`-- devel/pygobject3-common (no python version-spec, any version)
    `-- devel/gobject-introspection (3.4+)

And I don't see how 'installing it manually' can fix *this* issue.

You cant install py36-gobject-introspection and have it be a satisfactory dependency for py27-pygobject3-common

You also cant install py36-pygobject3-common to get py36-gobject-introspection and have either of those satisfy math/gnumeric which only ever wants/supports (at present) 2.7.

If I'm misunderstanding what you meant, please elaborate with an example of 'installing it manually, not as a automatic, depends', that fixes *this* issue, which is:

 a) a port requiring one specific version (eg: only 2.7)
 b) cannot be built when one of its dependencies (eg: gobject-introspection)
 c) requires a version that does not overlap with its own (eg: 3.4+)

But I can't see how that's possible, as making (a)(b)(c) work is precluded fundamentally just by the the Python ecosystem works.
Comment 8 Mikhail Teterin freebsd_committer 2019-06-16 11:08:21 UTC
(In reply to Kubilay Kocak from comment #7)
> please elaborate with an example of 'installing it manually, not as a
> automatic, depends', that fixes *this* issue, which is:
>
> a) a port requiring one specific version (eg: only 2.7)
> b) cannot be built when one of its dependencies (eg: gobject-introspection)
> c) requires a version that does not overlap with its own (eg: 3.4+)

The version of Python, that's used by a dependency, may not actually matter, when the dependency is not itself a Python-package -- that is, is not used by the dependent as "import foo". Or, when Python is merely a BUILD-dependency for the dependency... (Both of these examples fall under item 2. in my Comment #6.)

This is why, after you install gir manually -- allowing it to use whatever version it wants -- build works.

Things break, when the build of gir is invoked automatically by the build of a depending port, because in this case the Python-version is EXPLICITLY PRESCRIBED (by adding PYTHON_VERSION to the environment, I think)...
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-16 11:59:19 UTC
(In reply to Mikhail Teterin from comment #8)

Sure, got it. Thanks for helping me get there. And the dependency is satisfied after manual installation because it's a binary/filename *_DEPENDS, not a package type.

So for case (2) of your two types, this type, one would need to be able to, minimally at least, based on what we know now:

a) more finely declare the dependency kind (so as not to propagate for it)
b) declare it on a per-dependency basis (have it not apply to any other things where it may not apply)

Independent to the feasibility, something like that is going to require some very clear and precise "spec'ing" out before it goes to portmgr, as I suspect it will need to. 

This is because I don't see this as being a bug, but rather a feature to more finely control (perhaps more precisely: ability to constrain) python version propagation, to take into account how a dependency is used.

It might be worth us chewing the fat on IRC (#freebsd-python) and coming up with an unambiguous and lightweight "PEP" for it. Who knows, we may even be able to come up with a hack^W workaround in the short term, or other alternatives
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-16 12:03:35 UTC
I'll take this for now, and will re-summarise/classify accordingly.
Comment 11 takeda 2019-06-20 04:41:58 UTC
Similar thing is happening with meson when used as a build dependency for deluge. We will have more of these occurrences as 2.7 is becoming EOL and packages are dropping support for it.
Comment 12 Alfredo Dal'Ava Junior freebsd_committer 2019-07-04 16:25:11 UTC
I reproduce the same issue on powerpc64 using a clean FreeBSD 13/current:

# cd /usr/ports/www/firefox && make

It crashed with "needs Python 3.4 at least, but 2.7 was specified." on 'devel/gobject-introspection'

# cd /usr/ports/devel/gobject-introspection && make

It crashed with "needs Python 3.4 at least, but 2.7 was specified." now on 'devel/meson'


I did workaround this by doing:


# cd /usr/ports/devel/meson && make && make install
# cd /usr/ports/devel/gobject-introspection && make && make install
# cd /usr/ports/www/firefox && make
Comment 13 Walter Schwarzenfeld freebsd_triage 2019-08-19 10:50:40 UTC
/usr/ports/MOVED:
lang/python34|lang/python36|2018-12-18|Has expired: Fails to build with recent OpenSSL
Comment 14 Ting-Wei Lan 2019-08-19 11:34:09 UTC
(In reply to Walter Schwarzenfeld from comment #13)
What did you mean by posting this line? It doesn't look relevant to the issue.
Comment 15 Mikhail Teterin freebsd_committer 2019-08-23 04:03:09 UTC
(In reply to Kubilay Kocak from comment #10)
Right now this is breaking firefox build -- firefox wants both python-2.7 and llvm80. The latter wants python-3.6 (why?!)

So, if llvm80 is not already installed, the build of firefox tries to build it -- which breaks, because firefox passes 2.7 as the requirement, which llvm80 rejects.

The work-around is to build llvm80 first. It should not be necessary.
Comment 16 Bryan Drewery freebsd_committer 2019-08-28 16:13:31 UTC
(In reply to Mikhail Teterin from comment #3)

This is dumb. Portmgr uses Poudriere which builds everything from source.

Portupgrade doesn't support FLAVORs. Move on.
Comment 17 Mikhail Teterin freebsd_committer 2019-08-28 21:44:31 UTC
(In reply to Bryan Drewery from comment #16)
> This is dumb. Portmgr uses Poudriere which builds everything from source.

This is dumb -- because it does not, not quite. Poudriere performs (the equivalent of) the following:

   1. make -C /usr/ports/depend/ency install clean
   2. make -C /usr/ports/dep/endent install clean

This still works. For example, llvm (the dependency) builds against Python-3.6, while Firefox then uses llvm to build against Python-2.7. And those using Podriere exclusively are unaware of the problem -- hence my frustration with portmgr@ (overly) relying on the tool.

What happens, when one REALLY builds everything from source is:

   1. make -C /usr/ports/dep/endent install clean

In the above scenario, the dependent passes its preferred FLAVORs to the dependencies. Which breaks, if one of the dependencies disagrees on the flavor. This is a design flaw in the FLAVORs implementation, which needs to be corrected.

This was already explained, in detail, in comment #8 -- and accepted by koobs@ in comment #9. But you saw me "unfairly attacking" your team and rushed to defend it without even reading to the end, evidently...

You can reproduce the problem today -- by building Firefox on a machine without llvm80 already installed. FROM SOURCE.

> Portupgrade doesn't support FLAVORs. Move on.

I'll take this information, however irrelevant to the ticket, under advisement. Thank you.
Comment 18 Walter Schwarzenfeld freebsd_triage 2019-08-28 23:36:46 UTC
python.mk:
457 # Pass PYTHON_VERSION down the dependency chain. This ensures that
    458 # port A -> B -> C all will use the same python version and do not
    459 # try to find a different one, if the passed version fits into
    460 # the supported version range.
    461 PYTHON_VERSION?=        python${_PYTHON_VERSION}
    462 .if !defined(PYTHON_NO_DEPENDS) && \
    463     ${PYTHON_VERSION} != python${PYTHON_DEFAULT}
    464 DEPENDS_ARGS+=          PYTHON_VERSION=${PYTHON_VERSION}
    465 .endif


....that is the cause...
Comment 19 Chris Hutchinson 2019-12-11 06:57:28 UTC
Just adding a "me too" here, with all the gory details:

I spent all day yesterday on a fresh world/kernel
fighting to install ports -- most of which failed
to build due to python (version) related errors.
I finally blew everything off the disk, and built/
installed a new world/kernel today, and made another
attempt, which (of course) also failed.

Procedure:
make/install perl5.30, python27 && python37.
cd /usr/ports/x11/xorg; make config-recursive until
the dialogs will no longer appear.
make

This is the result:
===>  Staging for xorg-7.7_3
===>   xorg-7.7_3 depends on file: /usr/local/libdata/pkgconfig/dri.pc - not
found
===>   mesa-dri-18.3.2_9 depends on package: wayland-protocols>=1.8 - found
===>   mesa-dri-18.3.2_9 depends on file:
/usr/local/libdata/pkgconfig/pthread-stubs.pc - found
===>   mesa-dri-18.3.2_9 depends on executable: bison - found
===>   mesa-dri-18.3.2_9 depends on executable: msgfmt - found
===>   mesa-dri-18.3.2_9 depends on executable: gmake - found
===>   mesa-dri-18.3.2_9 depends on package: pkgconf>=1.3.0_1 - found
===>   mesa-dri-18.3.2_9 depends on file: /usr/local/bin/python2.7 - found
===>   mesa-dri-18.3.2_9 depends on package: llvm80>=3.9.0_4 - not found
===>  llvm80-8.0.1_3 needs Python 3.6 at least, but 2.7 was specified.
*** Error code 1

These (python) related errors were the only ones generated all day
yesterday. As a result; most nothing can be installed when more
than one version of python is installed.
It is clear that python multiplicity is no longer supported.
Someone forgot to mention that in UPDATING. Please make an entry
ASAP so others will not have to suffer needlessly.

WORKING ENV:
make.conf(5)
DEFAULT_VERSIONS+=tcl=8.7 tk=8.7

FreeBSD eleventhree 11.3-RELEASE-p5 FreeBSD 11.3-RELEASE-p5 #0 r355580: Tue Dec
10 13:22:30 PST 2019
root@eleventhree:/usr/obj/usr/src/sys/ELEVENTHREE  amd64

Path: /usr/src
Working Copy Root Path: /usr/src
URL: svn://svn.freebsd.org/base/releng/11.3
Relative URL: ^/releng/11.3
Repository Root: svn://svn.freebsd.org/base
Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Revision: 355580
Node Kind: directory
Schedule: normal
Last Changed Author: gordon
Last Changed Rev: 354654
Last Changed Date: 2019-11-12 10:13:51 -0800 (Tue, 12 Nov 2019)

Path: /usr/ports
Working Copy Root Path: /usr/ports
URL: svn://svn.freebsd.org/ports/head
Relative URL: ^/head
Repository Root: svn://svn.freebsd.org/ports
Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
Revision: 519768
Node Kind: directory
Schedule: normal
Last Changed Author: pizzamig
Last Changed Rev: 519768
Last Changed Date: 2019-12-10 09:39:01 -0800 (Tue, 10 Dec 2019)

--Chris

@Walter
Good find! :)
Comment 20 Chris Hutchinson 2019-12-12 01:46:11 UTC
Created attachment 209881 [details]
Mk/Uses/python.mk: remove broken conditional

Mk/Uses/python.mk
This svn(1) diff(1) fixes a problem introduced to
Mk/Uses/python.mk
Its intent was to create a usable range of python versions
for any group of build/install targets. But was actually a
conditional/binary. As a result, it resulted in the failure
to build a great many ports.
This patch corrects the problem by removing the offending lines.
A more appropriate solution would be to use a greater than ||
less than, in an attempt to find a suitable range for the build
targets involved.
In any case. This patch will work correctly until a better
solution is found.
Tested on
FreeBSD eleventhree 11.3-RELEASE-p5 FreeBSD 11.3-RELEASE-p5 #0 r355580:
Tue Dec 10 13:22:30 PST 2019
root@eleventhree:/usr/obj/usr/src/sys/ELEVENTHREE  amd64

/usr/ports
519768

Special thanks to:
Walter Schwarzenfeld, pr 237795 comment 18, for the initial insight
Antoine Brodin, pr 233723 comment 7, for the "clue bat"


--Chris
Comment 21 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-13 23:38:08 UTC
*** Bug 240155 has been marked as a duplicate of this bug. ***
Comment 22 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-13 23:53:58 UTC
Comment on attachment 209881 [details]
Mk/Uses/python.mk: remove broken conditional

^Triage: Patch is in bug 242614, the now dependent bug dont, duplicate it here
Comment 23 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-15 10:38:02 UTC
*** Bug 240429 has been marked as a duplicate of this bug. ***
Comment 24 Tommy P 2020-01-26 07:11:33 UTC
I have similar issue with these dependent ports:

  devel/py-ioflo
  net/py-raet

If I build the ports interactively within poudriere, they build successfully and are linked to python 2.7.  They're ignored in poudriere because they're trying to link with python 3.8 when 

  USES=           python:2.7

is specified in the Makefiles.  My DEFAULT_VERSIONS:

  grep -e '^DEFAULT' /etc/make.conf

  DEFAULT_VERSIONS+=bind=9.14 bdb=6
  DEFAULT_VERSIONS+=mysql=10.4m pgsql=12 samba=4.10 ssl=openssl
  DEFAULT_VERSIONS+=perl5=5.30 python=3.8 python2=2.7 python3=3.8 ruby=2.6 tcltk=87
  DEFAULT_VERSIONS+=apache=2.4

Main port to build:

    sysutils/py-salt
Comment 25 Walter Schwarzenfeld freebsd_triage 2020-01-26 09:25:03 UTC
(In reply to Walter Schwarzenfeld from comment #18)
The conditioinal is removed in ports r522485.