Bug 227260 - net/ceph: unbreak build with python3 as default
Summary: net/ceph: unbreak build with python3 as default
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Jason E. Hale
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-04-03 16:07 UTC by Dima Panov
Modified: 2018-08-28 02:35 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (wjw)


Attachments
net/ceph: unbreak build with python3 as default (752 bytes, patch)
2018-04-03 16:07 UTC, Dima Panov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dima Panov freebsd_committer 2018-04-03 16:07:48 UTC
Created attachment 192173 [details]
net/ceph: unbreak build with python3 as default

net/ceph try to use "sphinx-build" for making docs, which is absent in requested "py27-flavoured" package with python3 set as default.

Solution: use "sphinx-build-2.7" instead of "sphinx-build" in cmake scripts
Comment 1 Yuri Victorovich freebsd_committer 2018-04-04 08:56:12 UTC
When I replace python:2,7 -> python:3.6, it breaks in poudriere:

CMake Error at /usr/local/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE) (Required is at
  least version "2.7")
Call Stack (most recent call first):
  /usr/local/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/share/cmake/Modules/FindPythonInterp.cmake:152 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:435 (find_package)
Comment 2 Dima Panov freebsd_committer 2018-04-04 09:07:25 UTC
(In reply to Yuri Victorovich from comment #1)

ceph itself should still use py2.7, my patch is only enable correct dependency use when python3 is set as default for system
Comment 3 Yuri Victorovich freebsd_committer 2018-04-04 09:27:57 UTC
I'm leaving this to the maintainer, since the patch doesn't really unbreak it for 3.6. It possibly unbreaks it on the system with python3 as a default, but normally setting USES=python:3.6 should work, and it doesn't.

Maintainer, please fix it so that it works when USES=python:3.4+.
Comment 4 Willem Jan Withagen 2018-04-04 10:21:15 UTC
I will try to fix this, once I get to building a new luminous release.
Problem is that I really do not have much of a clue as to how the whole
2.7 <> 3.x python stuff should work.
Let alone with the fact that most of Ceph only care about the Linux implementation and they already have enough to worry about with all the different distros and the versions that thay run.

That said, I do blieve that Ceph is able to run in a 3.x only environment.

But currently too busy with $WORK jobs.
Comment 5 Dima Panov freebsd_committer 2018-04-04 11:42:15 UTC
(In reply to Willem Jan Withagen from comment #4)

At least, building with py2 in py3-enabled world should be fixed asap :)
Comment 6 Yuri Victorovich freebsd_committer 2018-04-04 15:26:18 UTC
(In reply to Dima Panov from comment #5)

Yes, but its behavior is highly unusual. -) I didn't see even on other port that would depend on default python version like this.
Comment 7 Willem Jan Withagen 2018-04-04 15:44:00 UTC
(In reply to Yuri Victorovich from comment #6)

Right,
I claim complete innocence, since all the python-versioning was done after by others, after we got the versioning options in ports/pkg and likes.

And again, I'm far to busy atm. to atually figure out what should be done.
What I can do, is submit/approve suggested changes that I can test.
Note that I do not even have a python 3.x test enviroment.
Comment 8 Yuri Victorovich freebsd_committer 2018-04-04 16:06:49 UTC
(In reply to Willem Jan Withagen from comment #7)

> Note that I do not even have a python 3.x test enviroment.

I don't have a python3 testing environment, and I commit python ports all the time. This is because test environment isn't needed. The way python2/python3 works is that you can switch most of them from 2 to 3 and back and they should still work. Default shouldn't matter, it only affects what happens by default. Ceph is different because default matters (I didn't actually verify this).
Comment 9 Dima Panov freebsd_committer 2018-04-04 23:28:19 UTC
(In reply to Yuri Victorovich from comment #8)
Heh. It matters for ceph only by design error — rely on non-versioned symlinks, which are absent while py27 is not default for system.
Comment 10 commit-hook freebsd_committer 2018-04-20 20:09:24 UTC
A commit references this bug:

Author: jbeich
Date: Fri Apr 20 20:08:29 UTC 2018
New revision: 467840
URL: https://svnweb.freebsd.org/changeset/ports/467840

Log:
  net/ceph: drop bogus flavor specifier

  Unsuffixed binaries are only installed for default python version.

  $ DEFAULT_VERSIONS=python=3.6 make
  [...]
  ===>   ceph-12.2.4_1 depends on executable: sphinx-build - not found
  ===>   ceph-12.2.4_1 depends on executable: virtualenv - not found

  PR:		227260

Changes:
  head/net/ceph/Makefile
Comment 11 commit-hook freebsd_committer 2018-04-20 20:16:33 UTC
A commit references this bug:

Author: jbeich
Date: Fri Apr 20 20:15:51 UTC 2018
New revision: 467841
URL: https://svnweb.freebsd.org/changeset/ports/467841

Log:
  MFH: r467840

  net/ceph: drop bogus flavor specifier

  Unsuffixed binaries are only installed for default python version.

  $ DEFAULT_VERSIONS=python=3.6 make
  [...]
  ===>   ceph-12.2.4_1 depends on executable: sphinx-build - not found
  ===>   ceph-12.2.4_1 depends on executable: virtualenv - not found

  PR:		227260
  Approved by:	ports-secteam blanket

Changes:
_U  branches/2018Q2/
  branches/2018Q2/net/ceph/Makefile
Comment 12 Jan Beich freebsd_committer 2018-04-20 20:34:38 UTC
virtualenv isn't referenced by installed files...
Comment 13 commit-hook freebsd_committer 2018-08-27 16:10:04 UTC
A commit references this bug:

Author: jhale
Date: Mon Aug 27 16:09:22 UTC 2018
New revision: 478216
URL: https://svnweb.freebsd.org/changeset/ports/478216

Log:
  Fix build when DEFAULT_VERSIONS=python=3.[4-6] is set

  In a python 3.x environment, the unversioned symlink to sphinx-build points
  to the python 3.x version which will cause the build to fail. This port
  can only use python 2.7, so we need to specify the version of sphinx-build.

  This was supposedly fixed in r467840 and the PR was closed, but I was
  informed by fluffy via private mail that it was still broken.

  PR:		227260
  Submitted by:	fluffy
  Approved by:	portmgr (blanket - trivial build fix)

Changes:
  head/net/ceph/Makefile
Comment 14 Jan Beich freebsd_committer 2018-08-27 16:32:38 UTC
How did fluffy@ test? comment 10 assumed DEFAULT_VERSIONS=python=3.6, see https://ptpb.pw/Dr7Z
Common mistakes:
- Blindly overriding PY_FLAVOR without checking the port actually supports flavors
- Adding @${PY_FLAVOR} suffix for dependencies that are not imported but called as executables
Comment 15 Dima Panov freebsd_committer 2018-08-27 23:39:27 UTC
(In reply to Jan Beich from comment #14)
Just try to build ceph yourself, please. 

With py3 as default, we already have declared to use py27 instead at build stage and this provided a correct PY_FLAVOR, 27. But in this case, unversioned sphinx-build will point to py3 version rather than py27 on a real system and will be absent on clean poudriere environment, and I told this when filed PR.

In any case, rely on unversioned/unflavored binaries in flavored world is a huge design mistake and will produce more scenarios like we have there, please assume it. 

With python projects, unflavored binaries are only compat symlinks and created only 
when flavor == default-python-version.


Logs:
DEFAULT_VERSIONS+=python=3.6
DEFAULT_VERSIONS+=python3=3.6

Before (fails at unversioned sphinx): 
http://trueos.miwi.cc/data/fbsd11-dima-dimaports-py3/2018-08-26_15h22m28s/logs/errors/ceph-12.2.7_2.log

After (with picked up right, versioned sphinx): 
http://trueos.miwi.cc/data/fbsd11-dima-dimaports-py3/2018-08-28_07h23m56s/logs/ceph-12.2.7_2.log

===========
DEFAULT_VERSIONS+=python=2.7

And just almost default, with py27: 
http://trueos.miwi.cc/data/fbsd11-dima-dimaports-ssl/2018-08-28_07h01m11s/logs/ceph-12.2.7_2.log
Comment 16 Jan Beich freebsd_committer 2018-08-28 01:11:57 UTC
Ah, sorry. I didn't notice jhale@ reverted my fix here by bringing back PY_FLAVOR. DEFAULT_VERSIONS+=python=3.6 was indeed broken between ports r477944 to ports r478216. Porter's Handbook probably needs to document unflavored python dependencies aren't forbidden, otherwise adding PY_FLAVOR turns into a cargo cult like here or bug 227573.

(In reply to Dima Panov from comment #15)
> In any case, rely on unversioned/unflavored binaries in flavored
> world is a huge design mistake and will produce more scenarios like
> we have there, please assume it.

Python flavors allow ports depend on more than one python version. Why add extra churn for little benefit? Besides, patching build is more error prone than just using BINARY_ALIAS.
Comment 17 Dima Panov freebsd_committer 2018-08-28 01:47:47 UTC
(In reply to Jan Beich from comment #16)

Oh, I missed that BINARY_ALIAS does the trick not only for binutils/lld :(
Sure, for buildtime binaries this is a very good way


BTW, ceph can use python3 and do it on other platforms, so return flavors to deps is reasonable
Comment 18 Dima Panov freebsd_committer 2018-08-28 02:35:50 UTC
(In reply to Jan Beich from comment #16)

>DEFAULT_VERSIONS+=python=3.6 was indeed broken between ports r477944 to ports r478216

It was broken with any py3.x regardless flavors presence due to missed right unversioned symlink to sphinks-build-27