Bug 227428 - devel/cmake: fails to find suffixed libboost_python
Summary: devel/cmake: fails to find suffixed libboost_python
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-ports-bugs mailing list
URL:
Keywords: patch
Depends on:
Blocks: 227427
  Show dependency treegraph
 
Reported: 2018-04-10 19:30 UTC by Jan Beich
Modified: 2019-04-05 15:44 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (wjw)
jbeich: maintainer-feedback? (kde)


Attachments
workaround (2.89 KB, patch)
2018-04-16 05:54 UTC, Jan Beich
jbeich: maintainer-approval? (kde)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jan Beich freebsd_committer 2018-04-10 19:30:51 UTC
After bug 227427 lands this port will end up BROKEN. Can you help?
libboost_python.so is now libboost_python27.so
libboost_python3.so is now libboost_python36.so

$ fetch -qo- 'https://reviews.freebsd.org/D15030?download=true' | patch -Efsp0 -d /usr/ports
$ make all deinstall install clean -C/usr/ports/devel/boost-jam
$ make all deinstall install clean -C/usr/ports/devel/boost-libs
$ make all deinstall install clean -C/usr/ports/devel/boost-python-libs
$ make -C/usr/ports/net/ceph
[...]
CMake Error at /usr/local/share/cmake/Modules/FindBoost.cmake:1960 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.67.0

  Boost include path: /usr/local/include

  Could not find the following Boost libraries:

          boost_python
[...]

build log: https://ptpb.pw/8qfO
regressed by: https://github.com/boostorg/python/commit/d4d41d94aecc
Comment 1 Jan Beich freebsd_committer 2018-04-10 19:55:33 UTC
More ports are affected this time.

$ make -C/usr/ports/math/kig
[...]
-- Could NOT find BoostPython (missing: BoostPython_LIBRARIES BoostPython_INCLUDE_DIRS) 
[...]
===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: share/katepart5/syntax/python-kig.xml
[...]
https://ptpb.pw/RMyh
Comment 2 Willem Jan Withagen 2018-04-11 07:30:09 UTC
(In reply to Jan Beich from comment #1)

Hi Jan,

I think that the fallout stems from Cmake not understandig that the library is not called libboost_python any longer, so perhaps the fix should go into the Cmake module that tries to find and there account for the python-versioning.
Comment 3 Willem Jan Withagen 2018-04-11 07:54:34 UTC
(In reply to Willem Jan Withagen from comment #2)

The orignal code in Ceph is like:
  list(APPEND BOOST_COMPONENTS python)
  ......
  find_package(Boost 1.66 COMPONENTS ${BOOST_COMPONENTS} REQUIRED)

And then the CMAKE modules do their stuff.

Asking for a different python version there fixes the issue when there is that version library.
So it is fixable from Ceph, but I strongly think it needs to be fixed in the actual CMake code that tries to find the libboost_python{PY_VER}.
Comment 4 Adriaan de Groot freebsd_committer 2018-04-11 13:19:04 UTC
There is a CMake 3.11 update in exp-run, and I hear from tcberner@ that there is a hefty rewrite in FindBoost (again) that specifically tackles Python. So I'll check if that rewrite is in 3.11.0 release. If it is, this PR may go away automagically.
Comment 5 Jan Beich freebsd_committer 2018-04-13 23:43:39 UTC
CMake 3.11.0 doesn't help.
net/ceph: https://ptpb.pw/dFRs
math/kig: https://ptpb.pw/UUwO
Comment 6 Willem Jan Withagen 2018-04-14 00:02:57 UTC
(In reply to Jan Beich from comment #5)

Is there a ENV during CMake-ing that will tell the wanted Python version?

That will allow me to request either 2.7 or 3.x during CMake
Which will atleast work when libboost_python27.so is requested.
Comment 7 Jan Beich freebsd_committer 2018-04-16 05:54:45 UTC
Created attachment 192545 [details]
workaround

(In reply to Willem Jan Withagen from comment #6)
> Is there a ENV during CMake-ing that will tell the wanted Python version?

Like this? ;)
Comment 8 Willem Jan Withagen 2018-04-16 07:54:23 UTC
(In reply to Jan Beich from comment #7)

So to make sure:
    `CMAKE_ARGS+=	-DBOOST_PYTHON_SUFFIX:STRING=${PYTHON_SUFFIX}`
gets automagically added to the Cmake args if USES=python is set.

In which case I'll fix the Cmake build to differentiate on this.
Comment 9 Jan Beich freebsd_committer 2018-04-16 08:25:47 UTC
If -DBOOST_PYTHON_SUFFIX=27 is passed "find_package(Boost COMPONENTS python)" will look not only for libboost_python.so but also libboost_python27.so. Ideally, CMake should try Python_ADDITIONAL_VERSIONS for Boost then fall back to common suffixes. Requiring ugly Boost.Python ifdefs in consumers is non-starter unless a specifc Boost version is explicitly requested i.e., "find_package(Boost 1.67 ...)". And consumers cannot jump on the latest Boost version without alienating downstream[1].

[1] https://repology.org/metapackage/boost/versions

(In reply to Willem Jan Withagen from comment #8)
> In which case I'll fix the Cmake build to differentiate on this.

No need. With the patch net/ceph makes past "configure" stage until it hits C++ bustage (a la ports r467364).
Comment 10 Willem Jan Withagen 2018-04-16 10:06:03 UTC
(In reply to Jan Beich from comment #9)

Oke, I understand the solution, but the Ceph Cmake code is:
  list(APPEND BOOST_COMPONENTS python)
  ......
  find_package(Boost 1.66 COMPONENTS ${BOOST_COMPONENTS} REQUIRED)

Which is there because of all kinds of reasons, mainly having to do with different distro and versions of distros holding boost libs with a too low version.

So from you remark I conclude that this code WILL require handholding.
Even after things are patched?

I do expect to see some python version in Ceph in the near future, because they are running into 2.7 <> 3.x issues as well.

The other compilation errors are a "problem" as I have very little time at the moment. Trying to get $WORK projects finished before I have a little bit of R&R.
Comment 11 Adriaan de Groot freebsd_committer 2018-04-18 09:50:16 UTC
I see jbeich@ committing lots of workarounds for Boost::Python 1.67 .. how much of that would just go away if we fix the CMake module? Also, how much goes away by itself with CMake 3.11.0? There's a specific commit in there

FindBoost: Add support for Boost 1.67 with Python version suffixes

Which, if it doesn't work, needs fixing on our (my, CMake) end.
Comment 12 Willem Jan Withagen 2018-04-18 10:10:47 UTC
(In reply to Adriaan de Groot from comment #11)

If the fix in the Cmake stuff (either FreeBSD specific or not) allows for
this call
    find_package(Boost 1.66 COMPONENTS ${BOOST_COMPONENTS} REQUIRED)
to do "the right thing" (tm), then No changes to the ceph-source are needed.

As things go now I'll have to add a selector on FreeBSD to append the python-version. No biggie, but again specific code for a specific problem.

That said, to has started with a discussion on ceph-dev to determine what to do with 2.7 and how fast to fase it out, and make all 3.x code. 

That could/would leave FreeBSD users that are mainly on 2.7 in the cold.
Which should not be a major problem from opperational point, since you do not want to do much other on the ceph storage nodes. It could however conflict on compute nodes where scripting depends on 2.7 and 2.7 compatible classes.

So I'm driven here by what Ceph does in the code, and by FreeBSD packages in what is possible and desirable. And that in the negative time I have available ATM.
Comment 13 Jan Beich freebsd_committer 2018-04-18 10:27:34 UTC
(In reply to Adriaan de Groot from comment #11)
> how much of that would just go away if we fix the CMake module?

None. My ${PY_BOOST_LIB} fixes are for non-CMake ports or CMake ports with their own way to find Boost.Python.

> how much goes away by itself with CMake 3.11.0?

None. CMake added "pythonXY" (versioned) support as Boost component. If a project doesn't switch from "python" to "pythonXY" CMake won't find libboost_pythonXY installed by Boost 1.67. If "pythonXY" is used then Boost < 1.67 users are out of luck. Obviously, downstream (e.g., FreeBSD Ports) maybe unhappy with whatever Python version was requested, often preferring a more recent one.
Comment 14 Jan Beich freebsd_committer 2018-04-18 10:40:25 UTC
(In reply to Jan Beich from comment #13)
> If "pythonXY" is used then Boost < 1.67 users are out of luck.

Err, mainly FreeBSD Ports users. According to FindBoost.cmake some distributions (Debian/Ubuntu, Gentoo, RPMs) implemented their own versioning *before* Boost 1.67.
Comment 15 commit-hook freebsd_committer 2018-04-18 13:58:42 UTC
A commit references this bug:

Author: jbeich
Date: Wed Apr 18 13:58:16 UTC 2018
New revision: 467712
URL: https://svnweb.freebsd.org/changeset/ports/467712

Log:
  devel/cmake: allow consumers to qualify boost_python version

  PR:		227428
  Exp-run by:	antoine

Changes:
  head/Mk/Uses/python.mk
  head/devel/cmake/Makefile
  head/devel/cmake/files/patch-Modules_FindBoost.cmake
Comment 16 Jan Beich freebsd_committer 2018-04-18 14:00:40 UTC
Boost 1.67 has landed. Try commenting out -DBOOST_PYTHON_SUFFIX in Mk/Uses/python.mk then build devel/libarea in order to better understand CMake 3.11.0 shortcomings.
Comment 17 Willem Jan Withagen 2018-04-18 15:29:33 UTC
(In reply to Jan Beich from comment #16)

extracts from the Cpeh-python discussion:

Mimic - after a "best effort" to migrate, some Python code still runs on Python 2, but Shaman has a Python 3 build flavor and teuthology is capable of testing in Python 3-only environment.

Nautilus - all remaining packaged Python code has been migrated and explicitly runs (and is tested) on Python 3. Python 2 versions of our code do not need to be provided or packaged, because all target platforms support Python 3. There might still be some Python 2 scripts in the git repo, but these are either special cases or for developer use only, and are not packaged. 

(These are the 2 upcoming releases)

And also:
> I'm not objecting to making the leap (it's better than trying to
> support 2 and 3 at the same time!), but let's make sure we understand
> consequences for users.

Meaning that Ceph never intended to have both a 2.7 and 3.x python version.
And it is currently 2.7 for most of the consumer facing items.
Some of the test parts are actually already 3.x compatible, and or running on 3.x
Comment 18 Willem Jan Withagen 2018-04-23 08:05:23 UTC
(In reply to Willem Jan Withagen from comment #17)

Yesterday I had some time to work a bit on this, and with boost 1.67 and CMake 3.11.0_2 I was able to build and test correctly.
This was using a default environment based on python 2.7, even though Ceph starts to move to python3. (I have both installed)

So a challenge might be to have a poudriere env that will be able to have both python 2.7 AND python3? As I have the feeling that this is not really what the ports team is aiming for?
Comment 19 Willem Jan Withagen 2018-04-26 08:04:48 UTC
(In reply to Willem Jan Withagen from comment #17)

Hi Jan,

There is an update to the luminous tree: 12.2.5.
So I ran the current net/ceph port through poudriere after fixing the version and distinfo. 

That makes it compile without much problems.
 - virtenv is indeed not needed.
 - boost_pythons complains with an error:
---
Error: /usr/local/bin/ceph-mgr is linked to /usr/local/lib/libboost_python.so.1.66.0 from devel/boost-python-libs but it is not declared as
 a dependency
Warning: you need LIB_DEPENDS+=libboost_python.so:devel/boost-python-libs
---

Which also shows the "main problem". it is still linked against Boost 1.66.
That even though I portsnapped to update the complete ports tree for the poudriere  env.

Is this expected?
Comment 20 Jan Beich freebsd_committer 2018-04-26 08:16:10 UTC
(In reply to Willem Jan Withagen from comment #19)
> Error: /usr/local/bin/ceph-mgr is linked to /usr/local/lib/libboost_python.so.1.66.0 from devel/boost-python-libs but it is not declared as
 a dependency
> Warning: you need LIB_DEPENDS+=libboost_python.so:devel/boost-python-libs

Did you use an old ports/ tree?
https://lists.freebsd.org/pipermail/svn-ports-head/2018-March/170962.html
https://svnweb.freebsd.org/changeset/ports/463782

> Which also shows the "main problem". it is still linked against Boost 1.66.

HEAD has Boost 1.67.0
2018Q2 has Boost 1.66.0
2017Q4 and 2018Q1 has Boost 1.65.1
2017Q3 has Boost 1.64.0
...

If "pkg -vv | fgrep url" returns /quarterly at the end then you're on 2018Q2.
Comment 21 Willem Jan Withagen 2018-04-26 11:00:37 UTC
(In reply to Jan Beich from comment #20)

I update the ports tree with

poudriere ports -p local12 -u

So where would I find the tree/database to run pkg against?

Or does that always run against the pkg.conf in /usr/local/etc?
Comment 22 Willem Jan Withagen 2018-04-26 17:08:18 UTC
(In reply to Willem Jan Withagen from comment #21)

Must have been a pilot error, since I reloaded ports into the jail.
And now poudriere just builds fine. And I'll submit an upgrade.

Note that I can compile ceph/HEAD without your patch, so I'll test your boost-patch to see what the compiler complains about, and use that to submit a PR to Ceph if needed.
Comment 23 commit-hook freebsd_committer 2019-04-05 11:47:05 UTC
A commit references this bug:

Author: adridg
Date: Fri Apr  5 11:46:46 UTC 2019
New revision: 497948
URL: https://svnweb.freebsd.org/changeset/ports/497948

Log:
  Update CMake to latest release, 3.14.1

  Changelog: https://cmake.org/cmake/help/v3.14/release/3.14.html
  Local patches: fixes for Boost, Python, and OpenMP

  Affected ports:
   - opencpn, hugin needed help in finding wx
   - kadu is over-enthusiastic in finding non-existent X11 components
   - xlife++ does weird things in parsing help output from cmake

  Thanks antoine@ for multiple exp-runs.
  Thanks tcberner@ for much prep-work.

  PR:		236534 223678 227428

Changes:
  head/astro/opencpn/files/patch-CMakeLists.txt
  head/devel/cmake/Makefile
  head/devel/cmake/distinfo
  head/devel/cmake/files/patch-Modules_FindBoost.cmake
  head/devel/cmake/files/patch-Modules_FindLua.cmake
  head/devel/cmake/files/patch-Modules_FindOpenMP.cmake
  head/devel/cmake/files/patch-git_ff06b35
  head/devel/cmake/pkg-plist
  head/devel/cmake-doc/Makefile
  head/devel/cmake-doc/pkg-plist
  head/devel/cmake-gui/Makefile
  head/graphics/hugin/files/patch-sf_f6f56eb9
  head/math/xlife++/files/patch-cmake_xlifepp__user__management.cmake
  head/polish/kadu/files/patch-kadu-core_CMakeLists.txt
  head/polish/kadu/files/patch-plugins_idle_CMakeLists.txt
  head/polish/kadu/files/patch-plugins_pcspeaker_CMakeLists.txt
  head/polish/kadu/files/patch-plugins_screenshot_CMakeLists.txt