Bug 252277 - cmake find_package(python3...) fails to get default python version
Summary: cmake find_package(python3...) fails to get default python version
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-kde (Team)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-12-30 00:01 UTC by Shane
Modified: 2021-05-06 19:32 UTC (History)
4 users (show)

See Also:


Attachments
simple cmake file for testing (214 bytes, text/plain)
2021-04-24 08:46 UTC, Shane
no flags Details
test port makefile (201 bytes, text/plain)
2021-04-24 09:57 UTC, Shane
no flags Details
Makefile using EXACT (281 bytes, text/plain)
2021-05-06 19:31 UTC, Loïc Bartoletti
no flags Details
CMakeLists using EXACT (152 bytes, text/plain)
2021-05-06 19:32 UTC, Loïc Bartoletti
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Shane 2020-12-30 00:01:12 UTC
In older cmake versions, we would use
  find_package(PythonInterp 3.4 REQUIRED)
  find_package(PythonLibs 3.4 REQUIRED)

with cmake >= 3.12 we can replace that with
  find_package(Python3 3.4 REQUIRED COMPONENTS Interpreter Development)

The old method will find the python version matching the python version set in DEFAULT_VERSIONS

The newer python3 method always gets the largest python version installed, not the default version.


example using old method (CORRECT) - find_package(PythonLibs 3.4 REQUIRED)

-- Found PythonLibs: /usr/local/lib/libpython3.7m.so (found suitable version "3.7.9", minimum required is "3.4") 

change this to new method (WRONG) - find_package(Python3 3.4 REQUIRED COMPONENTS Interpreter Development)

-- Found Python3: /usr/local/bin/python3.9 (found suitable version "3.9.0", minimum required is "3.4") found components: Interpreter Development Development.Module Development.Embed 


The new way should find the same py3.7

This is using cmake 3.18.5 and still applies to 3.19.2 found in bug #251920
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2020-12-30 11:36:42 UTC
So, is this a bug against cmake, or a bug against the ports that use cmake?  It is not clear to me from the description.
Comment 2 Shane 2020-12-31 00:12:55 UTC
Against cmake, any port that calls find_package(python3 ...) will get the wrong version. At least when more than one python version is installed.

The old method using find_package(PythonInterp ...) still works as expected.

The newer method find_package(Python3 ...) does not find the default version, it always finds the newest python version. This differs from the previous methods behaviour.

This new method was added to cmake v3.12 and has not yet been adopted by every port using cmake and python.
Comment 3 Torfinn Ingolfsen 2021-03-22 18:26:13 UTC
This is still a problem with cmake 3.19.6:
root@kg-core2# pkg info cmake* python3*
cmake-3.19.6
python37-3.7.10
python38-3.8.8

trying to build and install libSavitar:
--->  Installing '/usr/ports/devel/libsavitar'
===>  Deinstalling for libSavitar
===>   libSavitar not installed, skipping
===>  Installing for libSavitar-4.5.0_5
===>  Checking if libSavitar is already installed
===>   Registering installation for libSavitar-4.5.0_5
pkg-static: Unable to access file /usr/ports/devel/libsavitar/work/stage/usr/local/lib/python3.7/site-packages/Savitar.so:No such file or directory
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/devel/libsavitar
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/libsavitar
** Command failed (exit code 1): env MAKE_JOBS_NUMBER_LIMIT=12 make reinstall
--->  Restoring the old version
--->  Installing '/var/tmp/pkg_replace.dDeaiZ/libSavitar-4.5.0_3/libSavitar-4.5.0_3.txz'
Installing libSavitar-4.5.0_3...
Extracting libSavitar-4.5.0_3: 100%
--->  Cleaning the preserved shared libraries
** Fix the problem and try again.
--->  ** [5/5] - 0 done, 4 ignored, 0 skipped, 1 failed
--->  Listing the results (!:failed)
        ! libSavitar-4.5.0_3 (install error)

and if I check the relevant directory in work:
root@kg-core2# l /usr/ports/devel/libsavitar/work/stage/usr/local/lib
./                   cmake/               libSavitar.so.0@
../                  debug/               libSavitar.so.0.1.1*
X11/                 libSavitar.so@       python3.8/
root@kg-core2# l /usr/ports/devel/libsavitar/work/stage/usr/local/lib/python3.8/
./             ../            site-packages/
root@kg-core2# l /usr/ports/devel/libsavitar/work/stage/usr/local/lib/python3.8/site-packages/
./          ../         Savitar.so*

This on 
root@kg-core2# freebsd-version -ku
11.4-RELEASE-p5
11.4-RELEASE-p5
ports tree updated yesterday (March 21st, 2021).
Comment 4 Adriaan de Groot freebsd_committer 2021-04-23 21:54:40 UTC
Assigning to kde@ as the maintainers of CMake (effectively, to me). It'd be really useful to have a couple of sample CMakeLists.txt attached to this PR that demonstrate "this works, this doesn't" so I don't have to come up with complete examples while also debugging the actual CMake module code.
Comment 5 Shane 2021-04-24 08:46:35 UTC
Created attachment 224394 [details]
simple cmake file for testing

The simplest setup is a CMakeLists.txt that only has the two python searches. Multiple python versions need to be installed.

run `cmake .` in the same dir

The old style find package finds py36
-- Found PythonInterp: /usr/local/bin/python3.6 (found suitable version "3.6.12", minimum required is "3.6") 
-- Found PythonLibs: /usr/local/lib/libpython3.6m.so (found suitable version "3.6.12", minimum required is "3.6") 


The new variation finds py39
-- Found Python3: /usr/local/bin/python3.9 (found suitable version "3.9.0", minimum required is "3.6") found components: Interpreter Development Development.Module Development.Embed 


They should find the same default py version.
Comment 6 Shane 2021-04-24 09:57:08 UTC
Created attachment 224396 [details]
test port makefile

Actually it would be better from a port, to use the ports env

Make a new port dir (misc/cmakepytest) and put the CmakeLists.txt in files as files/CMakeLists.txt.in

With this I see three versions --

-- Found PythonInterp: /usr/local/bin/python3.6 (found suitable version "3.6.12", minimum required is "3.6") 
-- Found PythonLibs: /usr/local/lib/libpython3.7m.so (found suitable version "3.7.9", minimum required is "3.6") 
-- Found Python3: /usr/local/bin/python3.9 (found suitable version "3.9.0", minimum required is "3.6") found components: Interpreter Development Development.Module Development.Embed
Comment 7 Loïc Bartoletti freebsd_committer 2021-05-06 19:31:45 UTC
Created attachment 224735 [details]
Makefile using EXACT
Comment 8 Loïc Bartoletti freebsd_committer 2021-05-06 19:32:12 UTC
Created attachment 224736 [details]
CMakeLists using EXACT
Comment 9 Loïc Bartoletti freebsd_committer 2021-05-06 19:32:44 UTC
Indeed, this method will search for the highest version available on the system.

Normally, we can use the Python3_ROOT_DIR hints, but unless I misuse it, it doesn't work properly.

cmake -DPython3_ROOT_DIR=/usr/local/lib/python3.8 should select py3.8, but cmake continue to take the highest version available (eg. 3.9 on my system).


Another way to do this, which works well, is to use EXACT.

example:
find_package(Python3 3.6 EXACT REQUIRED COMPONENTS Interpreter Development)

One possible workaround for the ports would be to patch as I did in Makefile_EXACT and CMakeLists_EXACT.txt