Bug 252277

Summary: cmake find_package(python3...) fails to get default python version
Product: Ports & Packages Reporter: Shane <FreeBSD>
Component: Individual Port(s)Assignee: freebsd-kde (Team) <kde>
Status: New ---    
Severity: Affects Some People CC: adridg, lbartoletti, linimon, tingox+freebsd
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
simple cmake file for testing
none
test port makefile
none
Makefile using EXACT
none
CMakeLists using EXACT none

Description Shane 2020-12-30 00:01:12 UTC
In older cmake versions, we would use
  find_package(PythonInterp 3.4 REQUIRED)
  find_package(PythonLibs 3.4 REQUIRED)

with cmake >= 3.12 we can replace that with
  find_package(Python3 3.4 REQUIRED COMPONENTS Interpreter Development)

The old method will find the python version matching the python version set in DEFAULT_VERSIONS

The newer python3 method always gets the largest python version installed, not the default version.


example using old method (CORRECT) - find_package(PythonLibs 3.4 REQUIRED)

-- Found PythonLibs: /usr/local/lib/libpython3.7m.so (found suitable version "3.7.9", minimum required is "3.4") 

change this to new method (WRONG) - find_package(Python3 3.4 REQUIRED COMPONENTS Interpreter Development)

-- Found Python3: /usr/local/bin/python3.9 (found suitable version "3.9.0", minimum required is "3.4") found components: Interpreter Development Development.Module Development.Embed 


The new way should find the same py3.7

This is using cmake 3.18.5 and still applies to 3.19.2 found in bug #251920
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2020-12-30 11:36:42 UTC
So, is this a bug against cmake, or a bug against the ports that use cmake?  It is not clear to me from the description.
Comment 2 Shane 2020-12-31 00:12:55 UTC
Against cmake, any port that calls find_package(python3 ...) will get the wrong version. At least when more than one python version is installed.

The old method using find_package(PythonInterp ...) still works as expected.

The newer method find_package(Python3 ...) does not find the default version, it always finds the newest python version. This differs from the previous methods behaviour.

This new method was added to cmake v3.12 and has not yet been adopted by every port using cmake and python.
Comment 3 Torfinn Ingolfsen 2021-03-22 18:26:13 UTC
This is still a problem with cmake 3.19.6:
root@kg-core2# pkg info cmake* python3*
cmake-3.19.6
python37-3.7.10
python38-3.8.8

trying to build and install libSavitar:
--->  Installing '/usr/ports/devel/libsavitar'
===>  Deinstalling for libSavitar
===>   libSavitar not installed, skipping
===>  Installing for libSavitar-4.5.0_5
===>  Checking if libSavitar is already installed
===>   Registering installation for libSavitar-4.5.0_5
pkg-static: Unable to access file /usr/ports/devel/libsavitar/work/stage/usr/local/lib/python3.7/site-packages/Savitar.so:No such file or directory
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/devel/libsavitar
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/libsavitar
** Command failed (exit code 1): env MAKE_JOBS_NUMBER_LIMIT=12 make reinstall
--->  Restoring the old version
--->  Installing '/var/tmp/pkg_replace.dDeaiZ/libSavitar-4.5.0_3/libSavitar-4.5.0_3.txz'
Installing libSavitar-4.5.0_3...
Extracting libSavitar-4.5.0_3: 100%
--->  Cleaning the preserved shared libraries
** Fix the problem and try again.
--->  ** [5/5] - 0 done, 4 ignored, 0 skipped, 1 failed
--->  Listing the results (!:failed)
        ! libSavitar-4.5.0_3 (install error)

and if I check the relevant directory in work:
root@kg-core2# l /usr/ports/devel/libsavitar/work/stage/usr/local/lib
./                   cmake/               libSavitar.so.0@
../                  debug/               libSavitar.so.0.1.1*
X11/                 libSavitar.so@       python3.8/
root@kg-core2# l /usr/ports/devel/libsavitar/work/stage/usr/local/lib/python3.8/
./             ../            site-packages/
root@kg-core2# l /usr/ports/devel/libsavitar/work/stage/usr/local/lib/python3.8/site-packages/
./          ../         Savitar.so*

This on 
root@kg-core2# freebsd-version -ku
11.4-RELEASE-p5
11.4-RELEASE-p5
ports tree updated yesterday (March 21st, 2021).
Comment 4 Adriaan de Groot freebsd_committer 2021-04-23 21:54:40 UTC
Assigning to kde@ as the maintainers of CMake (effectively, to me). It'd be really useful to have a couple of sample CMakeLists.txt attached to this PR that demonstrate "this works, this doesn't" so I don't have to come up with complete examples while also debugging the actual CMake module code.
Comment 5 Shane 2021-04-24 08:46:35 UTC
Created attachment 224394 [details]
simple cmake file for testing

The simplest setup is a CMakeLists.txt that only has the two python searches. Multiple python versions need to be installed.

run `cmake .` in the same dir

The old style find package finds py36
-- Found PythonInterp: /usr/local/bin/python3.6 (found suitable version "3.6.12", minimum required is "3.6") 
-- Found PythonLibs: /usr/local/lib/libpython3.6m.so (found suitable version "3.6.12", minimum required is "3.6") 


The new variation finds py39
-- Found Python3: /usr/local/bin/python3.9 (found suitable version "3.9.0", minimum required is "3.6") found components: Interpreter Development Development.Module Development.Embed 


They should find the same default py version.
Comment 6 Shane 2021-04-24 09:57:08 UTC
Created attachment 224396 [details]
test port makefile

Actually it would be better from a port, to use the ports env

Make a new port dir (misc/cmakepytest) and put the CmakeLists.txt in files as files/CMakeLists.txt.in

With this I see three versions --

-- Found PythonInterp: /usr/local/bin/python3.6 (found suitable version "3.6.12", minimum required is "3.6") 
-- Found PythonLibs: /usr/local/lib/libpython3.7m.so (found suitable version "3.7.9", minimum required is "3.6") 
-- Found Python3: /usr/local/bin/python3.9 (found suitable version "3.9.0", minimum required is "3.6") found components: Interpreter Development Development.Module Development.Embed
Comment 7 Loïc Bartoletti freebsd_committer 2021-05-06 19:31:45 UTC
Created attachment 224735 [details]
Makefile using EXACT
Comment 8 Loïc Bartoletti freebsd_committer 2021-05-06 19:32:12 UTC
Created attachment 224736 [details]
CMakeLists using EXACT
Comment 9 Loïc Bartoletti freebsd_committer 2021-05-06 19:32:44 UTC
Indeed, this method will search for the highest version available on the system.

Normally, we can use the Python3_ROOT_DIR hints, but unless I misuse it, it doesn't work properly.

cmake -DPython3_ROOT_DIR=/usr/local/lib/python3.8 should select py3.8, but cmake continue to take the highest version available (eg. 3.9 on my system).


Another way to do this, which works well, is to use EXACT.

example:
find_package(Python3 3.6 EXACT REQUIRED COMPONENTS Interpreter Development)

One possible workaround for the ports would be to patch as I did in Makefile_EXACT and CMakeLists_EXACT.txt