Bug 227939 - devel/boost-python-libs vs ctypes.util.find_library
Summary: devel/boost-python-libs vs ctypes.util.find_library
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-python (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2018-05-02 21:04 UTC by Wiktor Niesiobedzki
Modified: 2019-06-08 03:22 UTC (History)
6 users (show)

See Also:
jbeich: maintainer-feedback+
koobs: maintainer-feedback+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Wiktor Niesiobedzki 2018-05-02 21:04:25 UTC
boost-python-libs installs following files:
lib/libboost_python%%PYTHON_SUFFIX%%.a
lib/libboost_python%%PYTHON_SUFFIX%%.so
lib/libboost_python%%PYTHON_SUFFIX%%.so.%%BOOST_SHARED_LIB_VER%%

What we are missing is:
lib/libboost_python%%PYTHON_SUFFIX%%.so.1

Because of this, when you do:
$ python3
Python 3.6.5 (default, Apr  5 2018, 01:15:08)
[GCC 4.2.1 Compatible FreeBSD Clang 4.0.0 (tags/RELEASE_400/final 297347)] on freebsd11
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library
>>> find_library("boost_python36")
>>>

After creating this symlink manually, there is no problem. As far as I checked all my installed libraries have symlink *.so.%%MAJOR_VERSION%%. The only exception are boost libraries, which have only .so and .so.%%MAJOR%%.%%MINOR%%.

ctypes depends on ldconfig to find library. And indeed:
$ ldconfig -r | grep boost_
        388:-lboost_python36.1 => /usr/local/lib/libboost_python36.so.1

(for which I've made symlink, for example boost_atomic is missing).

This works on Linux, because ldconfig -p has entries regardless of missing symlink.
Comment 1 Jan Beich freebsd_committer freebsd_triage 2018-05-03 02:37:06 UTC
Before adding a workaround I'd like python@ folks to weigh in. It's not hard to find ports that install lib<NAME>.so.<MAJOR>.<MINOR>.<PATCH> lib<NAME>.so.<MAJOR>.<MINOR> without lib<NAME>.so.<MAJOR> symlink. All those fail ctypes.util.find_library(<NAME>).

  archivers/ark
  astro/gpstk
  astro/libkgeomap
  audio/codec2
  audio/libmusicxml
  audio/sndio
  benchmarks/nosqlbench
  biology/bamtools
  cad/kicad-devel
  comms/libsdr-gui
  comms/libsdr
  comms/qtel
  comms/svxlink
  databases/hiredis
  databases/kbibtex
  databases/kexi
  databases/linux-oracle-instantclient-basic
  databases/soci
  databases/sqlclient
  databases/tarantool-c
  deskutils/fbreader
  devel/adaid
  devel/bullet
  devel/cpprestsdk
  devel/eblob
  devel/gdcm
  devel/gflags
  devel/libght
  devel/libjson++
  devel/libosmo-sccp
  devel/libtuntap
  devel/llvm38
  devel/loki
  devel/performance
  devel/pyside
  devel/sfml
  devel/sfml1
  devel/skalibs
  devel/smack
  devel/trio
  emulators/mgba
  games/doomsday
  games/kolf
  games/py-fife
  graphics/ampasCTL
  graphics/caffe
  graphics/hugin
  graphics/libkexiv2
  graphics/libkipi
  graphics/mapnik
  graphics/ogre3d
  graphics/qgis
  graphics/waylandpp
  graphics/zint
  lang/cling
  lang/elan
  lang/execline
  lang/harbour
  lang/jimtcl
  lang/python35
  lang/python36
  lang/swi-pl
  mail/libcmime
  mail/osbf-lua
  mail/spmfilter-clamav
  mail/spmfilter
  math/cryptominisat
  math/reed-solomon
  math/vtk5
  multimedia/vdr-plugin-xineliboutput
  multimedia/vdr
  multimedia/vid.stab
  multimedia/win32-codecs
  net-im/vacuum-im
  net-mgmt/icinga2
  net-mgmt/snmp++
  net/freerdp1
  net/nanomsg
  net/olsrd
  net/yate
  print/epson-inkjet-printer-201401w
  print/epson-inkjet-printer-201601w
  science/dlib-cpp
  science/libefp
  security/ike
  security/libpwstor
  sysutils/s6-rc
  sysutils/s6
  textproc/liblingoteach
  www/mod_md-devel
  x11-toolkits/fltk-devel
  x11-toolkits/mygui
  x11/leechcraft
Comment 2 Jan Beich freebsd_committer freebsd_triage 2018-05-03 02:49:53 UTC
(In reply to Jan Beich from comment #1)
>  lang/python35
>  lang/python36

This is not a typo:

  $ pkg install python36
  $ python3.6
  Python 3.6.5 (default, Apr  4 2018, 12:24:22)
  [GCC 4.2.1 Compatible FreeBSD Clang 4.0.0 (tags/RELEASE_400/final 297347)] on freebsd11
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from ctypes.util import find_library
  >>> find_library("python3.6m")
  >>>

vs.

  $ pkg install python27
  $ python2.7
  Python 2.7.14 (default, Dec 19 2017, 15:01:12)
  [GCC 4.2.1 Compatible FreeBSD Clang 4.0.0 (tags/RELEASE_400/final 297347)] on freebsd11
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from ctypes.util import find_library
  >>> find_library("python2.7")
  'libpython2.7.so.1'
  >>>
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-11 08:54:07 UTC
Hi Jan, apologies for the delay.

See Also:

(1) https://bugs.python.org/issue23817 (remove freebsd SOVERSION special case)
(2) https://github.com/osmcode/pyosmium/issues/55 (origin of this issue)
(3)  https://bugs.python.org/issue1610795 (original freebsd find_library python code)

(1) ended up in only 3.5+, though I requested a backport to 3.4 and 2.7 so the behaviour was consistent in all versions. This explains the discrepancy between Python versions in comment 2. CC'ing bapt@ here too for opinions/thoughts, he originated the issue and patch upstream.

We could backport it ourselves, and/or add symlink in the python ports, and/or modify the SOVERSION/INSTSONAME for the python port builds, *but* we'd like to avoid doing so if the better/correct/permanent/rootcause resolution is elsewhere.

Correct me if my reading is off, but it sounds like we have the following options/paths, or combination thereof, so that we can have all supported freebsd/python versions work out of the box.

- Fix ports that don't install symlinks to do so
- Fix python ctypes/find_library to be correct/great on all FreeBSD versions.
- Fix ldconfig something something in the absence of symlinks / other condition(s).

Are some/all of the above what you referred to when you mentioned a workaround?
Comment 4 Wiktor Niesiobedzki 2018-05-14 17:03:05 UTC
I agree on the possible options laid out by koobs@.

I didn't found any relevant documentation for FreeBSD, but there is something for Linux:
http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

It specifies, that:
lib<NAME>.so.<MAJOR>

Is a fully-qualified soname for library and ldconfig is responsible (at least on Linux) on setting the link to library realname (in this case - libboost_python%%PYTHON_SUFFIX%%.so.%%BOOST_SHARED_LIB_VER%%), while libboost_python%%PYTHON_SUFFIX%%.so is only a linker name - used during compilation, not during library load.

I'm not sure if FreeBSD follows the same convention for ldconfig, but if so, then either ldconfig or install scripts should ensure that fully-qualified soname is created for installed library. So I lean more towards that's multiple-port issue or base system issue.
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-15 04:00:58 UTC
CC Ed (worked/working on lld) who may be able to shed more light on the expectations we can/should have on the linker/ldconfig across FreeBSD versions.
Comment 6 Jan Beich freebsd_committer freebsd_triage 2018-09-09 22:47:09 UTC
(In reply to Kubilay Kocak from comment #3)
> - Fix ports that don't install symlinks to do so

.so.X.Y -> .so symlink is required by ld(1) but why do we need .so.X.Y -> .so.X or .so.X.Y.Z -> .so.X.Y -> .so.X as well? According to semantic versioning[1] major version bump is only required for incompatible changes. Boost breaks API/ABI on each minor release where API incompatibility is minor (i.e., few breaking changes) but ABI has no guarantees at all. As SONAME already encodes .so.X.Y.Z there's nothing to fix in Boost.

For one, find_library() works fine on CentOS 7 without .so.X.Y.Z -> .so.X symlink.

$ ls -1 /usr/lib64/libboost_python*
/usr/lib64/libboost_python-mt.so.1.53.0
/usr/lib64/libboost_python.so.1.53.0

>>> find_library("boost_python")
'libboost_python.so.1.53.0'

[1] https://semver.org/
Comment 7 Jan Beich freebsd_committer freebsd_triage 2018-09-09 23:25:02 UTC
(In reply to Kubilay Kocak from comment #3)
> - Fix python ctypes/find_library to be correct/great on all FreeBSD versions.

See also ports r322328.
Comment 8 Jan Beich freebsd_committer freebsd_triage 2018-09-09 23:51:29 UTC
(In reply to Wiktor Niesiobedzki from comment #4)
> ldconfig is responsible (at least on Linux) on setting the link to library realname

ldconfig on FreeBSD doesn't grok SONAME unlike ld.so aka rtld-elf. Many libraries without .so.X symlinks actually use SONAME instead. As rtld-aout doesn't support SONAME maybe ldconfig was simply overlooked as ELF wasn't default on FreeBSD in the past.
Comment 9 Wiktor Niesiobedzki 2018-09-10 17:00:02 UTC
(In reply to Jan Beich from comment #6)
> For one, find_library() works fine on CentOS 7 without .so.X.Y.Z -> .so.X symlink.

Probably because find_library has another strategy available on Linux - invoking gcc in trace mode and extracting library path from there.

Maybe it's the way to go to implement the same behaviour for FreeBSD (though use clang instead of gcc).

For libraries that do not provide ABI compatibility, either they need to install .so.XYZ link to satisfy ldconfig requirements or ldconfig needs to be fixed to look for so.X.Y.Z. Security note in ldconfig(8) man page specifies that library is missing from hints file it will not be loaded by setuid binary. So if I'll have setuid binary linked with boost-python it will fail to run, as the linker will refuse to load boost-python as non-safe lib?
Comment 10 Jan Beich freebsd_committer freebsd_triage 2018-09-11 09:27:15 UTC
(In reply to Wiktor Niesiobedzki from comment #9)
> library is missing from hints file

Have you actually looked in /var/run/ld-elf.so.hints? It doesn't contain library names, only search directories.
Comment 11 Jan Beich freebsd_committer freebsd_triage 2018-09-11 10:34:13 UTC
(In reply to Wiktor Niesiobedzki from comment #9)
> another strategy available on Linux - invoking gcc in trace mode and extracting library path from there.

Why Python cannot use dlinfo()?

$ cat a.c
#define _GNU_SOURCE
#include <dlfcn.h>
#include <link.h>
#include <stdio.h>

int main()
{
  void *boost, *python;
  struct link_map *map;

  /* libboost_python assumes libpython is preloaded by Python interpreter */
  if (!(python = dlopen("libpython3.6m.so", RTLD_LAZY | RTLD_GLOBAL))) {
    printf("dlopen(python) failed\n");
    return 1;
  }

  if (!(boost = dlopen("libboost_python36.so", RTLD_LAZY))) {
    printf("dlopen(boost) failed\n");
    return 1;
  }
  dlinfo(boost, RTLD_DI_LINKMAP, &map);
  dlclose(boost);
  dlclose(python);

  printf("%s\n", map->l_name);

  return 0;
}

$ cc -ldl a.c
$ ./a.out
/usr/local/lib/libboost_python36.so
Comment 12 Jan Beich freebsd_committer freebsd_triage 2019-06-08 03:22:50 UTC
Boost >= 1.70 (see ports r498851) provides .so.X symlink but SONAME still contains the full version.