Bug 227939
| Summary: | devel/boost-python-libs vs ctypes.util.find_library | ||
|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Wiktor Niesiobedzki <bsd> |
| Component: | Individual Port(s) | Assignee: | freebsd-python (Nobody) <python> |
| Status: | Open --- | ||
| Severity: | Affects Some People | CC: | bapt, emaste, fcharlier, koobs, office, python |
| Priority: | --- | Keywords: | needs-qa |
| Version: | Latest | Flags: | jbeich:
maintainer-feedback+
koobs: maintainer-feedback+ |
| Hardware: | Any | ||
| OS: | Any | ||
| See Also: | https://github.com/osmcode/pyosmium/issues/55 | ||
Before adding a workaround I'd like python@ folks to weigh in. It's not hard to find ports that install lib<NAME>.so.<MAJOR>.<MINOR>.<PATCH> lib<NAME>.so.<MAJOR>.<MINOR> without lib<NAME>.so.<MAJOR> symlink. All those fail ctypes.util.find_library(<NAME>). archivers/ark astro/gpstk astro/libkgeomap audio/codec2 audio/libmusicxml audio/sndio benchmarks/nosqlbench biology/bamtools cad/kicad-devel comms/libsdr-gui comms/libsdr comms/qtel comms/svxlink databases/hiredis databases/kbibtex databases/kexi databases/linux-oracle-instantclient-basic databases/soci databases/sqlclient databases/tarantool-c deskutils/fbreader devel/adaid devel/bullet devel/cpprestsdk devel/eblob devel/gdcm devel/gflags devel/libght devel/libjson++ devel/libosmo-sccp devel/libtuntap devel/llvm38 devel/loki devel/performance devel/pyside devel/sfml devel/sfml1 devel/skalibs devel/smack devel/trio emulators/mgba games/doomsday games/kolf games/py-fife graphics/ampasCTL graphics/caffe graphics/hugin graphics/libkexiv2 graphics/libkipi graphics/mapnik graphics/ogre3d graphics/qgis graphics/waylandpp graphics/zint lang/cling lang/elan lang/execline lang/harbour lang/jimtcl lang/python35 lang/python36 lang/swi-pl mail/libcmime mail/osbf-lua mail/spmfilter-clamav mail/spmfilter math/cryptominisat math/reed-solomon math/vtk5 multimedia/vdr-plugin-xineliboutput multimedia/vdr multimedia/vid.stab multimedia/win32-codecs net-im/vacuum-im net-mgmt/icinga2 net-mgmt/snmp++ net/freerdp1 net/nanomsg net/olsrd net/yate print/epson-inkjet-printer-201401w print/epson-inkjet-printer-201601w science/dlib-cpp science/libefp security/ike security/libpwstor sysutils/s6-rc sysutils/s6 textproc/liblingoteach www/mod_md-devel x11-toolkits/fltk-devel x11-toolkits/mygui x11/leechcraft (In reply to Jan Beich from comment #1) > lang/python35 > lang/python36 This is not a typo: $ pkg install python36 $ python3.6 Python 3.6.5 (default, Apr 4 2018, 12:24:22) [GCC 4.2.1 Compatible FreeBSD Clang 4.0.0 (tags/RELEASE_400/final 297347)] on freebsd11 Type "help", "copyright", "credits" or "license" for more information. >>> from ctypes.util import find_library >>> find_library("python3.6m") >>> vs. $ pkg install python27 $ python2.7 Python 2.7.14 (default, Dec 19 2017, 15:01:12) [GCC 4.2.1 Compatible FreeBSD Clang 4.0.0 (tags/RELEASE_400/final 297347)] on freebsd11 Type "help", "copyright", "credits" or "license" for more information. >>> from ctypes.util import find_library >>> find_library("python2.7") 'libpython2.7.so.1' >>> Hi Jan, apologies for the delay. See Also: (1) https://bugs.python.org/issue23817 (remove freebsd SOVERSION special case) (2) https://github.com/osmcode/pyosmium/issues/55 (origin of this issue) (3) https://bugs.python.org/issue1610795 (original freebsd find_library python code) (1) ended up in only 3.5+, though I requested a backport to 3.4 and 2.7 so the behaviour was consistent in all versions. This explains the discrepancy between Python versions in comment 2. CC'ing bapt@ here too for opinions/thoughts, he originated the issue and patch upstream. We could backport it ourselves, and/or add symlink in the python ports, and/or modify the SOVERSION/INSTSONAME for the python port builds, *but* we'd like to avoid doing so if the better/correct/permanent/rootcause resolution is elsewhere. Correct me if my reading is off, but it sounds like we have the following options/paths, or combination thereof, so that we can have all supported freebsd/python versions work out of the box. - Fix ports that don't install symlinks to do so - Fix python ctypes/find_library to be correct/great on all FreeBSD versions. - Fix ldconfig something something in the absence of symlinks / other condition(s). Are some/all of the above what you referred to when you mentioned a workaround? I agree on the possible options laid out by koobs@. I didn't found any relevant documentation for FreeBSD, but there is something for Linux: http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html It specifies, that: lib<NAME>.so.<MAJOR> Is a fully-qualified soname for library and ldconfig is responsible (at least on Linux) on setting the link to library realname (in this case - libboost_python%%PYTHON_SUFFIX%%.so.%%BOOST_SHARED_LIB_VER%%), while libboost_python%%PYTHON_SUFFIX%%.so is only a linker name - used during compilation, not during library load. I'm not sure if FreeBSD follows the same convention for ldconfig, but if so, then either ldconfig or install scripts should ensure that fully-qualified soname is created for installed library. So I lean more towards that's multiple-port issue or base system issue. CC Ed (worked/working on lld) who may be able to shed more light on the expectations we can/should have on the linker/ldconfig across FreeBSD versions. (In reply to Kubilay Kocak from comment #3) > - Fix ports that don't install symlinks to do so .so.X.Y -> .so symlink is required by ld(1) but why do we need .so.X.Y -> .so.X or .so.X.Y.Z -> .so.X.Y -> .so.X as well? According to semantic versioning[1] major version bump is only required for incompatible changes. Boost breaks API/ABI on each minor release where API incompatibility is minor (i.e., few breaking changes) but ABI has no guarantees at all. As SONAME already encodes .so.X.Y.Z there's nothing to fix in Boost. For one, find_library() works fine on CentOS 7 without .so.X.Y.Z -> .so.X symlink. $ ls -1 /usr/lib64/libboost_python* /usr/lib64/libboost_python-mt.so.1.53.0 /usr/lib64/libboost_python.so.1.53.0 >>> find_library("boost_python") 'libboost_python.so.1.53.0' [1] https://semver.org/ (In reply to Kubilay Kocak from comment #3) > - Fix python ctypes/find_library to be correct/great on all FreeBSD versions. See also ports r322328. (In reply to Wiktor Niesiobedzki from comment #4) > ldconfig is responsible (at least on Linux) on setting the link to library realname ldconfig on FreeBSD doesn't grok SONAME unlike ld.so aka rtld-elf. Many libraries without .so.X symlinks actually use SONAME instead. As rtld-aout doesn't support SONAME maybe ldconfig was simply overlooked as ELF wasn't default on FreeBSD in the past. (In reply to Jan Beich from comment #6) > For one, find_library() works fine on CentOS 7 without .so.X.Y.Z -> .so.X symlink. Probably because find_library has another strategy available on Linux - invoking gcc in trace mode and extracting library path from there. Maybe it's the way to go to implement the same behaviour for FreeBSD (though use clang instead of gcc). For libraries that do not provide ABI compatibility, either they need to install .so.XYZ link to satisfy ldconfig requirements or ldconfig needs to be fixed to look for so.X.Y.Z. Security note in ldconfig(8) man page specifies that library is missing from hints file it will not be loaded by setuid binary. So if I'll have setuid binary linked with boost-python it will fail to run, as the linker will refuse to load boost-python as non-safe lib? (In reply to Wiktor Niesiobedzki from comment #9) > library is missing from hints file Have you actually looked in /var/run/ld-elf.so.hints? It doesn't contain library names, only search directories. (In reply to Wiktor Niesiobedzki from comment #9) > another strategy available on Linux - invoking gcc in trace mode and extracting library path from there. Why Python cannot use dlinfo()? $ cat a.c #define _GNU_SOURCE #include <dlfcn.h> #include <link.h> #include <stdio.h> int main() { void *boost, *python; struct link_map *map; /* libboost_python assumes libpython is preloaded by Python interpreter */ if (!(python = dlopen("libpython3.6m.so", RTLD_LAZY | RTLD_GLOBAL))) { printf("dlopen(python) failed\n"); return 1; } if (!(boost = dlopen("libboost_python36.so", RTLD_LAZY))) { printf("dlopen(boost) failed\n"); return 1; } dlinfo(boost, RTLD_DI_LINKMAP, &map); dlclose(boost); dlclose(python); printf("%s\n", map->l_name); return 0; } $ cc -ldl a.c $ ./a.out /usr/local/lib/libboost_python36.so Boost >= 1.70 (see ports r498851) provides .so.X symlink but SONAME still contains the full version. |
boost-python-libs installs following files: lib/libboost_python%%PYTHON_SUFFIX%%.a lib/libboost_python%%PYTHON_SUFFIX%%.so lib/libboost_python%%PYTHON_SUFFIX%%.so.%%BOOST_SHARED_LIB_VER%% What we are missing is: lib/libboost_python%%PYTHON_SUFFIX%%.so.1 Because of this, when you do: $ python3 Python 3.6.5 (default, Apr 5 2018, 01:15:08) [GCC 4.2.1 Compatible FreeBSD Clang 4.0.0 (tags/RELEASE_400/final 297347)] on freebsd11 Type "help", "copyright", "credits" or "license" for more information. >>> from ctypes.util import find_library >>> find_library("boost_python36") >>> After creating this symlink manually, there is no problem. As far as I checked all my installed libraries have symlink *.so.%%MAJOR_VERSION%%. The only exception are boost libraries, which have only .so and .so.%%MAJOR%%.%%MINOR%%. ctypes depends on ldconfig to find library. And indeed: $ ldconfig -r | grep boost_ 388:-lboost_python36.1 => /usr/local/lib/libboost_python36.so.1 (for which I've made symlink, for example boost_atomic is missing). This works on Linux, because ldconfig -p has entries regardless of missing symlink.