libtool (ltdl) includes a check to determine whether shared libraries on the host OS correctly handle dependencies or whether libltdl needs to manually manage shared library dependencies. This check is implemented by hard-wiring the answer for various OS's, rather than testing the actual OS behaviour using a test case. If an OS is not listed, libltdl assumes that the OS cannot handle shared library dependencies. FreeBSD is not listed in the known OSs and libltdl therefore defaults to manual dependency loading. This results in massive overheads in loading programs that use many shared libraries. As an example, gnucash uses 122 shared libraries and took 15 minutes of system time to start on an XP-1800 and a system call trace shows 24e6 open() calls. With libltdl_cv_sys_dlopen_deplibs=yes, the startup time drops to 10 seconds. The critical change is the following change in output from configure: -checking whether deplibs are loaded by dlopen... unknown +checking whether deplibs are loaded by dlopen... yes This bug also affects other programs with ltdl embedded in them. Based on my collection of ports, this includes guile (prior to 1.7.2) and ImageMagick-6.2.5.5. Fix: Ideally, libtool should implement a test case to determine whether an OS tracks shared library dependencies or now. The following patches just flag that FreeBSD does track dependencies. Any one of the following patches should fix the problem on libtool and any application that embeds libltdl. How-To-Repeat: Start gnucash and watch the system call rate.
Responsible Changed From-To: freebsd-ports-bugs->ade Over to the libtool maintainer
[ FWIW: the message has not appeared on bug-libtool yet, although I approved it several hours ago; seems the list server is still being worked on; anyway this is: http://www.freebsd.org/cgi/query-pr.cgi?pr=94826 ] Hi Peter, * Peter Jeremy wrote on Wed, Mar 22, 2006 at 09:27:31AM CET: > > libtool (ltdl) includes a check to determine whether shared > libraries on the host OS correctly handle dependencies or whether > libltdl needs to manually manage shared library dependencies. > > This check is implemented by hard-wiring the answer for various > OS's, rather than testing the actual OS behaviour using a test case. > If an OS is not listed, libltdl assumes that the OS cannot handle > shared library dependencies. FreeBSD is not listed in the known > OSs and libltdl therefore defaults to manual dependency loading. Confirmed. > This results in massive overheads in loading programs that use > many shared libraries. As an example, gnucash uses 122 shared > libraries and took 15 minutes of system time to start on an XP-1800 > and a system call trace shows 24e6 open() calls. > With libltdl_cv_sys_dlopen_deplibs=yes, the startup time drops > to 10 seconds. Ouch. That's disastrous. > The critical change is the following change in output from configure: > -checking whether deplibs are loaded by dlopen... unknown > +checking whether deplibs are loaded by dlopen... yes Yes. > This bug also affects other programs with ltdl embedded in them. > Based on my collection of ports, this includes guile (prior to 1.7.2) > and ImageMagick-6.2.5.5. Yes. > >Fix: > Ideally, libtool should implement a test case to determine whether > an OS tracks shared library dependencies or now. This should definitely be fixed. However, I really don't know how to write such a test so that it also works well in the cross compiling case. So for that we would still need the table. But at least in the non-cross-compiling case it would also fix the cases where we are not sure whether the system supports it right, or where the system needs some patch/upgrade to fix this issue. I'll put adding such a test on the TODO list. I'll also put reworking the manual loading algorithm on the TODO list, it shouldn't scale nonlinearly in the number of loaded deplibs, however many modules we open. > The following > patches just flag that FreeBSD does track dependencies. Any one > of the following patches should fix the problem on libtool and any > application that embeds libltdl. Thanks for the patch. A question before I apply it: this should work on DragonFly as well, right? Could somebody confirm this, to avoid another bug report to this end? Also, has the FreeBSD rtld always supported loading dependent libraries upon dlopen() (including honoring DT_RPATH entries in the module) or was this not working at some point in the past? Cheers, and thanks very much for the bug report, Ralf > --- ltdl.m4~ Mon Dec 19 08:53:17 2005 > +++ ltdl.m4 Wed Mar 22 18:43:11 2006 > @@ -127,6 +127,9 @@ > # If you are looking for one http://www.opendarwin.org/projects/dlcompat > libltdl_cv_sys_dlopen_deplibs=yes > ;; > + freebsd*) > + libltdl_cv_sys_dlopen_deplibs=yes > + ;; > gnu* | linux* | kfreebsd*-gnu | knetbsd*-gnu) > # GNU and its variants, using gnu ld.so (Glibc) > libltdl_cv_sys_dlopen_deplibs=yes
I should acknowledge that the problem and fix were pointed out by joerg@britannica.bec.de On Wed, 2006-Mar-22 17:28:11 +0100, Ralf Wildenhues wrote: >I'll put adding such a test on the TODO list. I'll also put reworking >the manual loading algorithm on the TODO list, it shouldn't scale >nonlinearly in the number of loaded deplibs, however many modules we >open. I've still got a trace of all calls to lt_dlopenext() during gnucash startup but I've cleaned up the more detailed debugging output that I collected (though I could probably re-create it). I suspect it's a combination of: - both dlopen() and lt_dlopen() are recursively attempting to load dependencies - The .la file can contain duplicate dependencies (which don't appear to be stripped out) - It looks like duplicate entries in the library search path (inherited from LD_LIBRARY_PATH and added via .la) aren't deleted - Tracking lt_dlopen/lt_dlclose shows that libraries are being opened and closed multiple times during startup. Before joerg pointed me in the right direction, I was looking at adding caching to lt_dlopenext() or lt_dlopen() but hadn't worked out how to correctly track reference counts across open/close calls. >Thanks for the patch. A question before I apply it: this should work on >DragonFly as well, right? Could somebody confirm this, to avoid another >bug report to this end? joerg found the problem using KDE on DragonFly. >Also, has the FreeBSD rtld always supported loading dependent libraries >upon dlopen() (including honoring DT_RPATH entries in the module) or was >this not working at some point in the past? AFAIK, it's worked since FreeBSD transitioned to ELF (FreeBSD3.0). I'm not sure about the a.out rtld but FreeBSD 2.x is ancient history. -- Peter Jeremy
* Peter Jeremy wrote on Wed, Mar 22, 2006 at 07:42:33PM CET: > I should acknowledge that the problem and fix were pointed out by > joerg@britannica.bec.de Thanks, I'll mention Joerg. > On Wed, 2006-Mar-22 17:28:11 +0100, Ralf Wildenhues wrote: > >I'll put adding such a test on the TODO list. I'll also put reworking > >the manual loading algorithm on the TODO list, it shouldn't scale > >nonlinearly in the number of loaded deplibs, however many modules we > >open. > > I've still got a trace of all calls to lt_dlopenext() during gnucash > startup but I've cleaned up the more detailed debugging output that > I collected (though I could probably re-create it). > > I suspect it's a combination of: > - both dlopen() and lt_dlopen() are recursively attempting to load > dependencies Right. > - The .la file can contain duplicate dependencies (which don't appear > to be stripped out) This is "normal": there are systems where you actually need to link each library against all its deplibs. So we store all of them. The linker and runtime linker on FreeBSD both follow DT_NEEDED entries and look in DT_RPATH entries inside libraries for finding and loading indirect library dependencies, though (I think). So it should be possible to improve quite a bit upon your 10 seconds startup time, by changing Libtool to not link in indirect deplibs on FreeBSD (and DragonFly). This would be similar to the changes that Debian currently has in their Libtool package. We plan to fix need_all_deplibs=no (it will have a different name then, and it has a couple of known issues at the moment), and then make it available for all systems that support this. (You'd have to relink all libraries once for this, though.) > - It looks like duplicate entries in the library search path > (inherited from LD_LIBRARY_PATH and added via .la) aren't deleted Ah, ok. This is another limitation in libltdl. > - Tracking lt_dlopen/lt_dlclose shows that libraries are being opened > and closed multiple times during startup. Hmm. Surely for FreeBSD this should be a non-issue now; but for systems where we _do_ need to load all deplibs, I'm not sure whether we can get around unloading them in the case that the module itself could not be opened. After all, those deplibs could have been the wrong ones to use in the first place, but we're not able to know until we've tried loading the module itself. Luckily this issue is important on very few systems only, today. > Before joerg pointed me in the right direction, I was looking at > adding caching to lt_dlopenext() or lt_dlopen() but hadn't worked out > how to correctly track reference counts across open/close calls. I think that should be possible, but then again the details may depend on the system in question. I'll try to test on Tru64 when I have some more time at hand. Cheers, Ralf
* Peter Jeremy wrote on Wed, Mar 22, 2006 at 07:42:33PM CET: > On Wed, 2006-Mar-22 17:28:11 +0100, Ralf Wildenhues wrote: > > >Thanks for the patch. A question before I apply it: this should work on > >DragonFly as well, right? Could somebody confirm this, to avoid another > >bug report to this end? > > joerg found the problem using KDE on DragonFly. > > >Also, has the FreeBSD rtld always supported loading dependent libraries > >upon dlopen() (including honoring DT_RPATH entries in the module) or was > >this not working at some point in the past? > > AFAIK, it's worked since FreeBSD transitioned to ELF (FreeBSD3.0). I'm > not sure about the a.out rtld but FreeBSD 2.x is ancient history. OK, thanks. FWIW, for the historically interested, http://www.freebsd.org/cgi/cvsweb.cgi/src/libexec/rtld-aout/Attic/rtld.c?hideattic=0 indicates this to be working in aout from rev. 1.25, backported to FreeBSD 2.1. I've applied the following patches into Libtool branch-1-5 and HEAD, respectively. Thanks again. Cheers, Ralf branch-1-5: * ltdl.m4 (AC_LTDL_SYS_DLOPEN_DEPLIBS) [ freebsd, dragonfly ]: Set libltdl_cv_sys_dlopen_deplibs to yes. Fixes excessive lt_dlopen times on these systems. * NEWS, THANKS: Update. Bug reported by Peter Jeremy <peterjeremy@optushome.com.au>, patch by Joerg Sonnenberger <joerg@netbsd.org>. Index: NEWS =================================================================== RCS file: /cvsroot/libtool/libtool/NEWS,v retrieving revision 1.109.2.44 diff -u -r1.109.2.44 NEWS --- NEWS 16 Mar 2006 20:48:52 -0000 1.109.2.44 +++ NEWS 23 Mar 2006 19:27:05 -0000 @@ -6,6 +6,9 @@ * Fix regression on OpenBSD, NetBSD, DragonFly, and other systems with hardcode_direct=yes that wrongly removed paths to uninstalled libraries during link mode. +* Let libltdl know that FreeBSD and DragonFly dlopen causes dependent + modules to be loaded. This fixes excessive load times for modules + with large library dependency graphs. * Bug Fixes. New in 1.5.22: 2005-12-18; CVS version 1.5.21a, Libtool team: Index: THANKS =================================================================== RCS file: /cvsroot/libtool/libtool/THANKS,v retrieving revision 1.34.2.17 diff -u -r1.34.2.17 THANKS --- THANKS 16 Mar 2006 20:48:52 -0000 1.34.2.17 +++ THANKS 23 Mar 2006 19:27:05 -0000 @@ -102,6 +102,7 @@ Patrick Welche prlw1@newn.cam.ac.uk Paul Eggert eggert@twinsun.com Peter Eisentraut peter_e@gmx.net + Peter Jeremy peterjeremy@optushome.com.au Rainer Orth ro@TechFak.Uni-Bielefeld.DE Ralf Menzel menzel@ls6.cs.uni-dortmund.de Robert Ögren lists@roboros.com Index: ltdl.m4 =================================================================== RCS file: /cvsroot/libtool/libtool/Attic/ltdl.m4,v retrieving revision 1.47.2.11 diff -u -r1.47.2.11 ltdl.m4 --- ltdl.m4 18 Dec 2005 22:14:06 -0000 1.47.2.11 +++ ltdl.m4 23 Mar 2006 19:27:05 -0000 @@ -127,6 +127,9 @@ # If you are looking for one http://www.opendarwin.org/projects/dlcompat libltdl_cv_sys_dlopen_deplibs=yes ;; + freebsd* | dragonfly*) + libltdl_cv_sys_dlopen_deplibs=yes + ;; gnu* | linux* | kfreebsd*-gnu | knetbsd*-gnu) # GNU and its variants, using gnu ld.so (Glibc) libltdl_cv_sys_dlopen_deplibs=yes HEAD: * libltdl/m4/ltdl.m4 (LT_SYS_DLOPEN_DEPLIBS) [ freebsd ] [ dragonfly ]: Set libltdl_cv_sys_dlopen_deplibs to yes. Fixes excessive lt_dlopen times on these systems. * NEWS, THANKS: Update. Bug reported by Peter Jeremy <peterjeremy@optushome.com.au>, patch by Joerg Sonnenberger <joerg@netbsd.org>. Index: NEWS =================================================================== RCS file: /cvsroot/libtool/libtool/NEWS,v retrieving revision 1.191 diff -u -r1.191 NEWS --- NEWS 12 Jan 2006 22:02:55 -0000 1.191 +++ NEWS 23 Mar 2006 09:47:43 -0000 @@ -20,7 +20,7 @@ * Support for linux-dietlibc (`diet' as well as `diet-dyn', separately). * Shell optimizations which break use of the stdin file descriptor in libtool. * `libtoolize --install' now also installs `install-sh'. -* Support (mostly) for DragonFly BSD. +* Support for DragonFly BSD, improved support for FreeBSD. * Allow shell special characters like `$' in source file names, but not in object names, to enhance GCJ support. * Detection of compiler wrappers like distcc/ccache and $host_alias prefix. Index: THANKS =================================================================== RCS file: /cvsroot/libtool/libtool/THANKS,v retrieving revision 1.54 diff -u -r1.54 THANKS --- THANKS 16 Mar 2006 20:47:54 -0000 1.54 +++ THANKS 23 Mar 2006 09:47:43 -0000 @@ -102,6 +102,7 @@ Patrick Welche prlw1@newn.cam.ac.uk Paul Eggert eggert@twinsun.com Peter Eisentraut peter_e@gmx.net + Peter Jeremy peterjeremy@optushome.com.au Rainer Orth ro@TechFak.Uni-Bielefeld.DE Ralf Menzel menzel@ls6.cs.uni-dortmund.de Robert Ögren lists@roboros.com Index: libltdl/m4/ltdl.m4 =================================================================== RCS file: /cvsroot/libtool/libtool/libltdl/m4/ltdl.m4,v retrieving revision 1.25 diff -u -r1.25 ltdl.m4 --- libltdl/m4/ltdl.m4 20 Nov 2005 08:45:54 -0000 1.25 +++ libltdl/m4/ltdl.m4 23 Mar 2006 09:47:43 -0000 @@ -341,6 +341,9 @@ # If you are looking for one http://www.opendarwin.org/projects/dlcompat lt_cv_sys_dlopen_deplibs=yes ;; + freebsd* | dragonfly*) + lt_cv_sys_dlopen_deplibs=yes + ;; gnu* | linux* | kfreebsd*-gnu | knetbsd*-gnu) # GNU and its variants, using gnu ld.so (Glibc) lt_cv_sys_dlopen_deplibs=yes
On Wed, Mar 22, 2006 at 05:28:11PM +0100, Ralf Wildenhues wrote: > * Peter Jeremy wrote on Wed, Mar 22, 2006 at 09:27:31AM CET: > > >Fix: > > Ideally, libtool should implement a test case to determine whether > > an OS tracks shared library dependencies or now. > > This should definitely be fixed. However, I really don't know how to > write such a test so that it also works well in the cross compiling > case. So for that we would still need the table. > > But at least in the non-cross-compiling case it would also fix the cases > where we are not sure whether the system supports it right, or where the > system needs some patch/upgrade to fix this issue. A native build could use the test but compare the result with the tabulated value used for cross-compiling, perhaps asking the user to report discrepancies.
Hi Noah, * Noah Misch wrote on Fri, Mar 24, 2006 at 01:45:33AM CET: > On Wed, Mar 22, 2006 at 05:28:11PM +0100, Ralf Wildenhues wrote: > > * Peter Jeremy wrote on Wed, Mar 22, 2006 at 09:27:31AM CET: > > > Ideally, libtool should implement a test case to determine whether > > > an OS tracks shared library dependencies or now. > > > > This should definitely be fixed. However, I really don't know how to > > write such a test so that it also works well in the cross compiling > > case. So for that we would still need the table. > > > > But at least in the non-cross-compiling case it would also fix the cases > > where we are not sure whether the system supports it right, or where the > > system needs some patch/upgrade to fix this issue. > > A native build could use the test but compare the result with the tabulated > value used for cross-compiling, perhaps asking the user to report discrepancies. Yes. But then I think it's not so much better than a good testsuite test that reports this, but carries the overhead that every package has to ship with this test. Let's be realistic here: Libtool is a package that pretty much needs to be adapted for each new system it should support (except that static libraries should work out-of-the-box); this was recognized a long time ago, and consequently, it has very few runtime tests (compared to the number of decisions) in its configuration, which at least takes away some of the configure-time-overhead burden away from the end user. So, when we accept the fact that Libtool needs active porting, the remaining point is IMVHO that we should improve the test suite insofar that it has a chance to expose as many missing or wrong settings. Then a passing testsuite is a good indication of a good port. Actually, I think we're moving in the right direction with the test suite of CVS HEAD Libtool. Also please note that a portable way of testing above feature would incur quite a large overhead: basically the libltdl loaders would need to be provided from within `configure', the right one compiled, and then the test itself. I don't think that is such a good idea. Cheers, Ralf
I haven't seen any action with this bug for a while, but it would be really great to have it fixed. I've built a set of patches for the following ports: lang/guile devel/libtool15 devel/libltdl15 The patch appears to solve the GnuCash problem locally: http://www.varju.ca/alex/freebsd/patch-94826.tgz Alex.
What the status on this bug? -- Anish Mistry amistry@am-productions.biz AM Productions http://am-productions.biz/
On Thu, 2006-Jun-29 16:18:43 -0400, Anish Mistry wrote: >What the status on this bug? I have been running with the patch since March with no problems. It has been imported into the upstream libtool. What is now required is to upgrade devel/libtool15 and any ports that embed their own libltdl (guile that I know of). -- Peter Jeremy
On Jun 30, 2006, at 07:50 , Peter Jeremy wrote: > > I have been running with the patch since March with no problems. > It has > been imported into the upstream libtool. What is now required is to > upgrade devel/libtool15 and any ports that embed their own libltdl > (guile that I know of). The question now becomes whether we either wait for a new libtool/ libltdl release, or incorporate the patches into the ports/ version, along with hacking up any other ports that use their own versions of libltdl. I'm open to suggestions on this one. -aDe
State Changed From-To: open->analyzed Depending on how things work out with the libtool-2.0 release, I'm going to hold off on this as a global libtool change for now. There is other autotools-related work that I need to do that will require a full set of package builds to be run, so hopefully I can merge things at that point.
How about we have bsd.autotools.mk just set the autoconf cache variable to yes if the port depends on libltdl. That should avoid having to patch all the problem ports, yet provide a usable workaround until the next libtool release etc. --- Mk/bsd.autotools.mk 5 Jul 2006 02:18:08 -0000 1.24 +++ Mk/bsd.autotools.mk 3 Sep 2006 03:55:59 -0000 @@ -176,6 +176,7 @@ # .if defined(AUTOTOOL_libltdl) LIB_DEPENDS+= ltdl.4:${PORTSDIR}/devel/libltdl15 +CONFIGURE_ENV+= libltdl_cv_sys_dlopen_deplibs=yes .endif .if defined(AUTOTOOL_libtool)
State Changed From-To: analyzed->closed Committed, thanks!
ade 2007-01-31 01:45:16 UTC FreeBSD ports repository Modified files: devel/libltdl15 Makefile devel/libtool15 Makefile Added files: devel/libtool15/files patch-libltdl-acinclude.m4 patch-ltdl.m4 Log: Remove massive overheads from loading programs that use many libtool-generated shared libraries by forcing libtool into understanding that FreeBSD can correctly handle shared library dependencies. This code already exists in the libtool CVS repository but without knowing the release schedule (if any) for a new libtool 1.5.x series, and the exceptional performance gains (gnucash for example uses 122 shared libraries and with things rebuilt now starts up in ~10 seconds as opposed to 15 *minutes*). Further, a number of people have assisted in extensive testing of this code in a series of different environments, so I am therefore adding this patch to the tree. PR: 94826 Submitted by: Peter Jeremy <peterjeremy@optushome.com.au> Tested by: too many to list - thank you all Revision Changes Path 1.8 +1 -1 ports/devel/libltdl15/Makefile 1.54 +1 -1 ports/devel/libtool15/Makefile 1.1 +8 -0 ports/devel/libtool15/files/patch-libltdl-acinclude.m4 (new) 1.1 +8 -0 ports/devel/libtool15/files/patch-ltdl.m4 (new) _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"