Created attachment 143686 [details] patch During staging libtool may relink libA when it links to another library in the work directory libB. The reason is that libA created during build phase has its runpath set to the location of libB in the work directory. This allows running an executable that links to libA from within the work directory. The relink removes this extra runpath. Executables that link to a library in the work directory would normally also need to have their runpath set to allow running them from the work directory, but instead libtool creates a wrapper script that runs the real executable with LD_LIBRARY_PATH set to the location of the library. When libtool relinks libA it replaces "libB.la" on the linker command line with "-L${STAGEDIR}${PREFIX}/lib -lB" with the intention to link to the libB in the stage directory but this -L flag isn't necessarily the first so another libB may be linked instead. For instance, there may be "libC.la" on the linker command line before "libB.la" and the dependency_libs field in libC.la may contain "-L/usr/local/lib -lintl" because it uses gettext. In this case libB from /usr/local/lib may be picked up. Rather than fixing the order of these -L flags which isn't always possible I think, the attached patch eliminates relinking by modifying ltmain.sh. The first sed command prevents that .la files contain a relink_command so they will never be relinked. The second sed command makes sure that libraries don't get the extra runpath during build phase. This should still allow running executables that link to these libraries from the work directory because due to libtool overlinking the executables will link to all libraries they depend on and any library from the work directory will have its location added to LD_LIBRARY_PATH by the executables' wrapper scripts. An exp-run is needed to make sure this is really the case. These sed commands apply to libtool versions 1.4 and up. It doesn't look like earlier versions ever relink libraries. (This fixes building devel/apr1 in unclean environments.)
That looks good to me an exp-run is needed
Take
I noticed in the exp-run results that gstreamer-plugins fails when linking an executable with libgstaudio which in turn links to libgstpbutils. The linker complains that it cannot find libgstpbutils. This patch has removed the location of libgstpbutils from the runpath of libgstaudio. The location of libgstpbutils does appear in libgstaudio.la and I expected libtool to "overlink" the executable with libgstpbutils this way, but USES=libtool also sets link_all_deplibs=no which eliminates some overlinking for executables. Back to the drawing board I guess.
It turns out that gstreamer uses libtool from Debian which contains extra changes. The proposed patch works correctly with standard libtool. This is why we patch individual ports and don't patch devel/libtool. You leak these kind of changes out to other projects if you do that...
As you noticed, the results of the exp-run are available at: http://package23.nyi.freebsd.org/build.html?mastername=91amd64-default-pr190941&build=2014-06-12_05h45m13s libreoffice did not build on baseline either yesterday, so no possible to tell if the patch affected it or not 7 new failures and more than 1600 skipped in total: + {"origin"=>"audio/gnomoradio", "pkgname"=>"gnomoradio-0.15.1_17", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"graphics/libchamplain", "pkgname"=>"libchamplain-0.8.1_5", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"multimedia/gstreamer-plugins", "pkgname"=>"gstreamer-plugins-0.10.36_4,3", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"multimedia/gstreamer1-plugins", "pkgname"=>"gstreamer1-plugins-1.2.4", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"net-mgmt/netxms", "pkgname"=>"netxms-1.2.14", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"science/gchemutils", "pkgname"=>"gchemutils-0.14.7_2", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"security/gnome-keyring", "pkgname"=>"gnome-keyring-2.32.1_4", "phase"=>"build", "errortype"=>"linker_error"} Error logs: http://package23.nyi.freebsd.org/data/91amd64-default-pr190941/2014-06-12_05h45m13s/logs/errors/gnomoradio-0.15.1_17.log http://package23.nyi.freebsd.org/data/91amd64-default-pr190941/2014-06-12_05h45m13s/logs/errors/libchamplain-0.8.1_5.log http://package23.nyi.freebsd.org/data/91amd64-default-pr190941/2014-06-12_05h45m13s/logs/errors/gstreamer-plugins-0.10.36_4,3.log http://package23.nyi.freebsd.org/data/91amd64-default-pr190941/2014-06-12_05h45m13s/logs/errors/gstreamer1-plugins-1.2.4.log http://package23.nyi.freebsd.org/data/91amd64-default-pr190941/2014-06-12_05h45m13s/logs/errors/netxms-1.2.14.log http://package23.nyi.freebsd.org/data/91amd64-default-pr190941/2014-06-12_05h45m13s/logs/errors/gchemutils-0.14.7_2.log http://package23.nyi.freebsd.org/data/91amd64-default-pr190941/2014-06-12_05h45m13s/logs/errors/gnome-keyring-2.32.1_4.log
Created attachment 143834 [details] patch2 New patch with a new approach. There are 4 new sed commands now. The first two apply some of the changes that Debian made to their libtool. The first command applies to libtool versions 1.4 and up. The second command is somewhat more elaborate but essentially it uses the sed hold space to move an "elif...fi" block down. It applies to 2.x. Together these reduce overlinking to unpatched .la files (from ports which don't have USES=libtool yet but also .la files in the work directory). The third and fourth command fix relinking. When relinking libtool replaces "libfoo.la" on the command line with "-L${STAGEDIR}${PREFIX}/lib -lfoo". I tried to change this into "${STAGEDIR}${PREFIX}/lib/libfoo.so" but libtool 1.4.x does not know about DESTDIR. This was fixed in libtool 1.5. (Staged ports that use libtool 1.4.x always relink with installed libraries instead of freshly built ones at the moment.) So instead these two commands make relink the same as a normal link. This means libtool will relink with libraries from the work directory instead of the stage directory. This applies to libtool 1.4 and up.
take for exp-run
Hi, The results on 91amd64 were good, no new failure: http://package18.nyi.freebsd.org/build.html?mastername=91amd64-default-pr190941&build=2014-06-17_05h09m31s On 10amd64 (still ongoing), there are linker errors: http://package18.nyi.freebsd.org/build.html?mastername=10amd64-default-pr190941&build=2014-06-18_05h31m05s linker error on devel/gettext: http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_05h31m05s/logs/errors/gettext-0.18.3.1_1.log linker error on net/libnetdude: http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_05h31m05s/logs/errors/libnetdude-0.12_1.log
Created attachment 143898 [details] patch3 New patch with fixes for gettext and libnetdude. The libtool.mk patch removes internal overlinking (inside a port). This can expose missing libraries on FreeBSD 10 because it requires library dependencies to be listed explicitly. FreeBSD < 10 can search dependencies of dependencies for missing libraries so that's why there are no problems there.
New results available at: http://package18.nyi.freebsd.org/build.html?mastername=10amd64-default-pr190941&build=2014-06-18_16h28m49s I do not have access to 10amd64 baseline results at the moment but I will provide you the list of new failures tonight. If you search for "missing" in the failed ports, this will give you a good estimate http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/gdbm-1.11_2.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/zziplib-0.13.62_1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libbonobo-2.32.0_2.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/open-motif-2.3.4_1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libgnomecanvas-2.30.3_3.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libgweather-2.30.3_3.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/rpm4-4.11.2_1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/swfdec-0.8.4_5.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/mjpegtools-2.1.0_2.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libgii-1.0.2_4.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/xchat-2.8.8_2.log etc.
New failures: + {"origin"=>"archivers/rpm4", "pkgname"=>"rpm4-4.11.2_1", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"databases/gdbm", "pkgname"=>"gdbm-1.11_2", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"devel/gitg0", "pkgname"=>"gitg0-0.0.8_1", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"devel/libbonobo", "pkgname"=>"libbonobo-2.32.0_2", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"devel/libgii", "pkgname"=>"libgii-1.0.2_4", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"devel/ois", "pkgname"=>"ois-1.2.0_4", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"devel/zziplib", "pkgname"=>"zziplib-0.13.62_1", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"graphics/devil", "pkgname"=>"devil-1.7.8_17,1", "phase"=>"build", "errortype"=>"linker_error"} + {"origin"=>"graphics/libgnomecanvas", "pkgname"=>"libgnomecanvas-2.30.3_3", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"graphics/swfdec", "pkgname"=>"swfdec-0.8.4_5", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"irc/xchat", "pkgname"=>"xchat-2.8.8_2", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"multimedia/mjpegtools", "pkgname"=>"mjpegtools-2.1.0_2", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"net/gtk-vnc", "pkgname"=>"gtk-vnc-0.3.10_10", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"net/libgweather", "pkgname"=>"libgweather-2.30.3_3", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"security/sssd", "pkgname"=>"sssd-1.9.6_5", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"sysutils/tracker-client", "pkgname"=>"tracker-client-0.6.95_18", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"www/gtkhtml3", "pkgname"=>"gtkhtml3-3.32.1_3", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"x11-toolkits/gdl", "pkgname"=>"gdl-2.30.1_3", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"x11-toolkits/libgdiplus", "pkgname"=>"libgdiplus-2.10_4", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"x11-toolkits/open-motif", "pkgname"=>"open-motif-2.3.4_1", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"x11-wm/mutter", "pkgname"=>"mutter-2.31.2_4", "phase"=>"build", "errortype"=>"missing"} Failure logs: http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/rpm4-4.11.2_1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/gdbm-1.11_2.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/gitg0-0.0.8_1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libbonobo-2.32.0_2.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libgii-1.0.2_4.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/ois-1.2.0_4.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/zziplib-0.13.62_1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/devil-1.7.8_17,1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libgnomecanvas-2.30.3_3.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/swfdec-0.8.4_5.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/xchat-2.8.8_2.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/mjpegtools-2.1.0_2.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/gtk-vnc-0.3.10_10.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libgweather-2.30.3_3.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/sssd-1.9.6_5.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/tracker-client-0.6.95_18.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/gtkhtml3-3.32.1_3.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/gdl-2.30.1_3.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/libgdiplus-2.10_4.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/open-motif-2.3.4_1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-18_16h28m49s/logs/errors/mutter-2.31.2_4.log
Created attachment 143932 [details] patch4 New patch that fixes all previous build failures.
Exp-run results available at: http://package18.nyi.freebsd.org/build.html?mastername=10amd64-default-pr190941&build=2014-06-20_04h59m46s I will check against baseline tonight, it seems the new failures are: databases/evolution-data-server devel/gnome-vfs graphics/geomview graphics/libggi http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-20_04h59m46s/logs/errors/gnome-vfs-2.24.4_3.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-20_04h59m46s/logs/errors/evolution-data-server-2.32.1_8.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-20_04h59m46s/logs/errors/libggi-2.2.2_6,1.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-20_04h59m46s/logs/errors/geomview-1.9.5.log
I confirm that there were only those 4 new failures
Created attachment 143960 [details] patch5 New patch with more build fixes.
http://package18.nyi.freebsd.org/build.html?mastername=10amd64-default-pr190941&build=2014-06-20_16h33m47s 3 new failures: + {"origin"=>"devel/anjuta", "pkgname"=>"anjuta-2.32.1.1_12", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"mail/evolution", "pkgname"=>"evolution-2.32.1_10", "phase"=>"build", "errortype"=>"missing"} + {"origin"=>"x11/gnome-panel", "pkgname"=>"gnome-panel-2.32.1_6", "phase"=>"build", "errortype"=>"missing"} http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-20_16h33m47s/logs/errors/anjuta-2.32.1.1_12.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-20_16h33m47s/logs/errors/evolution-2.32.1_10.log http://package18.nyi.freebsd.org/data/10amd64-default-pr190941/2014-06-20_16h33m47s/logs/errors/gnome-panel-2.32.1_6.log
Created attachment 143993 [details] patch6 More build fixes.
results available at http://package18.nyi.freebsd.org/build.html?mastername=10amd64-default-pr190941&build=2014-06-21_13h34m20s Patch approved, thanks!
Committed in r358784.
A commit references this bug: Author: tijl Date: Sun Jun 22 10:44:37 UTC 2014 New revision: 358784 URL: http://svnweb.freebsd.org/changeset/ports/358784 Log: Add 4 new sed commands to USES=libtool. The first two apply some of the changes that Debian made to their libtool. The first command applies to libtool versions 1.4 and up. The second command is somewhat more elaborate but essentially it uses the sed hold space to move an "elif...fi" block down. It applies to 2.x. Together these reduce overlinking to unpatched .la files (from ports that don't have USES=libtool yet but also .la files in the work directory). The third and fourth command fix relinking. During staging libtool may relink libA when it links to another library in the work directory libB. The reason is that libA created during build phase has its runpath set to the location of libB in the work directory. This allows running an executable that links to libA from within the work directory. The relink removes this extra runpath. When libtool relinks libA it replaces "libB.la" on the linker command line with "-L${STAGEDIR}${PREFIX}/lib -lB" with the intention to link to libB in the stage directory but this -L flag isn't necessarily the first so another libB may be linked instead. The two sed commands make relink the same as a normal link. This means libtool will relink with libraries from the work directory using a path similar to "../srcB/.libs/libB.so" without -L flags. This applies to libtool 1.4 and up. Earlier versions don't seem to relink libraries. (This fixes ports like devel/apr1 so they link with freshly built libraries instead of installed libraries.) Fix all ports with missing libraries. Additionally: archivers/rpm4: USES=patchfix. databases/gdbm: INSTALL_TARGET=install-strip. devel/gnome-vfs: remove patch that doesn't change anything. devel/ois: INSTALL_TARGET=install-strip and use standard USE_AUTOTOOLS. devel/zziplib: INSTALL_TARGET=install-strip, MAKE_CMD. multimedia/mjpegtools: remove USE_AUTOTOOLS, use modern compiler on i386 instead of disabling optimisations. net/libnetdude: disable static plugins. PR: 190941 Exp-run: antoine Approved by: portmgr (antoine) Changes: head/Mk/Uses/libtool.mk head/archivers/rpm4/Makefile head/databases/evolution-data-server/Makefile head/databases/gdbm/Makefile head/devel/anjuta/Makefile head/devel/gettext/files/patch-gettext-tools_src_Makefile.in head/devel/gitg0/Makefile head/devel/gnome-vfs/Makefile head/devel/gnome-vfs/files/patch-ac head/devel/libbonobo/files/patch-samples-echo-Makefile.in head/devel/libgii/Makefile head/devel/ois/Makefile head/devel/zziplib/Makefile head/devel/zziplib/files/patch-configure head/graphics/devil/Makefile head/graphics/geomview/Makefile head/graphics/libggi/Makefile head/graphics/libgnomecanvas/files/patch-demos-Makefile.in head/graphics/swfdec/Makefile head/irc/xchat/Makefile head/mail/evolution/Makefile head/multimedia/mjpegtools/Makefile head/multimedia/mjpegtools/files/patch-mplex__Makefile.in head/net/gtk-vnc/Makefile head/net/libgweather/Makefile head/net/libnetdude/Makefile head/net/libnetdude/files/patch-plugins__ChecksumFix__Makefile.in head/net/libnetdude/files/patch-tools-Makefile.in head/net/libnetdude/pkg-plist head/security/sssd/Makefile head/sysutils/tracker-client/Makefile head/www/gtkhtml3/Makefile head/x11/gnome-panel/Makefile head/x11-toolkits/gdl/Makefile head/x11-toolkits/libgdiplus/Makefile head/x11-toolkits/open-motif/Makefile head/x11-wm/mutter/Makefile