Failed runs: https://ci.freebsd.org/job/FreeBSD-head-i386-build/9799/console https://ci.freebsd.org/job/FreeBSD-head-armv7-build/2129/console Output (from FreeBSD-head-i386-build/9799) The order looks correct but might be the timing is too close or some path issues. 00:10:50.832 --- lib/libgcc_s__L --- 00:10:50.832 sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/usr/src/i386.i386/tmp/lib/libgcc_s.so.1 /usr/obj/usr/src/i386.i386/tmp/usr/lib/libgcc_s.so ... 00:10:50.836 --- lib/libgssapi__L --- 00:10:50.836 /usr/obj/usr/src/i386.i386/tmp/usr/bin/ld: error: unable to find library -lgcc_s ... 00:10:50.838 --- lib/libgssapi__L --- 00:10:50.838 cc: error: linker command failed with exit code 1 (use -v to see invocation)
Back on 2018-June-18 Bryan Drewery wrote in a response for something else that looked initially to be a possible race: If it was -lgcc_s then it's a known rare build race due to tools/install.sh not handling -S.
Occurred in a private CI build of mine: https://cirrus-ci.com/task/6381946174177280
Indeed, the build installs libgcc_s twice to the same location: % grep 'install.*libgcc_s.so$' ~/Downloads/6381946174177280-main.log sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so
In a successful build we actually install libgcc_s.so three times to the same location.
(In reply to Ed Maste from comment #4) Is there a installworld vs. buildworld confusion? (I'd expect ld to be a buildworld issue, not a installworld one.) I see from recent activity, using: egrep -r '(--- installworld ---|install.*libgcc_s.so)' /root/sys_typescripts/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld --- /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld --- /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -s -o root -g wheel -m 444 -S libgcc_s.so.1 /lib/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/lib/debug/lib/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755 /lib/libgcc_s.so.1 /usr/lib/libgcc_s.so /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -s -o root -g wheel -m 444 -S libgcc_s.so.1 /usr/lib32/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/lib/debug/usr/lib32/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755 libgcc_s.so.1 /usr/lib32/libgcc_s.so /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-22:15:07:14:Command: env __MAKE_CONF=/root/src.configs/make.conf SRCCONF=/dev/null SRC_ENV_CONF=/root/src.configs/src.conf. (So all from the same log file: one for installworld.) No matches found in files without installworld showing. Similarly for all the log history the grep went through. (I run buildworld and installworld in separate runs, not together, so separate log files in the directory.) The logs span amd64, armv7, aarch64, powerpc64, and powerpc builds. I see no evidence of buildworld doing install's of libgcc_s.so materials.
(In reply to Mark Millard from comment #5) The installs are done during buildworld as part of the libraries target. In a submake the _prereq_libs, _startup_libs and _generic_libs targets are built in turn and each of these builds and installs libgcc_s.so (see the ${_lib}__PL and ${_lib}__L targets in Makefile.inc1)
(In reply to Ed Maste from comment #6) Ahh, I needed to look in the .meta files. Sorry. What you are reporting may mean that some .meta files are written more than once, so I'd see content from only the last pass for each. Messy. For reference (for a first time build of a head -r345520 based tree): # grep -lr 'CMD .*install.*libgcc_s\.so' /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta So 4 meta files, 2 for lib32 activity and 2 not: # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug libgcc_s.so.1.full libgcc_s.so.1 CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444 libgcc_s.so.1 libgcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc TARGET libgcc_s.so.1 # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 -S libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/ CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/debug/lib/ CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/libgcc_s.so.1 /usr/obj/powerpc64 vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/libgcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc TARGET _libinstall # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug libgcc_s.so.1.full libgcc_s.so.1 CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444 libgcc_s.so.1 libgcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc TARGET libgcc_s.so.1 # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 -S libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/ CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib/de bug/usr/lib32/ CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755 libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/lib gcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc TARGET _libinstall
(In reply to Mark Millard from comment #7) And, finally what I should have looked for in the overall logs, using a aarch64 build this time (no lib32): # egrep '(libgcc_s\.so|libgcc_s.*_libinstall)' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22 Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.full --- libgcc_s.so.1.full --- building shared library libgcc_s.so.1 Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.debug Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1 Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall So 3 _libinstall 's for libgcc_s . Looking at other things in the build there are a lot of pairs of _libinstall 's and libgcc_eh has 3: FBSDFSSD# egrep '/_libinstall' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22 | sort | more Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libdtrace/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall . . . Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall . . .
On Tue, 30 Jul 2019 at 13:38, <jenkins-admin@freebsd.org> wrote: > > FreeBSD-head-powerpcspe-build - Build #12186 (r350451) - Failure > > Build information: https://ci.freebsd.org/job/FreeBSD-head-powerpcspe-build/12186/ > Full change log: https://ci.freebsd.org/job/FreeBSD-head-powerpcspe-build/12186/changes > Full build log: https://ci.freebsd.org/job/FreeBSD-head-powerpcspe-build/12186/console > 13:37:03 /usr/obj/usr/src/powerpc.powerpcspe/tmp/usr/bin/ld: cannot find -lgcc_s PR 233769 Possible build race: ld: error: unable to find library -lgcc_s https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233769
This is due to tools/install.sh not supporting -S. If that is fixed then this is easily fixed.
Just passing through -S in tools/install.sh is not enough. 1. Maybe this tool is expected to work on Linux where -S does something different. 2. The symlink/hardlink support call ln(1) which are is atomic for symlinks. It does unlink(2) followed by symlink(2). Reworking the build to avoid double installs here is possible but would be more hackish IMO. We would need to muck with the SUBDIR lists for gnu/lib to fix this specific case so that it only installs during the right phase of buildworld (like we do for llvm kinda). But that's JUST gnu/lib. The same problem could come up for other libraries eventually given enough install-to-worldtmp threads. META_MODE+filemon may eventually learn how to avoid the double installs but it has not yet. DIRDEPS_BUILD learned how. It's not hard to implement just risky and needs time devoted to testing. Of course this then requires filemon which isn't the solution some people want to hear. The double install/relink issue isn't causing a problem beyond this race which is why I think we just need to make the install atomic rather than wack-a-mole with SUBDIR hacks.
Oh and it's actually the symlink causing the problem here not the library itself.