Failed runs: https://ci.freebsd.org/job/FreeBSD-head-i386-build/9799/console https://ci.freebsd.org/job/FreeBSD-head-armv7-build/2129/console Output (from FreeBSD-head-i386-build/9799) The order looks correct but might be the timing is too close or some path issues. 00:10:50.832 --- lib/libgcc_s__L --- 00:10:50.832 sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/usr/src/i386.i386/tmp/lib/libgcc_s.so.1 /usr/obj/usr/src/i386.i386/tmp/usr/lib/libgcc_s.so ... 00:10:50.836 --- lib/libgssapi__L --- 00:10:50.836 /usr/obj/usr/src/i386.i386/tmp/usr/bin/ld: error: unable to find library -lgcc_s ... 00:10:50.838 --- lib/libgssapi__L --- 00:10:50.838 cc: error: linker command failed with exit code 1 (use -v to see invocation)
Back on 2018-June-18 Bryan Drewery wrote in a response for something else that looked initially to be a possible race: If it was -lgcc_s then it's a known rare build race due to tools/install.sh not handling -S.
Occurred in a private CI build of mine: https://cirrus-ci.com/task/6381946174177280
Indeed, the build installs libgcc_s twice to the same location: % grep 'install.*libgcc_s.so$' ~/Downloads/6381946174177280-main.log sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1 /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so
In a successful build we actually install libgcc_s.so three times to the same location.
(In reply to Ed Maste from comment #4) Is there a installworld vs. buildworld confusion? (I'd expect ld to be a buildworld issue, not a installworld one.) I see from recent activity, using: egrep -r '(--- installworld ---|install.*libgcc_s.so)' /root/sys_typescripts/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld --- /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld --- /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -s -o root -g wheel -m 444 -S libgcc_s.so.1 /lib/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/lib/debug/lib/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755 /lib/libgcc_s.so.1 /usr/lib/libgcc_s.so /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -s -o root -g wheel -m 444 -S libgcc_s.so.1 /usr/lib32/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/lib/debug/usr/lib32/ /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755 libgcc_s.so.1 /usr/lib32/libgcc_s.so /root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-22:15:07:14:Command: env __MAKE_CONF=/root/src.configs/make.conf SRCCONF=/dev/null SRC_ENV_CONF=/root/src.configs/src.conf. (So all from the same log file: one for installworld.) No matches found in files without installworld showing. Similarly for all the log history the grep went through. (I run buildworld and installworld in separate runs, not together, so separate log files in the directory.) The logs span amd64, armv7, aarch64, powerpc64, and powerpc builds. I see no evidence of buildworld doing install's of libgcc_s.so materials.
(In reply to Mark Millard from comment #5) The installs are done during buildworld as part of the libraries target. In a submake the _prereq_libs, _startup_libs and _generic_libs targets are built in turn and each of these builds and installs libgcc_s.so (see the ${_lib}__PL and ${_lib}__L targets in Makefile.inc1)
(In reply to Ed Maste from comment #6) Ahh, I needed to look in the .meta files. Sorry. What you are reporting may mean that some .meta files are written more than once, so I'd see content from only the last pass for each. Messy. For reference (for a first time build of a head -r345520 based tree): # grep -lr 'CMD .*install.*libgcc_s\.so' /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta So 4 meta files, 2 for lib32 activity and 2 not: # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug libgcc_s.so.1.full libgcc_s.so.1 CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444 libgcc_s.so.1 libgcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc TARGET libgcc_s.so.1 # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 -S libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/ CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/debug/lib/ CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/libgcc_s.so.1 /usr/obj/powerpc64 vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/libgcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc TARGET _libinstall # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug libgcc_s.so.1.full libgcc_s.so.1 CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444 libgcc_s.so.1 libgcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc TARGET libgcc_s.so.1 # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 -S libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/ CMD sh /usr/src/tools/install.sh -o root -g wheel -m 444 libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib/de bug/usr/lib32/ CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755 libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/lib gcc_s.so CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc TARGET _libinstall
(In reply to Mark Millard from comment #7) And, finally what I should have looked for in the overall logs, using a aarch64 build this time (no lib32): # egrep '(libgcc_s\.so|libgcc_s.*_libinstall)' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22 Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.full --- libgcc_s.so.1.full --- building shared library libgcc_s.so.1 Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.debug Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1 Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall So 3 _libinstall 's for libgcc_s . Looking at other things in the build there are a lot of pairs of _libinstall 's and libgcc_eh has 3: FBSDFSSD# egrep '/_libinstall' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22 | sort | more Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libdtrace/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall . . . Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall . . .