Bug 233769 - Possible build race: ld: error: unable to find library -lgcc_s
Summary: Possible build race: ld: error: unable to find library -lgcc_s
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-04 09:58 UTC by Li-Wen Hsu
Modified: 2019-03-26 21:46 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Li-Wen Hsu freebsd_committer 2018-12-04 09:58:06 UTC
Failed runs:

https://ci.freebsd.org/job/FreeBSD-head-i386-build/9799/console
https://ci.freebsd.org/job/FreeBSD-head-armv7-build/2129/console

Output (from FreeBSD-head-i386-build/9799)

The order looks correct but might be the timing is too close or some path issues.

00:10:50.832 --- lib/libgcc_s__L ---
00:10:50.832 sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/usr/src/i386.i386/tmp/lib/libgcc_s.so.1  /usr/obj/usr/src/i386.i386/tmp/usr/lib/libgcc_s.so

...

00:10:50.836 --- lib/libgssapi__L ---
00:10:50.836 /usr/obj/usr/src/i386.i386/tmp/usr/bin/ld: error: unable to find library -lgcc_s

...

00:10:50.838 --- lib/libgssapi__L ---
00:10:50.838 cc: error: linker command failed with exit code 1 (use -v to see invocation)
Comment 1 Mark Millard 2018-12-04 20:17:48 UTC
Back on 2018-June-18 Bryan Drewery wrote in a response for
something else that looked initially to be a possible race:

If it was -lgcc_s then it's a known rare build race due to
tools/install.sh not handling -S.
Comment 2 Ed Maste freebsd_committer 2019-03-26 00:16:19 UTC
Occurred in a private CI build of mine: https://cirrus-ci.com/task/6381946174177280
Comment 3 Ed Maste freebsd_committer 2019-03-26 03:40:26 UTC
Indeed, the build installs libgcc_s twice to the same location:

% grep 'install.*libgcc_s.so$' ~/Downloads/6381946174177280-main.log
sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so
sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so
Comment 4 Ed Maste freebsd_committer 2019-03-26 16:54:12 UTC
In a successful build we actually install libgcc_s.so three times to the same location.
Comment 5 Mark Millard 2019-03-26 19:50:30 UTC
(In reply to Ed Maste from comment #4)

Is there a installworld vs. buildworld confusion? (I'd expect ld to be a
buildworld issue, not a installworld one.)

I see from recent activity, using:

egrep -r '(--- installworld ---|install.*libgcc_s.so)' /root/sys_typescripts/


/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld ---
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld ---
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -s -o root -g wheel -m 444   -S  libgcc_s.so.1 /lib/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/lib/debug/lib/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755  /lib/libgcc_s.so.1  /usr/lib/libgcc_s.so
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -s -o root -g wheel -m 444   -S  libgcc_s.so.1 /usr/lib32/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/lib/debug/usr/lib32/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755  libgcc_s.so.1 /usr/lib32/libgcc_s.so
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-22:15:07:14:Command: env __MAKE_CONF=/root/src.configs/make.conf SRCCONF=/dev/null SRC_ENV_CONF=/root/src.configs/src.conf.

(So all from the same log file: one for installworld.)

No matches found in files without installworld showing. Similarly for
all the log history the grep went through. (I run buildworld and
installworld in separate runs, not together, so separate log files in
the directory.)

The logs span amd64, armv7, aarch64, powerpc64, and powerpc builds.
I see no evidence of buildworld doing install's of libgcc_s.so materials.
Comment 6 Ed Maste freebsd_committer 2019-03-26 19:57:40 UTC
(In reply to Mark Millard from comment #5)
The installs are done during buildworld as part of the libraries target. In a submake the _prereq_libs, _startup_libs and _generic_libs targets are built in turn and each of these builds and installs libgcc_s.so (see the ${_lib}__PL and ${_lib}__L targets in Makefile.inc1)
Comment 7 Mark Millard 2019-03-26 20:56:45 UTC
(In reply to Ed Maste from comment #6)

Ahh, I needed to look in the .meta files. Sorry.

What you are reporting may mean that some .meta files are
written more than once, so I'd see content from only the
last pass for each. Messy.

For reference (for a first time build of a head -r345520
based tree):

# grep -lr 'CMD .*install.*libgcc_s\.so' /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta

So 4 meta files, 2 for lib32 activity and 2 not:

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta
CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug  libgcc_s.so.1.full libgcc_s.so.1
CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444  libgcc_s.so.1 libgcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc
TARGET libgcc_s.so.1

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta
CMD sh /usr/src/tools/install.sh   -o root -g wheel -m 444   -S  libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/
CMD sh /usr/src/tools/install.sh  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/debug/lib/
CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/libgcc_s.so.1  /usr/obj/powerpc64
vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/libgcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc
TARGET _libinstall

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta
CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug  libgcc_s.so.1.full libgcc_s.so.1
CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444  libgcc_s.so.1 libgcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc
TARGET libgcc_s.so.1

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta
CMD sh /usr/src/tools/install.sh   -o root -g wheel -m 444   -S  libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/
CMD sh /usr/src/tools/install.sh  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib/de
bug/usr/lib32/
CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755  libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/lib
gcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc
TARGET _libinstall
Comment 8 Mark Millard 2019-03-26 21:46:15 UTC
(In reply to Mark Millard from comment #7)

And, finally what I should have looked for in the overall
logs, using a aarch64 build this time (no lib32):

# egrep '(libgcc_s\.so|libgcc_s.*_libinstall)' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.full
--- libgcc_s.so.1.full ---
building shared library libgcc_s.so.1
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.debug
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall

So 3 _libinstall 's for libgcc_s .

Looking at other things in the build there are a lot of pairs of _libinstall 's
and libgcc_eh has 3:

FBSDFSSD# egrep '/_libinstall' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22 | sort | more
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libdtrace/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall
. . .
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall
. .  .