Bug 182944 - FreeBSD 10 (and 11) build failures with ccache
Summary: FreeBSD 10 (and 11) build failures with ccache
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: Unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: Bryan Drewery
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-10-13 04:40 UTC by Kenta Suzumoto
Modified: 2015-11-08 00:50 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kenta Suzumoto 2013-10-13 04:40:00 UTC
Using ccache, "make buildworld" is failing. I found
this mailing list post describing a similar issue:

https://lists.freebsd.org/pipermail/freebsd-current/2013-May/042035.html

The main issue is:

/usr/bin/ld: this linker was not configured to use sysroots

It fails like so:

/usr/local/libexec/ccache/cc -m32 -march=native -DCOMPAT_32BIT  -isystem /usr/obj/usr/src/lib32/usr/include/  -L/usr/obj/usr/src/lib32/usr/lib32  -B/usr/obj/usr/src/lib32/usr/lib32 --sysroot=/usr/obj/usr/src/tmp -fpic -DPIC  -O2 -pipe -Qunused-parameter -Qunused-arguments -I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include -I/usr/src/lib/libc/i386 -DNLS  -D__DBINTERFACE_PRIVATE -I/usr/src/lib/libc/../../contrib/gdtoa -I/usr/src/lib/libc/../../contrib/libc-vis -I/usr/obj/lib32/usr/src/lib/libc -I/usr/src/lib/libc/resolv -D_ACL_PRIVATE -DPOSIX_MISTAKE -I/usr/src/lib/libc/../../contrib/jemalloc/include -I/usr/src/lib/libc/../../contrib/tzcode/stdtime -I/usr/src/lib/libc/stdtime  -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP -DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING -DNDEBUG -std=gnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -Wno-ta
 utological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr/src/lib/libc/string/wmemset.c -o wmemset.So
building shared library libc.so.7
/usr/bin/ld: this linker was not configured to use sysroots
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[4]: stopped in /usr/src/lib/libc
*** Error code 1

Stop.
make[3]: stopped in /usr/src
*** Error code 1

Stop.
make[2]: stopped in /usr/src
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src

Fix: 

"the base binutils could be fixed to enable --sysroot support."
Comment 1 Bryan Drewery freebsd_committer 2015-11-08 00:50:43 UTC
Fixed for head when using WITH_CCACHE_BUILD.
Comment 2 commit-hook freebsd_committer 2015-11-08 00:50:47 UTC
A commit references this bug:

Author: bdrewery
Date: Sun Nov  8 00:50:19 UTC 2015
New revision: 290526
URL: https://svnweb.freebsd.org/changeset/base/290526

Log:
  Add built-in ccache build support via WITH_CCACHE_BUILD option.

  ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
  used to achieve a safe pseudo-incremental build.  This is explained in
  more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
  cache, but with a full cache saves 30-50% in build times.  When combined with
  the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
  resulting dependency file, which it does not do when using mkdep(1)/'CC
  -E'.  Stats are provided at the end of this message.

  This removes the need to modify /etc/make.conf with the CC:= and CXX:=
  lines which conflicted with external compiler support [3] (causing the
  bootstrap compiler to not be built which lead to obscure failures [4]),
  incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
  Clang errors with parenthesis, and did not work with META_MODE.

  The option name was picked to match the existing option in ports.  This
  feature is available for both in-src and out-of-src builds that use
  /usr/share/mk.

  Linking, assembly compiles, and pre-processing avoid using ccache since it is
  only overhead.  ccache does nothing special in these modes, although there is
  no harm in calling it for them.

  CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
  compiler to hash the content of the compiler binary to determine if it
  should be a cache miss.  For external compilers the 'mtime' option is used
  as it is more efficient and likely to be correct.  Future work may optimize the
  'content' check using the same checks as whether a bootstrap compiler is needed
  to be built.

  The CCACHE_CPP2 pessimization is currently default in our devel/ccache
  port due to Clang requiring it.  Clang's -Wparentheses-equality,
  -Wtautological-compare, and -Wself-assign warnings do not mix well with
  compiling already-pre-processed code that may have expanded macros that
  trigger the warnings.  GCC has so far not had this issue so it is allowed to
  disable the CCACHE_CPP2 default in our port.

  Sharing a cache between multiple checkouts, or systems, is explained in
  the ccache manual.  Sharing a cache over NFS would likely not be worth
  it, but syncing cache directories between systems may be useful for an
  organization.  There is also a memcached backend available [5].  Due to using
  an object directory outside of the source directory though you will need to
  ensure that both are in the same prefix and all users use the same layout.  A
  possible working layout is as follows:
    Source: /some/prefix/src1
    Source: /some/prefix/src2
    Source: /some/prefix/src3
    Objdir: /some/prefix/obj
    Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
  This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
  all absolute paths to be relative.  Using something like this is required due
  to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
  adds into its hash for the object without CCACHE_BASEDIR.

  distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
  I have not personally tested this and assume it will not mix well with
  using the bootstrap compiler.

  The cache from buildworld can be reused in a subdir by first running
  'make buildenv' (from r290424).

  Note that the cache is currently different depending on whether -j is
  used or not due to ccache enabling -fdiagnostics-color automatically if
  stderr is a TTY, which bmake only does if not using -j.

  The system I used for testing was:
    WITNESS
    Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
    DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
          The arc was fully populated with src tree files and ccache objects.
    RAM: 76GiB
    CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
         2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

  The WITH_FAST_DEPEND feature was used for comparison here as well to show
  the dramatic time savings with a full cache.

  buildworld:
    x buildworld-before
    + buildworld-ccache-empty
    * buildworld-ccache-full
    % buildworld-ccache-full-fastdep
    # buildworld-fastdep
    +-------------------------------------------------------------------------------+
    |%            *                               #                                +|
    |%            *                               #                                +|
    |%            *                               #             xxx                +|
    |                                                           |A                  |
    |                                                                              A|
    |             A                                                                 |
    |A                                                                              |
    |                                             A                                 |
    +-------------------------------------------------------------------------------+
        N           Min           Max        Median           Avg        Stddev
    x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
    +   3          4519       4525.04       4520.73       4521.59     3.1104823
    Difference at 95.0% confidence
            758.027 +/- 43.4565
            20.1412% +/- 1.15466%
            (Student's t, pooled s = 19.1726)
    *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
    Difference at 95.0% confidence
            -1938.26 +/- 43.298
            -51.5007% +/- 1.15045%
            (Student's t, pooled s = 19.1026)
    %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
    Difference at 95.0% confidence
            -2491.3 +/- 44.3704
            -66.1952% +/- 1.17895%
            (Student's t, pooled s = 19.5758)
    #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
    Difference at 95.0% confidence
            -609.33 +/- 43.1943
            -16.1902% +/- 1.1477%
            (Student's t, pooled s = 19.0569)

  buildkernel:
    x buildkernel-before
    + buildkernel-ccache-empty
    * buildkernel-ccache-empty-fastdep
    % buildkernel-ccache-full
    # buildkernel-ccache-full-fastdep
    @ buildkernel-fastdep
    +-------------------------------------------------------------------------------+
    |#                        @   %                  *                              |
    |#                        @   %                  *     x                      + |
    |#                        @   %                  *     xx                     ++|
    |                                                      MA                       |
    |                                                                             MA|
    |                                                A                              |
    |                             A                                                 |
    |A                                                                              |
    |                         A                                                     |
    +-------------------------------------------------------------------------------+
        N           Min           Max        Median           Avg        Stddev
    x   3        571.57        573.94        571.79     572.43333     1.3094401
    +   3        727.97        731.91        728.06     729.31333     2.2492295
    Difference at 95.0% confidence
            156.88 +/- 4.17129
            27.4058% +/- 0.728695%
            (Student's t, pooled s = 1.84034)
    *   3         527.1        528.29        528.08     527.82333    0.63516402
    Difference at 95.0% confidence
            -44.61 +/- 2.33254
            -7.79305% +/- 0.407478%
            (Student's t, pooled s = 1.02909)
    %   3         400.4        401.05        400.62        400.69     0.3306055
    Difference at 95.0% confidence
            -171.743 +/- 2.16453
            -30.0023% +/- 0.378128%
            (Student's t, pooled s = 0.954969)
    #   3        201.94        203.34        202.28        202.52    0.73020545
    Difference at 95.0% confidence
            -369.913 +/- 2.40293
            -64.6212% +/- 0.419774%
            (Student's t, pooled s = 1.06015)
    @   3        369.12        370.57         369.3     369.66333    0.79033748
    Difference at 95.0% confidence
            -202.77 +/- 2.45131
            -35.4225% +/- 0.428227%
            (Student's t, pooled s = 1.0815)

  [1] https://ccache.samba.org/performance.html
  [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
  [3] https://reviews.freebsd.org/D3484
  [5] https://github.com/jrosdahl/ccache/pull/30

  PR:		182944 [4]
  MFC after:	3 weeks
  Sponsored by:	EMC / Isilon Storage Division
  Relnotes:	yes

Changes:
  head/Makefile
  head/Makefile.inc1
  head/share/mk/bsd.compiler.mk
  head/share/mk/bsd.dep.mk
  head/share/mk/bsd.lib.mk
  head/share/mk/bsd.opts.mk
  head/share/mk/bsd.prog.mk
  head/share/mk/local.meta.sys.mk
  head/share/mk/sys.mk
  head/sys/conf/kern.post.mk
  head/sys/conf/kern.pre.mk
  head/tools/build/options/WITH_CCACHE_BUILD