Bug 260099
| Summary: | Fix build when building with WITH_ASAN and WITH_UBSAN | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Xin LI <delphij> | ||||
| Component: | bin | Assignee: | Dimitry Andric <dim> | ||||
| Status: | Open --- | ||||||
| Severity: | Affects Some People | CC: | arichardson, delphij, dim, dmgk, emaste, freebsd, marklmi26-fbsd, tech-lists | ||||
| Priority: | --- | Keywords: | needs-qa | ||||
| Version: | CURRENT | Flags: | koobs:
maintainer-feedback?
(dim) koobs: maintainer-feedback? (delphij) |
||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| See Also: | https://reviews.freebsd.org/D32805 | ||||||
| Attachments: |
|
||||||
I've been trying to build world with MK_ASAN=yes and MK_UBSAN=yes, but it always fails at a very early stage, with something entirely unexpected:
--------------------------------------------------------------
>>> stage 2.1: cleaning up the object tree
--------------------------------------------------------------
cd /home/dim/src/main; time env MACHINE_ARCH=amd64 MACHINE=amd64 CPUTYPE= CC="cc -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin" CXX="c++ -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin" CPP="cpp -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin" AS="as" AR="ar" ELFCTL="elfctl" LD="ld" LLVM_LINK="" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" STRIPBIN="strip" INSTALL="install -U" PATH=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/sbin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/sbin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin SYSROOT=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp make -f Makefile.inc1 BWPHASE=cleanobj DESTDIR=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp _NO_INCLUDE_COMPILERMK=t cleandir
===> lib (cleandir)
make[3] warning: /lib: Permission denied.
make[3]: "/home/dim/src/main/lib/Makefile" line 190: "Requested build with sanitizers but cannot build runtime libraries!"
make[3]: stopped in /home/dim/src/main/lib
*** Error code 1
For some reason, at this point COMPILER_TYPE is not correctly defined, and the Makefile dies.
Adding arichardson@ on CC, maybe he knows more about this...
Maybe
diff --git a/lib/Makefile b/lib/Makefile
index 2cba5feaee19..bd28b974c673 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -185,7 +185,7 @@ SUBDIR.${MK_STATS}+= libstats
${MACHINE_CPUARCH} == "arm" || ${MACHINE_CPUARCH} == "i386" || \
${MACHINE_CPUARCH} == "powerpc")
_libclang_rt= libclang_rt
-.elif ${MK_ASAN} != "no" || ${MK_UBSAN} != "no"
+.elif (${MK_ASAN} != "no" || ${MK_UBSAN} != "no") && make(all)
.error "Requested build with sanitizers but cannot build runtime libraries!"
.endif
https://reviews.freebsd.org/D32805 for my change I was seeing two linker errors (one with WITH_UBSAN, the other WITH_ASAN, notably I don't use MK_*SAN), reporters in September in: https://www.mail-archive.com/freebsd-current@freebsd.org/msg184811.html The first (ftpd.full) was resolved via: https://reviews.freebsd.org/D31945 and my latest buildworld has passed that, leaving only the following: ld: error: undefined symbol: lldb_private::formatters::CMTimeSummaryProvider ld: error: undefined symbol: lldb_private::AppleObjCRuntimeV1 (and V2) ld: error: undefined symbol: lldb_private::CFBasicHash ld: error: undefined symbol: lldb_private::lldb_renderscript ld: error: undefined symbol: NewCommandObjectRenderScriptScriptGroup(lldb_private::CommandInterpreter&) These errors appear similar (same?) as those reported in 2017: https://lists.freebsd.org/pipermail/freebsd-current/2017-July/066648.html Where Dim mentions in a follow-up: Apparently in some scenarios more object files are needed than the minimum set I put in liblldb's Makefile. Is that still the case now? If so, its interesting that I haven't come across other reports of the issue (unless no-one is testing *SAN yet) @Xin Do you think your patch addresses the undefined symbols listed in my comment 4 or is it possible there are more SRCS's that may be required? Testing build on latest current sources with attachment 229774 [details] now
I got a successful build with:
# Speed
MALLOC_PRODUCTION=yes
WITHOUT_LLVM_ASSERTIONS=ses
# Hardening
WITH_PIE=yes
WITH_BIND_NOW=yes
WITH_RETPOLINE=yes
WITH_SSP=yes
# Sanitizers
WITH_ASAN=yes
WITH_UBSAN=yes
WITH_CCACHE_BUILD=yes
CCACHE_PREFIX=/usr/local/bin/distcc
7256.14 real 4002.80 user 1113.36 sys
--- buildworld_epilogue ---
--------------------------------------------------------------
>>> World build completed on Sat Dec 18 09:07:13 UTC 2021
>>> World built in 35606 seconds, ncpu: 2, make -j8
Hi, This doesn't work in very recent -current on arm64.aarch64 system: FreeBSD 14.0-CURRENT (CERES) #2 main-n251742-cfca19c28e5: Thu Dec 16 23:54:49 GMT 2021 hardware: raspberry pi4b+ / 8GB % gvs srctree last updated: Mon Dec 20 12:49:31 2021 +0100 version: 251840 build results here: https://cloud.zyxst.net/~john/FreeBSD/aarch64.rpi4.mitigations.src-current/buildfail.txt src.conf, src-env.conf, make.conf are here: https://cloud.zyxst.net/~john/FreeBSD/aarch64.rpi4.mitigations.src-current/ I *wasn't* able to apply the patch in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260099#c0 sorry, I meant https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260099#c6 (In reply to tech-lists from comment #9) Can you apply the changes manually FYI: Manually applying this patch and trying buildworld built lldb. This is based on also using:
# git -C /usr/main-src/ diff /usr/main-src/usr.bin/clang/
diff --git a/usr.bin/clang/llvm.prog.mk b/usr.bin/clang/llvm.prog.mk
index 3a708805d3ea..74bed2ecd314 100644
--- a/usr.bin/clang/llvm.prog.mk
+++ b/usr.bin/clang/llvm.prog.mk
@@ -25,6 +25,7 @@ PACKAGE= clang
.if ${.MAKE.OS} == "FreeBSD" || !defined(BOOTSTRAPPING)
LIBADD+= execinfo
LIBADD+= tinfow
+LIBADD+= z
.endif
LIBADD+= pthread
to get by an earlier stopping point.
The next stopping point now is:
--- all_subdir_usr.bin/clang/llvm-pdbutil ---
Building /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/usr.bin/clang/llvm-pdbutil/llvm-pdbutil.full
. . .
--- all_subdir_usr.bin/clang ---
--- llvm-pdbutil.full ---
ld: error: undefined symbol: llvm::codeview::limitSymbolArrayToScope(llvm::VarStreamArray<llvm::codeview::CVRecord<llvm::codeview::SymbolKind>, llvm::VarStreamArrayExtractor<llvm::codeview::CVRecord<llvm::codeview::SymbolKind> > > const&, unsigned int)
>>> referenced by ModuleDebugStream.cpp:91 (/usr/main-src/contrib/llvm-project/llvm/lib/DebugInfo/PDB/Native/ModuleDebugStream.cpp:91)
>>> ModuleDebugStream.o:(llvm::pdb::ModuleDebugStreamRef::getSymbolArrayForScope(unsigned int) const) in archive /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/lib/clang/libllvm/libllvm.a
c++: error: linker command failed with exit code 1 (use -v to see invocation)
*** [llvm-pdbutil.full] Error code 1
(In reply to Mark Millard from comment #12) I took a guess of: diff --git a/lib/clang/libllvm/Makefile b/lib/clang/libllvm/Makefile index 6df32d8b452a..9e37d8cdf829 100644 --- a/lib/clang/libllvm/Makefile +++ b/lib/clang/libllvm/Makefile @@ -470,6 +470,7 @@ SRCS_MIN+= DebugInfo/CodeView/RecordSerialization.cpp SRCS_MIN+= DebugInfo/CodeView/SimpleTypeSerializer.cpp SRCS_EXT+= DebugInfo/CodeView/StringsAndChecksums.cpp SRCS_MIN+= DebugInfo/CodeView/SymbolDumper.cpp +SRCS_EXT+= DebugInfo/CodeView/SymbolRecordHelpers.cpp SRCS_MIN+= DebugInfo/CodeView/SymbolRecordMapping.cpp SRCS_EXT+= DebugInfo/CodeView/SymbolSerializer.cpp SRCS_MIN+= DebugInfo/CodeView/TypeDumpVisitor.cpp and that let buildworld complete in my context. (I've no clue if _EXT is actually the right category overall.) I will note that some earlier experiments with trying -flto=thin based builds means that I do have some adde __attribute__((used)) __attribute((retain)) usage in a few places. I've no clue if any of things contributed to having completion or not. For the earlier-noted: # git -C /usr/main-src/ diff /usr/main-src/usr.bin/clang/ diff --git a/usr.bin/clang/llvm.prog.mk b/usr.bin/clang/llvm.prog.mk index 3a708805d3ea..74bed2ecd314 100644 --- a/usr.bin/clang/llvm.prog.mk +++ b/usr.bin/clang/llvm.prog.mk @@ -25,6 +25,7 @@ PACKAGE= clang .if ${.MAKE.OS} == "FreeBSD" || !defined(BOOTSTRAPPING) LIBADD+= execinfo LIBADD+= tinfow +LIBADD+= z .endif LIBADD+= pthread I also do not know if the LIBADD+= z is correctly handled for the 4 combinations of values for each of ${.MAKE.OS} == "FreeBSD" and !defined(BOOTSTRAPPING) (or if other contexts would need checking). I just took a guess. (In reply to Mark Millard from comment #13) Just as a quick cross check, I did installworld into /usr/obj/DESTDIRs/main-amd64-xSAN-chroot/ then: # sysctl kern.elf64.aslr.pie_enable=0 # chroot /usr/obj/DESTDIRs/main-amd64-xSAN-chroot # ls -Tld /* /usr/main-src/lib/libc/stdio/fread.c:133:10: runtime error: applying zero offset to null pointer SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/main-src/lib/libc/stdio/fread.c:133:10 in -r--r--r-- 1 root wheel 6109 Jan 7 09:30:01 2022 /COPYRIGHT drwxr-xr-x 2 root wheel 46 Jan 7 09:29:49 2022 /bin drwxr-xr-x 14 root wheel 65 Jan 7 09:30:01 2022 /boot dr-xr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /dev drwxr-xr-x 28 root wheel 104 Jan 7 09:30:03 2022 /etc drwxr-xr-x 5 root wheel 69 Jan 7 09:29:48 2022 /lib drwxr-xr-x 3 root wheel 5 Jan 7 09:29:40 2022 /libexec drwxr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /media drwxr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /mnt drwxr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /net dr-xr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /proc drwxr-xr-x 2 root wheel 149 Jan 7 09:29:48 2022 /rescue drwxr-xr-x 2 root wheel 7 Jan 7 09:30:02 2022 /root drwxr-xr-x 2 root wheel 148 Jan 7 09:29:50 2022 /sbin drwxrwxrwt 2 root wheel 2 Jan 7 09:29:38 2022 /tmp drwxr-xr-x 14 root wheel 14 Jan 7 09:29:38 2022 /usr drwxr-xr-x 24 root wheel 24 Jan 7 09:29:38 2022 /var So the build works to some extent. |
Created attachment 229774 [details] Proposed patch I've discovered a few build issues when playing with the two options (note that due to incompatibility with ASLR I haven't looked deeper).