Created attachment 229774 [details] Proposed patch I've discovered a few build issues when playing with the two options (note that due to incompatibility with ASLR I haven't looked deeper).
I've been trying to build world with MK_ASAN=yes and MK_UBSAN=yes, but it always fails at a very early stage, with something entirely unexpected: -------------------------------------------------------------- >>> stage 2.1: cleaning up the object tree -------------------------------------------------------------- cd /home/dim/src/main; time env MACHINE_ARCH=amd64 MACHINE=amd64 CPUTYPE= CC="cc -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin" CXX="c++ -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin" CPP="cpp -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin" AS="as" AR="ar" ELFCTL="elfctl" LD="ld" LLVM_LINK="" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" STRIPBIN="strip" INSTALL="install -U" PATH=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/sbin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/sbin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin SYSROOT=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp make -f Makefile.inc1 BWPHASE=cleanobj DESTDIR=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp _NO_INCLUDE_COMPILERMK=t cleandir ===> lib (cleandir) make[3] warning: /lib: Permission denied. make[3]: "/home/dim/src/main/lib/Makefile" line 190: "Requested build with sanitizers but cannot build runtime libraries!" make[3]: stopped in /home/dim/src/main/lib *** Error code 1 For some reason, at this point COMPILER_TYPE is not correctly defined, and the Makefile dies. Adding arichardson@ on CC, maybe he knows more about this...
Maybe diff --git a/lib/Makefile b/lib/Makefile index 2cba5feaee19..bd28b974c673 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -185,7 +185,7 @@ SUBDIR.${MK_STATS}+= libstats ${MACHINE_CPUARCH} == "arm" || ${MACHINE_CPUARCH} == "i386" || \ ${MACHINE_CPUARCH} == "powerpc") _libclang_rt= libclang_rt -.elif ${MK_ASAN} != "no" || ${MK_UBSAN} != "no" +.elif (${MK_ASAN} != "no" || ${MK_UBSAN} != "no") && make(all) .error "Requested build with sanitizers but cannot build runtime libraries!" .endif
https://reviews.freebsd.org/D32805 for my change
I was seeing two linker errors (one with WITH_UBSAN, the other WITH_ASAN, notably I don't use MK_*SAN), reporters in September in: https://www.mail-archive.com/freebsd-current@freebsd.org/msg184811.html The first (ftpd.full) was resolved via: https://reviews.freebsd.org/D31945 and my latest buildworld has passed that, leaving only the following: ld: error: undefined symbol: lldb_private::formatters::CMTimeSummaryProvider ld: error: undefined symbol: lldb_private::AppleObjCRuntimeV1 (and V2) ld: error: undefined symbol: lldb_private::CFBasicHash ld: error: undefined symbol: lldb_private::lldb_renderscript ld: error: undefined symbol: NewCommandObjectRenderScriptScriptGroup(lldb_private::CommandInterpreter&) These errors appear similar (same?) as those reported in 2017: https://lists.freebsd.org/pipermail/freebsd-current/2017-July/066648.html Where Dim mentions in a follow-up: Apparently in some scenarios more object files are needed than the minimum set I put in liblldb's Makefile. Is that still the case now? If so, its interesting that I haven't come across other reports of the issue (unless no-one is testing *SAN yet)
@Xin Do you think your patch addresses the undefined symbols listed in my comment 4 or is it possible there are more SRCS's that may be required?
Testing build on latest current sources with attachment 229774 [details] now
I got a successful build with: # Speed MALLOC_PRODUCTION=yes WITHOUT_LLVM_ASSERTIONS=ses # Hardening WITH_PIE=yes WITH_BIND_NOW=yes WITH_RETPOLINE=yes WITH_SSP=yes # Sanitizers WITH_ASAN=yes WITH_UBSAN=yes WITH_CCACHE_BUILD=yes CCACHE_PREFIX=/usr/local/bin/distcc 7256.14 real 4002.80 user 1113.36 sys --- buildworld_epilogue --- -------------------------------------------------------------- >>> World build completed on Sat Dec 18 09:07:13 UTC 2021 >>> World built in 35606 seconds, ncpu: 2, make -j8
Hi, This doesn't work in very recent -current on arm64.aarch64 system: FreeBSD 14.0-CURRENT (CERES) #2 main-n251742-cfca19c28e5: Thu Dec 16 23:54:49 GMT 2021 hardware: raspberry pi4b+ / 8GB % gvs srctree last updated: Mon Dec 20 12:49:31 2021 +0100 version: 251840 build results here: https://cloud.zyxst.net/~john/FreeBSD/aarch64.rpi4.mitigations.src-current/buildfail.txt src.conf, src-env.conf, make.conf are here: https://cloud.zyxst.net/~john/FreeBSD/aarch64.rpi4.mitigations.src-current/
I *wasn't* able to apply the patch in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260099#c0
sorry, I meant https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260099#c6
(In reply to tech-lists from comment #9) Can you apply the changes manually
FYI: Manually applying this patch and trying buildworld built lldb. This is based on also using: # git -C /usr/main-src/ diff /usr/main-src/usr.bin/clang/ diff --git a/usr.bin/clang/llvm.prog.mk b/usr.bin/clang/llvm.prog.mk index 3a708805d3ea..74bed2ecd314 100644 --- a/usr.bin/clang/llvm.prog.mk +++ b/usr.bin/clang/llvm.prog.mk @@ -25,6 +25,7 @@ PACKAGE= clang .if ${.MAKE.OS} == "FreeBSD" || !defined(BOOTSTRAPPING) LIBADD+= execinfo LIBADD+= tinfow +LIBADD+= z .endif LIBADD+= pthread to get by an earlier stopping point. The next stopping point now is: --- all_subdir_usr.bin/clang/llvm-pdbutil --- Building /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/usr.bin/clang/llvm-pdbutil/llvm-pdbutil.full . . . --- all_subdir_usr.bin/clang --- --- llvm-pdbutil.full --- ld: error: undefined symbol: llvm::codeview::limitSymbolArrayToScope(llvm::VarStreamArray<llvm::codeview::CVRecord<llvm::codeview::SymbolKind>, llvm::VarStreamArrayExtractor<llvm::codeview::CVRecord<llvm::codeview::SymbolKind> > > const&, unsigned int) >>> referenced by ModuleDebugStream.cpp:91 (/usr/main-src/contrib/llvm-project/llvm/lib/DebugInfo/PDB/Native/ModuleDebugStream.cpp:91) >>> ModuleDebugStream.o:(llvm::pdb::ModuleDebugStreamRef::getSymbolArrayForScope(unsigned int) const) in archive /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/lib/clang/libllvm/libllvm.a c++: error: linker command failed with exit code 1 (use -v to see invocation) *** [llvm-pdbutil.full] Error code 1
(In reply to Mark Millard from comment #12) I took a guess of: diff --git a/lib/clang/libllvm/Makefile b/lib/clang/libllvm/Makefile index 6df32d8b452a..9e37d8cdf829 100644 --- a/lib/clang/libllvm/Makefile +++ b/lib/clang/libllvm/Makefile @@ -470,6 +470,7 @@ SRCS_MIN+= DebugInfo/CodeView/RecordSerialization.cpp SRCS_MIN+= DebugInfo/CodeView/SimpleTypeSerializer.cpp SRCS_EXT+= DebugInfo/CodeView/StringsAndChecksums.cpp SRCS_MIN+= DebugInfo/CodeView/SymbolDumper.cpp +SRCS_EXT+= DebugInfo/CodeView/SymbolRecordHelpers.cpp SRCS_MIN+= DebugInfo/CodeView/SymbolRecordMapping.cpp SRCS_EXT+= DebugInfo/CodeView/SymbolSerializer.cpp SRCS_MIN+= DebugInfo/CodeView/TypeDumpVisitor.cpp and that let buildworld complete in my context. (I've no clue if _EXT is actually the right category overall.) I will note that some earlier experiments with trying -flto=thin based builds means that I do have some adde __attribute__((used)) __attribute((retain)) usage in a few places. I've no clue if any of things contributed to having completion or not. For the earlier-noted: # git -C /usr/main-src/ diff /usr/main-src/usr.bin/clang/ diff --git a/usr.bin/clang/llvm.prog.mk b/usr.bin/clang/llvm.prog.mk index 3a708805d3ea..74bed2ecd314 100644 --- a/usr.bin/clang/llvm.prog.mk +++ b/usr.bin/clang/llvm.prog.mk @@ -25,6 +25,7 @@ PACKAGE= clang .if ${.MAKE.OS} == "FreeBSD" || !defined(BOOTSTRAPPING) LIBADD+= execinfo LIBADD+= tinfow +LIBADD+= z .endif LIBADD+= pthread I also do not know if the LIBADD+= z is correctly handled for the 4 combinations of values for each of ${.MAKE.OS} == "FreeBSD" and !defined(BOOTSTRAPPING) (or if other contexts would need checking). I just took a guess.
(In reply to Mark Millard from comment #13) Just as a quick cross check, I did installworld into /usr/obj/DESTDIRs/main-amd64-xSAN-chroot/ then: # sysctl kern.elf64.aslr.pie_enable=0 # chroot /usr/obj/DESTDIRs/main-amd64-xSAN-chroot # ls -Tld /* /usr/main-src/lib/libc/stdio/fread.c:133:10: runtime error: applying zero offset to null pointer SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/main-src/lib/libc/stdio/fread.c:133:10 in -r--r--r-- 1 root wheel 6109 Jan 7 09:30:01 2022 /COPYRIGHT drwxr-xr-x 2 root wheel 46 Jan 7 09:29:49 2022 /bin drwxr-xr-x 14 root wheel 65 Jan 7 09:30:01 2022 /boot dr-xr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /dev drwxr-xr-x 28 root wheel 104 Jan 7 09:30:03 2022 /etc drwxr-xr-x 5 root wheel 69 Jan 7 09:29:48 2022 /lib drwxr-xr-x 3 root wheel 5 Jan 7 09:29:40 2022 /libexec drwxr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /media drwxr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /mnt drwxr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /net dr-xr-xr-x 2 root wheel 2 Jan 7 09:29:38 2022 /proc drwxr-xr-x 2 root wheel 149 Jan 7 09:29:48 2022 /rescue drwxr-xr-x 2 root wheel 7 Jan 7 09:30:02 2022 /root drwxr-xr-x 2 root wheel 148 Jan 7 09:29:50 2022 /sbin drwxrwxrwt 2 root wheel 2 Jan 7 09:29:38 2022 /tmp drwxr-xr-x 14 root wheel 14 Jan 7 09:29:38 2022 /usr drwxr-xr-x 24 root wheel 24 Jan 7 09:29:38 2022 /var So the build works to some extent.