Bug 260099 - Fix build when building with WITH_ASAN and WITH_UBSAN
Summary: Fix build when building with WITH_ASAN and WITH_UBSAN
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Dimitry Andric
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2021-11-29 03:08 UTC by Xin LI
Modified: 2022-01-07 09:39 UTC (History)
6 users (show)

See Also:
koobs: maintainer-feedback? (dim)
koobs: maintainer-feedback? (delphij)


Attachments
Proposed patch (2.46 KB, patch)
2021-11-29 03:08 UTC, Xin LI
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Xin LI freebsd_committer 2021-11-29 03:08:39 UTC
Created attachment 229774 [details]
Proposed patch

I've discovered a few build issues when playing with the two options (note that due to incompatibility with ASLR I haven't looked deeper).
Comment 1 Dimitry Andric freebsd_committer 2021-12-07 17:42:03 UTC
I've been trying to build world with MK_ASAN=yes and MK_UBSAN=yes, but it always fails at a very early stage, with something entirely unexpected:

--------------------------------------------------------------
>>> stage 2.1: cleaning up the object tree
--------------------------------------------------------------
cd /home/dim/src/main; time env MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE= CC="cc -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin" CXX="c++  -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin"  CPP="cpp -target x86_64-unknown-freebsd14.0 --sysroot=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp -B/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin"  AS="as" AR="ar" ELFCTL="elfctl" LD="ld"  LLVM_LINK="" NM=nm OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" STRIPBIN="strip"  INSTALL="install -U"  PATH=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/sbin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/usr/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/sbin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/bin:/home/dim/obj/home/dim/src/main/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin  SYSROOT=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp make  -f Makefile.inc1  BWPHASE=cleanobj  DESTDIR=/home/dim/obj/home/dim/src/main/amd64.amd64/tmp _NO_INCLUDE_COMPILERMK=t cleandir
===> lib (cleandir)
make[3] warning: /lib: Permission denied.
make[3]: "/home/dim/src/main/lib/Makefile" line 190: "Requested build with sanitizers but cannot build runtime libraries!"

make[3]: stopped in /home/dim/src/main/lib
*** Error code 1

For some reason, at this point COMPILER_TYPE is not correctly defined, and the Makefile dies.

Adding arichardson@ on CC, maybe he knows more about this...
Comment 2 Ed Maste freebsd_committer 2021-12-07 17:50:13 UTC
Maybe

diff --git a/lib/Makefile b/lib/Makefile
index 2cba5feaee19..bd28b974c673 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -185,7 +185,7 @@ SUBDIR.${MK_STATS}+=        libstats
     ${MACHINE_CPUARCH} == "arm" || ${MACHINE_CPUARCH} == "i386" || \
     ${MACHINE_CPUARCH} == "powerpc")
 _libclang_rt=  libclang_rt
-.elif ${MK_ASAN} != "no" || ${MK_UBSAN} != "no"
+.elif (${MK_ASAN} != "no" || ${MK_UBSAN} != "no") && make(all)
 .error "Requested build with sanitizers but cannot build runtime libraries!"
 .endif
Comment 3 Ed Maste freebsd_committer 2021-12-08 19:40:06 UTC
https://reviews.freebsd.org/D32805 for my change
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-14 22:20:54 UTC
I was seeing two linker errors (one with WITH_UBSAN, the other WITH_ASAN, notably I don't use MK_*SAN), reporters in September in: https://www.mail-archive.com/freebsd-current@freebsd.org/msg184811.html

The first (ftpd.full) was resolved via: https://reviews.freebsd.org/D31945 and my latest buildworld has passed that, leaving only the following:

 ld: error: undefined symbol: lldb_private::formatters::CMTimeSummaryProvider
 ld: error: undefined symbol: lldb_private::AppleObjCRuntimeV1 (and V2)
 ld: error: undefined symbol: lldb_private::CFBasicHash
 ld: error: undefined symbol: lldb_private::lldb_renderscript
 ld: error: undefined symbol: NewCommandObjectRenderScriptScriptGroup(lldb_private::CommandInterpreter&)

These errors appear similar (same?) as those reported in 2017:

  https://lists.freebsd.org/pipermail/freebsd-current/2017-July/066648.html

Where Dim mentions in a follow-up:

 Apparently in some scenarios more object files are needed
 than the minimum set I put in liblldb's Makefile.

Is that still the case now? If so, its interesting that I haven't come across other reports of the issue (unless no-one is testing *SAN yet)
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-17 22:34:01 UTC
@Xin Do you think your patch addresses the undefined symbols listed in my comment 4 or is it possible there are more SRCS's that may be required?
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-18 01:06:38 UTC
Testing build on latest current sources with attachment 229774 [details] now
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-18 22:59:56 UTC
I got a successful build with:

# Speed
MALLOC_PRODUCTION=yes
WITHOUT_LLVM_ASSERTIONS=ses

# Hardening
WITH_PIE=yes
WITH_BIND_NOW=yes
WITH_RETPOLINE=yes
WITH_SSP=yes

# Sanitizers
WITH_ASAN=yes
WITH_UBSAN=yes

WITH_CCACHE_BUILD=yes
CCACHE_PREFIX=/usr/local/bin/distcc


     7256.14 real      4002.80 user      1113.36 sys
--- buildworld_epilogue ---
--------------------------------------------------------------
>>> World build completed on Sat Dec 18 09:07:13 UTC 2021
>>> World built in 35606 seconds, ncpu: 2, make -j8
Comment 8 tech-lists 2021-12-20 13:41:40 UTC
Hi,

This doesn't work in very recent -current on arm64.aarch64

system:
FreeBSD 14.0-CURRENT (CERES) #2 main-n251742-cfca19c28e5: Thu Dec 16 23:54:49 GMT 2021

hardware: raspberry pi4b+ / 8GB

% gvs
srctree last updated: Mon Dec 20 12:49:31 2021 +0100 version: 251840

build results here:
https://cloud.zyxst.net/~john/FreeBSD/aarch64.rpi4.mitigations.src-current/buildfail.txt

src.conf, src-env.conf, make.conf are here:
https://cloud.zyxst.net/~john/FreeBSD/aarch64.rpi4.mitigations.src-current/
Comment 9 tech-lists 2021-12-20 13:44:10 UTC
I *wasn't* able to apply the patch in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260099#c0
Comment 10 tech-lists 2021-12-20 13:45:14 UTC
sorry, I meant https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260099#c6
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-23 07:10:15 UTC
(In reply to tech-lists from comment #9)

Can you apply the changes manually
Comment 12 Mark Millard 2022-01-07 05:46:02 UTC
FYI: Manually applying this patch and trying buildworld built lldb. This is based on also using:

# git -C /usr/main-src/ diff /usr/main-src/usr.bin/clang/
diff --git a/usr.bin/clang/llvm.prog.mk b/usr.bin/clang/llvm.prog.mk
index 3a708805d3ea..74bed2ecd314 100644
--- a/usr.bin/clang/llvm.prog.mk
+++ b/usr.bin/clang/llvm.prog.mk
@@ -25,6 +25,7 @@ PACKAGE=      clang
 .if ${.MAKE.OS} == "FreeBSD" || !defined(BOOTSTRAPPING)
 LIBADD+=       execinfo
 LIBADD+=       tinfow
+LIBADD+=       z
 .endif
 LIBADD+=       pthread

to get by an earlier stopping point.

The next stopping point now is:

--- all_subdir_usr.bin/clang/llvm-pdbutil ---
Building /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/usr.bin/clang/llvm-pdbutil/llvm-pdbutil.full
. . .
--- all_subdir_usr.bin/clang ---
--- llvm-pdbutil.full ---
ld: error: undefined symbol: llvm::codeview::limitSymbolArrayToScope(llvm::VarStreamArray<llvm::codeview::CVRecord<llvm::codeview::SymbolKind>, llvm::VarStreamArrayExtractor<llvm::codeview::CVRecord<llvm::codeview::SymbolKind> > > const&, unsigned int)
>>> referenced by ModuleDebugStream.cpp:91 (/usr/main-src/contrib/llvm-project/llvm/lib/DebugInfo/PDB/Native/ModuleDebugStream.cpp:91)
>>>               ModuleDebugStream.o:(llvm::pdb::ModuleDebugStreamRef::getSymbolArrayForScope(unsigned int) const) in archive /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/lib/clang/libllvm/libllvm.a
c++: error: linker command failed with exit code 1 (use -v to see invocation)
*** [llvm-pdbutil.full] Error code 1
Comment 13 Mark Millard 2022-01-07 09:17:39 UTC
(In reply to Mark Millard from comment #12)

I took a guess of:

diff --git a/lib/clang/libllvm/Makefile b/lib/clang/libllvm/Makefile
index 6df32d8b452a..9e37d8cdf829 100644
--- a/lib/clang/libllvm/Makefile
+++ b/lib/clang/libllvm/Makefile
@@ -470,6 +470,7 @@ SRCS_MIN+=  DebugInfo/CodeView/RecordSerialization.cpp
 SRCS_MIN+=     DebugInfo/CodeView/SimpleTypeSerializer.cpp
 SRCS_EXT+=     DebugInfo/CodeView/StringsAndChecksums.cpp
 SRCS_MIN+=     DebugInfo/CodeView/SymbolDumper.cpp
+SRCS_EXT+=     DebugInfo/CodeView/SymbolRecordHelpers.cpp
 SRCS_MIN+=     DebugInfo/CodeView/SymbolRecordMapping.cpp
 SRCS_EXT+=     DebugInfo/CodeView/SymbolSerializer.cpp
 SRCS_MIN+=     DebugInfo/CodeView/TypeDumpVisitor.cpp

and that let buildworld complete in my context. (I've no clue
if _EXT is actually the right category overall.)

I will note that some earlier experiments with trying -flto=thin
based builds means that I do have some adde
__attribute__((used)) __attribute((retain)) usage in a few places.
I've no clue if any of things contributed to having completion or
not.

For the earlier-noted:

# git -C /usr/main-src/ diff /usr/main-src/usr.bin/clang/
diff --git a/usr.bin/clang/llvm.prog.mk b/usr.bin/clang/llvm.prog.mk
index 3a708805d3ea..74bed2ecd314 100644
--- a/usr.bin/clang/llvm.prog.mk
+++ b/usr.bin/clang/llvm.prog.mk
@@ -25,6 +25,7 @@ PACKAGE=      clang
 .if ${.MAKE.OS} == "FreeBSD" || !defined(BOOTSTRAPPING)
 LIBADD+=       execinfo
 LIBADD+=       tinfow
+LIBADD+=       z
 .endif
 LIBADD+=       pthread

I also do not know if the LIBADD+= z is correctly handled for the
4 combinations of values for each of ${.MAKE.OS} == "FreeBSD" and
!defined(BOOTSTRAPPING) (or if other contexts would need checking).
I just took a guess.
Comment 14 Mark Millard 2022-01-07 09:39:09 UTC
(In reply to Mark Millard from comment #13)

Just as a quick cross check, I did installworld into
/usr/obj/DESTDIRs/main-amd64-xSAN-chroot/ then:

# sysctl kern.elf64.aslr.pie_enable=0
# chroot /usr/obj/DESTDIRs/main-amd64-xSAN-chroot
# ls -Tld /*
/usr/main-src/lib/libc/stdio/fread.c:133:10: runtime error: applying zero offset to null pointer
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/main-src/lib/libc/stdio/fread.c:133:10 in 
-r--r--r--   1 root  wheel  6109 Jan  7 09:30:01 2022 /COPYRIGHT
drwxr-xr-x   2 root  wheel    46 Jan  7 09:29:49 2022 /bin
drwxr-xr-x  14 root  wheel    65 Jan  7 09:30:01 2022 /boot
dr-xr-xr-x   2 root  wheel     2 Jan  7 09:29:38 2022 /dev
drwxr-xr-x  28 root  wheel   104 Jan  7 09:30:03 2022 /etc
drwxr-xr-x   5 root  wheel    69 Jan  7 09:29:48 2022 /lib
drwxr-xr-x   3 root  wheel     5 Jan  7 09:29:40 2022 /libexec
drwxr-xr-x   2 root  wheel     2 Jan  7 09:29:38 2022 /media
drwxr-xr-x   2 root  wheel     2 Jan  7 09:29:38 2022 /mnt
drwxr-xr-x   2 root  wheel     2 Jan  7 09:29:38 2022 /net
dr-xr-xr-x   2 root  wheel     2 Jan  7 09:29:38 2022 /proc
drwxr-xr-x   2 root  wheel   149 Jan  7 09:29:48 2022 /rescue
drwxr-xr-x   2 root  wheel     7 Jan  7 09:30:02 2022 /root
drwxr-xr-x   2 root  wheel   148 Jan  7 09:29:50 2022 /sbin
drwxrwxrwt   2 root  wheel     2 Jan  7 09:29:38 2022 /tmp
drwxr-xr-x  14 root  wheel    14 Jan  7 09:29:38 2022 /usr
drwxr-xr-x  24 root  wheel    24 Jan  7 09:29:38 2022 /var

So the build works to some extent.