LLVM's sanitizer runtime recently started defining a hexdump() symbol for some unstated reason: https://reviews.llvm.org/D110471 Now, executables in the base system which define a hexdump(), like mpsutil, fail to build because of a duplicate hexdump symbol definition. I tend to think that LLVM shouldn't be intercepting this function: it's a common function name and it's pretty easy to build libutil with ASAN enabled if so desired. The alternative would be to rename symbols in the base system, but that's a bit annoying.
https://github.com/llvm/llvm-project/pull/79378
Looks like we'll want to import the upstream revert that was committed?
(In reply to Dimitry Andric from comment #2) Yes please! Sadly, that's not enough for the WITH_ASAN build to succeed. Now it fails while linking llvm-ar: ld: error: undefined symbol: llvm::MachO::Symbol::operator==(llvm::MachO::Symbol const&) const >>> referenced by Symbol.h:142 (/home/markj/sb/main/src/contrib/llvm-project/llvm/include/llvm/TextAPI/Symbol.h:142) >>> InterfaceFile.o:(bool llvm::operator==<llvm::DenseMap<llvm::SymbolsMapKey, llvm::MachO::Symbol*, llvm::DenseMapInfo<llvm::SymbolsMapKey, void>, llvm::detail::DenseMapPair<llvm::SymbolsMapKey, llvm::MachO::Symbol*>>, llvm::DenseMapInfo<llvm::SymbolsMapKey, void>, llvm::detail::DenseMapPair<llvm::Symb olsMapKey, llvm::MachO::Symbol*>>(llvm::DenseMapBase<llvm::DenseMap<llvm::SymbolsMapKey, llvm::MachO::Symbol*, llvm::DenseMapInfo<llvm::SymbolsMapKey, void>, llvm::detail::DenseMapPair<llvm::SymbolsMapKey, llvm::MachO::Symbol*>>, llvm::SymbolsMapKey, llvm::MachO::Symbol*, llvm::DenseMapInfo<llvm::SymbolsMapKey, void> , llvm::detail::DenseMapPair<llvm::SymbolsMapKey, llvm::MachO::Symbol*>> const&, llvm::DenseMapBase<llvm::DenseMap<llvm::SymbolsMapKey, llvm::MachO::Symbol*, llvm::DenseMapInfo<llvm::SymbolsMapKey, void>, llvm::detail::DenseMapPair<llvm::SymbolsMapKey, llvm::MachO::Symbol*>>, llvm::SymbolsMapKey, llvm::MachO::Symbol* , llvm::DenseMapInfo<llvm::SymbolsMapKey, void>, llvm::detail::DenseMapPair<llvm::SymbolsMapKey, llvm::MachO::Symbol*>> const&)) in archive /usr/obj/home/markj/sb/main/src/amd64.amd64/lib/clang/libllvm/libllvm.a
(In reply to Mark Johnston from comment #3) Looks like libllvm/Makefile never includes TextAPI/Symbol.cpp in SRCS. Fixing that fixes that particular problem (there is at least one other instance). Though, it's a bit weird that a plain WITH_LLVM_BINUTILS build doesn't hit this.
(In reply to Mark Johnston from comment #4) The optimization to avoid building some cpp files works often, but not always, due to subtle reasons. In some cases, if you turn on debug information, or change optimization levels, previously unreferenced symbols now do get referenced. In any case, I'm getting different errors here: --- all_subdir_libexec --- ld: error: undefined symbol: __asan_option_detect_stack_use_after_return >>> referenced by reloc.c:64 (/home/dim/src/freebsd/src/libexec/rtld-elf/amd64/reloc.c:64) >>> reloc.o:(do_copy_relocations) >>> referenced by reloc.c:129 (/home/dim/src/freebsd/src/libexec/rtld-elf/amd64/reloc.c:129) >>> reloc.o:(reloc_non_plt) >>> referenced by reloc.c:361 (/home/dim/src/freebsd/src/libexec/rtld-elf/amd64/reloc.c:361) >>> reloc.o:(reloc_jmpslots) >>> referenced 53 more times ld: error: undefined symbol: __asan_stack_malloc_1 >>> referenced by reloc.c:64 (/home/dim/src/freebsd/src/libexec/rtld-elf/amd64/reloc.c:64) >>> reloc.o:(do_copy_relocations) >>> referenced by rtld.c:1976 (/home/dim/src/freebsd/src/libexec/rtld-elf/rtld.c:1976) >>> rtld.o:(find_symdef) >>> referenced by rtld.c:4028 (/home/dim/src/freebsd/src/libexec/rtld-elf/rtld.c:4028) >>> rtld.o:(dlvsym) >>> referenced 10 more times ld: error: undefined symbol: __asan_report_load8_noabort >>> referenced by reloc.c:70 (/home/dim/src/freebsd/src/libexec/rtld-elf/amd64/reloc.c:70) >>> reloc.o:(do_copy_relocations) >>> referenced by reloc.c:70 (/home/dim/src/freebsd/src/libexec/rtld-elf/amd64/reloc.c:70) >>> reloc.o:(do_copy_relocations) >>> referenced by reloc.c:72 (/home/dim/src/freebsd/src/libexec/rtld-elf/amd64/reloc.c:72) >>> reloc.o:(do_copy_relocations) >>> referenced 1769 more times in stage 4.4: building everything.
(In reply to Dimitry Andric from comment #5) That's strange, I wouldn't expect rtld and ASAN to play well together, and indeed, libexec/rtld-elf/Makefile has MK_ASAN=no.
(In reply to Mark Johnston from comment #6) Oh that is my bad, I ran buildworld with MK_ASAN=yes as the last argument to make, which causes it to override any local setting. That won't work in this case, so I'll have to pass it via the evironment or make.conf.
(In reply to Dimitry Andric from comment #7) Why not run "make buildworld WITH_ASAN="? That's what I've been doing.
(In reply to Mark Johnston from comment #8) I dislike the WITH and WITHOUT syntax, as I find it utterly confusing. But no matter what you do, you should really not pass any MK, WITH or WITHOUT settings as make command line variables, since they cannot be overridden in any sub-Makefile. I fall into this trap every time. )
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=5c38ea60a04199dae2c6e5db82461e7275c7def6 commit 5c38ea60a04199dae2c6e5db82461e7275c7def6 Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2024-01-27 21:51:08 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2024-01-27 21:51:08 +0000 Add libllvm and liblldb source files to enable WITH_ASAN build This is another part of fixing the WITH_ASAN build. Some additional source files had to be added to libllvm and liblldb, since the ASan instrumentation causes symbols in those files to be referenced. Reported by: markj PR: 276597 MFC after: 3 days lib/clang/liblldb/Makefile | 5 +++++ lib/clang/libllvm/Makefile | 2 ++ 2 files changed, 7 insertions(+)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=31aedef26fc0613f5d61204c624451d57e716ed9 commit 31aedef26fc0613f5d61204c624451d57e716ed9 Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2024-01-27 21:48:58 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2024-01-27 21:48:58 +0000 Merge commit 16a1ef86cbc5 from llvm-project (by David CARLIER): [compiler-rt] remove hexdump interception. (#79378) a freebsd dev member reported a symbol conflict and intercepting this had little value anyway. This is one part of fixing the WITH_ASAN build. Some executables in the base system define their own hexdump() symbol, which would otherwise conflict with the ASan-intercepted one. Reported by: markj PR: 276597 MFC after: 3 days .../sanitizer_common/sanitizer_common_interceptors.inc | 15 --------------- .../sanitizer_common/sanitizer_platform_interceptors.h | 1 - 2 files changed, 16 deletions(-)
I checked that it now works for me (famous last words :), but if you can verify that too, please do so. I will then MFC those commits after 3 days.
(In reply to Dimitry Andric from comment #12) The build succeeds for me, thank you. When I try to boot a VM built from a world compiled with WITH_ASAN, every executable segfaults. The crash is happening in libc, in __archlevel_resolve(). Specifically, the "environ == NULL" check in env_archlevel() is somehow triggering a segfault - I think the assembly is checking the shadow map but it's not set up yet.
(In reply to Mark Johnston from comment #13) Disabling the sanitizer for amd64_archlevel.c fixes the segfaults, so I'll retry the test suite with that patch.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=31c3fc40dedddadaaa143f0ddf3d61ee7b2493ee commit 31c3fc40dedddadaaa143f0ddf3d61ee7b2493ee Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2024-01-27 21:48:58 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2024-01-30 17:53:14 +0000 Merge commit 16a1ef86cbc5 from llvm-project (by David CARLIER): [compiler-rt] remove hexdump interception. (#79378) a freebsd dev member reported a symbol conflict and intercepting this had little value anyway. This is one part of fixing the WITH_ASAN build. Some executables in the base system define their own hexdump() symbol, which would otherwise conflict with the ASan-intercepted one. Reported by: markj PR: 276597 MFC after: 3 days (cherry picked from commit 31aedef26fc0613f5d61204c624451d57e716ed9) .../sanitizer_common/sanitizer_common_interceptors.inc | 15 --------------- .../sanitizer_common/sanitizer_platform_interceptors.h | 1 - 2 files changed, 16 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6ccc51fd6f28af3a4516cc058135f519cbeb30e0 commit 6ccc51fd6f28af3a4516cc058135f519cbeb30e0 Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2024-01-27 21:51:08 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2024-01-30 17:53:25 +0000 Add libllvm and liblldb source files to enable WITH_ASAN build This is another part of fixing the WITH_ASAN build. Some additional source files had to be added to libllvm and liblldb, since the ASan instrumentation causes symbols in those files to be referenced. Reported by: markj PR: 276597 MFC after: 3 days (cherry picked from commit 5c38ea60a04199dae2c6e5db82461e7275c7def6) lib/clang/liblldb/Makefile | 5 +++++ lib/clang/libllvm/Makefile | 2 ++ 2 files changed, 7 insertions(+)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e68356676e9c4e5a36dd20560f19845ae073dab3 commit e68356676e9c4e5a36dd20560f19845ae073dab3 Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2024-01-27 21:51:08 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2024-01-30 17:53:51 +0000 Add libllvm and liblldb source files to enable WITH_ASAN build This is another part of fixing the WITH_ASAN build. Some additional source files had to be added to libllvm and liblldb, since the ASan instrumentation causes symbols in those files to be referenced. Reported by: markj PR: 276597 MFC after: 3 days (cherry picked from commit 5c38ea60a04199dae2c6e5db82461e7275c7def6) lib/clang/liblldb/Makefile | 5 +++++ lib/clang/libllvm/Makefile | 2 ++ 2 files changed, 7 insertions(+)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3bd3724e4c8685d98bfc6feb3024b8bb213eed49 commit 3bd3724e4c8685d98bfc6feb3024b8bb213eed49 Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2024-01-27 21:48:58 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2024-01-30 17:53:39 +0000 Merge commit 16a1ef86cbc5 from llvm-project (by David CARLIER): [compiler-rt] remove hexdump interception. (#79378) a freebsd dev member reported a symbol conflict and intercepting this had little value anyway. This is one part of fixing the WITH_ASAN build. Some executables in the base system define their own hexdump() symbol, which would otherwise conflict with the ASan-intercepted one. Reported by: markj PR: 276597 MFC after: 3 days (cherry picked from commit 31aedef26fc0613f5d61204c624451d57e716ed9) .../sanitizer_common/sanitizer_common_interceptors.inc | 15 --------------- .../sanitizer_common/sanitizer_platform_interceptors.h | 1 - 2 files changed, 16 deletions(-)
Do you want to keep this open, now world at least builds?
(In reply to Dimitry Andric from comment #19) Yep, we can close it. The segfaults are fixed by commit 4dedcb1bb54cbbe8043c79ad733f966b6ffc6972, which I merged to stable/14. A quick smoke test suggests that everything is working. Thanks for the quick fixes.