Summary: | databases/mysql80-server: 8.0.27 routerfuzz_router_uri crashes (SIGILL) on Penryn CPU (missing popcnt) | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Ports & Packages | Reporter: | Eric Rucker <bhtooefr> | ||||||||
Component: | Individual Port(s) | Assignee: | Jochen Neumeister <joneum> | ||||||||
Status: | Closed FIXED | ||||||||||
Severity: | Affects Some People | CC: | busheval, chris, dim, joneum, sascha, viktor.stujber+freebsd-bugs_v4CCPfay | ||||||||
Priority: | --- | Keywords: | crash, needs-qa | ||||||||
Version: | Latest | Flags: | bugzilla:
maintainer-feedback?
(joneum) koobs: maintainer-feedback+ |
||||||||
Hardware: | amd64 | ||||||||||
OS: | Any | ||||||||||
See Also: | https://github.com/llvm/llvm-project/issues/52893 | ||||||||||
Attachments: |
|
Description
Eric Rucker
2021-08-29 23:05:56 UTC
Hi I have same problem on my test machine on Intel(R) Celeron(R) CPU 900@2.20GHz (Penryn). I try CPUTYPE?=penryn, CPUTYPE?=core2 and CPUTYPE?=native with and without MAKE_JOBS_UNSAFE=yes in /etc/make.conf, but no result. Still have same error on build mysql80-server-8.0.26. FreeBSD 12.2-RELEASE-p7 GENERIC amd64 root@test:/home/test # c++ --version FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) Target: x86_64-unknown-freebsd12.2 Thread model: posix InstalledDir: /usr/bin Preparing corpus for routerfuzz_router_uri cd /usr/ports/databases/mysql80-server/work/.build/router/tests/fuzzers && ./routerfuzz_router_uri -merge=1 -verbosity=0 -merge_control_file="/usr/ports/databases/mysql80-server/work/.build/router/tests/fuzzers/routerfuzz_router_uri.control" /usr/ports/databases/mysql80-server/work/mysql-8.0.26/router/tests/fuzzers/corpus/fuzz_router_uri /usr/ports/databases/mysql80-server/work/.build/router/tests/fuzzers/corpus/routerfuzz_router_uri 2> /dev/null Illegal instruction (core dumped) *** [router/tests/fuzzers/routerfuzz_router_uri] Error code 132 make[4]: *** router/tests/fuzzers/routerfuzz_router_uri removed make[4]: stopped in /usr/ports/databases/mysql80-server/work/.build 1 error make[4]: stopped in /usr/ports/databases/mysql80-server/work/.build *** [router/tests/fuzzers/CMakeFiles/routerfuzz_router_uri.dir/all] Error code 2 make[3]: stopped in /usr/ports/databases/mysql80-server/work/.build 1 error make[3]: stopped in /usr/ports/databases/mysql80-server/work/.build *** [all] Error code 2 make[2]: stopped in /usr/ports/databases/mysql80-server/work/.build 1 error make[2]: stopped in /usr/ports/databases/mysql80-server/work/.build ===> Compilation failed unexpectedly. Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer. *** Error code 1 Stop. make[1]: stopped in /usr/ports/databases/mysql80-server *** Error code 1 Stop. make: stopped in /usr/ports/databases/mysql80-server ===>>> make build failed for databases/mysql80-server ===>>> Aborting update ===>>> You can restart from the point of failure with this command line: portmaster <flags> databases/mysql80-server This command has been saved to ~/portmasterfail.txt I cannot reproduce the problem. I tested it on 12.2 and 13.0. both on i386 and on amd64 the build ran through without problems (In reply to Jochen Neumeister from comment #2) Just tested again on 12.2, still failing in the same place. What processor are you testing this on? Anything Core i series or newer, or AMD Phenom or newer, won't exhibit this problem, because they have the popcnt instruction. And, as 8.0.26_1 is out, just confirmed, the issue is still occurring on 8.0.26_1. Hi, In 8.0.26_1 I still have same error in build with 'make DISABLE_VULNERABILITIES=yes'. I can confirm occurrence of this problem on 13.0-RELEASE-p4 amd64, Xeon E5420 (Harpertown). For those that are affected or able to reproduce this issue: - Is the issue reproducible with an empty /etc/make.conf - Please include (as a single attachment): - uname -a output - /var/run/dmesg.boot output - /etc/src.conf contents if not empty - kernel configuration if not GENERIC ^Triage: Request feedback from dim @ toolchain (In reply to Kubilay Kocak from comment #7) Additionally in your attachment: - /etc/make.conf contents if not empty when reproducible Created attachment 230433 [details] Requested information from affected system (In reply to Kubilay Kocak from comment #8) Issue is reproducible with empty /etc/make.conf (I normally run one, though, and received some warnings from this build about OpenSSL as a result). /etc/src.conf does not exist on this system, and kernel is GENERIC, so it's just uname -a and /var/run/dmesg.boot in this file. Hi, I looked into this a bit and asked around. In my case at least, I concluded that the bytecode in /usr/lib/clang/11.0.1/lib/freebsd/libclang_rt.fuzzer-x86_64.a includes multiple cases of 'popcnt'. The related llvm source code is pure C++ (no inline asm) and involves simple arithmetic, so this is most likely a micro-optimization meant for applicable cpus. This implies that buildworld mis-compiled contrib/llvm-project/compiler-rt/lib/fuzzer, and who knows what else. My cpu is old intel atom. My target is x86_64-unknown-freebsd13.0. Kernel and world are 13-release from Aug 20. Invoking "clang -### -x c -march=native - -E </dev/null" gives an argument list that includes "-popcnt", so this instruction is supposed to be avoided. However, for some reason, for this fuzzer library, it wasn't. The rest of the OS runs fine, so the issue might only exist in specific clang support libraries. I am not sure how to investigate further. I considered escalating this to the maintainer of contrib/llvm-project (dim@). CURRENT is already at llvm-13.0 so this might have been silently fixed already. For the moment, it is possible to complete the build of mysql80-server by doing 'make -i' on the port directory, to resume the aborted source build and skip over all build errors. Created attachment 230461 [details]
Avoid hardcoded popcnt if not available for the target CPU
Please try this patch. Either build and install world, or rebuild and reinstall the lib/libclang_rt/fuzzer and lib/libclang_rt/fuzzer_no_main directories. Then attempt to build the mysql80 port again.
Thank you very much, ATTRIBUTE_TARGET_POPCNT is indeed the root cause, and I should have noticed it when first looking at the llvm sources - it even has popcnt in the name. At that time I did not know what I was looking at, since the macro is obscuring the true meaning. I have discussed this in #llvm and wrote up an issue report at https://github.com/llvm/llvm-project/issues/52893 with all the supporting info. Your patch probably works, but depending on what exactly it does, it might be simpler to just remove the whole thing. Let's see what the llvm devs have to say about it. This bad code was in there since at least 2016. Apparently no freebsd port until now used -fsanitize=fuzzer, or something. Until mysql resolved an issue that prevented a bunch of their tests from running. This fix was included in release 8.0.26, and it finally revealed this long-dormant llvm bug. A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=133180557479cd9676758e6f3f93a9d3e1c6b532 commit 133180557479cd9676758e6f3f93a9d3e1c6b532 Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2021-12-30 09:53:25 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2021-12-30 09:55:49 +0000 Avoid emitting popcnt in libclang_rt.fuzzer*.a if unsupported Since popcnt is only supported by CPUTYPE=nehalem and later, ensure that this instruction is only emitted when appropriate. Otherwise, programs using the library can abort with SIGILL. See also: https://github.com/llvm/llvm-project/issues/52893 PR: 258156 Reported by: Eric Rucker <bhtooefr@bhtooefr.org> MFC after: 3 days contrib/llvm-project/compiler-rt/lib/fuzzer/FuzzerPlatform.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Is there anything that can be done in ports to workaround this until all supported FreeBSD versions have this fix/change ? A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=802ff7fcee24cb224ea430ac45bece8f8743791f commit 802ff7fcee24cb224ea430ac45bece8f8743791f Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2021-12-30 09:53:25 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2022-01-02 12:02:19 +0000 Avoid emitting popcnt in libclang_rt.fuzzer*.a if unsupported Since popcnt is only supported by CPUTYPE=nehalem and later, ensure that this instruction is only emitted when appropriate. Otherwise, programs using the library can abort with SIGILL. See also: https://github.com/llvm/llvm-project/issues/52893 PR: 258156 Reported by: Eric Rucker <bhtooefr@bhtooefr.org> MFC after: 3 days (cherry picked from commit 133180557479cd9676758e6f3f93a9d3e1c6b532) contrib/llvm-project/compiler-rt/lib/fuzzer/FuzzerPlatform.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=50b0f99010bc485e7f60e948ff3bc590fbbeea43 commit 50b0f99010bc485e7f60e948ff3bc590fbbeea43 Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2021-12-30 09:53:25 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2022-01-02 12:02:14 +0000 Avoid emitting popcnt in libclang_rt.fuzzer*.a if unsupported Since popcnt is only supported by CPUTYPE=nehalem and later, ensure that this instruction is only emitted when appropriate. Otherwise, programs using the library can abort with SIGILL. See also: https://github.com/llvm/llvm-project/issues/52893 PR: 258156 Reported by: Eric Rucker <bhtooefr@bhtooefr.org> MFC after: 3 days (cherry picked from commit 133180557479cd9676758e6f3f93a9d3e1c6b532) contrib/llvm-project/compiler-rt/lib/fuzzer/FuzzerPlatform.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (In reply to Kubilay Kocak from comment #14) Disable the fuzzer based components, maybe? Unfortunately the hardcoded popcnt instructions ended up in a system library (libclang_rt.fuzzer-*.a) so the only way to avoid it is to stop linking against this library. (In reply to Kubilay Kocak from comment #14) To unbreak the build, you can disable MySQL unit tests by adding -DWITH_UNIT_TESTS=OFF to CMAKE_ARGS in the ports Makefile. (In reply to Sascha Klauder from comment #18) If this toggle works and disables tests for both individual subcomponents as well as the huge mysql-test/ directory, it would be worthwhile to add a TEST port option, for speeding up the build, as well as working around this temporary issue. |