Making a long story short: clang: error: clang frontend command failed with exit code 134 (use -v to see invocation) FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152) Target: aarch64-unknown-freebsd14.0 Thread model: posix InstalledDir: /usr/bin #0 0x0000000004a3a0e8 (/usr/bin/clang+0x4a3a0e8) #1 0x0000000004a384c4 (/usr/bin/clang+0x4a384c4) #2 0x00000000049e99f4 (/usr/bin/clang+0x49e99f4) #3 0x0000000045e67034 (/lib/libthr.so.3+0x2b034) PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: Preprocessed source(s) and associated run script(s) are located at: clang: note: diagnostic msg: /tmp/token-f9bcce.c clang: note: diagnostic msg: /tmp/token-f9bcce.sh clang: note: diagnostic msg:
I can't attach both of these files cause of some ridiculous limitations.
First C code: https://gist.github.com/dmilith/2d64a90aca7b149002e6d8a1ad88f207 Sh code: https://gist.github.com/dmilith/419f27309b9ed60c0e95312de8365e17
To reproduce, build Python 3.9+ (also the same for Python 3.11, 3.12) with --enable-optimizations - which will enable the PGO on which Clang will fail.
Also, it's worth noting that this works on FreeBSD up to version 13.2 without any issues.
^Triage: assign to responsible group. Bugmeister comment: I'm sorry that you don't like the filesize restrictions, but they are necessary for us to be able to keep Bugzilla running.
I could download the repro .c and .sh files from GitHub, but apparently this also needs a "code.profclangd" file which has been produced by an earlier profiling run: + clang -cc1 -triple aarch64-unknown-freebsd14.0 -emit-llvm-bc '-flto=full' -flto-unit -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name token.c -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition '-mframe-pointer=non-leaf' '-ffp-contract=on' -fno-rounding-math -mconstructor-aliases '-funwind-tables=2' -target-cpu generic -target-feature +neon -target-feature +v8a -target-abi aapcs -mllvm -treat-scalable-fixed-error-as-warning '-debug-info-kind=standalone' '-dwarf-version=5' '-debugger-tuning=gdb' '-fprofile-instrument-use-path=code.profclangd' '-fcoverage-compilation-dir=/Software/Python312/.src_34340eab1ef9bee6/Python-3.12.1' -D NDEBUG -D Py_BUILD_CORE -O3 -Wformat -Wformat-security -Wformat -Wformat-security '-Werror=implicit-function-declaration' -w '-std=c11' '-fdebug-compilation-dir=/Software/Python312/.src_34340eab1ef9bee6/Python-3.12.1' -ferror-limit 1 '-fvisibility=hidden' -fwrapv -pthread -stack-protector 1 -stack-protector-buffer-size 4 -stack-protector-buffer-size 4 -fno-signed-char '-fgnuc-version=4.2.1' -vectorize-loops -vectorize-slp -faddrsig '-D__GCC_HAVE_DWARF2_CFI_ASM=1' -x c token-7986e1.c error: Error in reading profile code.profclangd: No such file or directory If you still have this file, you can also upload it GitHub?
I'm sorry. Didn't want to be rude. I noticed the generated file size later. Didn't expect a 1MiB C file :)
It's not exceptional for a preprocessed .c file to be more than 1MB. If you preprocess some C++ sources it gets even crazier. :) In any case, you can save quite a bit of space by xz'ing such files, as they compress very well.
(In reply to Dimitry Andric from comment #6) I did a clean build of Python312 with "--enable-optimizations". Here I've packed everything: https://software.verknowsys.com/fbsd-bug-reports/cleanbuild-Python-3.12.1-FreeBSD14.0-arm64.zip The archive contains all code* files generated in the Python build dir. All files generated by the clang during the crash are under "tmp_generated_by_the_build". There are 4 kinds of crashes from what I understand.
I can't reproduce any crashes here, at least on my 15-CURRENT machine. Since 15-CURRENT uses a newer version of clang, I have used an older binary which identifies itself as "FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)", corresponding to your version. With the unpacked contents of your zipfile, I can run all .sh files (I only replaced the clang command in there to point at clang 16.0.6) successfully, and they all produce .bc files as intended: $ ls -l *.bc -rw-r--r-- 1 dim dim 258948 2024-01-07 15:17:06 pegen-d7bb62.bc -rw-r--r-- 1 dim dim 52752 2024-01-07 15:17:07 pegen_errors-c153bb.bc -rw-r--r-- 1 dim dim 4244 2024-01-07 15:17:07 python-50eaf7.bc -rw-r--r-- 1 dim dim 14520 2024-01-07 15:17:07 token-7986e1.bc -rw-r--r-- 1 dim dim 14428 2024-01-07 15:17:07 token-9bf20d.bc However, as indicated, this is on an amd64 host with 15-CURRENT and ample RAM. I am suspecting that something else may be going on on your system? Do you see any out of memory errors or other indications like segfaults in dmesg or system logs? Can you also share the full error message shown when an instance of clang crashes? There should be some more information about what is exactly causing the 134 exit code, which usually indicates an assertion failure.
I'm quite certain about the VM software. I have used the same Qemu 6.1 +HVF version for several years now, as for all other build hosts. On the FreeBSD v13.2-aarch64 VM, the PGO works flawlessly. The full stdout/err output from the compilation is like this https://gist.github.com/dmilith/4ba2f5dbdc3026f1638f801f317834b1
Also, it's 100% reproducible. Even from the sh file: #16:11:12 vks5-14-0 /tmp λ cp /Software/Python312/.src_34340eab1ef9bee6/Python-3.12.1/code.profclangd ./ #16:11:37 vks5-14-0 /tmp λ sh ./pegen_errors-927930.sh Expected<T> must be checked before access or destruction. Expected<T> value was in success state. (Note: Expected<T> values in success mode must still be checked prior to being destroyed). PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script. Stack dump: 0. Program arguments: /usr/bin/clang -cc1 -triple aarch64-unknown-freebsd14.0 -emit-llvm-bc -flto=thin -flto-unit -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name pegen_errors.c -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +neon -target-feature +v8a -target-abi aapcs -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=standalone -dwarf-version=5 -debugger-tuning=gdb -fprofile-instrument-use-path=code.profclangd -fcoverage-compilation-dir=/Software/Python312/.src_34340eab1ef9bee6/Python-3.12.1 -D NDEBUG -D Py_BUILD_CORE -O3 -Wformat -Wformat-security -Wformat -Wformat-security -Werror=implicit-function-declaration -w -std=c11 -fdebug-compilation-dir=/Software/Python312/.src_34340eab1ef9bee6/Python-3.12.1 -ferror-limit 1 -fvisibility=hidden -ftrapv -fwrapv -pthread -stack-protector 1 -stack-protector-buffer-size 4 -stack-protector-buffer-size 4 -fno-signed-char -fgnuc-version=4.2.1 -vectorize-loops -vectorize-slp -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -x c pegen_errors-927930.c #0 0x0000000004a3a0e8 (/usr/bin/clang+0x4a3a0e8) #1 0x0000000004a384c4 (/usr/bin/clang+0x4a384c4) #2 0x0000000004a3a7c8 (/usr/bin/clang+0x4a3a7c8) #3 0x0000000045e67034 (/lib/libthr.so.3+0x2b034) zsh: exit 134 sh ./pegen_errors-927930.sh
(In reply to dmilith from comment #12) So on what host are you running this? A 14.0-RELEASE arm64 one? I vaguely remember this "Expected<T> must be checked before access or destruction" showing up in the past for some lldb users, but that was never solved as it was not reproducible... At least not for me.
Never mind, I can reproduce here on a 14.0-RELEASE-p3 arm64 VM, with the system clang which is 16.0.6. However, I also tried clang 17 from ports and that works fine with all the files. Since I am currently working on MFCing llvm 17 to stable/14 and stable/13, I'm unsure if it is worth the time to spend on figuring out which upstream commit fixed this. That said, for 14.0 release it might be fixed in an errata notice, but it is quite a lot of work, and this problem only occurs in a very specific use case (which can be worked around easily), and will be made moot anyway after 17 is MFCd...
Cool, it means the fix is going to land ~14.1, right? Thanks for the confirmation.
If you need to work around it now, you can do "pkg install llvm17" from ports, and use that one instead, to get a PGO'd Python. (I hope that runs faster than regular Python ... :)
I already did this by simply adding "--disable-optimizations" for Python39, Python311 and Python312 on my FreeBSD-14.0 build system. It works :) Thank you!
(In reply to Dimitry Andric from comment #16) Well, Python can be significantly faster with PGO indeed :) I wonder why "--enable-optimizations" is not enabled for Python port builds?
(In reply to dmilith from comment #12) Looks like this is an internal report not directly about the source code being compiled. See, for example: https://reviews.llvm.org/D138781 where another example of this type of message is referenced and, apparently, fixed for RISCV.
Well, it's worth to mention that the issue is: - Present on Python 3.9+, but also any PGO optimization of any software will crash the compiler on arm64/aarch64. - If port definitions would use "--enable-optimizations" configuration option - it would affect 100% of the users. Cheers!
(In reply to dmilith from comment #20) A quick workaround would be to disable the PGO option by default in the port, *iff* the architecture is arm64. I think there are more examples of that in the ports tree.
Well, the issue is still the case for 14.1-RELEASE. It crashes similarly for any software that uses PGO during the build on an aarch64/arm64 machine.
(In reply to dmilith from comment #22) FYI: Any PGO builds have failed on a recent 14.2 under aarch64. The regression has continued since 13.x
This doesn't seem to be limited to aarch64. I have exactly the same problem (crash on the very first .c file after "Rebuilding with profile guided optimizations", which happens to be python.c) for Python 3.13.1. FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9) Target: x86_64-unknown-freebsd14.1
(In reply to Guillaume Outters from comment #24) Can you specify the exact configure flags you used, and the exact make command(s)? I have tried here with v3.13.1 on 15-CURRENT with clang 19.1.5, and: CC=cc CXX=c++ /share/dim/src/python/cpython/configure --enable-optimizations gmake -j12 V=1 and it works just fine. I don't see any specific PGO options passed to the compiler invocations in the "Rebuilding with profile guided optimizations" part, though. So for some reason it might not always do this, even with the --enable-optimizations configure option?
In a PkgBase stable/14 based poudriere aarch64 jail, I tried building lang/python310 based on: # git -C /usr/ports diff lang/python310 diff --git a/lang/python310/Makefile b/lang/python310/Makefile index b352e8a2f9d7..d19406a47897 100644 --- a/lang/python310/Makefile +++ b/lang/python310/Makefile @@ -27,8 +27,8 @@ SHEBANG_FILES+= Lib/test/ziptestdata/exe_with_z64 \ DISABLED_EXTENSIONS= _sqlite3 _tkinter _gdbm CONFIGURE_ARGS+= --enable-shared --without-ensurepip \ - --with-system-ffi -CONFIGURE_ENV+= OPT="" # Null out OPT to respect user CFLAGS and remove optimizations + --with-system-ffi --enable-optimizations +#CONFIGURE_ENV+= OPT="" # Null out OPT to respect user CFLAGS and remove optimizations INSTALL_TARGET= altinstall # Don't want cloberring of unprefixed files @@ -45,7 +45,7 @@ PLIST_SUB= ABI=${ABIFLAGS} \ OSMAJOR=${OSVERSION:C/([0-9]*)[0-9]{5}/\1/} # For plat-freebsd* in pkg-plist. https://bugs.python.org/issue19554 OPTIONS_DEFINE= DEBUG IPV6 LIBMPDEC LTO NLS PYMALLOC -OPTIONS_DEFAULT= LIBMPDEC PYMALLOC +OPTIONS_DEFAULT= LIBMPDEC PYMALLOC LTO OPTIONS_EXCLUDE_riscv64= LTO OPTIONS_RADIO= HASH OPTIONS_RADIO_HASH= FNV SIPHASH poudriere-devel got the result: [00:05:35] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:10:49] [01] [00:05:14] Finished lang/python310 | python310-3.10.16: Success The log file has the line: Rebuilding with profile guided optimizations: Also reported was: --CONFIGURE_ARGS-- --enable-shared --without-ensurepip --with-system-ffi --enable-optimizations --without-pydebug --enable-ipv6 --with-system-libmpdec --with-lto --with-pymalloc --prefix=/usr/local ${_LATE_CONFIGURE_ARGS} --End CONFIGURE_ARGS-- and the cc commands show use of: -flto I did similarly under main [so: 15 as stands] and got the same sort of results. I've tried without commenting out the CONFIGURE_ENV+= as well. Similarly without LTO being enabled. It would seem that more context than use of --enable-optimizations is at issue for stable/14 and main these days. What changes to lang/python310/Makefile are sufficient to lead to the poudriere(-devel) based builds showing the issue? (Not necessarily for stable/14 and main. But I'd likely try such changes under stable/14 or under main or both, just for curiosity.) For reference . . . Note: Windows DevKit 2023, 8 aarch64 FreeBSD cpus, 32 GiBytes of RAM, swap partition not active. Also that jail had no packages previously built at the start and: [00:00:03] Building 11 packages using up to 8 builders In use was (or, if indicted, was not): ALLOW_MAKE_JOBS=yes No use of MAKE_JOBS_NUMBER_LIMIT or the like. USE_TMPFS=all python310 not listed in TMPFS_BLACKLIST . UFS context, not ZFS.
(In reply to Mark Millard from comment #26) I've now also tried under a PkgBase 14.2-RELEASE-p2 based poudriere aarch64 jail: [00:05:32] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:11:08] [01] [00:05:36] Finished lang/python310 | python310-3.10.16: Success Again: no problem observed with both lto and --enable-optimizations in use (including PGO activity). It seems that there is more to the context that contributes to the failing cases.
(In reply to Mark Millard from comment #27) I've now also tried under a PkgBase 14.1-RELEASE-p6 based poudriere aarch64 jail: [00:05:29] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:11:27] [01] [00:05:58] Finished lang/python310 | python310-3.10.16: Success Again: no problem observed with both lto and --enable-optimizations in use (including PGO activity). It seems that there is more to the context that contributes to the failing cases. FYI: when I watch with top I see examples of: ./python -m test --pgo --timeout=1200{python}
(In reply to Mark Millard from comment #27) Sorry for my silly reference to -p2 for 14.2-RELEASE . In reality: # poudriere jail -l | grep -E '^JAILNAME|-aarch64' JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP PATH R141p6-aarch64 14.1-RELEASE-p6 aarch64 pkgbase 2024-12-29 08:13:01 /usr/local/poudriere/jails/R141p6-aarch64 release-aarch64 14.2-RELEASE aarch64 pkgbase 2024-12-28 23:40:41 /usr/local/poudriere/jails/release-aarch64 official-aarch64 14.2-STABLE aarch64 pkgbase 2024-12-28 22:35:51 /usr/local/poudriere/jails/official-aarch64 main-aarch64 15.0-CURRENT aarch64 pkgbase 2024-12-07 22:13:14 /usr/local/poudriere/jails/main-aarch64-poud
In reply to @Dimitri Andric about comment #25: After having cleaned up my env, I could bissect the crash to a single: export CC=clang before running the configure. Probably the configure chooses its profiling tool depending on the compiler's declared name (at least before it inspects the default cc to determine it's a clang). I start from the original unmodified Python (not the ports patched one), at http://www.python.org/ftp/python/3.13.1/Python-3.13.1.tar.xz on a stock FreeBSD 14.1-RELEASE with a minimalistic configure: ./configure --enable-optimizations
(In reply to Guillaume Outters from comment #30) I tried the following with the final WANRINGs at the end shown below: # tar -xpf Python-3.13.1.tar.xz -C ./ # cd Python-3.13.1 # ./configure --enable-optimizations . . . configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly. configure: WARNING: Platform "aarch64-unknown-freebsd15.0" with compiler "clang" is not supported by the CPython core team, see https://peps.python.org/pep-0011/ for more information. I then did the same in another directory tree, using export CC=clang before the "./configure --enable-optimizations". I also captured and diff'd the outputs of the 2 runs, other than "cc" vs. "clang" text, they matched. # diff -rq Python-3.13.1-noCC/ Python-3.13.1-CC_clang/ | more Files Python-3.13.1-noCC/Makefile and Python-3.13.1-CC_clang/Makefile differ Files Python-3.13.1-noCC/Makefile.pre and Python-3.13.1-CC_clang/Makefile.pre differ Files Python-3.13.1-noCC/config.log and Python-3.13.1-CC_clang/config.log differ Files Python-3.13.1-noCC/config.status and Python-3.13.1-CC_clang/config.status differ There are interesting Makefile differences and Makefile.pre differences, both, in part, tied to PGO_* and LLVM_PROF_* definition differences: # diff -r Python-3.13.1-noCC/ Python-3.13.1-CC_clang/ | less diff -r Python-3.13.1-noCC/Makefile Python-3.13.1-CC_clang/Makefile 38,39c38,39 < CC= cc -pthread < CXX= c++ -pthread --- > CC= clang -pthread > CXX= clang++ -pthread 51,54c51,54 < PGO_PROF_GEN_FLAG= < PGO_PROF_USE_FLAG= < LLVM_PROF_MERGER= < LLVM_PROF_FILE= --- > PGO_PROF_GEN_FLAG=-fprofile-instr-generate > PGO_PROF_USE_FLAG=-fprofile-instr-use="$(shell pwd)/code.profclangd" > LLVM_PROF_MERGER= /usr/bin/llvm-profdata merge -output="$(shell pwd)/code.profclangd" "$(shell pwd)"/*.profclangr > LLVM_PROF_FILE=LLVM_PROFILE_FILE="$(shell pwd)/code-%p.profclangr" 377c377 < CONFIG_ARGS= '--enable-optimizations' --- > CONFIG_ARGS= '--enable-optimizations' 'CC=clang' diff -r Python-3.13.1-noCC/Makefile.pre Python-3.13.1-CC_clang/Makefile.pre 37,38c37,38 < CC= cc -pthread < CXX= c++ -pthread --- > CC= clang -pthread > CXX= clang++ -pthread 50,53c50,53 < PGO_PROF_GEN_FLAG= < PGO_PROF_USE_FLAG= < LLVM_PROF_MERGER= < LLVM_PROF_FILE= --- > PGO_PROF_GEN_FLAG=-fprofile-instr-generate > PGO_PROF_USE_FLAG=-fprofile-instr-use="$(shell pwd)/code.profclangd" > LLVM_PROF_MERGER= /usr/bin/llvm-profdata merge -output="$(shell pwd)/code.profclangd" "$(shell pwd)"/*.profclangr > LLVM_PROF_FILE=LLVM_PROFILE_FILE="$(shell pwd)/code-%p.profclangr" 376c376 < CONFIG_ARGS= '--enable-optimizations' --- > CONFIG_ARGS= '--enable-optimizations' 'CC=clang' This much does not seem to be a FreeBSD issue.
That was enough context for me to replicate getting a failure via a 14.1-RELEASE-p6 poudriere jail based build based on, for example, just adding a CC+clang to CONFIGURE_ENV : diff --git a/lang/python310/Makefile b/lang/python310/Makefile index b352e8a2f9d7..ece416cb52dc 100644 --- a/lang/python310/Makefile +++ b/lang/python310/Makefile @@ -27,8 +27,9 @@ SHEBANG_FILES+= Lib/test/ziptestdata/exe_with_z64 \ DISABLED_EXTENSIONS= _sqlite3 _tkinter _gdbm CONFIGURE_ARGS+= --enable-shared --without-ensurepip \ - --with-system-ffi -CONFIGURE_ENV+= OPT="" # Null out OPT to respect user CFLAGS and remove optimizations + --with-system-ffi --enable-optimizations +CONFIGURE_ENV+= CC=clang +#CONFIGURE_ENV+= OPT="" # Null out OPT to respect user CFLAGS and remove optimizations INSTALL_TARGET= altinstall # Don't want cloberring of unprefixed files @@ -45,7 +46,7 @@ PLIST_SUB= ABI=${ABIFLAGS} \ OSMAJOR=${OSVERSION:C/([0-9]*)[0-9]{5}/\1/} # For plat-freebsd* in pkg-plist. https://bugs.python.org/issue19554 OPTIONS_DEFINE= DEBUG IPV6 LIBMPDEC LTO NLS PYMALLOC -OPTIONS_DEFAULT= LIBMPDEC PYMALLOC +OPTIONS_DEFAULT= LIBMPDEC PYMALLOC LTO OPTIONS_EXCLUDE_riscv64= LTO OPTIONS_RADIO= HASH OPTIONS_RADIO_HASH= FNV SIPHASH It got: [00:00:05] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:04:35] [01] [00:04:30] Saving lang/python310 | python310-3.10.16 wrkdir [00:05:06] [01] [00:05:01] Saved lang/python310 | python310-3.10.16 wrkdir to: /usr/local/poudriere/data/wrkdirs/R141p6-aarch64-default/default/python310-3.10.16.tbz [00:05:06] [01] [00:05:01] Finished lang/python310 | python310-3.10.16: Failed: build I'll check the other test jails as well.
stable/14 and main build the test case fine but 14.2-RELEASE does not. It appears that LLVM19's clang has the issue fixed. Some details . . . 14.2-RELEASE gets: [00:00:07] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:04:14] [01] [00:04:07] Finished lang/python310 | python310-3.10.16: Failed: build stable/14 (some 1402501 vintage) gets: [00:00:06] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:07:09] [01] [00:07:03] Finished lang/python310 | python310-3.10.16: Success main [so: 15 as stands] (some 1500029 vintage) gets: [00:00:19] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:08:33] [01] [00:08:14] Finished lang/python310 | python310-3.10.16: Success
(In reply to Mark Millard from comment #33) By the way: Do not take my Makefile diff as appropriate for committal. For example, the OPTIONS_DEFAULT ignores riscv64 apparently needing to avoid LTO. I just was not building for riscv64. Doing Makefile diff's was just a easy way to configure and show a type of experiment.
Very interesting. My build process uses -O3 with LTO enabled, so maybe that's why it still fails on my aarch64 build hosts. I will double-check that later.
(In reply to Guillaume Outters from comment #30) Yeah, I've done that too now, and CC=clang CXX=clang++ makes it at least generate and use profiling information, e.g.: clang -pthread -c -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-instr-use=/home/dim/obj/cpython-3.13.1/code.profclangd -I/share/dim/src/python/cpython/Include/internal -I/share/dim/src/python/cpython/Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I/share/dim/src/python/cpython/Include -DPy_BUILD_CORE -o Programs/python.o /share/dim/src/python/cpython/Programs/python.c But I see no crashes at all, it builds just fine to the end. Currently running the test suite. I'm not sure if LTO is enabled in this case, though.
(In reply to dmilith from comment #35) Looking in the build log of the successful stable/14 build, for example: clang -pthread -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -flto -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-instr-generate -I./Include/internal -I. -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Programs/python.o ./Programs/python.c So both -O3 and -O2 were listed, in that order. For reference: --MAKE_ENV-- OPENSSLBASE=/usr OPENSSLDIR=/etc/ssl OPENSSLINC=/usr/include OPENSSLLIB=/usr/lib XDG_DATA_HOME=/wrkdirs/usr/ports/lang/python310/work XDG_CONFIG_HOME=/wrkdirs/usr/ports/lang/python310/work XDG_CACHE_HOME=/wrkdirs/usr/ports/lang/python310/work/.cache HOME=/wrkdirs/usr/ports/lang/python310/work TMPDIR="/tmp" PATH=/wrkdirs/usr/ports/lang/python310/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin PKG_CONFIG_LIBDIR=/wrkdirs/usr/ports/lang/python310/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local LOCALBASE=/usr/local CC="cc" CFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " CPP="cpp" CPPFLAGS="-I/usr/local/include -I/usr/local/include" LDFLAGS=" -L/usr/local/lib -fstack-protector-strong " LIBS="-L/usr/local/lib -lintl" CXX="c++" CXXFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " BSD_INSTALL_PROGRAM="install -s -m 555" BSD_INSTALL_LIB="install -s -m 0644" BSD_INSTALL_SCRIPT="install -m 555" BSD_INSTALL_DATA="install -m 0644" BSD_INSTALL_MAN="install -m 444" --End MAKE_ENV-- which has both: CFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " CXXFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing "
(In reply to Dimitry Andric from comment #36) FYI: # grep "clang[+][+]" /usr/local/poudriere/data/logs/bulk/official-aarch64-default/2024-12-30_17h08m08s/logs/python310-3.10.16.log # grep "c[+][+]" /usr/local/poudriere/data/logs/bulk/official-aarch64-default/2024-12-30_17h08m08s/logs/python310-3.10.16.log OPENSSLBASE=/usr OPENSSLDIR=/etc/ssl OPENSSLINC=/usr/include OPENSSLLIB=/usr/lib XDG_DATA_HOME=/wrkdirs/usr/ports/lang/python310/work XDG_CONFIG_HOME=/wrkdirs/usr/ports/lang/python310/work XDG_CACHE_HOME=/wrkdirs/usr/ports/lang/python310/work/.cache HOME=/wrkdirs/usr/ports/lang/python310/work TMPDIR="/tmp" PATH=/wrkdirs/usr/ports/lang/python310/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin PKG_CONFIG_LIBDIR=/wrkdirs/usr/ports/lang/python310/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local LOCALBASE=/usr/local CC="cc" CFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " CPP="cpp" CPPFLAGS="-I/usr/local/include -I/usr/local/include" LDFLAGS=" -L/usr/local/lib -fstack-protector-strong " LIBS="-L/usr/local/lib -lintl" CXX="c++" CXXFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " BSD_INSTALL_PROGRAM="install -s -m 555" BSD_INSTALL_LIB="install -s -m 0644" BSD_INSTALL_SCRIPT="install -m 555" BSD_INSTALL_DATA="install -m 0644" BSD_INSTALL_MAN="install -m 444" checking whether c++ also accepts flags for thread support... yes Looks to me like there is no actual use of C++ in the build.
The stable/14 test context got: [00:00:07] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:07:55] [01] [00:07:48] Finished lang/python310 | python310-3.10.16: Success for: # cat ~/mmjnk.txt diff --git a/lang/python310/Makefile b/lang/python310/Makefile index b352e8a2f9d7..f09113380dcc 100644 --- a/lang/python310/Makefile +++ b/lang/python310/Makefile @@ -27,8 +27,10 @@ SHEBANG_FILES+= Lib/test/ziptestdata/exe_with_z64 \ DISABLED_EXTENSIONS= _sqlite3 _tkinter _gdbm CONFIGURE_ARGS+= --enable-shared --without-ensurepip \ - --with-system-ffi -CONFIGURE_ENV+= OPT="" # Null out OPT to respect user CFLAGS and remove optimizations + --with-system-ffi --enable-optimizations +CONFIGURE_ENV+= CC=clang +#CONFIGURE_ENV+= OPT="" # Null out OPT to respect user CFLAGS and remove optimizations +CFLAGS+= -O3 INSTALL_TARGET= altinstall # Don't want cloberring of unprefixed files @@ -45,7 +47,7 @@ PLIST_SUB= ABI=${ABIFLAGS} \ OSMAJOR=${OSVERSION:C/([0-9]*)[0-9]{5}/\1/} # For plat-freebsd* in pkg-plist. https://bugs.python.org/issue19554 OPTIONS_DEFINE= DEBUG IPV6 LIBMPDEC LTO NLS PYMALLOC -OPTIONS_DEFAULT= LIBMPDEC PYMALLOC +OPTIONS_DEFAULT= LIBMPDEC PYMALLOC LTO OPTIONS_EXCLUDE_riscv64= LTO OPTIONS_RADIO= HASH OPTIONS_RADIO_HASH= FNV SIPHASH that had commands like: clang -pthread -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -O2 -pipe -O3 -fstack-protector-strong -fno-strict-aliasing -flto -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-instr-generate -I./Include/internal -I. -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Programs/python.o ./Programs/python.c So, the order: -O3 -O2 -O3 It does not appear like -O3 leads to any crashes for LLVM19's clang.
I'll note that for that stable/14 test: # grep "no profile data" /usr/local/poudriere/data/logs/bulk/official-aarch64-default/2024-12-30_17h08m08s/logs/python310-3.10.16.log warning: no profile data available for file "_zoneinfo.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_lsprof.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "rotatingtree.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "grpmodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "syslogmodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "mmapmodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_xxsubinterpretersmodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "audioop.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_csv.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_testclinic.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_testimportmultiple.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_testmultiphase.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_xxtestfuzz.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "fuzzer.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_curses_panel.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_cryptmodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_cursesmodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "termios.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "ossaudiodev.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "nismodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_ctypes_test.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "_uuidmodule.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "posixshmem.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "xxlimited.c" [-Wprofile-instr-unprofiled] warning: no profile data available for file "xxlimited_35.c" [-Wprofile-instr-unprofiled]
(In reply to Mark Millard from comment #39) For the -O3 test with lto and --enable-optimizations use on aarch64, main [so: 15 as stands] got: [00:00:09] [01] [00:00:00] Building lang/python310 | python310-3.10.16 [00:09:31] [01] [00:09:22] Finished lang/python310 | python310-3.10.16: Success as well.
(In reply to Mark Millard from comment #38) > Looks to me like there is no actual use of C++ in the build. It's just a habit of passing CC and CXX. :) (In reply to Mark Millard from comment #40) > warning: no profile data available for file "_zoneinfo.c" [-Wprofile-instr-unprofiled] These can be safely ignored. It looks like they are explicitly excluded from profiling. Upstream will probably have their reasons. In any case, I also tried a build on 15-CURRENT main-n273771-e8263ace39c8 aarch64, and with both --enable-optimizations and --with-lto, but it all worked fine.
(In reply to Dimitry Andric from comment #42) The presence of the checking for "no profile data available for file", and the limited number of files reported for lack of coverage, suggests use of profile data for the rest of the files. In other words: my build was probably valid relative to PGO.
After some more investigation I found the cause, which is (again) the fact that assertions are _disabled_ on stable and release branches. This particular "error" is occurring in contrib/llvm-project/clang/lib/CodeGen/CodeGenModule.cpp, on line 422: 414 if (CodeGenOpts.hasProfileClangUse()) { 415 auto ReaderOrErr = llvm::IndexedInstrProfReader::create( 416 CodeGenOpts.ProfileInstrumentUsePath, *FS, 417 CodeGenOpts.ProfileRemappingFile); 418 // We're checking for profile read errors in CompilerInvocation, so if 419 // there was an error it should've already been caught. If it hasn't been 420 // somehow, trip an assertion. 421 assert(ReaderOrErr); 422 PGOReader = std::move(ReaderOrErr.get()); 423 } llvm::IndexedInstrProfReader::create() returns a llvm::Expected<> instance, which is similar to std::expected: it can contain either a "good" return value, or an error. In some llvm build configurations, when you attempt to use the return value without checking whether it contains an error, you get "Expected<T> must be checked before access or destruction". This occurs even if the Expected<> object contains a valid value, in which case you get "Expected<T> value was in success state". When llvm is built with assertions (WITH_LLVM_ASSERTIONS, which is on by default on -CURRENT, but not on stable or release branches), the assert() statement on line 421 is calling llvm::Expected<>::operator bool(), which checks whether the contained object is valid, and if so it resets the checked state: /// Bool conversion. Returns true if this Error is in a failure state, /// and false if it is in an accept state. If the error is in a Success state /// it will be considered checked. explicit operator bool() { setChecked(getPtr() == nullptr); return getPtr() != nullptr; } Then in line 422 the contained object is moved out of the 'ReaderOrErr' variable, into the 'PGOReader' variable. Finally, the 'ReaderOrErr' variable, which is now empty, is destroyed. However, when llvm is built without assertions (WITHOUT_LLVM_ASSERTIONS), the assert() statement on line 421 does nothing, and the 'ReaderOrErr' variable is unaffected. Then in line 422, llvm::Expected<>::get() is called: /// Returns a reference to the stored T value. reference get() { assertIsChecked(); return *getStorage(); } The first thing get() does is looking if the Expected<> object was checked, and if not, it prints "Expected<T> must be checked before access or destruction" and the whole program dies. Finally, assertIsChecked() is only really doing something if the global llvm macro LLVM_ENABLE_ABI_BREAKING_CHECKS is defined: void assertIsChecked() const { #if LLVM_ENABLE_ABI_BREAKING_CHECKS if (LLVM_UNLIKELY(Unchecked)) fatalUncheckedExpected(); #endif } In the upstream build system, LLVM_ENABLE_ABI_BREAKING_CHECKS is typically 0 when assertions are off, and 1 when assertions are on. In the FreeBSD case, this use to always be defined as 1 in lib/clang/include/llvm/Config/abi-breaking.h, until base 1c83996beda7 ("Adjust LLVM_ENABLE_ABI_BREAKING_CHECKS depending on NDEBUG"): 15 /* Define to enable checks that alter the LLVM C++ ABI */ 16 #ifdef NDEBUG 17 #define LLVM_ENABLE_ABI_BREAKING_CHECKS 0 18 #else 19 #define LLVM_ENABLE_ABI_BREAKING_CHECKS 1 20 #endif Unfortunately 14.2 does not have this commit, so the combination of MK_LLVM_ASSERTIONS=no and LLVM_ENABLE_ABI_BREAKING_CHECKS=1 leads to the scenario describe above: the assert() statement in CodeGenModule.cpp line 421 does nothing, but ReaderOrErr.get() still performs the check and aborts. I merged base 1c83996beda7 to stable/14 in base 86de9cd1f1b5, and to stable/13 in base 44be5a00bedd, but this is unfortunately not available in 14.2-RELEASE or 13.4-RELEASE, so for those releases there isn't much I can do except to try to roll it into a Erratum.
(In reply to Dimitry Andric from comment #44) I'm happy to report that PGO works again in the FreeBSD 14.3-RELEASE :)