Created attachment 184965 [details] poudriere log $ poudriere jail -cj 111aarch64 -x -a arm64.aarch64 -v 11.1-RELEASE $ poudriere bulk -Ctj 111aarch64 www/firefox # fails in lang/rust $ poudriere jail -sj 111aarch64 <configure network in the jail> $ env -i TERM=$TERM /usr/sbin/jexec 111aarch64-default /bin/sh $ export ABI=FreeBSD:11:aarch64 UNAME_r=11.1-RELEASE $ pkg install -qy rust $ echo 'fn main() {}' >a.rs $ rustc a.rs Fatal error 'exception should be rethrown' at line 129 in file /usr/src/lib/libthr/thread/thr_exit.c (errno = 2) qemu: uncaught target signal 6 (Abort trap) - core dumped Abort trap
Just for my sanity, does this run on real hardware?
Yep, comment 0 test works fine on ref11-aarch64 (base r319064 atm) and ref12-aarch64. lang/rust uses prebuilt version of rustc during build, and at least 1.18.0 built fine: http://thunderx1.nyi.freebsd.org/data/latest-per-pkg/rust/1.18.0/
rust-1.20.0 seems to build/run fine if one restarts cargo/rustc every time it randomly hangs or crashes. I've even managed to build unpatched Firefox 58 this babysitting way. $ file rustc-*-aarch64-unknown-freebsd/rustc/bin/rustc rustc-1.17.0-aarch64-unknown-freebsd/rustc/bin/rustc: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100122), FreeBSD-style, not stripped rustc-1.18.0-aarch64-unknown-freebsd/rustc/bin/rustc: ELF 64-bit LSB shared object, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100513), FreeBSD-style, not stripped rustc-1.19.0-aarch64-unknown-freebsd/rustc/bin/rustc: ELF 64-bit LSB shared object, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100513), FreeBSD-style, not stripped rustc-1.20.0-aarch64-unknown-freebsd/rustc/bin/rustc: ELF 64-bit LSB shared object, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100513), FreeBSD-style, not stripped $ rustc-1.17.0-aarch64-unknown-freebsd/rustc/bin/rustc -vV rustc 1.17.0 (56124baa9 2017-04-24) binary: rustc commit-hash: 56124baa9e73f28c0709e59e74783cf234a978cf commit-date: 2017-04-24 host: aarch64-unknown-freebsd release: 1.17.0 LLVM version: 3.9 Fatal error 'exception should be rethrown' at line 128 in file /poudriere/jails/head-aarch64/usr/src/lib/libthr/thread/thr_exit.c (errno = 0) Segmentation fault $ rustc-1.18.0-aarch64-unknown-freebsd/rustc/bin/rustc -vV rustc 1.18.0 (03fc9d622 2017-06-06) binary: rustc commit-hash: 03fc9d622e0ea26a3d37f5ab030737fcca6928b9 commit-date: 2017-06-06 host: aarch64-unknown-freebsd release: 1.18.0 LLVM version: 3.9 Fatal error 'exception should be rethrown' at line 128 in file /poudriere/jails/head-aarch64/usr/src/lib/libthr/thread/thr_exit.c (errno = 0) Segmentation fault $ rustc-1.19.0-aarch64-unknown-freebsd/rustc/bin/rustc -vV rustc 1.19.0 (0ade33941 2017-07-17) binary: rustc commit-hash: 0ade339411587887bf01bcfa2e9ae4414c8900d4 commit-date: 2017-07-17 host: aarch64-unknown-freebsd release: 1.19.0 LLVM version: 4.0 Fatal error 'exception should be rethrown' at line 128 in file /poudriere/jails/head-aarch64/usr/src/lib/libthr/thread/thr_exit.c (errno = 0) Segmentation fault $ rustc-1.20.0-aarch64-unknown-freebsd/rustc/bin/rustc -vV rustc 1.20.0 (f3d6973f4 2017-08-27) binary: rustc commit-hash: f3d6973f41a7d1fb83029c9c0ceaf0f5d4fd7208 commit-date: 2017-08-27 host: aarch64-unknown-freebsd release: 1.20.0 LLVM version: 4.0 $ make -C lang/rust [...] Compiling num-traits v0.1.40 Running `/wrkdirs/usr/ports/lang/rust/work/rustc-1.21.0-src/build/aarch64-unknown-freebsd/stage0/bin/rustc --crate-name num_traits src/vendor/num-traits/src/lib.rs --crate-type lib --emit=dep-info,link -C debug-assertions=off -C overflow-checks=on -C metadata=70b5f35f2a57f99e -C extra-filename=-70b5f35f2a57f99e --out-dir /wrkdirs/usr/ports/lang/rust/work/rustc-1.21.0-src/build/bootstrap/debug/deps -L dependency=/wrkdirs/usr/ports/lang/rust/work/rustc-1.21.0-src/build/bootstrap/debug/deps --cap-lints allow` qemu:handle_cpu_signal received signal outside vCPU context @ pc=0x6022324e load: 1.40 cmd: qemu-aarch64-static 23867 [uwait] 109.59r 5.71u 0.03s 0% 254640k
Created attachment 187465 [details] poudriere log (rust-1.21.0)
Looks like rust-1.20.0 only worked by luck. $ pkg install rust $ rustc -vV rustc 1.21.0 binary: rustc commit-hash: unknown commit-date: unknown host: aarch64-unknown-freebsd release: 1.21.0 LLVM version: 4.0 Fatal error 'exception should be rethrown' at line 128 in file /poudriere/jails/head-aarch64/usr/src/lib/libthr/thread/thr_exit.c (errno = 0) Abort trap
This is still a problem and actual since enabling building of lang/rust on armv6/armv7/aarch64 in ports r493268. The arvm7+QEMU build of lang/rust failed after taking > 21600 seconds to compile the cfg-if crate [1]. aarch64+QEMU isn't any better and hangs eventually too. [1] http://beefy16.nyi.freebsd.org/data/head-armv7-default/p493345_s344269/logs/rust-1.32.0.log
Created attachment 202230 [details] patch It's only a problem on armv6/7, we use real hw for building pkg on aarch64 and ppc64. The attached patch prevents building rust with qemu-user and try to clarify the situation on aarch64
(In reply to mikael.urankar from comment #7) +.if ${ARCH} == aarch64 +WARNING+= "Due to a bug in rtld, rust fails to run on 11.2-RELEASE and 12.0-RELEASE \ + you can either update /libexec/ld-elf.so.1 from -STABLE or -CURRENT \ + or run -STABLE or -CURRENT" +.endif Which revisions do users need to upgrade to? We might as well set IGNORE for older OSVERSIONs on aarch64. +.ifdef QEMU_EMULATING +IGNORE= qemu-user-static isn't able to build lang/rust, but it builds fine on a real hardware +.endif Ok.
(In reply to Tobias Kortkamp from comment #8) for 12-stable: base r342847 for -current: base r342113 no mfc on 11-stable afaict
A commit references this bug: Author: tobik Date: Thu Feb 21 19:03:17 UTC 2019 New revision: 493523 URL: https://svnweb.freebsd.org/changeset/ports/493523 Log: lang/rust: Ignore with qemu-user-static and on aarch64 without fixed ld-elf.so.1 - Rust will not run without a fixed ld-elf.so.1 on aarch64 - Builds with qemu-user-static currently hang after a while PR: 221185 Submitted by: Mika?l Urankar <mikael.urankar@gmail.com> Changes: head/lang/rust/Makefile
Could this issue be closed? It looks like some patches were committed...
(In reply to Mateusz Piotrowski from comment #11) qemu-user-static can't build lang/rust, it's still an issue.
This might be fixed, but I've got a local patch that will almost certainly solve this issue -- _umtx_op handling has been the bane of my existence, and I'm looking to offload it entirely to the kernel going forward. My local patch allows this by introducing a 32bit flag to _umtx_op commands which qemu-bsd-user can then set to indicate to the kernel that it's looking at 32-bit structures. The new logic is used for any target where the endianness matches the host's, so 64-bit amd64 can natively handle armv7 or aarch64 while it's forced to fallback to the still-slightly-buggy emulation of _umtx_op for, e.g., mips.
*** Bug 254694 has been marked as a duplicate of this bug. ***
Hello, rust-1.56.0 won't build in a qemu-poudriere for arm64 on amd64 on recent stable/13 with todays portstree. Error is "fails to build with qemu-user-static". For clarity: 1. is this the same bug? 2. If it is, is this bug a "won'tfix" thanks
Hi, I also ran into this bug under 13.0/amd64 and with rust 1.57.0, trying to build it for armv6. It would be good to know if this can be fixed and what is missing. Thank you for an answer.
(In reply to Kyle Evans from comment #13) Kyle, is any progress here?
If I'm correct, this issue is blocking py-cryptography from upgrading to an OpenSSL 3 compatible version because newer versions of py-cryptography require rust: bug 254853.
Even if this is fixed running rustc under qemu-user-static will be extremely slow, especially with ports 967022fd812c or lto=true in Cargo.toml. Converting lang/rust into a cross-compiler and hooking into poudriere a la native-xtools may be more practical.
(In reply to Enji Cooper from comment #19) This is not a blocker for py-cryptography. We build rust natively on all tier 1 arches.
(In reply to Mikael Urankar from comment #21) But, for example, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254853#c33 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254853#c49 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254853#c80 all indicate that this is effectively a blocker based on how things are actually being handled. That submittal lists this submittal as something it depends on. That official builds work on the ampere* aarch64 based builders (or analogous) without qemu use is not the criteria actually being used.
This bug report seems over specific when I look at the official port->package builds done via qemu. These days that is only for armv6 but it used to be for armv7 as well. http://beefy8.nyi.freebsd.org/build.html?mastername=main-armv6-default&build=p72871fad39eb_sd1d7a27370 failed to build 275 ports, leading to 22091 skipped ports. Of those 275 failures, 176 are "core dump" failures. (Rust was not part of the queued ports.) (I used to try to use qemu based port builds and I've occasionally checked what the official qemu results use looked like for armv[67] since then. Its armv* targeting reliability never seemed good. The efforts to make it work well for 32-bit on 64-bit never seem to have caught up with the problems over the years, as evidenced by the modern armv6 results. It still looks like a huge effort to make it generally work reliably for building ports into packages.)
(In reply to Mark Millard from comment #23) Looking around to check on selective memory (plus irregular sampling): =>> Building ports-mgmt/pkg build started at Sun Dec 11 03:55:47 UTC 2022 port directory: /usr/ports/ports-mgmt/pkg package name: pkg-1.18.4 building for: FreeBSD main-armv6-default-job-01 14.0-CURRENT FreeBSD 14.0-CURRENT 1400074 arm maintained by: pkg@FreeBSD.org Makefile ident: Poudriere version: 3.2.8-23-ga7f8d188 Host OSVERSION: 1400073 Jail OSVERSION: 1400074 looks to have built 23646 ports into packages, of 33314 queued, failed 208, and skipped 8725. Only 8 of the failures are listed as "coredump". This seems to be the most recent of the among-the-best results for armv6 when 30000+ ports were queued. So things got much worse this year compared to the end of last year. Even the one started on 22 Dec 2022 skipped 18411 ports.
Are there any bugs for this issue 'upstream'? I can't see a reference to it in https://wiki.freebsd.org/QemuUserModeToDo or https://github.com/seanbruno/qemu-bsd-user/issues
(In reply to Nathan Reilly from comment #25) We shifted the project over to here: https://github.com/qemu-bsd-user/qemu-bsd-user/issues but I don't think we have anything about this specifically, no.