Bug 216143
| Summary: | lang/rust, devel/cargo: generate bootstraps for Tier2 architectures via cross-compilation | ||
|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Jan Beich <jbeich> |
| Component: | Individual Port(s) | Assignee: | Jean-Sébastien Pédron <dumbbell> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | CC: | ahmedsayeed1982, dumbbell, gecko, lantw44, linimon, mikael, mmel, riggs, tablosazi.farahan |
| Priority: | --- | Keywords: | dogfood, needs-patch |
| Version: | Latest | Flags: | dumbbell:
maintainer-feedback+
|
| Hardware: | Any | ||
| OS: | Any | ||
| Bug Depends on: | |||
| Bug Blocks: | 225279 | ||
|
Description
Jan Beich
2017-01-16 15:04:45 UTC
Hi! I started to look into this. == Updating lang/rust-nightly == I'm updating lang/rust-nightly to a recent snapshot and add support for i386 at the same time. This is not directly related to this issue but will help to make sure we don't have regression. I'm currently testing the port in Poudriere for HEAD/amd64 and 11.0/i386. == Cargo on i386 == I'm updating devel/cargo to 0.15.0 and add i386 support at the same time. There cargo prebuilt binaries available for FreeBSD/i386 since last October, I missed that. I'm currently testing the port in Poudriere for HEAD/amd64. Unfortunately, I can't test 11.0/i386 because cargo(1) for i386 segfaults on amd64. The executable runs successfully in ref11-i386. I will try to build the port there. == Rust and Cargo on !x86 == Finally, I started to look for guides to bootstrap Rust and Cargo on unsupported hosts. I'm going to take aarch64 as the first victim. I found the following article from someone who did several ports of Rust: https://www.reddit.com/r/rust/comments/5ag60z/how_do_i_bootstrap_rust_to_crosscompile_for_a_new/ The following article might be useful too; I'm not sure yet: http://os.phil-opp.com/cross-compile-libcore.html I will keep you posted. The patch to update Cargo to 0.15.0 and enable it on i386 is up for review: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216143 Sorry, wrong copy/paste... The review is this one: https://reviews.freebsd.org/D9273 A commit references this bug: Author: dumbbell Date: Sun Jan 22 10:45:30 UTC 2017 New revision: 432111 URL: https://svnweb.freebsd.org/changeset/ports/432111 Log: devel/cargo: Update to 0.15.0, enable on i386 The new version requires cargo(1) to be available. Therefore, we let the Ports system to extract the bootstrap cargo (it was done by the Cargo build system before) and set a configure flag to point to it. We need to set OPENSSL_DIR as a Make environment variable because the openssl-sys crate's build.rs doesn't know about FreeBSD. Finally, enable devel/cargo on i386. A bootstrap cargo executable is available for FreeBSD/i386 since last October but I missed it. We can now use it to build a port/package for i386. PR: 216143 Approved by: bapt (mentor) Differential Revision: https://reviews.freebsd.org/D9273 Changes: head/devel/cargo/Makefile head/devel/cargo/distinfo head/devel/cargo/distinfo.amd64 head/devel/cargo/distinfo.i386 head/devel/cargo/files/ A commit references this bug: Author: dumbbell Date: Mon Jan 23 21:55:49 UTC 2017 New revision: 432293 URL: https://svnweb.freebsd.org/changeset/ports/432293 Log: devel/cargo: Fix bootstrap checksum + use ${OPENSSLBASE} Because devel/cargo and lang/rust-nightly use a different DISTFILE with the same name, the wrong checksum and size was selected in devel/cargo. To fix this, keep the parent directory (a date) in the DISTFILE name. Also, depending on the value of DEFAULT_VERSION=ssl= and whether OpenSSL from ports is installed, the openssl-sys crate could fail to build. Passing OpenSSL location to make was way too complicated anyway. So simplify it and use ${OPENSSLBASE}. This was reported in PR 216385. Bump PORTREVISION to 1. PR: 216143, 216385 Reported by: gja822@narod.ru (216385, SSL build failure) Approved by: antoine (mentor), jbeich Differential Revision: https://reviews.freebsd.org/D9295 Changes: head/devel/cargo/Makefile head/devel/cargo/distinfo.amd64 head/devel/cargo/distinfo.i386 A commit references this bug: Author: jbeich Date: Mon Jan 23 22:05:31 UTC 2017 New revision: 432294 URL: https://svnweb.freebsd.org/changeset/ports/432294 Log: www/firefox: re-enable RUST on i386 after r432111 PR: 216143 Changes: head/www/firefox/Makefile Quick status update on boostrapping on aarch64. I'm following japaric's guide [1]. Step 1 (manually cross-compiling the `core` crate) and step 2 (teaching `rustc` about aarch64-unknown-freebsd) are done. I'm at step 3 (cross-compiling `std`). I made the required changes to the `libc` crate and `std` builds but doesn't link: https://gist.github.com/dumbbell/f20431fb05b7879488d771b1110f4032 The `panic_unwind` crate builds fine, but it is not picked by `std`. I asked for help on IRC. I submitted no pull request so far: I want to be sure the existing patches are fine. [1] https://www.reddit.com/r/rust/comments/5ag60z/how_do_i_bootstrap_rust_to_crosscompile_for_a_new/ Another quick update: I could cross-compile rust-std for aarch64, using `x.py`, Clang 3.7 and a cross-compiled buildworld. So far, it was much simpler than the guide I linked before. However, `x.py` doesn't even try to compile rustc. I need to firgure out why. Status update:
I could finally cross-compile a working rustc for aarch64:
dumbbell@ref12-aarch64:~/rust % uname -a
FreeBSD ref12-aarch64.freebsd.org 12.0-CURRENT FreeBSD 12.0-CURRENT #0
r306376: Tue Sep 27 20:33:16 UTC 2016
peter@build-12.freebsd.org:/usr/obj/arm64.aarch64/usr/src/sys/CLUSTER12
arm64
dumbbell@ref12-aarch64:~/rust % ./nightly/bin/rustc -V
rustc 1.16.0-nightly (3f4718c82 2017-01-26)
dumbbell@ref12-aarch64:~/rust % ./nightly/bin/rustc -o hello hello.rs
dumbbell@ref12-aarch64:~/rust % ./hello
Hello from Rust on FreeBSD/arm64!
The next steps are:
- bootstrapping Cargo;
- testing a native build of Rust using the cross-compiled compiler and Cargo;
- submitting pull requests;
- asking upstream to enable automated buildd of nightlies.
Status update:
Cargo is bootstrapped! That was easy compared to Rust.
dumbbell@ref12-aarch64:~/rust % ./nightly/bin/cargo --version
cargo-0.17.0-nightly (38f52ea8 2017-02-02)
dumbbell@ref12-aarch64:~/rust % cargo new hello_world --bin
Created binary (application) `hello_world` project
dumbbell@ref12-aarch64:~/rust % cd hello_world/
dumbbell@ref12-aarch64:~/rust/hello_world % cargo build
Compiling hello_world v0.1.0 (file:///home/dumbbell/rust/hello_world)
Finished dev [unoptimized + debuginfo] target(s) in 3.50 secs
dumbbell@ref12-aarch64:~/rust/hello_world % ./target/debug/hello_world
Hello, world!
dumbbell@ref12-aarch64:~/rust/hello_world % cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
Running `target/debug/hello_world`
Hello, world!
I could cross-compile Rust for aarch64 again from scratch, just to verify my notes, scripts and patches. I put the results here (including Cargo), in case people want to play with them: https://people.freebsd.org/~dumbbell/rust/aarch64/ There is an install script in the three archives. Patches are in those two Git branches: https://github.com/dumbbell/rust/tree/support-aarch64-unknown-freebsd https://github.com/dumbbell/libc/tree/support-aarch64-unknown-freebsd I installed them on ref12-arm64.freebsd.org and now, I'm building Rust there to see if it can self-host. Once this is done, I can finally submit patches :) Rust can self-host on FreeBSD/aarch64:
...
Copying stage2 compiler (aarch64-unknown-freebsd)
Copying stage2 std from stage1 (... -> aarch64-unknown-freebsd)
Copying stage2 test from stage1 (... -> aarch64-unknown-freebsd)
Copying stage2 rustc from stage1 (... -> aarch64-unknown-freebsd)
Build completed in 2:04:16
dumbbell@ref12-aarch64:~/rust/rust %
I submitted two pull requests:
https://github.com/rust-lang/rust/pull/39491
https://github.com/rust-lang/libc/pull/512
I put my notes in a gist: https://gist.github.com/dumbbell/b587da50ef014078da9e732a4331ebad The `libc` pull request was approved, tested and merged! The `rust` pull request was approved and is being tested. The pull request to rust-lang/rust was merged too. I'm going to ask about when/how the libc Git submodule is updated and how to enable official FreeBSD/aarch64 bootstraps. The pull request to update the libc Git submodule in Rust was just merged: https://github.com/rust-lang/rust/pull/39601 Thus all patches are in to build for and run on FreeBSD/aarch64. The next step is asking upstream how to enable automatic creation of official FreeBSD/aarch64 bootstraps. In the meantime, it's probably possible we produce our own use it to build packages of lang/rust-nightly, perhaps lang/rust too. For Cargo, I'm wondering if we would need a devel/cargo-nightly. I'll play around that as soon as I can. For sparc64, the patches are ready, but I can't get a working C++ cross-compiler: either I get compilation/link failures or the produced executable segfaults. To be exact, I could obtain a working hello world with base's libstdc++, but LLVM (built as part of the Rust build) requires libstdc++ 4.7+. I'm waiting for the following review to be approved (updating rust-nightly to a recent snapshot): https://reviews.freebsd.org/D9286 Then I will modify the port to build on FreeBSD/aarch64. About sparc64, I'm building lang/gcc6 on the target because the cross-compiler (sparc64-gcc) is 6.3.0 and it doesn't want libstdc++ from gcc 4.9. Unfortunately, I'm having some troubles: comparison of stage2 and stage3 fails. I'm now rebuilding the port with the `BOOTSTRAP` option. About armv6, Mikaël Urankar could bootstrap Rust and Cargo. However, Cargo segfaults while verifying if Rust can self-host. Status update for sparc64 I could finally build lang/gcc6 natively and use its libstdc++ with sparc64-gcc cross-compiler. Building the bootstrap of Rust and Cargo was straightforward after that. I uploaded the bootstrap on freefall: https://people.freebsd.org/~dumbbell/rust/sparc64/ I could build and run a Hello World with Cargo. Note that the bootstrapped rustc requires libstdc++ from lang/gcc6. I'm going to test Rust self-hosting now. A commit references this bug: Author: dumbbell Date: Wed Feb 22 19:04:05 UTC 2017 New revision: 434631 URL: https://svnweb.freebsd.org/changeset/ports/434631 Log: lang/rust, lang/rust-nightly: Update to 1.15.1 and 1.17.0 respectively Both versions of Rust seen changes to their build system. Now they require a bootstrap of Cargo and thus, a snapshot of Cargo's registry. Each library has its own suffix instead of a common suffix for a given version of Rust. Thus all future updates of the ports will include changes to their plist. $DISTFILES are now all under the `rust` directory in `$DISTDIR`. A similar change will be committed to devel/cargo later. lang/rust-nightly is enabled on i386 (bugzilla 216143). The source snapshot also contains everything to build it on aarch64, but enabling it will be committed later. PR: 216143 Approved by: riggs, bapt (mentor), antoine (mentor) Differential Revision: https://reviews.freebsd.org/D9286 Changes: head/lang/rust/Makefile head/lang/rust/distinfo head/lang/rust/distinfo.amd64 head/lang/rust/distinfo.i386 head/lang/rust/distinfo.x86_64 head/lang/rust/files/patch-src_bootstrap_compile.rs head/lang/rust/files/patch-src_bootstrap_config.rs head/lang/rust/files/patch-src_bootstrap_install.rs head/lang/rust/files/patch-src_librustc__llvm_build.rs head/lang/rust/pkg-plist head/lang/rust/pkg-plist.amd64 head/lang/rust/pkg-plist.i386 head/lang/rust-nightly/Makefile head/lang/rust-nightly/distinfo head/lang/rust-nightly/files/patch-mk_main.mk head/lang/rust-nightly/files/patch-mk_rt.mk head/lang/rust-nightly/files/patch-src_etc_get-stage0.py head/lang/rust-nightly/files/patch-src_librustc__llvm_build.rs head/lang/rust-nightly/pkg-plist head/lang/rust-nightly/pkg-plist.amd64 head/lang/rust-nightly/pkg-plist.i386 I still have change waiting for approval before I can start with lang/rust* on aarch64. Sorry it takes so long. About sparc64, I made little progress. Rust is having issues with kqueue(2) there. Also gcc segfaults quite often. Any news on aarch64 in lang/rust (1.17+) + devel/cargo (0.18+)? I can probably keep --disable-rust availability for a while longer (ideally until ESR59) as Mozilla doesn't plan to remove non-oxidated codepaths just yet. Here's how FF55 version currently looks like: https://github.com/jbeich/gecko-dev/commit/5528caa9d548 I resumed the work on aarch64 last week and this week-end. I have some patches almost ready for the port, but the bootstrap compiler I created previously is too old and can't compile Rust anymore. I'm now creating a new bootstrap compiler based on Rust 1.17.0. Note that Rust 1.17.0 has the commits for FreeBSD/aarch64 so it should not require any additional patches. The patches I was talking about are the changes to lang/rust and lang/rust-nightly themselves. Here is the patch on Phabricator to enable lang/rust on FreeBSD/aarch64: https://reviews.freebsd.org/D10857 And here is the patch on Phabricator to enable devel/cargo on FreeBSD/aarch64: https://reviews.freebsd.org/D10867 This patch is *UNTESTED* for now. A commit references this bug: Author: dumbbell Date: Sat Jun 17 14:44:30 UTC 2017 New revision: 443777 URL: https://svnweb.freebsd.org/changeset/ports/443777 Log: lang/rust: Enable on FreeBSD/aarch64 + various fixes This port now uses an unofficial bootstrap of Rust 1.17.0 and Cargo 0.18.0 for FreeBSD/aarch64. Here are my notes to create this boostrap: https://gist.github.com/dumbbell/b587da50ef014078da9e732a4331ebad The port's Makefile was changed to: - allow to override the versions and directories of the bootstrap archives; - patch the bootstrap script and manifest with those versions/directories. Beside changes to support FreeBSD/aarch64, the port received the following fixes: - Pass python path to the configure script. - Fix and simplify the extraction of rust-std bootstrap by re-using existing variables. - Make sure tar(1) doesn't produce sparse file entries when it recreates the rust-std bootstrap because Python 2's tarfile module doesn't support them (PR 219842). PR: 216143, 219842 Differential Revision: https://reviews.freebsd.org/D10857 Changes: head/lang/rust/Makefile head/lang/rust/distinfo head/lang/rust/files/extra-patch-ino64 Support for FreeBSD/aarch64 was just enabled in lang/rust. Next step is to do the same in devel/cargo. No ETA :) A commit references this bug: Author: dumbbell Date: Sat Jun 17 15:37:26 UTC 2017 New revision: 443780 URL: https://svnweb.freebsd.org/changeset/ports/443780 Log: devel/cargo: Update to 0.19.0 + enable on FreeBSD/aarch64 See r443777 for details on the FreeBSD/aarch64 bootstrap. PR: 216143 Differential Revision: https://reviews.freebsd.org/D10867 Changes: head/devel/cargo/Makefile head/devel/cargo/distinfo A commit references this bug: Author: jbeich Date: Thu Jun 22 01:19:17 UTC 2017 New revision: 444090 URL: https://svnweb.freebsd.org/changeset/ports/444090 Log: www/firefox: enable RUST on aarch64 after r443777 and r443780 PR: 216143 Changes: head/www/firefox/Makefile www/firefox wasn't ready: https://bugzilla.mozilla.org/show_bug.cgi?id=1376411 Any news on armv6? (In reply to Jan Beich from comment #23) > I can probably keep --disable-rust availability for a while longer > (ideally until ESR59) as Mozilla doesn't plan to remove non-oxidated > codepaths just yet. https://bugzilla.mozilla.org/show_bug.cgi?id=1261841 made it non-trivial, so FF56 (circa 2017-09-26) will probably be the end. powerpc* and sparc64 are already out of hope due Skia not supporting big-endian archs since FF53. (In reply to Jan Beich from comment #33) I gave it a shot to not avail. I was able to bootstrap rust and cargo, unfortunately cargo segfault at startup and rust wasn't able to compile itself. I'll try again in august. FWIW firefox doesn't build on armv6. (In reply to mikael.urankar from comment #34) > FWIW firefox doesn't build on armv6. www/firefox hasn't appeared on pkg-fallout@ since 51.0 when it was blocked by a Clang crash. Now it's blocked by math/fftw3 and maybe more ports. I can't really begin on a fix without prebuilt packages as my build of devel/llvm40 is killed as runaway after ~20 hours spent on trying to link bin/clang-check. Mikael, have you tried to bootstrap Rust on armv7? For one, FreeBSD armv7 appears to assume NEON. $ cc -v FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn) Target: armv7-unknown-freebsd12.0-gnueabihf Thread model: posix InstalledDir: /nxb-bin/usr/bin $ cc -dM -E -</dev/null 2>&1 | fgrep -i -e vfp -e neon #define __ARM_NEON 1 #define __ARM_NEON_FP 0x4 #define __ARM_NEON__ 1 #define __ARM_PCS_VFP 1 #define __ARM_VFPV3__ 1 #define __VFP_FP__ 1 Jan, I have rust bootstrapped for armv6 and armv7. The problem is that bootstrap is built on head/12. The bootstrap on stable/11 fails in final phase (when making dist packages, with strange linker error) and I haven't access to releng/11.0 machine. But I'm current trapped by fixing jemalloc / libc TLS bugs in head, so rustc is on hold now :( MARKED AS SPAM A commit references this bug: Author: tobik Date: Mon Feb 18 15:30:45 UTC 2019 New revision: 493268 URL: https://svnweb.freebsd.org/changeset/ports/493268 Log: lang/rust: Add aarch64, armv{6,7}, and powerpc64 bootstraps PR: 216143, 228892 Submitted by: Mika?l Urankar <mikael.urankar@gmail.com> Differential Revision: https://reviews.freebsd.org/D18367 Changes: head/lang/rust/Makefile head/lang/rust/distinfo head/lang/rust/files/patch-src_bootstrap_bootstrap.py head/lang/rust/files/patch-src_bootstrap_native.rs head/lang/rust/files/patch-src_libbacktrace_fileline.c head/lang/rust/files/patch-src_libcompiler__builtins_build.rs head/lang/rust/files/patch-src_liblibc_src_unix_bsd_freebsdlike_freebsd_arm.rs head/lang/rust/files/patch-src_liblibc_src_unix_bsd_freebsdlike_freebsd_mod.rs head/lang/rust/files/patch-src_liblibc_src_unix_bsd_freebsdlike_freebsd_powerpc64.rs head/lang/rust/files/patch-src_liblibc_src_unix_bsd_freebsdlike_mod.rs head/lang/rust/files/patch-src_librustc__llvm_build.rs head/lang/rust/files/patch-src_librustc__target_spec_armv6__unknown__freebsd.rs head/lang/rust/files/patch-src_librustc__target_spec_armv7__unknown__freebsd.rs head/lang/rust/files/patch-src_librustc__target_spec_mod.rs head/lang/rust/files/patch-src_librustc__target_spec_powerpc64__unknown__freebsd.rs head/lang/rust/files/patch-src_libstd_build.rs head/lang/rust/files/patch-src_libstd_os_raw_mod.rs head/lang/rust/files/patch-src_libstd_sys_unix_stack__overflow.rs head/lang/rust/files/patch-src_llvm_utils_llvm-build_llvmbuild_main.py head/lang/rust/files/patch-vendor_backtrace_src_backtrace_libunwind.rs head/lang/rust/files/patch-vendor_libc_src_unix_bsd_freebsdlike_freebsd_arm.rs head/lang/rust/files/patch-vendor_libc_src_unix_bsd_freebsdlike_freebsd_mod.rs head/lang/rust/files/patch-vendor_libc_src_unix_bsd_freebsdlike_freebsd_powerpc64.rs head/lang/rust/files/patch-vendor_libc_src_unix_bsd_freebsdlike_mod.rs head/lang/rust/files/patch-vendor_rustc-ap-rustc__target_spec_armv6__unknown__freebsd.rs head/lang/rust/files/patch-vendor_rustc-ap-rustc__target_spec_armv7__unknown__freebsd.rs head/lang/rust/files/patch-vendor_rustc-ap-rustc__target_spec_mod.rs head/lang/rust/files/patch-vendor_rustc-ap-rustc__target_spec_powerpc64__unknown__freebsd.rs Many Tier2 are supported after ports r493268. The rest is mostly Tier3, not represented on the package cluster. MARKED AS SPAM |