Summary: | www/firefox (56+) fails to build using lang/rust (1.21+) built with PORT_LLVM=on: error[E0432]: unresolved import `x86::sse2` | ||||||
---|---|---|---|---|---|---|---|
Product: | Ports & Packages | Reporter: | Dimitry Andric <dim> | ||||
Component: | Individual Port(s) | Assignee: | freebsd-gecko (Nobody) <gecko> | ||||
Status: | Closed FIXED | ||||||
Severity: | Affects Some People | CC: | dumbbell, fullermd, gja822, jakub_lach, jbeich, jcfyecrayz, keren_sky, mi, vvd, w.schwarzenfeld | ||||
Priority: | --- | Flags: | bugzilla:
maintainer-feedback?
(gecko) |
||||
Version: | Latest | ||||||
Hardware: | Any | ||||||
OS: | Any | ||||||
Bug Depends on: | 223342 | ||||||
Bug Blocks: | |||||||
Attachments: |
|
I can generally confirm it as rust related; I rebuild the new PORTREVISION of firefox on one system before updating rust, and it went fine; doing it again from the same ports tree after the rust update fails as listed. (In reply to fullermd from comment #1) "before updating rust" meaning with 1.20.0. I have the same error on amd64 too. $ grep ^OPTIONS /var/db/ports/www_firefox/options OPTIONS_FILE_SET+=BUNDLED_CAIRO OPTIONS_FILE_UNSET+=CANBERRA OPTIONS_FILE_UNSET+=DBUS OPTIONS_FILE_UNSET+=DEBUG OPTIONS_FILE_UNSET+=DTRACE OPTIONS_FILE_SET+=FFMPEG OPTIONS_FILE_SET+=GCONF OPTIONS_FILE_UNSET+=INTEGER_SAMPLES OPTIONS_FILE_SET+=LIBPROXY OPTIONS_FILE_SET+=OPTIMIZED_CFLAGS OPTIONS_FILE_UNSET+=PGO OPTIONS_FILE_UNSET+=PROFILE OPTIONS_FILE_UNSET+=TEST OPTIONS_FILE_SET+=ALSA OPTIONS_FILE_UNSET+=JACK OPTIONS_FILE_UNSET+=PULSEAUDIO OPTIONS_FILE_UNSET+=SNDIO $ grep CPUTYPE?= /etc/make.conf CPUTYPE?=core2 With rust 1.20 compiled fine, with rust 1.21 - this error. Comment on attachment 187555 [details] Poudriere log for www/firefox Hmm, I can't reproduce on 11.1-RELEASE amd64 using the same port options and the same CPUTYPE set. Build log: http://sprunge.us/EYDH Can you try rebuilding lang/rust without CPUTYPE defined in make.conf? *** Bug 223302 has been marked as a duplicate of this bug. *** Why does Summary field mentions i386 when every reporter so far is on amd64? I have CPUTYPE?=bdver2 in make.conf and the same error. (And without poudriere, just from make command within /usr/ports/www/firefox directory.) Options: OPTIONS_FILE_UNSET+=BUNDLED_CAIRO OPTIONS_FILE_UNSET+=CANBERRA OPTIONS_FILE_SET+=DBUS OPTIONS_FILE_UNSET+=DEBUG OPTIONS_FILE_UNSET+=DTRACE OPTIONS_FILE_SET+=FFMPEG OPTIONS_FILE_SET+=GCONF OPTIONS_FILE_UNSET+=INTEGER_SAMPLES OPTIONS_FILE_UNSET+=LIBPROXY OPTIONS_FILE_SET+=OPTIMIZED_CFLAGS OPTIONS_FILE_UNSET+=PGO OPTIONS_FILE_UNSET+=PROFILE OPTIONS_FILE_SET+=TEST OPTIONS_FILE_SET+=ALSA OPTIONS_FILE_UNSET+=JACK OPTIONS_FILE_SET+=PULSEAUDIO OPTIONS_FILE_UNSET+=SNDIO (In reply to Jan Beich from comment #7) > Why does Summary field mentions i386 when every reporter so far is on amd64? Sorry about it, that was a complete misinterpretation on my side. I had accidentally thought that the system I saw this on was i386, but it was amd64, in fact. I have now fixed the subject. I too have the same issue building Firefox. FreeBSD 11.1 amd64. I believe it occurred after updating rust to 1.21 Thank you. (In reply to Jan Beich from comment #4) Removed CPUTYPE from /etc/make.conf, rebuilded rust 1.21.0_1, firefox build with same error: Compiling simd v0.2.0 error[E0432]: unresolved import `x86::sse2` --> /tmp/work/usr/ports/www/firefox/work/firefox-56.0.2/third_party/rust/simd/src/common.rs:17:10 | 17 | use x86::sse2::common; | ^^^^ Could not find `sse2` in `x86` error: aborting due to previous error error: Could not compile `simd`. P.S. FreeBSD 11.1, amd64. For information: $ grep ^OPTIONS /var/db/ports/lang_rust/options OPTIONS_FILE_UNSET+=DOCS OPTIONS_FILE_UNSET+=GDB OPTIONS_FILE_UNSET+=LLNEXTGEN OPTIONS_FILE_SET+=PORT_LLVM OPTIONS_FILE_UNSET+=SOURCES (In reply to vvd from comment #12) > OPTIONS_FILE_SET+=PORT_LLVM Can you try without? Rust uses patched LLVM 4.0 by default which maybe required for cfg_target_feature to detect SSE2. https://github.com/rust-lang/rust/pull/31709 https://github.com/rust-lang/rust/blob/690ff045949a/src/rustllvm/PassWrapper.cpp#L200 https://github.com/rust-lang/rust/blob/690ff045949a/src/bootstrap/compile.rs#L543 https://github.com/rust-lang/rust/blob/690ff045949a/src/bootstrap/lib.rs#L505 Rust 1.21.0 has https://github.com/rust-lang/rust/commit/881a724660 which would explain why simd crate built fine with Rust 1.20.0. If you really want PORT_LLVM=on maybe drop --enable-rust-simd from Mk/bsd.gecko.mk. It's mainly used by https://bugzilla.mozilla.org/show_bug.cgi?id=1261841 (In reply to Jan Beich from comment #13) > Can you try without? In progress. Will write result in about 1 hour… I hope. :-D > If you really want PORT_LLVM=on Too long time for compile the rust with embedded llvm. And I already have the llvm 4.0.1 installed. IMHO, it's bad solution to build and install a huge port llvm twice in the system… (In reply to vvd from comment #16) > Too long time for compile the rust with embedded llvm. It's a stripped down version: no compiler-rt, no clang, no extra clang tools, no lld, no lldb, no openmp. lang/rust also takes a lot of time to build because it does a poor job of parallelizing release builds. cf. https://internals.rust-lang.org/t/help-test-out-thinlto/6017 > And I already have the llvm 4.0.1 installed. PORT_LLVM=on statically links LLVM, so you don't save space. Given both bundled llvm and devel/llvm* have similar dependencies if one of those is obsolete poudriere would force a rebuild unless you trick it into not looking at the whole dependency chain. I could reproduce in 11.1-RELEASE i386 jail. PORT_LLVM=on is really the culprit. $ git clone https://github.com/rust-lang-nursery/simd $ cd simd $ git checkout v0.2.0 $ export RUSTC_BOOTSTRAP=1 # cheat Nightly requirement $ cargo build --release Compiling simd v0.2.0 (file:///tmp/simd) error[E0432]: unresolved import `x86::sse2` --> src/common.rs:17:10 | 17 | use x86::sse2::common; | ^^^^ Could not find `sse2` in `x86` error: aborting due to previous error $ rustc --print target-features Target features help is not supported by this LLVM version. FWIW, PORT_LLVM=on also means you're going to miss out on bug 223314. (In reply to vvd from comment #16) >>> OPTIONS_FILE_SET+=PORT_LLVM >> Can you try without? > In progress. Will write result in about 1 hour… I hope. :-D Tested. Firefox build fine on amd64 using rust with OPTIONS_FILE_UNSET+=PORT_LLVM and CPUTYPE?=core2 in /etc/make.conf. Testing i386. Right, after rebuilding rust without PORT_LLVM, I get the features list: $ rustc --print target-features Available features for this target: 16bit-mode - 16-bit mode (i8086). 32bit-mode - 32-bit mode (80386). 3dnow - Enable 3DNow! instructions. 3dnowa - Enable 3DNow! Athlon instructions. 64bit - Support 64-bit instructions. 64bit-mode - 64-bit mode (x86_64). adx - Support ADX instructions. aes - Enable AES instructions. atom - Intel Atom processors. avx - Enable AVX instructions. avx2 - Enable AVX2 instructions. avx512bw - Enable AVX-512 Byte and Word Instructions. avx512cd - Enable AVX-512 Conflict Detection Instructions. avx512dq - Enable AVX-512 Doubleword and Quadword Instructions. avx512er - Enable AVX-512 Exponential and Reciprocal Instructions. avx512f - Enable AVX-512 instructions. avx512ifma - Enable AVX-512 Integer Fused Multiple-Add. avx512pf - Enable AVX-512 PreFetch Instructions. avx512vbmi - Enable AVX-512 Vector Byte Manipulation Instructions. avx512vl - Enable AVX-512 Vector Length eXtensions. bmi - Support BMI instructions. bmi2 - Support BMI2 instructions. call-reg-indirect - Call register indirect. clflushopt - Flush A Cache Line Optimized. clwb - Cache Line Write Back. cmov - Enable conditional move instructions. cx16 - 64-bit with cmpxchg16b. f16c - Support 16-bit floating point conversion instructions. fast-lzcnt - LZCNT instructions are as fast as most simple integer ops. fast-partial-ymm-write - Partial writes to YMM registers are fast. fast-scalar-fsqrt - Scalar SQRT is fast (disable Newton-Raphson). fast-vector-fsqrt - Vector SQRT is fast (disable Newton-Raphson). fma - Enable three-operand fused multiple-add. fma4 - Enable four-operand fused multiple-add. fsgsbase - Support FS/GS Base instructions. fxsr - Support fxsave/fxrestore instructions. hle - Support HLE. idivl-to-divb - Use 8-bit divide for positive values less than 256. idivq-to-divl - Use 32-bit divide for positive values less than 2^32. invpcid - Invalidate Process-Context Identifier. lea-sp - Use LEA for adjusting the stack pointer. lea-uses-ag - LEA instruction needs inputs at AG stage. lzcnt - Support LZCNT instruction. mmx - Enable MMX instructions. movbe - Support MOVBE instruction. mpx - Support MPX instructions. mwaitx - Enable MONITORX/MWAITX timer functionality. pad-short-functions - Pad short functions. pclmul - Enable packed carry-less multiplication instructions. pcommit - Enable Persistent Commit. pku - Enable protection keys. popcnt - Support POPCNT instruction. prefetchwt1 - Prefetch with Intent to Write and T1 Hint. prfchw - Support PRFCHW instructions. rdrnd - Support RDRAND instruction. rdseed - Support RDSEED instruction. rtm - Support RTM instructions. sahf - Support LAHF and SAHF instructions. sgx - Enable Software Guard Extensions. sha - Enable SHA instructions. slm - Intel Silvermont processors. slow-bt-mem - Bit testing of memory is slow. slow-incdec - INC and DEC instructions are slower than ADD and SUB. slow-lea - LEA instruction with certain arguments is slow. slow-pmulld - PMULLD instruction is slow. slow-shld - SHLD instruction is slow. slow-unaligned-mem-16 - Slow unaligned 16-byte memory access. slow-unaligned-mem-32 - Slow unaligned 32-byte memory access. smap - Supervisor Mode Access Protection. soft-float - Use software floating point features.. sse - Enable SSE instructions. sse-unaligned-mem - Allow unaligned memory operands with SSE instructions. sse2 - Enable SSE2 instructions. sse3 - Enable SSE3 instructions. sse4.1 - Enable SSE 4.1 instructions. sse4.2 - Enable SSE 4.2 instructions. sse4a - Support SSE 4a instructions. ssse3 - Enable SSSE3 instructions. tbm - Enable TBM instructions. vmfunc - VM Functions. x87 - Enable X87 float instructions. xop - Enable XOP instructions. xsave - Support xsave instructions. xsavec - Support xsavec instructions. xsaveopt - Support xsaveopt instructions. xsaves - Support xsaves instructions. Use +feature to enable a feature, or -feature to disable it. For example, rustc -C -target-cpu=mycpu -C target-feature=+feature1,-feature2 And then www/firefox does build to the end. (In reply to vvd from comment #20) >>>> OPTIONS_FILE_SET+=PORT_LLVM >>> Can you try without? >> In progress. Will write result in about 1 hour… I hope. :-D > Tested. Firefox build fine on amd64 using rust with OPTIONS_FILE_UNSET+=PORT_LLVM and CPUTYPE?=core2 in /etc/make.conf. > > Testing i386. Tested on i386: the firefox rebuilded without errors with same configuration for the rust: OPTIONS_FILE_UNSET+=PORT_LLVM. The rust's option PORT_LLVM is the key. Same here, PORT_LLVM. FreeBSD 11.1-STABLE #0 r324952 amd64, core2 CPU. If I'm not mistaken, PORT_LLVM uses already installed port's LLVM if present (not building the one bundled with rust), thus speeding up things considerably? *** Bug 223570 has been marked as a duplicate of this bug. *** A commit references this bug: Author: dumbbell Date: Mon Nov 27 18:16:15 UTC 2017 New revision: 454983 URL: https://svnweb.freebsd.org/changeset/ports/454983 Log: lang/rust: Remove the LLVM_PORT option FTR, this option allowed to used LLVM from ports instead of building the bundled copy. The problem is that this combination isn't really tested upstream. This led to regressions which are difficult to diagnose. For instance, in Rust 1.21.0, the bundled LLVM provided a new API to query the features supported by the target arch. The equivalent code inside Rust was removed to use that new API. Unfortunately, building Rust 1.21.0+ with a copy of LLVM not providing this API didn't failed but instead made that list of CPU features empty. This resulted in the following obscure build failure in Firefox: error[E0432]: unresolved import `x86::sse2` To avoid future pain for both end users and maintainers, we decided to remove that option. Yes, it will increase the (already long) time to build Rust, but it should save time wasted on debugging what is not really supported anyway. PR: 223342, 223300 Reported by: Many users Changes: head/lang/rust/Makefile head/lang/rust/files/config.toml This is "fixed" in the sense that the LLVM_PORT option is now gone. After last update of rust-1.22.1 to 1.22.1_1 and firefox-57.0_2,1 to 57.0_3,1 error was returned: Compiling simd v0.2.0 error[E0432]: unresolved import `x86::sse2` --> /usr/obj/usr/ports/www/firefox/work/firefox-57.0/third_party/rust/simd/src/common.rs:17:10 | 17 | use x86::sse2::common; | ^^^^ Could not find `sse2` in `x86` Compiling term v0.4.5 error: aborting due to previous error error: Could not compile `simd`. warning: build failed, waiting for other jobs to finish... error: build failed ======================== It's FreeBSD 11.1 i386. On 11.1 amd64 no errors. What is the output of `rustc --print target-features`? I think this commit broke ff build: https://svnweb.freebsd.org/ports/head/lang/rust/files/patch-src_librustc__back_target_i686__unknown__freebsd.rs?revision=454995&view=markup&sortby=date The test in progress - I'll confirm it soon. Probably because -march= and -C target-cpu= are out of sync e.g., when CPUTYPE is defined in make.conf. The fix is to pass RUSTFLAGS+=-C target-feature=+sse2 explicitly on i386 when MACHINE_CPU contains sse2. Ofc CPUTYPE defined: CPUTYPE?=prescott The fix is to check CPUTYPE and turn on/off SSE2 automatically instead of turn off it on ALL i386 by default. A commit references this bug: Author: jbeich Date: Tue Nov 28 12:57:57 UTC 2017 New revision: 455039 URL: https://svnweb.freebsd.org/changeset/ports/455039 Log: gecko: respect CPUTYPE for Rust code After r454995 build may fail if SSE2 is only enabled for C/C++ code as simd crate expects SSE2 based on MACHINE_CPU. PR: 223415 223300 Reported by: vvd@unislabs.com Changes: head/Mk/bsd.gecko.mk Now rustc can emit SSE3 for CPUTYPE=prescott as well. Whether it will probably depends on auto-vectorizer. $ rustc --print target-features | grep sse sse - Enable SSE instructions. sse-unaligned-mem - Allow unaligned memory operands with SSE instructions. sse2 - Enable SSE2 instructions. sse3 - Enable SSE3 instructions. sse4.1 - Enable SSE 4.1 instructions. sse4.2 - Enable SSE 4.2 instructions. sse4a - Support SSE 4a instructions. ssse3 - Enable SSSE3 instructions. With rust-1.22.1 firefox-57.0_3,1 build fine. Testing build with rust-1.22.1_1 and new Mk/bsd.gecko.mk now. > Testing build with rust-1.22.1_1 and new Mk/bsd.gecko.mk now.
Test passed without errors.
(In reply to commit-hook from comment #32) The commit changes gecko.mk to derive the target-cpu setting given to rustc from the CFLAGS. This is a very elegant solution and would've been perfect, if not for the common case of "-march=native". Because, sadly, "native" to rustc currently means "generic", instead of "check the features of the currently-used processor and enable them all". And "generic" means "no SSE2". And, because it means generic, the firefox build fails with that same error about SSE2, when CPUTYPE is set to "native". I'm not sure, how to fix it correctly, but a warning advising user to set CPUTYPE explicitly may be helpful. (In reply to Mikhail T. from comment #36) > "native" to rustc currently means "generic" I cannot reproduce. $ RUSTC_BOOTSTRAP=1 rustc -C target-cpu=native --print cfg debug_assertions target_arch="x86" target_endian="little" target_env="" target_family="unix" target_feature="avx" target_feature="avx2" target_feature="bmi" target_feature="bmi2" target_feature="fma" target_feature="lzcnt" target_feature="popcnt" target_feature="rdrnd" target_feature="rdseed" target_feature="sse" target_feature="sse2" target_feature="sse3" target_feature="sse4.1" target_feature="sse4.2" target_feature="ssse3" target_has_atomic="16" target_has_atomic="32" target_has_atomic="64" target_has_atomic="8" target_has_atomic="ptr" target_os="freebsd" target_pointer_width="32" target_vendor="unknown" unix > I cannot reproduce. > $ RUSTC_BOOTSTRAP=1 rustc -C target-cpu=native --print cfg For me the above command prints something different: debug_assertions target_arch="x86" target_endian="little" target_env="" target_family="unix" target_os="freebsd" target_pointer_width="32" unix Maybe, that's because I compiled rust itself (1.22.1) with "march=native"? BTW, if you are trying to set RUSTFLAGS intelligently in bsd.gecko.mk, you may also want to add the `-C codegen-units=${MAKE_JOBS_NUMBER}` to parallelize the compilations. (In reply to Mikhail T. from comment #38) > For me the above command prints something different: Err, sorry, with the RUSTC_BOOTSTRAP=1 I get the longer list below -- but still without SSE2: debug_assertions target_arch="x86" target_endian="little" target_env="" target_family="unix" target_has_atomic="16" target_has_atomic="32" target_has_atomic="64" target_has_atomic="8" target_has_atomic="ptr" target_os="freebsd" target_pointer_width="32" target_vendor="unknown" unix (In reply to Mikhail T. from comment #38) > Maybe, that's because I compiled rust itself (1.22.1) with "march=native"? Well, I use CPUTYPE=native myself (on -CURRENT amd64) and on head-i386 poudriere jail. Obviously, MACHINE_CPU had to be populated manually because bsd.cpu.mk is dumb (see bug 112997). > may also want to add the `-C codegen-units=${MAKE_JOBS_NUMBER}` to parallelize the compilations. No, -C codegen-units pessimizes LTO which is enabled by default for release builds. ThinLTO isn't ready yet. https://internals.rust-lang.org/t/help-test-out-thinlto/6017 (In reply to Jan Beich from comment #40) > Well, I use CPUTYPE=native myself (on -CURRENT amd64) and on head-i386 So, why would my rustc have a different cfg-output? I have it built with the following options: # Options for rust-1.22.1_2 _OPTIONS_READ=rust-1.22.1_2 _FILE_COMPLETE_OPTIONS_LIST=DOCS GDB LLNEXTGEN SOURCES OPTIONS_FILE_UNSET+=DOCS OPTIONS_FILE_UNSET+=GDB OPTIONS_FILE_SET+=LLNEXTGEN OPTIONS_FILE_UNSET+=SOURCES The CPUs in this machine are old -- could it be, it does not find SSE3 (or some other later addition) and presumes, SSE2 is not there either? Try using freebsd.org package. If it works bisect your environment by replicating it inside a poudriere jail or find another user with the same issue. If it doesn't work try checking if llvm40 has working -march=native then try booting a FreeBSD kernel from newer release (keep COMPAT_* in config). |
Created attachment 187555 [details] Poudriere log for www/firefox After updating my ports tree to r453055, lang/rust got updated to 1.21.0, and www/firefox to 56.0.2_3,1. Building under poudriere now results in an error (it used to build OK in the past): ... gmake[6]: Entering directory '/wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/obj-x86_64-unknown-freebsd11.1/toolkit/library/rust' force-cargo-library-build env RUSTC_BOOTSTRAP=1 RUSTFLAGS=' ' CARGO_TARGET_DIR=/wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/obj-x86_64-unknown-freebsd11.1/toolkit/library RUSTC=/usr/local/bin/rustc MOZ_SRC=/wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2 MOZ_DIST=/wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/obj-x86_64-unknown-freebsd11.1/dist LIBCLANG_PATH="/usr/local/llvm40/lib" CLANG_PATH="/usr/local/llvm40/bin/clang" PKG_CONFIG_ALLOW_CROSS=1 RUST_BACKTRACE=1 MOZ_TOPOBJDIR=/wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/obj-x86_64-unknown-freebsd11.1 MOZ_CARGO_WRAP_LDFLAGS="-pthread -Wl,--as-needed -fstack-protector -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-rpath-link,/wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/obj-x86_64-unknown-freebsd11.1/dist/bin -Wl,-rpath-link,/usr/local/lib" MOZ_CARGO_WRAP_LD=" /usr/bin/cc -std=gnu99" CARGO_TARGET_X86_64_UNKNOWN_FREEBSD_LINKER=/wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/build/cargo-linker /usr/local/bin/cargo build --release --frozen --manifest-path /wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/toolkit/library/rust/Cargo.toml --lib --target=x86_64-unknown-freebsd --features "servo bindgen simd-accel no-static-ideograph-encoder-tables" Compiling nom v1.2.4 Compiling rayon-core v1.2.0 Compiling nsstring_vendor v0.1.0 (file:///wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/servo/components/style/gecko_bindings/nsstring_vendor) Compiling either v1.1.0 Compiling smallvec v0.4.0 Compiling stable_deref_trait v1.0.0 Compiling same-file v0.1.3 Compiling precomputed-hash v0.1.0 Compiling scopeguard v0.3.2 Compiling void v1.0.2 Compiling regex-syntax v0.4.0 Compiling fnv v1.0.5 Compiling odds v0.2.25 Compiling libc v0.2.24 Compiling rustc-serialize v0.3.22 Compiling simd v0.2.0 error[E0432]: unresolved import `x86::sse2` --> /wrkdirs/usr/ports/www/firefox/work/firefox-56.0.2/third_party/rust/simd/src/common.rs:17:10 | 17 | use x86::sse2::common; | ^^^^ Could not find `sse2` in `x86` Compiling unicode-xid v0.0.4 Compiling unicode-normalization v0.1.4 error: aborting due to previous error error: Could not compile `simd`. I guess either the sse2 component got removed from rust, or the firefox component got updated to now include it?