Comment 1 Dimitry Andric 2021-04-24 11:04:17 UTC

From the stack trace, it looks like this is using the devel/llvm12 port? I'm adding Brooks on CC, because if any fix is identified, it will have to be added to that port.

Yes, it's using llvm12 package (autoinstalled as a build dependency).

FWIW, I just have completed firefox built without LTO.

Comment 3 Jan Beich 2021-04-25 00:51:13 UTC

Cross-language LTO likely requires Clang and rustc to use the same LLVM version. lang/rust at 1.51.0 bundles LLVM 11.0.1, so to use LTO add DEFAULT_VERSIONS+=llvm=11 to /etc/make.conf or www/firefox/Makefile.local. However, the next lang/rust update to 1.52.0 (scheduled on 2021-05-06) will have LLVM 12.0.0.

If LTO is important it needs to be enabled by default to avoid regressions due to changes in framework, dependencies or Firefox updates.

Comment 4 Dimitry Andric 2021-04-26 08:13:26 UTC

(In reply to Jan Beich from comment #3)
Ah yes, the rust+llvm11 and firefox+llvm12 mismatch is probably the cause, then. On my latest poudriere run I see the same "Wrong types for attribute" errors, followed by lld SIGABRT'ing.

Indeed llvm12 was only pulled in because of Firefox, so I'll try with llvm11 instead.

I still see the problem with llvm12 (on FreeBSD 13).

An earlier attempt to build firefox with llvm12 with LTO option & MAKE_JOBS_UNSAFE=yes had failed.

Not using LTO option, there is no "broken module" error from llvm and firefox build succeeds.

(In reply to jakub_lach from comment #2)

I've just finished building firefox with LTO, with

.if ${.CURDIR:M*/www/firefox}
DEFAULT_VERSIONS+=llvm=11
.endif

Comment 7 Dimitry Andric 2021-05-05 10:02:41 UTC

Hm I've put DEFAULT_VERSIONS+=llvm=11 in my make.conf, but it seems at least the mesa-dri ports still hardcode llvm10. :) So that's a little annyoing, but I guess it can't be helped?

(In reply to Dimitry Andric from comment #7)

See: https://gitter.im/FreeBSDDesktop/Lobby?at=6021a47aaa6a6f319d1326b5

To override LLVM ver for mesa-libs, one should locally patch the ports tree. And if one patches the ports tree, it makes more sense to ditch mesa-libs and use mesa-devel instead.

I don't know why x11@ didn't go the bsd.gecko.mk route (ignore llvm default in ports but respect make.conf).

FWIW, LLVM dependency is optional with mesa-devel, which IMO is much more flexible/configurable/usable for users who consume ports the Gentoo-way.

Comment 9 Dimitry Andric 2021-05-05 13:40:04 UTC

(In reply to Evgeniy Khramtsov from comment #8)
I'm unsure if mesa can "just" compile with different LLVM versions than what it was released for, as it may be that they depend on specific APIs.

The LLVM_DEFAULT=  10 in graphics/mesa-dri/Makefile.common was put there on 2020-10-12 here: https://cgit.freebsd.org/ports/commit/?id=57ce68a0c0cab56c3d08120e6f55c11a8c248f06

Note that it says "=" not "?=" which must be for a reason. :)

In any case, I can at least confirm that LTO works fine for www/firefox, if I set DEFAULT_VERSIONS+=llvm=11.

I think it would be handy if in general the LLVM versions for firefox and rust are kept the same?

(In reply to Dimitry Andric from comment #9)

I don't feel like hijacking the topic of this bug for discussion of x11@ policy, but I think I should clarify:

> if mesa can compile

Upstream accepts LLVM versions >= 8.0.0 https://gitlab.freedesktop.org/mesa/mesa/-/blob/mesa-20.2.3/meson.build#L1449-1455

For example, Arch Linux already builds against LLVM 11 (see llvm-libs version): https://archlinux.org/packages/extra/x86_64/mesa/

The current LLVM 10 for mesa-libs is outdated.

> Note that it says "=" not "?=" which must be for a reason. :)

x11@ favors apparent stability and the use of binary packages.

They also think that putting artificial limitations like non-overridable LLVM version for people who build from source is a good idea, maybe because handling possible reports is too hard. Not to mention the time and carbon footprint of the hardcoded LLVM version.

(In reply to Evgeniy Khramtsov from comment #10)

Note: Arch also built against 11 when mesa was at 20. This one is important.

Comment 12 Jan Beich 2021-05-05 15:02:35 UTC

(In reply to Dimitry Andric from comment #9)
> I think it would be handy if in general the LLVM versions for firefox and rust are kept the same?

Without enabling LTO by default it'd double (or triple as LTO builds are longer) the amount of pre-commit testing. ports/ committers have to test changes on their own hardware as poudriere requires root which universe* machines[1] don't provide. Without poudriere[2] testing is fragile and hard to automate, so would likely trip on complex desktop ports with many dependencies (like www/firefox). Some folks experimented[3] with offloading testing to third-party CI.

For example, SNDIO used to break often until it was enabled by default.

[1] https://www.freebsd.org/internal/machines/
[2] https://people.freebsd.org/~dinoex/howto/README.txt
[3] https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=comment:pipelines+status:closed

Comment 13 Jan Beich 2021-05-05 15:14:19 UTC

Kinda forgot, LLVM_DEFAULT desync can be caused by either gecko@ bumping too early and rust@ forgetting to bump.

Comment 14 commit-hook 2021-05-05 15:47:34 UTC

A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=b1c90afe23f93698805a22802f2dd02acbe4488a

commit b1c90afe23f93698805a22802f2dd02acbe4488a
Author:     Jan Beich <jbeich@FreeBSD.org>
AuthorDate: 2021-05-05 15:19:58 +0000
Commit:     Jan Beich <jbeich@FreeBSD.org>
CommitDate: 2021-05-05 15:46:08 +0000

    gecko: wait for rust before swiching llvm* next time

    Cross-language LTO in www/firefox requires devel/llvm* and bundled
    LLVM in lang/rust to be of the same major version. Adjust the comment
    to delay future bumps but until LTO is enabled desync can still happen.

    PR:             255359
    Suggested by:   dim

 Mk/bsd.gecko.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 15 Jan Beich 2021-05-05 15:52:29 UTC

Can someone check LTO build after applying review D30099? Don't forget to remove DEFAULT_VERSION+=llvm=11 workaround.

$ rustc -vV
rustc 1.52.0
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-freebsd
release: 1.52.0
LLVM version: 12.0.0   <--------- ATTENTION HERE

(In reply to Jan Beich from comment #13)

Yes, I've tried building now without llvm11 hack with LTO, wasn't successful.

LLVM ERROR: Broken module found, compilation aborted!
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
#0 0x0000000803159779 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/llvm12/lib/libLLVM-12.so+0x2959779)
#1 0x00000008031579b5 llvm::sys::RunSignalHandlers() (/usr/local/llvm12/lib/libLLVM-12.so+0x29579b5)
#2 0x000000080315a181 (/usr/local/llvm12/lib/libLLVM-12.so+0x295a181)
#3 0x000000080078fe10 (/lib/libthr.so.3+0x19e10)
clang++: error: unable to execute command: Abort trap (core dumped)
clang++: error: linker command failed due to signal (use -v to see invocation)
gmake[5]: *** [/usr/obj/usr/ports/www/firefox/work/firefox-88.0.1/config/rules.mk:545: libxul.so] Error 254
gmake[5]: Leaving directory '/usr/obj/usr/ports/www/firefox/work/.build/toolkit/library/build'
gmake[4]: *** [/usr/obj/usr/ports/www/firefox/work/firefox-88.0.1/config/recurse.mk:72: toolkit/library/build/target] Error 2
gmake[4]: Leaving directory '/usr/obj/usr/ports/www/firefox/work/.build'
gmake[3]: *** [/usr/obj/usr/ports/www/firefox/work/firefox-88.0.1/config/recurse.mk:34: compile] Error 2
gmake[3]: Leaving directory '/usr/obj/usr/ports/www/firefox/work/.build'
gmake[2]: *** [/usr/obj/usr/ports/www/firefox/work/firefox-88.0.1/config/rules.mk:355: all] Error 2
gmake[2]: Leaving directory '/usr/obj/usr/ports/www/firefox/work/.build'

Comment 17 Dimitry Andric 2021-05-06 11:19:24 UTC

(In reply to Jan Beich from comment #15)
How does review D30099 pull in LLVM 12 exactly? I don't see that detail anywhere in the patch. There doesn't seem to be any change to LLVM_DEFAULT, nor does the port have any dependency on another llvm port.

Maybe it's got its own vendored copy of llvm 12 embedded? That's awful of course, but probably unavoidable if they're doing incompatible changes to it.

(In reply to Jan Beich from comment #15)

I can't reproduce this with llvm=12 and new rust from this review. To mimic a more vanilla environment, I built firefox without DankBSD lightweight hardening and with X11 on. I can post a compressed poudriere log if you need this. Sorry for not posting yesterday, my build machine is kinda slow. The options were:

---Begin OPTIONS List---
===> The following configuration options are available for firefox-88.0.1,2:
     CANBERRA=off: Sound theme alerts
     DBUS=on: D-Bus IPC system support
     DEBUG=off: Build with debugging support
     FFMPEG=on: FFmpeg support (WMA, AIFF, AC3, APE...)
     LIBPROXY=off: Proxy support via libproxy
     LTO=on: Use Link-Time Optimization
     OPTIMIZED_CFLAGS=on: Use extra compiler optimizations
     PROFILE=off: Build with profiling support
     TEST=off: Build and/or run tests
====> Extra cubeb audio backends (OSS is always available)
     ALSA=off: ALSA audio architecture support
     JACK=off: JACK audio server support
     PULSEAUDIO=on: PulseAudio sound server support
     SNDIO=off: Sndio audio support
===> Use 'make config' to modify these settings
---End OPTIONS List---

LTO likely requires more memory to build, but I have 16 GB swap so the build passed. Note that profiling on may increase the requirements for an LTO build. The reporter should check if there are any out of memory messages in dmesg.

Comment 19 Charlie Li 2021-05-07 01:33:14 UTC

(In reply to Dimitry Andric from comment #17)
rust 1.52.0's copy/fork of LLVM is 12. Remember that we had an OPTION to use ports LLVM but the rust folks added a target-features bit that the gecko@ ports need to build, per bug 223342 and bug 223300.

review D30099 finally finished building on my machine, so will check LTO builds for both www/firefox and mail/thunderbird soon.

Comment 20 Charlie Li 2021-05-09 03:21:49 UTC

(In reply to Charlie Li from comment #19)
LTO with rust 1.52.0 and LLVM 12 works with www/firefox. The combination fails with mail/thunderbird for an unrelated reason (fails with rust 1.51.0 and LLVM 11 there).

Comment 21 Jan Beich 2021-05-11 18:11:49 UTC

Thanks for confirming. Fixed by ports d8837418ea74 and ensured by ports 6ec985b72d58.

Comment 22 Dimitry Andric 2021-05-16 15:24:52 UTC

Hmm, after that rust update I get:

% make -C /share/dim/ports/devel/rust-cbindgen
===>  License MPL20 accepted by the user
===>   rust-cbindgen-0.19.0_1 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by rust-cbindgen-0.19.0_1 for building
===>  Extracting for rust-cbindgen-0.19.0_1
=> SHA256 Checksum OK for rust/crates/cbindgen-0.19.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/ansi_term-0.11.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/atty-0.2.14.tar.gz.
=> SHA256 Checksum OK for rust/crates/autocfg-1.0.1.tar.gz.
=> SHA256 Checksum OK for rust/crates/bitflags-1.2.1.tar.gz.
=> SHA256 Checksum OK for rust/crates/cfg-if-0.1.10.tar.gz.
=> SHA256 Checksum OK for rust/crates/clap-2.33.3.tar.gz.
=> SHA256 Checksum OK for rust/crates/cloudabi-0.0.3.tar.gz.
=> SHA256 Checksum OK for rust/crates/getrandom-0.1.15.tar.gz.
=> SHA256 Checksum OK for rust/crates/hashbrown-0.9.1.tar.gz.
=> SHA256 Checksum OK for rust/crates/heck-0.3.1.tar.gz.
=> SHA256 Checksum OK for rust/crates/hermit-abi-0.1.16.tar.gz.
=> SHA256 Checksum OK for rust/crates/indexmap-1.6.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/itoa-0.4.6.tar.gz.
=> SHA256 Checksum OK for rust/crates/lazy_static-1.4.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/libc-0.2.77.tar.gz.
=> SHA256 Checksum OK for rust/crates/lock_api-0.3.4.tar.gz.
=> SHA256 Checksum OK for rust/crates/log-0.4.11.tar.gz.
=> SHA256 Checksum OK for rust/crates/parking_lot-0.10.2.tar.gz.
=> SHA256 Checksum OK for rust/crates/parking_lot_core-0.7.2.tar.gz.
=> SHA256 Checksum OK for rust/crates/ppv-lite86-0.2.9.tar.gz.
=> SHA256 Checksum OK for rust/crates/proc-macro2-1.0.21.tar.gz.
=> SHA256 Checksum OK for rust/crates/quote-1.0.7.tar.gz.
=> SHA256 Checksum OK for rust/crates/rand-0.7.3.tar.gz.
=> SHA256 Checksum OK for rust/crates/rand_chacha-0.2.2.tar.gz.
=> SHA256 Checksum OK for rust/crates/rand_core-0.5.1.tar.gz.
=> SHA256 Checksum OK for rust/crates/rand_hc-0.2.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/redox_syscall-0.1.57.tar.gz.
=> SHA256 Checksum OK for rust/crates/remove_dir_all-0.5.3.tar.gz.
=> SHA256 Checksum OK for rust/crates/ryu-1.0.5.tar.gz.
=> SHA256 Checksum OK for rust/crates/scopeguard-1.1.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/serde-1.0.116.tar.gz.
=> SHA256 Checksum OK for rust/crates/serde_derive-1.0.116.tar.gz.
=> SHA256 Checksum OK for rust/crates/serde_json-1.0.57.tar.gz.
=> SHA256 Checksum OK for rust/crates/serial_test-0.5.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/serial_test_derive-0.5.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/smallvec-1.4.2.tar.gz.
=> SHA256 Checksum OK for rust/crates/strsim-0.8.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/syn-1.0.41.tar.gz.
=> SHA256 Checksum OK for rust/crates/tempfile-3.1.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/textwrap-0.11.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/toml-0.5.6.tar.gz.
=> SHA256 Checksum OK for rust/crates/unicode-segmentation-1.6.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/unicode-width-0.1.8.tar.gz.
=> SHA256 Checksum OK for rust/crates/unicode-xid-0.2.1.tar.gz.
=> SHA256 Checksum OK for rust/crates/vec_map-0.8.2.tar.gz.
=> SHA256 Checksum OK for rust/crates/wasi-0.9.0+wasi-snapshot-preview1.tar.gz.
=> SHA256 Checksum OK for rust/crates/winapi-0.3.9.tar.gz.
=> SHA256 Checksum OK for rust/crates/winapi-i686-pc-windows-gnu-0.4.0.tar.gz.
=> SHA256 Checksum OK for rust/crates/winapi-x86_64-pc-windows-gnu-0.4.0.tar.gz.
===>  Moving crates to /wrkdirs/share/dim/ports/devel/rust-cbindgen/work/cbindgen-0.19.0/cargo-crates
===>  Patching for rust-cbindgen-0.19.0_1
===>   rust-cbindgen-0.19.0_1 depends on package: rust>=1.52.1 - found
===>  Configuring for rust-cbindgen-0.19.0_1
*** Signal 11

Stop.
make: stopped in /share/dim/ports/devel/rust-cbindgen

Has anybody seen this? Maybe it could be solved by rebuilding all my ports yet again... :/

(In reply to Dimitry Andric from comment #22)

Building ports outside of poudriere is fragile: ports build behavior can vary depending on environment (packages installed, configurations, variables, etc). Building outside of poudriere doesn't ensure that port consumers will rebuild if OPTIONS or PKGVERSION change.