Bug 263798

Summary: lang/rust - gecko ports with LTO enabled are dysfunctional
Product: Ports & Packages Reporter: Jakub Šebek <jamertwo>
Component: Individual Port(s)Assignee: FreeBSD Rust Team <rust>
Status: Open ---    
Severity: Affects Some People CC: cmt, drtr0jan, freebsd, gnuserland, grahamperrin, jwb, mi, mmance, osidorkin, salvadore, sm, tomek, wolfgang
Priority: --- Flags: bugzilla: maintainer-feedback? (gecko)
Version: Latest   
Hardware: amd64   
OS: Any   

Description Jakub Šebek 2022-05-05 15:12:55 UTC
After upgrading to [100.0_1,2], I am unable to run firefox or thunderbird. Running `firefox --safe-mode` yields a "Trace/BPT trap (core dumped)". Previously it was logging `[Child][PCompositorManagerChild] Error: SendAndWait(msgname=PWebRenderBridge::Msg_EnsureConnected) Channel error: cannot send/recv` too. I haven't debugged ff in the past, any other useful info I can provide?
Comment 1 freebsd 2022-05-05 17:28:11 UTC
I see the same thing, so the Importance field of this bug should be at least "Affects some people".
Comment 2 Jason W. Bacon freebsd_committer freebsd_triage 2022-05-05 18:09:01 UTC
Same here.

Thunderbird also crashes on startup.

Falkon works fine.

I tried "mv .mozilla mozilla.save" before starting firefox, in case it's an issue with old config files, but it made no difference.

This happens on my ThinkCenter M92P-Tiny, my ThinkPad T430, and in a VirtualBox guest, all running 13.0-RELEASE.
Comment 3 Jakub Šebek 2022-05-05 18:19:40 UTC
Solved: This has actually been fixed a few days ago in 100.0_3,2 - A result of buggy LLVM, buggy Rust and buggy package repos that haven't built the patch for latest FreeBSD:13:amd64, but they did for quarterly. Be better.
Comment 4 freebsd 2022-05-05 19:07:20 UTC
Can confirm, I just spent a bit of time compiling firefox from ports, it works again.
Comment 5 Jason W. Bacon freebsd_committer freebsd_triage 2022-05-05 20:47:05 UTC
FreeBSD moray.acadix  bacon ~ 952: lldb -c firefox.core /usr/local/bin/firefox 
(lldb) target create "/usr/local/bin/firefox" --core "firefox.core"
Core file '/home/bacon/firefox.core' (x86_64) was loaded.

(lldb) bt
* thread #1, name = 'glean.ping_director', stop reason = signal SIGTRAP
  * frame #0: 0x000000080b23771c libxul.so
    frame #1: 0x000000080e60ff20
    frame #2: 0x000000080b1d869a libxul.so`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4c15c0a441e53c93 + 138
    frame #3: 0x0000000806286fa3 libxul.so`std::sys::unix::thread::Thread::new::thread_start::hb718d4d977950b4a + 35
    frame #4: 0x000000080123982b libthr.so.3`___lldb_unnamed_symbol11$$libthr.so.3 + 331
Comment 6 Jason W. Bacon freebsd_committer freebsd_triage 2022-05-05 20:48:04 UTC
FreeBSD moray.acadix  bacon ~ 954: lldb -c thunderbird.core /usr/local/bin/thunderbird 
(lldb) target create "/usr/local/bin/thunderbird" --core "thunderbird.core"
Core file '/home/bacon/thunderbird.core' (x86_64) was loaded.

(lldb) bt
* thread #1, name = 'SwComposite', stop reason = signal SIGTRAP
  * frame #0: 0x000000080ad94a77 libxul.so
    frame #1: 0x000000080525d476 libxul.so
    frame #2: 0x000000080abc5d2a libxul.so`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hde4974977d69c0bf + 122
    frame #3: 0x000000080627c7d3 libxul.so`std::sys::unix::thread::Thread::new::thread_start::hb718d4d977950b4a + 35
    frame #4: 0x000000080123482b libthr.so.3`___lldb_unnamed_symbol11$$libthr.so.3 + 331
Comment 7 Jason W. Bacon freebsd_committer freebsd_triage 2022-05-05 20:50:58 UTC
Ooops, didn't see the replies before uploading the backtraces.  I'll try building from source and report back.
Comment 8 Santiago Martinez 2022-05-06 11:34:20 UTC
Hi gents, i see the same for firefox. 


smartinez@tucho /u/h/smartinez> lldb -c firefox.core firefox
(lldb) target create "firefox" --core "firefox.core"
Core file '/usr/home/smartinez/firefox.core' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'SwComposite', stop reason = signal SIGTRAP
  * frame #0: 0x00002fce507f3737 libxul.so
    frame #1: 0x00002fce4aca3df2 libxul.so
    frame #2: 0x00002fce5062481a libxul.so`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc4e0714f2e52ea49 + 122
    frame #3: 0x00002fce4bc86fa3 libxul.so`std::sys::unix::thread::Thread::new::thread_start::hb718d4d977950b4a + 35
    frame #4: 0x00002fce1ad6f83a libthr.so.3`thread_start(curthread=0x00002fcef8290a00) at thr_create.c:292:16
Comment 9 Christoph Moench-Tegeder freebsd_committer freebsd_triage 2022-05-06 11:39:56 UTC
fixed in firefox-100.0_3,2 firefox-esr-91.9.0_2,1 thunderbird-91.9.0_3 - see commit ports fc0a979a30ea
Comment 10 Jason W. Bacon freebsd_committer freebsd_triage 2022-05-06 14:47:52 UTC
Confirmed that building from source works here as well.
Comment 11 Tomasz "CeDeROM" CEDRO 2022-05-06 17:06:01 UTC
Thank you for the quick fix in the ports! Rebuilding updated port fixes the issue :-)

However I also notice functional problems with this Firefox on various sites, like hanging after login, not following the internal redirections and updates, etc. Probably some other internals also changed and there is a silent compatibility problem. Removing site data (cookies and local store) fixes the issue.

Thank you :-)
Comment 12 wolfgang 2022-05-07 11:23:22 UTC
I still got this error on firefox-100.0_3,2 until I recompiled it with LTO disabled.
Comment 13 Tomasz "CeDeROM" CEDRO 2022-05-07 12:04:33 UTC
Yes, as described in the patch commit message, LTO needs to be disabled.

After updating port it is good to `make rmconfig` to make sure default config is used. It is disabled now as default.
Comment 14 Christoph Moench-Tegeder freebsd_committer freebsd_triage 2022-05-08 12:33:35 UTC
*** Bug 263800 has been marked as a duplicate of this bug. ***
Comment 15 Oleg Sidorkin 2022-05-12 11:30:03 UTC
Rust switched to LLVM14 in 1.60, so linker from llvm13 that links firefox binary doesn't know how to work with llvm14 libs.

I've built firefox with devel/llvm14 and LTO, and it works (I patched bsd.gecko.mk and created wasi-compiler14 port).
Comment 16 Marc Mance 2022-06-09 14:47:32 UTC
I recently upgraded as well and could not open firefox.  I removed my configs and now I can use firefox.  It is still configured with LTO.

When I went to use thunderbird, it also would crash on start.  I found Bug 263800 and configured thunderbird without LTO, but it gives me the same crash.

❯ thunderbird
[calBackendLoader] Using Thunderbird's libical backend
[LDAPModuleLoader] Using LDAPDirectory.jsm
[MsgSendModuleLoader] Using MessageSend.jsm
[SmtpModuleLoader] Using SmtpService.jsm
console.error: (new Error("couldn't open library libc.so: /usr/lib/libc.so: invalid file format", "resource:///modules/CLib.jsm", 46))
Exiting due to channel error.
[1]    43809 trace trap  thunderbird
Comment 17 Marc Mance 2022-06-09 15:02:30 UTC
It did not remove LTO last time (user error?).  I removed the option and now it works.
Comment 18 Mikhail Teterin freebsd_committer freebsd_triage 2022-06-10 15:08:40 UTC
(In reply to Oleg Sidorkin from comment #15)
> Rust switched to LLVM14 in 1.60, so linker from llvm13 that links firefox binary doesn't know how to work with llvm14 libs.

Thanks for the investigation, Oleg. This seems like something someone ought to report to LLVM project -- the 13.x linker either ought to do "the right thing", or die with a noisy error :(

Meanwhile, in my humble opinion, lang/rust ought to use the ports-provided llvm -- the ongoing flip-flopping between that and using the bundled version needs to stop.

If llvm14 is really mandatory for the newer Rust versions, then lang/llvm14  ought to become a dependency. But the ci.yml files inside Rust's sources indicate llvm-12 as the minimum requirement, so users ought to be able to use those already installed older versions too.

This would reduce build-times significantly and eliminate the mixing of toolchains, which is a bad idea in general and, according to Oleg's investigation, is the actual source of problem in this case in particular.

(In reply to Christoph Moench-Tegeder from comment #9)
> fixed in firefox-100.0_3,2 firefox-esr-91.9.0_2,1 thunderbird-91.9.0_3 - see commit ports fc0a979a30ea

Disabling an otherwise valid option is not a fix, it is a work-around :( I understand, that gecko-maintainers cannot do much more than that -- except to, maybe, set BROKEN for those, who have LTO explicitly turned on in the options already (so we don't waste hours recompiling the heavy ports)...
Comment 19 Graham Perrin freebsd_committer freebsd_triage 2023-04-23 15:31:35 UTC
(In reply to Mikhail Teterin from comment #18)

salvadore@ please, do you have any thoughts on this? 

(Keyword: llvm)
Comment 20 Lorenzo Salvadore freebsd_committer freebsd_triage 2023-04-25 10:18:52 UTC
(In reply to Graham Perrin from comment #19)

I am not a maintainer for any of the ports involved in this bug report and I do not know much about them, but since you ask I am happy to give my opinion: I agree with Mikhail Teterin, using an existing llvm port as dependency rather than a bundled version would be much better. But if it has not been done until now, there must be a reason. I have no idea which one is it.