Bug 260005 - devel/wasi-libcxx: not built to spec and not really useable
Summary: devel/wasi-libcxx: not built to spec and not really useable
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Christoph Moench-Tegeder
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2021-11-23 21:02 UTC by Christoph Moench-Tegeder
Modified: 2021-11-25 17:24 UTC (History)
2 users (show)

See Also:
greg: maintainer-feedback+


Attachments
disable exceptions and fix more feature flags in wasi-libcxx (4.71 KB, patch)
2021-11-23 21:02 UTC, Christoph Moench-Tegeder
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Christoph Moench-Tegeder freebsd_committer 2021-11-23 21:02:24 UTC
Created attachment 229684 [details]
disable exceptions and fix more feature flags in wasi-libcxx

platform is FreeBSD 13.0-RELEASE-p5, amd64, poudriere.

Trying to use wasi-libcxx (e.g. when building firefox beta with web assembly sandbox) does not work:

  wasm-ld: error: /usr/local/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.o): undefined symbol: __cxa_allocate_exception
  wasm-ld: error: /usr/local/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.o): undefined symbol: __cxa_throw

wasi-sdk doesn't support exception, so more diagnosis:
wasi-libcxx was built with exceptions, there is no "-fno-exceptions" in the build log, even as the port's Makefile specifies "-DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF". Digging further, I find that wasi-libcxx's CMakeFiles report

  -- Performing Test LIBCXX_SUPPORTS_FNO_EXCEPTIONS_FLAG
  -- Performing Test LIBCXX_SUPPORTS_FNO_EXCEPTIONS_FLAG - Failed

among others like that (e.g. "LIBCXX_SUPPORTS_WALL_FLAG - Failed"). More digging ensues.

=> First:
CMakeFiles/CMakeErrors.log for libcxxabi reports

  Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
  Compiler: /usr/bin/c++

That's the wrong compiler - base clang (11.0 in 13) does not know about "--target=wasm32-wasi".
The reason is that this cmake was run in post-configure under CONFIGURE_ENV, which does not contain CC/CXX/etc., that's only in MAKE_ENV by default (see any poudrier log, all the environments are listed right at the top).
Anyways, that results in many of the compilier feature Tests failing, as the compiler does not know about the target.

=> Second:
Even when using the correct compiler, many feature Tests still fail:

  wasm-ld: error: cannot open crt1.o: No such file or directory
  wasm-ld: error: unable to find library -lc++
  wasm-ld: error: unable to find library -lc++abi
  wasm-ld: error: unable to find library -lc

Well duh, we're getting into chicken-and-egg territory here.
The very ugly workaround is to just pass the Test results as flags to cmake, which will then do the right thing (for at least some cases).

Attached patch will:
- bump PORTREVISION
- put CC/CFLAGS/CXX/CXXFLAGS into CONFIGURE_ENV
- pass the Test results into cmake (I'm being generous with the flags here, and I hope I got it all correct - but then "it works for me").

Another nitpick: poudriere complains about

  pkg-static: DEVELOPER_MODE: Error: arch "FreeBSD:13:*" -- package installs architecture specific files

but I'm not touching that right now, between this and firefox beta (it builds!) I've enough for today.
Comment 1 Fernando Apesteguía freebsd_committer 2021-11-24 06:58:57 UTC
^Triage: Reporter is committer, assign accordingly.

Q/A:  Makefile: [34]: use a tab (not space) after a variable name
 Makefile: [35]: use a tab (not space) after a variable name
 Makefile: CFLAGS/CXXFLAGS are not needed in CONFIGURE_ENV as they are already added there in bsd.port.mk.

Thanks!
Comment 2 Greg V 2021-11-24 12:07:56 UTC
Patch looks fine. I guess I haven't caught the chicken-and-egg situation because I haven't really used a version built in a clean environment, sorry.

> pkg-static: DEVELOPER_MODE: Error: arch "FreeBSD:13:*" -- package installs architecture specific files

Well, pkg has no idea about what wasm is, but it likes to detect ELFs and ar archives and blindly assume they "have to" be specific to the *CPU* architecture rather than something portable like wasm.

Reported to pkg: https://github.com/freebsd/pkg/issues/2008

BTW initially it actually complained about *malformed* libraries even: https://github.com/freebsd/pkg/issues/1892
Comment 3 Christoph Moench-Tegeder freebsd_committer 2021-11-24 12:23:55 UTC
(In reply to Fernando Apesteguía from comment #1)
> Makefile: CFLAGS/CXXFLAGS are not needed in CONFIGURE_ENV as they are already added there in bsd.port.mk.

I beg to differ:
poudriere logs (before patch)
  --CONFIGURE_ENV--
  XDG_DATA_HOME=/wrkdirs/usr/ports/devel/wasi-libcxx/work
  XDG_CONFIG_HOME=/wrkdirs/usr/ports/devel/wasi-libcxx/work
  XDG_CACHE_HOME=/wrkdirs/usr/ports/devel/wasi-libcxx/work/.cache
  HOME=/wrkdirs/usr/ports/devel/wasi-libcxx/work
  TMPDIR="/tmp"
  PATH=/wrkdirs/usr/ports/devel/wasi-libcxx/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
  SHELL=/bin/sh
  CONFIG_SHELL=/bin/sh

and that, per my analysis, is causing some of the problems here (I could be wrong, but then we'd need to find out what else is happening).
Comment 4 Fernando Apesteguía freebsd_committer 2021-11-24 12:38:10 UTC
(In reply to Christoph Moench-Tegeder from comment #3)

You are right. Those variables are added to CONFIGURE_ENV only if HAS_CONFIGURE is set. Sorry for the noise.
Comment 5 commit-hook freebsd_committer 2021-11-24 21:14:20 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=bf143897d312fc861061eeca33d4cf3e777b6728

commit bf143897d312fc861061eeca33d4cf3e777b6728
Author:     Christoph Moench-Tegeder <cmt@FreeBSD.org>
AuthorDate: 2021-11-24 20:58:57 +0000
Commit:     Christoph Moench-Tegeder <cmt@FreeBSD.org>
CommitDate: 2021-11-24 20:58:57 +0000

    devel/wasi-libcxx: actually disable exceptions

    This port was accidentially using the base system compiler for some
    parts of the configure stages - which gives wrong results in the
    compiler feature tests, as base clang on FreeBSD 13.0 (llvm 11) does
    not know about the wasm32 target.
    Worse, even with the correct compiler some of the feature tests depend
    on a present and usable libc++ for the wasi target, which we don't
    have yet (this port will build one, but before...).
    The end result was that the build system failed to figure out the
    compiler flags for disabling exceptions (-fno-exceptions in clang's
    case) and built the wasm libc++ with exceptions enabled. But exceptions
    are not (yet) supported in wasi-sdk, so trying to build any code against
    this wasm libc++ failed with linker errors like
      wasm-ld: error: /usr/local/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.o): undefined symbol: __cxa_allocate_exception
      wasm-ld: error: /usr/local/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.o): undefined symbol: __cxa_throw

    To solve that, we have to force the configure stages to use the correct
    compiler by pushing CC/CXX and related variables into CONFIGURE_ENV
    and specify the results of the compiler feature tests via cmake
    arguments. That is a horrible workaround, but short of a full bootstrap
    build, I can't see any other solution to the chicken-and-egg problem
    with wasi-libcxx requiring a functional wasi-libc++ to build.

    PR:             260005
    Approved by:    greg at unrelenting dot technology (maintainer)

 devel/wasi-libcxx/Makefile | 75 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 74 insertions(+), 1 deletion(-)
Comment 6 Christoph Moench-Tegeder freebsd_committer 2021-11-24 21:16:07 UTC
committed, thanks.
Comment 7 Christoph Moench-Tegeder freebsd_committer 2021-11-24 21:17:42 UTC
I will need wasi-libc and wasi-libcxx version 13 for firefox 95, but 2021Q4 has only version 11. Do you know if that's safe to cherry-pick over?
Comment 8 Greg V 2021-11-25 17:24:39 UTC
(In reply to Christoph Moench-Tegeder from comment #7)
Why wouldn't it be? So far these ports do not have any dependents, etc.