Bug 231411 - lang/rust: fails to build with non-default SSL library
Summary: lang/rust: fails to build with non-default SSL library
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: FreeBSD Rust Team
URL:
Keywords: regression
Depends on:
Blocks: 229826 230470
  Show dependency treegraph
 
Reported: 2018-09-16 22:17 UTC by rozhuk.im
Modified: 2018-09-21 11:16 UTC (History)
12 users (show)

See Also:
bugzilla: maintainer-feedback? (rust)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description rozhuk.im 2018-09-16 22:17:10 UTC
...
   Compiling serde_json v1.0.24
error: failed to run custom build command for `libssh2-sys v0.2.8`
process didn't exit successfully: `/tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/build/x86_64-unknown-freebsd/stage2-tools/release/build/libssh2-sys-2d66401c8012e623/build-script-build` (exit code: 101)
...
[ 40%] Building C object src/CMakeFiles/libssh2.dir/kex.c.o
gmake[4]: Leaving directory '/tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/build/x86_64-unknown-freebsd/stage2-tools/x86_64-unknown-freebsd/release/build/libssh2-sys-149db94635ec1367/out/build'
gmake[3]: Leaving directory '/tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/build/x86_64-unknown-freebsd/stage2-tools/x86_64-unknown-freebsd/release/build/libssh2-sys-149db94635ec1367/out/build'
gmake[2]: Leaving directory '/tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/build/x86_64-unknown-freebsd/stage2-tools/x86_64-unknown-freebsd/release/build/libssh2-sys-149db94635ec1367/out/build'

--- stderr
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS


/tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/src/vendor/libssh2-sys/libssh2/src/hostkey.c:526:10: error: use of undeclared identifier 'LIBSSH2_HOSTKEY_HASH_SHA256'
    case LIBSSH2_HOSTKEY_HASH_SHA256:
         ^
1 error generated.
gmake[4]: *** [src/CMakeFiles/libssh2.dir/build.make:141: src/CMakeFiles/libssh2.dir/hostkey.c.o] Error 1
gmake[4]: *** Waiting for unfinished jobs....
/tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/src/vendor/libssh2-sys/libssh2/src/channel.c:2262:44: error: use of undeclared identifier 'LIBSSH2_ERROR_CHANNEL_WINDOW_FULL'
            return _libssh2_error(session, LIBSSH2_ERROR_CHANNEL_WINDOW_FULL,
                                           ^
1 error generated.
gmake[4]: *** [src/CMakeFiles/libssh2.dir/build.make:89: src/CMakeFiles/libssh2.dir/channel.c.o] Error 1
...


grep -R "LIBSSH2_HOSTKEY_HASH_SHA256" /tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/
/tmp/ports/usr/ports/lang/rust/work/rustc-1.29.0-src/src/vendor/libssh2-sys/libssh2/include/libssh2.h:#define LIBSSH2_HOSTKEY_HASH_SHA256                         3


I have installed libssh2-1.8.0,3 and looks like rust include libssh2 .h file from local base instead rust source.
Comment 1 Charlie Li 2018-09-17 02:03:43 UTC
Interesting: https://github.com/alexcrichton/ssh2-rs/pull/88

Probably what happened here.
Comment 2 Jan Beich freebsd_committer 2018-09-17 02:24:56 UTC
The package cluster is unlikely to try before 2018-09-18 01:00 UTC but rust-1.29.0 builds fine for me on all supported releases/architectures:
- 10.4 amd64, see https://ptpb.pw/3-0-
- 10.4  i386, see https://ptpb.pw/NkD9
- 11.1 amd64, see https://ptpb.pw/705e
- 11.1  i386, see https://ptpb.pw/Blzu
- 11.2 amd64, see https://ptpb.pw/RmgF
- 11.2  i386, see https://ptpb.pw/zV5K
- 12.0 amd64, see https://ptpb.pw/NM9f
- 12.0  i386, see https://ptpb.pw/v65a

A few guesses without error log:
- lang/rust doesn't respect verbose build, so maybe C*FLAGS aren't respected thus USES=localbase (via USES=libedit) fails to convert -I to -isystem in order to make /usr/local/include appear last for include search.
- Maybe lang/rust respects C*FLAGS but due to a pilot error make.conf uses = instead of += which ends up clobbering values set by Mk/* including -isystem.
Comment 3 Vladimir Omelchuk 2018-09-17 10:25:54 UTC
I have the same error!

Part of log:
[...]
--- stderr
fatal: not a git repository (or any parent up to mount point /usr)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS

/usr/ports/lang/rust/work/rustc-1.29.0-src/src/vendor/libssh2-sys/libssh2/src/channel.c:2262:44: error: use of undeclared identifier 'LIBSSH2_ERROR_CHANNEL_WINDOW_FULL'
            return _libssh2_error(session, LIBSSH2_ERROR_CHANNEL_WINDOW_FULL,
                                           ^
1 error generated.
gmake[3]: *** [src/CMakeFiles/libssh2.dir/build.make:89: src/CMakeFiles/libssh2.dir/channel.c.o] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:91: src/CMakeFiles/libssh2.dir/all] Error 2
gmake[1]: *** [Makefile:152: all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', vendor/cmake/src/lib.rs:643:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

command did not execute successfully: "/usr/ports/lang/rust/work/rustc-1.29.0-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" "build" "--target" "x86_64-unknown-freebsd" "-j" "1" "--release" "--frozen" "--manifest-path" "/usr/ports/lang/rust/work/rustc-1.29.0-src/src/tools/cargo/Cargo.toml" "--features" "" "--message-format" "json"
expected success, got: exit code: 101
Traceback (most recent call last):
  File "/usr/ports/lang/rust/work/rustc-1.29.0-src/x.py", line 20, in <module>
    bootstrap.main()
  File "/usr/ports/lang/rust/work/rustc-1.29.0-src/src/bootstrap/bootstrap.py", line 843, in main
    bootstrap(help_triggered)
  File "/usr/ports/lang/rust/work/rustc-1.29.0-src/src/bootstrap/bootstrap.py", line 834, in bootstrap
    run(args, env=env, verbose=build.verbose)
  File "/usr/ports/lang/rust/work/rustc-1.29.0-src/src/bootstrap/bootstrap.py", line 148, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /usr/ports/lang/rust/work/rustc-1.29.0-src/build/bootstrap/debug/bootstrap build --verbose --config ./config.toml --jobs 1
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/rust
** Command failed [exit code 1]: /usr/bin/script -qa /tmp/portupgrade20180917-33283-1cbjzz6 env UPGRADE_TOOL=portupgrade UPGRADE_PORT=rust-1.27.1 UPGRADE_PORT_VER=1.27.1 make
** Fix the problem and try again.
** Listing the failed packages (-:ignored / *:skipped / !:failed)
	! lang/rust (rust-1.27.1)	(unknown build error)
[...]

<uname>
FreeBSD 11.2-RELEASE-p3 amd64
Comment 4 Rob Belics 2018-09-17 12:43:33 UTC
(In reply to Vladimir Omelchuk from comment #3)
Same for me.
Comment 5 Konstantin Belousov freebsd_committer 2018-09-17 15:32:50 UTC
I put the poudriere run log at https://www.kib.kiev.ua/kib/rust-1.29.0.log
Comment 6 Charlie Li 2018-09-17 19:11:04 UTC
Comparing my poudriere build log (12.0 amd64) to the package cluster, the only differences are my using ccache and ssl=libressl-devel compared to the package cluster using base OpenSSL.

As a test, I removed the security/libssh2 dependency and unset LIBSSH2 from ftp/curl so as to not have any system library to fall into. The build succeeds. This strongly suggests clashing between the bundled and system libraries, especially in the future, as the crate's upstream maintainer forked libssh2 itself:

https://github.com/alexcrichton/ssh2-rs/commit/4e3c4d84434f91b99fd2f57e66001ec6b2543703
Comment 7 Cy Schubert freebsd_committer 2018-09-17 19:33:03 UTC
(In reply to Charlie Li from comment #1)
Yes. This is the cause.

We need to remove the dependency on libssh2. Either way it is now redundant.

I suspect (I haven't looked at this) that the include path might need some attention too.
Comment 8 Konstantin Belousov freebsd_committer 2018-09-17 20:19:58 UTC
For me it fails to build with the default ports openssl.
Comment 9 Jan Beich freebsd_committer 2018-09-17 20:34:41 UTC
(In reply to Konstantin Belousov from comment #8)
USES=ssl defaults to in-base OpenSSL aka DEFAULT_VERSIONS+=ssl=base unless world is built WITHOUT_OPENSSL or one of the alternatives is already installed. See SSL_DEFAULT logic in Mk/bsd.default-versions.mk.
Comment 10 Jan Beich freebsd_committer 2018-09-18 00:39:14 UTC
The port doesn't respect C*FLAGS as MAKE_ENV is lost when x.py is run. OTOH, libssh2 fixed include order issue in https://github.com/libssh2/libssh2/commit/54bef4c5dad8#diff-95e351a3805a1dafa85bf20b81d086e6

Unlike Rust upstream (e.g., via rustup.rs) we do have control of both rust and libssh2 packages thus enforce the same libssl is used. Our libssh2 package maybe old (unlike libgit2) but rust still builds fine against it, see https://ptpb.pw/dNxy.

Maybe apply libssh2 fix for now then try to switch to LIBSSH2_SYS_USE_PKG_CONFIG=1 (and LIBGIT2_SYS_USE_PKG_CONFIG=1) after 2018Q4 branches. Why rust directly links against libssh2 when using system libgit2, anyway?
Comment 11 Craig Leres freebsd_committer 2018-09-18 02:22:24 UTC
I use ssl=openssl.

One thing I see is that removing security/libssh2 from LIB_DEPENDS is the right thing to do it's also a noop; rust depends on ftp/curl which depends on libssh2. 
I tried to figure out how to add -I${WRKSRC}/src/vendor/libssh2-sys/libssh2/include to CFLAGS for the build of src/vendor/libssh2-sys but I *really* don't understand cargo...
Comment 12 Charlie Li 2018-09-18 03:07:26 UTC
(In reply to Craig Leres from comment #11)
You also need to unset the LIBSSH2 option from ftp/curl in order to not have security/libssh2 in the environment. And I believe LIBSSH2 is unset by default.
Comment 13 Craig Leres freebsd_committer 2018-09-18 06:24:00 UTC
(In reply to Charlie Li from comment #12)
The rust port must build no matter what options the user selects for curl. This is definitely possible, we just need a cargo pilot explain how to add to CFLAGS when the rust-local libssh2 is built.
Comment 14 Cy Schubert freebsd_committer 2018-09-19 03:25:50 UTC
(In reply to Jan Beich from comment #10)
libssh2-devel port possibly as well?
Comment 15 Craig Leres freebsd_committer 2018-09-19 03:30:24 UTC
FYI, I opened a github issue for this issue:

    https://github.com/rust-lang/rust/issues/54340

I'm sure the fix is simple and it really belongs upstream (although I'd be happy with a patch that let the current version build).
Comment 16 Charlie Li 2018-09-20 04:33:57 UTC
(In reply to Craig Leres from comment #15)
Please close the github issue. It has nothing to do with the Rust language itself (which is where you opened the issue in); it has nothing to do with cargo; it has everything to do with the libssh2-sys crate.

I'm almost done working on a patch that will fix this problem for at least this Rust release in our ports tree. It turns out that the upstream maintainer for libssh2-sys, its dependencies and many more crates (and he's part of the Rust Core Team) has gotten fed up with using cmake in building the C/C++ parts and is jettisoning cmake at first opportunity:

https://github.com/alexcrichton/curl-rust/pull/225

Unfortunately for libssh2-sys, the cmake removal comes after a few previous commits and releases so we will have to update the crate wholesale. Its forward and reverse dependencies also specify newer crates than bundled in the rustc-1.29.0 tarball, so they are getting updated too. Couple that with how we non-negotiably forbid any network use during portbuild, and we have a recipe for a giant, nasty and probably unavoidable patch coming up.

So far with libssh2-sys sans cmake, the updated crate builds successfully during the portbuild. The new (shell) environment variable added and found in the bundled crate serves as a simple gatekeeper to using ports security/libssh2; the environment variable only needs to exist for the crate to use the ports library. The libgit2 crate has and originated this same bit, albeit with devel/libgit2. Side note, this is how the cargo/the cc crate receives CFLAGS, etc; rust couldn't care less about MAKE_ENV. The environment variables are passed as such before or with invoking cargo, or in this case x.py.

Patching the port like this won't be maintainable/sustainable in the long run. The crates bundled in the rustc tarball are often outdated by the time rust stable gets released, and we've been bitten multiple times in not-so-edge cases.
Comment 17 commit-hook freebsd_committer 2018-09-20 10:08:11 UTC
A commit references this bug:

Author: jbeich
Date: Thu Sep 20 10:07:38 UTC 2018
New revision: 480162
URL: https://svnweb.freebsd.org/changeset/ports/480162

Log:
  lang/rust: unbreak if ssl != base

    [ 13%] Building C object src/CMakeFiles/libssh2.dir/channel.c.o
    /usr/bin/cc -DLIBSSH2_CLEAR_MEMORY -DLIBSSH2_DH_GEX_NEW=1 -DLIBSSH2_HAVE_ZLIB=1 -DLIBSSH2_OPENSSL -I/usr/local/include -Isrc/vendor/libssh2-sys/libssh2/include -Ibuild/x86_64-unknown-freebsd/stage2-tools/x86_64-unknown-freebsd/release/build/libssh2-sys-149db94635ec1367/out/build/src  -ffunction-sections -fdata-sections -fPIC -ffunction-sections -fdata-sections -fPIC -m64 -m64 -O3 -DNDEBUG   -std=gnu90 -o CMakeFiles/libssh2.dir/channel.c.o   -c src/vendor/libssh2-sys/libssh2/src/channel.c
    src/vendor/libssh2-sys/libssh2/src/channel.c:2262:44: error: use of undeclared identifier 'LIBSSH2_ERROR_CHANNEL_WINDOW_FULL'
  	      return _libssh2_error(session, LIBSSH2_ERROR_CHANNEL_WINDOW_FULL,
  					     ^

  PR:		231411
  Reported by:	many
  Obtained from:	upstream (libssh2)
  Approved by:	portmgr blanket

Changes:
  head/lang/rust/files/patch-src_vendor_libssh2-sys_libssh2_src_CMakeLists.txt
Comment 18 Jan Beich freebsd_committer 2018-09-20 10:40:43 UTC
I've landed bandaid for now which maintains statu quo. Respecting *FLAGS or switching to system libraries have different rationales thus can be done later.

(In reply to Charlie Li from comment #16)
> Side note, this is how the cargo/the cc crate receives CFLAGS, etc;
> rust couldn't care less about MAKE_ENV. The environment variables
> are passed as such before or with invoking cargo, or in this case
> x.py.

How do you think ports pass CC/CFLAGS/CXX/CXXFLAGS/LD/LDFLAGS to vendor build system? MAKE_ENV isn't strictly for make-style builds e.g., see USES=cargo implementation.
Comment 19 Jean-Sébastien Pédron freebsd_committer 2018-09-20 10:45:08 UTC
Thank you for working on it and committing a fix!
Comment 20 Charlie Li 2018-09-20 10:51:13 UTC
(In reply to Jan Beich from comment #18)
In Uses/cargo.mk, the CARGO_CARGO_RUN helper invokes ${SETENV} ${MAKE_ENV} ${CARGO_ENV} on the same line as the cargo invocation itself. Without that part, MAKE_ENV, *FLAGS, etc are not acknowledged. This follows what the cargo documentation describes:

https://doc.rust-lang.org/cargo/reference/build-scripts.html#inputs-to-the-build-script
Comment 21 commit-hook freebsd_committer 2018-09-21 11:16:14 UTC
A commit references this bug:

Author: jbeich
Date: Fri Sep 21 11:16:08 UTC 2018
New revision: 480253
URL: https://svnweb.freebsd.org/changeset/ports/480253

Log:
  lang/rust: unbundle libgit2/libssh2

  - libgit2 in ports is nowadays newer than bundled version
  - libssh2 was already unbundled until 1.29.0 update
  - Unbundle libssh2 for USES=cargo (unused atm)

  PR:		231411
  Approved by:	rust (dumbbell, tobik)

Changes:
  head/Mk/Uses/cargo.mk
  head/lang/rust/Makefile