Bug 251191

Summary: Mk/Uses/ninja.mk: 'USES=cmake emacs' results in build error with ninja
Product: Ports & Packages Reporter: Yasuhiro Kimura <yasu>
Component: Individual Port(s)Assignee: Port Management Team <portmgr>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: adridg, kde, lwhsu, tcberner
Priority: --- Flags: yasu: exp-run?
Version: Latest   
Hardware: Any   
OS: Any   
URL: https://reviews.freebsd.org/D27494
Attachments:
Description Flags
Patch file
none
Updated patch file
none
Updated patch file none

Description Yasuhiro Kimura freebsd_committer freebsd_triage 2020-11-16 17:50:17 UTC
Created attachment 219737 [details]
Patch file

1. Summary

'USES=cmake emacs' results in build error with ninja. This is bug of ports build system. Attached patch fixes it.

2. Background

I'm trying to create port of following software.

libegit2: Emacs bindings for libgit2
https://github.com/magit/libegit2

3. What is the problem?

To port libegit2 following 2 steps are necessary.

a. Build libegit2.so with cmake and install it to ${PREFI}/lib
b. Byte-compile libgit.el and install them to
   ${PREFIX}/${EMACS_VERSION_SITE_LISPDIR}

So I need to add 'USES=cmake emacs' in Makefile. But it causes error of ninja such as following.

----------------------------------------------------------------------
===>  License GPLv3+ accepted by the user
===>   libegit2-emacs27_nox-0.0.1.20200316 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by libegit2-emacs27_nox-0.0.1.20200316 for building
===>  Extracting for libegit2-emacs27_nox-0.0.1.20200316
=> SHA256 Checksum OK for magit-libegit2-0.0.1.20200316-0ef8b13_GH0.tar.gz.
===>  Patching for libegit2-emacs27_nox-0.0.1.20200316
===>  Applying FreeBSD patches for libegit2-emacs27_nox-0.0.1.20200316 from /usr0/freebsd/ports/git/devel/libegit2/files
===>   libegit2-emacs27_nox-0.0.1.20200316 depends on file: /usr/local/bin/cmake - found
===>   libegit2-emacs27_nox-0.0.1.20200316 depends on executable: ninja - found
===>   libegit2-emacs27_nox-0.0.1.20200316 depends on file: /usr/local/bin/emacs-27.1 - found
===>   libegit2-emacs27_nox-0.0.1.20200316 depends on package: pkgconf>=1.3.0_1 - found
===>   libegit2-emacs27_nox-0.0.1.20200316 depends on shared library: libgit2.so - found (/usr/local/lib/libgit2.so)
===>  Configuring for libegit2-emacs27_nox-0.0.1.20200316
===>  Performing out-of-source build
/bin/mkdir -p /usr0/freebsd/ports/work/usr0/freebsd/ports/git/devel/libegit2/work-nox/.build
-- The C compiler identification is Clang 10.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is Clang 10.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: pkgconf (found version "1.7.3") 
-- Checking for module 'libgit2'
--   Found libgit2, version 1.0.1
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_FLAGS_DEBUG
    CMAKE_CXX_FLAGS_RELEASE
    CMAKE_EXE_LINKER_FLAGS
    CMAKE_MODULE_LINKER_FLAGS
    CMAKE_VERBOSE_MAKEFILE
    THREADS_HAVE_PTHREAD_ARG


-- Build files have been written to: /usr0/freebsd/ports/work/usr0/freebsd/ports/git/devel/libegit2/work-nox/.build
===>  Building for libegit2-emacs27_nox-0.0.1.20200316
ninja: error: unknown target 'EMACS=/usr/local/bin/emacs-27.1'
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make: stopped in /usr0/freebsd/ports/git/devel/libegit2

Command exit status: 1
----------------------------------------------------------------------

I tried to build libegit2 according to the steps described in its document and it succeeded without any error. Moreover I selected some ports that use 'USES=cmake', intentionally change them to 'USES=cmake emacs' and tried to build them. Then build failed with same error for all of them. So this is bug of ports build system.

4. Why the problem happens?

There is following line in Mk/Uses/emacs.mk

----------------------------------------------------------------------
MAKE_ARGS+=	EMACS=${EMACS_CMD}
----------------------------------------------------------------------

So if both 'cmake' and 'emacs' is added to USES, then ninja is invoked with something like `ninja -v EMACS=/usr/local/bin/emacs-27.1`. But according to the output of `ninja --help`, ninja doesn't accept such argument as 'NAME=VALUE' that `make` accepts. So it results in the error that happens when I try to build my devel/libegit2 port.

I checked Mk/Uses/*.mk and found following ones also add extra arguments to MAKE_ARGS.

* Mk/Uses/gnustep.mk
* Mk/Uses/scons.mk
* Mk/Uses/waf.mk

Moreover each port that uses 'USES=cmake' may do same thing. Therefore this problem should be fixed by changing Mk/Uses/ninja.mk.

5. Solution

In Mk/bsd.port.mk MAKE_ARGS is used in following targets.

* do-build
* do-install
* do-test

So the solution is to re-define them in Mk/Uses/ninja.mk. As for do-install and do-test, DO_MAKE_BUILD and DO_MAKE_TEST are defined and used when each target is called respectively. So re-define them in ninja.mk so ninja is directly invoked without using MAKE_ARGS. As for do-install DO_MAKE_INSTALL isn't defined. So re-define the target with same way in ninja.mk.

As the result re-definition of following variables in ninja.mk gets unnecessary.

* MAKE_ARGS
* MAKE_CMD
* MAKE_FLAGS
* MAKE_JOBS
* MAKEFILE

So remove them.

Additionally define NINJA_ARGS whose default value is empty and add it to argument of executing ninja. It is intended that some ports may want to add extra argument when executing ninja.

6. Test

Following branch includes both devel/libegit2 port and this fix.

https://github.com/yasuhirokimura/freebsd-ports/tree/libegit2/
(Note: This branch will be removed once devel/libegit2 is committed.)

It can be build without any error. I also confirmed some ports that uses 'USES=cmake' can be built successfully with attached patch. But it should be tested more with exp-run.
Comment 1 Li-Wen Hsu freebsd_committer freebsd_triage 2020-11-16 17:56:24 UTC
Over to maintainer
Comment 2 Tobias C. Berner freebsd_committer freebsd_triage 2020-11-16 18:47:29 UTC
Moin moin 

You could try cmake:noninja in the meantime.

https://svnweb.freebsd.org/ports/head/Mk/Uses/cmake.mk?revision=488341&view=markup#l10

This flag to cmake.mk was added specifically to disable usage of ninja.


mfg Tobias
Comment 3 Yasuhiro Kimura freebsd_committer freebsd_triage 2020-11-16 20:45:42 UTC
The problem is found. If a port uses 'USES=cmake' and also defines do-install target, then `make describe` produces such warning message as below.

make: "/usr0/freebsd/ports/head/sysutils/osquery/Makefile" line 78: warning: duplicate script for target "do-install" ignored
make: "/usr0/freebsd/ports/head/Mk/Uses/ninja.mk" line 29: warning: using previous script for "do-install" defined here

I'll investigate and update patch to fix it. So please suspend exp-run until new patch is unloaded.
Comment 4 Yasuhiro Kimura freebsd_committer freebsd_triage 2020-11-17 08:30:05 UTC
Created attachment 219766 [details]
Updated patch file

Fix warning messages with following changes.

* In Mk/bsd.port.mk define DO_MAKE_INSTALL and use it in do-install target.
* In Mk/Uses/ninja.mk re-define DO_MAKE_INSTALL so ninja is directly invoked.

So please do exp-run with attached patch instead of original one.
Comment 5 Antoine Brodin freebsd_committer freebsd_triage 2020-11-24 20:19:48 UTC
(In reply to Yasuhiro KIMURA from comment #4)

Can you open a review on https://reviews.freebsd.org/ ?
Comment 6 Yasuhiro Kimura freebsd_committer freebsd_triage 2020-12-06 22:32:57 UTC
Created attachment 220329 [details]
Updated patch file

Change custom targets of ports using ninja so they don't use MAKE_{ARGS,CMD,ENV}.
Comment 7 Yasuhiro Kimura freebsd_committer freebsd_triage 2020-12-06 22:46:38 UTC
(In reply to Antoine Brodin from comment #5)

Sorry for late response. I created following review.

https://reviews.freebsd.org/D27494
Comment 8 Adriaan de Groot freebsd_committer freebsd_triage 2022-01-24 23:08:39 UTC
This is very much bogged down in the linked review. I wonder if it is worth pursuing further, because it's *really hard* to get things right for multiple build systems, even as yasu@ has demonstrated that it is possible to get ninja and make straight.

To summarize, though:
- `USES=emacs` assumes that the build is going to use make, and sets make-relevant variables,
- `USES=ninja`, or `USES=cmake` (which implies ninja), switch the build system to ninja which does not understand the make-relevant variables,
- and some ports do convoluted things with special targets in the ports Makefiles.

I wonder if it is feasible to split this into smaller steps?

- Introduce helpers that make it easier to switch build systems (make vs ninja) for specific kinds of commands. I think those are called `DO_MAKE_INSTALL` in the review, and similar.
- Avoid defining `do-*` targets in the files in Mk/Uses, to improve interoperability and safety (but maybe this is going to come down to a documentation issue).
- Then start switching over the problematic ports one-by-one.
Comment 9 Adriaan de Groot freebsd_committer freebsd_triage 2022-05-05 22:44:51 UTC
I'm going to close this since libegit2 has landed, there's a workaround, and fixing this "the right way" has gotten so bogged down. I believe it might be done easier if emacs.mk could detect that ninja is (or will be) used -- maybe it could check for USES=cmake and do something sensible for CMake instead, since overall it seems unlikely that a CMake-generated set of Makefiles would expect such a variable. (Indeed, the EMACS variable isn't used in the generated Makefiles)
Comment 10 commit-hook freebsd_committer freebsd_triage 2022-05-05 22:46:10 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=07ebebf97e7cc575bc90b37f5d46097179201285

commit 07ebebf97e7cc575bc90b37f5d46097179201285
Author:     Adriaan de Groot <adridg@FreeBSD.org>
AuthorDate: 2022-05-05 22:23:35 +0000
Commit:     Adriaan de Groot <adridg@FreeBSD.org>
CommitDate: 2022-05-05 22:45:34 +0000

    USES=cmake: be explicit about incompatibilities with ninja

    Document how `USES=gmake` and `USES=emacs` are incompatible with
    the default `USES=cmake` behavior of using ninja as generator.
    Using a naive `USES=cmake emacs` line used to get you a strange
    ninja error when ${MAKE_ARGS} is passed to it, now there is a
    meaningful error message beforehand.

    The linked PR examines ways to make it "just work", but that
    got bogged down. Do a partial fix so it is easier to track
    down next time.

    PR:             251191

 Mk/Uses/cmake.mk | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)