Created attachment 175704 [details] Proposed patch (since 421635 revision) Patch to make configurable verbose build output, with using NO_VERBOSE_BUILD define, e.g. NO_VERBOSE_BUILD=yes in /etc/make.conf or `make NO_VERBOSE_BUILD=yes` for concrete port to disable verbose build output or without it to enable. The ports r421635 changed defaults from compact (so called "silent") to verbose build output unconditionally. While changes for defaults are understandable, there are no objective restrictions to forbid usage of previous approach (i.e. compact build output), especially from point of view of backward compatibility, where Make (including configure scripts), CMake, Ninja build systems designed with compact build output by default, with ability to select verbose build as an option. Possible to change the name for "NO_VERBOSE_BUILD" variable and/or description of it. Some examples of compact and verbose build outputs: Compact ports-mgmt/pkg: http://beefy6.nyi.freebsd.org/data/101amd64-default/419462/logs/pkg-1.8.7_1.log Verbose ports-mgmt/pkg: http://beefy6.nyi.freebsd.org/data/101amd64-default/423739/logs/pkg-1.8.7_3.log Compact devel/llvm39: http://beefy6.nyi.freebsd.org/data/101amd64-default/421280/logs/llvm39-3.9.0.log Verbose devel/llvm39: http://beefy6.nyi.freebsd.org/data/101amd64-default/423487/logs/llvm39-3.9.0_2.log
Created attachment 175705 [details] Proposed patch for ports I also created a (preliminary) patch for ports, where verbose builds used. Undecided: https://svnweb.freebsd.org/ports/head/games/q2p/files/patch-Makefile?revision=361182&view=markup#l16 https://svnweb.freebsd.org/ports/head/math/speedcrunch/Makefile?revision=385637&view=markup#l18 For check (already in patch): https://svnweb.freebsd.org/ports/head/emulators/dolphin-emu/Makefile?revision=423032&view=markup#l64 https://svnweb.freebsd.org/ports/head/graphics/sekrit-twc-zimg/Makefile?revision=423586&view=markup#l50 https://svnweb.freebsd.org/ports/head/irc/kvirc/Makefile?revision=421956&view=markup#l48 https://svnweb.freebsd.org/ports/head/net/mpich2/Makefile?revision=412348&view=markup#l84
This was already discussed. We specifically want unconditional verbose builds.
Yes, like Dmitry said, we already talked about it. We *always* want the builds to be verbose. Always.
We also specifically don't want any extra complication in individual ports. But thanks for pointing out some remaining conditionals, these need to be removed.
Any build that chains make/gmake supports $ make -s ninja/waf may need a hack e.g., -MAKE_ARGS+= -v +MAKE_ARGS+= ${ECHO:Ntrue:S/echo/-v/} (In reply to Dmitry Marakasov from comment #2) > This was already discussed. We specifically want unconditional verbose builds. Maybe provide a link, so it doesn't look like closed doors decision. I only remember a recurrent idea on ports@ list.
Previous discussion: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212075
I contacted with core@ (and some other port's developers, including jbeich@, which email probably doesn't work, as stated in his comments currently) about this PR, where I included related links. I shared some concerns: - Increased I/O usage in console - Hard to find a warning messages, especially for CMake based projects, like LLVM/Clang, where verbose build output disables colored output, one line build messages, with duplicate for warning messages. This also may be related to the size of the console buffer, where compact build output is more lightweight on this. - There are people who use regular make, ports-mgmt/portmaster, ports-mgmt/synth, etc. and not just ports-mgmt/poudriere, where verbose output was enabled, because of BATCH=yes define. If there are concerns about NO_VERBOSE_BUILD usage inside Makefile(s), there is a possibility to move description of it to "special behavior" place: https://svnweb.freebsd.org/ports/head/Mk/bsd.port.mk?revision=421635&view=markup#l981 Also, I didn't ask to apply related patches for other ports, but may use NO_VERBOSE_BUILD check for my (private) ports. I'm not going to "play" close/open PR "games".
Created attachment 175718 [details] Proposed patch (since 423928 revision) Updated patch after ports r423928 changes.
Created attachment 175728 [details] Proposed patch for ports (since 423928 revision) Updated patch after ports r423912 and ports r423928 changes.
Created attachment 175729 [details] Proposed patch for ports (since 423928 revision) Cosmetic fixes.
Need to note, that I mentioned in comment #7 about BATCH=yes for ports-mgmt/poudriere: https://github.com/freebsd/poudriere/blob/534a66bb6399cf0722f534e75496b0efa9517ae2/src/share/poudriere/common.sh#L1809 The good example of it is multimedia/assimp port, where it was verbose build output: before ports r421635: http://beefy6.nyi.freebsd.org/data/101amd64-default/419094/logs/assimp-3.3.1.log and after ports r421635: http://beefy6.nyi.freebsd.org/data/101amd64-default/423739/logs/assimp-3.3.1.log because of: https://svnweb.freebsd.org/ports/head/Mk/Uses/cmake.mk?revision=411970&view=markup#l87 https://svnweb.freebsd.org/ports/head/Mk/Uses/cmake.mk?revision=411970&view=markup#l92 https://svnweb.freebsd.org/ports/head/Mk/Uses/cmake.mk?revision=421635&view=markup#l82 It wasn't for devel/llvm39, before ports r421635 as in reference links in comment #0, because of Ninja build system usage: https://svnweb.freebsd.org/ports/head/devel/llvm39/Makefile?revision=421280&view=markup#l31 https://svnweb.freebsd.org/ports/head/Mk/Uses/ninja.mk?revision=411970&view=markup#l20 but after ports r421635: https://svnweb.freebsd.org/ports/head/Mk/Uses/ninja.mk?revision=421635&view=markup#l17
(In reply to comment #11) > It wasn't for devel/llvm39, before ports r421635 as in reference links in > comment #0, because of Ninja build system usage which also related to following comment: (In reply to comment #7) > - Hard to find a warning messages, especially for CMake based projects, like > LLVM/Clang, where verbose build output disables colored output, > one line build messages, with duplicate for warning messages. where "one line build messages" is a feature of Ninja build system (in case of devel/llvm39), without verbose build output.
Created attachment 175768 [details] Proposed patch for ports (since 423973 revision) Updated patch after ports r423964 and ports r423973 changes.
(In reply to comment #5) > Any build that chains make/gmake supports > $ make -s I may confirm, that this works for multimedia/assimp. (In reply to comment #5) > ninja/waf may need a hack e.g., > > -MAKE_ARGS+= -v > +MAKE_ARGS+= ${ECHO:Ntrue:S/echo/-v/} but not for ports-mgmt/pkg and devel/llvm39, for example. And what I proposed is not a hacks, which solve the consequences, but solution(s) to solve a causes of it, i.e. concrete build flags used to enable verbose build output, instead of less verbose. Possible to encapsulate other MAKE_ARGS+=V=1 or MAKE_ENV+=V=1 cases, used in some ports, to some Mk/Uses file (verbose.mk? USES+=verbose; USES+=verbose:env). This way, verbose build output may be isolated to Mk directory and there may be no need to use ".ifndef NO_VERBOSE_BUILD" check for actual ports, but some Mk/Uses file.
Created attachment 175772 [details] Proposed Mk/Uses/verbose.mk file As I said in comment #14, there is a possibility to create Mk/Uses/verbose.mk file, which provides support for verbose build output, with ability to disable this feature (by using mentioned NO_VERBOSE_BUILD define), when needed. I attached some variant of it. Some examples of usage: 1) USES+= verbose -8<-- % make -V MAKE_ARGS -V=1 ... -->8- 2) USES+= verbose:env -8<-- % make -V MAKE_ENV -V=1 ... -->8- 3) USES+= verbose:env,var -8<-- % make -V MAKE_ARGS -V MAKE_ENV -V=1 ... -V=1 ... -->8- 4) USES+= verbose:env VERBOSE_FLAG= BUILD_VERBOSE=1 -8<-- % make -V MAKE_ENV BUILD_VERBOSE=1 ... -->8- 5) USES+= verbose VERBOSE_VAR= FOO BAR -8<-- % make -V FOO -V BAR -V=1 -V=1 -->8- 6) USES+= verbose:env,var VERBOSE_VAR= FOO VERBOSE_FLAG= verbose=yes -8<-- % make -V FOO -V MAKE_ENV verbose=yes verbose=yes ... -->8- 7) USES+= verbose VERBOSE_VAR= CMAKE_ARGS VERBOSE_FLAG= -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -8<-- % make -V CMAKE_ARGS -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON % make NO_VERBOSE_BUILD=yes -V CMAKE_ARGS -->8-
Created attachment 175774 [details] Proposed Mk/Uses/verbose.mk file Changed value of VERBOSE_FLAG to V=1. Need to note, that there are possible drawbacks in current implementation, which, I guess, possible to solve with using approach used for OPTIONS or MASTER_SITES defines (e.g. VERBOSE_VAR= foo:bar, VERBOSE_VAR_foo= MAKE_ARGS, VERBOSE_FLAG_bar= V=1).
Created attachment 175775 [details] Proposed Mk/Uses/verbose.mk file
(In reply to comment #16) > e.g. VERBOSE_VAR= foo:bar, VERBOSE_VAR_foo= MAKE_ARGS, VERBOSE_FLAG_bar= V=1). or: VERBOSE_VAR= MAKE_ARGS:first MAKE_ENV:second CMAKE_ARGS:third first= V=1 second= VERBOSE=yes third= -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON which may give: MAKE_ARGS+= V=1 MAKE_ENV+= VERBOSE=yes CMAKE_ARGS+= -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON Possible to use default value also (i.e. defined ${VERBOSE_FLAG}), in case of no ":" usage, I guess.
Other possible variable variants, instead of NO_VERBOSE_BUILD: NO_VERBOSE, LESS_VERBOSE, VERBOSE_LESS, VERBOSE_SILENT, etc. -8<-- # cd /usr/ports/ports-mgmt/pkg # make extract # cd work/pkg-1.8.7 # ./configure --help | grep silent-rules --enable-silent-rules less verbose build output (undo: "make V=1") --disable-silent-rules verbose build output (undo: "make V=0") -->8- -8<-- % cmake --help-full | grep -h8 CMAKE_VERBOSE_MAKEFILE | tail -9 CMAKE_VERBOSE_MAKEFILE ---------------------- Enable verbose output from Makefile builds. This variable is a cache entry initialized (to ``FALSE``) by the ``project()`` command. Users may enable the option in their local build tree to get more verbose output from Makefile builds and show each command line as it is launched. -->8-
Created attachment 175812 [details] Some example for complete variant Attached some example for complete variant. - Changed name of NO_VERBOSE_BUILD variable to LESS_VERBOSE. - Moved description of LESS_VERBOSE define to "special behavior" place of Mk/bsd.port.mk file. - Added another variant of Mk/Uses/verbose.mk file, with some fallback code in case of USES+=verbose:list (which may be removed, if not needed). I didn't test, but possible, that various MAKE_ENV+=V=1, MAKE_ENV+=VERBOSE=yes, MAKE_ARGS+=VERBOSE=yes defines may be converted to USES+=verbose, without the need to redefine VERBOSE_VAL value or USES+=verbose:env usage.
(In reply to comment #14) > > (In reply to comment #5) > > ninja/waf may need a hack e.g., > > > > -MAKE_ARGS+= -v > > +MAKE_ARGS+= ${ECHO:Ntrue:S/echo/-v/} > > but not for ports-mgmt/pkg and devel/llvm39, for example. Turns out, that possible to use `make MAKE_ARGS=V=0` for ports-mgmt/pkg port to get less (or compact) verbose build output. This may cover cases with "--disable-silent-rules" for configure scripts. On the other hand, the devel/tig port with MAKE_ARGS=V=1 in Makefile: https://svnweb.freebsd.org/ports/head/devel/tig/Makefile?view=markup&pathrev=423383#l20 doesn't build with `make MAKE_ARGS=V=0`. The `make MAKE_ENV=V=0` doesn't give less verbose build output. The `make -s` disables all build output for this case. The case with Ninja build system (for devel/llvm39 port) is different, because there is no option to switch off (https://svnweb.freebsd.org/ports/head/Mk/Uses/ninja.mk?view=markup&pathrev=421635#l17): -8<-- % ninja --help 2>&1 | grep building -v show all command lines while building -->8-
(In reply to comment #21) > On the other hand, the devel/tig port with MAKE_ARGS=V=1 in Makefile: > https://svnweb.freebsd.org/ports/head/devel/tig/Makefile?view=markup&pathrev=423383#l20 > doesn't build with `make MAKE_ARGS=V=0`. It shows less verbose build output with using `make MAKE_ARGS=` command. (In reply to comment #21) > The case with Ninja build system (for devel/llvm39 port) is different, because > there is no option to switch off Also above command works for devel/llvm39 port. Therefore, this depends.
(In reply to comment #21) > Turns out, that possible to use `make MAKE_ARGS=V=0` `make MAKE_ARGS=V=0 build` (In reply to comment #22) > It shows less verbose build output with using `make MAKE_ARGS=` command. `make MAKE_ARGS= build`, because of other defines in MAKE_ARGS (e.g. DESTDIR), related to `make stage`.
(In reply to comment #0) > with using NO_VERBOSE_BUILD define, e.g. NO_VERBOSE_BUILD=yes in > /etc/make.conf or `make NO_VERBOSE_BUILD=yes` for concrete port to > disable verbose build output or without it to enable. There is also following possible use case (with mentioned changes from comment #20) in /etc/make.conf: -8<-- .ifndef VERBOSE LESS_VERBOSE=yes .endif -->8- which allows to use `make VERBOSE=yes` to get full verbose build output when needed (and less verbose build output otherwise), as it was before ports r421635 changes. Needed to say, that previously this was more granular, with usage of CMAKE_VERBOSE, NINJA_VERBOSE, etc. (and possible CONFIGURE_VERBOSE, WAF_VERBOSE proposals). This was possible to configure for concrete build cluster (not just BATCH=yes), without the need to force mandatory full verbose build output usage for ports framework's users, in my opinion. Moreover, there was no need to use "good" and "bad" biased opinions about different levels (or types) of the same information for documentation: https://www.freebsd.org/doc/en/books/porters-handbook/dads-verbose-logs.html https://svnweb.freebsd.org/doc/head/en_US.ISO8859-1/books/porters-handbook/porting-dads/chapter.xml?view=markup&pathrev=49378#l325 But possible explanation about preferences when submitting (full verbose) build logs to FreeBSD Bugzilla (or maintainer(s)). As you may know, the (current) portmgr@ team (or some of them) chose more simple, but restrictive methods.
Created attachment 178899 [details] The custom.bsd.port.mk file to use for BSDPORTMK variable People, who interested to solve this issue on user level, may create wrappers for build utilities. For example: cmake: -8<-- #!/bin/sh if [ -z "$ALLOW_VERBOSE" ] || [ -n "$LESS_VERBOSE" ]; then args=$(echo -n "$@" | sed 's/-\<DCMAKE_VERBOSE_MAKEFILE[:[:alnum:]]*=[[:alnum:]]*\>//gI') elif [ -n "$FLAG_VERBOSE" ]; then args="$@ $FLAG_VERBOSE" else args="$@" fi /path/to/cmake $args -->8- gmake (but some issue(s) possible with VPATH): -8<-- #!/bin/sh if [ -z "$ALLOW_VERBOSE" ] || [ -n "$LESS_VERBOSE" ]; then args=$(echo -n "$@" | sed 's/\<V=[[:alnum:]]*\>//gI ; s/\<VERBOSE=[[:alnum:]]*\>//gI') unset BUILD_VERBOSE V VERBOSE elif [ -n "$FLAG_VERBOSE" ]; then args="$@ $FLAG_VERBOSE" else args="$@" fi /path/to/gmake $args -->8- ninja: -8<-- #!/bin/sh if [ -z "$ALLOW_VERBOSE" ] || [ -n "$LESS_VERBOSE" ]; then args=$(echo -n "$@" | sed 's/-\<v\>//g') elif [ -n "$FLAG_VERBOSE" ]; then args="$@ $FLAG_VERBOSE" else args="$@" fi /path/to/ninja $args -->8- Also, there is a possibility to define BSDPORTMK variable (e.g. BSDPORTMK=/path/to/custom.bsd.port.mk in /etc/make.conf file) with path to custom bsd.port.mk file, which filters needed variables (or for some other purposes): custom.bsd.port.mk: -8<-- .include "${PORTSDIR}/Mk/bsd.port.mk" .ifndef ALLOW_VERBOSE .if !defined(_SET_LATE_CONFIGURE_ARGS_CHANGED) && defined(GNU_CONFIGURE) && defined(SET_LATE_CONFIGURE_ARGS) _SET_LATE_CONFIGURE_ARGS_CHANGED= yes SET_LATE_CONFIGURE_ARGS+= \ if [ ! -z "`${CONFIGURE_CMD} --help 2>&1 | ${GREP} -- '--enable-silent-rules'`" ]; then \ _LATE_CONFIGURE_ARGS="$${_LATE_CONFIGURE_ARGS} --enable-silent-rules" ; fi ; .endif .if !defined(_CMAKE_ARGS_CHANGED) && defined(_INCLUDE_USES_CMAKE_MK) && defined(CMAKE_ARGS) _CMAKE_ARGS_CHANGED= yes CMAKE_ARGS+= -DCMAKE_VERBOSE_MAKEFILE:BOOL=OFF .endif .if !defined(_NINJA_MAKE_ARGS_CHANGED) && defined(_INCLUDE_USES_NINJA_MK) && defined(MAKE_ARGS) _NINJA_MAKE_ARGS_CHANGED= yes MAKE_ARGS:= ${MAKE_ARGS:N-v} .endif .if !defined(_WAF_MAKE_ARGS_CHANGED) && defined(_INCLUDE_USES_WAF_MK) && defined(MAKE_ARGS) _WAF_MAKE_ARGS_CHANGED= yes MAKE_ARGS:= ${MAKE_ARGS:N--verbose} .endif .if !defined(_QMAKE_ARGS_CHANGED) && defined(_INCLUDE_USES_QMAKE_MK) && defined(QMAKE_VERBOSE) && defined(QMAKE_ARGS) _QMAKE_ARGS_CHANGED= yes QMAKE_ARGS:= ${QMAKE_ARGS:N-d} .endif .endif -->8- This was possible because of https://svnweb.freebsd.org/base/head/share/mk/bsd.port.mk?revision=287436&view=markup#l20 -8<-- BSDPORTMK?= ${PORTSDIR}/Mk/bsd.port.mk -->8- For example, I found this useful for waf build system, where mutlimedia/mpv port builds 8 times faster (on my configuration) with less verbose build output, compared to full verbose build output: -8<-- # time make -C /usr/ports/multimedia/mpv build <...> 88.732u 16.573s 0:36.56 288.0% 19218+338k 1+62io 0pf+0w # make -C /usr/ports/multimedia/mpv clean <...> # time make -C /usr/ports/multimedia/mpv ALLOW_VERBOSE=1 build 331.393u 43.463s 4:53.40 127.7% 5144+212k 0+62io 0pf+0w -->8- Build times for other systems may vary (or almost the same). I attached custom.bsd.port.mk file, just in case, but its contents may be outdated by changes in ports framework. Therefore, you may use it on your own risk.
(In reply to comment #25) > Build times for other systems may vary (or almost the same). "or almost the same" - I meant, with almost the same build time for less and full verbose build outputs, depending from used build utilities.
(In reply to comment #25) Results for single threaded build of multimedia/mpv: -8<-- # time make -C /usr/ports/multimedia/mpv MAKE_JOBS_UNSAFE=yes build <...> 'build' finished successfully (1m26.735s) 83.180u 12.030s 1:34.93 100.2% 19949+330k 19+60io 0pf+0w # make -C /usr/ports/multimedia/mpv clean <...> # time make -C /usr/ports/multimedia/mpv MAKE_JOBS_UNSAFE=yes ALLOW_VERBOSE=yes build <...> 'build' finished successfully (5m39.394s) 325.951u 22.101s 5:47.56 100.1% 5460+212k 0+61io 0pf+0w -->8- About 3.66 (3.91) times faster for less verbose build. I did this test to minimize effect of multiple threads.
Created attachment 180096 [details] The custom.bsd.port.mk file to use for BSDPORTMK variable (extended version) (In reply to comment #25) Added some (user level) solution for following cases: MAKE_ARGS+=V=1 https://github.com/freebsd/freebsd-ports/blob/b83b2ba2ca7ca69c7c3a5ccd4dd00ce146bfe9e8/devel/cgit/Makefile#L26 MAKE_ARGS+=VERBOSE=1 https://github.com/freebsd/freebsd-ports/blob/4687be62e6ec86ba48a6447122a6cab409334540/devel/icu/Makefile#L27 MAKE_ARGS+=verbose=yes https://github.com/freebsd/freebsd-ports/blob/6c1e55158cb18ae1b2213a452b9f1bd861b14944/multimedia/libvpx/Makefile#L42 MAKE_ARGS+=verbose=1 https://github.com/freebsd/freebsd-ports/blob/38843cd6da919e19590c9aa97c12d9cffffdf537/devel/xsd/Makefile#L35 MAKE_ENV+=V=1 https://github.com/freebsd/freebsd-ports/blob/70558fc7a7b0973c0ca323e82e585d11f0df5593/devel/git/Makefile#L31 MAKE_ENV+=VERBOSE=1 https://github.com/freebsd/freebsd-ports/blob/fce8f50432e2ce4ade7801cfaa86793c2b971955/x11-toolkits/girara/Makefile#L20 MAKE_ENV+=verbose=yes https://github.com/freebsd/freebsd-ports/blob/d7cb557b47478b4047ff701030e9309f957ab6b2/multimedia/xawtv/Makefile#L27 and other V=*, VERBOSE=* (case insensitive) variants: -8<-- .ifndef ALLOW_CUSTOM_VERBOSE .if !defined(_MAKE_ARGS_CHECKED) && !defined(_NINJA_MAKE_ARGS_CHANGED) && !defined(_WAF_MAKE_ARGS_CHANGED) && defined(MAKE_ARGS) _MAKE_ARGS_CHECKED= yes .if ${MAKE_ARGS:M[vV]=*} MAKE_ARGS:= ${MAKE_ARGS:N[vV]=*} _MAKE_ARGS_CHECKED+= v .endif .if ${MAKE_ARGS:M[vV][eE][rR][bB][oO][sS][eE]=*} MAKE_ARGS:= ${MAKE_ARGS:N[vV][eE][rR][bB][oO][sS][eE]=*} _MAKE_ARGS_CHECKED+= verbose .endif .endif .if !defined(_MAKE_ENV_CHECKED) && defined(MAKE_ENV) _MAKE_ENV_CHECKED= yes .if ${MAKE_ENV:M[vV]=*} MAKE_ENV:= ${MAKE_ENV:N[vV]=*} _MAKE_ENV_CHECKED+= v .endif .if ${MAKE_ENV:M[vV][eE][rR][bB][oO][sS][eE]=*} MAKE_ENV:= ${MAKE_ENV:N[vV][eE][rR][bB][oO][sS][eE]=*} _MAKE_ENV_CHECKED+= verbose .endif .endif .endif -->8- Possible to disable this with using ALLOW_CUSTOM_VERBOSE or ALLOW_VERBOSE variable(s): % make ALLOW_CUSTOM_VERBOSE=1 or (also to disable other checks): % make ALLOW_VERBOSE=1 To check what cases were used (v and/or verbose): % make -V _MAKE_ARGS_CHECKED -V _MAKE_ENV_CHECKED To check what was removed from MAKE_ARGS and/or MAKE_ENV variables (with using shells/bash): $ diff <(make ALLOW_VERBOSE=1 -V MAKE_ARGS -V MAKE_ENV) <(make -V MAKE_ARGS -V MAKE_ENV) For example: % cd /usr/ports/devel/icu % make -V _MAKE_ARGS_CHECKED -V _MAKE_ENV_CHECKED yes verbose yes % bash $ diff <(make ALLOW_VERBOSE=1 -V MAKE_ARGS -V MAKE_ENV) <(make -V MAKE_ARGS -V MAKE_ENV) 1c1 < VERBOSE=1 DESTDIR=/usr/ports/devel/icu/work/stage --- > DESTDIR=/usr/ports/devel/icu/work/stage $ exit The mentioned solution provided as is, without warranty (especially, because it may change variables, which may have different meaning in different context).