Bug 276478
| Summary: | Mk/*: Build with a clean environment | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Tijl Coosemans <tijl> | ||||||||||||||||||
| Component: | Ports Framework | Assignee: | Tijl Coosemans <tijl> | ||||||||||||||||||
| Status: | Closed FIXED | ||||||||||||||||||||
| Severity: | Affects Only Me | CC: | acm, amdmi3, arrowd, bdrewery, eduardo, freebsd, gnome, go, haskell, java, kde, olgeni, perl, ports-bugs, python, ruby, rust, saper, x11 | ||||||||||||||||||
| Priority: | --- | Flags: | antoine:
exp-run+
|
||||||||||||||||||
| Version: | Latest | ||||||||||||||||||||
| Hardware: | Any | ||||||||||||||||||||
| OS: | Any | ||||||||||||||||||||
| Bug Depends on: | |||||||||||||||||||||
| Bug Blocks: | 272216 | ||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||
I think there is a minimum environment to keep to avoid breaking things, like: UNAME_r UNAME_v OSVERSION UNAME_m UNAME_p Some new failure logs: https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/mongodb-tools-100.9.4.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/rubygem-do_postgres-0.10.17_2.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/bazel-buildtools-3.2.1_15.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/elixir-hex-2.0.6.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/elixir-make-0.4.2.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/coreos-etcd31-3.1.20_20.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/coreos-etcd32-3.2.32_18.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/git-codereview-1.8.0_1.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/goredo-1.32.0_1.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/yandex-ddns-g20200613_14.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/python27-2.7.18_2.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/python310-3.10.13.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/python311-3.11.7.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/python38-3.8.18.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/python39-3.9.18.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/tauthon-2.8.5.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/mailhog-1.0.1_15.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/gap-4.12.2.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/reduce-psl-20231218_1.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/cryptoballot-g20181015_18.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/carbon-relay-ng-0.10.0_1.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/intel-snap-0.0.1_16.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/kapacitor-1.5.1_15.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/mqtt2prometheus-0.1.6_15.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/nvidia_gpu_prometheus_exporter-g20181028_16.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/prometheus-collectd-exporter-0.5.0_16.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/prometheus1-1.8.2_18.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/cf-6.49.0_16.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/revsocks-2.8_1.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/nss-3.95.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/cbsd-mq-api-0.3_15.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/cbsd-mq-router-0.2_16.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/hared-1.0.45_16.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/jest-3.0.16_17.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/gron-0.6.1_14.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/rubygem-prism-0.19.0.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-01-21_08h18m04s/logs/nginx-vts-exporter-0.10.7_15.log I do not think this is needed. portmgr@ is demoting support for building on the host (as a consequence, all the tools that do that - `portmaster`, `synth`, etc). We shouldn't bother ourselves with build-on-host scenarios. Yes, this includes running make in a ports directory to test its build. What does it mean that "build on host" is demoted? @Tijl: I think TERM should be explicitly left out. I am not sure if the colors are that important (for example, I like to diff good/failed build logs to find out what went wrong and ANSI color sequences make it diffiult or ugly). Created attachment 247897 [details]
patch2
To fix the build failures this patch introduces a WRK_ENV variable and adds the following environment variables to it: HOME (elixir ports), PWD (go ports), UNAME_* (nss, python, reduce-psl), PATH (rubygem ports).
Also add TERM to restore colored output to terminals. This does not affect output to files. And TMPDIR for users who define that.
math/gap: This build failure is unrelated. The port Makefile defines SHEBANG_FILES=. which means shebangfix touches every file changing their timestamp. This causes the build to think some Makefile.in files are out-of-date so it tries to regenerate them with automake without regenerating prerequisite autotools files. Perhaps shebangfix could restore timestamps like run-autotools-fixup does, but that's for somebody else to look into.
(In reply to Mathieu Arnold from comment #2) I don't mind making poudriere the default but that shouldn't mean patches for the non-poudriere case are rejected. I think that would only lead to unneeded drama in the community. In this case I think the patch also helps poudriere because it also eliminates environment variables with common names like ARCH that some ports need patches for currently. ocaml native fails with this environ: https://pkg-status.freebsd.org/package22/data/132amd64-default-foo/2024-02-02_19h17m47s/logs/errors/ocaml-ocamlbuild-0.14.2_3.log Created attachment 248176 [details]
patch3
Fix devel/ocaml-ocamlbuild: Define OCAML_NATIVE=true explicitly as explained in the upstream Readme.md. The upstream configure.make defines this if ARCH is defined and that's no longer the case.
After looking into this change more closely I have to say that I actually like it. Thanks for working on this, Tijl! Created attachment 248232 [details]
patch4
- Put PWD in quotes because it may contain spaces (archivers/unarchiver and devel/sabre).
- Add OSVERSION to WRK_ENV for net/freebsd-telnetd, which runs uname -K.
- Revert changes to devel/ocaml-ocamlbuild and fix the problem in lang/ocaml instead. This also fixes math/ocaml-num. More details in commit log.
- Fix games/legesmotus by removing patches.
- Let sip print a build log in graphics/py-python-poppler-qt5 because I cannot reproduce it.
- Fix net/libnatpmp.
(In reply to Tijl Coosemans from comment #9) It seems there is no need in extra '$' in PWD="$${PWD}" ? (In reply to Gleb Popov from comment #10) It is needed to let the shell evaluate ${PWD}. In a command like (cd dir && env PWD="${PWD}" command ...), make's idea of PWD isn't up-to-date. Some new failure logs: https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-02-11_08h14m44s/logs/errors/zynaddsubfx-3.0.6_2,2.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-02-11_08h14m44s/logs/errors/turbo-g20230621.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-02-11_08h14m44s/logs/errors/iortcw-1.51c_3,1.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-02-11_08h14m44s/logs/errors/q3cellshading-1.0_4.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-02-11_08h14m44s/logs/errors/tremulous-1.1.0_13.log https://pkg-status.freebsd.org/pb2/data/140i386-default-foo/2024-02-11_08h14m44s/logs/errors/boringssl-0.0.0.0.2024.02.03.01.log https://pkg-status.freebsd.org/package18/data/140amd64-default-foo/2024-02-11_22h21m21s/logs/errors/turbo-g20230621.log https://pkg-status.freebsd.org/package18/data/140amd64-default-foo/2024-02-11_22h21m21s/logs/errors/iortcw-1.51c_3,1.log https://pkg-status.freebsd.org/package23/data/132i386-default-foo/2024-02-11_07h49m56s/logs/errors/zynaddsubfx-3.0.6_2,2.log https://pkg-status.freebsd.org/package23/data/132i386-default-foo/2024-02-11_07h49m56s/logs/errors/turbo-g20230621.log https://pkg-status.freebsd.org/package23/data/132i386-default-foo/2024-02-11_07h49m56s/logs/errors/iortcw-1.51c_3,1.log https://pkg-status.freebsd.org/package23/data/132i386-default-foo/2024-02-11_07h49m56s/logs/errors/q3cellshading-1.0_4.log https://pkg-status.freebsd.org/package23/data/132i386-default-foo/2024-02-11_07h49m56s/logs/errors/tremulous-1.1.0_13.log https://pkg-status.freebsd.org/package23/data/132i386-default-foo/2024-02-11_07h49m56s/logs/errors/boringssl-0.0.0.0.2024.02.03.01.log Created attachment 248511 [details]
patch5
audio/zynaddsubfx: I cannot reproduce this. The process seems to have died without flushing stdout so I've added a patch that redirects stdout to stderr and that adds some more printfs.
editors/turbo: Unrelated, fixed by cmake 3.28.3 update.
games/iortcw, games/q3cellshading, games/tremulous: All using Quake 3 engine. Fix use of ARCH. Reduce diff between FreeBSD code and Linux code.
security/boringssl: Unrelated, already fixed by maintainer.
If audio/zynaddsubfx fails again can you make the work directory available somewhere for me to download?
Created attachment 248512 [details]
patch6
Fix an issue with the audio/zynaddsubfx patch.
(In reply to Tijl Coosemans from comment #7) > The ports tree adds its definition of ARCH to the MAKEFLAGS environment variable, which is interpreted by sub-makes as command line arguments, which means that any definition of ARCH in upstream makefiles was overridden. The following ports required fixes now that this is no longer the case. From what I can tell, it is already possible to set NOPRECIOUSMAKEVARS=yes for this exact same purpose, or at least this is sufficient to remove the workaound from lang/ocaml. (In reply to Benjamin Jacobs from comment #15) Yes, but that only gets rid of variables passed through MAKEFLAGS. It does not get rid of MAKEFLAGS itself. On i386 I have those 2 failures: https://pkg-status.freebsd.org/gohan04/data/140i386-default-foo/2024-02-18_19h53m07s/logs/zynaddsubfx-3.0.6_2,2.log https://pkg-status.freebsd.org/gohan04/data/140i386-default-foo/2024-02-18_19h53m07s/logs/iortcw-1.51c_4,1.log workdir in https://people.freebsd.org/~antoine/zynaddsubfx-work.tar Created attachment 248731 [details]
textproc/mxml 3.3.1
textproc/mxml: Update to 3.3.1
Also add a patch to fix a bounds check problem:
When writing XML data into a buffer a pointer is used to keep track of the current position. When the end of the buffer is reached the writing stops but the pointer continues to be incremented to determine how many bytes would have been written had the buffer been large enough. The problem is that the bounds check that stops the writing did not handle the case where a large amount of data causes the pointer to wrap around to 0. This can happen for example when the buffer is allocated on the stack and the stack is close to the end of the address space.
This should fix audio/zynaddsubfx. The new build environment is smaller, thereby causing the buffer mentioned above to be allocated closer to the end of the address space on FreeBSD i386.
Created attachment 248732 [details]
patch7
audio/zynaddsubfx: Remove debug printfs again.
games/iortcw: Fix pkg-plist problem.
Exp-run looks fine A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=572f2361692640bc27729191b1267aa3fcc354a7 commit 572f2361692640bc27729191b1267aa3fcc354a7 Author: Tijl Coosemans <tijl@FreeBSD.org> AuthorDate: 2024-01-20 11:38:46 +0000 Commit: Tijl Coosemans <tijl@FreeBSD.org> CommitDate: 2024-02-29 20:21:37 +0000 Mk/*: Build with a clean environment Both our make and gmake use the MAKEFLAGS environment variable but the values aren't compatible and the latest version of gmake complains about that. To rule out that any environment variable can cause problems like this, add a new command SETENVI=/usr/bin/env -i that clears the environment, and use it to run upstream build systems with a clean environment. Introduce a new variable WRK_ENV that contains the environment to use with SETENVI in all targets that run upstream build commands. Variables that are common between CONFIGURE_ENV and MAKE_ENV could be moved to WRK_ENV but for now it just contains a minimal environment: HOME=${WRKDIR}: Fixes USES=elixir ports that were using the user's HOME. OSVERSION: For cross building; determines the output of uname -K and getosreldate(3); affects net/freebsd-telnetd for example. PATH: Fixes USES=gem ports that were using the user's PATH. PWD=$${PWD}: Preserve current working directory; affects USES=go ports. TERM: To preserve colored output to terminals. TMPDIR: For users who define that. UNAME_*: For cross building; determines the output of uname(1); affects lang/python* for example. This commit deals with everything under Mk/. Ports that have their own targets running upstream build commands can switch to SETENVI later. The ports tree adds its definition of ARCH to the MAKEFLAGS environment variable, which is interpreted by sub-makes as command line arguments, which means that any definition of ARCH in upstream makefiles was overridden. The following ports required fixes now that this is no longer the case. games/iortcw, games/q3cellshading, games/tremulous: These use Quake 3 engine code. Fix use of ARCH. Reduce diff between FreeBSD code and Linux code. games/legesmotus: Remove ARCH related patches. lang/ocaml: Patch configure script so it detects amd64 correctly. Also make the powerpc case consistent with the other architectures. This also affects other ocaml ports like devel/ocaml-ocamlbuild and math/ocaml-num that include a Makefile.config installed by lang/ocaml. While here, use SETENVI in check-test target. net/libnatpmp: Use of upstream definition of ARCH triggers installation in PREFIX/lib64 on amd64. Disable this. PR: 276478 Approved by: portmgr (antoine) Exp-run by: antoine CHANGES | 17 +- Mk/Uses/angr.mk | 2 +- Mk/Uses/cabal.mk | 16 +- Mk/Uses/cargo.mk | 5 +- Mk/Uses/cmake.mk | 9 +- Mk/Uses/elixir.mk | 2 +- Mk/Uses/erlang.mk | 2 +- Mk/Uses/gem.mk | 8 +- Mk/Uses/go.mk | 16 +- Mk/Uses/imake.mk | 2 +- Mk/Uses/lazarus.mk | 2 +- Mk/Uses/mate.mk | 2 +- Mk/Uses/perl5.mk | 10 +- Mk/Uses/pyqt.mk | 4 +- Mk/Uses/pytest.mk | 2 +- Mk/Uses/python.mk | 20 +- Mk/Uses/qmake.mk | 2 +- Mk/Uses/ruby.mk | 10 +- Mk/bsd.commands.mk | 1 + Mk/bsd.java.mk | 8 +- Mk/bsd.port.mk | 25 +- games/iortcw/Makefile | 6 +- games/iortcw/files/patch-MP_Makefile (new) | 73 ++++++ .../files/patch-MP_code_qcommon_q__platform.h | 42 ++-- games/iortcw/files/patch-SP_Makefile (new) | 73 ++++++ .../files/patch-SP_code_qcommon_q__platform.h | 42 ++-- games/legesmotus/files/patch-Makefile (gone) | 11 - games/legesmotus/files/patch-common.mk | 18 +- games/q3cellshading/Makefile | 19 +- games/q3cellshading/files/patch-code-unix-Makefile | 268 ++++++++------------- games/tremulous/Makefile | 4 +- games/tremulous/files/patch-Makefile | 124 +++++----- lang/ocaml/Makefile | 16 +- lang/ocaml/files/patch-configure | 34 ++- net/libnatpmp/files/patch-Makefile (new) | 10 + 35 files changed, 487 insertions(+), 418 deletions(-) A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=9827f3b51002e7d07c4f3fcd76484f665e95cefe commit 9827f3b51002e7d07c4f3fcd76484f665e95cefe Author: Tijl Coosemans <tijl@FreeBSD.org> AuthorDate: 2024-02-24 20:20:15 +0000 Commit: Tijl Coosemans <tijl@FreeBSD.org> CommitDate: 2024-02-29 20:21:34 +0000 textproc/mxml: Update to 3.3.1 Also add a patch to fix a bounds check problem discovered during an exp-run for bug 276478. PR: 276478 textproc/mxml/Makefile | 7 +- textproc/mxml/distinfo | 6 +- textproc/mxml/files/patch-mxml-file.c (new) | 105 ++++++++++++++++++++++++++++ 3 files changed, 112 insertions(+), 6 deletions(-) Tijl, should these lines be adapted somehow? https://github.com/freebsd/freebsd-ports/blob/main/Mk/bsd.port.mk#L2015 Maybe this block should append to WRK_ENV only? (In reply to Gleb Popov from comment #23) Yes, everything that is common between CONFIGURE_ENV and MAKE_ENV can probably be moved to WRK_ENV. A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=bf8e12d3e72868d97f4515c7dee6a22dd0f30e74 commit bf8e12d3e72868d97f4515c7dee6a22dd0f30e74 Author: Gleb Popov <arrowd@FreeBSD.org> AuthorDate: 2024-03-28 06:21:00 +0000 Commit: Gleb Popov <arrowd@FreeBSD.org> CommitDate: 2024-03-28 11:40:52 +0000 Framework: Use WRK_ENV when handling USE_LOCALE PR: 276478 Mk/bsd.port.mk | 3 +-- lang/ghc/Makefile | 2 +- math/scilab-toolbox-swt/Makefile | 4 ++-- 3 files changed, 4 insertions(+), 5 deletions(-) |
Created attachment 247801 [details] patch Both our make and gmake use the MAKEFLAGS environment variable but the values aren't compatible and the latest version of gmake complains about that. To prevent that environment variables interfere with the build process like this, this patch adds a new command SETENVI=/usr/bin/env -i that clears the environment. The idea is to use SETENVI instead of SETENV when running build tools and scripts so their environment only contains variables from CONFIGURE_ENV or MAKE_ENV or TEST_ENV or similar. This way it doesn't matter what variables our make uses or what environment variables a user might have set. This patch takes care of Mk/* and Mk/Uses/*. Individual ports can be changed later. If you are the maintainer of any of the Mk/Uses/* files please review the changes. A minor open problem is that the TERM environment variable is now missing which means that error messages are no longer colored for instance. I'd like to introduce a new variable WRK_ENV that would contain common environment variables like TERM, PATH, and SHELL, and perhaps other variables that are common between CONFIGURE_ENV and MAKE_ENV. It would be used in every make target while CONFIGURE_ENV, MAKE_ENV, and TEST_ENV contain target specific environment variables. Any objections to this?