With ALLOW_MAKE_JOBS=yes, the last part are as follows.
This looks like the failure mode reported on Bug 255007.

In both cases (ALLOW_MAKE_JOBS=yes or not), this happenes very late stage of builds, toooo time consuming to reproduce further. Giving up. :-(

--- .obj/qquickwebenginefaviconprovider.o ---
ck -I/usr/local/include/qt5/QtGui/5.15.8 -I/usr/local/include/qt5/QtGui/5.15.8/QtGui -I/usr/local/include/qt5/QtQmlModels/5.15.8 -I/usr/local/include/qt5/QtQmlModels/5.15.8/QtQmlModels -I/usr/local/include/qt5/QtQml/5.15.8 -I/usr/local/include/qt5/QtQml/5.15.8/QtQml -I/usr/local/include/qt5/QtCore/5.15.8 -I/usr/local/include/qt5/QtCore/5.15.8/QtCore -I.moc -I/usr/local/include -I/usr/local/include -I/usr/local/lib/qt5/mkspecs/freebsd-clang -o .obj/qquickwebenginefaviconprovider.o /wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/webengine/api/qquickwebenginefaviconprovider.cpp
--- .obj/qquickwebengineloadrequest.o ---
c++ -B/usr/local/bin -c -O2 -pipe -march=haswell -fstack-protector-strong -fno-strict-aliasing -std=c++1z -fvisibility=hidden -fvisibility-inlines-hidden -fno-exceptions -Wall -Wextra -Wdate-time -Winconsistent-missing-override -pthread -fPIC -DQT_NO_LINKED_LIST -DQT_BUILD_WEBENGINE_LIB -DQT_NO_NARROWING_CONVERSIONS_IN_CONNECT -DQT_BUILD_WEBENGINE_LIB -DQT_BUILDING_QT -DQT_NO_CAST_TO_ASCII -DQT_ASCII_CAST_WARNINGS -DQT_MOC_COMPAT -DQT_USE_QSTRINGBUILDER -DQT_DEPRECATED_WARNINGS -DQT_DISABLE_DEPRECATED_BEFORE=0x050000 -DQT_DEPRECATED_WARNINGS_SINCE=0x060000 -DQT_NO_EXCEPTIONS -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -DQT_NO_DEBUG -DQT_WEBENGINECORE_LIB -DQT_QUICK_LIB -DQT_GUI_LIB -DQT_QMLMODELS_LIB -DQT_WEBCHANNEL_LIB -DQT_QML_LIB -DQT_NETWORK_LIB -DQT_POSITIONING_LIB -DQT_CORE_LIB -DQT_WEBENGINECORE_LIB -DQT_QUICK_LIB -DQT_GUI_LIB -DQT_QMLMODELS_LIB -DQT_WEBCHANNEL_LIB -DQT_QML_LIB -DQT_NETWORK_LIB -DQT_POSITIONING_LIB -DQT_CORE_LIB -I/wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/webengine -I. -I/wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/webengine -I/wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/webengine/api -I/wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/core -I/wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/core/api -I../../include -I../../include/QtWebEngine -I../../include/QtWebEngine/5.15.8 -I../../include/QtWebEngine/5.15.8/QtWebEngine -I../../include/QtWebEngineCore -I/usr/local/include/qt5 -I/usr/local/include/qt5/QtQuick -I/usr/local/include/qt5/QtGui -I/usr/local/include/qt5/QtQmlModels -I/usr/local/include/qt5/QtWebChannel -I/usr/local/include/qt5/QtQml -I/usr/local/include/qt5/QtNetwork -I/usr/local/include/qt5/QtPositioning -I/usr/local/include/qt5/QtCore -I../../include/QtWebEngineCore/5.15.8 -I../../include/QtWebEngineCore/5.15.8/QtWebEngineCore -I/usr/local/include/qt5/QtQuick/5.15.8 -I/usr/local/include/qt5/QtQuick/5.15.8/QtQuick -I/usr/local/include/qt5/QtGui/5.15.8 -I/usr/local/include/qt5/QtGui/5.15.8/QtGui -I/usr/local/include/qt5/QtQmlModels/5.15.8 -I/usr/local/include/qt5/QtQmlModels/5.15.8/QtQmlModels -I/usr/local/include/qt5/QtQml/5.15.8 -I/usr/local/include/qt5/QtQml/5.15.8/QtQml -I/usr/local/include/qt5/QtCore/5.15.8 -I/usr/local/include/qt5/QtCore/5.15.8/QtCore -I.moc -I/usr/local/include -I/usr/local/include -I/usr/local/lib/qt5/mkspecs/freebsd-clang -o .obj/qquickwebengineloadrequest.o /wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/webengine/api/qquickwebengineloadrequest.cpp
--- sub-process-make_first ---
--- ../../libexec/QtWebEngineProcess ---
c++ -B/usr/local/bin -Wl,--as-needed -fstack-protector-strong -pthread -Wl,-rpath,/usr/local/lib/qt5 -Wl,-rpath-link,/wrkdirs/usr/ports/www/qt5-webengine/work/.build/lib -o ../../libexec/QtWebEngineProcess .obj/main.o   -L/wrkdirs/usr/ports/www/qt5-webengine/work/.build/lib -L/usr/local/lib /usr/local/lib/qt5/libQt5Gui.so /usr/local/lib/qt5/libQt5Core.so -lGL /wrkdirs/usr/ports/www/qt5-webengine/work/.build/lib/libQt5WebEngineCore.so /usr/local/lib/qt5/libQt5Quick.so /usr/local/lib/qt5/libQt5QmlModels.so /usr/local/lib/qt5/libQt5WebChannel.so /usr/local/lib/qt5/libQt5Qml.so /usr/local/lib/qt5/libQt5Network.so /usr/local/lib/qt5/libQt5Gui.so /usr/local/lib/qt5/libQt5Positioning.so /usr/local/lib/qt5/libQt5Core.so
c++: error: no such file or directory: '/wrkdirs/usr/ports/www/qt5-webengine/work/.build/lib/libQt5WebEngineCore.so'
*** [../../libexec/QtWebEngineProcess] Error code 1

make[3]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/process
1 error

make[3]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/process
*** [sub-process-make_first] Error code 2

make[2]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src
--- sub-webengine-make_first ---
*** [sub-module-pro-make_first] Error code 6

make[3]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/webengine
1 error

make[3]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/webengine
*** [sub-webengine-make_first] Error code 2

make[2]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src
--- sub-tools-qwebengine_convert_dict-make_first ---
*** [sub-tools-qwebengine_convert_dict-make_first] Error code 6

make[2]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src
--- sub-pdf-make_first ---
WARNING at the command-line "--args":1:1128: Build argument has no effect.
use_qt=true init_stack_vars=false is_component_build=false is_shared=true enable_debugallocation=false enable_media_remoting=false enable_message_center=false enable_nacl=false enable_remoting=false enable_reporting=false enable_resource_allowlist_generation=false enable_swiftshader=false enable_swiftshader_vulkan=false angle_enable_swiftshader=false enable_web_speech=false enable_widevine=true forbid_non_component_debug_builds=false has_native_accessibility=false safe_browsing_mode=0 skia_use_dawn=false toolkit_views=false treat_warnings_as_errors=false use_allocator_shim=false use_allocator="none" use_custom_libcxx=false chrome_pgo_phase=0 enable_hangout_services_extension=true optimize_webui=false enable_js_type_check=false v8_use_external_startup_data=false strip_absolute_paths_from_debug_symbols=false use_jumbo_build=true jumbo_file_merge_limit=8 jumbo_build_excluded=["browser"] enable_precompiled_headers=false is_official_build=true is_debug=false symbol_level=0 blink_symbol_level=0 remove_v8base_debug_symbols=true use_cups=false use_gio=false use_gnome_keyring=false use_udev=true use_bundled_fontconfig=false use_sysroot=false enable_session_service=false is_cfi=false use_ozone=true use_x11=false ozone_auto_platforms=false ozone_platform_headless=false ozone_platform_external=true ozone_platform="qt" ozone_extra_path="/wrkdirs/usr/ports/www/qt5-webengine/work/qtwebengine-everywhere-src-5.15.8/src/core/ozone/ozone_extra.gni" use_gold=false use_lld=false is_clang=true clang_use_chrome_plugins=false clang_use_default_sample_profile=false clang_base_path="/usr -B/usr" custom_toolchain="/wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/toolchain:target" host_toolchain="/wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/toolchain:host" host_cpu="x64" pkg_config="pkg-config" host_pkg_config="/usr/local/bin/pkg-config" use_system_zlib=true use_system_minizip=true pdfium_use_system_zlib=true use_system_libpng=true pdfium_use_system_libpng=true use_system_libjpeg=true use_system_freetype=true use_system_harfbuzz=true use_glib=false enable_basic_printing=true enable_print_preview=true use_dbus=true use_udev=false use_nss_certs=false pdf_enable_v8=false pdf_enable_xfa=false pdf_enable_xfa_bmp=false pdf_enable_xfa_gif=false pdf_enable_xfa_png=false pdf_enable_xfa_tiff=false qtwebengine_target="/wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/pdf/release:QtPdf"
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ^----
The variable "use_bundled_fontconfig" was set as a build argument
but never appeared in a declare_args() block in any buildfile.

To view all possible args, run "gn args --list <out_dir>"

The build continued as if that argument was unspecified.

Done. Made 14471 targets from 2283 files in 6259ms
*** [sub-gn_run-pro-make_first] Error code 6

make[3]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/pdf
1 error

make[3]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src/pdf
*** [sub-pdf-make_first] Error code 2

make[2]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src
4 errors

make[2]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build/src
*** [sub-src-make_first] Error code 2

make[1]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build
1 error

make[1]: stopped in /wrkdirs/usr/ports/www/qt5-webengine/work/.build
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make: stopped in /usr/ports/www/qt5-webengine
=>> Cleaning up wrkdir
===>  Cleaning for qt5-webengine-5.15.8
build of www/qt5-webengine | qt5-webengine-5.15.8 ended at Mon Mar  6 19:30:55 JST 2023
build time: 03:35:45
!!! build failure encountered !!!

Linking the library libQt5WebEngineCore.so.5.15.8 failed because
ulimit was unable to expand the open files limit for the process:

  ulimit -n 4096 ...
  ulimit: bad limit: Operation not permitted


By default the jail poudriere uses has a limit of 1024:

  --Resource limits--
  ...
  open files                      (-n)  1024

I increased it for the build of qt5-webengine in poudriere.conf:

  # How many file descriptors to limit each jail process to (default: 1024)
  # This can also be set per PKGBASE, such as MAX_FILES_RStudio=2048.
  # Package names with hyphens (-) should be replaced with underscores (_).
  #MAX_FILES=1024
  MAX_FILES_qt5_webengine=4096

This fixed the build problem.

(In reply to rk from comment #2)

Thanks! I've missed the option.
Built fine by adding it in /usr/local/etc/poudriere.conf.

Closed as "Not a Bug".

But I still can't understand why this raising of limit is required,
as www/qt5-webengine seems to be based on www/chromium according to
the commit message of 5.15.8, but giant www/chromium haven't fail as such
on poudriere builds.

(In reply to Tomoaki AOKI from comment #3)

I noticed that qt6-webengine built fine using the defaults
of 1024 files. So I looked into the log file and found this:

  /wrkdirs/usr/ports/www/qt6-webengine/work/.build/linker_ulimit.sh /usr/local/libexec/ccache/c++ ...
  ulimit: bad limit: Operation not permitted

So they apparently changed the way the do the ulimit call.
In qt5-webengine it is hardcoded in the command line as follows:

  ulimit -n 4096 && c++ -B/usr/local/bin ...

So the whole build fails when ulimit returns an error. In
qt6-webengine it was wrapped inside a script and wasn't
used to start the linker/compiler with an "&&" operator.
Anyway, the linker was able to operate with the 1024 file limit
in qt6-webengine. Maybe some step needed more open files in some
circumstances so someone added these ulimit calls.

(In reply to rk from comment #4)

Thanks.
So backporting the script is (hopefully) needed?
Or, possibly, next release include it?

How does the FreeBSD project's package builder solve this?  Do they have a non-default poudriere.conf that has a higher limit for MAX_FILES?

It might be nice to be able to commit something to a particular port's Makefile that provides information for how poudriere should set up environment for a particular port build (instead of editing poudriere.conf).  Then you wouldn't have to worry about manually configuring poudriere to accommodate individual ports.

It would be helpful to test the resource limit on open files EARLY in the build so you don't have to suffer through a LONG poudriere build of qt5-webengine, only to fail near the end.  Whether the build _really_ needs 4096 descriptors or the poudriere [current] default of 1024 (or some other value) would suffice, checking the minimum resource limit early would be a fairly simple addition to the port.

I just did a test link of libQtWebEngineCore with truss.  The largest file descriptor it opened (in a 12.x/amd64 poudriere environment) was 516.

I don't know why the ulimit is there.  Maybe at one point upstream had a problem with some runaway linking issue?  And they decided to limit the open file descriptors as a backstop to keep it from consuming too many system resources, perhaps.  It seems unlikely that it is likely to happen with released code in a downstream environment like FreeBSD.  But without knowing why it was added upstream, that's just a guess.

I think it would be fine to just remove the ulimit or ignore any failure from it (ulimit -n 4096 || true).  There's already a limit in the poudriere environment (at a default of 1024 descriptors, that's lower than the 4096), so the backstop is already in place at a higher level for poudriere builds.

Created attachment 241886 [details]
[patch] remove troublesome ulimit

This patch just removes the ulimit.

(In reply to John Hein from comment #9)
Feel free to re-open this if you feel this patch is useful.

I stumbled across the bug in mid August again and had to wait two times in poudriere 6 hours to get to the failure. After the 2nd time I search with Don Google to get to this fix...

The issue should be re-opened again (I don't know how to do so) and the maintainer should fix this port appropriately. Thanks

Reopened per requests.

Comment 13 Kai Knoblich 2023-08-15 21:40:18 UTC

(In reply to Tomoaki AOKI from comment #0)
(In reply to rk from comment #2)
(In reply to John Hein from comment #8)

Thanks for the report and your findings so far.

I was able to reproduce the issue, if a build jail uses a non-root user with the following conditions:

- devel/ccache is installed

poudriere.conf:

- BUILD_AS_NON_ROOT=yes (default)
- CCACHE_DIR=/path/to/ccache/folder
- CCACHE_DIR_NON_ROOT_SAFE=yes
- MAX_FILES=1024 (default)


I was not able to reproduce it, if a build jail uses the root user with the following setup:

- devel/ccache is installed

poudriere.conf:

- BUILD_AS_NON_ROOT=yes (default, but is changed to "no" as CCACHE_DIR_NON_ROOT_SAFE is not set)
- CCACHE_DIR=/path/to/ccache/folder
- CCACHE_DIR_NON_ROOT_SAFE is not set
- MAX_FILES=1024 (default)


AFAIK the official FreeBSD builders use (by default?) the root user for the build jails and don't seem to use devel/ccache at all so the reported issue with www/qt5-webengine didn't occur there.


(In reply to John Hein from comment #9)

Many thanks for the patch, which looks promising.  I'm going commit it in a few hours/days once the test builds are completed.  I also plan to do a MFH for it.

Comment 14 commit-hook 2023-08-17 10:32:14 UTC

A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=2f01d32332494380dc4971717d2757415dcbd27e

commit 2f01d32332494380dc4971717d2757415dcbd27e
Author:     John Hein <jcfyecrayz@liamekaens.com>
AuthorDate: 2023-08-17 10:21:16 +0000
Commit:     Kai Knoblich <kai@FreeBSD.org>
CommitDate: 2023-08-17 10:30:45 +0000

    www/qt5-webengine: Remove unnecessary ulimit calls

    Before linking, which is far at the end of the long building process,
    the ulimit is tried to be set to 4096.  If this fails, it will cause
    the whole linking process to abort and thus lead to a failed build,
    which is the case when qt5-webengine is build with a non-root user in
    a Poudriere jail.

    The problem could be probably solved by substituting "&&" with ";"
    in the affected lines, but in the course of bug 270041 it has become
    apparent that an ulimit of 1024, which is poudriere's default,
    is sufficient. [1]

    This could also be successfully verified with qt5-webengine builds (with
    either ALSA/PULSEAUDIO/SNDIO enabled and DEBUG set/unset) in non-root
    poudriere jail hence remove the ulimit calls to fix the problem.

    PR:             270041
    Reported by:    Tomoaki AOKI, John Hein [1]
    MFH:            2023Q3
    With hat:       kde

 www/qt5-webengine/files/patch-src_core_core__module.pro | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

Comment 15 commit-hook 2023-08-17 10:35:17 UTC

A commit in branch 2023Q3 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=6ee34b44b28c4a593e334f5f21c1d28f372c24b6

commit 6ee34b44b28c4a593e334f5f21c1d28f372c24b6
Author:     John Hein <jcfyecrayz@liamekaens.com>
AuthorDate: 2023-08-17 10:21:16 +0000
Commit:     Kai Knoblich <kai@FreeBSD.org>
CommitDate: 2023-08-17 10:33:35 +0000

    www/qt5-webengine: Remove unnecessary ulimit calls

    Before linking, which is far at the end of the long building process,
    the ulimit is tried to be set to 4096.  If this fails, it will cause
    the whole linking process to abort and thus lead to a failed build,
    which is the case when qt5-webengine is build with a non-root user in
    a Poudriere jail.

    The problem could be probably solved by substituting "&&" with ";"
    in the affected lines, but in the course of bug 270041 it has become
    apparent that an ulimit of 1024, which is poudriere's default,
    is sufficient. [1]

    This could also be successfully verified with qt5-webengine builds (with
    either ALSA/PULSEAUDIO/SNDIO enabled and DEBUG set/unset) in non-root
    poudriere jail hence remove the ulimit calls to fix the problem.

    PR:             270041
    Reported by:    Tomoaki AOKI, John Hein [1]
    MFH:            2023Q3
    With hat:       kde

    (cherry picked from commit 2f01d32332494380dc4971717d2757415dcbd27e)

 www/qt5-webengine/files/patch-src_core_core__module.pro | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

Comment 16 Kai Knoblich 2023-08-17 14:01:41 UTC

Committed to main and MFH'ed to the 2023Q3 branch and the reported issue should now be solved.