Created attachment 146414 [details] patch When building X-based packages, there are long pauses with lots of disk-thrashing when font packages are installed, while fc-cache, mkfontscale, and mkfontdir are run. Oftentimes installing those font packages takes longer than building the actual port, especially with ccache running. I suspect that those steps can be safely skipped in a package-building environment. This patch alters the fc, fcfontsdir, and fontsdir keywords to simply skip those steps when PACKAGE_BUILDING is defined. It would be interesting to see whether it breaks stuff in an exp-run.
I do like that idea, beside we need to think about all the possible impacts (I can't think of a bad one so far) I would like to see what it does on an exp
I think that the provided patch doesn't work
I do not see why it won't work given we do not sanitize the env. At worst the only thing we will have to do it to add PACKAGE_BUILDING to the env when installing deps with pkg
oh I miss read the patch s/%%PACKAGE_BUILDING%%/${PACKAGE_BUILDING} should be done because otherwise packages procudes by the builders (hence installed by the user) will never run the fonts commands
I think this will lead to strange violations/leftovers in qa mode
Created attachment 146429 [details] second patch bapt you're totally right. I don't know why I thought that environment wouldn't get passed through, but I've set it to use $PACKAGE_BUILDING instead.
(In reply to Antoine Brodin from comment #5) > I think this will lead to strange violations/leftovers in qa mode I've been building some ports in various poudriere jails and I haven't seen any qa errors yet.
Created attachment 146432 [details] patch Now that it just uses $PACKAGE_BUILDING from the environment, there's no need to touch bsd.port.mk. Updated patch to remove bsd.port.mk stuff.
Take for QAT exp-run
Hi, Exp-run results are available at: http://package23.nyi.freebsd.org/build.html?mastername=91amd64-default-PR193082&build=2014-09-01_23h55m04s There was 1 new "hard" failure: + {"origin"=>"www/chromium", "pkgname"=>"chromium-37.0.2062.94", "phase"=>"stage", "errortype"=>"???"} And 15 new violations: + {"origin"=>"audio/amarok-kde4", "pkgname"=>"amarok-2.8.0_3", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"databases/grass", "pkgname"=>"grass-6.4.4_1,2", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"devel/py-stevedore", "pkgname"=>"py27-stevedore-0.15", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"graphics/gegl", "pkgname"=>"gegl-0.2.0_9", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"graphics/gnash", "pkgname"=>"gnash-0.8.10_13", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"graphics/oyranos", "pkgname"=>"oyranos-0.9.5_2", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"graphics/p5-GraphViz2", "pkgname"=>"p5-GraphViz2-2.32", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"graphics/qgis", "pkgname"=>"qgis-2.4.0_1", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"graphics/synfigstudio", "pkgname"=>"synfigstudio-0.63.05_4", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"lang/racket", "pkgname"=>"racket-6.1", "phase"=>"stage_fs_violation", "errortype"=>"stage"} + {"origin"=>"math/gretl", "pkgname"=>"gretl-1.9.13_4", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"net-im/kopete-kde4", "pkgname"=>"kopete-4.12.5_5", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"print/kpdftool", "pkgname"=>"kpdftool-0.23.1_3", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"textproc/xqilla", "pkgname"=>"xqilla-2.3.0_2,1", "phase"=>"build_fs_violation", "errortype"=>"mtree"} + {"origin"=>"www/kdewebdev4", "pkgname"=>"kdewebdev-4.12.5_1", "phase"=>"build_fs_violation", "errortype"=>"mtree"} Maybe chromium can be changed to not depend on a fonts.dir file? For the violations, maybe [ -z "${PACKAGE_BUILDING}" ] can be replaced with [ -z "${PACKAGE_BUILDING}" -o -n "${DEVELOPER_MODE}" ] this hack would run the commands in poudriere when qa mode is enabled, not sure if it's sustainable and not sure about tinderbox too ??
(In reply to Antoine Brodin from comment #10) > [ -z "${PACKAGE_BUILDING}" -o -n "${DEVELOPER_MODE}" ] > > this hack would run the commands in poudriere when qa mode is enabled, not > sure if it's sustainable and not sure about tinderbox too ?? I would recommend DEVELOPER instead. DEVELOPER_MODE is a pkg(8) thing and mostly obsoleted now.
(In reply to Bryan Drewery from comment #11) > I would recommend DEVELOPER instead. DEVELOPER_MODE is a pkg(8) thing and > mostly obsoleted now. Okay, I'll complete my testing with DEVELOPER instead of DEVELOPER_MODE.
I will redo the test, but if i remember correctly when i did a "printenv" from the keyword, i wasn't seeing DEVELOPER
Extract from a testport build log, I don't have DEVELOPER in env when installing the depends: =======================<phase: lib-depends >============================ ===> libevtx-a.20140901 depends on shared library: libevt.so - not found ===> Verifying for libevt.so in /usr/ports/devel/libevt ===> Installing existing package /packages/All/libevt-a.20140831.txz [92amd64-custom] Installing libevt-a.20140831... done printenv STATUS=1 OPSYS=FreeBSD ARCH=amd64 POSIXLY_CORRECT=1 SAVED_TERM=screen NO_WARNING_PKG_INSTALL_EOL=yes MASTERMNT=/poudriere/data/.m/92amd64-custom/ref PKG_PREFIX=/usr/local MAIL=/var/mail/root MAKEFLAGS= ARCH=amd64 OPSYS=FreeBSD OSREL=9.2 OSVERSION=902001 SYSTEMVERSION= __MKLVL__=1 PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin POUDRIERE_BUILD_TYPE=bulk OSREL=9.2 HTTP_USER_AGENT=pkg/1.3.7 SYSTEMVERSION= PWD=/usr/ports/devel/libevtx MASTERNAME=92amd64-custom DEVELOPER_MODE=yes USER=root HOME=/root POUDRIERE_VERSION=3.1-pre USE_PACKAGE_DEPENDS_ONLY=1 TRYBROKEN=yes UNAME_r=9.2-RELEASE-p10 LOCALBASE=/usr/local PACKAGE_BUILDING=yes OSVERSION=902001 UNAME_v=FreeBSD 9.2-RELEASE-p10 BLOCKSIZE=K ===> Returning to build of libevtx-a.20140901 ===> libevtx-a.20140901 depends on shared library: libintl.so - found (/usr/local/lib/libintl.so.9) ===> libevtx-a.20140901 depends on shared library: libiconv.so.3 - found (/usr/local/lib/libiconv.so.3) =========================================================================== ====>> Status for build devel/libevtx: configure ====>> Recording filesystem state for prebuild... done
(In reply to Antoine Brodin from comment #14) > Extract from a testport build log, I don't have DEVELOPER in env when > installing the depends: > > =======================<phase: lib-depends >============================ > ===> libevtx-a.20140901 depends on shared library: libevt.so - not found > ===> Verifying for libevt.so in /usr/ports/devel/libevt > ===> Installing existing package /packages/All/libevt-a.20140831.txz > [92amd64-custom] Installing libevt-a.20140831... done > printenv > STATUS=1 > OPSYS=FreeBSD > ARCH=amd64 > POSIXLY_CORRECT=1 > SAVED_TERM=screen > NO_WARNING_PKG_INSTALL_EOL=yes > MASTERMNT=/poudriere/data/.m/92amd64-custom/ref > PKG_PREFIX=/usr/local > MAIL=/var/mail/root > MAKEFLAGS= ARCH=amd64 OPSYS=FreeBSD OSREL=9.2 OSVERSION=902001 SYSTEMVERSION= > __MKLVL__=1 > PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin: > /root/bin > POUDRIERE_BUILD_TYPE=bulk > OSREL=9.2 > HTTP_USER_AGENT=pkg/1.3.7 > SYSTEMVERSION= > PWD=/usr/ports/devel/libevtx > MASTERNAME=92amd64-custom > DEVELOPER_MODE=yes Well DEVELOPER_MODE here is not guaranteed to stay. I forget why I left it in all phases, but am thinking now it only makes sense for 'package' phase. Why use DEVELOPER* at all? PACKAGE_BUILDING seems enough to me to prevent building caches. I don't see the need to build them when doing port testing. It is known which files/dirs it will touch.
I can confirm that with "-n ${DEVELOPER}" the QA errors are present and they disappear with "-n ${DEVELOPER_MODE}". Bryan the issue as I understand it is that there are some ports that seem to run the fontconfig stuff themselves if the cache isn't already there. This creates QA violations where ${PREFIX} is modified during build. My understanding is then that, to utilize this PR, the options are: 1) Accept that there are 14 ports that will produce mtree QA errors 2) Count on poudriere exporting a specific variable when running in testport vs bulk mode 3) Teach the QA script to ignore changes in ${LOCALBASE}/var/db/fontconfig/*.cache-4
(In reply to Adam Weinberger from comment #16) > I can confirm that with "-n ${DEVELOPER}" the QA errors are present and they > disappear with "-n ${DEVELOPER_MODE}". > > Bryan the issue as I understand it is that there are some ports that seem to > run the fontconfig stuff themselves if the cache isn't already there. This > creates QA violations where ${PREFIX} is modified during build. > > My understanding is then that, to utilize this PR, the options are: > > 1) Accept that there are 14 ports that will produce mtree QA errors > 2) Count on poudriere exporting a specific variable when running in testport > vs bulk mode > 3) Teach the QA script to ignore changes in > ${LOCALBASE}/var/db/fontconfig/*.cache-4 I'm leaning on #3 when PACKAGE_BUILDING is set. Tinderbox will barf on this, but it already is broken in many other ways.
Now that massive concurrency in port-building is an established thing, there's just no real benefit to shaving 10 seconds off builds.