This patch allows ports to build with make -jx on multi-CPU machines (where x is typically 1 + the number of CPUs in the system) in order to speed up the build process. All a port maintainer has to do is set the ALLOW_MAKE_JOBS variable in a port Makefile, whereas the user has to set ENABLE_MAKE_JOBS in his /etc/make.conf. If these two variables are set, the code in the patch will get the number of cpus, add 1, then call make with, e.g., -j3. Here's a description of the variables involved: ENABLE_MAKE_JOBS: The master switch that enables or disables the whole thing. The user is supposed to set it in his /etc/make.conf . If this variable isn't set, the ports build like they always did. ALLOW_MAKE_JOBS: Goes into a port's makefile. The port maintainer indicates with it that the port can be built with multiple make jobs. MAKE_JOBS_WHITELIST: Allows the user to override ALLOW_MAKE_JOBS. Any port whose UNIQUENAME is listed in MAKE_JOBS_WHITELIST will have its ALLOW_MAKE_JOBS defined. The user would put something like this in his /etc/make.conf: MAKE_JOBS_WHITELIST=kdebase gtk20 The following are new internal variables I introduced: CPUS: The number of CPUs in the system. MAKE_JOBS_NJOBS: The number of make jobs that will be used. Currently, this is ${CPUS} + 1, which benchmarks have shown to be the most efficient number on my dual-core system. MAKE_JOBS_ARGS: The argument that is passed to make. BUILD_FAILMSG: A message that is printed if the do-build target fails. Note that this variable can be used by any part of the ports system, not just the MAKE_JOBS part. To use it, write code like BUILD_FAILMSG+= "Hello World" Each message added to BUILD_FAILMSG like this will be printed at the end of the do-build stage (if it fails), one paragraph per message. The MAKE_JOBS code currently uses this to inform the user that he needn't bother to send bug reports if MAKE_JOBS_WHITELIST is used. I have tested this patch throughout the X.org 7.2 upgrade process with several hundred port builds, it seems to work well; many ports seem to support building with multiple make jobs.
Responsible Changed From-To: freebsd-ports-bugs->portmgr Affects bsd.port.mk.
Instead of echo-ing through grep, something like ALLOW_MAKE_JOBS= ${MAKE_JOBS_WHITELIST:M${UNIQUENAME}} should be used to avoid spawning off extra processes. Also, I would use the word PARALLEL instead of JOBS. Figuring out the number of CPUs should also be inside the conditional so as not to bother for most ports. But these are all just nit-picking... This is a fine approach for "power-users", but ultimately the ports-maintainers should start adding something as simple as ALL_TARGET= -j`${SYSCTL} -n hw.ncpu` to their ports to keep it all "automagic"... -mi
Mikhail Teterin wrote: > Instead of echo-ing through grep, something like > > ALLOW_MAKE_JOBS= ${MAKE_JOBS_WHITELIST:M${UNIQUENAME}} > > should be used to avoid spawning off extra processes. I'll have to evaluate that, although I think I might have tried that before and found it didn't work... not sure anymore. > Also, I would use the word PARALLEL instead of JOBS. I had that before. I changed it to JOBS because there is ongoing work to make it possible to run several different port builds in parallel, and I wanted to avoid name clashes and confusion. > Figuring out the number of CPUs should also be inside the conditional so as > not to bother for most ports. But these are all just nit-picking... I figured this variable might be useful for other things in the future, so my idea is to establish it as a standard ports variable. > This is a fine approach for "power-users", but ultimately the > ports-maintainers should start adding something as simple as > > ALL_TARGET= -j`${SYSCTL} -n hw.ncpu` > > to their ports to keep it all "automagic"... The WHITELIST part is, yes. The automagic part I've thought of as well: as soon as this patch has made it into the tree, I plan to start petitioning port maintainers to add ALLOW_MAKE_JOBS to their ports. Once this is done, all the user has to do is to set ENABLE_MAKE_JOBS globally (or it could even become the default, although given the conservativity of the FreeBSD project (don't get me wrong, I like that aspect), I don't see that happening). Thanks for your comments. Cheers Benjamin
=D7=A6=D7=D4=CF=D2=CF=CB 21 =D3=C5=D2=D0=C5=CE=D8 2007 09:52 =C4=CF, Benjam= in Lutz =F7=C9 =CE=C1=D0=C9=D3=C1=CC=C9: > The WHITELIST part is, yes. The automagic part I've thought of as well: > as soon as this patch has made it into the tree, I plan to start > petitioning port maintainers to add ALLOW_MAKE_JOBS to their ports. Well, that's the thing. Since maintainers will need to be petitioned anyway= ,=20 they may as well as be asked to modify the ALL_TARGET directly... I=20 personally am already doing that in some of my ports, for example -- most=20 recently in graphics/vips and graphics/nip2. Yours, -mi
Can you change: MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 to: _MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 MAKE_JOBS_NJOBS?= ${_MAKE_JOBS_NJOBS} This would allow my sysutils/bsdadminscripts which implement their own parallel make magic to silently fall back to this patch if it gets committed. Thanks.
On Wednesday 12 March 2008 17:10:47 Dominic Fandrey wrote: > Can you change: > > MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 > > to: > > _MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 > MAKE_JOBS_NJOBS?= ${_MAKE_JOBS_NJOBS} > > > This would allow my sysutils/bsdadminscripts which implement their > own parallel make magic to silently fall back to this patch if it > gets committed. Can you not hook your fallback mechanism into the MAKE_JOBS_ARGS ?= assignment? I'm not in favor of arbitrarily changing the internal mechanics of this patch for the benefit of tools not directly associated with the ports. Cheers Benjamin
Benjamin Lutz wrote: > On Wednesday 12 March 2008 17:10:47 Dominic Fandrey wrote: >> Can you change: >> >> MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 >> >> to: >> >> _MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 >> MAKE_JOBS_NJOBS?= ${_MAKE_JOBS_NJOBS} >> >> >> This would allow my sysutils/bsdadminscripts which implement their >> own parallel make magic to silently fall back to this patch if it >> gets committed. > > Can you not hook your fallback mechanism into the MAKE_JOBS_ARGS ?= > assignment? I'm not in favor of arbitrarily changing the internal > mechanics of this patch for the benefit of tools not directly > associated with the ports. > > Cheers > Benjamin Yes I can, but what if it's used for more than -j one day? By its name it sounds more generic than MAKE_JOBS_NJOBS and I'm afraid overwriting it, might break something one day. Anyway, I will follow your suggestion now and deal with future problems when they arise. Greetings Dominic
Benjamin Lutz wrote: > On Wednesday 12 March 2008 17:10:47 Dominic Fandrey wrote: >> Can you change: >> >> MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 >> >> to: >> >> _MAKE_JOBS_NJOBS!= ${EXPR} ${CPUS} + 1 >> MAKE_JOBS_NJOBS?= ${_MAKE_JOBS_NJOBS} >> >> >> This would allow my sysutils/bsdadminscripts which implement their >> own parallel make magic to silently fall back to this patch if it >> gets committed. > > Can you not hook your fallback mechanism into the MAKE_JOBS_ARGS ?= > assignment? I'm not in favor of arbitrarily changing the internal > mechanics of this patch for the benefit of tools not directly > associated with the ports. > > Cheers > Benjamin I have come to the concluseion that I have to do something like the following in order to hook in clearly: # Determine weather native parallel builds are available. .if !defined(BUILDFLAGS_DUMMY) && (defined(SUBTHREADS) || defined(MAKE_JOBS)) CPUS!= ${MAKE} -DBUILDFLAGS_DUMMY ${.MAKEFLAGS} ${.TARGETS} \ -VCPUS .if ! ${CPUS:M*} .undef CPUS .endif ALLOW_MAKE_JOBS!= ${MAKE} -DBUILDFLAGS_DUMMY ${.MAKEFLAGS} ${.TARGETS} \ -VALLOW_MAKE_JOBS .if ! ${ALLOW_MAKE_JOBS:M*} .undef ALLOW_MAKE_JOBS .endif .endif This has a severe performance punishment. Not having a way of manually setting the amount of jobs makes without such evil trickery makes my life very difficult. I'm running distcc to distribute builds and the number of cores on the main system is not an indicator for a sensible amount of parallel jobs in such a setting.
State Changed From-To: open->closed Similar but different patch committed