Bug 237290

Summary: [PATCH] build(7): PORTS_MODULES does not build in parallel
Product: Base System Reporter: rozhuk.im
Component: confAssignee: freebsd-bugs mailing list <bugs>
Status: Open ---    
Severity: Affects Some People CC: bdrewery, grahamperrin, rozhuk.im
Priority: --- Keywords: needs-qa, patch, performance
Version: CURRENT   
Hardware: Any   
OS: Any   
Description Flags
more clean MAKEFLAGS
more env vars to ports rozhuk.im: maintainer-approval?

Description rozhuk.im 2019-04-14 23:42:23 UTC
Then some ports set via PORTS_MODULES then they builded with single thread, with takes a lot of time.

Add to /etc/make.conf
PORTS_MODULES+= graphics/drm-fbsd12.0-kmod

build kernel:
/usr/bin/time -h make -j`getconf NPROCESSORS_ONLN` -s -DALWAYS_CHECK_MAKE buildkernel

top show only 1 cpu usage during drm-fbsd12.0-kmod port building.
Comment 1 rozhuk.im 2019-04-16 01:12:18 UTC
Created attachment 203705 [details]

Works for me.

Known issue:  if in /etc/make.conf set WRKDIRPREFIX?=/tmp/ports or WRKDIRPREFIX=/tmp/ports then it overwrites value that set in kern.post.mk (/usr/obj...).
Comment 2 Bryan Drewery freebsd_committer 2019-04-16 21:41:01 UTC
Don't remove -B please. We do not want to pass -j down to the ports build. Ports does not support -j. It uses an internal mechanism of MAKE_JOBS. Some ports support it and others do not. Passing down -j doesn't do what you would expect. 

Separate issue, but it's also not supported to build multiple ports at once simply via 'make'. Something like Poudriere is needed for that.

#               - Override the number of make jobs to be used.  User settable.
#               - Set a limit for maximum number of make jobs allowed to be
#                 used.

MAKE_JOBS_NUMBER defaults to the number of CPU threads on the system as long as you do not set DISABLE_MAKE_JOBS and the port isn't MAKE_JOBS_UNSAFE.
Comment 3 rozhuk.im 2019-04-16 21:51:22 UTC
(In reply to Bryan Drewery from comment #2)

Keeping -B is terrible idea: compilation takes too many time.

-j can be filtered out here:
MAKEFLAGS="${MAKEFLAGS:M*:tW:S/^-m /-m_/g:S/ -m / -m_/g:tw:N-m_*:NMK_AUTO_OBJ=*}"
also I can extract "-j N" N arg and pass it to MAKE_JOBS_NUMBER.
I dont know why -m filered out here.

Now MAKEFLAGS contains -j 8 -s -J15,16 and some other strange staff.
Probably better not pass MAKEFLAGS or keep only known good flags, like -s.
Comment 4 rozhuk.im 2019-04-16 21:59:30 UTC
(In reply to rozhuk.im from comment #3)
MAKEFLAGS:  -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64
Comment 5 rozhuk.im 2019-04-17 21:30:26 UTC
Created attachment 203755 [details]
more clean MAKEFLAGS

src MAKEFLAGS:  -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64

Passed to ports:
Comment 6 rozhuk.im 2019-07-22 04:10:43 UTC
Created attachment 205980 [details]
more env vars to ports

Pass more ports specific settings via env.
Now all ports build into kernel OBJ dir.