Bug 237290 - [PATCH] build(7): PORTS_MODULES does not build in parallel
Summary: [PATCH] build(7): PORTS_MODULES does not build in parallel
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: conf (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords: needs-qa, patch, performance
Depends on:
Blocks:
 
Reported: 2019-04-14 23:42 UTC by rozhuk.im
Modified: 2019-08-18 20:17 UTC (History)
3 users (show)

See Also:


Attachments
patch (968 bytes, patch)
2019-04-16 01:12 UTC, rozhuk.im
no flags Details | Diff
more clean MAKEFLAGS (1.53 KB, patch)
2019-04-17 21:30 UTC, rozhuk.im
no flags Details | Diff
more env vars to ports (2.44 KB, patch)
2019-07-22 04:10 UTC, rozhuk.im
rozhuk.im: maintainer-approval?
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description rozhuk.im 2019-04-14 23:42:23 UTC
Then some ports set via PORTS_MODULES then they builded with single thread, with takes a lot of time.

Reproduce:
Add to /etc/make.conf
PORTS_MODULES+= graphics/drm-fbsd12.0-kmod

build kernel:
/usr/bin/time -h make -j`getconf NPROCESSORS_ONLN` -s -DALWAYS_CHECK_MAKE buildkernel

top show only 1 cpu usage during drm-fbsd12.0-kmod port building.
Comment 1 rozhuk.im 2019-04-16 01:12:18 UTC
Created attachment 203705 [details]
patch

Works for me.

Known issue:  if in /etc/make.conf set WRKDIRPREFIX?=/tmp/ports or WRKDIRPREFIX=/tmp/ports then it overwrites value that set in kern.post.mk (/usr/obj...).
Comment 2 Bryan Drewery freebsd_committer 2019-04-16 21:41:01 UTC
Don't remove -B please. We do not want to pass -j down to the ports build. Ports does not support -j. It uses an internal mechanism of MAKE_JOBS. Some ports support it and others do not. Passing down -j doesn't do what you would expect. 

Separate issue, but it's also not supported to build multiple ports at once simply via 'make'. Something like Poudriere is needed for that.

# MAKE_JOBS_NUMBER
#               - Override the number of make jobs to be used.  User settable.
# MAKE_JOBS_NUMBER_LIMIT
#               - Set a limit for maximum number of make jobs allowed to be
#                 used.

MAKE_JOBS_NUMBER defaults to the number of CPU threads on the system as long as you do not set DISABLE_MAKE_JOBS and the port isn't MAKE_JOBS_UNSAFE.
Comment 3 rozhuk.im 2019-04-16 21:51:22 UTC
(In reply to Bryan Drewery from comment #2)

Keeping -B is terrible idea: compilation takes too many time.

-j can be filtered out here:
MAKEFLAGS="${MAKEFLAGS:M*:tW:S/^-m /-m_/g:S/ -m / -m_/g:tw:N-m_*:NMK_AUTO_OBJ=*}"
also I can extract "-j N" N arg and pass it to MAKE_JOBS_NUMBER.
I dont know why -m filered out here.

Now MAKEFLAGS contains -j 8 -s -J15,16 and some other strange staff.
Probably better not pass MAKEFLAGS or keep only known good flags, like -s.
Comment 4 rozhuk.im 2019-04-16 21:59:30 UTC
(In reply to rozhuk.im from comment #3)
MAKEFLAGS:  -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64
Comment 5 rozhuk.im 2019-04-17 21:30:26 UTC
Created attachment 203755 [details]
more clean MAKEFLAGS

src MAKEFLAGS:  -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64

Passed to ports:
MAKE_JOBS_NUMBER: 8
MAKEFLAGS: -s -D ALWAYS_CHECK_MAKE -s -D ALWAYS_CHECK_MAKE -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64
Comment 6 rozhuk.im 2019-07-22 04:10:43 UTC
Created attachment 205980 [details]
more env vars to ports

Pass more ports specific settings via env.
Now all ports build into kernel OBJ dir.