Bug 237290
| Summary: | build(7): PORTS_MODULES does not build in parallel | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Ivan Rozhuk <rozhuk.im> | ||||||||||||||||||
| Component: | conf | Assignee: | freebsd-bugs (Nobody) <bugs> | ||||||||||||||||||
| Status: | Open --- | ||||||||||||||||||||
| Severity: | Affects Some People | CC: | Alexander88207, agh, bdrewery, grahamperrin, rozhuk.im, tatsuki_makino | ||||||||||||||||||
| Priority: | --- | Keywords: | needs-qa, performance | ||||||||||||||||||
| Version: | CURRENT | Flags: | rozhuk.im:
maintainer-feedback-
|
||||||||||||||||||
| Hardware: | Any | ||||||||||||||||||||
| OS: | Any | ||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||
|
Description
Ivan Rozhuk
2019-04-14 23:42:23 UTC
Created attachment 203705 [details]
patch
Works for me.
Known issue: if in /etc/make.conf set WRKDIRPREFIX?=/tmp/ports or WRKDIRPREFIX=/tmp/ports then it overwrites value that set in kern.post.mk (/usr/obj...).
Don't remove -B please. We do not want to pass -j down to the ports build. Ports does not support -j. It uses an internal mechanism of MAKE_JOBS. Some ports support it and others do not. Passing down -j doesn't do what you would expect. Separate issue, but it's also not supported to build multiple ports at once simply via 'make'. Something like Poudriere is needed for that. # MAKE_JOBS_NUMBER # - Override the number of make jobs to be used. User settable. # MAKE_JOBS_NUMBER_LIMIT # - Set a limit for maximum number of make jobs allowed to be # used. MAKE_JOBS_NUMBER defaults to the number of CPU threads on the system as long as you do not set DISABLE_MAKE_JOBS and the port isn't MAKE_JOBS_UNSAFE. (In reply to Bryan Drewery from comment #2) Keeping -B is terrible idea: compilation takes too many time. -j can be filtered out here: MAKEFLAGS="${MAKEFLAGS:M*:tW:S/^-m /-m_/g:S/ -m / -m_/g:tw:N-m_*:NMK_AUTO_OBJ=*}" also I can extract "-j N" N arg and pass it to MAKE_JOBS_NUMBER. I dont know why -m filered out here. Now MAKEFLAGS contains -j 8 -s -J15,16 and some other strange staff. Probably better not pass MAKEFLAGS or keep only known good flags, like -s. (In reply to rozhuk.im from comment #3) MAKEFLAGS: -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64 Created attachment 203755 [details]
more clean MAKEFLAGS
src MAKEFLAGS: -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64
Passed to ports:
MAKE_JOBS_NUMBER: 8
MAKEFLAGS: -s -D ALWAYS_CHECK_MAKE -s -D ALWAYS_CHECK_MAKE -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64
Created attachment 205980 [details]
more env vars to ports
Pass more ports specific settings via env.
Now all ports build into kernel OBJ dir.
maintainer timeout (In reply to rozhuk.im from comment #3) What's a terrible idea is making a change you don't understand. The -j flag has 0 impact on the ports build except to BREAK it. -j is not how ports are allowed to use make jobs. These fields are: # MAKE_JOBS_UNSAFE # - Disallow multiple jobs even when user set a global override. # To be used with known bad ports. # DISABLE_MAKE_JOBS # - Set to disable the multiple jobs feature. User settable. # MAKE_JOBS_NUMBER # - Override the number of make jobs to be used. User settable. # MAKE_JOBS_NUMBER_LIMIT # - Set a limit for maximum number of make jobs allowed to be # used. 77 # Read ports defaults. 78 PORTSDIR?= $(${MAKE} -V PORTSDIR) 79 DISTDIR?= $(${MAKE} -V DISTDIR) 80 PORT_DBDIR?= $(${MAKE} -V PORT_DBDIR) 81 OPTIONS_SET?= $(${MAKE} -V OPTIONS_SET) 82 OPTIONS_UNSET?= $(${MAKE} -V OPTIONS_UNSET) 83 NO_CCACHE?= $(${MAKE} -V NO_CCACHE) 84 CCACHE_DIR?= $(${MAKE} -V CCACHE_DIR) 85 WITH_CCACHE_BUILD?= $(${MAKE} -V WITH_CCACHE_BUILD) 86 DEFAULT_VERSIONS?= $(${MAKE} -V DEFAULT_VERSIONS) 87 WITH_DEBUG_PORTS?= $(${MAKE} -V WITH_DEBUG_PORTS) This is completely redundant and not needed. This is no such thing as maintainer timeout on this code as it is not ports. I suggest if you want this fixed you explore why the MAKE_JOBS settings are not used and why those env vars were needed when they are already available in ports. Passing -j1 instead of -B seems to work when using a proof of concept test. Can you try that please? (In reply to Bryan Drewery from comment #9) Probably there is misunderstanding. MAKEFLAGS_PORTS=${MAKEFLAGS:M*:tW:S/^-m /-m_/g:S/ -m / -m_/g:S/^-j /-m_/g:S/ -j / -m_/g:S/^-J /-m_/g:S/ -J / -m_/g:tw:N-m_*:NMK_AUTO_OBJ=*} As I remember this thing removes -j args from flags. MAKE_JOBS_NUMBER=${.MAKE.JOBS} - is passed to ports after my patch. Created attachment 234277 [details]
add FLAVOR support
Now allow to use FLAVOR:
PORTS_MODULES= graphics/gpu-firmware-radeon-kmod@caicos
If the -B of make -B is only to prevent parallel execution, I think it is excessive.
For example, the value of .MAKE.MODE changes from normal to compat.
Just make -- and make -j 1 also makes a difference.
Options like -J 15,16 come across in the MAKEFLAGS environment variable.
Wouldn't it be better to pass MAKEFLAGS with the -[Jj] option dropped in the environment variable?
MAKEFLAGS="${MAKEFLAGS:M*:C/((^| )-[Jjm]) /\1/gW:N-[Jjm]*:NMK_AUTO_OBJ=*}"
The number of jobs when building ports is left to MAKE_JOBS_NUMBER on the ports side.
If make buildkernel is executed with -j 1, but MAKE_JOBS_NUMBER is 4, then build the port with 4 jobs.
(In reply to Ivan Rozhuk from comment #11)
:M*@*:C/.*@/FLAVOR=/g is great :)
Created attachment 234307 [details]
patch for stable/12
This is a patch for stable/12, but the buildkernel and installkernel on stable/12 went fine.
The flat plain of load average 1.00 did not even occur when -j 3 buildkernel.
The ports build was invoked as follows.
===> Ports module graphics/drm-fbsd12.0-kmod (all)
cd ${PORTSDIR:-/usr/ports}/graphics/drm-fbsd12.0-kmod; env -u CC -u CXX -u CPP -u MAKESYSPATH -u MK_AUTO_OBJ -u MAKEOBJDIR MAKEFLAGS="-D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNCONF=GENERIC KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64" SYSDIR=/usr/src/sys PATH=/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin SRC_BASE=/usr/src OSVERSION=1203507 WRKDIRPREFIX=/usr/obj/usr/src/amd64.amd64/sys/GENERIC make clean build
===> Ports module graphics/gpu-firmware-radeon-kmod@caicos (all)
cd ${PORTSDIR:-/usr/ports}/graphics/gpu-firmware-radeon-kmod; env -u CC -u CXX -u CPP -u MAKESYSPATH -u MK_AUTO_OBJ -u MAKEOBJDIR MAKEFLAGS="-D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNCONF=GENERIC KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64" SYSDIR=/usr/src/sys PATH=/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin SRC_BASE=/usr/src OSVERSION=1203507 WRKDIRPREFIX=/usr/obj/usr/src/amd64.amd64/sys/GENERIC make FLAVOR=caicos clean build
Created attachment 247153 [details]
patch for 15-CURRENT
Hello,
Attached patch based on Tatsuki Makino's. Having ${PORTS_MODULES} handle flavours is a massive Quality of Life (QOL) improvement for those of use building on one host, and distributing system updates via NFS.
(In reply to Alastair Hogge from comment #14) $${PORTSDIR -> ${PORTSDIR Created attachment 247166 [details]
patch 14/stable
(In reply to Ivan Rozhuk from comment #15) > $${PORTSDIR -> ${PORTSDIR Results in: > MAKE="make" sh /usr/src/sys/conf/newvers.sh KOTO > --- vers.o --- > cc -target x86_64-unknown-freebsd15.0 --sysroot=/net/fafnir/obj/usr/src/amd64.amd64/tmp -B/net/fafnir/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -fdebug-prefix-map=./i386=/usr/src/sys/i386/include -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error=tautological-compare -Wno-error=empty-body -Wno-error=parentheses-equality -Wno-error=unused-function -Wno-error=pointer-sign -Wno-error=shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -mno-aes -mno-avx -std=gnu99 -Werror vers.c > --- kernel --- > linking kernel > objcopy --strip-debug kernel > text data bss dec hex filename > 11990200 1052822 3137536 16180558 0xf6e54e kernel > make[2]: Unknown modifier "-/usr/ports" > make[2]: Unknown modifier "-/usr/ports" > make[2]: Unknown modifier "-/usr/ports" > --- all --- > ===> Ports module graphics/drm-515-kmod (all) > cd /graphics/drm-515-kmod; env -u CC -u CXX -u CPP -u MAKESYSPATH -u MK_AUTO_OBJ -u MAKEOBJDIR MAKEFLAGS="-D NO_MODULES_OBJ KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64" SYSDIR=/usr/src/sys PATH=/net/fafnir/obj/usr/src/amd64.amd64/tmp/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/usr/sbin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/usr/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin SRC_BASE=/usr/src OSVERSION=1500007 WRKDIRPREFIX=/net/fafnir/obj/usr/src/amd64.amd64/sys/KOTO make clean build > cd: /graphics/drm-515-kmod: No such file or directory > *** [all] Error code 2 > > make[2]: stopped in /net/fafnir/obj/usr/src/amd64.amd64/sys/KOTO > 1 error > > make[2]: stopped in /net/fafnir/obj/usr/src/amd64.amd64/sys/KOTO > 42.12 real 281.49 user 132.20 sys > > make[1]: stopped in /usr/src > > make: stopped in /usr/src (In reply to Alastair Hogge from comment #17) >(In reply to Ivan Rozhuk from comment #15) >> $${PORTSDIR -> ${PORTSDIR $${PORTSDIR:-/usr/ports} -> ${PORTSDIR:U/usr/ports} It seems that the way default values are given has been changed from sh to make side? Created attachment 252906 [details]
patch for stable/14
I don't know how to do it best, but I decided to apply this to stable/14.
And the command to generate the PORTS_MODULES is as follows.
# generate from installed files.
find -s /boot/modules/ -name '*.ko' -exec pkg which -q {} \; | sort -u | xargs -- sh -c 'for i ; do { pkg query %o "$i" ; pkg query "%o@%At@%Av" "$i" | sed -ne "/@flavor@/{s,@flavor@,@,;p;};" ; } | tail -n 1 ; done' sh
# generate from loaded modules
kldstat | sed -e '1d; s,^[[:space:]]*,,;' | cut -f 5 -w | xargs -J % -n 1 -- find /boot/ -name % | xargs -- pkg which -q | sort -u | xargs -- sh -c 'for i ; do { pkg query %o "$i" ; pkg query "%o@%At@%Av" "$i" | sed -ne "/@flavor@/{s,@flavor@,@,;p;};" ; } | tail -n 1 ; done' sh
|
