or: gg:Recursive 'make' considered harmful (and not using 'make' too)
how to reproduce: e.g. # portmaster -F kde
(download all distfiles needed to build kde)
When portmaster recurses through the dependecies, it processes entries on which more than one other entries depends over and over again, which is fairly useless provided nothing changes during one run, increasing (by a factor of estimated several 10^x) the runtime.
IMHO the correct way to handle the graph of dependencies is what (I guess) poudriere (or 'make' when it is given a correct makefile) does: the graph needs to be stratified before diving into the dependencies (side benefit: allows parallel processing), and then must be processed /bottom-up/.
In contrast, (from to output of portmaster I guess that) portmaster processes the graph top-down-depth-first ("by-feet", i.e. does not use 'make') and does not remember what was already finished. In the example above the "basic" ports like automoc, qmake4, cmake, several py-... etc. are processed several hundred (1000's?) times.
Thus, if there's a way for portmaster to contruct a (or use an already existent) Makefile for a given task and then launch 'make' to find a (optimal) way to do the actual work, I'd say that's much better than to do it "by-feet" in a sub-optimal way.
For the example above, this magic should be sufficient:
"portsdir=/usr/ports; target=x11/kde4; cd $portsdir/$target; make fetch-recursive"
Basically, most of what 'portmaster' would have to do is to find the right directory name for a given target (x11/kde4 <-> kde).
Comparison: after more than one hour, portmaster was still recursing through the dependencies and downloaded nothing, whereas direct use of 'make' as above started to download the 1st missing distfile after a few seconds.
nobody is maintaining this port:
Frankly, very few committers care about it (Personally I don't see the point since pkg(8) arrived with Poudriere)
If you have the capability to fix it, I'd suggest becoming the maintainer. Otherwise, don't be surprised if this PR doesn't get a lot of love.
PM has new maintainer, reassign PR.
I plan to complete rewrite that part of portmaster, but this will be a major effort and I'm not sure, when I get to it.
The current version was designed to work with the "old" ports system, more than a decade ago. It was extended in a backwards compatible way, whenever the ports and packages systems evolved (still supporting old features and ways).
A redesign is required, and I have plans for it, but portmaster is so complex and hard to understand in detail (due to implicit and undocumented effects of global variables on the sequence of actions taken), that any change comes with a significant risk of regressions - a major rewrite much more so.
I have been using a rewrite of portmaster for nearly one year, which implements a planning phase in which the port makefiles and information from the pkg database is used to gather information in associative arrays and develop a build strategy based on this information. That version does not only build ports in the same way and order as the official portmaster port, but it does also offer build modes where ports are build in a clean jail (and either used to just create a repository or installed at the end of the run).
While that version works for me, I have now started to convert it from a shell script to LUA, to be able to take advantage of the data structures it provides. This will offer many opportunities for optimizations, e.g. to find the best build strategy for each invocation, but also to improve the performance.
I plan to publish that version before the end of the first quarter of 2020.
To both John and Stefan: I do appreciate what you guys are doing.
And yes, I was not aware that portmaster did not have a maintainer. Since it was (is still?) mentioned in the FreeBSD Handbook, I assumed it's use is implicitely considered standard behaviour. I made the mistake to conclude from the intuitive name of the port to it's quality... ;) and did not know about it's history. Stefan, you took a good decision to change to LUA.