Bug 199572 - ports-mgmt/portmaster: useless processing of depends over and over again
Summary: ports-mgmt/portmaster: useless processing of depends over and over again
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Stefan Esser
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-04-21 01:52 UTC by torsten.eichstaedt
Modified: 2020-01-25 22:29 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description torsten.eichstaedt 2015-04-21 01:52:56 UTC
or: gg:Recursive 'make' considered harmful (and not using 'make' too)

how to reproduce: e.g. # portmaster -F kde
(download all distfiles needed to build kde)

When portmaster recurses through the dependecies, it processes entries on which more than one other entries depends over and over again, which is fairly useless provided nothing changes during one run, increasing (by a factor of estimated several 10^x) the runtime.

IMHO the correct way to handle the graph of dependencies is what (I guess) poudriere (or 'make' when it is given a correct makefile) does: the graph needs to be stratified before diving into the dependencies (side benefit: allows parallel processing), and then must be processed /bottom-up/.

In contrast, (from to output of portmaster I guess that) portmaster processes the graph top-down-depth-first ("by-feet", i.e. does not use 'make') and does not remember what was already finished.  In the example above the "basic" ports like automoc, qmake4, cmake, several py-... etc. are processed several hundred (1000's?) times. 

Thus, if there's a way for portmaster to contruct a (or use an already existent)  Makefile for a given task and then launch 'make' to find a (optimal) way to do the actual work, I'd say that's much better than to do it "by-feet" in a sub-optimal way.

For the example above, this magic should be sufficient:
"portsdir=/usr/ports; target=x11/kde4; cd $portsdir/$target; make fetch-recursive"
Basically, most of what 'portmaster' would have to do is to find the right directory name for a given target (x11/kde4 <-> kde).
 
Comparison: after more than one hour, portmaster was still recursing through the dependencies and downloaded nothing, whereas direct use of 'make' as above started to download the 1st missing distfile after a few seconds.
Comment 1 John Marino freebsd_committer 2015-04-21 13:46:34 UTC
nobody is maintaining this port:
http://www.freshports.org/ports-mgmt/portmaster

Frankly, very few committers care about it (Personally I don't see the point since pkg(8) arrived with Poudriere)


If you have the capability to fix it, I'd suggest becoming the maintainer.  Otherwise, don't be surprised if this PR doesn't get a lot of love.
Comment 2 John Marino freebsd_committer 2016-02-10 18:43:05 UTC
PM has new maintainer, reassign PR.
Comment 3 Stefan Esser freebsd_committer 2017-12-18 17:05:11 UTC
I plan to complete rewrite that part of portmaster, but this will be a major effort and I'm not sure, when I get to it.

The current version was designed to work with the "old" ports system, more than a decade ago. It was extended in a backwards compatible way, whenever the ports and packages systems evolved (still supporting old features and ways).

A redesign is required, and I have plans for it, but portmaster is so complex and hard to understand in detail (due to implicit and undocumented effects of global variables on the sequence of actions taken), that any change comes with a significant risk of regressions - a major rewrite much more so.
Comment 4 Stefan Esser freebsd_committer 2020-01-25 22:29:21 UTC
I have been using a rewrite of portmaster for nearly one year, which implements a planning phase in which the port makefiles and information from the pkg database is used to gather information in associative arrays and develop a build strategy based on this information. That version does not only build ports in the same way and order as the official portmaster port, but it does also offer build modes where ports are build in a clean jail (and either used to just create a repository or installed at the end of the run).

While that version works for me, I have now started to convert it from a shell script to LUA, to be able to take advantage of the data structures it provides. This will offer many opportunities for optimizations, e.g. to find the best build strategy for each invocation, but also to improve the performance.

I plan to publish that version before the end of the first quarter of 2020.