Bug 199572 - ports-mgmt/portmaster: useless processing of depends over and over again
Summary: ports-mgmt/portmaster: useless processing of depends over and over again
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Stefan Esser
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-04-21 01:52 UTC by torsten.eichstaedt
Modified: 2017-12-18 17:05 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description torsten.eichstaedt 2015-04-21 01:52:56 UTC
or: gg:Recursive 'make' considered harmful (and not using 'make' too)

how to reproduce: e.g. # portmaster -F kde
(download all distfiles needed to build kde)

When portmaster recurses through the dependecies, it processes entries on which more than one other entries depends over and over again, which is fairly useless provided nothing changes during one run, increasing (by a factor of estimated several 10^x) the runtime.

IMHO the correct way to handle the graph of dependencies is what (I guess) poudriere (or 'make' when it is given a correct makefile) does: the graph needs to be stratified before diving into the dependencies (side benefit: allows parallel processing), and then must be processed /bottom-up/.

In contrast, (from to output of portmaster I guess that) portmaster processes the graph top-down-depth-first ("by-feet", i.e. does not use 'make') and does not remember what was already finished.  In the example above the "basic" ports like automoc, qmake4, cmake, several py-... etc. are processed several hundred (1000's?) times. 

Thus, if there's a way for portmaster to contruct a (or use an already existent)  Makefile for a given task and then launch 'make' to find a (optimal) way to do the actual work, I'd say that's much better than to do it "by-feet" in a sub-optimal way.

For the example above, this magic should be sufficient:
"portsdir=/usr/ports; target=x11/kde4; cd $portsdir/$target; make fetch-recursive"
Basically, most of what 'portmaster' would have to do is to find the right directory name for a given target (x11/kde4 <-> kde).
 
Comparison: after more than one hour, portmaster was still recursing through the dependencies and downloaded nothing, whereas direct use of 'make' as above started to download the 1st missing distfile after a few seconds.
Comment 1 John Marino freebsd_committer 2015-04-21 13:46:34 UTC
nobody is maintaining this port:
http://www.freshports.org/ports-mgmt/portmaster

Frankly, very few committers care about it (Personally I don't see the point since pkg(8) arrived with Poudriere)


If you have the capability to fix it, I'd suggest becoming the maintainer.  Otherwise, don't be surprised if this PR doesn't get a lot of love.
Comment 2 John Marino freebsd_committer 2016-02-10 18:43:05 UTC
PM has new maintainer, reassign PR.
Comment 3 Stefan Esser freebsd_committer 2017-12-18 17:05:11 UTC
I plan to complete rewrite that part of portmaster, but this will be a major effort and I'm not sure, when I get to it.

The current version was designed to work with the "old" ports system, more than a decade ago. It was extended in a backwards compatible way, whenever the ports and packages systems evolved (still supporting old features and ways).

A redesign is required, and I have plans for it, but portmaster is so complex and hard to understand in detail (due to implicit and undocumented effects of global variables on the sequence of actions taken), that any change comes with a significant risk of regressions - a major rewrite much more so.