Bug 199572 - ports-mgmt/portmaster: useless processing of depends over and over again
Summary: ports-mgmt/portmaster: useless processing of depends over and over again
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Stefan Eßer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-04-21 01:52 UTC by Walter von Entferndt
Modified: 2020-08-30 23:43 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Walter von Entferndt 2015-04-21 01:52:56 UTC
or: gg:Recursive 'make' considered harmful (and not using 'make' too)

how to reproduce: e.g. # portmaster -F kde
(download all distfiles needed to build kde)

When portmaster recurses through the dependecies, it processes entries on which more than one other entries depends over and over again, which is fairly useless provided nothing changes during one run, increasing (by a factor of estimated several 10^x) the runtime.

IMHO the correct way to handle the graph of dependencies is what (I guess) poudriere (or 'make' when it is given a correct makefile) does: the graph needs to be stratified before diving into the dependencies (side benefit: allows parallel processing), and then must be processed /bottom-up/.

In contrast, (from to output of portmaster I guess that) portmaster processes the graph top-down-depth-first ("by-feet", i.e. does not use 'make') and does not remember what was already finished.  In the example above the "basic" ports like automoc, qmake4, cmake, several py-... etc. are processed several hundred (1000's?) times. 

Thus, if there's a way for portmaster to contruct a (or use an already existent)  Makefile for a given task and then launch 'make' to find a (optimal) way to do the actual work, I'd say that's much better than to do it "by-feet" in a sub-optimal way.

For the example above, this magic should be sufficient:
"portsdir=/usr/ports; target=x11/kde4; cd $portsdir/$target; make fetch-recursive"
Basically, most of what 'portmaster' would have to do is to find the right directory name for a given target (x11/kde4 <-> kde).
 
Comparison: after more than one hour, portmaster was still recursing through the dependencies and downloaded nothing, whereas direct use of 'make' as above started to download the 1st missing distfile after a few seconds.
Comment 1 John Marino freebsd_committer 2015-04-21 13:46:34 UTC
nobody is maintaining this port:
http://www.freshports.org/ports-mgmt/portmaster

Frankly, very few committers care about it (Personally I don't see the point since pkg(8) arrived with Poudriere)


If you have the capability to fix it, I'd suggest becoming the maintainer.  Otherwise, don't be surprised if this PR doesn't get a lot of love.
Comment 2 John Marino freebsd_committer 2016-02-10 18:43:05 UTC
PM has new maintainer, reassign PR.
Comment 3 Stefan Eßer freebsd_committer 2017-12-18 17:05:11 UTC
I plan to complete rewrite that part of portmaster, but this will be a major effort and I'm not sure, when I get to it.

The current version was designed to work with the "old" ports system, more than a decade ago. It was extended in a backwards compatible way, whenever the ports and packages systems evolved (still supporting old features and ways).

A redesign is required, and I have plans for it, but portmaster is so complex and hard to understand in detail (due to implicit and undocumented effects of global variables on the sequence of actions taken), that any change comes with a significant risk of regressions - a major rewrite much more so.
Comment 4 Stefan Eßer freebsd_committer 2020-01-25 22:29:21 UTC
I have been using a rewrite of portmaster for nearly one year, which implements a planning phase in which the port makefiles and information from the pkg database is used to gather information in associative arrays and develop a build strategy based on this information. That version does not only build ports in the same way and order as the official portmaster port, but it does also offer build modes where ports are build in a clean jail (and either used to just create a repository or installed at the end of the run).

While that version works for me, I have now started to convert it from a shell script to LUA, to be able to take advantage of the data structures it provides. This will offer many opportunities for optimizations, e.g. to find the best build strategy for each invocation, but also to improve the performance.

I plan to publish that version before the end of the first quarter of 2020.
Comment 5 Walter von Entferndt 2020-06-02 18:31:34 UTC
To both John and Stefan: I do appreciate what you guys are doing.
And yes, I was not aware that portmaster did not have a maintainer.  Since it was (is still?) mentioned in the FreeBSD Handbook, I assumed it's use is implicitely considered standard behaviour.  I made the mistake to conclude from the intuitive name of the port to it's quality... ;) and did not know about it's history.  Stefan, you took a good decision to change to LUA.
Comment 6 Chris Collins 2020-08-30 23:37:55 UTC
I think I am hitting this now.

Usually I use portupgrade which has some very intelligent behaviour, its a very good ports-mgmt tool that the more newer tools have failed to replicate.

I have decided to start trying alternatives, I really hate Poudriere, just dont like it, and whats especially bad about Poudriere as there seems to be a push to make it the defacto option for ports management, which has led to the other choices been almost abandoned.  But anyway back to the issue at hand.

I ran a recursive recompile of python dependencies after I had changed the default python version from 2.7 to 3.6.

I like the fact that portmaster does nested dependencies, but I have found the documentation incredibly confusing in regards to the -r -R and -t flags, but I after checking dry run lists, determined that -R was the way to go.

The first bug issue I consider is that the -n dry run option, does not alert to the fact that you get this looping behaviour, that part of the output is omitted.

So what seems to be happening is if multiple ports have the same dependency e.g. many ports depend on perl, then portmaster does not have the intelligence to only recompile it once and then skip it over it the proceeding times, but what I cannot work out because the text is whizzing by on my terminal, is if it is stuck in a loop, as I appear to be constantly getting m4 -> some other port -> some other port -> perl, either many ports have that dependency tree, or its stuck in a loop, I think its already recompiled perl circa 20-30 times now, thankfully ccache is a thing on this server but no point continuing if its in a perpetual loop.

Its great that you are working on this stefan, but 5 years is a very long time, you now appear to have something that works as an improvement but now want to recode it a 2nd time in a different language and add more delay? if that is the case I will probably abandon this idea of using portmaster sadly.

I see now currently its on this point.

===>>> Updating dependent ports >> m4-1.4.18_1,1 >> texinfo-6.7_3,1 >> p5-Unicode-EastAsianWidth-12.0 (175/72)

===>>> Returning to dependency check for textproc/p5-Unicode-EastAsianWidth
===>>> Dependency check complete for textproc/p5-Unicode-EastAsianWidth

===>>> Updating dependent ports >> m4-1.4.18_1,1 >> texinfo-6.7_3,1 >> p5-Unicode-EastAsianWidth-12.0 (175/72)

===>  Cleaning for p5-Unicode-EastAsianWidth-12.0

I suspect its not a perpetual loop but rather its slowly progressing but just having to recompile perl and gettext many times as many in the list depend on it.
Comment 7 Chris Collins 2020-08-30 23:43:25 UTC
I aborted, this is the queue that was added to it.

===>>> Exiting due to signal
===>>> The following actions were performed:
        Re-installation of py36-setuptools-44.0.0
        Re-installation of py36-setuptools-44.0.0
        Re-installation of py36-setuptools-44.0.0
        Re-installation of py36-pytz-2020.1,1
        Re-installation of py36-setuptools-44.0.0
        Re-installation of py36-Babel-2.8.0
        Re-installation of py36-setuptools-44.0.0
        Re-installation of py36-setuptools-44.0.0
        Re-installation of py36-chardet-3.0.4_3
        Re-installation of py36-setuptools-44.0.0
        Re-installation of libiconv-1.16
        Re-installation of perl5-5.32.0
        Re-installation of perl5-5.32.0
        Re-installation of p5-Text-Unidecode-1.30
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-tools-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of indexinfo-0.3.1
        Re-installation of gmake-4.3_2
        Re-installation of ncurses-6.2.20200822
        Re-installation of libiconv-1.16
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-tools-0.21
        Re-installation of perl5-5.32.0
        Re-installation of libiconv-1.16
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of perl5-5.32.0
        Re-installation of p5-Locale-libintl-1.31
        Re-installation of perl5-5.32.0
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-tools-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of indexinfo-0.3.1
        Re-installation of gmake-4.3_2
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-tools-0.21
        Re-installation of perl5-5.32.0
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of perl5-5.32.0
        Re-installation of p5-Locale-gettext-1.07
        Re-installation of perl5-5.32.0
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of libiconv-1.16
        Re-installation of ncurses-6.2.20200822
        Re-installation of ncurses-6.2.20200822
        Re-installation of indexinfo-0.3.1
        Re-installation of libtextstyle-0.21
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-tools-0.21
        Re-installation of perl5-5.32.0
        Re-installation of libiconv-1.16
        Re-installation of indexinfo-0.3.1
        Re-installation of gettext-runtime-0.21
        Re-installation of perl5-5.32.0
        Re-installation of p5-Locale-gettext-1.07
        Re-installation of perl5-5.32.0
        Re-installation of indexinfo-0.3.1
        Re-installation of help2man-1.47.16
        Re-installation of indexinfo-0.3.1
        Re-installation of perl5-5.32.0
        Re-installation of perl5-5.32.0
        Re-installation of p5-Unicode-EastAsianWidth-12.0


===>>> You can restart from the point of failure with this command line:
       portmaster <flags> devel/scons@py36 devel/m4 print/texinfo textproc/py-docutils@py36 textproc/py-pygments@py36 textproc/py-sphinx@py36 devel/py-Jinja2@py36 textproc/py-MarkupSafe@py36 devel/py-packaging@py36 devel/py-pyparsing@py36 devel/py-six@py36 graphics/py-imagesize@py36 security/ca_root_nss security/openssl textproc/py-alabaster@py36 textproc/py-snowballstemmer@py36 textproc/py-sphinxcontrib-applehelp@py36 textproc/py-sphinxcontrib-devhelp@py36 textproc/py-sphinxcontrib-htmlhelp@py36 textproc/py-sphinxcontrib-jsmath@py36 textproc/py-sphinxcontrib-qthelp@py36 textproc/py-sphinxcontrib-serializinghtml@py36 www/py-requests@py36 dns/py-idna@py36 net/py-urllib3@py36 security/py-certifi@py36 security/py-cryptography@py36 devel/py-asn1crypto@py36 devel/py-cffi@py36 devel/libffi devel/py-pycparser@py36 security/py-openssl@py36 textproc/py-sphinxcontrib-websupport@py36 textproc/asciidoc devel/autoconf devel/autoconf-wrapper devel/automake textproc/docbook-xsl textproc/docbook textproc/docbook-sgml textproc/iso8879 textproc/xmlcatmgr textproc/docbook-xml textproc/xmlcharent textproc/sdocbook-xml textproc/libxml2 textproc/libxslt security/libgcrypt security/libgpg-error devel/meson devel/ninja security/py-fail2ban@py36 databases/py-sqlite3@py36 databases/sqlite3 devel/libedit