Bug 228360 - ports-mgmt/poudriere: option to fetch distfiles at once?
Summary: ports-mgmt/poudriere: option to fetch distfiles at once?
Status: Closed Feedback Timeout
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Bryan Drewery
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-05-19 13:07 UTC by Anonymized Account
Modified: 2021-09-27 20:50 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (bdrewery)


Attachments
Make list of fetched ports. (442 bytes, text/plain)
2018-05-19 13:14 UTC, Anonymized Account
no flags Details
Fetch distfiles in parallel. (278 bytes, text/plain)
2018-05-19 13:15 UTC, Anonymized Account
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Anonymized Account freebsd_committer freebsd_triage 2018-05-19 13:07:36 UTC
There seems to be no option to fetch all needed distfiles at once. This would be very useful on intermittent connection or when one wants to limit Internet access to certain times.

Simply running "make fetch recursive" in Ports before running Poudriere is very slow because on each port it starts anew instead of making a list of what to build.

I tried making a custom script that uses "make missing" of Ports and it works most of the time, except that it reguarly forgets about some distfiles, something like Linux compat and I still see them downloaded by Poudriere later.

Another problem is, downloading the distfiles is not parallel and CPU is almost idle.
Comment 1 Anonymized Account freebsd_committer freebsd_triage 2018-05-19 13:14:28 UTC
Created attachment 193530 [details]
Make list of fetched ports.

Attaching the ugly scripts I'm using to give the idea.
Comment 2 Anonymized Account freebsd_committer freebsd_triage 2018-05-19 13:15:07 UTC
Created attachment 193531 [details]
Fetch distfiles in parallel.
Comment 3 Anonymized Account freebsd_committer freebsd_triage 2018-05-19 13:16:08 UTC
(unparallelised download is also inefficient because it does not use the Internet bandwidth fully -- would it be better to run more fetch jobs than CPUs?)
Comment 4 Walter Schwarzenfeld freebsd_triage 2019-08-14 23:11:20 UTC
Maintainer feedback, please!
Comment 5 Bryan Drewery freebsd_committer freebsd_triage 2021-09-27 20:50:11 UTC
https://github.com/freebsd/poudriere/issues/882