Bug 241148 - Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times
Summary: Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file sever...
Status: Closed Overcome By Events
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Port Management Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-10-09 09:52 UTC by Ruslan Garipov
Modified: 2022-09-07 15:23 UTC (History)
4 users (show)

See Also:
brigadir15: maintainer-feedback?

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Ruslan Garipov 2019-10-09 09:52:19 UTC 
After r507705[1] fetch-list for ports with DISTFILES having entries with subdirectories, produces shell script that causes downloading of the same file several time.

For example:

$ cd /usr/ports/lang/rust
$ make fetch-list
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr  -S 96163304 https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz  || env /usr/bin/fetch -Fpr  -S 96163304 http://distcache.FreeBSD.org/ports-distfiles/rust/rustc-1.38.0-src.tar.xz  || echo "rustc-1.38.0-src.tar.xz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz  || echo "2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz  || echo "2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz" not fetched; }

Execution of that script ends up with this result:

$ su root -c "/usr/home/user/fetch_rust.sh"
Password:
fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size unknown
fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size of remote file is not known
rustc-1.38.0-src.tar.xz                                 91 MB 4241 kBps    22s
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd          67 MB 3724 kBps    08s
fetch: http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
2019-08-15/rust-std-1.37.0-x86_64-unknown-free          83 MB 4212 kBps    21s
fetch: http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd        6554 kB 3455 kBps    02s
fetch: http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz: Not Found
fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz: Not Found

As you can see, the script tries to download, for example, rustc-1.37.0-x86_64-unknown-freebsd.tar.gz five times.  Even after the script has downloaded it successfully on the first try.

That happens because of the following lines in fetch-list's script:

mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr  -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz  || echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; }

It makes "2019-08-15", downloads (successfully) rustc-1.37.0-x86_64-unknown-freebsd.tar.gz from https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz (having zero exit status), skips `mkdir -p "2019-08-15"` (because of "no errors" exit status), and fetches the file again after `&&` (once again -- because of zero exit status).  This `fetch` is subsequently failing, but `mkdir` after that `fetch` succeeds (because it had `-p` specified).  The script `fetch`es again.  And so on.

I believe `mkdir` and `fetch` must be grouped together between all those `||` operators.

See also: bug 239293[2]

P.S. If someone will fix this, please also take bug 239761[3] into account, where my patch allows to combine several fetching lines into single one, as usual.

[1] https://svnweb.freebsd.org/ports?view=revision&revision=507705
[2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239293
[3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239761
Comment 1 Tobias Kortkamp freebsd_committer freebsd_triage 2019-10-09 09:59:37 UTC 
This has little to do with lang/rust (except that it was used as an example)
and rust@ does not need to be involved here.
Comment 2 Ruslan Garipov 2019-10-09 10:25:47 UTC 
(In reply to Tobias Kortkamp from comment #1)

> This has little to do with lang/rust
Yes.  But assignment to the rust team was made automatically, I didn't assigned them:

> Bugzilla Automation 2019-10-09 09:52:19 UTC
> Assignee: ports-bugs@FreeBSD.org → rust@FreeBSD.org

So, rust@, I'm sorry for bother.
Comment 3 Adam Weinberger freebsd_committer freebsd_triage 2019-10-09 10:40:22 UTC 
Tobias wasn't chiding you, and yes bugzilla does the initial assignment automatically. He was adding a note documenting why he reassigned it to portmgr.
Comment 4 Tobias C. Berner freebsd_committer freebsd_triage 2022-09-07 15:23:58 UTC 
Moin moin 

Closing this, due to old age. Please re-open the issue, if it is still relevant.


mfg Tobias