I can't get `fetch-list` for the port: ``` $ cd /usr/ports/lang/rust $ make fetch-list mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 98707920 https://static.rust-lang.org/dist/rustc-1.36.0-src.tar.xz || env /usr/bin/fetch -Fpr -S 98707920 http://distcache.FreeBSD.org/ports-distfiles/rust/rustc-1.36.0-src.tar.xz || echo "rustc-1.36.0-src.tar.xz" not fetched; } mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir: 2019-05-23: Permission denied *** Error code 1 Stop. make: stopped in /usr/ports/lang/rust ``` On other ports I need `make fetch-list` generate the lists without errors. And I never `fetch-list` for lang/rust before (therefore, I'm not sure that this error appears near the r506903). The ports tree is at r506903. FreeBSD 13.0-CURRENT and 12.0-RELEASE-p4.
This works just fine. It looks like you do not have permissions to create /usr/ports/distfiles/rust/2019-05-23 for whatever reason. Make sure that you have write permissions to /usr/ports/distfiles/rust.
(In reply to Tobias Kortkamp from comment #1) > It looks like you do not have permissions to create > /usr/ports/distfiles/rust/2019-05-23 for whatever reason Yes, I definitely don't. I run `make fetch-list` from a non-root user. Always. And I've never got that ``Permission denied'' error. That's why I didn't try to run `make fetch-list` in /usr/ports/lang/rust from root. Moreover, I didn't think that `make fetch-list` actually change my file system in any way. As I already said: all other ports show their "list" just fine. And thanks for your tip: yes, when I `make fetch-list` from root user it works fine. Thanks a lot!
Ok, I am able to reproduce it if I do not have write permissions to /usr/ports/lang/rust. For some reason fetch-list or fetch-url-list attempt to mkdir /usr/ports/lang/rust/2019-05-23 on fetch-list or fetch-url-list, which does not make a whole lot of sense since it is not even under /usr/ports/distfiles/. But then again fetch-list should probably not even attempt creating any directories at all. This also affects other ports that have a DISTFILES with a / in them like devel/rust-cbindgen or devel/rust-bindgen. This seems like bad behavior of or an unhandled edge case in do-fetch.sh (or somewhere else?) to me. Assigning to portmgr.
Created attachment 205879 [details] v0 Here's a patch that seems to fix it.
(In reply to Tobias Kortkamp from comment #4) I'm sorry for delay. Your patch works fine for me, thanks.
Created attachment 206151 [details] Add `mkdir` to Tobias Kortkamp's v0 patch (In reply to Tobias Kortkamp from comment #4) Oh, sorry, I hastened to say ``the patch looks good''. Unfortunately, it doesn't fix the problem completely: it prevents mkdir(1) on fetch-list, but the result script itself ends up with issues: $ cd /usr/ports/lang/rush $ make fetch-list {elided} mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 73812596 -o 2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz ...{elided} mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 83588135 -o 2019-05-23/rust-std-1.35.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-05-23/rust-std-1.35.0-x86_64-unknown-freebsd.tar.gz ... {elided} {elided} As one can see, the script generated by fetch-list creates /usr/ports/distfiles/rust directory, but fetch(1) writes its output to /usr/ports/distfiles/rust/2019-05-23 directory, which doesn't exist. Therefore, fetch(1) fails with errors like the following ones: fetch: 2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz: open(): No such file or directory fetch: 2019-05-23/rust-std-1.35.0-x86_64-unknown-freebsd.tar.gz: open(): No such file or directory Tobias, I've modified your patch slightly, and tested on lang/rust, shells/bash, www/firefox and x11/xorg (meta)port. For the former the new patch gives the following script: $ cd /usr/ports/lang/rust $ make fetch-list {the first row is elided} mkdir -p "/usr/ports/distfiles/rust/2019-05-23" && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 73812596 -o 2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org... {elided} {elided} which works fine on a machine having the Internet access. Are there any mandatory tests I should run against the updated do-fetch.sh script?
Sorry for being silent, I was looking into it. (but slowly because you don't want to rush things). You found out what was missing from Tobias's patch, but I don't like yours either, it could be made in a much simpler way.
(As a side note, your patch does not work, file is not defined when it runs.)
https://reviews.freebsd.org/D21112
(In reply to Mathieu Arnold from comment #9) Thanks for your answers and the revision. > it could be made in a much simpler way I just wanted to concentrate all the echos in one place. > your patch does not work For fetch/fetch-recursive may be?.. Yes, it's my fault -- I didn't test those actions; I just checked fetch-list/fetch-recursive-list. Sorry.
(In reply to Ruslan Garipov from comment #10) > (In reply to Mathieu Arnold from comment #9) > Thanks for your answers and the revision. > > > it could be made in a much simpler way > I just wanted to concentrate all the echos in one place. It was a noble thought, but I (re)wrote this code a few years ago, if it could be made simpler, it would :-) > > your patch does not work > For fetch/fetch-recursive may be?.. Yes, it's my fault -- I didn't test > those actions; I just checked fetch-list/fetch-recursive-list. Sorry. Well, fetch-list, yes. You were using ${file} before it was defined, so it did not do what you wanted, at all.
(In reply to Mathieu Arnold from comment #11) Sorry for off topic, but I just want to understand where I've failed and I don't see why I was > ... using ${file} before it was defined ? I believe the `file` is defined on the 30th line[1]. Unconditionally, just as the first statement of the first `for` loop. And my patch modified the 110th line in the file. The variable is used above and beyond the 110th line -- but why it's undefined there? I also see no `unset` in the file... Am I missed something? > so it did not do what you wanted, at all. I'm sorry, but it did -- as I already said in the comment #6 I successfully fetch-list for lang/rust, shells/bash, www/firefox and x11/xorg ports. Was it a lucky break? [1] https://svnweb.freebsd.org/ports/head/Mk/Scripts/do-fetch.sh?revision=462544&view=markup#l30
A commit references this bug: Author: mat Date: Wed Jul 31 10:10:36 UTC 2019 New revision: 507705 URL: https://svnweb.freebsd.org/changeset/ports/507705 Log: Fix fetch-list when running as a user. The fetch-list target is used to generate a shell script that will more or less replicates what do-fetch does. It allows one to do most things as a regular user, and generate that script to run, say, on another machine, if the one where you build things does not have access to the internet, or has much slower access. It was failing when DISTDIR was not writable by the current user, and the port had a distribution file with a path in it. (Not using DIST_SUBDIR, something else, like lang/rust does.) It was failing because it was trying to create that subdirectory unconditionally, instead of only creating the subdirectory if actually had to. This also fixes the bug that the generated script did not have the appropriate mkdirs for those directories. PR: 239293 Submitted by: tobik (earlier version) Reported by: Ruslan Garipov Differential Revision: https://reviews.freebsd.org/D21112 Changes: head/Mk/Scripts/do-fetch.sh
(In reply to Mathieu Arnold from comment #9) (In reply to commit-hook from comment #13) Please add -n to your `echo`: Index: do-fetch.sh =================================================================== --- do-fetch.sh (revision 508165) +++ do-fetch.sh (working copy) @@ -128,7 +128,7 @@ */*) case ${dp_TARGET} in fetch-list|fetch-url-list-int) - echo "mkdir -p \"${file%/*}\" && " + echo -n "mkdir -p \"${file%/*}\" && " ;; *) mkdir -p "${file%/*}" like in all `echo`s in the file. I'm usually fetch-list several ports into one file, and then remove duplicated lines. Without -n in the `echo` above, I'm unable to do that. Thanks.