Bug 239293 - fetch-list: broken handling of DISTFILES when it has entries with subdirectories
Summary: fetch-list: broken handling of DISTFILES when it has entries with subdirectories
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Port Management Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-07-19 06:23 UTC by Ruslan Garipov
Modified: 2019-08-11 04:21 UTC (History)
1 user (show)

See Also:
tobik: maintainer-feedback+


Attachments
v0 (502 bytes, patch)
2019-07-19 07:20 UTC, Tobias Kortkamp
no flags Details | Diff
Add `mkdir` to Tobias Kortkamp's v0 patch (982 bytes, patch)
2019-07-30 08:37 UTC, Ruslan Garipov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ruslan Garipov 2019-07-19 06:23:35 UTC
I can't get `fetch-list` for the port:

```
$ cd /usr/ports/lang/rust
$ make fetch-list
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr  -S 98707920 https://static.rust-lang.org/dist/rustc-1.36.0-src.tar.xz  || env /usr/bin/fetch -Fpr  -S 98707920 http://distcache.FreeBSD.org/ports-distfiles/rust/rustc-1.36.0-src.tar.xz  || echo "rustc-1.36.0-src.tar.xz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir: 2019-05-23: Permission denied
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/rust
```

On other ports I need `make fetch-list` generate the lists without errors.  And I never `fetch-list` for lang/rust before (therefore, I'm not sure that this error appears near the r506903).

The ports tree is at r506903.

FreeBSD 13.0-CURRENT and 12.0-RELEASE-p4.
Comment 1 Tobias Kortkamp freebsd_committer 2019-07-19 06:34:05 UTC
This works just fine.

It looks like you do not have permissions to create
/usr/ports/distfiles/rust/2019-05-23 for whatever reason.
Make sure that you have write permissions to /usr/ports/distfiles/rust.
Comment 2 Ruslan Garipov 2019-07-19 06:51:50 UTC
(In reply to Tobias Kortkamp from comment #1)
> It looks like you do not have permissions to create
> /usr/ports/distfiles/rust/2019-05-23 for whatever reason

Yes, I definitely don't.  I run `make fetch-list` from a non-root user.
Always.  And I've never got that ``Permission denied'' error.  That's
why I didn't try to run `make fetch-list` in /usr/ports/lang/rust from
root.  Moreover, I didn't think that `make fetch-list` actually change
my file system in any way.

As I already said: all other ports show their "list" just fine.

And thanks for your tip: yes, when I `make fetch-list` from root user it
works fine.  Thanks a lot!
Comment 3 Tobias Kortkamp freebsd_committer 2019-07-19 07:12:09 UTC
Ok, I am able to reproduce it if I do not have write permissions to
/usr/ports/lang/rust.  For some reason fetch-list or fetch-url-list
attempt to mkdir /usr/ports/lang/rust/2019-05-23 on fetch-list or
fetch-url-list, which does not make a whole lot of sense since it
is not even under /usr/ports/distfiles/.  But then again fetch-list
should probably not even attempt creating any directories at all.

This also affects other ports that have a DISTFILES with a / in
them like devel/rust-cbindgen or devel/rust-bindgen.  This seems
like bad behavior of or an unhandled edge case in do-fetch.sh (or
somewhere else?) to me.  Assigning to portmgr.
Comment 4 Tobias Kortkamp freebsd_committer 2019-07-19 07:20:19 UTC
Created attachment 205879 [details]
v0

Here's a patch that seems to fix it.
Comment 5 Ruslan Garipov 2019-07-29 10:41:34 UTC
(In reply to Tobias Kortkamp from comment #4)
I'm sorry for delay.  Your patch works fine for me, thanks.
Comment 6 Ruslan Garipov 2019-07-30 08:37:23 UTC
Created attachment 206151 [details]
Add `mkdir` to Tobias Kortkamp's v0 patch

(In reply to Tobias Kortkamp from comment #4)
Oh, sorry, I hastened to say ``the patch looks good''.

Unfortunately, it doesn't fix the problem completely: it prevents
mkdir(1) on fetch-list, but the result script itself ends up with
issues:

$ cd /usr/ports/lang/rush
$ make fetch-list
{elided}
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 73812596 -o 2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz ...{elided}
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 83588135 -o 2019-05-23/rust-std-1.35.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-05-23/rust-std-1.35.0-x86_64-unknown-freebsd.tar.gz ... {elided}
{elided}

As one can see, the script generated by fetch-list creates
/usr/ports/distfiles/rust directory, but fetch(1) writes its output to
/usr/ports/distfiles/rust/2019-05-23 directory, which doesn't exist.
Therefore, fetch(1) fails with errors like the following ones:

fetch: 2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz: open(): No such file or directory
fetch: 2019-05-23/rust-std-1.35.0-x86_64-unknown-freebsd.tar.gz: open(): No such file or directory

Tobias, I've modified your patch slightly, and tested on lang/rust,
shells/bash, www/firefox and x11/xorg (meta)port.  For the former the
new patch gives the following script:

$ cd /usr/ports/lang/rust
$ make fetch-list
{the first row is elided}
mkdir -p "/usr/ports/distfiles/rust/2019-05-23" && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 73812596 -o 2019-05-23/rustc-1.35.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org... {elided}
{elided}

which works fine on a machine having the Internet access.

Are there any mandatory tests I should run against the updated
do-fetch.sh script?
Comment 7 Mathieu Arnold freebsd_committer 2019-07-30 13:12:37 UTC
Sorry for being silent, I was looking into it. (but slowly because you don't want to rush things).
You found out what was missing from Tobias's patch, but I don't like yours either, it could be made in a much simpler way.
Comment 8 Mathieu Arnold freebsd_committer 2019-07-30 13:14:37 UTC
(As a side note, your patch does not work, file is not defined when it runs.)
Comment 9 Mathieu Arnold freebsd_committer 2019-07-30 13:18:44 UTC
https://reviews.freebsd.org/D21112
Comment 10 Ruslan Garipov 2019-07-30 15:25:26 UTC
(In reply to Mathieu Arnold from comment #9)
Thanks for your answers and the revision.

> it could be made in a much simpler way
I just wanted to concentrate all the echos in one place.

> your patch does not work
For fetch/fetch-recursive may be?..  Yes, it's my fault -- I didn't test those actions; I just checked fetch-list/fetch-recursive-list.  Sorry.
Comment 11 Mathieu Arnold freebsd_committer 2019-07-30 15:47:42 UTC
(In reply to Ruslan Garipov from comment #10)
> (In reply to Mathieu Arnold from comment #9)
> Thanks for your answers and the revision.
> 
> > it could be made in a much simpler way
> I just wanted to concentrate all the echos in one place.

It was a noble thought, but I (re)wrote this code a few years ago, if it could be made simpler, it would :-)

> > your patch does not work
> For fetch/fetch-recursive may be?..  Yes, it's my fault -- I didn't test
> those actions; I just checked fetch-list/fetch-recursive-list.  Sorry.

Well, fetch-list, yes.  You were using ${file} before it was defined, so it did not do what you wanted, at all.
Comment 12 Ruslan Garipov 2019-07-30 17:18:32 UTC
(In reply to Mathieu Arnold from comment #11)
Sorry for off topic, but I just want to understand where I've failed
and I don't see why I was

> ... using ${file} before it was defined
?  I believe the `file` is defined on the 30th line[1].
Unconditionally, just as the first statement of the first `for` loop.
And my patch modified the 110th line in the file.  The variable is used
above and beyond the 110th line -- but why it's undefined there?  I
also see no `unset` in the file...  Am I missed something?

> so it did not do what you wanted, at all.
I'm sorry, but it did -- as I already said in the comment #6 I
successfully fetch-list for lang/rust, shells/bash, www/firefox and
x11/xorg ports.  Was it a lucky break?

[1] https://svnweb.freebsd.org/ports/head/Mk/Scripts/do-fetch.sh?revision=462544&view=markup#l30
Comment 13 commit-hook freebsd_committer 2019-07-31 10:11:08 UTC
A commit references this bug:

Author: mat
Date: Wed Jul 31 10:10:36 UTC 2019
New revision: 507705
URL: https://svnweb.freebsd.org/changeset/ports/507705

Log:
  Fix fetch-list when running as a user.

  The fetch-list target is used to generate a shell script that will more
  or less replicates what do-fetch does. It allows one to do most things
  as a regular user, and generate that script to run, say, on another
  machine, if the one where you build things does not have access to the
  internet, or has much slower access.

  It was failing when DISTDIR was not writable by the current user, and
  the port had a distribution file with a path in it. (Not using
  DIST_SUBDIR, something else, like lang/rust does.) It was failing
  because it was trying to create that subdirectory unconditionally,
  instead of only creating the subdirectory if actually had to.  This also
  fixes the bug that the generated script did not have the appropriate
  mkdirs for those directories.

  PR:		239293
  Submitted by:	tobik (earlier version)
  Reported by:	Ruslan Garipov
  Differential Revision:	https://reviews.freebsd.org/D21112

Changes:
  head/Mk/Scripts/do-fetch.sh
Comment 14 Ruslan Garipov 2019-08-05 12:52:09 UTC
(In reply to Mathieu Arnold from comment #9)
(In reply to commit-hook from comment #13)
Please add -n to your `echo`:

Index: do-fetch.sh
===================================================================
--- do-fetch.sh (revision 508165)
+++ do-fetch.sh (working copy)
@@ -128,7 +128,7 @@
                        */*)
                                case ${dp_TARGET} in
                                fetch-list|fetch-url-list-int)
-                                       echo "mkdir -p \"${file%/*}\" && "
+                                       echo -n "mkdir -p \"${file%/*}\" && "
                                        ;;
                                *)
                                        mkdir -p "${file%/*}"

like in all `echo`s in the file.

I'm usually fetch-list several ports into one file, and then remove
duplicated lines.  Without -n in the `echo` above, I'm unable to do
that.  Thanks.