Bug 251216 - fetch(1) not able to fetch file
Summary: fetch(1) not able to fetch file
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.2-RELEASE
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-11-17 13:32 UTC by vermaden
Modified: 2020-11-18 14:05 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description vermaden 2020-11-17 13:32:21 UTC
As in title, the fetch(1) is not able to download file from GitHub:

% fetch https://github.com/amir/sctd/archive/0.2.0.tar.gz 

fetch: https://github.com/amir/sctd/archive/0.2.0.tar.gz: Moved Temporarily

But it is not problem to download the same file with wget(1) for example:

% wget https://github.com/amir/sctd/archive/0.2.0.tar.gz
--2020-11-17 14:31:42--  https://github.com/amir/sctd/archive/0.2.0.tar.gz
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/amir/sctd/tar.gz/0.2.0 [following]
--2020-11-17 14:31:43--  https://codeload.github.com/amir/sctd/tar.gz/0.2.0
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘0.2.0.tar.gz’

0.2.0.tar.gz                     [ <=>      ]   8.43K  --.-KB/s    in 0.001s  

2020-11-17 14:31:43 (14.2 MB/s) - ‘0.2.0.tar.gz’ saved [8629]



Regards.
Comment 1 Tom Jones freebsd_committer 2020-11-17 13:42:49 UTC
Was this maybe a temporay issue with github?

I cannot reproduce on 12.1, 12-stable or 13-current and someone on irc couldn't on 12.2-stable or 11.4-stable
Comment 2 Fernando Apesteguía freebsd_committer 2020-11-17 16:09:38 UTC
I can not reproduce in a 12.2 machine:

fernape@12:~ $ uname -a
FreeBSD 12.2amd64 12.2-RELEASE FreeBSD 12.2-RELEASE r367095 GENERIC  amd64
fernape@12:~ $ fetch -v https://github.com/amir/sctd/archive/0.2.0.tar.gz
resolving server address: github.com:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.3 connection established using TLS_AES_128_GCM_SHA256
Certificate subject: /C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
Certificate issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
requesting https://github.com/amir/sctd/archive/0.2.0.tar.gz
302 redirect to https://codeload.github.com/amir/sctd/tar.gz/0.2.0
resolving server address: codeload.github.com:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.3 connection established using TLS_AES_128_GCM_SHA256
Certificate subject: /C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=*.github.com
Certificate issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
requesting https://codeload.github.com/amir/sctd/tar.gz/0.2.0
remote size / mtime: 8629 / 0
0.2.0.tar.gz                                          8629  B   31 MBps    00s
Comment 3 vermaden 2020-11-17 16:50:47 UTC
Just tried again, same results:

% fetch https://github.com/amir/sctd/archive/0.2.0.tar.gz 
fetch: https://github.com/amir/sctd/archive/0.2.0.tar.gz: Moved Temporarily

% uname -a
FreeBSD w520.local 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC  amd64

% freebsd-version -u
12.2-RELEASE

% freebsd-version -k
12.2-RELEASE

% host github.com
github.com has address 140.82.121.3
github.com mail is handled by 1 aspmx.l.google.com.
github.com mail is handled by 10 alt3.aspmx.l.google.com.
github.com mail is handled by 10 alt4.aspmx.l.google.com.
github.com mail is handled by 5 alt1.aspmx.l.google.com.
github.com mail is handled by 5 alt2.aspmx.l.google.com.

% dog !$
dog github.com
A github.com. 54s   140.82.121.3

Its just that fetch(1) has a problem with HTTP 302 redirection.

When you see the wget(1) output then after redirection it is downloaded from https://codeload.github.com/amir/sctd/tar.gz/0.2.0 location and this one works even with fetch(1). 

% fetch https://codeload.github.com/amir/sctd/tar.gz/0.2.0
fetch: https://codeload.github.com/amir/sctd/tar.gz/0.2.0: size of remote file is not known
0.2.0                                                 8629  B   10 MBps    00s

% ls -l 0.2.0 0.2.0.tar.gz
-rw-r--r-- 1 vermaden vermaden 8629 2020-11-17 17:49 0.2.0
-rw-r--r-- 1 vermaden vermaden 8629 2020-11-17 14:31 0.2.0.tar.gz

% file 0.2.0 0.2.0.tar.gz
0.2.0:        gzip compressed data, from Unix, original size modulo 2^32 40960
0.2.0.tar.gz: gzip compressed data, from Unix, original size modulo 2^32 40960
Comment 4 Fernando Apesteguía freebsd_committer 2020-11-17 17:25:10 UTC
(In reply to vermaden from comment #3)
In my execution fetch is accepting 302 fine. Could you run fetch -v and see where your output and mine differ?
Comment 5 vermaden 2020-11-17 17:58:41 UTC
(In reply to Fernando Apesteguía from comment #4)

Here:

% fetch -v https://github.com/amir/sctd/archive/0.2.0.tar.gz
resolving server address: github.com:443
SSL options: 82004854
TLSv1.3 connection established using TLS_AES_128_GCM_SHA256
Certificate subject: /C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
Certificate issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
requesting https://github.com/amir/sctd/archive/0.2.0.tar.gz
fetch: https://github.com/amir/sctd/archive/0.2.0.tar.gz: Moved Temporarily
Comment 6 Fernando Apesteguía freebsd_committer 2020-11-17 18:13:27 UTC
(In reply to vermaden from comment #5)
...
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
...

Someone at IRC suggested using --no-verify-peer.
Comment 7 Ed Maste freebsd_committer 2020-11-17 18:17:14 UTC
Also not reproducible here unless I add the -A/--no-redirect option.
Comment 8 andrew 2020-11-18 13:23:51 UTC
(In reply to vermaden from comment #5)

(I am the "someone on IRC" previously mentioned)

As far as I can tell, it's impossible to get this verbose output:

> requesting https://github.com/amir/sctd/archive/0.2.0.tar.gz
> fetch: https://github.com/amir/sctd/archive/0.2.0.tar.gz: Moved Temporarily

unless the -A flag is somehow becoming set in libfetch. Without that flag, that code path would always lead through code that outputs this:

> 302 redirect to https://codeload.github.com/amir/sctd/tar.gz/0.2.0

What if you run it in a clean environment with env -i?
Comment 9 vermaden 2020-11-18 14:05:50 UTC
I am sorry.

I had an alias for fetch with -A ...

% which fetch
fetch: aliased to fetch -ApRr --no-verify-peer --no-verify-hostname

My bad.

Sorry for your lost time.

Regards.