Bug 235548 - fetch(1): Options -s and -S cause Bad Request when used with https URL and proxy.
Summary: fetch(1): Options -s and -S cause Bad Request when used with https URL and pr...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Dag-Erling Smørgrav
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-02-06 05:54 UTC by mickey242
Modified: 2019-02-26 08:00 UTC (History)
1 user (show)

See Also:


Attachments
Output of fetch -vv -s (3.44 KB, text/plain)
2019-02-14 14:37 UTC, mickey242
no flags Details
Output of fetch -vv -S 599712 (4.42 KB, text/plain)
2019-02-14 14:39 UTC, mickey242
no flags Details
Output of fetch -vv -s using http instead of https (4.33 KB, text/plain)
2019-02-24 13:20 UTC, mickey242
no flags Details
Output of fetch -vv -S 599712 using http instead of https (4.57 KB, text/plain)
2019-02-24 13:25 UTC, mickey242
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description mickey242 2019-02-06 05:54:14 UTC
The commandline options '-s' and '-S' of fetch(1) to report/require remote file size fail with 'Bad Request' when used with a https URL through a (squid) proxy server (no SSL bump, connect through proxy). Without proxy or using http instead of https both works.

$ env HTTP_PROXY=http://10.6.6.1:3128 fetch -s https://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz
fetch: Bad Request

$ env HTTP_PROXY=http://10.6.6.1:3128 fetch -s http://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz
599712

$ fetch -s https://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz
599712

$ env HTTP_PROXY=http://10.6.6.1:3128 fetch -S 599712 https://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz
fetch: https://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz: Bad Request

$ env HTTP_PROXY=http://10.6.6.1:3128 fetch -S 599712 http://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz
powerdevil-5.14.5.tar.xz              585 kB  329 kBps   02s

$ fetch -S 599712 https://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz
powerdevil-5.14.5.tar.xz              585 kB  312 kBps   02s

The proxy log shows two requests for each attempt that failed with 'Bad Request'. In case of the '-s' option it shows:

"CONNECT download.kde.org:443 HTTP/1.1" 200 6863 TCP_TUNNEL:HIER_DIRECT
"HEAD /pub/mirrors/ftp.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz HTTP/1.1" 400 301 NONE:HIER_NONE

And with the '-S 599712' option:

"CONNECT download.kde.org:443 HTTP/1.1" 200 7243 TCP_TUNNEL:HIER_DIRECT
"GET /pub/mirrors/ftp.kde.org/pub/kde/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz HTTP/1.1" 400 3973 NONE:HIER_NONE

This is particularly bad cause the ports system uses these options to fetch the required distfiles. On ports that use https URLs the first attempt will always fail and cause a fallback to another site.
Comment 1 Dag-Erling Smørgrav freebsd_committer 2019-02-08 10:33:43 UTC
Could you please re-run these commands with `-vv`?
Comment 2 mickey242 2019-02-14 14:37:50 UTC
Created attachment 202026 [details]
Output of fetch -vv -s
Comment 3 mickey242 2019-02-14 14:39:14 UTC
Created attachment 202027 [details]
Output of fetch -vv -S 599712
Comment 4 mickey242 2019-02-14 14:43:11 UTC
Reran both failing requests with -vv commandline options added. Output attached:

1) env HTTP_PROXY=http://10.6.6.1:3128 fetch -vv -s https://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz

2) env HTTP_PROXY=http://10.6.6.1:3128 fetch -vv -S 599712 https://download.kde.org/stable/plasma/5.14.5/powerdevil-5.14.5.tar.xz
Comment 5 Dag-Erling Smørgrav freebsd_committer 2019-02-22 14:05:21 UTC
It looks like a Squid configuration error to me, but I need to see the logs from the successful downloads as well to be sure.
Comment 6 mickey242 2019-02-24 13:20:08 UTC
Created attachment 202325 [details]
Output of fetch -vv -s using http instead of https
Comment 7 mickey242 2019-02-24 13:25:34 UTC
Created attachment 202326 [details]
Output of fetch -vv -S 599712 using http instead of https

These two logs show the verbose output when using http instead of https through the proxy, both of which succeed.

To me it looks like the failing requests are missing two elements: 1) An URI scheme and 2) a host name. That is why squid responds with bad request.
Comment 8 Dag-Erling Smørgrav freebsd_committer 2019-02-24 21:35:09 UTC
You are right.  It looks like libfetch forgets that it is using a proxy after processing the redirect.
Comment 9 mickey242 2019-02-26 08:00:40 UTC
Interestingly when using the http URL, the server first forces the use of https by redirecting to the same URL using https, before it finally redirects to some mirror site using http. Yet in this scenario it does not fail.