Bug 203861
| Summary: | 'fetch' command fails when HTTP_PROXY env. variable is set, and there is a http->https redirect | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | mvharding | ||||||
| Component: | bin | Assignee: | Dag-Erling Smørgrav <des> | ||||||
| Status: | Open --- | ||||||||
| Severity: | Affects Some People | CC: | des, kirill | ||||||
| Priority: | --- | ||||||||
| Version: | 10.2-STABLE | ||||||||
| Hardware: | Any | ||||||||
| OS: | Any | ||||||||
| Attachments: |
|
||||||||
|
Description
mvharding
2015-10-18 23:11:46 UTC
Regular fetch works fine (I have a squid proxy on 192.168.0.2 on my local network): $ fetch http://pypi.python.org/packages/source/c/cryptography/cryptography-1.0.2.tar.gz fetch http://pypi.python.org/packages/source/c/cryptography/cryptography-1.0.2.tar.gz cryptography-1.0.2.tar.gz 100% of 325 kB 822 kBps 00m01s Setting a proxy does not work: $ HTTP_PROXY=http://192.168.0.2 fetch http://pypi.python.org/packages/source/c/cryptography/cryptography-1.0.2.tar.gz HTTP_PROXY=http://192.168.0.2 fetch http://pypi.python.org/packages/source/c/cryptography/cryptography-1.0.2.tar.gz fetch: http://pypi.python.org/packages/source/c/cryptography/cryptography-1.0.2.tar.gz: Not Found This seems (to me) to affect all pypi packages, as there is a 301 redirect from http to https. If I try to get the 'https' stuff directly, it works fine. $ HTTP_PROXY=http://192.168.0.2 fetch https://pypi.python.org/packages/source/c\ /cryptography/cryptography-1.0.2.tar.gz HTTP_PROXY=http://192.168.0.2 fetch https://pypi.python.org/packages/source/c/c\ ryptography/cryptography-1.0.2.tar.gz cryptography-1.0.2.tar.gz 100% of 325 kB 825 kBps 00m00s This could, I guess, be worked around by changing the base for the pypi fetches to 'https'. Right, now, most Python package fetches fail unless I disable the proxy. I did some runs with '-vvv' but and can see the 301 redirect (I can paste the whole session here, but it's easy to recreate...). Can you please attach your log? Created attachment 164322 [details]
log with a proxy set
Created attachment 164323 [details]
log with no proxy set
logs attatched (generated with 'script') The problem is that libfetch does not reevaluate which proxy to use if a URL is redirected to another with a different scheme. This is difficult to fix as the decision to use a proxy is made outside `http_request()`. It also overlaps / conflicts with #220468. Forgot to add: python.org has added HSTS since then, so if you test using the same URL today, you will get a 403 with a Strict-Transport-Security header. We should treat this (403 + STS) the same as a 301. |
