Bug 252123

Summary: fetch(3): Fix wrong usage of proxy when request is redirected to different URL
Product: Base System Reporter: Yasuhiro Kimura <yasu>
Component: binAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Only Me CC: des
Priority: --- Flags: yasu: mfc-stable13?
yasu: mfc-stable12?
yasu: mfc-stable11?
Version: CURRENT   
Hardware: Any   
OS: Any   
URL: https://reviews.freebsd.org/D28437
Attachments:
Description Flags
Patch file none

Description Yasuhiro Kimura freebsd_committer 2020-12-25 04:26:44 UTC
Created attachment 220905 [details]
Patch file

1. Summary

Under some proxy settings fetch(3) makes wrong usage of proxy when request is redirected to different URL. The source of the problem is that current implementation decides proxy usage against initial URL of request and reuses it even if the request is redirected to different one. So it can be fixed by changing http_requset_body() in lib/libfetch/http.c so proxy usage is re-evaluated against new URL after request is redirected to different one.

2. Detail of the problem

In order to explain the detail of the problem, let's assume an imaginary company, its network environment and some server hosts. Domain name of the company is "example.com". Network of example.com is isolated one and can't access public network directly. So the employee of example.com need to use proxy to access ftp/web servers on public network. Following 3 servers are connected to the network of example.com

(a) www.example.com

Web server of example.com.

(b) ftp-proxy.example.com

Proxy server of example.com. It only permits FTP access to public network.

(c) http-proxy.example.com

Another proxy server of example.com. It only permit HTTP and HTTPS assess to public network.

And following 2 servers are also used to explain the problem.

(d) ftp.example.net

Ftp server on the public network.

(e) www.example.net

Web server on the public network.

Under this situation the employee adds following proxy settings in his .cshrc file.

----------------------------------------------------------------------
setenv ftp_proxy 'http://ftp-proxy.example.com:8080/'
setenv FTP_PROXY 'http://ftp-proxy.example.com:8080/'
setenv http_proxy 'http://http-proxy.example.com:8080/'
setenv HTTP_PROXY 'http://http-proxy.example.com:8080/'
setenv no_proxy 'example.com,localhost'
setenv no_proxy 'example.com,localhost'
----------------------------------------------------------------------

Now let's provide 2 situations and think what happens.

Case 1: Access to www.example.com is redirected to www.example.net

Let's assume www.example.com is configured so access to "http://www.example.com/example.net/" is redirected to "http://www.example.net/", and the employee executed "fetch -o  http://www.example.com/example.net". If fetch(1) works correctly it should work as following.

a. fetch(1) directly accesses www.example.com and sends request for http://www.example.com/example.net/
b. www.example.com returns response that http://www.example.com/example.net/ is redirected to http://www.example.net/
c. fetch(1) accesses www.example.net through http-proxy.example.com and sends request for http://www.example.net/
d. www.example.net returns response for http://www.example.net/
e. fetch(1) writes body of response to stdout.

But what really happens is as following.

a. fetch(1) directly accesses www.example.com and sends request for http://www.example.com/example.net/
b. www.example.com returns response that http://www.example.com/example.net/ is redirected to http://www.example.net/
c'. fetch(1) tries to access www.example.net directly.
d'. Since it's impossible to make direct access to public network, fetch(1) exits with error.

Case 2: Access to www.example.net is redirected to ftp.example.net

Let's assume www.example.net is configured so access to "http://www.example.net/ftp/foo.txt" is redirected to "ftp://ftp.example.net/foo.txt". If fetch(1) works correctly it should work as following.

a. fetch(1) accesses www.example.net through http-proxy.examole.com and sends request for http://www.example.net/ftp/foo.txt
b. www.example.net returns response that http://www.example.net/ftp/foo.txt is redirected to ftp://ftp.example.net/foo.txt
c. fetch(1) accesses ftp.example.net through ftp-proxy.example.com and sends request for ftp://ftp.example.net/foo.txt
d. ftp.example.net returns response for ftp://ftp.example.net/foo.txt
e. fetch(1) writes body of response to stdout.

But what really happens is as following.

a. fetch(1) accesses www.example.net through http-proxy.examole.com and sends request for http://www.example.net/ftp/foo.txt
b. www.example.net returns response that http://www.example.net/ftp/foo.txt is redirected to ftp://ftp.example.net/foo.txt
c'. fetch(1) tries to access ftp.example.net through http-proxy.example.com
d'. Since http-proxy.example.com doesn't permit FTP access to public network, It returns error result.
e'. fetch(1) exits with error.

3. Source of the problem

In general proxy usage depends on URL of request. So if a request to one URL is redirected to other URL, it may be necessary to change proxy usage. But current fetch(3) implementation keeps same proxy usage even if request is redirected to different URL. So under some situation fetch(3) makes wrong proxy usage against redirected URL.

4. Solution

Attached patch fixes the problem. It changes http_request_body() of lib/libfetch/http.c So `purl` is reallocated with new value of `url` each time redirection happens.
Comment 1 Yasuhiro Kimura freebsd_committer 2021-01-31 17:08:54 UTC
I created the review for the patch.

https://reviews.freebsd.org/D28437