Bug 219518

Fetch (and I suspect libfetch) is unable to use an FTP proxy as specified by the FTP_PROXY environment variable.  Attempting to download a file through an FTP proxy (e.g. Squid) results in fetch timing out.  The error appears to be that fetch is unable to interact with or properly negotiate with the proxy server.  I suspect that this bug has not been noticed before because common utilities that rely on fetch and proxies often use the HTTP protocol.

My setup:
    Client Machine -> Squid ver. 3.5.24 -> FTP Server

Following environment variables are set:

    setenv HTTP_PROXY http://192.168.1.1:3128
    setenv FTP_PROXY ftp://192.168.1.1:3128

Test Commands and Output:

Test Case 1 (to show Fetch Working with HTTP Proxy):

fetch -vvv http://www.cnn.com
scheme:   [http]
user:     []
password: []
host:     [www.cnn.com]
port:     [0]
document: [/]
scheme:   [http]
user:     []
password: []
host:     [192.168.1.1]
port:     [3128]
document: [/]
---> 192.168.1.1:3128
looking up 192.168.1.1
connecting to 192.168.1.1:3128
requesting http://www.cnn.com/
>>> GET http://www.cnn.com/ HTTP/1.1
>>> Host: www.cnn.com
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 200 OK
<<< access-control-allow-origin: *
<<< Cache-Control: max-age=60
<<< content-security-policy: default-src 'self' blob: https://*.cnn.com:* http://*.cnn.com:* *.cnn.io:* *.cnn.net:* *.turner.com:* *.turner.io:* *.ugdturner.com:* *.vgtf.net:*; script-src 'unsafe-eval' 'unsafe-inline' 'self' *; style-src 'unsafe-inline' 'self' blob: *; child-src 'self' blob: *; frame-src 'self' *; object-src 'self' *; img-src 'self' data: blob: *; media-src 'self' blob: *; font-src 'self' data: *; connect-src 'self' *;
<<< Content-Type: text/html; charset=utf-8
<<< x-content-type-options: nosniff
<<< x-frame-options: SAMEORIGIN
<<< x-servedByHost: ::ffff:172.17.106.8
<<< x-xss-protection: 1; mode=block
<<< Fastly-Debug-Digest: 46be59e687681f2cbdc5286ab50024ed035dc360065b1aec7ce355bf418daeb9
<<< Content-Length: 139907
<<< Accept-Ranges: bytes
content length: [139907]
<<< Date: Thu, 25 May 2017 02:20:38 GMT
<<< Age: 21
<<< Set-Cookie: countryCode=US; Domain=.cnn.com; Path=/
<<< Set-Cookie: geoData=marietta|GA|30062|US|NA; Domain=.cnn.com; Path=/
<<< X-Served-By: cache-iad2146-IAD, cache-atl6230-ATL
<<< X-Cache: HIT, HIT
<<< X-Cache-Hits: 2, 4
<<< X-Timer: S1495678839.626481,VS0,VE0
<<< Vary: Accept-Encoding, Fastly-SSL, Fastly-SSL
<<< X-Cache: MISS from pfsense.johnllyon.com
<<< X-Cache-Lookup: HIT from pfsense.johnllyon.com:3128
<<< Via: 1.1 varnish, 1.1 varnish, ICAP/1.0 pfSense.johnllyon.com (C-ICAP/0.4.4 SquidClamav/Antivirus service ), 1.1 pfsense.johnllyon.com (squid/3.5.24)
<<< Connection: close
<<<
offset 0, length -1, size -1, clength 139907
local size / mtime: 139503 / 1495678456
remote size / mtime: 139907 / 0
www.cnn.com                                   100% of  136 kB   95 MBps 00m00s

Test Case 2 and 3 and Output (to show Fetch Timeout with Multiple FTP Servers -- expected behavior is for requested text files to be displayed to stdout):

root@zeus# fetch -vvv ftp://ftp.netbsd.org/robots.txt
scheme:   [ftp]
user:     []
password: []
host:     [ftp.netbsd.org]
port:     [0]
document: [/robots.txt]
scheme:   [ftp]
user:     []
password: []
host:     [192.168.1.1]
port:     [3128]
document: [/]
---> 192.168.1.1:3128
looking up 192.168.1.1
connecting to 192.168.1.1:3128
fetch: transfer timed out


fetch -vvv ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.0-RELEASE/MANIFEST
scheme:   [ftp]
user:     []
password: []
host:     [ftp.freebsd.org]
port:     [0]
document: [/pub/FreeBSD/releases/amd64/11.0-RELEASE/MANIFEST]
scheme:   [ftp]
user:     []
password: []
host:     [192.168.1.1]
port:     [3128]
document: [/]
---> 192.168.1.1:3128
looking up 192.168.1.1
connecting to 192.168.1.1:3128
fetch: transfer timed out

Test Case 4:

I setup my browser (Chrome) to use the Squid FTP proxy and navigated to the two FTP servers used in Test Case 2 and Test Case 3.  In both cases, I could access the FTP servers and files and I was presented with the Squid logo in browser to notify me I was using the Squid proxy.  Hence the conclusion that the bug is in fetch.