Bug 153211 - freebsd-update(8) can not fetch updates over a proxy with mandatory authorization
Summary: freebsd-update(8) can not fetch updates over a proxy with mandatory authoriza...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 8.1-RELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-12-16 13:00 UTC by Mario Kleinboelting
Modified: 2019-03-12 23:00 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mario Kleinboelting 2010-12-16 13:00:18 UTC
I installed a FreeBSD 8.1 i386 Server from CD-ROM.
After installation i tried to update the server with freebsd-update and portsnap. 
The freebsd-update did not work properly in our environment.
freebsd-update can not fetch updates because the internet connection is established with an auth proxy. Our proxy requires username an password.

Two environment variables are set for the proxy connection. http_proxy and ftp_proxy.
Both contains the proxy URL and the proxy credentials and are exported.

http_proxy="http://username:password@http-proxy:3128"
ftp_proxy="http://username:password@ftp-proxy:3428"

The start of freebsd-update -v debug fetch says:
----------------------------------------------------------------------
# freebsd-update -v debug fetch
Looking up update.FreeBSD.org mirrors... none found.
Fetching metadata signature for 8.1-RELEASE from update.FreeBSD.org... 
latest.ssl                                    100% of  512  B 1667 kBps
done.
Fetching metadata index... 
14b0dfddc908e278d231e3290507fe1ab01e6ceb553a9d100% of  225  B  552 kBps
done.
Fetching 2 metadata files... 
/usr/libexec/phttpget update.FreeBSD.org 8.1-RELEASE/i386/m/61dfab427765793984bc911a1eab992cffc22c26bba438fe11a665820fcf97f2.gz 8.1-RELEASE/i386/m/e376c4c604af7362453546c3ebb4ad439f900173000d8412aabcf0101c92311d.gz
phttpget: host = username, port = password@http-proxy:3128: servname not supported for ai_socktype
failed.
----------------------------------------------------------------------
freebsd-update seems to evaluate the variable and cuts server and port with ":".
Now we have ":" two times. Delimiter for username and passwort and delimiter for proxyserver and port.


I have tested the variable http_proxy and ftp_proxy with only the proxy server and the variable http_proxy_auth with the proxy credentials. 

Then freebsd-update says:
----------------------------------------------------------------------
# export http_proxy="http://http-proxy:3128"
# export ftp_proxy="http://ftp-proxy:3428"
# export http_proxy_auth="basic:*:username:password"
# freebsd-update -v debug fetch
Looking up update.FreeBSD.org mirrors... none found.
Fetching metadata signature for 8.1-RELEASE from update.FreeBSD.org... 
fetch: http://update.FreeBSD.org/8.1-RELEASE/i386/latest.ssl: Proxy Authentication Required
failed.
No mirrors remaining, giving up.
----------------------------------------------------------------------
It seems that freebsd-update did not recognize the http_proxy_auth variable

portsnap and /usr/bin/fetch works fine with our proxy an handles the authentication info in http_proxy and ftp_proxy correct.

Our Proxyserver is squid on solaris. Every connection to the internet must use username and a password by corporate policy.

FYI: Username and password are not my real proxy credentials...

Fix: 

Rewrite of /usr/libexec/phttpget and freebsd-update to work with auth information in proxy URL's.
How-To-Repeat: Use freebsd-update behind a (squid) proxy with mandatory username and password authorization.
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2010-12-16 16:31:46 UTC
Responsible Changed
From-To: freebsd-www->cperciva

Over to maintainer.
Comment 2 Mario Kleinboelting 2010-12-17 20:21:48 UTC
I did a little bit of testing to reproduce the problem in my private 
environment.
I can reproduce the problem on two patched FreeBSD 8.1 systems with 
8.1-RELEASE-p2 in my LAN also.

# uname -a
FreeBSD hera 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:55:53 UTC 
2010     root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

# uname -a
FreeBSD horus 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 
UTC 2010     root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64


To build a testproxy I used this squid config to reconfigure my 
testproxy (squid-3.0.25_2 of FreeBSD 8.1):
auth_param basic program /usr/local/libexec/squid/ncsa_auth 
/usr/local/etc/squid/squidpasswd
acl password proxy_auth REQUIRED
http_access allow password

/usr/local/etc/squid/squidpasswd contains the credentials in htpasswd 
format.


When /var/db/freebsd-update contains old cached data the freebsd-update 
call is not a problem:
------------------------------------------------
# export http_proxy="http://username:password@192.168.0.5:3128/"
# find /var/db/freebsd-update/
/var/db/freebsd-update/
/var/db/freebsd-update/files
/var/db/freebsd-update/files/61dfab427765793984bc911a1eab992cffc22c26bba438fe11a665820fcf97f2.gz
/var/db/freebsd-update/files/e376c4c604af7362453546c3ebb4ad439f900173000d8412aabcf0101c92311d.gz
/var/db/freebsd-update/tINDEX.present
/var/db/freebsd-update/serverlist_tried
/var/db/freebsd-update/serverlist_full
/var/db/freebsd-update/serverlist
/var/db/freebsd-update/pub.ssl
/var/db/freebsd-update/tag
# freebsd-update -v debug fetch
Looking up update.FreeBSD.org mirrors... 4 mirrors found.
Fetching metadata signature for 8.1-RELEASE from update5.FreeBSD.org...
latest.ssl                                    100% of  512  B 1004 kBps
done.
Fetching metadata index...
14b0dfddc908e278d231e3290507fe1ab01e6ceb553a9d100% of  225  B  426 kBps
done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 8.1-RELEASE-p2.
------------------------------------------------


When the cached content in /var/db/freebsd-update/files is deleted or 
the directory /var/db/freebsd-update is empty (on a fresh installed 
system) the problem is there:
------------------------------------------------
# export http_proxy="http://username:password@192.168.0.5:3128/"
# rm 
/var/db/freebsd-update/files/61dfab427765793984bc911a1eab992cffc22c26bba438fe11a665820fcf97f2.gz 
/var/db/freebsd-update/files/e376c4c604af7362453546c3ebb4ad439f900173000d8412aabcf0101c92311d.gz
# freebsd-update -v debug fetch
Looking up update.FreeBSD.org mirrors... 4 mirrors found.
Fetching metadata signature for 8.1-RELEASE from update5.FreeBSD.org...
latest.ssl                                    100% of  512  B  948 kBps
done.
Fetching metadata index...
14b0dfddc908e278d231e3290507fe1ab01e6ceb553a9d100% of  225  B   39 kBps
done.
Fetching 2 metadata files...
/usr/libexec/phttpget update5.FreeBSD.org 
8.1-RELEASE/i386/m/61dfab427765793984bc911a1eab992cffc22c26bba438fe11a665820fcf97f2.gz 
8.1-RELEASE/i386/m/e376c4c604af7362453546c3ebb4ad439f900173000d8412aabcf0101c92311d.gz
phttpget: host = username, port = password@192.168.0.5:3128: servname 
not supported for ai_socktype
failed.
------------------------------------------------


Regards
Mario
Comment 3 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:41:42 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 4 Gerald Aryeetey 2019-01-14 21:42:02 UTC
This is likely because environment variables are case sensitive and as seen in the documentation the expected variables are "HTTP_PROXY", "HTTP_PROXY_AUTH", "HTTP_USER_AGENT" and "HTTP_TIMEOUT".

Interestingly, lowercase "http_proxy" is accepted but for the other variables case matters.

HTTP_PROXY does not accept username:password. It only expects the host name and potentially the port number. All authentication information should be given through HTTP_PROXY_AUTH.
Comment 5 Colin Percival freebsd_committer 2019-03-12 23:00:17 UTC
Drop freebsd-update PRs which were assigned to me.  I'm not working on this code any more.