Bug 220468 - libfetch: Does not handle 407 (proxy auth) when connecting to HTTPS using connect tunnel
Summary: libfetch: Does not handle 407 (proxy auth) when connecting to HTTPS using con...
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 11.0-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Renato Botelho
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-07-04 13:51 UTC by Egil Hasting
Modified: 2021-04-12 05:03 UTC (History)
16 users (show)

See Also:
koobs: mfc-stable13?
garga: mfc-stable12?
koobs: mfc-stable11?


Attachments
patched http.c file allowing authed connect tunnel with https as a target (52.13 KB, patch)
2017-07-04 13:51 UTC, Egil Hasting
no flags Details | Diff
patch of the diff from freebsd 11.0 release src (3.28 KB, patch)
2017-07-04 18:15 UTC, Egil Hasting
no flags Details | Diff
patch of the diff from freebsd 11.0 release src (3.48 KB, patch)
2017-07-05 11:39 UTC, Egil Hasting
no flags Details | Diff
simple patch (1.93 KB, patch)
2019-11-27 12:58 UTC, yamayan
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Egil Hasting 2017-07-04 13:51:21 UTC
Created attachment 184056 [details]
patched http.c file allowing authed connect tunnel with https as a target

Using:
export HTTP_PROXY_AUTH="basic:*:username:password"
export HTTP_PROXY="http://<proxy_ip>:3128"


following will FAIL with 407:
fetch https://<pkgrepohost>/repo/meta.txz  

following will WORK:
fetch http://<pkgrepohost>/repo/meta.txz


this is also affecting pkgng which are compiling libfetch.
Comment 1 Egil Hasting 2017-07-04 13:54:28 UTC
Patch allows 
fetch https://<pkgrepohost>/repo/meta.txz  

to WORK, if that was not clear in previous comment.
Comment 2 Baptiste Daroussin freebsd_committer 2017-07-04 15:35:34 UTC
Can you send a patch rather than the full file patched?

diff -u http.c.orig http.c > http.c.patch

should make one for you if you don't know how to make one.
Comment 3 Egil Hasting 2017-07-04 18:15:43 UTC
Created attachment 184057 [details]
patch of the diff from freebsd 11.0 release src

Added patch on request
Comment 4 Egil Hasting 2017-07-05 11:39:30 UTC
Created attachment 184069 [details]
patch of the diff from freebsd 11.0 release src

Removed a segfault when not supplying auth information in either url or HTTP_PROXY_AUTH
improved error message and exit message on fail.
Comment 5 Eugene V. Lyapin 2017-10-18 13:44:48 UTC
We also have big troubles with fetch, no credentials are sent to remote host when CONNECT method used. Please fix it ASAP.

$ export HTTP_PROXY_AUTH='basic:*:proxy_user:PROXY_PASS'
$ export HTTP_PROXY='http://local.proxy.me:3128/'
$ export HTTPS_PROXY='http://local.proxy.me:3128/'

fetch HTTP url via PROXY:

$ fetch http://google.com -vv
scheme:   "http"
user:     ""
password: ""
host:     "google.com"
port:     "0"
document: "/"
scheme:   "http"
user:     ""
password: ""
host:     "local.proxy.me"
port:     "3128"
document: "/"
---> local.proxy.me:3128
resolving server address: local.proxy.me:3128
requesting http://google.com/
>>> GET http://google.com/ HTTP/1.1
>>> Host: google.com
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 407 Proxy Authentication Required
proxy requires authorization
<<< Proxy-Authenticate: NEGOTIATE
<<< Proxy-Authenticate: NTLM
<<< Proxy-Authenticate: BASIC realm="IWA3"
<<< Cache-Control: no-cache
<<< Pragma: no-cache
<<< Content-Type: text/html; charset=utf-8
<<< Proxy-Connection: close
<<< Set-Cookie: BCSI-CS-e773a25e87ae05cc=2; Path=/
<<< Connection: close
<<< Content-Length: 849
<<<
content length: [849]
---> local.proxy.me:3128
resolving server address: local.proxy.me:3128
requesting http://google.com/
>>> GET http://google.com/ HTTP/1.1
>>> Host: google.com
basic: usr: [proxy_user]
basic: pwd: [PROXY_PASS]
>>> Proxy-Authorization: Basic c3ZjX2VzbWd43m9ib3Q6SFA4X325KjkjekgsXF5jP1UwTiI=
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 302 Found
<<< Content-Type: text/html; charset=UTF-8
<<< Referrer-Policy: no-referrer
<<< Location: http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw
<<< Content-Length: 268
302 redirect to http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw
scheme:   "http"
user:     ""
password: ""
host:     "www.google.ru"
port:     "0"
document: "/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw"
<<< Date: Wed, 18 Oct 2017 13:31:02 GMT
content length: [268]
<<< Cache-Control: private, proxy-revalidate
<<< Connection: close
<<<
---> local.proxy.me:3128
resolving server address: local.proxy.me:3128
requesting http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw
>>> GET http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw HTTP/1.1
>>> Host: www.google.ru
basic: usr: [proxy_user]
basic: pwd: [PROXY_PASS]
>>> Proxy-Authorization: Basic c3ZjX2VzbWd43m9ib3Q6SFA4X325KjkjekgsXF5jP1UwTiI=
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 200 OK
<<< Date: Wed, 18 Oct 2017 13:31:02 GMT
<<< Expires: -1
<<< Content-Type: text/html; charset=windows-1251
<<< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
<<< Server: gws
<<< X-XSS-Protection: 1; mode=block
<<< X-Frame-Options: SAMEORIGIN
<<< Accept-Ranges: none
<<< Vary: Accept-Encoding
<<< Transfer-Encoding: chunked
<<< Cache-Control: private, max-age=0, proxy-revalidate
<<< Connection: close
<<< Set-Cookie: 1P_JAR=2017-10-18-13; expires=Wed, 25-Oct-2017 13:31:02 GMT; path=/; domain=.google.ru
<<< Set-Cookie: NID=114=BN3CH2k6S-NantH3YSo7BDamqqS4zq65i3TCQfxjPtiPwJ3cWwy-Ck3uFavI_ZoDw_4Kw_5gSKNUmxZp-zowexGOC0pywbNpIIAoGX7p_-HYEWpPtDjMalnCCj9BGf8I; expires=Thu, 19-Apr-2018 13:31:02 GMT; path=/; domain=.google.ru; HttpOnly
<<<
offset 0, length -1, size -1, clength -1
fetch: http://google.com: size of remote file is not known
local size / mtime: 11314 / 1508333405
google.com                                               0  B    0  Bps<<< 2c39
http_new_chunk(): new chunk: 11321 (11321)
<<< 0
http_new_chunk(): end of last chunk
google.com                                              11 kB  134 MBps 00m00s

fetch HTTPS url via PROXY:

$ fetch https://google.com -vv
scheme:   "https"
user:     ""
password: ""
host:     "google.com"
port:     "0"
document: "/"
scheme:   "http"
user:     ""
password: ""
host:     "local.proxy.me"
port:     "3128"
document: "/"
---> local.proxy.me:3128
resolving server address: local.proxy.me:3128
>>> CONNECT google.com:443 HTTP/1.1
>>> Host: google.com:443
>>>
<<< HTTP/1.1 407 Proxy Authentication Required
fetch: https://google.com: Proxy Authentication Required
Comment 6 Conrad Meyer freebsd_committer 2017-12-28 18:03:12 UTC
DES - Ping.  Don't want this to get dropped on the floor.
Comment 7 Lars Engels freebsd_committer 2019-04-18 08:04:54 UTC
Ping again.
Comment 8 tim1kopplow 2019-11-20 12:09:30 UTC
I encountered the same issue today and was wondering why the patch hasn't been implemented yet.
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2019-11-22 01:06:55 UTC
^Triage: 

- Assignee timeout, reset assignee. Open to take
- 10.x is EoL, cancel MFC to stable/10

@Egil Could you please check that attachment 184069 [details] still applies to CURRENT (head), and rebase/replace it if it doesn't. Thank you
Comment 10 yamayan 2019-11-27 12:58:39 UTC
Created attachment 209479 [details]
simple patch

I have a simple patch.

previous patch seems complex for me.
and it seems broken.

setting conn->err after conn is freed with fetch_close
can cause SIGSEGV.
Comment 11 iz-rpi03 2020-09-01 06:46:27 UTC
I ran into the same problem using a base r364806 CURRENT.

After tracing down the cause, I found this bug report and used yamayan's
simple_patch (comment #10). It fixed the bug for me very well. Thank you.

Bug #235548 seems to be a duplicate. It is about fetch(1) but the described
problem looks like this libfetch bug.
Comment 12 Eugene Grosbein freebsd_committer 2020-09-01 07:14:59 UTC
Reset Assignee after 10 months of inactivity.
Comment 13 Kirill 2020-09-24 08:26:58 UTC
Hi. Can anyone help with adding this latest patch?
Comment 14 Renato Botelho freebsd_committer 2021-03-31 19:48:15 UTC
I just want to note that this bug still happens and this patch still works.  We added it to pfSense to fix a problem reported by our users
Comment 15 commit-hook freebsd_committer 2021-04-01 21:07:41 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=345c30a94f6425954163f1e0b075a75f603d27cd

commit 345c30a94f6425954163f1e0b075a75f603d27cd
Author:     Renato Botelho <garga@FreeBSD.org>
AuthorDate: 2021-04-01 21:02:57 +0000
Commit:     Renato Botelho <garga@FreeBSD.org>
CommitDate: 2021-04-01 21:02:57 +0000

    libfetch: Retry with proxy auth when server returns 407

    PR:             220468
    Submitted by:   Egil Hasting <egil.hasting@higen.org> (based on)
    Reviewed by:    kevans, kp
    Approved by:    kp
    MFC after:      2 weeks
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D29533

 lib/libfetch/http.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 49 insertions(+), 6 deletions(-)
Comment 16 Kubilay Kocak freebsd_committer freebsd_triage 2021-04-02 02:29:49 UTC
(In reply to commit-hook from comment #15)

Is this targeted for releng/13 and stable/* ? That would be lovely.

^Triage: Assign to committer resolving
Comment 17 Renato Botelho freebsd_committer 2021-04-05 13:25:55 UTC
(In reply to Kubilay Kocak from comment #16)
It's too late for releng/13.0, we already have an RC5.  I plan to merge it to all supported stable branches
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2021-04-12 05:03:09 UTC
(In reply to Renato Botelho from comment #17)

^Triage: Thanks, please include PR: references in those merges :)