Created attachment 184056 [details] patched http.c file allowing authed connect tunnel with https as a target Using: export HTTP_PROXY_AUTH="basic:*:username:password" export HTTP_PROXY="http://<proxy_ip>:3128" following will FAIL with 407: fetch https://<pkgrepohost>/repo/meta.txz following will WORK: fetch http://<pkgrepohost>/repo/meta.txz this is also affecting pkgng which are compiling libfetch.
Patch allows fetch https://<pkgrepohost>/repo/meta.txz to WORK, if that was not clear in previous comment.
Can you send a patch rather than the full file patched? diff -u http.c.orig http.c > http.c.patch should make one for you if you don't know how to make one.
Created attachment 184057 [details] patch of the diff from freebsd 11.0 release src Added patch on request
Created attachment 184069 [details] patch of the diff from freebsd 11.0 release src Removed a segfault when not supplying auth information in either url or HTTP_PROXY_AUTH improved error message and exit message on fail.
We also have big troubles with fetch, no credentials are sent to remote host when CONNECT method used. Please fix it ASAP. $ export HTTP_PROXY_AUTH='basic:*:proxy_user:PROXY_PASS' $ export HTTP_PROXY='http://local.proxy.me:3128/' $ export HTTPS_PROXY='http://local.proxy.me:3128/' fetch HTTP url via PROXY: $ fetch http://google.com -vv scheme: "http" user: "" password: "" host: "google.com" port: "0" document: "/" scheme: "http" user: "" password: "" host: "local.proxy.me" port: "3128" document: "/" ---> local.proxy.me:3128 resolving server address: local.proxy.me:3128 requesting http://google.com/ >>> GET http://google.com/ HTTP/1.1 >>> Host: google.com >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 407 Proxy Authentication Required proxy requires authorization <<< Proxy-Authenticate: NEGOTIATE <<< Proxy-Authenticate: NTLM <<< Proxy-Authenticate: BASIC realm="IWA3" <<< Cache-Control: no-cache <<< Pragma: no-cache <<< Content-Type: text/html; charset=utf-8 <<< Proxy-Connection: close <<< Set-Cookie: BCSI-CS-e773a25e87ae05cc=2; Path=/ <<< Connection: close <<< Content-Length: 849 <<< content length: [849] ---> local.proxy.me:3128 resolving server address: local.proxy.me:3128 requesting http://google.com/ >>> GET http://google.com/ HTTP/1.1 >>> Host: google.com basic: usr: [proxy_user] basic: pwd: [PROXY_PASS] >>> Proxy-Authorization: Basic c3ZjX2VzbWd43m9ib3Q6SFA4X325KjkjekgsXF5jP1UwTiI= >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 302 Found <<< Content-Type: text/html; charset=UTF-8 <<< Referrer-Policy: no-referrer <<< Location: http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw <<< Content-Length: 268 302 redirect to http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw scheme: "http" user: "" password: "" host: "www.google.ru" port: "0" document: "/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw" <<< Date: Wed, 18 Oct 2017 13:31:02 GMT content length: [268] <<< Cache-Control: private, proxy-revalidate <<< Connection: close <<< ---> local.proxy.me:3128 resolving server address: local.proxy.me:3128 requesting http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw >>> GET http://www.google.ru/?gfe_rd=cr&dcr=0&ei=llfnWaf3F7HG7gT7p4-gBw HTTP/1.1 >>> Host: www.google.ru basic: usr: [proxy_user] basic: pwd: [PROXY_PASS] >>> Proxy-Authorization: Basic c3ZjX2VzbWd43m9ib3Q6SFA4X325KjkjekgsXF5jP1UwTiI= >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 200 OK <<< Date: Wed, 18 Oct 2017 13:31:02 GMT <<< Expires: -1 <<< Content-Type: text/html; charset=windows-1251 <<< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." <<< Server: gws <<< X-XSS-Protection: 1; mode=block <<< X-Frame-Options: SAMEORIGIN <<< Accept-Ranges: none <<< Vary: Accept-Encoding <<< Transfer-Encoding: chunked <<< Cache-Control: private, max-age=0, proxy-revalidate <<< Connection: close <<< Set-Cookie: 1P_JAR=2017-10-18-13; expires=Wed, 25-Oct-2017 13:31:02 GMT; path=/; domain=.google.ru <<< Set-Cookie: NID=114=BN3CH2k6S-NantH3YSo7BDamqqS4zq65i3TCQfxjPtiPwJ3cWwy-Ck3uFavI_ZoDw_4Kw_5gSKNUmxZp-zowexGOC0pywbNpIIAoGX7p_-HYEWpPtDjMalnCCj9BGf8I; expires=Thu, 19-Apr-2018 13:31:02 GMT; path=/; domain=.google.ru; HttpOnly <<< offset 0, length -1, size -1, clength -1 fetch: http://google.com: size of remote file is not known local size / mtime: 11314 / 1508333405 google.com 0 B 0 Bps<<< 2c39 http_new_chunk(): new chunk: 11321 (11321) <<< 0 http_new_chunk(): end of last chunk google.com 11 kB 134 MBps 00m00s fetch HTTPS url via PROXY: $ fetch https://google.com -vv scheme: "https" user: "" password: "" host: "google.com" port: "0" document: "/" scheme: "http" user: "" password: "" host: "local.proxy.me" port: "3128" document: "/" ---> local.proxy.me:3128 resolving server address: local.proxy.me:3128 >>> CONNECT google.com:443 HTTP/1.1 >>> Host: google.com:443 >>> <<< HTTP/1.1 407 Proxy Authentication Required fetch: https://google.com: Proxy Authentication Required
DES - Ping. Don't want this to get dropped on the floor.
Ping again.
I encountered the same issue today and was wondering why the patch hasn't been implemented yet.
^Triage: - Assignee timeout, reset assignee. Open to take - 10.x is EoL, cancel MFC to stable/10 @Egil Could you please check that attachment 184069 [details] still applies to CURRENT (head), and rebase/replace it if it doesn't. Thank you
Created attachment 209479 [details] simple patch I have a simple patch. previous patch seems complex for me. and it seems broken. setting conn->err after conn is freed with fetch_close can cause SIGSEGV.
I ran into the same problem using a base r364806 CURRENT. After tracing down the cause, I found this bug report and used yamayan's simple_patch (comment #10). It fixed the bug for me very well. Thank you. Bug #235548 seems to be a duplicate. It is about fetch(1) but the described problem looks like this libfetch bug.
Reset Assignee after 10 months of inactivity.
Hi. Can anyone help with adding this latest patch?
I just want to note that this bug still happens and this patch still works. We added it to pfSense to fix a problem reported by our users
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=345c30a94f6425954163f1e0b075a75f603d27cd commit 345c30a94f6425954163f1e0b075a75f603d27cd Author: Renato Botelho <garga@FreeBSD.org> AuthorDate: 2021-04-01 21:02:57 +0000 Commit: Renato Botelho <garga@FreeBSD.org> CommitDate: 2021-04-01 21:02:57 +0000 libfetch: Retry with proxy auth when server returns 407 PR: 220468 Submitted by: Egil Hasting <egil.hasting@higen.org> (based on) Reviewed by: kevans, kp Approved by: kp MFC after: 2 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D29533 lib/libfetch/http.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 49 insertions(+), 6 deletions(-)
(In reply to commit-hook from comment #15) Is this targeted for releng/13 and stable/* ? That would be lovely. ^Triage: Assign to committer resolving
(In reply to Kubilay Kocak from comment #16) It's too late for releng/13.0, we already have an RC5. I plan to merge it to all supported stable branches
(In reply to Renato Botelho from comment #17) ^Triage: Thanks, please include PR: references in those merges :)