Bug 293312 - fetch fails to time out during either transfer or lookup (see 293124)
Summary: fetch fails to time out during either transfer or lookup (see 293124)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 14.3-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Dag-Erling Smørgrav
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2026-02-20 15:25 UTC by karl
Modified: 2026-02-26 18:17 UTC (History)
1 user (show)

See Also:
des: mfc-stable15+
des: mfc-stable14+
des: mfc-stable13+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description karl 2026-02-20 15:25:11 UTC
I'm not convinced this really is where it appears to be but since I'm redirected here we are.

I have seen this intermittently for a very long time but finally got a reproducing link from an associate.  Note that the target address DOES resolve (per "dig" or "curl" for that matter) and also DOES connect (per "curl" and complete.)

In EVERY CASE if fetch is run to this target it hangs.  It does NOT time out although it is specified to do so, and it is not intermittent either -- it hangs every time from multiple FreeBSD machines both on 14.3-RELEASE and also on 14.4-STABLE -- and the same problem is one I've noted for quite some time but did not report as I had no *reliable* way to make it happen -- if it hanged a second attempt would succeed, etc.

Now that I have a reproduceable link it merits being reported.  I am not convinced that the problem is in resolution simply because it works under curl and in addition drill, dig and nslookup all return a correct host address.  The target is a "CNAME" with both IPv6 and IPv6 addresses although using either "dig" or "drill" with no switches only returns (as expected) the CNAME and correct IPv4 resolution for it thus while it LOOKS like it is hanging there I'm not convinced (e.g. buffering of output to stdout may be involved here.)  Redirecting the command's output to a file along with stderr (e.g. >/tmp/xxx 2>&1) does not show when the ^C is delivered, obviously, but appears to indicate the *transfer* timed out rather than the lookup.

Target on emailed request (its not my site; I do know the owner personally) and entering the target into a browser -- or using curl -- works as expected.

Run with ktrace; 

It hangs with -vv in the same place.  Note that curl to the same target completes.

[karl@NewFS ~]$ ktrace fetch -T 2 -vv -o /tmp/fetch.tmp https:{elided}
scheme:   "https"
user:     ""
password: ""
host:     "{elided}"
port:     "0"
document: "......(elided).jpg"
---> {elided}:443
resolving server address: {elided}:443



^CSSL options: 82004850
Peer verification enabled
Using OpenSSL default CA cert file and path
Verify hostname
TLSv1.3 connection established using TLS_AES_256_GCM_SHA384
Certificate subject: /CN={elided}
Certificate issuer: /C=US/O=Let's Encrypt/CN=E7
requesting https:{elided}jpg
>>> GET {elided} HTTP/1.1
>>> Host: {elided}
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>

<<< HTTP/1.1 200 OK
<<< Server: nginx/1.28.1
<<< Date: Fri, 20 Feb 2026 14:26:11 GMT
<<< Content-Type: image/jpeg
<<< Content-Length: 348401
<<< Last-Modified: Thu, 05 Feb 2026 04:12:24 GMT
content length: [348401]
<<< Connection: close
last modified: [2026-02-05 04:12:24]
<<< ETag: "698418a8-550f1"
<<< Expires: Thu, 31 Dec 2037 23:55:55 GMT
<<< Cache-Control: max-age=315360000
<<< Strict-Transport-Security: max-age=63072000
<<< Accept-Ranges: bytes
<<<
offset 0, length -1, size -1, clength 348401
fetch: transfer timed out
fetch: transfer interrupted

$ kdump -tcst -f ktrace.out >ktrace.txt

Output (~2Mb) is at http:///www.denninger.net/ktrace.txt
Comment 1 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-02-20 18:02:38 UTC
I specifically asked for `kdump -E -tcst`, but ok, we can do without.

The log shows that SIGINT is delivered while we are waiting for DNS resolution.

The ktrace, however, shows that SIGINT is delivered after DNS resolution, while we are trying to connect to the server.  So this is, at root, a network issue, compounded by two separate issues in libfetch:

1) When DNS returns multiple addresses and connect(2) is interrupted, fetch_connect() should return immediately instead of trying the next address.  This is definitely a bug.

2) Libfetch does not attempt to enforce the timeout while connecting.  This was an intentional omission, because it's a genuinely hard problem to solve.  We should either fix it or document it.

Additionally, the fetch(3) manual page does not document fetchTimeout, and the fetch(1) manual page does not specify what the timeout applies to.
Comment 2 karl 2026-02-20 18:26:12 UTC
My bad on the requested options -- I've put that file at http://www.denninger.net/ktrace.txt2

A couple of additional notes on this -- I know the owner of the site in question; that particular call is for an image file, which in turn is a parsed thing (that is, the server's instance calls a fastcgi app that apparently does some indexing stuff to go into a repository, so its not a "straight" request from Ngnix.)  I suspect this is part of the issue and will speak to him about it, since if the v6 address is tried first and doesn't respond that is obviously a problem -- but one the timeout, I'd think, should apply to.

Curl when run in verbose mode does get all three IP addresses (two IPv6 and one IPv4) from the resolver and proceeds without incident; it connects on the IPv4 one from here.  It looks like curl is attempting both an IPv4 and v6 at the same time -- thus is it correct that libfetch is attempting them in series and ignoring the timeout during that attempt where curl is trying one of each address class in parallel?  Also, curl DOES have a "connect-timeout" and "max-time" pair of parameters (fetch only has -T which implies a timeout that is global in scope, but apparently as you note is not.)
Comment 3 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-02-20 19:01:59 UTC
I assure you nothing that is going on server-side has any bearing on this issue.

1. fetch() (in fetch(1)) sets a two-second alarm and calls fetchXGet() (in fetch(3)):

  5702 fetch    0.001722239 CALL  setitimer(ITIMER_REAL,0x19c4392eb30,0x19c4392eb10)
  5702 fetch    0.001725694 STRU  itimerval { .interval = {0, 0}, .value = {2, 0} }
  5702 fetch    0.001727675 STRU  itimerval { .interval = {0, 0}, .value = {0, 0} }
  5702 fetch    0.001729087 RET   setitimer 0

2. fetch_connect() performs DNS resolution:

  5702 fetch    0.002234669 CALL  connect(0x3,0x19c466e3924,0x10)
  5702 fetch    0.002236539 STRU  struct sockaddr { AF_INET, 127.0.0.1:53 }
  5702 fetch    0.002243647 RET   connect 0
  5702 fetch    0.002246855 CALL  sendto(0x3,0x256ace6832c0,0x27,0,0,0)
  5702 fetch    0.002254559 RET   sendto 39/0x27
  [...]
  5702 fetch    0.054034782 CALL  recvfrom(0x3,0x256ace672700,0x10000,0,0x19c4392d490,0x19c4392d07c)
  5702 fetch    0.054047302 STRU  struct sockaddr { AF_INET, 127.0.0.1:53 }
  5702 fetch    0.054049007 RET   recvfrom 119/0x77

3. fetch_connect() tries to connect to an IPv6 address, which hangs (and would have hung for 75 seconds if you had let it):

  5702 fetch    0.054203233 CALL  connect(0x3,0x256ace6550d0,0x1c)
  5702 fetch    0.054205332 STRU  struct sockaddr { AF_INET6, [2600:1702:6530:bdff:abac:7ca7:c15a:6646]:443 }

4. two seconds later, the alarm goes off and interrupts the connection attempt, the sigalrm flag is set:

  5702 fetch    2.002414880 RET   connect -1 errno 4 Interrupted system call
  5702 fetch    2.002429133 PSIG  SIGALRM caught handler=0x19c476acd30 mask=0x0 code=SI_KERNEL
  [...]
  5702 fetch    2.002450388 RET   sigreturn JUSTRETURN

5. fetch_connect() eats the EINTR and tries the next IPv6 address, which also hangs (and would have hung for 75 seconds if you had let it):

  5702 fetch    2.002481274 CALL  connect(0x3,0x256ace655120,0x1c)
  5702 fetch    2.002483286 STRU  struct sockaddr { AF_INET6, [2600:1702:6530:bdff:66f1:4bd2:4045:4550]:443 }

6. you get impatient and hit Ctrl-C, the second connection attempt is interrupted, the sigint flag gets set:

  5702 fetch    17.485940955 RET   connect -1 errno 4 Interrupted system call
  5702 fetch    17.486238976 PSIG  SIGINT caught handler=0x19c476acd30 mask=0x0 code=SI_KERNEL
  [...]
  5702 fetch    17.486320626 RET   sigreturn JUSTRETURN

7. fetch_connect() eats the EINTR again and tries an IPv4 address, which succeeds:

  5702 fetch    17.486426436 CALL  connect(0x3,0x256ace62b070,0x10)
  5702 fetch    17.486435791 STRU  struct sockaddr { AF_INET, 108.226.59.220:443 }
  5702 fetch    17.554460606 RET   connect 0

8. the request gets sent

9. the response header is received

10. fetchXGet() returns a non-null FILE * to fetch()

11. fetch() finally notices the sigalrm and sigint flags, prints messages about both, and aborts.
Comment 4 karl 2026-02-20 19:10:41 UTC
Well, other than the server advertising two unreachable IPv6 addresses, which in general it should not expose to the Internet as a whole... :-)

Given that this is likely hard to fix I suppose the "best answer" for now is to use curl instead which has both timeouts and appears to try one of each address class in parallel.
Comment 5 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-02-20 19:14:37 UTC
It's not hard to fix.  Making libfetch respect the timeout while connecting is hard, making it give up when fetch's alarm goes off is trivial.

https://reviews.freebsd.org/D55406
Comment 6 commit-hook freebsd_committer freebsd_triage 2026-02-21 01:19:35 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=afbdcd402bb439bd3d487baaad63b68e95929265

commit afbdcd402bb439bd3d487baaad63b68e95929265
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-02-21 01:18:15 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2026-02-21 01:18:15 +0000

    libfetch: Fail hard if interrupted while connecting

    This fixes an issue where the first address that DNS returns is blocked
    by a packet filter, so we hang for a while, then the user hits Ctrl-C,
    interrupting connect(2), whereupon we move on to the next address, get
    a connection, request the file, and return to fetch(1), which sees that
    SIGINT was caught and bails.

    Note that we make no attempt to enforce fetchTimeout in the connection
    phase, and never have.  It's feasible, but non-trivial, so we'll leave
    it as an exercise for future us.

    PR:             293312
    MFC after:      1 week
    Reviewed by:    imp
    Differential Revision:  https://reviews.freebsd.org/D55406

 lib/libfetch/common.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Comment 7 commit-hook freebsd_committer freebsd_triage 2026-02-26 04:22:56 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=3b879f771bd462948320373dee7b0e5f281e8e08

commit 3b879f771bd462948320373dee7b0e5f281e8e08
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-02-21 01:18:15 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2026-02-26 04:02:28 +0000

    libfetch: Fail hard if interrupted while connecting

    This fixes an issue where the first address that DNS returns is blocked
    by a packet filter, so we hang for a while, then the user hits Ctrl-C,
    interrupting connect(2), whereupon we move on to the next address, get
    a connection, request the file, and return to fetch(1), which sees that
    SIGINT was caught and bails.

    Note that we make no attempt to enforce fetchTimeout in the connection
    phase, and never have.  It's feasible, but non-trivial, so we'll leave
    it as an exercise for future us.

    PR:             293312
    MFC after:      1 week
    Reviewed by:    imp
    Differential Revision:  https://reviews.freebsd.org/D55406

    (cherry picked from commit afbdcd402bb439bd3d487baaad63b68e95929265)

 lib/libfetch/common.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Comment 8 commit-hook freebsd_committer freebsd_triage 2026-02-26 04:22:59 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=cca6f5eadb796b03379eb21f38c74ca46a64e45b

commit cca6f5eadb796b03379eb21f38c74ca46a64e45b
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-02-21 01:18:15 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2026-02-26 04:03:43 +0000

    libfetch: Fail hard if interrupted while connecting

    This fixes an issue where the first address that DNS returns is blocked
    by a packet filter, so we hang for a while, then the user hits Ctrl-C,
    interrupting connect(2), whereupon we move on to the next address, get
    a connection, request the file, and return to fetch(1), which sees that
    SIGINT was caught and bails.

    Note that we make no attempt to enforce fetchTimeout in the connection
    phase, and never have.  It's feasible, but non-trivial, so we'll leave
    it as an exercise for future us.

    PR:             293312
    MFC after:      1 week
    Reviewed by:    imp
    Differential Revision:  https://reviews.freebsd.org/D55406

    (cherry picked from commit afbdcd402bb439bd3d487baaad63b68e95929265)

 lib/libfetch/common.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Comment 9 commit-hook freebsd_committer freebsd_triage 2026-02-26 04:24:00 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c0ec07280e920cf8c672cffa0730676e2dd40d72

commit c0ec07280e920cf8c672cffa0730676e2dd40d72
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-02-21 01:18:15 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2026-02-26 04:03:52 +0000

    libfetch: Fail hard if interrupted while connecting

    This fixes an issue where the first address that DNS returns is blocked
    by a packet filter, so we hang for a while, then the user hits Ctrl-C,
    interrupting connect(2), whereupon we move on to the next address, get
    a connection, request the file, and return to fetch(1), which sees that
    SIGINT was caught and bails.

    Note that we make no attempt to enforce fetchTimeout in the connection
    phase, and never have.  It's feasible, but non-trivial, so we'll leave
    it as an exercise for future us.

    PR:             293312
    MFC after:      1 week
    Reviewed by:    imp
    Differential Revision:  https://reviews.freebsd.org/D55406

    (cherry picked from commit afbdcd402bb439bd3d487baaad63b68e95929265)

 lib/libfetch/common.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Comment 10 commit-hook freebsd_committer freebsd_triage 2026-02-26 18:17:00 UTC
A commit in branch releng/14.4 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ca76ec7c4e563292a0c75781683b0421a4439a02

commit ca76ec7c4e563292a0c75781683b0421a4439a02
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-02-21 01:18:15 +0000
Commit:     Colin Percival <cperciva@FreeBSD.org>
CommitDate: 2026-02-26 18:00:38 +0000

    libfetch: Fail hard if interrupted while connecting

    This fixes an issue where the first address that DNS returns is blocked
    by a packet filter, so we hang for a while, then the user hits Ctrl-C,
    interrupting connect(2), whereupon we move on to the next address, get
    a connection, request the file, and return to fetch(1), which sees that
    SIGINT was caught and bails.

    Note that we make no attempt to enforce fetchTimeout in the connection
    phase, and never have.  It's feasible, but non-trivial, so we'll leave
    it as an exercise for future us.

    PR:             293312
    MFC after:      1 week
    Reviewed by:    imp
    Differential Revision:  https://reviews.freebsd.org/D55406

    (cherry picked from commit afbdcd402bb439bd3d487baaad63b68e95929265)
    (cherry picked from commit cca6f5eadb796b03379eb21f38c74ca46a64e45b)

 lib/libfetch/common.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)