Bug 20471

Summary: fetch(1) doesn't handle retrievals of unspecified length very well if FTP_PASSIVE_MODE is set
Product: Base System Reporter: malachai <malachai>
Component: binAssignee: Dag-Erling Smørgrav <des>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: 4.1-STABLE   
Hardware: Any   
OS: Any   

Description malachai 2000-08-07 23:10:00 UTC
When using fetch(1) to retrieve a file for which the server does not
specify the file's length, fetch reports retrieving 2^32-1 bytes.  It
will properly retrieve the file, but on completion it thinks the file
was truncated and then deletes it.

Note that unsetting FTP_PASSIVE_MODE causes fetch to work as expected.

Fix: 

Workaround:  make sure FTP_PASSIVE_MODE is not defined.
How-To-Repeat: First noticed when fetching packages from ftp.freebsd.org:

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/All/9e-1.0.tgz
Receiving 9e-1.0.tgz (4294967295 bytes): 0%
4343 bytes transferred in 3.4 seconds (1.25 kBps)
fetch: 9e-1.0.tgz appears to be truncated: 4343/4294967295 bytes

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/All/amaterus-0.32.5.tgz
Receiving amaterus-0.32.5.tgz (4294967295 bytes): 0%
30558 bytes transferred in 10.2 seconds (2.93 kBps)
fetch: amaterus-0.32.5.tgz appears to be truncated: 30558/4294967295 bytes

% unset FTP_PASSIVE_MODE
% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/All/amaterus-0.32.5.tgz
Receiving amaterus-0.32.5.tgz (30558 bytes): 100%
30558 bytes transferred in 10.2 seconds (2.93 kBps)
Comment 1 Sheldon Hearn freebsd_committer freebsd_triage 2000-08-08 09:51:03 UTC
Responsible Changed
From-To: freebsd-bugs->des

Over to the maintainer.
Comment 2 jeffe 2000-08-26 01:42:15 UTC
I have also seen this problem under 4.1-RELEASE

Debugging it, I have found that there is a problem in libfetch/http.c

The function _http_request declares the local variables clength, length,

and size to be of type of type size_t, which is an unsigned 32 bit int.
It then
immediately initializes them to -1, which will set them to 2^32-1.  If
no length or size
is ever gotten, then these remain with this value.  Ultimately, size is
copied into us->size (struct url_stat), which is of type off_t, which is

a signed 64 bit int, and the -1 meaning is then totally lost.  The main
fetch program does check for -1, but will never see it in this case.

The fix would be to either to change those three variables to type
ssize_t
(signed 32 bit int), and make everything that touches them match (a bit
of work), do the same thing, but change them to off_t (may be best), or
to
get rid of the '-1's  and replace them with SIZE_T_MAX,
and explicitly check for that when assigning  us->size,  setting it
directly
to -1 if so.

I have not checked to see if there are other places this might happen.

Jeff Eaton
Comment 3 jeffe 2000-08-26 02:23:17 UTC
OK, it wasn't that hard to fix.  I have appended a patch that
fixes the problem.  I didn't see this problem anywhere else in
the libfetch library.

I have tested this patch on cases that failed before, and it now
works.  I tested a few other cases, but I'm stuck behind a proxy,
and can't test very many cases.  It doesn't seem to have broken
anything.

Note that in my case, setting or unsetting FTP_PASSIVE_MODE didn't
help.  When it is using HTTP_PROXY, it should never even look at
that variable.  Shawn should test this in his case, and see if
this fixes his problem or not.

Jeff Eaton


--- /usr/src/lib/libfetch/http.c	Wed Jul 26 00:54:18 2000
+++ /tmp/libfetch/http.c	Fri Aug 25 17:49:12 2000
@@ -444,13 +444,13 @@
  * Parse a content-length header
  */
 static int
-_http_parse_length(char *p, size_t *length)
+_http_parse_length(char *p, off_t *length)
 {
-    size_t len;
+    off_t len;
     
     for (len = 0; *p && isdigit(*p); ++p)
 	len = len * 10 + (*p - '0');
-    DEBUG(fprintf(stderr, "content length: [\033[1m%d\033[m]\n", len));
+    DEBUG(fprintf(stderr, "content length: [\033[1m%lld\033[m]\n", len));
     *length = len;
     return 0;
 }
@@ -459,9 +459,9 @@
  * Parse a content-range header
  */
 static int
-_http_parse_range(char *p, off_t *offset, size_t *length, size_t *size)
+_http_parse_range(char *p, off_t *offset, off_t *length, off_t *size)
 {
-    int first, last, len;
+    off_t first, last, len;
 
     if (strncasecmp(p, "bytes ", 6) != 0)
 	return -1;
@@ -477,7 +477,7 @@
 	len = len * 10 + *p - '0';
     if (len < last - first + 1)
 	return -1;
-    DEBUG(fprintf(stderr, "content range: [\033[1m%d-%d/%d\033[m]\n",
+    DEBUG(fprintf(stderr, "content range: [\033[1m%lld-%lld/%lld\033[m]\n",
 		  first, last, len));
     *offset = first;
     *length = last - first + 1;
@@ -771,7 +771,7 @@
     int chunked, need_auth, noredirect, proxy, verbose;
     int code, fd, i, n;
     off_t offset;
-    size_t clength, length, size;
+    off_t clength, length, size;
     time_t mtime;
     char *p;
     FILE *f;
@@ -972,7 +972,7 @@
 	goto ouch;
     }
 
-    DEBUG(fprintf(stderr, "offset: %lld, length: %d, size: %d, clength: %d\n",
+    DEBUG(fprintf(stderr, "offset: %lld, length: %lld, size: %lld, clength: %lld\n",
 		  offset, length, size, clength));
     
     /* check for inconsistencies */
Comment 4 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2000-08-30 10:02:55 UTC
State Changed
From-To: open->closed

This was fixed five days before the PR was sent.