Bug 13691
| Summary: | tcpslice cannot extract over 2GB part of trace file | ||
|---|---|---|---|
| Product: | Base System | Reporter: | tate <tate> |
| Component: | bin | Assignee: | Bruce M Simpson <bms> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | ||
| Priority: | Normal | ||
| Version: | 3.1-RELEASE | ||
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
tate
1999-09-11 04:20:00 UTC
----Next_Part(Tue_Sep_14_21:36:15_1999)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit From: Sheldon Hearn <sheldonh@uunet.co.za> > Pity. Like I said, the patch wasn't tested and it really was a shot in > the dark. I'll take a closer look some time soon, but let me know anyway > how it went on 3.2R . It makes same errors even on 3.2R box. I looked the patch closer and I found a small bug. I attach a new patch (it is against to original tcpslice source). It seems to work! A bug is simple mistake about a cast came from original code. In function sf_find_end:search.c, the code say: u_int num_bytes; .... num_bytes = MAX_BYTES_FOR_DEFINITE_HEADER; if ( fseeko( pcap_file( p ), (off_t) -num_bytes, 2 ) < 0 ) return 0; The 2nd argument of fseeko() becomes 4294966960 instead of -336 because -num_bytes is still u_int type. I tested on a sample that I reported on send-pr. I'll split a whole file from now, however, I'm not sure I can report it works well since it will take a long time and output files are too large to check them. Thanks a lot! -- Takamichi ----Next_Part(Tue_Sep_14_21:36:15_1999)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit *** search.c.orig Wed Aug 23 14:18:57 1995 --- search.c Tue Sep 14 21:13:37 1999 *************** *** 281,287 **** * end of the file. */ num_bytes = MAX_BYTES_FOR_DEFINITE_HEADER; ! if ( fseek( pcap_file( p ), (long) -num_bytes, 2 ) < 0 ) return 0; buf = (u_char *)malloc((u_int) num_bytes); --- 281,287 ---- * end of the file. */ num_bytes = MAX_BYTES_FOR_DEFINITE_HEADER; ! if ( fseeko( pcap_file( p ), -(off_t)num_bytes, 2 ) < 0 ) return 0; buf = (u_char *)malloc((u_int) num_bytes); *************** *** 346,353 **** status = 1; /* Seek so that the next read will start at last valid packet. */ ! if ( fseek( pcap_file( p ), (long) -(bufend - hdrpos), 2 ) < 0 ) ! error( "final fseek() failed in sf_find_end()" ); done: free( (char *) buf ); --- 346,353 ---- status = 1; /* Seek so that the next read will start at last valid packet. */ ! if ( fseeko( pcap_file( p ), -(off_t) (bufend - hdrpos), 2 ) < 0 ) ! error( "final fseeko() failed in sf_find_end()" ); done: free( (char *) buf ); *************** *** 384,403 **** * negative value if the desired_time is outside the given range. */ ! static long ! interpolated_position( struct timeval *min_time, long min_pos, ! struct timeval *max_time, long max_pos, struct timeval *desired_time ) { double full_span = timeval_diff( max_time, min_time ); double desired_span = timeval_diff( desired_time, min_time ); ! long full_span_pos = max_pos - min_pos; double fractional_offset = desired_span / full_span; if ( fractional_offset < 0.0 || fractional_offset > 1.0 ) return -1; ! return min_pos + (long) (fractional_offset * (double) full_span_pos); } --- 384,403 ---- * negative value if the desired_time is outside the given range. */ ! static off_t ! interpolated_position( struct timeval *min_time, off_t min_pos, ! struct timeval *max_time, off_t max_pos, struct timeval *desired_time ) { double full_span = timeval_diff( max_time, min_time ); double desired_span = timeval_diff( desired_time, min_time ); ! off_t full_span_pos = max_pos - min_pos; double fractional_offset = desired_span / full_span; if ( fractional_offset < 0.0 || fractional_offset > 1.0 ) return -1; ! return min_pos + (fractional_offset * full_span_pos); } *************** *** 412,425 **** { struct pcap_pkthdr hdr; const u_char *buf; ! long pos; int status; for ( ; ; ) { struct timeval *timestamp; ! pos = ftell( pcap_file( p ) ); buf = pcap_next( p, &hdr ); if ( buf == 0 ) --- 412,425 ---- { struct pcap_pkthdr hdr; const u_char *buf; ! off_t pos; int status; for ( ; ; ) { struct timeval *timestamp; ! pos = ftello( pcap_file( p ) ); buf = pcap_next( p, &hdr ); if ( buf == 0 ) *************** *** 443,450 **** } } ! if ( fseek( pcap_file( p ), pos, 0 ) < 0 ) ! error( "fseek() failed in read_up_to()" ); return (status); } --- 443,450 ---- } } ! if ( fseeko( pcap_file( p ), pos, 0 ) < 0 ) ! error( "fseeko() failed in read_up_to()" ); return (status); } *************** *** 466,480 **** int sf_find_packet( pcap_t *p, ! struct timeval *min_time, long min_pos, ! struct timeval *max_time, long max_pos, struct timeval *desired_time ) { int status = 1; struct timeval min_time_copy, max_time_copy; u_int num_bytes = MAX_BYTES_FOR_DEFINITE_HEADER; int num_bytes_read; ! long desired_pos, present_pos; u_char *buf, *hdrpos; struct pcap_pkthdr hdr; --- 466,480 ---- int sf_find_packet( pcap_t *p, ! struct timeval *min_time, off_t min_pos, ! struct timeval *max_time, off_t max_pos, struct timeval *desired_time ) { int status = 1; struct timeval min_time_copy, max_time_copy; u_int num_bytes = MAX_BYTES_FOR_DEFINITE_HEADER; int num_bytes_read; ! off_t desired_pos, present_pos; u_char *buf, *hdrpos; struct pcap_pkthdr hdr; *************** *** 501,507 **** break; } ! present_pos = ftell( pcap_file( p ) ); if ( present_pos <= desired_pos && desired_pos - present_pos < STRAIGHT_SCAN_THRESHOLD ) --- 501,507 ---- break; } ! present_pos = ftello( pcap_file( p ) ); if ( present_pos <= desired_pos && desired_pos - present_pos < STRAIGHT_SCAN_THRESHOLD ) *************** *** 517,523 **** if ( desired_pos < min_pos ) desired_pos = min_pos; ! if ( fseek( pcap_file( p ), desired_pos, 0 ) < 0 ) error( "fseek() failed in sf_find_packet()" ); num_bytes_read = --- 517,523 ---- if ( desired_pos < min_pos ) desired_pos = min_pos; ! if ( fseeko( pcap_file( p ), desired_pos, 0 ) < 0 ) error( "fseek() failed in sf_find_packet()" ); num_bytes_read = *************** *** 540,546 **** desired_pos += (hdrpos - buf); /* Seek to the beginning of the header. */ ! if ( fseek( pcap_file( p ), desired_pos, 0 ) < 0 ) error( "fseek() failed in sf_find_packet()" ); if ( sf_timestamp_less_than( &hdr.ts, desired_time ) ) --- 540,546 ---- desired_pos += (hdrpos - buf); /* Seek to the beginning of the header. */ ! if ( fseeko( pcap_file( p ), desired_pos, 0 ) < 0 ) error( "fseek() failed in sf_find_packet()" ); if ( sf_timestamp_less_than( &hdr.ts, desired_time ) ) *** tcpslice.c.orig Thu Jan 21 03:33:13 1999 --- tcpslice.c Tue Sep 14 06:16:10 1999 *************** *** 451,459 **** extract_slice(char filename[], char write_file_name[], struct timeval *start_time, struct timeval *stop_time) { - long start_pos, stop_pos; struct timeval file_start_time, file_stop_time; struct pcap_pkthdr hdr; pcap_t *p; char errbuf[PCAP_ERRBUF_SIZE]; --- 451,459 ---- extract_slice(char filename[], char write_file_name[], struct timeval *start_time, struct timeval *stop_time) { struct timeval file_start_time, file_stop_time; struct pcap_pkthdr hdr; + off_t start_pos, stop_pos; pcap_t *p; char errbuf[PCAP_ERRBUF_SIZE]; *************** *** 462,468 **** error( "bad tcpdump file %s: %s", filename, errbuf ); snaplen = pcap_snapshot( p ); ! start_pos = ftell( pcap_file( p ) ); if ( ! dumper ) { --- 462,468 ---- error( "bad tcpdump file %s: %s", filename, errbuf ); snaplen = pcap_snapshot( p ); ! start_pos = ftello( pcap_file( p ) ); if ( ! dumper ) { *************** *** 483,489 **** error( "problems finding end packet of file %s", filename ); ! stop_pos = ftell( pcap_file( p ) ); /* sf_find_packet() requires that the time it's passed as its last --- 483,489 ---- error( "problems finding end packet of file %s", filename ); ! stop_pos = ftello( pcap_file( p ) ); /* sf_find_packet() requires that the time it's passed as its last ----Next_Part(Tue_Sep_14_21:36:15_1999)---- Hi there!
I know it's been a while since you reported your problem with tcpslice,
but I was wondering whether you could test out the following patch from
the author of tcpslice, without any of the modifications you made
yourself, and tell me whether the author has come up with a fix that
works for you.
I know that you raised concerns regarding the storage size used instead
of ssize_t, which is why I'd like your feedback on Ver's patch.
Thanks,
Sheldon.
--- /tmp/,RCSt1a08617 Mon Dec 6 01:32:06 1999
+++ /tmp/,RCSt2a08617 Mon Dec 6 01:32:06 1999
@@ -23,7 +23,7 @@
"@(#) Copyright (c) 1991, 1992, 1993, 1995, 1996, 1997\n\
The Regents of the University of California. All rights reserved.\n";
static const char rcsid[] =
- "@(#)$Header: tcpslice.c,v 1.21 97/01/24 13:36:07 leres Exp $ (LBL)";
+ "@(#)$Header: tcpslice.c,v 1.22 99/06/22 23:07:45 vern Exp $ (LBL)";
#endif
/*
@@ -470,7 +470,7 @@
error( "bad tcpdump file %s: %s", filename, errbuf );
snaplen = pcap_snapshot( p );
- start_pos = ftell( pcap_file( p ) );
+ start_pos = ftello( pcap_file( p ) );
if ( ! dumper )
{
@@ -491,7 +491,7 @@
error( "problems finding end packet of file %s",
filename );
- stop_pos = ftell( pcap_file( p ) );
+ stop_pos = ftello( pcap_file( p ) );
/* sf_find_packet() requires that the time it's passed as its last
Hi, From: Sheldon Hearn <sheldonh@uunet.co.za> > I know it's been a while since you reported your problem with tcpslice, > but I was wondering whether you could test out the following patch from > the author of tcpslice, without any of the modifications you made > yourself, and tell me whether the author has come up with a fix that > works for you. I'm sorry that I cannot do actual test since I already removed a large (4.8GB!) dump file because of disk shortage. However, I think the author of tcpslice makes the patch against the newest version of tcpslice released from LBL. # ftp://ftp.ee.lbl.gov/tcpslice-1.1a3.tar.Z The LBL version of tcpslice already uses portable "off_t" for file offsets. In past, I compiled LBL's tcpslice on the Large File Compilation Environment of SUN Solaris 2.6 (this environment is very similar to the nature of FreeBSD). I only replaced fseek/ftell with fseeko/ftello respectively. It works great for a dump file larger than 2GB. So, I think the author's patch is good enough for tcpslice.c of LBL's tcpslice. However, we should replace fseek/ftell in search.c also, I think. A small but important difference between FreeBSD's tcpslice and LBL's one is the definition of num_bytes in search.c:sf_find_end(). FreeBSD's code defines it as u_int num_bytes; on the other hand, LBL's code defines it as int num_bytes; This difference makes a sign expansion problem that I reported last time. LBL's code never makes the problem since it is signed and no need for a sign expansion. Thanks, TATEOKA, Takamichi Hi, From: Sheldon Hearn <sheldonh@uunet.co.za> > I know it's been a while since you reported your problem with tcpslice, > but I was wondering whether you could test out the following patch from > the author of tcpslice, without any of the modifications you made > yourself, and tell me whether the author has come up with a fix that > works for you. I'm sorry that I cannot do actual test since I already removed a large (4.8GB!) dump file because of disk shortage. However, I think the author of tcpslice makes the patch against the newest version of tcpslice released from LBL. # ftp://ftp.ee.lbl.gov/tcpslice-1.1a3.tar.Z The LBL version of tcpslice already uses portable "off_t" for file offsets. In past, I compiled LBL's tcpslice on the Large File Compilation Environment of SUN Solaris 2.6 (this environment is very similar to the nature of FreeBSD). I only replaced fseek/ftell with fseeko/ftello respectively. It works great for a dump file larger than 2GB. So, I think the author's patch is good enough for tcpslice.c of LBL's tcpslice. However, we should replace fseek/ftell in search.c also, I think. A small but important difference between FreeBSD's tcpslice and LBL's one is the definition of num_bytes in search.c:sf_find_end(). FreeBSD's code defines it as u_int num_bytes; on the other hand, LBL's code defines it as int num_bytes; This difference makes a sign expansion problem that I reported last time. LBL's code never makes the problem since it is signed and no need for a sign expansion. Thanks, TATEOKA, Takamichi Hi all-- Came across PR bin/13691 while I was researching a Y2K bug in tcpslice. Right now I see three problems with the version of tcpslice in the FreeBSD base distribution: 1. tcpslice doesn't work correctly on >2GB files. 2. Timezone conversions aren't right. (See PR bin/10633, filed by me.) 3. Y2K bug (tcpslice doesn't handle dates correctly on traces that start in year 2000 or later). To fix all three of these, I propose the following: A. Contribify tcpslice-1.1a3, as I described in bin/10633. This fixes #2 and, according to Tateoka-san's follow-up to bin/13691, will fix part of #1 by giving us a tcpslice that uses off_t instead of long for file offsets. B. Apply the patch attached here, to replace all of the fseek/ftell calls with fsetpos/fgetpos calls. Another way to do this is to replace fseek/ftell with fseeko/ftello. The patch from Vern (cited by Sheldon) replaces two of the ftell calls with ftello, but doesn't touch any of the fseek calls. (My patch uses fsetpos/fgetpos because that's what I was familiar with at the time. Using fseeko/ftello may in fact be better because it involves fewer code changes.) The attached patch also has a diff for gwtm2secs.c to fix the Y2K problem in #3. I realize this doesn't stand much of a chance of happening in the middle of the 4.0 code freeze, but I wanted to write this up while it's still fresh in my mind. (Vern: I sent you a CC on this in case you're still doing work on tcpslice. You can get some of the background info leading up to this discussion at: http://www.freebsd.org/cgi/query-pr-summary.cgi ) (David: I don't mean to hassle you over bin/10633, but I thought you'd probably be interested.) Thanks all! Bruce. diff -c -r tcpslice-1.1a3-dist/gwtm2secs.c tcpslice-1.1a3/gwtm2secs.c *** tcpslice-1.1a3-dist/gwtm2secs.c Sat Dec 21 19:56:52 1996 --- tcpslice-1.1a3/gwtm2secs.c Fri Feb 11 08:17:02 2000 *************** *** 62,67 **** --- 62,74 ---- else year += 2000; + /* Make sure our year is still >= 1970. We fix 3-digit years + * this way, because localtime(3) can return tm_year >= 100, + * starting in year 2000. + */ + if ( year < 1970 ) + year += 1900; + days = 0; for ( i = 1970; i < year; ++i ) { diff -c -r tcpslice-1.1a3-dist/search.c tcpslice-1.1a3/search.c *** tcpslice-1.1a3-dist/search.c Sat Dec 21 19:56:37 1996 --- tcpslice-1.1a3/search.c Wed Apr 21 08:00:42 1999 *************** *** 295,301 **** * end of the file. */ num_bytes = MAX_BYTES_FOR_DEFINITE_HEADER; ! if ( fseek( pcap_file( p ), (off_t) -num_bytes, 2 ) < 0 ) return 0; buf = (u_char *)malloc((u_int) num_bytes); --- 295,301 ---- * end of the file. */ num_bytes = MAX_BYTES_FOR_DEFINITE_HEADER; ! if ( fseek( pcap_file( p ), (off_t) -num_bytes, SEEK_END ) < 0 ) return 0; buf = (u_char *)malloc((u_int) num_bytes); *************** *** 360,366 **** status = 1; /* Seek so that the next read will start at last valid packet. */ ! if ( fseek( pcap_file( p ), (off_t) -(bufend - hdrpos), 2 ) < 0 ) error( "final fseek() failed in sf_find_end()" ); done: --- 360,366 ---- status = 1; /* Seek so that the next read will start at last valid packet. */ ! if ( fseek( pcap_file( p ), (off_t) -(bufend - hdrpos), SEEK_END ) < 0 ) error( "final fseek() failed in sf_find_end()" ); done: *************** *** 433,439 **** { struct timeval *timestamp; ! pos = ftell( pcap_file( p ) ); buf = pcap_next( p, &hdr ); if ( buf == 0 ) --- 433,439 ---- { struct timeval *timestamp; ! fgetpos( pcap_file( p ), &pos ); buf = pcap_next( p, &hdr ); if ( buf == 0 ) *************** *** 457,464 **** } } ! if ( fseek( pcap_file( p ), pos, 0 ) < 0 ) ! error( "fseek() failed in read_up_to()" ); return (status); } --- 457,464 ---- } } ! if ( fsetpos( pcap_file( p ), &pos ) < 0 ) ! error( "fsetpos() failed in read_up_to()" ); return (status); } *************** *** 515,521 **** break; } ! present_pos = ftell( pcap_file( p ) ); if ( present_pos <= desired_pos && desired_pos - present_pos < STRAIGHT_SCAN_THRESHOLD ) --- 515,521 ---- break; } ! fgetpos( pcap_file( p ), &present_pos ); if ( present_pos <= desired_pos && desired_pos - present_pos < STRAIGHT_SCAN_THRESHOLD ) *************** *** 531,538 **** if ( desired_pos < min_pos ) desired_pos = min_pos; ! if ( fseek( pcap_file( p ), desired_pos, 0 ) < 0 ) ! error( "fseek() failed in sf_find_packet()" ); num_bytes_read = fread( (char *) buf, 1, num_bytes, pcap_file( p ) ); --- 531,538 ---- if ( desired_pos < min_pos ) desired_pos = min_pos; ! if ( fsetpos( pcap_file( p ), &desired_pos ) < 0 ) ! error( "fsetpos() failed in sf_find_packet()" ); num_bytes_read = fread( (char *) buf, 1, num_bytes, pcap_file( p ) ); *************** *** 554,561 **** desired_pos += (hdrpos - buf); /* Seek to the beginning of the header. */ ! if ( fseek( pcap_file( p ), desired_pos, 0 ) < 0 ) ! error( "fseek() failed in sf_find_packet()" ); if ( sf_timestamp_less_than( &hdr.ts, desired_time ) ) { /* too early in the file */ --- 554,561 ---- desired_pos += (hdrpos - buf); /* Seek to the beginning of the header. */ ! if ( fsetpos( pcap_file( p ), &desired_pos ) < 0 ) ! error( "fsetpos() failed in sf_find_packet()" ); if ( sf_timestamp_less_than( &hdr.ts, desired_time ) ) { /* too early in the file */ diff -c -r tcpslice-1.1a3-dist/tcpslice.c tcpslice-1.1a3/tcpslice.c *** tcpslice-1.1a3-dist/tcpslice.c Fri Jan 24 13:36:09 1997 --- tcpslice-1.1a3/tcpslice.c Wed Apr 21 08:01:01 1999 *************** *** 470,476 **** error( "bad tcpdump file %s: %s", filename, errbuf ); snaplen = pcap_snapshot( p ); ! start_pos = ftell( pcap_file( p ) ); if ( ! dumper ) { --- 470,476 ---- error( "bad tcpdump file %s: %s", filename, errbuf ); snaplen = pcap_snapshot( p ); ! fgetpos( pcap_file( p ), &start_pos ); if ( ! dumper ) { *************** *** 491,497 **** error( "problems finding end packet of file %s", filename ); ! stop_pos = ftell( pcap_file( p ) ); /* sf_find_packet() requires that the time it's passed as its last --- 491,497 ---- error( "problems finding end packet of file %s", filename ); ! fgetpos( pcap_file( p ), &stop_pos ); /* sf_find_packet() requires that the time it's passed as its last On Fri, 11 Feb 2000 08:50:46 PST, Bruce A. Mah wrote:
> Came across PR bin/13691 while I was researching a Y2K bug in
> tcpslice. Right now I see three problems with the version of tcpslice
> in the FreeBSD base distribution:
Thanks. This is already on someone's list -- either asmodai's or ru's,
I can't remember. I'll keep an eye on the PR's after 4.0-RELEASE.
Ciao,
Sheldon.
Hi, From: bmah@CA.Sandia.GOV (Bruce A. Mah) > B. Apply the patch attached here, to replace all of the fseek/ftell > calls with fsetpos/fgetpos calls. Another way to do this is to > replace fseek/ftell with fseeko/ftello. The patch from Vern (cited by > Sheldon) replaces two of the ftell calls with ftello, but doesn't > touch any of the fseek calls. > > (My patch uses fsetpos/fgetpos because that's what I was familiar with > at the time. Using fseeko/ftello may in fact be better because it > involves fewer code changes.) I think your patch is good enough to work on FreeBSD box and fixes known problems of tcpslice. I think that It is better to use fseeko/ftello instead of fsetpos/fgetpos. Because it makes type mismatch for offset value. In fact, both of them are off_t in most case, however, it's not guaranteed. (see definition of fpos_t in /usr/include/stdio.h.) -- Takamichi TATEOKA (tate@cs.uec.ac.jp) Responsible Changed From-To: freebsd-bugs->fenner I only learned recently that Mr Fenner takes care of tcpdump. Responsible Changed From-To: fenner->bms i'll take this, my starter for ten bms 2006-09-23 21:12:23 UTC
FreeBSD src repository
Modified files:
usr.sbin/tcpdump/tcpslice gwtm2secs.c search.c tcpslice.c
Log:
Fix our ancient tcpslice for >2GB limits.
PR: bin/13691
MFC after: 1 week
Submitted by: Bruce A. Mah
Revision Changes Path
1.5 +7 -0 src/usr.sbin/tcpdump/tcpslice/gwtm2secs.c
1.5 +13 -13 src/usr.sbin/tcpdump/tcpslice/search.c
1.13 +3 -3 src/usr.sbin/tcpdump/tcpslice/tcpslice.c
_______________________________________________
cvs-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/cvs-all
To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
State Changed From-To: open->closed Committed, thanks! delphij 2006-10-18 03:21:37 UTC
FreeBSD src repository
Modified files: (Branch: RELENG_6)
usr.sbin/tcpdump/tcpslice gwtm2secs.c search.c tcpslice.1
tcpslice.c
Log:
MFC: Fix our ancient tcpslice for >2GB limits [1].
Also, mention that tcpslice in the base system is being
deprecated, and encourage users to install it from the
ports collection.
PR: bin/13691
Submitted by: Bruce A. Mah
Approved by: re (bmah)
Revision Changes Path
1.4.34.1 +7 -0 src/usr.sbin/tcpdump/tcpslice/gwtm2secs.c
1.4.34.1 +13 -13 src/usr.sbin/tcpdump/tcpslice/search.c
1.15.8.1 +10 -1 src/usr.sbin/tcpdump/tcpslice/tcpslice.1
1.12.14.1 +3 -3 src/usr.sbin/tcpdump/tcpslice/tcpslice.c
_______________________________________________
cvs-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/cvs-all
To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
|
