Bug 36998
| Summary: | rsync: -z option core dumps on large files | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Donal Diamond <ddiamond> | ||||||
| Component: | Individual Port(s) | Assignee: | Anders Nordby <anders> | ||||||
| Status: | Closed FIXED | ||||||||
| Severity: | Affects Only Me | ||||||||
| Priority: | Normal | ||||||||
| Version: | Latest | ||||||||
| Hardware: | Any | ||||||||
| OS: | Any | ||||||||
| Attachments: |
|
||||||||
Apologies, I have a typo for the command. Command should be: rsync -azv rsync://sunsite.dk/ftp//mirrors/mysql/Downloads/Contrib/MyAccess.mda . Also, forgot to state rsync version: $rsync --version rsync version 2.5.5 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 32-bit system inums, 64-bit internal inums Donal Thanks for your report.
I am going to check this next week, since I am on a official journey
from tomorrow morning until next week.
Regards,
Olli
[MAINTAINER UPDATE]
rsync -z works if compiled with CFLAGS including at least -O2
optimization level.
The following patch checks whether -O[2-6] is part of CFLAGS.
Otherwise -O2 will be appended.
[ From the gcc manual:
If you use multiple `-O' options, with or without level numbers,
the last such option is the one that is effective. ]
Bumped PORTREVISION
Regards,
Olli
--------------------------------------------------------------------------
diff -ruN rsync.old/Makefile rsync/Makefile
--- rsync.old/Makefile Mon Apr 15 23:13:44 2002
+++ rsync/Makefile Mon Apr 15 23:25:50 2002
@@ -8,7 +8,7 @@
PORTNAME= rsync
PORTVERSION= 2.5.5
-PORTREVISION= 0
+PORTREVISION= 1
CATEGORIES= net ipv6
MASTER_SITES= ftp://samba.anu.edu.au/pub/rsync/ \
ftp://sunsite.auc.dk/pub/unix/rsync/ \
@@ -18,6 +18,11 @@
MAINTAINER= obraun@informatik.unibw-muenchen.de
GNU_CONFIGURE= yes
+
+OPTIMIZATION!= ${ECHO} "${CFLAGS}" | grep '\-O[2-6]' || true
+.if empty(OPTIMIZATION)
+CFLAGS+= -O2
+.endif
.include <bsd.port.pre.mk>
State Changed From-To: open->closed Committed maintainers fix. Thanks for reporting! Hi, I'm the upstream rsync maintainer. Hopefully I have the right reply address. Although -O2 is the default for building rsync with gcc, I was not aware that it was required to get a properly functioning version. You'd have to suspect it was a compiler bug that the behaviour differs, but perhaps not. The crash is inside zlib, which is a 3rd-party library we ship with some modifications. Linking against the system's zlib will produce a similar error -- I don't suppose you were doing that in your port, were you? I don't have easy access to a FreeBSD box, but I'll try to reproduce this elsewhere. I'd really appreciate if somebody could work out what exactly is going wrong. Thanks, -- Martin On Saturday, 20 April 2002 at 19:42:00 -0700, Martin Pool wrote: > Hi, > > I'm the upstream rsync maintainer. G'day, Martin, pleased to meet you :-) > Hopefully I have the right reply address. It's a good start, since it gets entered in the gnats data base. You should know that most FreeBSD ports, including rsync, have a designated maintainer. You can find this in the MAINTAINER definition in the ports Makefile: MAINTAINER= obraun@informatik.unibw-muenchen.de From the commit logs I discover that this is Oliver Braun <obraun@informatik.unibw-muenchen.de>, and that he's only recently taken over the maintainership of this port. He's not a FreeBSD committer. I'm copying him to make sure he's in the loop. > You'd have to suspect it was a compiler bug that the behaviour > differs, but perhaps not. I'd find it difficult to find a different explanation for something which was dependent on optimization levels. > The crash is inside zlib, which is a 3rd-party library we ship with > some modifications. Linking against the system's zlib will produce > a similar error -- I don't suppose you were doing that in your port, > were you? It seems that we do use the rsync zlib, so that doesn't seem to be the issue. I think that's wrong, though. If there are bugs in zlib, we should fix them. We shouldn't use variants of libraries to work around the bugs. > I don't have easy access to a FreeBSD box, Hmm. I see: $ finger mbp Login: mbp Name: Martin Pool Directory: /home/mbp Shell: /usr/local/bin/bash Never logged in. No Mail. No Plan. > but I'll try to reproduce this elsewhere. I'd really appreciate if > somebody could work out what exactly is going wrong. You're welcome to use wantadilla. Send me your ssh keys; I don't have them. Greg -- See complete headers for address and phone numbers State Changed From-To: closed->open Re-open PR. This solution should be done another way. Responsible Changed From-To: freebsd-ports->anders I'll handle this. Hello,
On Sat, Apr 20, 2002 at 08:45:52PM -0701, Jos Backus wrote:
> Fwiw, I sent mail to the rsync list and cvs-all about Bruce Evans' workaround
> which we could commit to the rsync sources.
I can verify the rsync 2.5.5 bug on:
- 4.5-STABLE from yesterday
- -current from yesterday
Test case is simple:
# rsync -azv \
rsync://sunsite.dk/ftp//mirrors/mysql/Downloads/Contrib/MyAccess.mda .
If i add -lz to Makefile's LIBS (use the system libz) and skip building
ZLIBOBJ objects, I can not reproduce the bug. Why should rsync roll its
own libz when FreeBSD already does this? I think I would prefer it not
to. What does port maintainer think? What does the rest of you think?
Cheers,
--
Anders.
> If i add -lz to Makefile's LIBS (use the system libz) and skip building
> ZLIBOBJ objects, I can not reproduce the bug. Why should rsync roll its
> own libz when FreeBSD already does this? I think I would prefer it not
> to. What does port maintainer think? What does the rest of you think?
From zlib/README.rsync in the source distribution:
------
READ THIS BEFORE TRYING TO DYNAMICALLY LINK RSYNC AND ZLIB!
zlib has been adapted slightly for use in rsync. Please don't bother
the zlib authors with problems related to the use of zlib in rsync as
any bugs are likely to be our fault and not theirs.
Specific changes that have been made to zlib for rsync include:
- add Z_INSERT_ONLY to allow for efficient history updating without
actually emitting any data. This is used to compress the matched
blocks that don't cross the wire, which gives better compression
ratios on the literal data.
- fixed a number of minor compilation issues. (redefinition of MAX and
other such trivial things)
- include rsync.h to ensure that we get a consistent set of includes
for all C code in rsync and to take advantage of autoconf
As a result of the first item, the streams from rsync's version of
zlib are *not compatible* with those produced by the upstream version
of rsync. In other words, if you link rsync against your system's
copy, it will not be able to interoperate with any other version if
the -z option is used. (Sorry. Sometimes standard is better than
better.)
The rsync maintainers hope to fix this problem in the future by either
merging our changes into the upstream version, or backing them out of
rsync in a way that preserves wire compatibility. But in the meantime
this version must be maintained in parallel.
----
--
Martin
On 21 Apr 2002, Anders Nordby <anders@fix.no> wrote: > # rsync -azv \ > rsync://sunsite.dk/ftp//mirrors/mysql/Downloads/Contrib/MyAccess.mda . Works fine for me using rsync's upstream HEAD and 2.5.5 release. So it seems like it must be either something in BSD's patches, or something about your compiler, or possibly something else. Could you please try building from upstream source and let me know? -- Martin Hello, On Sun, Apr 21, 2002 at 08:29:36AM -0700, Martin Pool wrote: >> # rsync -azv \ >> rsync://sunsite.dk/ftp//mirrors/mysql/Downloads/Contrib/MyAccess.mda . > Works fine for me using rsync's upstream HEAD and 2.5.5 release. So > it seems like it must be either something in BSD's patches, or > something about your compiler, or possibly something else. > > Could you please try building from upstream source and let me know? Did a checkout of HEAD, built with CFLAGS: -O -pipe on FreeBSD -STABLE from 15th of april. Still segfaults: # export CVSROOT=:pserver:cvs@pserver.samba.org:/cvsroot # cvs co rsync (..) # cd rsync # CFLAGS="-O -pipe" ./configure --disable-debug --enable-ipv6 (..) # make (..) # ./rsync --version rsync version 2.5.6cvs protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 32-bit system inums, 64-bit internal inums rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. # ./rsync -azv rsync://sunsite.dk/ftp//mirrors/mysql/Downloads/Contrib/MyAccess.mda . (non-interesting text snipped..) receiving file list ... done MyAccess.mda rsync: connection unexpectedly closed (930 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) Compiler is gcc 2.95.3. Cheers, -- Anders. On Sun, Apr 21, 2002 at 08:29:36AM -0700, Martin Pool wrote: >> # rsync -azv \ >> rsync://sunsite.dk/ftp//mirrors/mysql/Downloads/Contrib/MyAccess.mda . > Works fine for me using rsync's upstream HEAD and 2.5.5 release. So > it seems like it must be either something in BSD's patches, or > something about your compiler, or possibly something else. > > Could you please try building from upstream source and let me know? The attached patch works around the bug so that it does not show up in -stable or -current. It's a workaround for GCC bugs in FreeBSD, made by bde@FreeBSD.org some weeks ago. Sorry I didn't see this before now. Rsync folks: For FreeBSD, I think this is the temporary fix for the port until GCC is fixed. I don't know if you want to include it in any way. BTW: http://rsync.samba.org/cvs.html (from the download page on rsync.samba.org) seems to be lost. Luckily, I had what I needed in ~/.cvspass. Oliver: may I commit this and revert your changes on the issue? Greg/Kris: MFC of this file in src/lib/libz/infcodes.c may save a lot of users for some trouble. The bug rsync suffers from in its home-rolled zlib 1.1.4 is also reproducable in -stable, I don't see why this isn't going to apply to other programs that uses zlib. Cheers, -- Anders. Is it OK if I fix this in rsync CVS so the next release won't need this patch
on FreeBSD?
--
Jos Backus _/ _/_/_/ Santa Clara, CA
_/ _/ _/
_/ _/_/_/
_/ _/ _/ _/
jos@catnook.com _/_/ _/_/_/ use Std::Disclaimer;
Hi,
On Sun, Apr 21, 2002 at 12:04:22PM -0700, Jos Backus wrote:
> Is it OK if I fix this in rsync CVS so the next release won't need this patch
> on FreeBSD?
Hm. Better make the patch like this. The ifdef was too narrow (sorry).
Cheers,
--
Anders.
* Anders Nordby <anders@fix.no> [2002-04-21 18:36]: > Oliver: may I commit this and revert your changes on the issue? Of course & thanks. Regards, Olli -- Department of Computing Science Federal Armed Forces University Munich http://ist.unibw-muenchen.de/People/obraun/ On Sun, Apr 21, 2002 at 09:24:32PM +0200, Anders Nordby wrote:
> Hm. Better make the patch like this. The ifdef was too narrow (sorry).
I wonder if the #ifdef is needed. I.e. is this a GCC bug that manifests itself
because of the way GCC is implemented on i386 FreeBSD, or is a problem with
GCC itself that manifests itself on i386 (perhaps because it is a bug in the
i386 code generator)?
Jos
On Sun, Apr 21, 2002 at 06:34:30PM +0200, Anders Nordby wrote:
> Greg/Kris: MFC of this file in src/lib/libz/infcodes.c may save a lot of
> users for some trouble. The bug rsync suffers from in its home-rolled
> zlib 1.1.4 is also reproducable in -stable, I don't see why this isn't
> going to apply to other programs that uses zlib.
I thought -stable still had zlib 1.1.3 - am I mistaken?
Kris
State Changed From-To: open->closed I committed the GCC workaround instead. Problem solved. On Sunday, 21 April 2002 at 15:24:50 +0200, Anders Nordby wrote: > On Sat, Apr 20, 2002 at 08:45:52PM -0701, Jos Backus wrote: > > If i add -lz to Makefile's LIBS (use the system libz) and skip building > ZLIBOBJ objects, I can not reproduce the bug. Why should rsync roll its > own libz when FreeBSD already does this? I think I would prefer it not > to. What does port maintainer think? What does the rest of you think? Well, mbp has explained why rsync has its own zlib, but I still don't like it. It would have been better to call it something else if it's not completely compatible with the released version of zlib. I think the best solution would be to get the rsync mods merged into the mainstream version of zlib. Has anybody tried using an rsync binary compiled under Linux? Greg -- See complete headers for address and phone numbers |
rsync port core dumps tansferring large files if you use -z/--compress option.(Problem does not occur if you install manually from source.) I made a basic rysnc port Makefile (no patches/options etc) and it still coredumps. Here's a gdb from a normal port install that was tweaked to enable debug symbols: Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/libc.so.4...done. Reading symbols from /usr/libexec/ld-elf.so.1...done. #0 0x8061823 in inflate_codes (s=0x807b040, z=0x80777c0, r=0) at zlib/infcodes.c:200 200 while (f < s->window) /* modulo window size-"while" instead */ (gdb) bt #0 0x8061823 in inflate_codes (s=0x807b040, z=0x80777c0, r=0) at zlib/infcodes.c:200 #1 0x806104c in inflate_blocks (s=0x807b040, z=0x80777c0, r=-5) at zlib/infblock.c:340 #2 0x806243b in inflate (z=0x80777c0, f=0) at zlib/inflate.c:221 #3 0x805852f in recv_deflated_token (f=0, data=0xbfbfe8dc) at token.c:412 #4 0x805872e in recv_token (f=0, data=0xbfbfe8dc) at token.c:506 #5 0x804bd38 in receive_data (f_in=0, buf=0x0, fd=1, fname=0x8074a80 "master-bin.095", total_size=203874289) at receiver.c:224 #6 0x804c254 in recv_files (f_in=0, flist=0x807a090, local_name=0x0, f_gen=4) at receiver.c:456 #7 0x804fe23 in do_recv (f_in=0, f_out=1, flist=0x807a090, local_name=0x0) at main.c:398 #8 0x80500da in do_server_recv (f_in=0, f_out=1, argc=2, argv=0xbfbff718) at main.c:491 #9 0x8050188 in start_server (f_in=0, f_out=1, argc=2, argv=0xbfbff718) at main.c:519 #10 0x804e048 in local_child (argc=2, argv=0xbfbff718, f_in=0xbfbff900, f_out=0xbfbff904) at util.c:212 #11 0x804f969 in do_cmd (cmd=0x0, machine=0x0, user=0x0, path=0x807a080 ".", f_in=0xbfbff900, f_out=0xbfbff904) at main.c:234 #12 0x8050752 in start_client (argc=2, argv=0x8078060) at main.c:772 #13 0x80509dc in main (argc=2, argv=0x8078060) at main.c:921 #14 0x804a01d in _start () (gdb) p s $1 = (inflate_blocks_statef *) 0x807b040 (gdb) p *s $2 = {dummy = 6} (gdb) p z $3 = 0x80777c0 (gdb) p *z $4 = {next_in = 0x8091ff6 "!.îYõºùúÕ", avail_in = 9, total_in = 16374, next_out = 0x809a000 <Address 0x809a000 out of bounds>, avail_out = 0, total_out = 98304, msg = 0x0, state = 0x8078100, zalloc = 0x8064f18 <zcalloc>, zfree = 0x8064f30 <zcfree>, opaque = 0x0, data_type = 0, adler = 0, reserved = 0} How-To-Repeat: transfer a large (binary?) file with compression on. local to local transfers also fail. rsync -azv / rsync://sunsite.dk/ftp//mirrors/mysql/Downloads/Contrib/MyAccess.mda . rsync: connection unexpectedly closed (930 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(150)