Created attachment 162210 [details] patch for gzip While running FreeBSD using space-constrained embedded platform like MIPS32 router, one may not have enough flash space for full-blown file system to keep variable configuration files persistent between reboots. For example, there may be only 64K or so flash space seen as /dev/map/cfg. This is not enough for newfs(1) but enough to write gzip-compressed tar archive that can store hundreds kilobytes of text configs like SSH host keys etc. We need a way to check integrity of those raw 64Kbytes of data and "gzip -qt" would do the job. However, "gzip -qt < /dev/map/cfg" does not work due to hardware limitations and we need to use "dd bs=64k if=/dev/map/cfg | gzip -qt". That feeds gzip with our compressed data and some trailing garbage. Our gzip ignores trailing garbage but returns exit status 1 (fatal error) even if CRC matches. GNU gzip returns exit status 2 (just warning) in such case. Attached patch makes our gzip return 2 in case of ignored trailing garbage.
Adding some of the folks who have touched gzip to quick review. I'm pretty sure this is a desirable thing. Our gzip seems to come from NetBSD. We should just double check that they don't already do this.
Thank you for the submission. The change seems reasonable to me. Are there other warnings we should capture as well? If you're keen I'd like to see mention of the exit status in the man page as well. AFAICT NetBSD doesn't do this today; it would also be worth suggesting this change to them (although that could happen before or after we incorporate the change).
> Are there other warnings we should capture as well? None I am aware of. > I'd like to see mention of the exit status in the man page as well. Feel free to add :-) English is not my nativa language and I'm afraid of bad wording.
(In reply to eugen from comment #3) > > Are there other warnings we should capture as well? > > None I am aware of. > > > I'd like to see mention of the exit status in the man page as well. > > Feel free to add :-) English is not my nativa language and I'm afraid of bad > wording. How does the following look? gzip complains "trailing garbage ignored" Some tar.gz files are padded with zeroes to ensure a size which is a multiple of a certain block size. This occurs in particular when the compressed tar file is on a device, such as a magnetic tape. When such files are extracted with a command such as gunzip < file.tar.gz | tar xvf - gtar xvzf /dev/rmt/0 gunzip correctly decompresses the tar.gz file, then attempts to decompress the rest of the input which consists of zeroes. Since those zeroes are not in gzip format, gzip can complain, but ignores them. The tar extract command still works correctly, since gzip has sent through the pipe all the data that tar needs. You can suppress this harmless warning using the -q option, as in: gunzip -q < file.tar.gz | tar xvf - GZIP=-q gtar xvzf /dev/rmt/0 # for bash, ksh, sh ... (setenv GZIP -q; gtar xvzf /dev/rmt/0) # for csh, tcsh, ... I simply used information already available on gzip's web site, and modified it slightly. Hope it helps! --Chris
(In reply to Chris Hutchinson from comment #4) Looks fine for me but lacks documentation for exit status (0 for success, 1 for fatal errors, 2 for warnings)
(In reply to eugen from comment #5) > (In reply to Chris Hutchinson from comment #4) > > Looks fine for me but lacks documentation for exit status (0 for success, 1 > for fatal errors, 2 for warnings) You mean something like the following (appended at the end) gzip complains with trailing garbage ignored Some tar.gz files are padded with zeroes to ensure a size which is a multiple of a certain block size. This occurs in particular when the compressed tar file is on a device, such as a magnetic tape. When such files are extracted with a command such as gunzip < file.tar.gz | tar xvf - gtar xvzf /dev/rmt/0 gunzip correctly decompresses the tar.gz file, then attempts to decompress the rest of the input which consists of zeroes. Since those zeroes are not in gzip format, gzip can complain, but ignores them. The tar extract command still works correctly, since gzip has sent through the pipe all the data that tar needs. You can suppress this harmless warning using the -q option, as in: gunzip -q < file.tar.gz | tar xvf - GZIP=-q gtar xvzf /dev/rmt/0 # for bash, ksh, sh ... (setenv GZIP -q; gtar xvzf /dev/rmt/0) # for csh, tcsh, ... EXIT STATUS gzip returns the following numeric status: 0 on SUCCESS 1 on FATAL errors 2 on [non-fatal] errors [warn] --Chris
(In reply to Chris Hutchinson from comment #6) Something like that, yes.
A commit references this bug: Author: delphij Date: Mon Oct 26 22:29:59 UTC 2015 New revision: 290024 URL: https://svnweb.freebsd.org/changeset/base/290024 Log: In gunzip(1), treat trailing garbage as a warning and not an error. This allows scripts to distinguish it between real fatal errors, for instance a CRC mismatch. Update manual page for the behavior change. PR: bin/203873 Submitted by: Eugene Grosbein <eugen grosbein net> MFC after: 2 weeks Changes: head/usr.bin/gzip/gzip.1 head/usr.bin/gzip/gzip.c
A commit references this bug: Author: delphij Date: Mon Nov 9 01:53:55 UTC 2015 New revision: 290569 URL: https://svnweb.freebsd.org/changeset/base/290569 Log: MFC r290024,290073: In gunzip(1), treat trailing garbage as a warning and not an error. This allows scripts to distinguish it between real fatal errors, for instance a CRC mismatch. Update manual page for the behavior change. PR: bin/203873 Submitted by: Eugene Grosbein <eugen grosbein net> Changes: _U stable/10/ stable/10/usr.bin/gzip/gzip.1 stable/10/usr.bin/gzip/gzip.c
Resolved long time ago.