Bug 178504 - [patch] switch logs compression from bzip2 to xz
Summary: [patch] switch logs compression from bzip2 to xz
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: conf (show other bugs)
Version: unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-05-11 10:00 UTC by Alex Kozlov
Modified: 2018-07-19 16:13 UTC (History)
2 users (show)

See Also:


Attachments
file.diff (1.89 KB, patch)
2013-05-11 10:00 UTC, Alex Kozlov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Kozlov 2013-05-11 10:00:00 UTC
Make newsyslog compress logs with xz instead of bzip2 to save space.

Fix: Patch attached with submission follows:
Comment 1 dfilter service freebsd_committer 2013-05-12 16:24:07 UTC
Author: eadler
Date: Sun May 12 15:23:59 2013
New Revision: 250565
URL: http://svnweb.freebsd.org/changeset/base/250565

Log:
  Make newsyslog compress logs with xz instead of bzip2 to save space.
  
  PR:		conf/178504
  Submitted by:	ak
  Reviewed by:	smh

Modified:
  head/etc/newsyslog.conf

Modified: head/etc/newsyslog.conf
==============================================================================
--- head/etc/newsyslog.conf	Sun May 12 13:42:49 2013	(r250564)
+++ head/etc/newsyslog.conf	Sun May 12 15:23:59 2013	(r250565)
@@ -17,23 +17,23 @@
 # future, these defaults may change to more conservative ones.
 #
 # logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
-/var/log/all.log			600  7	   *	@T00  J
-/var/log/amd.log			644  7	   100	*     J
-/var/log/auth.log			600  7     100  @0101T JC
-/var/log/console.log			600  5	   100	*     J
-/var/log/cron				600  3	   100	*     JC
-/var/log/daily.log			640  7	   *	@T00  JN
-/var/log/debug.log			600  7     100  *     JC
-/var/log/init.log			644  3	   100	*     J
-/var/log/kerberos.log			600  7	   100	*     J
-/var/log/lpd-errs			644  7	   100	*     JC
-/var/log/maillog			640  7	   *	@T00  JC
-/var/log/messages			644  5	   100	@0101T JC
-/var/log/monthly.log			640  12	   *	$M1D0 JN
-/var/log/pflog				600  3	   100	*     JB    /var/run/pflogd.pid
-/var/log/ppp.log	root:network	640  3	   100	*     JC
-/var/log/security			600  10	   100	*     JC
+/var/log/all.log			600  7	   *	@T00  X
+/var/log/amd.log			644  7	   100	*     X
+/var/log/auth.log			600  7     100  @0101T XC
+/var/log/console.log			600  5	   100	*     X
+/var/log/cron				600  3	   100	*     XC
+/var/log/daily.log			640  7	   *	@T00  XN
+/var/log/debug.log			600  7     100  *     XC
+/var/log/init.log			644  3	   100	*     X
+/var/log/kerberos.log			600  7	   100	*     X
+/var/log/lpd-errs			644  7	   100	*     XC
+/var/log/maillog			640  7	   *	@T00  XC
+/var/log/messages			644  5	   100	@0101T XC
+/var/log/monthly.log			640  12	   *	$M1D0 XN
+/var/log/pflog				600  3	   100	*     XB    /var/run/pflogd.pid
+/var/log/ppp.log	root:network	640  3	   100	*     XC
+/var/log/security			600  10	   100	*     XC
 /var/log/sendmail.st			640  10	   *	168   BN
 /var/log/utx.log			644  3	   *	@01T05 B
-/var/log/weekly.log			640  5	   *	$W6D0 JN
-/var/log/xferlog			600  7	   100	*     JC
+/var/log/weekly.log			640  5	   *	$W6D0 XN
+/var/log/xferlog			600  7	   100	*     XC
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Comment 2 Eitan Adler freebsd_committer freebsd_triage 2013-05-12 16:24:14 UTC
State Changed
From-To: open->closed

Committed. Thanks!
Comment 3 dfilter service freebsd_committer 2013-05-12 22:24:25 UTC
Author: eadler
Date: Sun May 12 21:24:18 2013
New Revision: 250579
URL: http://svnweb.freebsd.org/changeset/base/250579

Log:
  Revert r250565 which causes issues for older CPUs
  
  PR:		conf/178504
  Requested by:	many

Modified:
  head/etc/newsyslog.conf

Modified: head/etc/newsyslog.conf
==============================================================================
--- head/etc/newsyslog.conf	Sun May 12 20:44:28 2013	(r250578)
+++ head/etc/newsyslog.conf	Sun May 12 21:24:18 2013	(r250579)
@@ -17,23 +17,23 @@
 # future, these defaults may change to more conservative ones.
 #
 # logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
-/var/log/all.log			600  7	   *	@T00  X
-/var/log/amd.log			644  7	   100	*     X
-/var/log/auth.log			600  7     100  @0101T XC
-/var/log/console.log			600  5	   100	*     X
-/var/log/cron				600  3	   100	*     XC
-/var/log/daily.log			640  7	   *	@T00  XN
-/var/log/debug.log			600  7     100  *     XC
-/var/log/init.log			644  3	   100	*     X
-/var/log/kerberos.log			600  7	   100	*     X
-/var/log/lpd-errs			644  7	   100	*     XC
-/var/log/maillog			640  7	   *	@T00  XC
-/var/log/messages			644  5	   100	@0101T XC
-/var/log/monthly.log			640  12	   *	$M1D0 XN
-/var/log/pflog				600  3	   100	*     XB    /var/run/pflogd.pid
-/var/log/ppp.log	root:network	640  3	   100	*     XC
-/var/log/security			600  10	   100	*     XC
+/var/log/all.log			600  7	   *	@T00  J
+/var/log/amd.log			644  7	   100	*     J
+/var/log/auth.log			600  7     100  @0101T JC
+/var/log/console.log			600  5	   100	*     J
+/var/log/cron				600  3	   100	*     JC
+/var/log/daily.log			640  7	   *	@T00  JN
+/var/log/debug.log			600  7     100  *     JC
+/var/log/init.log			644  3	   100	*     J
+/var/log/kerberos.log			600  7	   100	*     J
+/var/log/lpd-errs			644  7	   100	*     JC
+/var/log/maillog			640  7	   *	@T00  JC
+/var/log/messages			644  5	   100	@0101T JC
+/var/log/monthly.log			640  12	   *	$M1D0 JN
+/var/log/pflog				600  3	   100	*     JB    /var/run/pflogd.pid
+/var/log/ppp.log	root:network	640  3	   100	*     JC
+/var/log/security			600  10	   100	*     JC
 /var/log/sendmail.st			640  10	   *	168   BN
 /var/log/utx.log			644  3	   *	@01T05 B
-/var/log/weekly.log			640  5	   *	$W6D0 XN
-/var/log/xferlog			600  7	   100	*     XC
+/var/log/weekly.log			640  5	   *	$W6D0 JN
+/var/log/xferlog			600  7	   100	*     JC
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Comment 4 Ulrich Spoerlein freebsd_committer 2013-05-15 14:50:09 UTC
This is not only an issue for CPU, but more for RAM. XZ uses a lot more
RAM to compress files, and if an attacker can spam logfiles fast enough,
especially those that are not rotated on size, like maillog, then they
can probably gobble up enough RAM to make it be a problem.

I guess the low entropy of logfiles works in our favor here, however.

Cheers,
Uli
Comment 5 David Chisnall freebsd_committer 2018-03-08 10:21:16 UTC
Do we have any measurements regarding the CPU / RAM overhead here?  Can this be addressed by using a smaller block size without impacting compression ratios too much?

A few measurements from my largest maillog file (852K uncompressed, much smaller than on a large installation):

bzip2 baseline: 

7948  maximum resident set size
File size: 77KB.

xz --fast:

4328  maximum resident set size
File size: 80K

xz --best:

59848  maximum resident set size
File size: 70KB

xz -5:
26196  maximum resident set size
File size: 74KB

xz -block-size=1M:
26160  maximum resident set size
70K

The default appears to be --best.  

If I concatenate multiple rolled-over maillogs to give a 4.6MB file, I get this

bzip2 baseline: 

8316  maximum resident set size
File size: 443K.

xz --fast:

4436  maximum resident set size
File size: 447K

xz --best:

110092  maximum resident set size
File size: 382K

xz -5:
61524  maximum resident set size
File size: 412K

xz -block-size=1M

27920  maximum resident set size
File size: 396K

It looks as if setting the block size to 1MB prevents too much blowup in memory for very large files, gives us better compression that bzip2, though at the cost of more CPU and RAM.

I think that the correct solution for this is probably:

* Make the block size a configurable parameter
* Make the default 1M
* Make xz the default

This should let us get the benefit from better compression, but without letting attackers consume too much memory (though they can consume CPU in proportion to the size of the input file, with a larger constant multiplier for xz than bzip2).
Comment 6 Conrad Meyer freebsd_committer 2018-07-19 16:08:53 UTC
(In reply to David Chisnall from comment #5)
Perhaps memory use could be scaled to system size?  I don't know if xz has a mode for that.  Regardless, we should figure out a path to xz by default instead of bzip2.
Comment 7 Conrad Meyer freebsd_committer 2018-07-19 16:09:42 UTC
Reopen — it was unfixed when reverted.
Comment 8 Conrad Meyer freebsd_committer 2018-07-19 16:13:30 UTC
(In reply to Conrad Meyer from comment #6)
E.g.,

       --memlimit-compress=limit
              Set a memory usage limit for compression.
...
              If  the  compression settings exceed the limit, xz will adjust the settings downwards so that the
              limit is no longer exceeded
...
              The limit can be specified in multiple ways:

              ·  The limit can be an absolute value in bytes.  Using an integer suffix like MiB can be  useful.
                 Example: --memlimit-compress=80MiB

              ·  The limit can be specified as a percentage of total physical memory (RAM).  This can be useful
                 especially when setting the XZ_DEFAULTS environment variable in a shell initialization  script
                 that  is  shared  between  different computers.  That way the limit is automatically bigger on
                 systems with more memory.  Example: --memlimit-compress=70%