Make newsyslog compress logs with xz instead of bzip2 to save space. Fix: Patch attached with submission follows:
Author: eadler Date: Sun May 12 15:23:59 2013 New Revision: 250565 URL: http://svnweb.freebsd.org/changeset/base/250565 Log: Make newsyslog compress logs with xz instead of bzip2 to save space. PR: conf/178504 Submitted by: ak Reviewed by: smh Modified: head/etc/newsyslog.conf Modified: head/etc/newsyslog.conf ============================================================================== --- head/etc/newsyslog.conf Sun May 12 13:42:49 2013 (r250564) +++ head/etc/newsyslog.conf Sun May 12 15:23:59 2013 (r250565) @@ -17,23 +17,23 @@ # future, these defaults may change to more conservative ones. # # logfilename [owner:group] mode count size when flags [/pid_file] [sig_num] -/var/log/all.log 600 7 * @T00 J -/var/log/amd.log 644 7 100 * J -/var/log/auth.log 600 7 100 @0101T JC -/var/log/console.log 600 5 100 * J -/var/log/cron 600 3 100 * JC -/var/log/daily.log 640 7 * @T00 JN -/var/log/debug.log 600 7 100 * JC -/var/log/init.log 644 3 100 * J -/var/log/kerberos.log 600 7 100 * J -/var/log/lpd-errs 644 7 100 * JC -/var/log/maillog 640 7 * @T00 JC -/var/log/messages 644 5 100 @0101T JC -/var/log/monthly.log 640 12 * $M1D0 JN -/var/log/pflog 600 3 100 * JB /var/run/pflogd.pid -/var/log/ppp.log root:network 640 3 100 * JC -/var/log/security 600 10 100 * JC +/var/log/all.log 600 7 * @T00 X +/var/log/amd.log 644 7 100 * X +/var/log/auth.log 600 7 100 @0101T XC +/var/log/console.log 600 5 100 * X +/var/log/cron 600 3 100 * XC +/var/log/daily.log 640 7 * @T00 XN +/var/log/debug.log 600 7 100 * XC +/var/log/init.log 644 3 100 * X +/var/log/kerberos.log 600 7 100 * X +/var/log/lpd-errs 644 7 100 * XC +/var/log/maillog 640 7 * @T00 XC +/var/log/messages 644 5 100 @0101T XC +/var/log/monthly.log 640 12 * $M1D0 XN +/var/log/pflog 600 3 100 * XB /var/run/pflogd.pid +/var/log/ppp.log root:network 640 3 100 * XC +/var/log/security 600 10 100 * XC /var/log/sendmail.st 640 10 * 168 BN /var/log/utx.log 644 3 * @01T05 B -/var/log/weekly.log 640 5 * $W6D0 JN -/var/log/xferlog 600 7 100 * JC +/var/log/weekly.log 640 5 * $W6D0 XN +/var/log/xferlog 600 7 100 * XC _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
State Changed From-To: open->closed Committed. Thanks!
Author: eadler Date: Sun May 12 21:24:18 2013 New Revision: 250579 URL: http://svnweb.freebsd.org/changeset/base/250579 Log: Revert r250565 which causes issues for older CPUs PR: conf/178504 Requested by: many Modified: head/etc/newsyslog.conf Modified: head/etc/newsyslog.conf ============================================================================== --- head/etc/newsyslog.conf Sun May 12 20:44:28 2013 (r250578) +++ head/etc/newsyslog.conf Sun May 12 21:24:18 2013 (r250579) @@ -17,23 +17,23 @@ # future, these defaults may change to more conservative ones. # # logfilename [owner:group] mode count size when flags [/pid_file] [sig_num] -/var/log/all.log 600 7 * @T00 X -/var/log/amd.log 644 7 100 * X -/var/log/auth.log 600 7 100 @0101T XC -/var/log/console.log 600 5 100 * X -/var/log/cron 600 3 100 * XC -/var/log/daily.log 640 7 * @T00 XN -/var/log/debug.log 600 7 100 * XC -/var/log/init.log 644 3 100 * X -/var/log/kerberos.log 600 7 100 * X -/var/log/lpd-errs 644 7 100 * XC -/var/log/maillog 640 7 * @T00 XC -/var/log/messages 644 5 100 @0101T XC -/var/log/monthly.log 640 12 * $M1D0 XN -/var/log/pflog 600 3 100 * XB /var/run/pflogd.pid -/var/log/ppp.log root:network 640 3 100 * XC -/var/log/security 600 10 100 * XC +/var/log/all.log 600 7 * @T00 J +/var/log/amd.log 644 7 100 * J +/var/log/auth.log 600 7 100 @0101T JC +/var/log/console.log 600 5 100 * J +/var/log/cron 600 3 100 * JC +/var/log/daily.log 640 7 * @T00 JN +/var/log/debug.log 600 7 100 * JC +/var/log/init.log 644 3 100 * J +/var/log/kerberos.log 600 7 100 * J +/var/log/lpd-errs 644 7 100 * JC +/var/log/maillog 640 7 * @T00 JC +/var/log/messages 644 5 100 @0101T JC +/var/log/monthly.log 640 12 * $M1D0 JN +/var/log/pflog 600 3 100 * JB /var/run/pflogd.pid +/var/log/ppp.log root:network 640 3 100 * JC +/var/log/security 600 10 100 * JC /var/log/sendmail.st 640 10 * 168 BN /var/log/utx.log 644 3 * @01T05 B -/var/log/weekly.log 640 5 * $W6D0 XN -/var/log/xferlog 600 7 100 * XC +/var/log/weekly.log 640 5 * $W6D0 JN +/var/log/xferlog 600 7 100 * JC _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
This is not only an issue for CPU, but more for RAM. XZ uses a lot more RAM to compress files, and if an attacker can spam logfiles fast enough, especially those that are not rotated on size, like maillog, then they can probably gobble up enough RAM to make it be a problem. I guess the low entropy of logfiles works in our favor here, however. Cheers, Uli
Do we have any measurements regarding the CPU / RAM overhead here? Can this be addressed by using a smaller block size without impacting compression ratios too much? A few measurements from my largest maillog file (852K uncompressed, much smaller than on a large installation): bzip2 baseline: 7948 maximum resident set size File size: 77KB. xz --fast: 4328 maximum resident set size File size: 80K xz --best: 59848 maximum resident set size File size: 70KB xz -5: 26196 maximum resident set size File size: 74KB xz -block-size=1M: 26160 maximum resident set size 70K The default appears to be --best. If I concatenate multiple rolled-over maillogs to give a 4.6MB file, I get this bzip2 baseline: 8316 maximum resident set size File size: 443K. xz --fast: 4436 maximum resident set size File size: 447K xz --best: 110092 maximum resident set size File size: 382K xz -5: 61524 maximum resident set size File size: 412K xz -block-size=1M 27920 maximum resident set size File size: 396K It looks as if setting the block size to 1MB prevents too much blowup in memory for very large files, gives us better compression that bzip2, though at the cost of more CPU and RAM. I think that the correct solution for this is probably: * Make the block size a configurable parameter * Make the default 1M * Make xz the default This should let us get the benefit from better compression, but without letting attackers consume too much memory (though they can consume CPU in proportion to the size of the input file, with a larger constant multiplier for xz than bzip2).
(In reply to David Chisnall from comment #5) Perhaps memory use could be scaled to system size? I don't know if xz has a mode for that. Regardless, we should figure out a path to xz by default instead of bzip2.
Reopen — it was unfixed when reverted.
(In reply to Conrad Meyer from comment #6) E.g., --memlimit-compress=limit Set a memory usage limit for compression. ... If the compression settings exceed the limit, xz will adjust the settings downwards so that the limit is no longer exceeded ... The limit can be specified in multiple ways: · The limit can be an absolute value in bytes. Using an integer suffix like MiB can be useful. Example: --memlimit-compress=80MiB · The limit can be specified as a percentage of total physical memory (RAM). This can be useful especially when setting the XZ_DEFAULTS environment variable in a shell initialization script that is shared between different computers. That way the limit is automatically bigger on systems with more memory. Example: --memlimit-compress=70%
Keyword: patch or patch-ready – in lieu of summary line prefix: [patch] * bulk change for the keyword * summary lines may be edited manually (not in bulk). Keyword descriptions and search interface: <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>