My primary trouble with the current version, is that it relies on the arg1 element of the logged string to contain the culprit. That is often a wrong assumption -- depending on the ruleset, arg1 may contain the local address, for example. The attached patch makes use of the relay= element of the string, processing its entire content -- not just the IP address. The patch also: . obviates the need for daily_status_mail_rejects_logs by using find(1) to find the maillog-s modified within the last two days . allows one to keep maillog-s bzip2-ed . removes one invocation of sort(1) and one of uniq(1) -- both, perl and awk are quite capable of counting and hashing . removes one invocation of date -- both awk and perl are capable of figuring out the todays date . stops the input processing as soon as the timestamp on the next line starts indicating today -- to save time . replaces perl with awk, because I don't know perl that well How-To-Repeat: Here is the typical output of the current version. Note, that ALDAN.ALGEBRA.COM and virtual-estates.net are local machines here and are listed because certain addresses at them are blocked. Checking for rejected mail hosts: 7 ALDAN.ALGEBRA.COM 4 virtual-estates.net 4 [200.35.80.90] 3 [207.252.175.120] 2 imail.com 2 [61.144.185.23] 1 tig249-rsby.isp.net.au 1 spiderman.webworld1.net 1 host-148-244-121-173.block.alestra.net.mx 1 [216.105.175.129] 1 [208.60.110.2] 1 [200.74.132.158] 1 61-222-234-50.HINET-IP.hinet.net Here are the same logs processed by the new version: Checking for rejected mail hosts: 4 [200.35.80.90] 3 [207.252.175.120] 2 auction.jackpot.com [64.70.22.158] (may be forged) 2 [64.86.133.90] 2 [61.144.185.23] 1 tig249-rsby.isp.net.au [203.202.67.250] 1 qm-2.dlbdirect.com [64.152.73.103] 1 opt.edirectnetwork.net [66.115.47.116] (may be forged) 1 mailer5.hispeedoffers.com [64.32.63.39] 1 mail8.transcentives.net [216.23.198.158] 1 mail.bigfoot.com [64.15.239.140] 1 laoutbound3.jackpot.com [64.70.22.156] 1 host-148-244-121-173.block.alestra.net.mx [148.244.121.173] 1 cc7.optinmail.cc [64.38.239.36] 1 cc43.optinmail.cc [64.38.239.149] 1 cc42.optinmail.cc [64.38.239.148] 1 cc13.optinmail.cc [64.38.239.77] 1 [216.105.175.129] 1 [211.20.3.171] 1 [208.60.110.2] 1 [203.247.158.7] 1 [200.74.132.158] 1 61-222-234-50.HINET-IP.hinet.net [61.222.234.50]
Responsible Changed From-To: freebsd-bugs->brian I've got some changes close to this in the pipeline
The attached new version of the patch fixes a problem for the single digits days (awk's strftime pads them with leading zeros, while syslogd pads with spaces) and adds the total number of the rejections. -mi --- 460.status-mail-rejects Tue Oct 31 18:47:27 2000 +++ 460.status-mail-rejects Tue Mar 12 11:57:03 2002 @@ -33,23 +33,36 @@ echo Checking for rejected mail hosts: - start=`date -v-1d '+%b %d' | sed 's/0\(.\)$/ \1/'` - n=$(($daily_status_mail_rejects_logs - 2)) rc=$({ - while [ $n -ge 0 ] + for f in `find /var/log -name maillog\* \ + \( -mtime 1 -o -mtime 2 \) | xargs ls -tr` do - if [ -f /var/log/maillog.$n ] - then - cat /var/log/maillog.$n - elif [ -f /var/log/maillog.$n.gz ] - then - zcat -fc /var/log/maillog.$n.gz - fi - n=$(($n - 1)) + case $f in + *.gz) zcat -fc $f;; + *.bz2) bzip2 -cd $f;; + *) cat $f;; + esac done - cat /var/log/maillog - } | - perl -ne "print \"\$2\n\" - if (/reject=/ and /^$start.*ruleset=check_\S+,\s+arg1=(<[^@]+@)?([^>,]+).*reject=/o);" | - sort -f | uniq -ic | sort -fnr | tee /dev/stderr | wc -l) + } | awk ' + BEGIN { + today=systime(); + yesterday=strftime("%b %d", today-86400); + today=strftime("%b %d", today); + gsub(" 0", " ", today); gsub(" 0", " ", yesterday); + } + { + relay=gensub("^" yesterday ".*, relay=([^,]+), reject=.*", + "\\1", 1); + if (relay != $0) + rejects[relay]++; + else if (match($0, "^" today)) + exit; + } + END { + for (relay in rejects) { + printf("%4d %s\n", rejects[relay], relay); + total += rejects[relay]; + } + printf("%4d TOTAL\n", total); + }' | sort -fnr | tee /dev/stderr | wc -l) [ $rc -gt 0 ] && rc=1 fi;;
Polished the patch so it applies cleanly against 1.13 and 1.8.2.5 of 460.status-mail-rejects. Also included a small if test so that the total line is not printed out unless there was any rejected mails. -- Morten Rodal // // PGP ID 2D75595B // 22DE D67A 1AEA EF94 872A 9384 6D67 B50B 2D75 595B //
State Changed From-To: open->feedback Is there any chance that the submitter could update this patch?
State Changed From-To: feedback->suspended It does not sound like this is being worked on right now.
State Changed From-To: suspended->closed No response from the submitter.
brian@FreeBSD.org ÎÁÐÉÓÁ×(ÌÁ): > Synopsis: [patch] enhancing daily/460.status-mail-rejects > > State-Changed-From-To: suspended->closed > State-Changed-By: brian > State-Changed-When: Tue May 26 07:48:41 UTC 2009 > State-Changed-Why: > No response from the submitter. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=35018 > For the record, the submitter would like to apologize for failing in his responsibilities and dropping the figurative ball four years ago. Identifying the problems seven years ago was not enough... Expecting a project member, who volunteered in 2002 to take the PR and have committed a number of modifications to the script since, to also fix the identified problems was clearly too much. I shall try harder the next time, if I dare disturb the esteemed project members again with such non-sense, that is. -mi
On Tue, 26 May 2009 11:33:44 -0400, "Mikhail T." <mi+thun@aldan.algebra.com= > wrote: > brian@FreeBSD.org =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=B2(=D0=BB=D0=B0= ): > > Synopsis: [patch] enhancing daily/460.status-mail-rejects > > > > State-Changed-From-To: suspended->closed > > State-Changed-By: brian > > State-Changed-When: Tue May 26 07:48:41 UTC 2009 > > State-Changed-Why:=20 > > No response from the submitter. > > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D35018 > > =20 > For the record, the submitter would like to apologize for failing in his > responsibilities and dropping the figurative ball four years ago. > Identifying the problems seven years ago was not enough... Expecting a > project member, who volunteered in 2002 to take the PR and have > committed a number of modifications to the script since, to also fix the > identified problems was clearly too much. >=20 > I shall try harder the next time, if I dare disturb the esteemed project > members again with such non-sense, that is. To be honest, I felt quite guilty just closing this. I only did so because substantial changes were made to the script since and as I haven't seen any of my local machines turn up in the reports I figured I wouldn't be in a position to reproduce the original problem. As you're alive and kicking, would you be able to send me some maillog lines that cause the problem (if you can dig any up)? I'll redo the fix and submit it. Thanks - and sorry for the mucking about. --=20 Brian Somers <brian@Awfulhak.org> Don't _EVER_ lose your sense of humour ! <brian@FreeBSD.org>
State Changed From-To: closed->open This bug needs to be re-opened. It turns out that arg1 is a good value to show for check_mail rule failures, but not for check_mail or check_rcpt.
Author: brian Date: Thu May 28 07:43:06 2009 New Revision: 192970 URL: http://svn.freebsd.org/changeset/base/192970 Log: Update this script so that it handles different ruleset failures differently. The output now shows the ruleset and shortens to slightly different text (using $daily_status_mail_rejects_shorten), but it should be more descriptive. PR: 35018 Inspired by: Mikhail Teterin - mi at aldan dot algebra dot com MFC after: 3 weeks Modified: head/etc/periodic/daily/460.status-mail-rejects Modified: head/etc/periodic/daily/460.status-mail-rejects ============================================================================== --- head/etc/periodic/daily/460.status-mail-rejects Thu May 28 07:37:49 2009 (r192969) +++ head/etc/periodic/daily/460.status-mail-rejects Thu May 28 07:43:06 2009 (r192970) @@ -12,10 +12,8 @@ then fi case "$daily_status_mail_rejects_shorten" in -[Yy][Ee][Ss]) - sed_output='\4 \3...';; -*) - sed_output='\2 (\3... \4)';; +[Yy][Ee][Ss]) shorten='cut -d" " -f2,3';; +*) shorten=cat;; esac case "$daily_status_mail_rejects_enable" in @@ -39,7 +37,8 @@ case "$daily_status_mail_rejects_enable" echo echo Checking for rejected mail hosts: - start=`date -v-1d '+%b %e'` + yesterday=$(date -v-1d '+%b %e') + today=$(date '+%b %e') n=$(($daily_status_mail_rejects_logs - 2)) rc=$({ while [ $n -ge 0 ] @@ -57,9 +56,14 @@ case "$daily_status_mail_rejects_enable" n=$(($n - 1)) done cat /var/log/maillog - } | - sed -n -E "s/^$start"'.*ruleset=check_[^ ]+, +arg1=<?([^@]+@)?([^>,]+).*reject=([^ ]+) .* ([^ ]+)$/'"$sed_output"'/p' | - sort -f | uniq -ic | sort -fnr | tee /dev/stderr | wc -l) + } | sed -Ene "/^$today/q" -e "/^$yesterday/{"' + s/.*ruleset=check_relay,.* relay=([^,]+), reject=([^ ]*).*/\2 check_relay \1/p + t end + s/.*ruleset=check_rcpt,.* arg1=<?([^>,]+).* reject=([^ ]+) .* ([^ ]+)/\2 check_rcpt \1 \3/p + t end + s/.*ruleset=check_([^,]+),.* arg1=<?([^@]+@)?([^>,]+).* reject=([^ ]+) .* ([^ ]+)/\4 check_\1 \3 \5/p + :end + }' | eval $shorten | sort -f | uniq -ic | sort -fnr | tee /dev/stderr | wc -l) [ $rc -gt 0 ] && rc=1 fi;; _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
State Changed From-To: open->patched Patch r192970 applied to head. I'll MFC in 3 weeks.
State Changed From-To: patched->closed Merged to stable/7: r194411
Author: brian Date: Thu Jun 18 01:11:10 2009 New Revision: 194411 URL: http://svn.freebsd.org/changeset/base/194411 Log: MFC: r192970: Handle different rulesets. PR: 35018 Modified: stable/7/etc/ (props changed) stable/7/etc/periodic/daily/460.status-mail-rejects Modified: stable/7/etc/periodic/daily/460.status-mail-rejects ============================================================================== --- stable/7/etc/periodic/daily/460.status-mail-rejects Wed Jun 17 23:34:58 2009 (r194410) +++ stable/7/etc/periodic/daily/460.status-mail-rejects Thu Jun 18 01:11:10 2009 (r194411) @@ -12,10 +12,8 @@ then fi case "$daily_status_mail_rejects_shorten" in -[Yy][Ee][Ss]) - sed_output='\4 \3...';; -*) - sed_output='\2 (\3... \4)';; +[Yy][Ee][Ss]) shorten='cut -d" " -f2,3';; +*) shorten=cat;; esac case "$daily_status_mail_rejects_enable" in @@ -39,7 +37,8 @@ case "$daily_status_mail_rejects_enable" echo echo Checking for rejected mail hosts: - start=`date -v-1d '+%b %e'` + yesterday=$(date -v-1d '+%b %e') + today=$(date '+%b %e') n=$(($daily_status_mail_rejects_logs - 2)) rc=$({ while [ $n -ge 0 ] @@ -57,9 +56,14 @@ case "$daily_status_mail_rejects_enable" n=$(($n - 1)) done cat /var/log/maillog - } | - sed -n -E "s/^$start"'.*ruleset=check_[^ ]+, +arg1=<?([^@]+@)?([^>,]+).*reject=([^ ]+) .* ([^ ]+)$/'"$sed_output"'/p' | - sort -f | uniq -ic | sort -fnr | tee /dev/stderr | wc -l) + } | sed -Ene "/^$today/q" -e "/^$yesterday/{"' + s/.*ruleset=check_relay,.* relay=([^,]+), reject=([^ ]*).*/\2 check_relay \1/p + t end + s/.*ruleset=check_rcpt,.* arg1=<?([^>,]+).* reject=([^ ]+) .* ([^ ]+)/\2 check_rcpt \1 \3/p + t end + s/.*ruleset=check_([^,]+),.* arg1=<?([^@]+@)?([^>,]+).* reject=([^ ]+) .* ([^ ]+)/\4 check_\1 \3 \5/p + :end + }' | eval $shorten | sort -f | uniq -ic | sort -fnr | tee /dev/stderr | wc -l) [ $rc -gt 0 ] && rc=1 fi;; _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"