Bug 88120

Summary: [PATCH] procmail again very slow on 5.x
Product: Ports & Packages Reporter: Andre.Albsmeier
Component: Individual Port(s)Assignee: Andrey A. Chernov <ache>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
file.diff none

Description Andre.Albsmeier 2005-10-28 10:10:13 UTC 
Years ago we found out that mail/procmail behaves badly
(very slow when delivering large emails) due to its way it
(ab)uses realloc(). Later, some logic was added to procmail
to detect systems with "inefficient realloc" and appropriate
measures are being taken.

This detection logic doesn't work anymore on FreeBSD 5.4 (and,
I suspect, on later versions as well) and procmail runs slow
again.

Fix: Well, I simply enabled the INEFFICIENTrealloc measure and
procmail ran about 50(!) times faster here when delivering
big emails:
How-To-Repeat: 
Deliver big emails through procmail on 5.4 and 4.11.
Measure and compare the time it takes.
Comment 1 Volker Stolz freebsd_committer freebsd_triage 2005-10-28 14:20:42 UTC 
Responsible Changed
From-To: freebsd-ports-bugs->ache

Over to maintainer
Comment 2 Andrey A. Chernov freebsd_committer freebsd_triage 2005-10-29 07:16:53 UTC 
State Changed
From-To: open->feedback

Please investigate why INEFFICIENTrealloc test fails on your system.  
I automatically got  
Sorry, repeatedly reallocing is inefficient 
on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4
Comment 3 Andre.Albsmeier 2005-10-29 11:21:04 UTC 
On Sat, 29-Oct-2005 at 06:20:05 +0000, Andrey A. Chernov wrote:
> Synopsis: [PATCH] procmail again very slow on 5.x
> 
> State-Changed-From-To: open->feedback
> State-Changed-By: ache
> State-Changed-When: Sat Oct 29 06:16:53 GMT 2005
> State-Changed-Why: 
> Please investigate why INEFFICIENTrealloc test fails on your system. 

I think the only person who can tell this is phk :-).

> I automatically got 
> Sorry, repeatedly reallocing is inefficient
> on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4

When trying it on my freshly booted FreeBSD 5.4 box at home,
I get the same results as you do. But when trying it on
FreeBSD 5.4 boxes at our company (with longer uptimes an
probably heavylier used memory) it sometimes works and
sometimes wont. Since the test routines try to come to a
decision regarding realloc() implementation, I assume they
depend very much on the malloc()/free() usage history since
boot (and maybe on the phase of the moon, the contents of
/dev/random and many more things :-)).

Since FreeBSD _always_ has an INEFFICIENTrealloc (in procmail's
eyes) but only _sometimes_ is able to detect this fact properly,
I suggest enabling this feature statically (using the patch
in my PR or by other means).

	-Andre

> 
> http://www.freebsd.org/cgi/query-pr.cgi?pr=88120
Comment 4 Andrey A. Chernov freebsd_committer freebsd_triage 2005-10-29 11:40:44 UTC 
On Sat, Oct 29, 2005 at 12:21:04PM +0200, Andre Albsmeier wrote:
> > Please investigate why INEFFICIENTrealloc test fails on your system. 
> 
> I think the only person who can tell this is phk :-).

Well, ask him.

> > I automatically got 
> > Sorry, repeatedly reallocing is inefficient
> > on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4
> 
> When trying it on my freshly booted FreeBSD 5.4 box at home,
> I get the same results as you do. But when trying it on
> FreeBSD 5.4 boxes at our company (with longer uptimes an
> probably heavylier used memory) it sometimes works and
> sometimes wont. Since the test routines try to come to a
> decision regarding realloc() implementation, I assume they
> depend very much on the malloc()/free() usage history since
> boot (and maybe on the phase of the moon, the contents of
> /dev/random and many more things :-)).

I think another chance are malloc options, i.e. /etc/malloc.conf

> Since FreeBSD _always_ has an INEFFICIENTrealloc (in procmail's
> eyes) but only _sometimes_ is able to detect this fact properly,
> I suggest enabling this feature statically (using the patch
> in my PR or by other means).

I fear to make any changes without understanding a problem, i.e. hacks.
I prefer this subject be clearly traced. Perhaps something wrong on your 
company machines, so purposed change only masks the real bug on yours (or 
somebody else) side.

-- 
http://ache.pp.ru/
Comment 5 Andre.Albsmeier 2005-10-30 09:30:37 UTC 
On Sat, 29-Oct-2005 at 14:40:44 +0400, Andrey Chernov wrote:
> On Sat, Oct 29, 2005 at 12:21:04PM +0200, Andre Albsmeier wrote:
> > > Please investigate why INEFFICIENTrealloc test fails on your system. 
> > 
> > I think the only person who can tell this is phk :-).
> 
> Well, ask him.
> 
> > > I automatically got 
> > > Sorry, repeatedly reallocing is inefficient
> > > on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4
> > 
> > When trying it on my freshly booted FreeBSD 5.4 box at home,
> > I get the same results as you do. But when trying it on
> > FreeBSD 5.4 boxes at our company (with longer uptimes an
> > probably heavylier used memory) it sometimes works and
> > sometimes wont. Since the test routines try to come to a
> > decision regarding realloc() implementation, I assume they
> > depend very much on the malloc()/free() usage history since
> > boot (and maybe on the phase of the moon, the contents of
> > /dev/random and many more things :-)).
> 
> I think another chance are malloc options, i.e. /etc/malloc.conf
> 
> > Since FreeBSD _always_ has an INEFFICIENTrealloc (in procmail's
> > eyes) but only _sometimes_ is able to detect this fact properly,
> > I suggest enabling this feature statically (using the patch
> > in my PR or by other means).
> 
> I fear to make any changes without understanding a problem, i.e. hacks.
> I prefer this subject be clearly traced. Perhaps something wrong on your 
> company machines, so purposed change only masks the real bug on yours (or 
> somebody else) side.

Yeah, you are right. I have now done tests on 10 different boxes.
It turned out that machines which are fast and idle and have fast
storage more often fail to detect INEFFICIENTrealloc properly
while others work almost always.

To make a long story short: procmail's funky_realloc_test isn't
always being built at all. To see what's happening let's patch
src/autoconf first:

--- procmail-3.22/src/autoconf.ORI	Tue Sep 11 06:55:46 2001
+++ procmail-3.22/src/autoconf	Sun Oct 30 09:28:04 2005
@@ -793,6 +793,7 @@
  echo '#define NO_enum' >>$ACONF
  not_ISO=yep
 fi
+md5 _autotst
 $RM _autotst.$O
 
 if test -n "$not_ISO"
@@ -977,9 +978,10 @@
 
 echo 'Checking realloc implementation'
 
-if $MAKE _autotst >$DEVNULL 2>&1
+if $MAKE _autotst >/tmp/autost.log 2>&1
  test -f _autotst
 then
+ md5 _autotst
  _autotst >_autotst.rrr 2>realloc.log
  case "$?" in
  0) cat _autotst.rrr >>$ACONF;;



Then we run make (on a fast and idle machine) and if the
INEFFICIENTrealloc detection failed, we have a look at
/tmp/autost.log:

cc -c -O -pipe -fno-ident  _autotst.c
`_autotst' is up to date.

OK, _autotst.o had been built but _autotst not. Now which
one is it using? Let's look at the output of make:

...
Proceeding with kernel-locking-support tests in the background
Testing for const
Testing for volatile
Testing for prototypes
Testing for enum
MD5 (_autotst) = 4b8059db3a117b09ae46272a5e73a0a4
Checking for POSIX and ANSI/ISO system include files
Checking for network/comsat/biff support
Testing for void*, size_t, off_t, pid_t, time_t, mode_t, uid_t & gid_t
Checking realloc implementation
MD5 (_autotst) = 4b8059db3a117b09ae46272a5e73a0a4
Testing for WIFEXITED(), WIFSTOPPED(), WEXITSTATUS() & WSIGTERM()
Testing for various struct passwd members
...


It uses the _autost from the enum tests :-). So my patch to
src/autoconf (which modified the realloc test code) worked
only by accident -- make has built _autotst and INEFFICIENTrealloc
was detected and I was happy. The right patch would be something
like:

--- src/autoconf.ORI	Sun Oct 30 10:01:45 2005
+++ src/autoconf	Sun Oct 30 10:03:56 2005
@@ -978,6 +978,7 @@
 
 echo 'Checking realloc implementation'
 
+$RM -f _autotst
 if $MAKE _autotst >$DEVNULL 2>&1
  test -f _autotst
 then


If have tested this now many times on the same machines and
it worked always.

	-Andre
Comment 6 Andrey A. Chernov freebsd_committer freebsd_triage 2005-10-30 09:53:23 UTC 
On Sun, Oct 30, 2005 at 10:30:37AM +0100, Andre Albsmeier wrote:
> was detected and I was happy. The right patch would be something
> like:
> 
> --- src/autoconf.ORI	Sun Oct 30 10:01:45 2005
> +++ src/autoconf	Sun Oct 30 10:03:56 2005

I just commit it. Please send this patch along with explanation to 
procmail author too.

-- 
http://ache.pp.ru/
Comment 7 Andrey A. Chernov freebsd_committer freebsd_triage 2005-10-30 09:58:56 UTC 
State Changed
From-To: feedback->closed

Last fix committed