Years ago we found out that mail/procmail behaves badly (very slow when delivering large emails) due to its way it (ab)uses realloc(). Later, some logic was added to procmail to detect systems with "inefficient realloc" and appropriate measures are being taken. This detection logic doesn't work anymore on FreeBSD 5.4 (and, I suspect, on later versions as well) and procmail runs slow again. Fix: Well, I simply enabled the INEFFICIENTrealloc measure and procmail ran about 50(!) times faster here when delivering big emails: How-To-Repeat: Deliver big emails through procmail on 5.4 and 4.11. Measure and compare the time it takes.
Responsible Changed From-To: freebsd-ports-bugs->ache Over to maintainer
State Changed From-To: open->feedback Please investigate why INEFFICIENTrealloc test fails on your system. I automatically got Sorry, repeatedly reallocing is inefficient on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4
On Sat, 29-Oct-2005 at 06:20:05 +0000, Andrey A. Chernov wrote: > Synopsis: [PATCH] procmail again very slow on 5.x > > State-Changed-From-To: open->feedback > State-Changed-By: ache > State-Changed-When: Sat Oct 29 06:16:53 GMT 2005 > State-Changed-Why: > Please investigate why INEFFICIENTrealloc test fails on your system. I think the only person who can tell this is phk :-). > I automatically got > Sorry, repeatedly reallocing is inefficient > on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4 When trying it on my freshly booted FreeBSD 5.4 box at home, I get the same results as you do. But when trying it on FreeBSD 5.4 boxes at our company (with longer uptimes an probably heavylier used memory) it sometimes works and sometimes wont. Since the test routines try to come to a decision regarding realloc() implementation, I assume they depend very much on the malloc()/free() usage history since boot (and maybe on the phase of the moon, the contents of /dev/random and many more things :-)). Since FreeBSD _always_ has an INEFFICIENTrealloc (in procmail's eyes) but only _sometimes_ is able to detect this fact properly, I suggest enabling this feature statically (using the patch in my PR or by other means). -Andre > > http://www.freebsd.org/cgi/query-pr.cgi?pr=88120
On Sat, Oct 29, 2005 at 12:21:04PM +0200, Andre Albsmeier wrote: > > Please investigate why INEFFICIENTrealloc test fails on your system. > > I think the only person who can tell this is phk :-). Well, ask him. > > I automatically got > > Sorry, repeatedly reallocing is inefficient > > on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4 > > When trying it on my freshly booted FreeBSD 5.4 box at home, > I get the same results as you do. But when trying it on > FreeBSD 5.4 boxes at our company (with longer uptimes an > probably heavylier used memory) it sometimes works and > sometimes wont. Since the test routines try to come to a > decision regarding realloc() implementation, I assume they > depend very much on the malloc()/free() usage history since > boot (and maybe on the phase of the moon, the contents of > /dev/random and many more things :-)). I think another chance are malloc options, i.e. /etc/malloc.conf > Since FreeBSD _always_ has an INEFFICIENTrealloc (in procmail's > eyes) but only _sometimes_ is able to detect this fact properly, > I suggest enabling this feature statically (using the patch > in my PR or by other means). I fear to make any changes without understanding a problem, i.e. hacks. I prefer this subject be clearly traced. Perhaps something wrong on your company machines, so purposed change only masks the real bug on yours (or somebody else) side. -- http://ache.pp.ru/
On Sat, 29-Oct-2005 at 14:40:44 +0400, Andrey Chernov wrote: > On Sat, Oct 29, 2005 at 12:21:04PM +0200, Andre Albsmeier wrote: > > > Please investigate why INEFFICIENTrealloc test fails on your system. > > > > I think the only person who can tell this is phk :-). > > Well, ask him. > > > > I automatically got > > > Sorry, repeatedly reallocing is inefficient > > > on FreeBSD 6 & 7, but don't have access to FreeBSD 5.4 > > > > When trying it on my freshly booted FreeBSD 5.4 box at home, > > I get the same results as you do. But when trying it on > > FreeBSD 5.4 boxes at our company (with longer uptimes an > > probably heavylier used memory) it sometimes works and > > sometimes wont. Since the test routines try to come to a > > decision regarding realloc() implementation, I assume they > > depend very much on the malloc()/free() usage history since > > boot (and maybe on the phase of the moon, the contents of > > /dev/random and many more things :-)). > > I think another chance are malloc options, i.e. /etc/malloc.conf > > > Since FreeBSD _always_ has an INEFFICIENTrealloc (in procmail's > > eyes) but only _sometimes_ is able to detect this fact properly, > > I suggest enabling this feature statically (using the patch > > in my PR or by other means). > > I fear to make any changes without understanding a problem, i.e. hacks. > I prefer this subject be clearly traced. Perhaps something wrong on your > company machines, so purposed change only masks the real bug on yours (or > somebody else) side. Yeah, you are right. I have now done tests on 10 different boxes. It turned out that machines which are fast and idle and have fast storage more often fail to detect INEFFICIENTrealloc properly while others work almost always. To make a long story short: procmail's funky_realloc_test isn't always being built at all. To see what's happening let's patch src/autoconf first: --- procmail-3.22/src/autoconf.ORI Tue Sep 11 06:55:46 2001 +++ procmail-3.22/src/autoconf Sun Oct 30 09:28:04 2005 @@ -793,6 +793,7 @@ echo '#define NO_enum' >>$ACONF not_ISO=yep fi +md5 _autotst $RM _autotst.$O if test -n "$not_ISO" @@ -977,9 +978,10 @@ echo 'Checking realloc implementation' -if $MAKE _autotst >$DEVNULL 2>&1 +if $MAKE _autotst >/tmp/autost.log 2>&1 test -f _autotst then + md5 _autotst _autotst >_autotst.rrr 2>realloc.log case "$?" in 0) cat _autotst.rrr >>$ACONF;; Then we run make (on a fast and idle machine) and if the INEFFICIENTrealloc detection failed, we have a look at /tmp/autost.log: cc -c -O -pipe -fno-ident _autotst.c `_autotst' is up to date. OK, _autotst.o had been built but _autotst not. Now which one is it using? Let's look at the output of make: ... Proceeding with kernel-locking-support tests in the background Testing for const Testing for volatile Testing for prototypes Testing for enum MD5 (_autotst) = 4b8059db3a117b09ae46272a5e73a0a4 Checking for POSIX and ANSI/ISO system include files Checking for network/comsat/biff support Testing for void*, size_t, off_t, pid_t, time_t, mode_t, uid_t & gid_t Checking realloc implementation MD5 (_autotst) = 4b8059db3a117b09ae46272a5e73a0a4 Testing for WIFEXITED(), WIFSTOPPED(), WEXITSTATUS() & WSIGTERM() Testing for various struct passwd members ... It uses the _autost from the enum tests :-). So my patch to src/autoconf (which modified the realloc test code) worked only by accident -- make has built _autotst and INEFFICIENTrealloc was detected and I was happy. The right patch would be something like: --- src/autoconf.ORI Sun Oct 30 10:01:45 2005 +++ src/autoconf Sun Oct 30 10:03:56 2005 @@ -978,6 +978,7 @@ echo 'Checking realloc implementation' +$RM -f _autotst if $MAKE _autotst >$DEVNULL 2>&1 test -f _autotst then If have tested this now many times on the same machines and it worked always. -Andre
On Sun, Oct 30, 2005 at 10:30:37AM +0100, Andre Albsmeier wrote: > was detected and I was happy. The right patch would be something > like: > > --- src/autoconf.ORI Sun Oct 30 10:01:45 2005 > +++ src/autoconf Sun Oct 30 10:03:56 2005 I just commit it. Please send this patch along with explanation to procmail author too. -- http://ache.pp.ru/
State Changed From-To: feedback->closed Last fix committed