On FreeBSD amd64, after about 196 days uptime, the clock_t data type rolls over
from 2147483647 (0x7FFFFFFF) to -2147483648 (0x80000000).

When this happens, cl_times()'s return value, which is unsigned long (64bit)
jumps from 0x000000007FFFFFFF to 0xFFFFFFFF80000000, wreaking all sorts of
havoc, e.g. total spam of messages like:

May 18 10:00:05 baweb01-tev heartbeat: [84855]: WARN: Gmain_timeout_dispatch:
Dispatch function for memory stats was delayed 144115187989455872 ms (> 5000
ms) before being called (GSource: 0x802531318)
May 18 10:00:05 baweb01-tev heartbeat: [84855]: WARN: Gmain_timeout_dispatch:
Dispatch function for memory stats was delayed 144115187989455879 ms (> 5000
ms) before being called (GSource: 0x802531318)

till heartbeat finally shuts down.

Furthermore, on this platform, ./configure does not detect that the rollover
workaround has to be used, as sizeof(long) is 8, which for some odd reason is
the condition for "clock_t is long enough".

Note: I placed the same bug report upstream on http://developerbugs.linux-foundation.org/show_bug.cgi?id=2596

Fix: I attached a patch doing the following changes:

- fix to ./configure to check size of clock_t, not long
- changed return type of cl_times() from unsigned long to clock_t
- make all callers able to cope with that change - time_longclock() is the only
caller that doesn't immediately convert cl_times()'s return value to a clock_t,
so only that one needed changes
- changed wraparound logic to no longer rely on unsigned long being "good
enough". Instead, the static variables are changed to longclock_t, and sign
extension is explicitly prevented by an AND with the maximum "unsigned" clock_t
value

It fixes the issue in our lab, and should do no harm on other platforms.

Patch attached with submission follows:
How-To-Repeat: Also, for testing purposes, I provided the source code of a "fake uptime"
preload library to fake the return value of times(). That way, the problem can
be reproduced without having access to a FreeBSD amd64 system with an uptime of
196 days. Usage:

gcc -shared -o /tmp/fu.so fu.c -Os -fPIC -s -Wall -Wextra
env LD_PRELOAD=/tmp/fu.so FU_TIMES=0x7FFF0000 /usr/local/etc/rc.d/heartbeat
start

Note that the library may need changes if libc is not /lib/libc.so.7

Source code:


#include <sys/times.h>
#include <err.h>
#include <stdbool.h>
#include <stdlib.h>
#include <dlfcn.h>

typedef clock_t (times_t) (struct tms *buffer);
static times_t *orig_times = NULL;
static bool initialized = false;
static clock_t offset = 0;

static void init()
{
        struct tms t;

        if(initialized)
                return;
        initialized = true;

        orig_times = (times_t *) dlfunc(RTLD_NEXT, "times");
        if(!orig_times)
        {
                void *libc = dlopen("/lib/libc.so.7", RTLD_LAZY);
                orig_times = (times_t *) dlfunc(libc, "times");
        }
        if(!orig_times)
                errx(1, "cannot find times()");

        const char *p = getenv("FU_TIMES");
        if(p)
        {
                clock_t ret = times(&t);
                offset = strtoll(p, NULL, 0) - ret;
        }
        else
                offset = 0;

        initialized = true;
}

clock_t times(struct tms *buffer)
{
        init();
        clock_t ret = orig_times(buffer);
        return ret + offset;
}