Corresponding discussion over email is available at: http://lists.freebsd.org/pipermail/freebsd-stable/2014-September/080296.html


We discovered that our newly upgraded 10.1-BETA2 (r271983) has started leaking kernel memory and the *wired* portion has been steadily growing over the last few days. The *routetbl* pool is getting bloated with time.


# while true; do date; vmstat -m | grep routetbl; sleep 60; done

         Type InUse   MemUse HighUse Requests  Size(s)
Mon Sep 29 18:27:55 UTC 2014
     routetbl 5988792 2888491K       - 14285826  32,64,128,256,512,2048
Mon Sep 29 18:28:55 UTC 2014
     routetbl 5990120 2889131K       - 14288972  32,64,128,256,512,2048
Mon Sep 29 18:29:55 UTC 2014
     routetbl 5991448 2889771K       - 14292352  32,64,128,256,512,2048
Mon Sep 29 18:30:55 UTC 2014
     routetbl 5992776 2890411K       - 14295464  32,64,128,256,512,2048
Mon Sep 29 18:31:55 UTC 2014
     routetbl 5994104 2891051K       - 14298576  32,64,128,256,512,2048
Mon Sep 29 18:32:55 UTC 2014
     routetbl 5995432 2891691K       - 14301904  32,64,128,256,512,2048
Mon Sep 29 18:33:55 UTC 2014
     routetbl 5996096 2892011K       - 14303624  32,64,128,256,512,2048
Mon Sep 29 18:34:55 UTC 2014
     routetbl 5997422 2892650K       - 14306980  32,64,128,256,512,2048
Mon Sep 29 18:35:55 UTC 2014
     routetbl 5998750 2893290K       - 14310092  32,64,128,256,512,2048
Mon Sep 29 18:36:55 UTC 2014
     routetbl 6000078 2893930K       - 14313204  32,64,128,256,512,2048
Mon Sep 29 18:37:55 UTC 2014
     routetbl 6001406 2894570K       - 14316532  32,64,128,256,512,2048
Mon Sep 29 18:38:55 UTC 2014
     routetbl 6002734 2895210K       - 14319644  32,64,128,256,512,2048
Mon Sep 29 18:39:55 UTC 2014
     routetbl 6004062 2895850K       - 14323024  32,64,128,256,512,2048
Mon Sep 29 18:40:56 UTC 2014
     routetbl 6004726 2896170K       - 14324745  32,64,128,256,512,2048
Mon Sep 29 18:41:56 UTC 2014
     routetbl 6006054 2896810K       - 14327857  32,64,128,256,512,2048
Mon Sep 29 18:42:56 UTC 2014
     routetbl 6007382 2897450K       - 14331185  32,64,128,256,512,2048
Mon Sep 29 18:43:56 UTC 2014
     routetbl 6008710 2898090K       - 14334297  32,64,128,256,512,2048


After some investigation we discovered that the memory leak is triggered by a reload of the PF rule set. A simple test case with 1 rule and 1 table reveals the bug and makes it easy to reproduce. Every time 'pfctl -f /etc/firewall/pf.conf' (or your corresponding pf.conf file) is run the output above shows an increase in memory. In between reloads the memory usage stays stable.

The following DTrace script could be used to watch the same behavior:

#!/usr/sbin/dtrace -s
#pragma D option quiet

BEGIN {
    printf("%-20s %20s %20s %20s\n", "FUNCTION", "ALLOCATED", "FREE", "TOTAL");
}

dtmalloc::$1:malloc {
    @malloc[probefunc] = sum(args[3]);
    @total[probefunc] = sum(args[3]);
}

dtmalloc::$1:free {
    @free[probefunc] = sum(args[3]);
    @total[probefunc] = sum(-args[3]);
}

tick-1s {
    printa("%-20.20s %20@d %20@d %20@d\n", @malloc, @free, @total);
}


Output:
# ./script2.d routetbl
FUNCTION  ALLOCATED    FREE   TOTAL
routetbl        512     512       0
...
routetbl      46592   46592       0
routetbl     808960  444416  364544
...
routetbl     861184  496640  364544
routetbl    1623552  894464  729088
...
routetbl    1641984  912896  729088



Further diagnostics:

#!/usr/sbin/dtrace -s
fbt:kernel:rt_msg2:entry {
    @rt_msg2[stack()] = count();
}
fbt:kernel:rn_addroute:entry {
    @rn_addroute[stack()] = count();
}

Output:
              kernel`sysctl_rtsock+0x274
              kernel`sysctl_root+0x214
              kernel`userland_sysctl+0x1d8
              kernel`sys___sysctl+0x74
              kernel`amd64_syscall+0x334
              kernel`0xffffffff80900e0b
             2340

              kernel`sysctl_rtsock+0x64c
              kernel`sysctl_root+0x214
              kernel`userland_sysctl+0x1d8
              kernel`sys___sysctl+0x74
              kernel`amd64_syscall+0x334
              kernel`0xffffffff80900e0b
             4680

This problem was reported to Gleb Smirnoff and he managed to reproduce and confirm the problem.