Bug 195407 - relayd crashes kernel after update to 10.1-RELEASE
Summary: relayd crashes kernel after update to 10.1-RELEASE
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.1-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-11-26 10:34 UTC by Andrej Kolontai
Modified: 2015-03-12 07:35 UTC (History)
4 users (show)

See Also:


Attachments
kernel crash report (79.92 KB, text/plain)
2014-11-26 10:34 UTC, Andrej Kolontai
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrej Kolontai 2014-11-26 10:34:00 UTC
Created attachment 149878 [details]
kernel crash report

We are using FreeBSD for our Firewalls and are actually happy with it. Since recently we use relayd (installed via pkg) to do some load balancing stuff. On a freshly installed machine running 10.0-RELEASE everything worked fine. 
On monday, I tried to upgrade to 10.1-RELEASE using freebsd-update as described in the handbook chapter 24. At first everything looked good but relayd wouldn't come up:

"Nov 24 10:50:48 flutters relayd[3300]: fatal: cannot add rule: Operation not supported by device
Nov 24 10:50:48 flutters relayd[3293]: lost child: pfe exited abnormally"

When I tried to start it with /usr/local/etc/rc.d/relayd start the kernel panicked. I had to roll back the update (which worked fine). However, I was able to reproduce this behavior on a virtual machine. 


My guess is it happens here:
#7  0xffffffff81a37954 in pfr_detach_table (kt=0x0)
    at /usr/src_10.1.0/sys/modules/pf/../../netpfil/pf/pf_table.c:2047

The corresponding code is:
void
pfr_detach_table(struct pfr_ktable *kt)
{

        PF_RULES_WASSERT();
        KASSERT(kt->pfrkt_refcnt[PFR_REFCNT_RULE] > 0, ("%s: refcount %d\n",
            __func__, kt->pfrkt_refcnt[PFR_REFCNT_RULE]));

        if (!--kt->pfrkt_refcnt[PFR_REFCNT_RULE])
                pfr_setflags_ktable(kt, kt->pfrkt_flags&~PFR_TFLAG_REFERENCED);
}

From what I know about C programming: kt is not supposed to be 0x0. 
My guess was that some data structure has changed between 10.0 and 10.1 kernels. So a recompile of relayd should fix that. It did. I compiled it from the ports and it worked. 

Here's a procedure to reproduce the situation:

* install FreeBSD 10.0-RELEASE, relayd (configure it, start pf and relayd)
* update to FreeBSD 10.1-RELEASE using freebsd-update as described in the handbook
* after some reboots start pf and relayd. After startup, relayd will simply crash. After the second startup (wait some seconds) the kernel will crash. 

doing a pkg update, pkg upgrade after freebsd-update won't help. In fact, the relayd binaries seem to be identical on 10.0 and 10.1.
Comment 1 jjasen 2015-03-05 14:34:21 UTC
I'd like to put in a +1 for this bug. I can confirm it on FreeBSD 10.1-RELEASE-p6, which is current.

If relayd is started, it will crash as the original poster indicated, with a "relayd[3300]: fatal: cannot add rule: Operation not supported by device".

Restarting relayd can, and usually does, result in a kernel panic.
Comment 2 jjasen 2015-03-06 18:01:19 UTC
I can confirm the original posters opinion: "My guess was that some data structure has changed between 10.0 and 10.1 kernels. So a recompile of relayd should fix that. It did. I compiled it from the ports and it worked."

a) A fresh installation of FreeBSD-10.1, updated to p6, and pkg install relayd will crash relayd. restarting relayd will cause a kernel panic.

b) A fresh installation of FreeBSD-10.1, updated to p6, with relayd compiled from ports does NOT experience either issue.
Comment 3 holger 2015-03-12 07:35:37 UTC
(In reply to jjasen from comment #2)

PF ABI changed from 10.0 to 10.1. You can already see it when starting with 10.0 and installing a 10.1 kernel and reboot. pfctl will not be able to add rules.

The other part is that pf's ioctl interface (and probably other parts) does not seem to be very robust against API mis-usage (e.g. see http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf_ioctl.c?rev=1.236&content-type=text/x-cvsweb-markup)