Bug 204735

I think I may have found a culprit - flowtable.

When comparing netinet/ip_output.c between 9-stable and 10-stable, I noticed the flowtable call is different. And looking into net/flowtable.c revealed that it was almost completely rewritten between 9.x and 10.x.

Flowtable is not included in GENERIC kernel, but our problematic host runs custom kernel with "options FLOWTABLE" defined (we had this in our kernel for 9.x and kept it when upgrading to 10.x). I've just disabled flowtable on the host via sysctl to check whether it helps (though it will take weeks to be sure, because the issue strikes just occasionally).

BTW, there are some fixes to flowtable in HEAD, specifically r292978 and r290276, both several months old. The latter was even scheduled to "MFC after 3 days", but it doesn't seem to be applied to 10-STABLE yet.

I faced the same problem. After some time some outgoing connections starts to balance on all available interfaces. Clearing and then applying default route helps for some time. I removed pf and rewrite all my rules for ipfw but problem is reappearing after some time.

 My kernels are mostly copied from 9 STABLE with additions from 10 STABLE and have too:
 options         FLOWTABLE               # per-cpu routing cache

Now i disabled it with:
sysctl net.flowtable.enable=0

I'll look if this problem happens again

After moving flowtable out of the picture, the affected router has been running with no problems for more than a month now. So I guess that described problem can be definitely considered flowtable-related (with r290276 - still not MFCed - being a fix candidate).

Comment 4 Oleksandr Tymoshenko 2019-02-01 22:30:51 UTC

flowtable feature was removed in base r321618, closing as OBE. If there are any plans to fix this in 11-STABLE, please reopen PR.