On high-load network router since some time (1-5 days) process flowcleaner get's 100% load on a one CPU's core. ssh works, remote reboot can't be done, only hard reboot. Bug was seen at many routers: http://forum.nag.ru/forum/index.php?showtopic=55485&st=0 Fix: Without "flowtable" option in kernel's config router works fine. How-To-Repeat: Bug can be seen only at high-load (more than 30 megabits) routers.
Responsible Changed From-To: freebsd-amd64->freebsd-net Probably not amd64-specific.
Hi! I observe a similar behaviour on a 8.0-RELEASE-p2 i386 GENERIC kernel. System receives 2 BGP4 fullfeeds (approx. 310K routes each). The system is still running, a few processes are unkillable or die only after a long amount (1-2h) of time. Here's the list of unkillable processes: 80871 ?? R 0:00.00 /bin/sh /etc/periodic/daily/470.status-named 76499 ?? Rs 0:00.01 sshd: [accepted] (sshd) 76922 ?? Rs 0:00.01 sshd: [accepted] (sshd) flowcleaner looks pretty busy (for an uptime of approx. 40h): 22 ?? RL 1209:50.98 [flowcleaner] 4:17PM up 1 day, 22:22, 2 users, load averages: 7.20, 6.53, 5.81 quagga is running on the system, bgpd mgmt cli is no longer reachable: # telnet 0 2605 Trying 0.0.0.0... Connected to 0. Escape character is '^]'. ^] telnet> close Connection closed. # What can I do to help to debug this ? No console access available right now, but can probably made available. This is a production host, but not yet super-critical, so... -- pi@opsec.eu +49 171 3101372 10 years to go !
FWIW, this flowtable problem still can be triggered in 8.1-RELEASE. Yesterday it was seen on a machine with approx. 300 routes from OSPF -- no BGP full views involved. The machine was unable to reboot cleanly, was stuck forever waiting for vnlru to exit. Can it be a locking issue? Thanks, Yar
Today I recognize also 100 % CPU load by flowcleaner. Machine is a fresh installed 8.1 stable with *no* router functionality but with opennms - a netmanagement application - installed. Nicola=
Hi folks, I hit this problem on a pair of anycast name servers. What I'm running: FreeBSD ns1.example.com 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 Here's a peak at ps: ns1b# ps auxwww | head USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 100.0 0.0 0 32 ?? RL 11Jan11 59960:15.43 [idle] root 21 100.0 0.0 0 16 ?? RL 11Jan11 1112:01.24 [flowcleaner] root 0 0.0 0.0 0 96 ?? DLs 11Jan11 0:02.94 [kernel] root 1 0.0 0.0 3204 556 ?? ILs 11Jan11 0:00.01 /sbin/init -- root 2 0.0 0.0 0 16 ?? DL 11Jan11 0:52.87 [g_event] root 3 0.0 0.0 0 16 ?? DL 11Jan11 0:10.10 [g_up] root 4 0.0 0.0 0 16 ?? DL 11Jan11 0:15.18 [g_down] root 5 0.0 0.0 0 16 ?? DL 11Jan11 0:00.00 [mpt_recovery0] The box is running Quagga with a single OSPF adjacency. It has about 500 routes. Both anycast instances of ns1 hit this problem, but neither instance of ns2, which are configured identically, saw the trouble. The ns1 name servers are much busier than ns2. It appears that one instance of ns1 died almost a week ago, which went unnoticed :-( This morning, the second instance died. At that point, it was hard not to notice :-) Traffic on the mailing list suggests that 'sysctl net.inet.flowtable.enable=0' is a work-around. We'll pursue that path and hope for a bug fix in the not-too-distant future. thanks, mark
Hi, things could be improved in HEAD and stable/8 (as of r217168 [1]). Please test and report back. /bz [1] http://svn.freebsd.org/viewvc/base?view=revision&revision=217168 -- Bjoern A. Zeeb You have to have visions! <ks> Going to jail sucks -- <bz> All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
State Changed From-To: open->feedback Submitter(s) were asked for feedback on a more up-to-date stable/8 or HEAD back in February.
Author: bz Date: Sat Mar 19 15:50:34 2011 New Revision: 219775 URL: http://svn.freebsd.org/changeset/base/219775 Log: For now remove options FLOWTABLE from the remaining GENERIC kernel configurations and make it opt-in for those who want it. LINT will still build it. While it may be a perfect win in some scenarios, it still troubles users (see PRs) in general cases. In addition we are still allocating resources even if disabled by sysctl and still leak arp/nd6 entries in case of interface destruction. Discussed with: qingli (2010-11-24, just never executed) Discussed with: juli (OCTEON1) PR: kern/148018, kern/155604, kern/144917, kern/146792 MFC after: 2 weeks Modified: head/sys/amd64/conf/GENERIC head/sys/i386/conf/GENERIC head/sys/ia64/conf/GENERIC head/sys/mips/conf/OCTEON1 Modified: head/sys/amd64/conf/GENERIC ============================================================================== --- head/sys/amd64/conf/GENERIC Sat Mar 19 14:44:24 2011 (r219774) +++ head/sys/amd64/conf/GENERIC Sat Mar 19 15:50:34 2011 (r219775) @@ -61,7 +61,6 @@ options KBD_INSTALL_CDEV # install a CD options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework -options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel Modified: head/sys/i386/conf/GENERIC ============================================================================== --- head/sys/i386/conf/GENERIC Sat Mar 19 14:44:24 2011 (r219774) +++ head/sys/i386/conf/GENERIC Sat Mar 19 15:50:34 2011 (r219775) @@ -62,7 +62,6 @@ options KBD_INSTALL_CDEV # install a CD options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework -options FLOWTABLE # per-cpu routing cache #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel Modified: head/sys/ia64/conf/GENERIC ============================================================================== --- head/sys/ia64/conf/GENERIC Sat Mar 19 14:44:24 2011 (r219774) +++ head/sys/ia64/conf/GENERIC Sat Mar 19 15:50:34 2011 (r219775) @@ -31,8 +31,6 @@ options COMPAT_FREEBSD7 # Compatible wi options DDB # Support DDB options DEADLKRES # Enable the deadlock resolver options FFS # Berkeley Fast Filesystem -#options FLOWTABLE # per-cpu routing cache (removed due to - # misaligned access -- see kern/148018) options GDB # Support remote GDB options GEOM_LABEL # Provides labelization options INCLUDE_CONFIG_FILE # Include this file in kernel Modified: head/sys/mips/conf/OCTEON1 ============================================================================== --- head/sys/mips/conf/OCTEON1 Sat Mar 19 14:44:24 2011 (r219774) +++ head/sys/mips/conf/OCTEON1 Sat Mar 19 15:50:34 2011 (r219775) @@ -73,7 +73,6 @@ options PRINTF_BUFR_SIZE=128 # Prevent options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework -options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Author: bz Date: Sat Apr 9 12:04:35 2011 New Revision: 220486 URL: http://svn.freebsd.org/changeset/base/220486 Log: MFC r219775: For now remove options FLOWTABLE from the remaining GENERIC kernel configurations and make it opt-in for those who want it. LINT will still build it. While it may be a perfect win in some scenarios, it still troubles users (see PRs) in general cases. In addition we are still allocating resources even if disabled by sysctl and still leak arp/nd6 entries in case of interface destruction. Discussed with: qingli (2010-11-24, just never executed) Discussed with: juli (OCTEON1) PR: kern/148018, kern/155604, kern/144917, kern/146792 Modified: stable/8/sys/amd64/conf/GENERIC stable/8/sys/i386/conf/GENERIC stable/8/sys/ia64/conf/GENERIC stable/8/sys/mips/conf/OCTEON1 Directory Properties: stable/8/sys/ (props changed) stable/8/sys/amd64/include/xen/ (props changed) stable/8/sys/cddl/contrib/opensolaris/ (props changed) stable/8/sys/contrib/dev/acpica/ (props changed) stable/8/sys/contrib/pf/ (props changed) Modified: stable/8/sys/amd64/conf/GENERIC ============================================================================== --- stable/8/sys/amd64/conf/GENERIC Sat Apr 9 11:03:04 2011 (r220485) +++ stable/8/sys/amd64/conf/GENERIC Sat Apr 9 12:04:35 2011 (r220486) @@ -72,7 +72,6 @@ options KBD_INSTALL_CDEV # install a CD options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework -options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel Modified: stable/8/sys/i386/conf/GENERIC ============================================================================== --- stable/8/sys/i386/conf/GENERIC Sat Apr 9 11:03:04 2011 (r220485) +++ stable/8/sys/i386/conf/GENERIC Sat Apr 9 12:04:35 2011 (r220486) @@ -73,7 +73,6 @@ options KBD_INSTALL_CDEV # install a CD options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework -options FLOWTABLE # per-cpu routing cache #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel Modified: stable/8/sys/ia64/conf/GENERIC ============================================================================== --- stable/8/sys/ia64/conf/GENERIC Sat Apr 9 11:03:04 2011 (r220485) +++ stable/8/sys/ia64/conf/GENERIC Sat Apr 9 12:04:35 2011 (r220486) @@ -31,7 +31,6 @@ options COMPAT_43TTY # BSD 4.3 TTY comp options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options FFS # Berkeley Fast Filesystem -options FLOWTABLE # per-cpu routing cache options GDB # Support remote GDB options GEOM_LABEL # Provides labelization options INCLUDE_CONFIG_FILE # Include this file in kernel Modified: stable/8/sys/mips/conf/OCTEON1 ============================================================================== --- stable/8/sys/mips/conf/OCTEON1 Sat Apr 9 11:03:04 2011 (r220485) +++ stable/8/sys/mips/conf/OCTEON1 Sat Apr 9 12:04:35 2011 (r220486) @@ -76,7 +76,6 @@ options PRINTF_BUFR_SIZE=128 # Prevent options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework -options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Have you tested this workload with 9.0 with the flowtable enabled? Is the problem as severe? Thanks
9.3 same issue
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
FLOWTABLE deorbited long time ago.