Bug 193452 - Dell PowerEdge 210 II -- Kernel panic bce (broadcom)
Summary: Dell PowerEdge 210 II -- Kernel panic bce (broadcom)
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.3-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-net mailing list
URL:
Keywords: crash, needs-qa
Depends on:
Blocks:
 
Reported: 2014-09-08 06:30 UTC by Tomasz
Modified: 2017-06-12 05:05 UTC (History)
3 users (show)

See Also:
koobs: mfc-stable9?
koobs: mfc-stable10?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tomasz 2014-09-08 06:30:43 UTC
We are experiencing multiple kernel panics per day on a series of Dell PowerEdge
R210 II


kgdb kernel.debug /var/crash/vmcore.6
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff809a1c47
stack pointer           = 0x28:0xffffff8211d25560
frame pointer           = 0x28:0xffffff8211d255c0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (irq265: bce0)
trap number             = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff809759f6 at kdb_backtrace+0x66
#1 0xffffffff8093ba1e at panic+0x1ce
#2 0xffffffff80d337a0 at trap_fatal+0x290
#3 0xffffffff80d33b01 at trap_pfault+0x211
#4 0xffffffff80d340b4 at trap+0x344
#5 0xffffffff80d1d3e3 at calltrap+0x8
#6 0xffffffff80a671d0 at ip_fragment+0x1c0
#7 0xffffffff80a68065 at ip_output+0xc15
#8 0xffffffff80a63f6a at ip_forward+0x2ea
#9 0xffffffff80a655ec at ip_input+0x5bc
#10 0xffffffff80a07098 at netisr_dispatch_src+0x218
#11 0xffffffff809fdc3d at ether_demux+0x14d
#12 0xffffffff809fdf1e at ether_nh_input+0x1fe
#13 0xffffffff80a07098 at netisr_dispatch_src+0x218
#14 0xffffffff8047f71a at bce_intr+0x47a
#15 0xffffffff8090be8d at intr_event_execute_handlers+0xfd
#16 0xffffffff8090d67d at ithread_loop+0x9d
#17 0xffffffff809088df at fork_exit+0x11f
Uptime: 3d21h56m4s
Dumping 602 out of 4043 MB:..3%..11%..22%..32%..43%..51%..62%..72%..83%..91%

#0  doadump (textdump=<value optimized out>) at pcpu.h:234
234             __asm("movq %%gs:%1,%0" : "=r" (td)

(kgdb)
(kgdb) backtrace
#0  doadump (textdump=<value optimized out>) at pcpu.h:234
#1  0xffffffff8093b4f6 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:449
#2  0xffffffff8093b9f7 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:637
#3  0xffffffff80d337a0 in trap_fatal (frame=0xc, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:879
#4  0xffffffff80d33b01 in trap_pfault (frame=0xffffff8211d254b0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:795
#5  0xffffffff80d340b4 in trap (frame=0xffffff8211d254b0) at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80d1d3e3 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff809a1c47 in m_copym (m=0x0, off0=1500, len=1480, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:542
#8  0xffffffff80a671d0 in ip_fragment (ip=0xfffffe003deb4980, m_frag=0xffffff8211d256c8, mtu=<value optimized out>, if_hwassist_flags=<value optimized out>, sw_csum=<value optimized out>)
    at /usr/src/sys/netinet/ip_output.c:813
#9  0xffffffff80a68065 in ip_output (m=0xfffffe003deb4900, opt=<value optimized out>, ro=0xffffff8211d25780, flags=1, imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:647
#10 0xffffffff80a63f6a in ip_forward (m=0xfffffe003deb4900, srcrt=<value optimized out>) at /usr/src/sys/netinet/ip_input.c:1494
#11 0xffffffff80a655ec in ip_input (m=0xfffffe003deb4900) at /usr/src/sys/netinet/ip_input.c:702
#12 0xffffffff80a07098 in netisr_dispatch_src (proto=1, source=<value optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1013
#13 0xffffffff809fdc3d in ether_demux (ifp=0xfffffe00032f4000, m=0xfffffe003deb4900) at /usr/src/sys/net/if_ethersubr.c:943
#14 0xffffffff809fdf1e in ether_nh_input (m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:762
#15 0xffffffff80a07098 in netisr_dispatch_src (proto=9, source=<value optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1013
#16 0xffffffff8047f71a in bce_intr (xsc=<value optimized out>) at /usr/src/sys/dev/bce/if_bce.c:6830
#17 0xffffffff8090be8d in intr_event_execute_handlers (p=<value optimized out>, ie=0xfffffe0003371600) at /usr/src/sys/kern/kern_intr.c:1272
#18 0xffffffff8090d67d in ithread_loop (arg=0xfffffe0003368ac0) at /usr/src/sys/kern/kern_intr.c:1285
#19 0xffffffff809088df in fork_exit (callout=0xffffffff8090d5e0 <ithread_loop>, arg=0xfffffe0003368ac0, frame=0xffffff8211d25b00) at /usr/src/sys/kern/kern_fork.c:992
#20 0xffffffff80d1d90e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606
#21 0x0000000000000000 in ?? ()
(kgdb) f 7
#7  0xffffffff809a1c47 in m_copym (m=0x0, off0=1500, len=1480, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:542

542                     if (off < m->m_len)
(kgdb) list
537             MBUF_CHECKSLEEP(wait);
538             if (off == 0 && m->m_flags & M_PKTHDR)
539                     copyhdr = 1;
540             while (off > 0) {
541                     KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
542                     if (off < m->m_len)
543                             break;
544                     off -= m->m_len;
545                     m = m->m_next;
546             }
(kgdb)
(kgdb) p m
$8 = (struct mbuf *) 0x0
(kgdb) p *m
Cannot access memory at address 0x0
(kgdb) f 8
#8  0xffffffff80a671d0 in ip_fragment (ip=0xfffffe003deb4980, m_frag=0xffffff8211d256c8, mtu=<value optimized out>, if_hwassist_flags=<value optimized out>, sw_csum=<value optimized out>)
    at /usr/src/sys/netinet/ip_output.c:813
813                     m->m_next = m_copym(m0, off, len, M_DONTWAIT);
(kgdb) p m
$1 = <value optimized out>
(kgdb) p *m
$2 = {m_hdr = {mh_next = 0x4a3b4002c045, mh_nextpkt = 0x3cfd10ac23dd0137, mh_data = 0x69a5040306eb3bd4 <Address 0x69a5040306eb3bd4 out of bounds>, mh_len = -939196416, mh_flags = -603652027,
    mh_type = 28687, pad = "@\000y\021ĘÁ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xdd9ebfc306eb3bd4, header = 0xf21bc8059869fea5, len = 1671299073, flowid = 4040902335, csum_flags = -802142021,
        csum_data = 4096, tso_segsz = 24314, PH_vt = {vt_vtag = 36678, vt_nrecs = 36678}, tags = {slh_first = 0x20746e6572726f54}}, MH_dat = {MH_ext = {
          ext_buf = 0x6c6f636f746f7270 <Address 0x6c6f636f746f7270 out of bounds>, ext_free = 0x500100000000000, ext_arg1 = 0x7631f35ad0ef60c0, ext_arg2 = 0x22d4d367a5c84854,
          ext_size = 2443806180, ref_cnt = 0x0, ext_type = 0},
        MH_databuf = "protocol\000\000\000\000\000\020\000\005Ŕ`ďĐZó1vTHČĽgÓÔ\"ä\205Š\221-yu3", '\0' <repeats 16 times>, "`Jë=\000ţ˙˙ \000\000\000\002\000\000\000\001", '\0' <repeats 23 times>, " ", '\0' <repeats 19 times>, "´akś\000\000\000\000\000\000\000\000\000\000L^\f$4ă\a˘\000\000\002\000\000\000\000\000\\\000˙˙˙˙\000\000ÔŽRÓhË\000\001\\$ç-\b"}},
    M_databuf = "Ô;ë\006Ăż\236ÝĽţi\230\005Č\033ň\001\000\236cżFŰđťH0Đ\000\020\000\000ú^F\217\023BitTorrent protocol\000\000\000\000\000\020\000\005Ŕ`ďĐZó1vTHČĽgÓÔ\"ä\205Š\221-yu3", '\0' <repeats 16 times>, "`Jë=\000ţ˙˙ \000\000\000\002\000\000\000\001", '\0' <repeats 23 times>, " ", '\0' <repeats 19 times>, "´akś\000\000\000\000\000\000\000\000\000\000L^\f$4ă\a˘\000\000\002\000\000\000\000\000\\\000˙˙˙˙\000\000ÔŽRÓhË\000\001\\$ç-\b"}}
(kgdb) p *m0
$3 = {m_hdr = {mh_next = 0xfffffe004a464d00, mh_nextpkt = 0x0, mh_data = 0xfffffe003deb4980 "EŔ\002@;J", mh_len = 60, mh_flags = 2, mh_type = 1, pad = "\000\000\000\000\000"}, M_dat = {MH = {
      MH_pkthdr = {rcvif = 0xfffffe00032f4000, header = 0x0, len = 576, flowid = 0, csum_flags = 0, csum_data = 0, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first = 0x0}},
      MH_dat = {MH_ext = {ext_buf = 0x2ce7245c01000000 <Address 0x2ce7245c01000000 out of bounds>, ext_free = 0x200002207, ext_arg1 = 0xffff000002520000, ext_arg2 = 0xca68d352aed40000,
          ext_size = 604790348, ref_cnt = 0x4a3b4002c045, ext_type = 601686327},
        MH_databuf = "\000\000\000\001\\$ç,\a\"\000\000\002\000\000\000\000\000R\002\000\000˙˙\000\000ÔŽRÓhĘL^\f$4Ţ\b\000EŔ\002@;J\000\0007\001Ý#Ź\020ý<Ô;ë\006\003\004Ľi\000\000\005ČE\000\005Ü\017p@\000y\021ĘÁÔ;ë\006Ăż\236ÝĽţi\230\005Č\033ň\001\000\236cżFŰđťH0Đ\000\020\000\000ú^F\217\023BitTorrent protocol\000\000\000\000\000\020\000\005Ŕ`ďĐZó1vTHČĽgÓÔ\"ä\205Š\221-yu3"}},
    M_databuf = "\000@/\003\000ţ˙˙\000\000\000\000\000\000\000\000@\002", '\0' <repeats 33 times>, "\001\\$ç,\a\"\000\000\002\000\000\000\000\000R\002\000\000˙˙\000\000ÔŽRÓhĘL^\f$4Ţ\b\000EŔ\002@;J\000\0007\001Ý#Ź\020ý<Ô;ë\006\003\004Ľi\000\000\005ČE\000\005Ü\017p@\000y\021ĘÁÔ;ë\006Ăż\236ÝĽţi\230\005Č\033ň\001\000\236cżFŰđťH0Đ\000\020\000\000ú^F\217\023BitTorrent protocol\000\000\000\000\000\020\000\005Ŕ`ďĐZó1vTHČĽgÓÔ\"ä\205Š\221-yu3"}}
(kgdb) p *off
Cannot access memory at address 0x5dc
(kgdb) p off
$4 = 1500
(kgdb) p len
$5 = 1480
(kgdb) list
808                             len = ip->ip_len - off;
809                             m->m_flags |= M_LASTFRAG;
810                     } else
811                             mhip->ip_off |= IP_MF;
812                     mhip->ip_len = htons((u_short)(len + mhlen));
813                     m->m_next = m_copym(m0, off, len, M_DONTWAIT);
814                     if (m->m_next == NULL) {        /* copy failed */
815                             m_free(m);
816                             error = ENOBUFS;        /* ??? */
817                             IPSTAT_INC(ips_odropped);


Kernel is Generic with
options DEVICE_POLLING
options DUMMYNET
options IPFILTER_LOG
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL_FORWARD
options IPFIREWALL_VERBOSE
options IPFIREWALL
options IPFILTER

FreeBSD ns2.test.pl 9.2-RELEASE-p10 FreeBSD 9.2-RELEASE-p10 #2 r269086M: Mon Aug 25 11:32:01 CEST 2014     wwt@ns2.test.pl:/usr/obj/usr/src/sys/AMD64VEGA  amd64

in sysctl I found something strange

dev.bce.0.%desc: Broadcom NetXtreme II BCM5716 1000Base-T (C0)
dev.bce.1.%desc: Broadcom NetXtreme II BCM5716 1000Base-T (C0)
dev.brgphy.0.%desc: BCM5709 10/100/1000baseT PHY
dev.brgphy.1.%desc: BCM5709 10/100/1000baseT PHY

Sep  6 03:16:24 <0.2> ns2 kernel: bce0: <Broadcom NetXtreme II BCM5716 1000Base-T (C0)> mem 0xc0000000-0xc1ffffff irq 16 at device 0.0 on pci2
Sep  6 03:16:24 <0.2> ns2 kernel: miibus0: <MII bus> on bce0
Sep  6 03:16:24 <0.2> ns2 kernel: brgphy0: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus0
Sep  6 03:16:24 <0.2> ns2 kernel: brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
Sep  6 03:16:24 <0.2> ns2 kernel: bce0: Ethernet address: d4:ae:52:d3:68:ca
Sep  6 03:16:24 <0.2> ns2 kernel: bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (7.4.0); Bufs (RX:8;TX:8;PG:32); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11)
Sep  6 03:16:24 <0.2> ns2 kernel: Coal (RX:6,6,18,18; TX:20,20,80,80)
Sep  6 03:16:24 <0.2> ns2 kernel: bce1: <Broadcom NetXtreme II BCM5716 1000Base-T (C0)> mem 0xc2000000-0xc3ffffff irq 17 at device 0.1 on pci2
Sep  6 03:16:24 <0.2> ns2 kernel: miibus1: <MII bus> on bce1
Sep  6 03:16:24 <0.2> ns2 kernel: brgphy1: <BCM5709 10/100/1000baseT PHY> PHY 1 on miibus1
Sep  6 03:16:24 <0.2> ns2 kernel: brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
Sep  6 03:16:24 <0.2> ns2 kernel: bce1: Ethernet address: d4:ae:52:d3:68:cb
Sep  6 03:16:24 <0.2> ns2 kernel: bce1: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (7.4.0); Bufs (RX:8;TX:8;PG:32); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11)
Sep  6 03:16:24 <0.2> ns2 kernel: Coal (RX:6,6,18,18; TX:20,20,80,80)


the NIC is BCM5709 or BCM5716 ????



------------------------------------------------------------------------
ps -axl

  UID   PID PPID CPU PRI NI    VSZ RSS MWCHAN   STAT TT         TIME COMMAND
    0     0    0   0 -92  0      0   0 -        DLs  ??      7:32.21 [kernel]
    0     1    0   0  20  0   6276   0 wait     DLs  ??      0:00.06 [init]
    0     2    0   0 -16  0      0   0 mps_scan DL   ??      0:00.59 [mps_scan0
    0     3    0   0 -16  0      0   0 waiting_ DL   ??      0:00.00 [sctp_iter
    0     4    0   0 -16  0      0   0 ccb_scan DL   ??      0:00.00 [xpt_thrd]
    0     5    0   0 -16  0      0   0 psleep   DL   ??      0:01.37 [pagedaemo
    0     6    0   0 -16  0      0   0 psleep   DL   ??      0:00.00 [vmdaemon]
    0     7    0   0 -16  0      0   0 pollid   DL   ??      0:00.20 [idlepoll]
    0     8    0   0 155  0      0   0 pgzero   DL   ??      0:00.00 [pagezero]
    0     9    0   0 -16  0      0   0 psleep   DL   ??      0:00.61 [bufdaemon
    0    10    0   0 -16  0      0   0 audit_wo DL   ??      0:00.00 [audit]
    0    11    0   0 155  0      0   0 -        RL   ??  22270:11.50 [idle]
    0    12    0   0 -76  0      0   0 -        WL   ??     12:34.22 [intr]
    0    13    0   0  -8  0      0   0 -        DL   ??      0:15.31 [geom]
    0    14    0   0 -16  0      0   0 -        DL   ??      0:17.70 [yarrow]
    0    15    0   0 -68  0      0   0 -        DL   ??      0:15.83 [usb]
    0    16    0   0 -16  0      0   0 tzpoll   DL   ??      0:02.40 [acpi_ther
    0    17    0   0 -16  0      0   0 cooling  DL   ??      0:00.57 [acpi_cool
    0    18    0   0 -16  0      0   0 vlruwt   DL   ??      0:01.77 [vnlru]
    0    19    0   0  16  0      0   0 syncer   DL   ??      3:48.53 [syncer]
    0    20    0   0 -16  0      0   0 sdflush  DL   ??      0:03.84 [softdepfl
    0   149    1   0  52  0   9952   0 pause    Ds   ??      0:00.00 [adjkerntz
    0   912    1   0  20  0  10376   0 select   Ds   ??      0:00.06 [devd]
    0  1024    1   0  52  0  12040   0 select   Ds   ??      0:00.00 [in.tftpd]
    0  1027    1   0  52  0  12040   0 select   Ds   ??      0:00.00 [in.tftpd]
    0  1030    1   0  52  0  12040   0 select   Ds   ??      0:00.00 [in.tftpd]
    0  1033    1   0  52  0  12040   0 select   Ds   ??      0:00.00 [in.tftpd]
    0  1036    1   0  52  0  12040   0 select   Ds   ??      0:00.00 [in.tftpd]
    0  1039    1   0  52  0  12040   0 select   Ds   ??      0:00.00 [in.tftpd]
    0  1042    1   0  20  0  12040   0 select   Ds   ??      0:00.13 [in.tftpd]
    0  1084    1   0  20  0  12084   0 select   Ds   ??      0:06.65 [syslogd]
    0  1234    1   0  20  0  48424   0 connec   Ds   ??      0:00.38 [snmptrapd
    0  1238    1   0  20  0  46592   0 select   D    ??     14:19.14 [snmpd]
65534  1247    1   0  20  0  34360   0 select   Ds   ??      0:03.60 [proftpd]
    0  1342    1   0  20  0  12028   0 nanslp   Ds   ??      0:02.99 [ipmon]
    0  1670    1   0  20  0  22256   0 select   Ds   ??      0:07.34 [ntpd]
    0  1680    1   0  20  0  12080   0 select   Ds   ??      0:00.85 [timed]
  279  1712    1   0  52  0  58420   0 nanslp   Ds   ??      5:05.73 [perl]
  174  1722    1   0  20  0  20648   0 select   Ds   ??      5:45.50 [flow-capt
    0  1730    1   0  20  0  46876   0 select   Ds   ??      0:00.00 [sshd]
    0  1735    1   0  20  0 303172   0 select   Ds   ??      0:04.09 [httpd]
    0  1748    1   0  20  0  14176   0 nanslp   Ds   ??      0:00.62 [cron]
    0  1781    1   0  52  0  16244   0 select   Ds   ??      0:00.00 [inetd]
    0  1800    1   0  20  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
    0  1801    1   0  52  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
    0  1802    1   0  52  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
    0  1803    1   0  52  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
    0  1804    1   0  52  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
    0  1805    1   0  52  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
    0  1806    1   0  52  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
    0  1807    1   0  52  0  12084   0 ttyin    Ds+  ??      0:00.00 [getty]
   80  1819 1735   0  20  0 458184   0 lockf    D    ??      0:00.36 [httpd]
   80  1820 1735   0  20  0 462280   0 kqread   D    ??      0:00.36 [httpd]
   80  1821 1735   0  20  0 462280   0 lockf    D    ??      0:00.37 [httpd]
   80  2138 1735   0  20  0 470472   0 lockf    D    ??      0:00.39 [httpd]
  136 28223    1   0  20  0  28452   0 select   Ds   ??      0:00.00 [dhcpd]
   53 86359    1   0  20  0 257236   0 kqread   Ds   ??      2:50.11 [named]

------------------------------------------------------------------------
vmstat -s

1603491677 cpu context switches
 24428073 device interrupts
345311090 software interrupts
 94892352 traps
577724534 system calls
       20 kernel threads created
   307967  fork() calls
    19828 vfork() calls
        0 rfork() calls
        7 swap pager pageins
        8 swap pager pages paged in
      104 swap pager pageouts
      229 swap pager pages paged out
     3259 vnode pager pageins
    18863 vnode pager pages paged in
        1 vnode pager pageouts
        4 vnode pager pages paged out
      262 page daemon wakeups
  6644592 pages examined by the page daemon
     1124 pages reactivated
 26052283 copy-on-write faults
    81649 copy-on-write optimized faults
 43302018 zero fill pages zeroed
      383 zero fill pages prezeroed
      179 intransit blocking page faults
 86192070 total VM faults taken
        0 pages affected by kernel thread creation
129553068 pages affected by  fork()
  9986952 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
 89416642 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
   108176 pages active
   440923 pages inactive
      614 pages in VM cache
   168200 pages wired down
   278536 pages free
     4096 bytes per page
251704866 total name lookups
          cache hits (95% pos + 1% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
       USBdev    23     3K       -       23  64,128
       isadev     7     1K       -        7  128
          USB    50    69K       -       57  16,32,64,128,2048,4096
         cdev     7     2K       -        7  256
      entropy  1024    64K       -     1024  64
        sigio     1     1K       -        3  64
     filedesc    70    63K       -   454208  16,32,64,128,512,1024,2048,4096
      kdtrace   495   114K       -   328349  64,256
         kenv    83    11K       -       91  16,32,64,128
       kqueue    10    18K       -     5847  256,512,2048
    proc-args    33     3K       -   215771  16,32,64,128,256
        hhook     2     1K       -        2  256
       kbdmux     6    18K       -        6  16,512,1024,2048
      ithread    94    16K       -       94  32,128,256
          LED     8     1K       -        8  16,128
       KTRACE   100    13K       -      100  128
       linker   164    11K       -      166  16,32,64,512
        lockf    39     4K       -   147436  64,128
   loginclass     2     1K       -     4520  64
       ip6ndp     9     1K       -       11  64,128
       ip6opt     2     1K       -   585547  256
         temp   259   329K       -  1003850  16,32,64,128,256,512,1024,2048,4096
       devbuf 21634 36823K       -    21898  16,32,64,128,256,512,1024,2048,4096
     pci_link    16     2K       -       16  64,128
       module   486    61K       -      486  128
     mtx_pool     2    16K       -        2
     pmchooks     1     1K       -        1  128
    acpi_perf     4     2K       -        4  512
      subproc   350   375K       -   328110  512,4096
         proc     2    16K       -        2
      session    33     5K       -     5703  128
         pgrp    33     5K       -     5892  128
         cred    54     9K       -  5760414  64,256
      uidinfo     8     3K       -     3059  128,2048
       plimit    13     4K       -    59314  256
    sysctltmp     0     0K       -    10461  16,32,64,128,256,4096
    sysctloid  3643   186K       -     3724  16,32,64,128
       sysctl     0     0K       -   181732  16,32,64
      tidhash     1    16K       -        1
      callout     3  1536K       -        3
         umtx  1026   129K       -     1026  128
     p1003.1b     1     1K       -        1  16
         SWAP     2  1097K       -        2  64
       feeder     7     1K       -        7  32
       bus-sc    69   546K       -     2627  16,32,64,128,256,512,1024,2048,4096
          bus  1244   104K       -     5107  16,32,64,128,256,1024
      devstat     4     9K       -        4  32,4096
 eventhandler    91     8K       -       91  64,128
   CAM periph     6     2K       -      139  16,32,64,128,256
         kobj   333  1332K       -      429  4096
      Per-cpu     1     1K       -        1  32
       DEVFS1   103    52K       -      113  512
       DEVFS3   232    58K       -      736  256
       DEVFS2   103     2K       -      106  16
         rman   239    29K       -      601  16,32,128
         sbuf     0     0K       -      588  16,32,64,128,256,512,1024,2048,4096
   DEVFS_RULE    55    26K       -      263  64,512
        DEVFS    30     1K       -       87  16,128
       DEVFSP     1     1K       -     1136  64
        stack     0     0K       -        2  256
    taskqueue    19     2K       -       19  16,32,128
       Unitno    13     1K       -   680377  32,64
          iov     0     0K       - 47110818  16,64,128,256,512
       select   356    45K       -      356  128
     ioctlops     0     0K       -  1830418  16,32,64,128,256,512,1024,2048,4096
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1
          tty    19    19K       -       29  1024,2048
          pts     0     0K       -        8  256
     mbuf_tag     0     0K       -       13  32
        shmfd     1     8K       -        1
          pcb    40  1174K       -  4813651  16,32,128,1024,2048
       soname     8     1K       - 30090179  16,32,128
          acl     0     0K       -     5128  4096
     vfscache     1  2048K       -        1
   cl_savebuf     0     0K       -    27805  64
     vfs_hash     1  1024K       -        1
       vnodes     2     1K       -        3  64,256
        mount    29     2K       -      151  16,32,64,128,256
  vnodemarker     0     0K       -    32771  512
          BPF    10    10K       -     4550  128,512,4096
  ether_multi    68     4K       -       80  16,32,64
       ifaddr    88    22K       -       88  32,64,128,256,512,4096
        ifnet     7    13K       -        7  128,2048
        clone     6    24K       -        6  4096
       arpcom     3     1K       -        3  16
      lltable    30    11K       -      164  256,512
     routetbl    64     8K       -   232529  32,64,128,256,512
         igmp     6     2K       -        6  256
     in_multi     3     1K       -        3  256
    sctp_iter     0     0K       -        9  256
     sctp_ifn     3     1K       -        3  128
     sctp_ifa    14     2K       -       14  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -        9  16
    hostcache     1    28K       -        1
     syncache     1   192K       -        1
    in6_multi    35     5K       -       35  32,256
          mld     6     1K       -        6  128
     dummynet     3     3K       -        3  512,1024
  IpFw/IpAcct     6    35K       -        6  16,64,128,1024
      NFS FHA     1     2K       -        1  2048
          rpc     2     1K       -        2  256
audit_evclass   180     6K       -      219  32
      jblocks     2     1K       -        2  128,256
     savedino     0     0K       -   124455  256
        sbdep     0     0K       -    10840  64
      jsegdep    20     2K       -  1169681  64
         jseg    14     2K       -    86391  128
    jfreefrag     0     0K       -    78913  128
      jnewblk     1     1K       -   466754  128
      jremref     0     0K       -   311135  128
      jaddref     0     0K       -   312879  128
      freedep     0     0K       -      696  64
     freework     1     1K       -   270333  16,128
    newdirblk     0     0K       -       46  64
       dirrem     0     0K       -   311123  128
        mkdir     0     0K       -       62  128
       diradd     0     0K       -   312817  128
     freefile     0     0K       -   125112  64
     freeblks     0     0K       -   181593  256
     freefrag     2     1K       -    78913  128
     indirdep     2     1K       -    12370  128
       newblk    19   133K       -   466755  256
    bmsafemap     3     9K       -   153404  256
     inodedep     7  1027K       -   451891  512
      pagedep     1   128K       -    10309  256
  ufs_dirhash  2261   461K       -   376525  16,32,64,128,256,512,1024
    ufs_quota     1  1024K       -        1
    ufs_mount     3    37K       -        3  512,4096
    vm_pgdata     2   513K       -        2  128
      UMAHash     3   273K       -       18  512,1024,2048,4096
    pfs_nodes    21     6K       -       21  256
         GEOM    43    16K       -      290  16,32,64,128,256,512,1024,2048
          mps    15   408K       -       66  16,32,64,128,256,512,2048,4096
      memdesc     1     4K       -        1  4096
     acpiintr     1     1K       -        1  64
       acpica  4537   470K       -   348404  16,32,64,128,256,512,1024,2048,4096
     acpitask     1     8K       -        1
CAM dev queue     4     1K       -        4  128
    raid_data     0     0K       -       42  32,128,256
         UART     6     4K       -        6  16,512,1024
md_nvidia_data     0     0K       -        6  512
      acpisem    68     9K       -       68  128
       apmdev     1     1K       -        1  128
  md_sii_data     0     0K       -        6  512
   madt_table     0     0K       -        1  4096
     CAM path     7     1K       -      360  32
      CAM CCB   257   514K       -      415  2048
      scsi_cd     0     0K       -       10  16
      io_apic     1     2K       -        1  2048
      CAM DEV     5    10K       -      200  2048
      CAM XPT    25     3K       -      269  16,32,64,128,256,1024,2048
          MCA    14     2K       -       14  32,128
          msi    12     2K       -       12  128
     nexusdev     3     1K       -        3  16
      scsi_da     0     0K       -       37  16
    CAM queue    16     5K       -     1243  16,32,64,128,256,512,4096
      CAM SIM     4     1K       -        4  256
      acpidev    51     4K       -       51  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,      90,      12,      90,   0,   0
UMA Zones:              896,      0,      90,       2,      90,   0,   0
UMA Slabs:              568,      0,   19772,       3,   27978,   0,   0
UMA RCntSlabs:          568,      0,    9442,       1,    9486,   0,   0
UMA Hash:               256,      0,       1,      14,       4,   0,   0
16 Bucket:              152,      0,      81,      69,     153,   0,   0
32 Bucket:              280,      0,     121,      33,     219,   0,   0
64 Bucket:              536,      0,      97,      15,     243,  57,   0
128 Bucket:            1048,      0,    1538,       1,    3211,6494,   0
VM OBJECT:              232,      0,   85895,   12089, 9718206,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 149234,     234,    1254,  740753,   0,   0
MAP ENTRY:              120,      0,    2499,    1717,21517589,   0,   0
fakepg:                 120,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     326,       1,     326,   0,   0
16:                      16,      0,    2505,     855,75975510,   0,   0
32:                      32,      0,    2126,     904, 7566495,   0,   0
64:                      64,      0,   13118,    1386, 6327568,   0,   0
128:                    128,      0,   10986,    2093, 2508151,   0,   0
256:                    256,      0,    1170,    1020, 5278125,   0,   0
512:                    512,      0,    1516,    1151, 1179398,   0,   0
1024:                  1024,      0,      57,     175,  327013,   0,   0
2048:                  2048,      0,     295,     107,    1935,   0,   0
4096:                  4096,      0,     441,     368,  454363,   0,   0
Files:                   80,      0,     182,     763,46301882,   0,   0
TURNSTILE:              136,      0,     514,      66,     514,   0,   0
rl_entry:                40,      0,     258,     162,     258,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,      56,     235,  327815,   0,   0
THREAD:                1160,      0,     437,      76,     532,   0,   0
SLEEPQUEUE:              80,      0,     514,      95,     514,   0,   0
VMSPACE:                392,      0,      37,     263,  327797,   0,   0
cpuset:                  72,      0,      59,      91,      59,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,   16321,    1995,11013022,   0,   0
mbuf:                   256,      0,    4083,    5273,46073237,   0,   0
mbuf_cluster:          2048, 524288,   18304,     246,   25472,   0,   0
mbuf_jumbo_page:       4096, 246339,       0,     167,    3223,   0,   0
mbuf_jumbo_9k:         9216, 393216,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 262144,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       0,    3975, 5950642,   0,   0
ttyinq:                 160,      0,     120,     240,     960,   0,   0
ttyoutq:                256,      0,      64,     191,     504,   0,   0
ata_request:            328,      0,       0,       0,       0,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
VNODE:                  504,      0,  119152,   12872, 6375571,   0,   0
VNODEPOLL:              112,      0,       1,      98,       2,   0,   0
S VFS Cache:            108,      0,  118707,   14250, 4989725,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,     127,   36245, 1396870,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,      80,56323293,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,    4207,    2525,   76117,   0,   0
Mountpoints:            824,      0,       3,       9,       7,   0,   0
pipe:                   728,      0,       6,     164,  257091,   0,   0
ksiginfo:               112,      0,     279,     414,   54850,   0,   0
itimer:                 344,      0,       1,      32,       2,   0,   0
KNOTE:                  128,      0,      43,     885,80076380,   0,   0
socket:                 680, 524292,      73,     449, 5515839,   0,   0
unpcb:                  240, 524288,      14,     210,   14941,   0,   0
ipq:                     56,  16443,       0,     315,    7121,   0,   0
udp_inpcb:              392, 524290,      42,     408, 5491487,   0,   0
udpcb:                   16, 524328,      42,     966, 5491487,   0,   0
tcp_inpcb:              392, 524290,      15,     115,    8128,   0,   0
tcpcb:                  976, 524288,      15,      69,    8128,   0,   0
tcptw:                   72,  27800,       0,     300,    1971,   0,   0
syncache:               152, 102400,       0,     150,    2727,   0,   0
hostcache:              136,  15372,       1,     139,      53,   0,   0
tcpreass:                40,  32844,       0,     336,      67,   0,   0
sackhole:                32,      0,       0,     303,     101,   0,   0
sctp_ep:               1384, 524288,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     360,      13,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400032,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392, 524290,       1,      89,    1273,   0,   0
rtentry:                200,      0,      41,      54,      41,   0,   0
IPFW dynamic rule:      120,  50003,       0,       0,       0,   0,   0
selfd:                   56,      0,     427,     707,13145220,   0,   0
SWAPMETA:               288, 498290,      63,      54,    2944,   0,   0
FFS inode:              168,      0,  119118,   12728, 6375425,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,  119118,   12792, 6375413,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq20: ehci0                      507306       9394
irq23: ehci1                      845559      15658
cpu0:timer                     380742666    7050790
irq264: mps0                     1982999      36722
irq265: bce0                    10274852     190275
irq266: bce1                    10817314     200320
irq267: ahci0                         43          0
cpu1:timer                       5208976      96462
cpu3:timer                       5541586     102621
cpu2:timer                       5285149      97873
Total                          421206450    7800119

------------------------------------------------------------------------
pstat -T

182/12328 files
0M/8191M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/da0b         16776952     1568 16775384     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
             da0              cd0            pass0             cpu
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
 28.22 36676 1010.64   0.00   0  0.00   0.00   0  0.00   1  0  0  0 99

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME


------------------------------------------------------------------------
ipcs -T

msginfo:
        msgmax:        16384    (max characters in a message)
        msgmni:           40    (# of message queues)
        msgmnb:         2048    (max characters in a message queue)
        msgtql:           40    (max # of messages in system)
        msgssz:            8    (size of a message segment)
        msgseg:         2048    (# of message segments in system)

shminfo:
        shmmax:    536870912    (max shared memory segment size)
        shmmin:            1    (min shared memory segment size)
        shmmni:          192    (max number of shared memory identifiers)
        shmseg:          128    (max shared memory segments per process)
        shmall:       131072    (max amount of shared memory in pages)

seminfo:
        semmni:           50    (# of semaphore identifiers)
        semmns:          340    (# of semaphores in system)
        semmnu:          150    (# of undo structures in system)
        semmsl:          340    (max # of semaphores per id)
        semopm:          100    (max # of operations per semop call)
        semume:           50    (max # of undo entries per process)
        semusz:          632    (size in bytes of undo structure)
        semvmx:        32767    (semaphore maximum value)
        semaem:        16384    (adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
        90078 packets sent
                21335 data packets (19985262 bytes)
                579 data packets (685476 bytes) retransmitted
                40 data packets unnecessarily retransmitted
                0 resends initiated by MTU discovery
                59829 ack-only packets (0 delayed)
                0 URG only packets
                0 window probe packets
                23 window update packets
                8304 control packets
        85518 packets received
                22543 acks (for 19809531 bytes)
                2522 duplicate acks
                0 acks for unsent data
                18927 packets (12956183 bytes) received in-sequence
                105 completely duplicate packets (10885 bytes)
                2 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                67 out-of-order packets (27031 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                64 window update packets
                0 packets received after close
                2 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
                0 discarded due to memory problems
        5468 connection requests
        2085 connection accepts
        0 bad connection attempts
        0 listen queue overflows
        111 ignored RSTs in the windows
        2895 connections established (including accepts)
        8113 connections closed (including 47 drops)
                637 connections updated cached RTT on close
                637 connections updated cached RTT variance on close
                285 connections updated cached ssthresh on close
        4638 embryonic connections dropped
        12456 segments updated rtt (of 16864 attempts)
        37361 retransmit timeouts
                3 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 Connections (fin_wait_2) dropped because of timeout
        4638 keepalive timeouts
                8 keepalive probes sent
                4630 connections dropped by keepalive
        2570 correct ACK header predictions
        13487 correct data packet header predictions
        2727 syncache entries added
                150 retransmitted
                104 dupsyn
                0 dropped
                2085 completed
                0 bucket overflow
                0 cache overflow
                609 reset
                20 stale
                0 aborted
                0 badack
                15 unreach
                0 zone failures
        2727 cookies sent
        2 cookies received
        53 hostcache entries added
                0 bucket overflow
        21 SACK recovery episodes
        242 segment rexmits in SACK recovery episodes
        337559 byte rexmits in SACK recovery episodes
        515 SACK options (SACK blocks) received
        7 SACK options (SACK blocks) sent
        0 SACK scoreboard overflow
        0 packets with ECN CE bit set
        0 packets with ECN ECT(0) bit set
        0 packets with ECN ECT(1) bit set
        0 successful ECN handshakes
        0 times ECN reduced the congestion window
udp:
        13734616 datagrams received
        0 with incomplete header
        1 with bad data length field
        5 with bad checksum
        3444890 with no checksum
        737440 dropped due to no socket
        11270 broadcast/multicast datagrams undelivered
        0 dropped due to full socket buffers
        0 not for hashed pcb
        12985900 delivered
        9614624 datagrams output
        0 times multicast source filter matched
ip:
        13933147 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        14406 fragments received
        0 fragments dropped (dup or out of space)
        11 fragments dropped after timeout
        7110 packets reassembled ok
        13923406 packets for this host
        137 packets for unknown/unsupported protocol
        1676 packets forwarded (0 packets fast forwarded)
        0 packets not forwardable
        0 packets received for unknown multicast group
        0 redirects sent
        10460598 packets sent from this host
        161 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        3 datagrams with bad address in header
icmp:
        675136 calls to icmp_error
        0 errors not generated in response to an icmp message
        Output histogram:
                echo reply: 79698
                destination unreachable: 675136
        0 messages with bad code fields
        0 messages less than the minimum length
        31 messages with bad checksum
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                echo reply: 138
                destination unreachable: 18050
                routing redirect: 78
                echo: 84931
                time exceeded: 164
        79698 message responses generated
        0 invalid return addresses
        0 no return routes
igmp:
        15 messages received
        0 messages received with too few bytes
        0 messages received with wrong TTL
        0 messages received with bad checksum
        0 V1/V2 membership queries received
        0 V3 membership queries received
        0 membership queries received with invalid field(s)
        0 general queries received
        0 group queries received
        0 group-source queries received
        0 group-source queries dropped
        7 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 V3 reports received without Router Alert
        0 membership reports sent
arp:
        137 ARP requests sent
        21918 ARP replies sent
        22441 ARP requests received
        85 ARP replies received
        22526 ARP packets received
        463 total packets dropped due to no ARP entry
        134 ARP entrys timed out
        0 Duplicate IPs seen
ip6:
        0 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        0 packets for this host
        0 packets forwarded
        0 packets not forwardable
        0 redirects sent
        7 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        1166714 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        0 multicast packets which we don't join
        Mbuf statistics:
                0 one mbuf
                0 one ext mbuf
                0 two or more ext mbuf
        0 packets whose headers are not contiguous
        0 tunneling packets that can't find gif
        0 packets discarded because of too many headers
        0 failures of source address selection
        Source addresses selection rule applied:
icmp6:
        0 calls to icmp6_error
        0 errors not generated in response to an icmp6 message
        0 errors not generated because of rate limitation
        Output histogram:
                neighbor solicitation: 2
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 bad redirect messages
        0 path MTU changes
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output

------------------------------------------------------------------------
netstat -m

20404/7268/27672 mbufs in use (current/cache/total)
16309/2241/18550/524288 mbuf clusters in use (current/cache/total/max)
16321/1995 mbuf+clusters out of packet secondary zone in use (current/cache)
0/167/167/246339 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/393216 9k jumbo clusters in use (current/cache/total/max)
0/0/0/262144 16k jumbo clusters in use (current/cache/total/max)
37719K/6967K/44686K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
Comment 1 Sean Bruno freebsd_committer 2016-01-25 15:46:59 UTC
Does the submitter have the ability to test 10 release or stable?  We have *one* of these machines in the FreeBSD cluster and I don't ever remember it doing this.  However, we track current pretty aggressively.
Comment 2 Márcio Elias 2016-08-24 11:56:18 UTC
Hi, we get the same problem with same hardware on a 10.3-RELEASE. We get this in a Dell R210 II and a T210, both with the same driver loaded, and both running IPNAT to translate a lot o private IP's to Public IP's (we are a ISP).

The strange is that this error not happens on the time of more load, is random, can occur any time.

After change sysctl variables related to network tunning the crashes get less frequent, but the max uptime that we can get is about 2 days.

Below is the Sysctl variables that we change.

net.bpf.optimize_writers=1
kern.random.sys.harvest.ethernet=0
kern.random.sys.harvest.point_to_point=0
kern.random.sys.harvest.interrupt=0
net.inet.raw.maxdgram=16384
net.inet.raw.recvspace=16384
net.route.netisr_maxqlen=2048
net.inet.ip.redirect=0
kern.ipc.shmmax=2147483648

And this on the /boot/loader.conf

net.link.ifqmaxlen=2048
net.isr.maxthreads=4
net.isr.defaultqlimit=2048

Also we enable LARGE_NAT on /usr/src/sys/contrib/ipfilter/netinet/ip_nat.h.