Bug 185427

Summary: [igb] [panic] freebsd 8.4, 9.1 and 9.2 panic Double-Fault with intel 82576 igb driver
Product: Base System Reporter: napTu
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: Closed DUPLICATE    
Severity: Affects Only Me CC: eugen, ports, sbruno, shurd
Priority: Normal Keywords: IntelNetworking, crash
Version: Unspecified   
Hardware: Any   
OS: Any   

Description napTu 2014-01-02 22:40:00 UTC
With Intel 82576 igb driver, FreeBSD periodic (0.5-3 times per day) panics with "Double fault" message.
With Intel 82571 em driver same routers work fine.

Both routers on AMD platforms.

FreeBSD 9.1-RELEASE and 9.2-RELEASE KDB backtraces and info files:
http://icenet.net.ua/crash/core.txt.1
http://icenet.net.ua/crash/core.txt.2
http://icenet.net.ua/crash/core.txt.4
http://icenet.net.ua/crash/core.txt.5
http://icenet.net.ua/crash/core.txt.6
http://icenet.net.ua/crash/core.txt.7
http://icenet.net.ua/crash/core.txt.8
http://icenet.net.ua/crash/core.txt.10

http://icenet.net.ua/crash/info.1
http://icenet.net.ua/crash/info.2
http://icenet.net.ua/crash/info.3
http://icenet.net.ua/crash/info.4
http://icenet.net.ua/crash/info.5
http://icenet.net.ua/crash/info.6
http://icenet.net.ua/crash/info.7
http://icenet.net.ua/crash/info.8
http://icenet.net.ua/crash/info.10

FreeBSD 8.4 crash:
http://icenet.net.ua/crash/vmcore.0

# kgdb kernel /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault:
eip = 0x808fcea5
esp = 0x85af0f10
ebp = 0x85af102c
cpuid = 2; apic id = 02
panic: double fault
cpuid = 2
Uptime: 1h25m4s
Physical memory: 1773 MB
Dumping 189 MB: 174 158 142 126 110 94 78 62 46 30 14

Reading symbols from /boot/kernel/if_em.ko...Reading symbols from /boot/kernel/if_em.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_em.ko
Reading symbols from /boot/kernel/ng_eiface.ko...Reading symbols from /boot/kernel/ng_eiface.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_eiface.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_bridge.ko...Reading symbols from /boot/kernel/ng_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_bridge.ko
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/kernel/ng_netflow.ko...Reading symbols from /boot/kernel/ng_netflow.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_netflow.ko
Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ksocket.ko
Reading symbols from /boot/kernel/ng_car.ko...Reading symbols from /boot/kernel/ng_car.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_car.ko
Reading symbols from /boot/kernel/ng_ipfw.ko...Reading symbols from /boot/kernel/ng_ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ipfw.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/ng_tee.ko...Reading symbols from /boot/kernel/ng_tee.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_tee.ko
#0  doadump () at pcpu.h:244
244             __asm("movl %%fs:0,%0" : "=r" (td));
(kgdb) where
#0  doadump () at pcpu.h:244
#1  0x808c81c7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:443
#2  0x808c8478 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:634
#3  0x80bb6b5b in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:1042
#4  0x808fcea5 in kvprintf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n", func=0x808fe620 <putchar>, arg=0x85af1048,
    radix=10, ap=0x85af110c "\f") at /usr/src/sys/kern/subr_prf.c:545
#5  0x808fe24b in vprintf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n", ap=0x85af110c "\f")
    at /usr/src/sys/kern/subr_prf.c:330
#6  0x808fe2bb in printf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n") at /usr/src/sys/kern/subr_prf.c:301
#7  0x80bb7949 in trap (frame=0x85af11a0) at /usr/src/sys/i386/i386/trap.c:316
#8  0x80b9e6ac in calltrap () at /usr/src/sys/i386/i386/exception.s:168
#9  0x808fbb18 in msgbuf_addchar (mbp=0x81862fe4, c=107) at /usr/src/sys/kern/subr_msgbuf.c:121
#10 0x808fe5d9 in msglogchar (c=107, pri=-1) at /usr/src/sys/kern/subr_prf.c:884
#11 0x808fe6f2 in putchar (c=107, arg=0x85af13b0) at /usr/src/sys/kern/subr_prf.c:403
#12 0x808fcf32 in kvprintf (fmt=0x80cb1890 "ernel trap %d with interrupts disabled\n", func=0x808fe620 <putchar>, arg=0x85af13b0,
    radix=10, ap=0x85af1474 "\f") at /usr/src/sys/kern/subr_prf.c:577
#13 0x808fe24b in vprintf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n", ap=0x85af1474 "\f")
    at /usr/src/sys/kern/subr_prf.c:330
#14 0x808fe2bb in printf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n") at /usr/src/sys/kern/subr_prf.c:301
#15 0x80bb7949 in trap (frame=0x85af1508) at /usr/src/sys/i386/i386/trap.c:316
#16 0x80b9e6ac in calltrap () at /usr/src/sys/i386/i386/exception.s:168
#17 0x808fbb18 in msgbuf_addchar (mbp=0x81862fe4, c=107) at /usr/src/sys/kern/subr_msgbuf.c:121
#18 0x808fe5d9 in msglogchar (c=107, pri=-1) at /usr/src/sys/kern/subr_prf.c:884
#19 0x808fe6f2 in putchar (c=107, arg=0x85af1718) at /usr/src/sys/kern/subr_prf.c:403
#20 0x808fcf32 in kvprintf (fmt=0x80cb1890 "ernel trap %d with interrupts disabled\n", func=0x808fe620 <putchar>, arg=0x85af1718,
    radix=10, ap=0x85af17dc "\f") at /usr/src/sys/kern/subr_prf.c:577
#21 0x808fe24b in vprintf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n", ap=0x85af17dc "\f")
    at /usr/src/sys/kern/subr_prf.c:330
#22 0x808fe2bb in printf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n") at /usr/src/sys/kern/subr_prf.c:301
#23 0x80bb7949 in trap (frame=0x85af1870) at /usr/src/sys/i386/i386/trap.c:316
#24 0x80b9e6ac in calltrap () at /usr/src/sys/i386/i386/exception.s:168
#25 0x808fbb18 in msgbuf_addchar (mbp=0x81862fe4, c=107) at /usr/src/sys/kern/subr_msgbuf.c:121
#26 0x808fe5d9 in msglogchar (c=107, pri=-1) at /usr/src/sys/kern/subr_prf.c:884
---Type <return> to continue, or q <return> to quit---
#27 0x808fe6f2 in putchar (c=107, arg=0x85af1a80) at /usr/src/sys/kern/subr_prf.c:403
#28 0x808fcf32 in kvprintf (fmt=0x80cb1890 "ernel trap %d with interrupts disabled\n", func=0x808fe620 <putchar>, arg=0x85af1a80,
    radix=10, ap=0x85af1b44 "\f") at /usr/src/sys/kern/subr_prf.c:577
#29 0x808fe24b in vprintf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n", ap=0x85af1b44 "\f")
    at /usr/src/sys/kern/subr_prf.c:330
#30 0x808fe2bb in printf (fmt=0x80cb188f "kernel trap %d with interrupts disabled\n") at /usr/src/sys/kern/subr_prf.c:301
#31 0x80bb7949 in trap (frame=0x85af1bd8) at /usr/src/sys/i386/i386/trap.c:316
#32 0x80b9e6ac in calltrap () at /usr/src/sys/i386/i386/exception.s:168
#33 0x808ebb01 in sched_choose () at /usr/src/sys/kern/sched_ule.c:2314
#34 0x808cfba8 in choosethread () at /usr/src/sys/kern/kern_switch.c:159
#35 0x808ee099 in sched_switch (td=0x85d438a0, newtd=0x0, flags=264) at /usr/src/sys/kern/sched_ule.c:1903
#36 0x808d0eaa in mi_switch (flags=264, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:466
#37 0x808ecf5a in sched_idletd (dummy=0x0) at /usr/src/sys/kern/sched_ule.c:2642
#38 0x8089cf26 in fork_exit (callout=0x808ecce0 <sched_idletd>, arg=0x0, frame=0x85af1d28) at /usr/src/sys/kern/kern_fork.c:872
#39 0x80b9e724 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:275
(kgdb)


Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.4-RELEASE-p4 #6: Fri Oct 25 06:02:16 EEST 2013
    root@icenet.net.ua:/usr/obj/usr/src/sys/ICENET3 i386
gcc version 4.2.1 20070831 patched [FreeBSD]
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Phenom(tm) II X4 965 Processor (3415.35-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f43  Family = 10  Model = 4  Stepping = 3
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x837ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,NodeId>
  TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 1819262976 (1734 MB)
ACPI APIC Table: <032612 APIC0905>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
ACPI Warning: Optional field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20101013/tbfadt-655)
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <032612 XSDT0905> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of ffb80000, 80000 (3) failed
acpi0: reservation of fec10000, 20 (3) failed
acpi0: reservation of fed40000, 5000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 6fe00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xb000-0xb0ff mem 0xd0000000-0xdfffffff,0xfdff0000-0xfdffffff,0xfde00000-0xfdefffff irq 18 at device 5.0 on pci
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.9 - 8> port 0xcc00-0xcc1f mem 0xfe8c0000-0xfe8dffff,0xfe400000-0xfe7fffff,0xfe8f8000-0xfe8fb
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:1b:21:8b:40:70
igb0: [ITHREAD]
igb0: Bound queue 0 to cpu 0
igb0: [ITHREAD]
igb0: Bound queue 1 to cpu 1
igb0: [ITHREAD]
igb0: Bound queue 2 to cpu 2
igb0: [ITHREAD]
igb0: Bound queue 3 to cpu 3
igb0: [ITHREAD]
igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.9 - 8> port 0xc880-0xc89f mem 0xfe8a0000-0xfe8bffff,0xfe000000-0xfe3fffff,0xfe8f4000-0xfe8f7
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 00:1b:21:8b:40:71
igb1: [ITHREAD]
igb1: Bound queue 0 to cpu 0
igb1: [ITHREAD]
igb1: Bound queue 1 to cpu 1
igb1: [ITHREAD]
igb1: Bound queue 2 to cpu 2
igb1: [ITHREAD]
igb1: Bound queue 3 to cpu 3
igb1: [ITHREAD]
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci3: <ACPI PCI bus> on pcib3
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfceff000-0xfcefffff,0xfcef8000-0xfcefbfff irq 16 at device 0.0
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000
re0: Ethernet address: 30:85:a9:43:ef:b3
re0: [ITHREAD]
atapci0: <ATI IXP700/800 SATA300 controller> port 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f mem 0xfddffc00-0xfddfffff irq
atapci0: [ITHREAD]
atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM supported
ata2: <ATA channel> at channel 0 on atapci0
ata2: [ITHREAD]
ata3: <ATA channel> at channel 1 on atapci0
ata3: [ITHREAD]
ata4: <ATA channel> at channel 2 on atapci0
ata4: [ITHREAD]
ata5: <ATA channel> at channel 3 on atapci0
ata5: [ITHREAD]
ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfddfe000-0xfddfefff irq 16 at device 18.0 on pci0
ohci0: [ITHREAD]
usbus0 on ohci0
ohci1: <OHCI (generic) USB controller> mem 0xfddfd000-0xfddfdfff irq 16 at device 18.1 on pci0
ohci1: [ITHREAD]
usbus1 on ohci1
ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfddff800-0xfddff8ff irq 17 at device 18.2 on pci0
ehci0: [ITHREAD]
usbus2: EHCI version 1.0
usbus2 on ehci0
ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfddfc000-0xfddfcfff irq 18 at device 19.0 on pci0
ohci2: [ITHREAD]
usbus3 on ohci2
ohci3: <OHCI (generic) USB controller> mem 0xfddfb000-0xfddfbfff irq 18 at device 19.1 on pci0
ohci3: [ITHREAD]
usbus4 on ohci3
ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfddff400-0xfddff4ff irq 19 at device 19.2 on pci0
ehci1: [ITHREAD]
usbus5: EHCI version 1.0
usbus5 on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci1: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
ata0: <ATA channel> at channel 0 on atapci1
ata0: [ITHREAD]
ata1: <ATA channel> at channel 1 on atapci1
ata1: [ITHREAD]
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci4: <ACPI PCI bus> on pcib4
rl0: <RealTek 8139 10/100BaseTX> port 0xe800-0xe8ff mem 0xfebffc00-0xfebffcff irq 20 at device 5.0 on pci4
miibus1: <MII bus> on rl0
rlphy0: <RealTek internal media interface> PHY 0 on miibus1
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: Ethernet address: 00:0e:2e:d4:3c:28
rl0: [ITHREAD]
fxp0: <Intel 82558 Pro/100 Ethernet> port 0xec00-0xec1f mem 0xfcfff000-0xfcffffff,0xfea00000-0xfeafffff irq 21 at device 6.0 on pci4
miibus2: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus2
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
fxp0: Ethernet address: 00:50:8b:5e:30:f7
fxp0: [ITHREAD]
ohci4: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfddfa000-0xfddfafff irq 18 at device 20.5 on pci0
ohci4: [ITHREAD]
usbus6 on ohci4
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
acpi_hpet1: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
device_attach: acpi_hpet1 attach returned 12
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xcf000-0xcf7ff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
ipfw2 initialized, divert enabled, nat enabled, rule-based forwarding enabled, default to accept, logging disabled
DUMMYNET 0 with IPv6 initialized (100409)
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
usbus6: 12Mbps Full Speed USB v1.0
ugen0.1: <ATI> at usbus0
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen6.1: <ATI> at usbus6
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ad6: 305245MB <SAMSUNG HD321KJ CP100-12> at ata3-master UDMA100 SATA 3Gb/s
ad10: 305245MB <Hitachi HCS5C3232SLA380 ST2OA37E> at ata5-master UDMA100 SATA 3Gb/s
uhub6: 2 ports with 2 removable, self powered
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub4: 3 ports with 3 removable, self powered
uhub2: 6 ports with 6 removable, self powered
uhub5: 6 ports with 6 removable, self powered
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
hwpmc: TSC/1/64/0x20<REA> K8/4/48/0x1ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA>
Trying to mount root from ufs:/dev/ad10s2a
WARNING: / was not properly dismounted
Bump sched buckets to 64 (was 0)

How-To-Repeat: set Intel 82576 adapter to the FreeBSD box and run router
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2014-01-03 02:06:34 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-net

Over to maintainer.
Comment 2 Sean Bruno freebsd_committer freebsd_triage 2015-07-25 22:28:29 UTC
(In reply to napTu from comment #0)
crash text URL's no longer resolve.


The doublefault seems to indicate that something is happening with memory access causing the cpu's to crash.

Does this still happen on 10.1 and 10.2r beta?
Comment 3 napTu 2015-07-26 10:59:38 UTC
New url to access to the dumps is http://icenet.net.ru/ ...
Comment 4 Sean Bruno freebsd_committer freebsd_triage 2015-07-26 17:21:18 UTC
(In reply to napTu from comment #3)
Doesn't seem to be resolvable.  Couldn't access the site and crashdumps.
Comment 6 Sean Bruno freebsd_committer freebsd_triage 2016-01-07 17:39:28 UTC
This may be a 32bit specific failure case here.  We'll have to try and build up an i386 system and test.  This doesn't happen on any of my amd64 hosts.
Comment 7 Eugene Grosbein 2016-01-07 18:29:37 UTC
(In reply to napTu from comment #5)

These crashes do not seem igb-specific for me.

It seems, you run something like mpd daemon serving lots of users with NAT and Netflow processing, do you? If so, have you applied some non-default system tuning? Specifically, what is your sysctl net.isr.dispatch setting for this router?

It is generally not recommended to run busy mpd servers with 32 bit FreeBSD version due to stability problems even if a router has less that 4GB RAM.
Have you read my notes on the topic (in Russian) http://dadv.livejournal.com/137221.html ?
Comment 8 napTu 2016-01-10 20:55:52 UTC
Sean Bruno, yes, possible 32 bit specific problem. 
Now, one of hosts work with FreeBSD10.2 amd64 and I can try again igb netcard.


Eugen,

FreeBSD8.4 host use only routing, without mpd, without NAT, WITH NETFLOW.
FreeBSD9.x host use routing with NAT and NetFlow, without mpd also.

net.isr.dispatch: direct

and simple tuning:
rc.local:sysctl net.inet.ip.fw.one_pass=0
rc.local:sysctl net.inet.tcp.blackhole=2
rc.local:sysctl net.inet.udp.blackhole=1
rc.local:sysctl kern.ipc.nmbclusters=32768

loader.conf:
ipfw_load="YES"
ipfw_nat_load="YES"
ipdivert_load="YES"
libalias_load="YES"
dummynet_load="YES"

8.4 host modules:
# kldstat
Id Refs Address    Size     Name
 1   37 0x80400000 bb5f84   kernel
 2    1 0x80fb6000 50ea0    if_em.ko
 3    1 0x86490000 3000     ng_eiface.ko
 4    9 0x86493000 b000     netgraph.ko
 5    1 0x86523000 4000     ng_bridge.ko
 6    1 0x86532000 4000     ng_ether.ko
 7    1 0x86536000 7000     ng_netflow.ko
 8    1 0x8653d000 5000     ng_ksocket.ko
 9    1 0x865c2000 4000     ng_car.ko
10    1 0x865c6000 2000     ng_ipfw.ko
11    1 0x87af7000 4000     ng_socket.ko
12    1 0x87b7e000 3000     ng_tee.ko
13    1 0x93416000 1d000    nfsclient.ko
14    1 0x93433000 3000     nfslock.ko
15    1 0x93436000 2000     nfs_common.ko
16    1 0x93438000 13000    krpc.ko

9.x host modules
# kldstat
Id Refs Address            Size     Name
 1   39 0xffffffff80200000 179ddb0  kernel
 2    1 0xffffffff8199e000 2f9b00   zfs.ko
 3    2 0xffffffff81c98000 6048     opensolaris.ko
 4    4 0xffffffff81c9f000 21a60    ipfw.ko
 5    1 0xffffffff81cc1000 7a40     ipfw_nat.ko
 6    2 0xffffffff81cc9000 16f70    libalias.ko
 7    1 0xffffffff81ce0000 8578     ipdivert.ko
 8    1 0xffffffff81ce9000 19a80    dummynet.ko
 9    1 0xffffffff81e11000 7430     ng_netflow.ko
10    6 0xffffffff81e19000 b9db     netgraph.ko
11    1 0xffffffff81e25000 4ae1     ng_ksocket.ko
12    1 0xffffffff81e2a000 40fa     ng_ether.ko
13    1 0xffffffff81e2f000 3c48     ng_eiface.ko
14    1 0xffffffff81e33000 3832     ng_socket.ko
15    1 0xffffffff81e37000 17d7     ng_tee.ko
Comment 9 Eugene Grosbein 2016-01-11 08:22:51 UTC
(In reply to napTu from comment #8)

There seem to be some problems with multithreading and/or locks within 9.x kernel/ng_nat code. 82576 just adds extra parallelism as it supports MSI-X and multiple receive interrupts and increase probability of races within NETGRAPH code. 82571 supports only single-threaded receive path.

Why do you use ng_nat if you do not run mpd?

Try to switch from ng_nat to ipfw nat and eliminate netgraph to simplify code path.

Your 8.x crash link is broken (404).
Comment 10 napTu 2016-01-15 19:49:41 UTC
I install igb card to FreeBsd10.2 amd64.
Uptime is 55hours.

Crash links must be changed from host icenet.net.ua to new host name icenet.net.ru


Eugen, 
I do not use ng_nat, but use ipfw_nat.
Netgraph is used only to collect netflow data.
Comment 11 napTu 2016-01-15 19:55:51 UTC
(In reply to napTu from comment #10)
yes,
link to FreeBSD 8.4 crash:
http://icenet.net.ru/crash/vmcore.0

is broken
Comment 12 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:49:43 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 13 Eugene Grosbein freebsd_committer freebsd_triage 2018-05-28 21:19:00 UTC
i386 system, long back traces and double faults show this is duplicate of the PR 219476.

*** This bug has been marked as a duplicate of bug 219476 ***