Bug 122928
| Summary: | [em] interface watchdog timeouts and stops receiving packets | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Neil Hoggarth <neil> |
| Component: | kern | Assignee: | jfv |
| Status: | Closed Overcome By Events | ||
| Severity: | Affects Only Me | CC: | sbruno |
| Priority: | Normal | Keywords: | IntelNetworking |
| Version: | 7.0-STABLE | ||
| Hardware: | Any | ||
| OS: | Any | ||
State Changed From-To: open->feedback Neil: Please additionally provide output of `ifconfig em0' and `sysctl hw.em' Over to maintainer(s). Responsible Changed From-To: freebsd-bugs->freebsd-net Neil: Please additionally provide output of `ifconfig em0' and `sysctl hw.em' Over to maintainer(s). On Sun, 20 Apr 2008, vwe@FreeBSD.org wrote: > Neil: Please additionally provide output of `ifconfig em0' and `sysctl hw.em' The output of "ifconfig em0" looks like this: em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:0e:0c:06:c2:3a inet 10.0.0.7 netmask 0xff000000 broadcast 10.255.255.255 media: Ethernet autoselect (1000baseTX <full-duplex>) status: active "sysctl hw.em" gives an "unknown oid" error. "sysctl dev.em" gives: dev.em.0.%desc: Intel(R) PRO/1000 Network Connection Version - 6.7.3 dev.em.0.%driver: em dev.em.0.%location: slot=9 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x100e subvendor=0x8086 subdevice=0x002e class=0x020000 dev.em.0.%parent: pci2 dev.em.0.debug: -1 dev.em.0.stats: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 For the record, the problem persists following the recent MFC of the em
driver and locking fix in delta 1.184.2.3 of if_em.c.
ifconfig em0:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0e:0c:06:c2:3a
inet 10.0.0.7 netmask 0xff000000 broadcast 10.255.255.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
sysctl dev.em:
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 6.9.5
dev.em.0.%driver: em
dev.em.0.%location: slot=9 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x100e subvendor=0x8086
subdevice=0x002e class=0x020000
dev.em.0.%parent: pci2
dev.em.0.debug: -1
dev.em.0.stats: -1
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_processing_limit: 100
State Changed From-To: feedback->open Feedback received. Responsible Changed From-To: freebsd-net->jfv Over to maintainer. This chipset is handled by lem(4) nowadays. I suspect it needs to have its watchdog handling updated, but I'm unable to test this specific chipset. If this is still a problem with 10.2r or stable/10, please reopen this issue and we'll look at porting the watchdog enhancements over to lem(4). |
I have a recently built system using an ASRock AM2NF3-VSTA motherboard and an AMD Athlon X2 BE-2400 dual-core processor, running 7-STABLE (amd64), with the GENERIC kernel. This system uses an Intel PRO/1000 MT PCI Gigabit Ethernet adaptor (with a 82540EM chip). The same adaptor was used for several years in my previous system, a uniprocessor AthlonXP system running 6-STALE (i386), where it worked without any visible issues. The Intel PCI card provides interface em0, which is connected to my home LAN (a 3Com 3C1670500 OfficeConnect Gigabit switch). The motherboard's built-in ethernet provides interface nfe0, which is connected to my ADSL router. The system uses ipf(5) for filtering incoming packets on nfe0, and ipnat(5) to NAT connections from other machines on the LAN to the public Internet. The machine frequently gets into a state where traffic no longer reliably passes through the em0 interface (devices on the LAN can no longer communicate with the machine itself or with the public Internet), and the kernel starts logging an endless succession of messages of the form: Apr 12 07:38:50 neilhoggarth-2 kernel: em0: watchdog timeout -- resetting Apr 12 07:38:50 neilhoggarth-2 kernel: em0: link state changed to DOWN Apr 12 07:38:52 neilhoggarth-2 kernel: em0: link state changed to UP Apr 12 07:38:57 neilhoggarth-2 kernel: em0: watchdog timeout -- resetting Apr 12 07:38:57 neilhoggarth-2 kernel: em0: link state changed to DOWN Apr 12 07:39:01 neilhoggarth-2 kernel: em0: link state changed to UP As far as I can tell from using tcpdump on the machine itself and a laptop connected to the internal LAN, the machine still sends packets out through the em0 interface, but does not receive incoming packets. The only way that I have found of recovering normal function is to reboot the machine. The same problem was present when I first installed 7.0-RELEASE. I have tried a variety of different things in an attempt to gather more information or find a workaround, including building a kernel with WITNESS and INVARIANTS (which didn't seem to provide any extra diagnostics), building a kernel with DEVICE_POLLING and enabling polling on the em0 interface, building a kernel without PREEMPTION, and building a kernel without the SMP option. I have also experimented with running the interface at various other speeds (all the way down to 10baseT half-duplex) rather than allowing it to autonegotiate 1000baseTX. The problem still occurs under all the circumstances that I've tried. Any suggestions on how I can try to debug this further would be appreciated. pciconf -lv: ============ hostb0@pci0:0:0:0: class=0x060000 card=0x00e11849 chip=0x00e110de rev=0xa1 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 Host/PCI Bridge' class = bridge subclass = HOST-PCI isab0@pci0:0:1:0: class=0x060100 card=0x00e01849 chip=0x00e010de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 LPC Interface Bridge' class = bridge subclass = PCI-ISA none0@pci0:0:1:1: class=0x0c0500 card=0x00e41849 chip=0x00e410de rev=0xa1 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 PCI System Management' class = serial bus subclass = SMBus ohci0@pci0:0:2:0: class=0x0c0310 card=0x00e71849 chip=0x00e710de rev=0xa1 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 OpenHCD USB Controller' class = serial bus subclass = USB ohci1@pci0:0:2:1: class=0x0c0310 card=0x00e71849 chip=0x00e710de rev=0xa1 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 OpenHCD USB Controller' class = serial bus subclass = USB ehci0@pci0:0:2:2: class=0x0c0320 card=0x00e81849 chip=0x00e810de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 Enhanced PCI to USB Controller' class = serial bus subclass = USB nfe0@pci0:0:5:0: class=0x068000 card=0x00df1849 chip=0x00df10de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'Marvell 88E1111 Network adapter' class = bridge atapci0@pci0:0:8:0: class=0x01018a card=0x00e51849 chip=0x00e510de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 Parallel ATA Controller' class = mass storage subclass = ATA atapci1@pci0:0:10:0: class=0x010185 card=0x00e31849 chip=0x00e310de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'nForce3 250 Serial ATA Controller' class = mass storage subclass = ATA pcib1@pci0:0:11:0: class=0x060400 card=0x00000000 chip=0x00e210de rev=0xa2 hdr=0x01 vendor = 'Nvidia Corp' device = 'nForce3 250 AGP Host to PCI Bridge' class = bridge subclass = PCI-PCI pcib2@pci0:0:14:0: class=0x060400 card=0x00000000 chip=0x00ed10de rev=0xa2 hdr=0x01 vendor = 'Nvidia Corp' device = 'nForce3 250 PCI-PCI Bridge' class = bridge subclass = PCI-PCI hostb1@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron HyperTransport Technology Configuration' class = bridge subclass = HOST-PCI hostb2@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Address Map' class = bridge subclass = HOST-PCI hostb3@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron DRAM Controller' class = bridge subclass = HOST-PCI hostb4@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Miscellaneous Control' class = bridge subclass = HOST-PCI vgapci0@pci0:1:0:0: class=0x030000 card=0x2062148c chip=0x59611002 rev=0x01 hdr=0x00 vendor = 'ATI Technologies Inc' device = 'RV280 ATI RADEON 9200 se agp' class = display subclass = VGA vgapci1@pci0:1:0:1: class=0x038000 card=0x2063148c chip=0x59411002 rev=0x01 hdr=0x00 vendor = 'ATI Technologies Inc' device = 'RV280 ATI Radeon 9200 - Secondary' class = display atapci2@pci0:2:8:0: class=0x010400 card=0x00011103 chip=0x00081103 rev=0x07 hdr=0x00 vendor = 'Triones Technologies Inc. (HighPoint)' device = 'HPT374 Rocket 154x/1640, RocketRAID 154x/1640 RAID EIDE Controller' class = mass storage subclass = RAID atapci3@pci0:2:8:1: class=0x010400 card=0x00011103 chip=0x00081103 rev=0x07 hdr=0x00 vendor = 'Triones Technologies Inc. (HighPoint)' device = 'HPT374 Rocket 154x/1640, RocketRAID 154x/1640 RAID EIDE Controller' class = mass storage subclass = RAID em0@pci0:2:9:0: class=0x020000 card=0x002e8086 chip=0x100e8086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet vmstat -i: ========== interrupt total rate irq14: ata0 525 0 irq15: ata1 246 0 irq16: drm0 172525 48 irq17: atapci2+ 132203 36 irq18: em0 72904 20 irq20: nfe0 ohci0 13790 3 irq21: ohci1+ 1 0 irq22: ehci0 1149 0 cpu0: timer 7146437 1999 cpu1: timer 7138376 1997 Total 14678156 4106 dmesg: ====== Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #0: Thu Apr 3 00:57:21 BST 2008 root@neilhoggarth-2.dsl.easynet.co.uk:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) X2 Dual Core Processor BE-2400 (2331.98-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x60fb2 Stepping = 2 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x2001<SSE3,CX16> AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!> AMD Features2=0x11f<LAHF,CMP,SVM,ExtAPIC,CR8,Prefetch> Cores per package: 2 usable memory = 8575258624 (8178 MB) avail memory = 8289517568 (7905 MB) ACPI APIC Table: <A_M_I OEMAPIC > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 2 ioapic0 <Version 1.1> irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: <A_M_I OEMRSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d5f00000 (3) failed Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: <ACPI CPU> on acpi0 powernow0: <PowerNow! K8> on cpu0 device_attach: powernow0 attach returned 6 cpu1: <ACPI CPU> on acpi0 powernow1: <PowerNow! K8> on cpu1 device_attach: powernow1 attach returned 6 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 agp0: <NVIDIA nForce3-250 AGP Controller> on hostb0 isab0: <PCI-ISA bridge> at device 1.0 on pci0 isa0: <ISA bus> on isab0 pci0: <serial bus, SMBus> at device 1.1 (no driver attached) ohci0: <OHCI (generic) USB controller> mem 0xfebff000-0xfebfffff irq 20 at device 2.0 on pci0 ohci0: [GIANT-LOCKED] ohci0: [ITHREAD] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: <OHCI (generic) USB controller> on ohci0 usb0: USB revision 1.0 uhub0: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 uhub0: 4 ports with 4 removable, self powered ohci1: <OHCI (generic) USB controller> mem 0xfebfe000-0xfebfefff irq 21 at device 2.1 on pci0 ohci1: [GIANT-LOCKED] ohci1: [ITHREAD] usb1: OHCI version 1.0, legacy support usb1: SMM does not respond, resetting usb1: <OHCI (generic) USB controller> on ohci1 usb1: USB revision 1.0 uhub1: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 uhub1: 4 ports with 4 removable, self powered ehci0: <NVIDIA nForce3 250 USB 2.0 controller> mem 0xfebfdc00-0xfebfdcff irq 22 at device 2.2 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb2: EHCI version 1.0 usb2: companion controllers, 4 ports each: usb0 usb1 usb2: <NVIDIA nForce3 250 USB 2.0 controller> on ehci0 usb2: USB revision 2.0 uhub2: <nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb2 uhub2: 8 ports with 8 removable, self powered uhub3: <vendor 0x0409 product 0x0059, class 9/0, rev 2.00/1.00, addr 2> on uhub2 uhub3: single transaction translator uhub3: 4 ports with 4 removable, self powered uscanner0: <EPSON EPSON Scanner 010F, class 255/0, rev 1.10/1.00, addr 3> on uhub3 uhid0: <Mega World USB 2-Axis 8-Button Gamepad, class 0/0, rev 1.10/1.01, addr 4> on uhub3 ukbd0: <vendor 0x1241 USB Keyboard, class 0/0, rev 1.10/2.80, addr 5> on uhub3 kbd2 at ukbd0 uhid1: <vendor 0x1241 USB Keyboard, class 0/0, rev 1.10/2.80, addr 5> on uhub3 nfe0: <NVIDIA nForce3 MCP7 Networking Adapter> port 0xec00-0xec07 mem 0xfebfc000-0xfebfcfff irq 20 at device 5.0 on pci0 miibus0: <MII bus> on nfe0 rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto nfe0: Ethernet address: 00:19:66:47:21:9d nfe0: [FILTER] atapci0: <nVidia nForce3 Pro UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 8.0 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] atapci1: <nVidia nForce3 Pro SATA150 controller> port 0xf80-0xf87,0xf00-0xf03,0xe80-0xe87,0xe00-0xe03,0xe000-0xe00f,0xd800-0xd87f irq 21 at device 10.0 on pci0 atapci1: [ITHREAD] ata2: <ATA channel 0> on atapci1 ata2: [ITHREAD] ata3: <ATA channel 1> on atapci1 ata3: [ITHREAD] pcib1: <ACPI PCI-PCI bridge> at device 11.0 on pci0 pci1: <ACPI PCI bus> on pcib1 vgapci0: <VGA-compatible display> port 0xa000-0xa0ff mem 0xe8000000-0xefffffff,0xfe9f0000-0xfe9fffff irq 16 at device 0.0 on pci1 vgapci1: <VGA-compatible display> mem 0xe0000000-0xe7ffffff,0xfe9e0000-0xfe9effff at device 0.1 on pci1 pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0 pci2: <ACPI PCI bus> on pcib2 atapci2: <HighPoint HPT374 (channel 0+1) UDMA133 controller> port 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc000-0xc0ff irq 17 at device 8.0 on pci2 atapci2: [ITHREAD] ata4: <ATA channel 0> on atapci2 ata4: [ITHREAD] ata5: <ATA channel 1> on atapci2 ata5: [ITHREAD] atapci3: <HighPoint HPT374 (channel 2+3) UDMA133 controller> port 0xc400-0xc407,0xbc00-0xbc03,0xb880-0xb887,0xb800-0xb803,0xb400-0xb4ff irq 17 at device 8.1 on pci2 atapci3: [ITHREAD] ata6: <ATA channel 0> on atapci3 ata6: [ITHREAD] ata7: <ATA channel 1> on atapci3 ata7: [ITHREAD] em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0xb080-0xb0bf mem 0xfeac0000-0xfeadffff,0xfeaa0000-0xfeabffff irq 18 at device 9.0 on pci2 em0: Ethernet address: 00:0e:0c:06:c2:3a em0: [FILTER] acpi_button0: <Power Button> on acpi0 sio0: configured irq 3 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 3 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <Generic IRDA-compatible device> port 0x2f8-0x2ff irq 3 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: <Parallel port bus> on ppc0 ppbus0: [ITHREAD] plip0: <PLIP network interface> on ppbus0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sio1: configured irq 4 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 4 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 on acpi0 sio1: type 16550A sio1: [FILTER] orm0: <ISA Option ROMs> at iomem 0xc0000-0xccfff,0xcd000-0xce7ff on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 uaudio0: <vendor 0x0ccd USB Audio, class 0/0, rev 1.10/1.00, addr 2> on uhub0 uaudio0: audio rev 1.00 pcm0: <USB Audio> on uaudio0 uhid2: <vendor 0x0ccd USB Audio, class 0/0, rev 1.10/1.00, addr 2> on uhub0 WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounters tick every 1.000 msec ad0: DMA limited to UDMA33, device found non-ATA66 cable ad0: FAILURE - SET_MULTI status=51<READY,DSC,ERROR> error=4<ABORTED> ad0: 976MB <TRANSCEND 2.0> at ata0-master UDMA33 ZFS filesystem version 6 ZFS storage pool version 6 acd0: DVDR <PIONEER DVD-RW DVR-109/1.40> at ata1-master UDMA66 ad8: 715404MB <Seagate ST3750640AS 3.AAK> at ata4-master UDMA133 ad10: 715404MB <Seagate ST3750640AS 3.AAK> at ata5-master UDMA133 acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x48 0x00 0x01 acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x48 0x00 0x01 SMP: AP CPU #1 Launched! cd0 at ata1 bus 0 target 0 lun 0 cd0: <PIONEER DVD-RW DVR-109 1.40> Removable CD-ROM SCSI-0 device cd0: 66.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present Trying to mount root from zfs:newtank IP Filter: v4.1.28 initialized. Default = pass all, Logging = enabled em0: link state changed to UP How-To-Repeat: The problem happens intermittently, typically with a frequency on the order of 2 or 3 times a week, and I have not found a way to repeatably trigger it.