Bug 179376 - xhci ehci irq storm
Summary: xhci ehci irq storm
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: usb (show other bugs)
Version: unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-usb mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-06-06 20:30 UTC by Michal Sviba
Modified: 2018-05-28 19:45 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michal Sviba 2013-06-06 20:30:00 UTC
CPU load is high because of to much interrupts.

#vmstat -i
interrupt                          total       rate
irq16: xhci0 ehci0           20060263963      14114
irq23: ehci1                     2844944          2
cpu0:timer                    1227368754        863
irq264: em0                     39704787         27
irq265: hdac0                         77          0
irq266: ahci0                    5757249          4
cpu1:timer                      30460523         21
cpu3:timer                      26554702         18
cpu2:timer                      35402259         24
Total                        21428357258      15077

#sysctl hw.usb.ehci.debug=1
#sysctl hw.usb.xhci.debug=1

#tail -f /var/log/messages
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: xhci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
Comment 1 Michal Sviba 2013-06-11 10:12:39 UTC
no device attached


CPU: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz (3392.37-MHz K8-class CPU)
   Origin = "GenuineIntel"  Id = 0x306a9  Family = 6  Model = 3a 
Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16401600512 (15641 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HPQOEM SLIC-BPC>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  2
  cpu2 (AP): APIC ID:  4
  cpu3 (AP): APIC ID:  6
ACPI Warning: FADT (revision 5) is longer than ACPI 2.0 version, 
truncating length 268 to 244 (20110527/tbfadt-320)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
ctl: CAM Target Layer loaded
cryptosoft0: <software crypto> on motherboard
acpi0: <HPQOEM SLIC-BPC> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 67, 1 (4) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 
0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
agp0: <IvyBridge desktop GT1 IG> on vgapci0
agp0: aperture size is 256M, detected 65532k stolen memory
xhci0: <Intel Panther Point USB 3.0 controller> mem 
0xf7c20000-0xf7c2ffff irq 16 at device 20.0 on pci0
xhci0: 32 byte context size.
usbus0 on xhci0
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms, UART> at device 22.3 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xf080-0xf09f mem 
0xf7c00000-0xf7c1ffff,0xf7c39000-0xf7c39fff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 6c:3b:e5:38:ca:00
ehci0: <Intel Panther Point USB 2.0 controller> mem 
0xf7c38000-0xf7c383ff irq 16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
hdac0: <Intel Panther Point HDA Controller> mem 0xf7c30000-0xf7c33fff 
irq 22 at device 27.0 on pci0
ehci1: <Intel Panther Point USB 2.0 controller> mem 
0xf7c37000-0xf7c373ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Panther Point AHCI SATA controller> port 
0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f 
mem 0xf7c36000-0xf7c367ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
driver bug: Unable to set devclass (devname: (null))
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROM> at iomem 0xce800-0xcf7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec


-- 
Michal Sviba
Comment 2 michal 2013-06-25 11:20:05 UTC
20days without restart made 10times higher irq16 rate



#uptime
12:10PM  up 35 days,  1:36, 1 user, load averages: 0.35, 0.41, 0.41

#vmstat -i
interrupt                          total       rate
irq16: xhci0 ehci0          312902807569     103275
irq23: ehci1                     6065098          2
cpu0:timer                    3038336324       1002
irq264: em0                     86937072         28
irq265: hdac0                         77          0
irq266: ahci0                   12706295          4
cpu1:timer                      88134649         29
cpu3:timer                      80433524         26
cpu2:timer                     103143958         34
Total                       316318564566     104402

-- 
Michal
Comment 3 michal_sjx 2013-06-26 09:28:08 UTC
I tried remove xhci support from kernel. But without success.
Maybe the problem isn't in xhci.

ms10~#uptime
10:13AM  up 12:46, 1 user, load averages: 0.20, 0.34, 0.32
 
 
                 ms10~#vmstat -i
interrupt                          total       rate
irq1: atkbd0                          18          0
irq16: ehci0                  9855800830     214485
irq23: ehci1                       91988          2
cpu0:timer                      51247683       1115
irq264: em0                      1095725         23
irq265: hdac0                         74          0
irq266: ahci0                     261277          5
cpu2:timer                       1566559         34
cpu3:timer                       1232863         26
cpu1:timer                       1234862         26
Total                         9912531879     215719
Comment 4 Michal Sviba 2013-06-26 09:32:46 UTC
I tried remove xhci support from kernel. But without success.
Maybe the problem isn't in xhci.

ms10~#uptime
10:13AM  up 12:46, 1 user, load averages: 0.20, 0.34, 0.32


                 ms10~#vmstat -i
interrupt                          total       rate
irq1: atkbd0                          18          0
irq16: ehci0                  9855800830     214485
irq23: ehci1                       91988          2
cpu0:timer                      51247683       1115
irq264: em0                      1095725         23
irq265: hdac0                         74          0
irq266: ahci0                     261277          5
cpu2:timer                       1566559         34
cpu3:timer                       1232863         26
cpu1:timer                       1234862         26
Total                         9912531879     215719
Comment 5 tin 2013-08-22 17:33:30 UTC
The same issue on FreeBSD 9.1-RELEASE GENERIC amd64:

# uname -rsmi
FreeBSD 9.1-RELEASE amd64 GENERIC

#sysctl hw.usb.xhci.debug=16 ; sleep 1; sysctl hw.usb.xhci.debug=0
Aug 22 19:00:00 pedpresa-NEW kernel: interrupt storm detected on "irq16:"; throttling interrupt source
Aug 22 19:00:31 pedpresa-NEW last message repeated 31 times
Aug 22 19:02:32 pedpresa-NEW last message repeated 121 times
Aug 22 19:12:33 pedpresa-NEW last message repeated 601 times
Aug 22 19:16:53 pedpresa-NEW last message repeated 260 times
Aug 22 19:16:54 pedpresa-NEW kernel: xhci_interrupt: real interrupt (sts=0x00000000, iman=0x00000002)
Aug 22 19:16:54 pedpresa-NEW last message repeated 279 times
Aug 22 19:16:54 pedpresa-NEW kernel: interrupt storm detected on "irq16:"; throttling interrupt source
Aug 22 19:16:54 pedpresa-NEW kernel: xhci_interrupt: real interrupt (sts=0x00000000, iman=0x00000002)
Aug 22 19:16:55 pedpresa-NEW last message repeated 311 times
Aug 22 19:16:55 pedpresa-NEW kernel: interrupt storm detected on "irq16:"; throttling interrupt source

# vmstat -i
interrupt                          total       rate
irq16: xhci0 ehci0            5588674976      53577
irq17: re0                       1247393         11
irq23: ehci1                      157524          1
cpu0:timer                      15492154        148
irq264: hdac0                         93          0
irq266: ahci1                     903811          8
cpu1:timer                      10026802         96
cpu7:timer                       9218769         88
cpu4:timer                       9207822         88
cpu2:timer                      10149700         97
cpu6:timer                      47883543        459
cpu5:timer                       8447886         80
cpu3:timer                      10476273        100
Total                         5711886746      54758

# top -SHb 1000 | grep "intr{irq"
   12 root     -88    -     0K   368K istorm  3 363:26  0.88% intr{irq16: xhci0 ehc}
   12 root     -92    -     0K   368K WAIT    5   0:06  0.00% intr{irq17: re0}
   12 root     -88    -     0K   368K WAIT    3   0:05  0.00% intr{irq266: ahci1}
   12 root     -88    -     0K   368K WAIT    0   0:00  0.00% intr{irq23: ehci1}
   12 root     -96    -     0K   368K WAIT    1   0:00  0.00% intr{irq264: hdac0}
   12 root     -84    -     0K   368K WAIT    0   0:00  0.00% intr{irq1: atkbd0}
   12 root     -84    -     0K   368K WAIT    0   0:00  0.00% intr{irq5: ppc1}
   12 root     -88    -     0K   368K WAIT    0   0:00  0.00% intr{irq265: ahci0}

#pciconf -lvc
xhci0@pci0:0:20:0:	class=0x0c0330 card=0x50071458 chip=0x1e318086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Panther Point USB xHCI Host Controller'
    class      = serial bus
    subclass   = USB
    cap 01[70] = powerspec 2  supports D0 D3  current D0
    cap 05[80] = MSI supports 8 messages, 64 bit 
---
ehci0@pci0:0:26:0:	class=0x0c0320 card=0x50061458 chip=0x1e2d8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Panther Point USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP


-- 
#include <regards.h>
/*    Pavel Shanaiev    */
/* Mob. +38(050)7777850 */
Comment 6 raitech 2013-10-21 13:45:34 UTC
The same irq storm occurs here when I plug a monitor in my ISPs router,
does not matter if the monitor is plugged via DVI, D-Sub (VGA) or HDMI.

Because it is under production now, I can not test on it. I am using a
custom kernel with ipfw fwd, verbosity and pass all, plus vimage enabled,
but can confirm that an installation over a new machine came up with the
same problem.

Both machines are running 9.2-RELEASE, and both are over Gigabyte
motherboards.

To reproduce, does not need to go further: just boot an installation media,
jump into shell and monitor irqs. Then, unplug the monitor from whatever
kind of connector it is connected. Wait some seconds and plug it again. Now
you got your own irq storrm!

Make shure that the test machine has USB 3.0.

Cheers,
Raimundo Santos

PS: my router was made over 9.1-RELEASE and I upgraded recently because of
a bug with routes, aliases and multiple FIBs. But I have the same issue
under 9.1-RELEASE.
Comment 7 Mark Linimon freebsd_committer freebsd_triage 2014-10-17 23:29:21 UTC
Reassign.
Comment 8 never 2014-12-08 18:50:36 UTC
Same thing with ehci on 9.3-STABLE r268744 amd64...
Comment 9 never 2014-12-08 18:53:12 UTC
$ vmstat -i
interrupt                          total       rate
irq1: atkbd0                          96          0
irq16: ehci0                  9991659232     137244
irq19: fwohci0                         2          0
irq23: ehci1                      199764          2
cpu0:timer                      19646164        269
irq265: em0                      4769531         65
irq266: hdac0                         89          0
irq267: ahci0                   26887191        369
irq268: ahci1                   14763358        202
cpu1:timer                      19904264        273
cpu4:timer                      20021623        275
cpu2:timer                      13360164        183
cpu7:timer                      18135949        249
cpu3:timer                      50290351        690
cpu5:timer                      19567099        268
cpu6:timer                      20678535        284
Total                        10219883412     140379


Dec  8 20:52:37 bafik kernel: ehci_timeout: xfer=0xffffff80053af148
Dec  8 20:52:37 bafik kernel: ehci_start_dma_delay: 
Dec  8 20:52:37 bafik kernel: ehci_start_dma_delay_second: 
Dec  8 20:52:37 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:37 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:37 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:40 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:44 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:48 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: Periodic is active
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: 
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: Async is active
Dec  8 20:52:52 bafik kernel: ehci_set_hw_power: Periodic is active
Comment 10 Hans Petter Selasky freebsd_committer 2014-12-08 21:19:35 UTC
Hi,

If you set hw.usb.ehci.debug=16

and it doesn't print a real interrupt has occurred, then it is most likely not the EHCI which is generating those interrupts. I guess something is broken in the PCI/ACPI IRQ interaction with the kernel, like how the interrupt is configured.

--HPS
Comment 11 raitech 2015-04-25 21:42:09 UTC
I've observed this behaviour within FreeBSD 10.1 RELEASE and with 10.1 STABLE and 11.0 CURRENT snapshots.

In FreeBSD 10.1 RELEASE, compiling a custom kernel just without ehci worked out the problem, now I can connect and disconnect my video monitor how many times I need.

The computer did lost its capacity of using USB devices through 2.0-only ports, but there are four 2.0-compatible USB 3.0 ports on the motherboard.

It is a GA-Z77X-D3H Gigabyte motherboard with an Core i5-3330 processor.

Thank you,
Raimundo Santos
Comment 12 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:45:29 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.