Bug 204378 - xhci fails on Cavium ThunderX
Summary: xhci fails on Cavium ThunderX
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: usb (show other bugs)
Version: CURRENT
Hardware: arm64 Any
: --- Affects Only Me
Assignee: freebsd-usb mailing list
URL: https://reviews.freebsd.org/D4698
Keywords: ThunderX
Depends on:
Blocks: 203349
  Show dependency treegraph
 
Reported: 2015-11-08 18:56 UTC by Ed Maste
Modified: 2015-12-26 09:47 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer 2015-11-08 18:56:27 UTC
ThunderX XHCI requires MSI

I tried booting with:

hw.usb.xhci.use_polling=0
hw.usb.xhci.debug=255

and I see:

xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x868000000000-0x8680001fffff,0x868000200000-0x8680002fffff at device 16.0 on pci0^M
xhci_init: ^M
xhci_init: CAPLENGTH=0x20^M
xhci_init: RUNTIMEOFFSET=0x440^M
xhci_init: DOOROFFSET=0x480^M
xhci_init: xHCI version = 0x0100^M
xhci_init: HCS0 = 0x0220f665^M
xhci0: 64 bytes context size, 64-bit DMA^M
xhci_init: Max slots: 64^M
xhci_init: HCS2=0x0c0000f1^M
xhci_init: Max scratch: 1^M
xhci0: Could not allocate IRQ^M
xhci_halt_controller: ^M
device_attach: xhci0 attach returned 6^M
xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x869000000000-0x8690001fffff,0x869000200000-0x8690002fffff at device 17.0 on pci0^M
xhci_init: ^M
xhci_init: CAPLENGTH=0x20^M
xhci_init: RUNTIMEOFFSET=0x440^M
xhci_init: DOOROFFSET=0x480^M
xhci_init: xHCI version = 0x0100^M
xhci_init: HCS0 = 0x0220f665^M
xhci0: 64 bytes context size, 64-bit DMA^M
xhci_init: Max slots: 64^M
xhci_init: HCS2=0x0c0000f1^M
xhci_init: Max scratch: 1^M
xhci0: Could not allocate IRQ^M
xhci_halt_controller: ^M
device_attach: xhci0 attach returned 6^M

I tried booting with xhci polling:

set hw.usb.xhci.use_polling=1
set hw.usb.xhci.debug=255

and I see:

xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x868000000000-0x8680001fffff,0x868000200000-0x8680002fffff at device 16.0 on pci0^M
xhci_init: ^M
xhci_init: CAPLENGTH=0x20^M
xhci_init: RUNTIMEOFFSET=0x440^M
xhci_init: DOOROFFSET=0x480^M
xhci_init: xHCI version = 0x0100^M
xhci_init: HCS0 = 0x0220f665^M
xhci0: 64 bytes context size, 64-bit DMA^M
xhci_init: Max slots: 64^M
xhci_init: HCS2=0x0c0000f1^M
xhci_init: Max scratch: 1^M
xhci0: Could not allocate IRQ^M
xhci0: Interrupt polling at 1000Hz^M
xhci_interrupt: real interrupt (status=0x00000001)^M
xhci_interrupt: host controller halted^M
xhci_halt_controller: ^M
xhci_start_controller: ^M
xhci_start_controller: CONFIG=0x00000000 -> 0x00000040^M
xhci_start_controller: ERSTSZ=0x00000000 -> 0x00000001^M
xhci_start_controller: ERDP(0)=0x0000000007600080^M
xhci_start_controller: ERSTBA(0)=0x0000000007600000^M
xhci_start_controller: CRCR=0x0000000007600d80^M
usbus0 on xhci0^M
xhci0: usbpf: Attached^M
random: harvesting attach, 8 bytes (4 bits) from usbus0^M
random: harvesting attach, 8 bytes (4 bits) from xhci0^M
xhci1: <Cavium ThunderX USB 3.0 controller> mem 0x869000000000-0x8690001fffff,0x869000200000-0x8690002fffff at device 17.0 on pci0^M
xhci_init: ^M
xhci_init: CAPLENGTH=0x20^M
xhci_init: RUNTIMEOFFSET=0x440^M
xhci_init: DOOROFFSET=0x480^M
xhci_init: xHCI version = 0x0100^M
xhci_init: HCS0 = 0x0220f665^M
xhci1: 64 bytes context size, 64-bit DMA^M
xhci_init: Max slots: 64^M
xhci_init: HCS2=0x0c0000f1^M
xhci_init: Max scratch: 1^M
xhci1: Could not allocate IRQ^M
xhci1: Interrupt polling at 1000Hz^M
xhci_interrupt: real interrupt (status=0x00000001)^M
xhci_interrupt: host controller halted^M
xhci_halt_controller: ^M
xhci_start_controller: ^M
xhci_start_controller: CONFIG=0x00000000 -> 0x00000040^M
xhci_start_controller: ERSTSZ=0x00000000 -> 0x00000001^M
xhci_start_controller: ERDP(0)=0x00000000066cd080^M
xhci_start_controller: ERSTBA(0)=0x00000000066cd000^M
xhci_start_controller: CRCR=0x00000000066cdd80^M
usbus1 on xhci1^M
xhci1: usbpf: Attached^M
random: harvesting attach, 8 bytes (4 bits) from usbus1^M
random: harvesting attach, 8 bytes (4 bits) from xhci1^M

and then every few seconds:

xhci_set_hw_power: 
xhci_set_hw_power: 
xhci_set_hw_power: 
xhci_set_hw_power: 
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001
xhci_roothub_exec: UR_GET_STATUS i=1
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001
xhci_roothub_exec: port status=0x000002a0
xhci_roothub_exec: UR_GET_STATUS i=1
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002
xhci_roothub_exec: port status=0x000002a0
xhci_roothub_exec: UR_GET_STATUS i=2
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002
xhci_roothub_exec: port status=0x000002a0
xhci_roothub_exec: UR_GET_STATUS i=2
xhci_roothub_exec: UR_GET_STATUS i=1
xhci_roothub_exec: port status=0x000002a0
xhci_roothub_exec: port status=0x000002a0
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002
xhci_roothub_exec: UR_GET_STATUS i=1
xhci_roothub_exec: UR_GET_STATUS i=2
xhci_roothub_exec: port status=0x000002a0
xhci_roothub_exec: port status=0x000002a0
xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002
xhci_roothub_exec: UR_GET_STATUS i=2
xhci_roothub_exec: port status=0x000002a0



pciconf -lvc shows:

none1@pci0:0:16:0:      class=0x0c0330 card=0xa11b177d chip=0xa01b177d rev=0x01 hdr=0x00
    vendor     = 'Cavium, Inc.'
    device     = 'THUNDERX xHCI USB Controller'
    class      = serial bus
    subclass   = USB
    cap 10[70] = PCI-Express 2 endpoint max data 128(128) link x0(x0)
    cap 11[b0] = MSI-X supports 4 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0xf0000]
    ecap 000e[100] = ARI 1
    ecap 000b[108] = Vendor 1 ID 160
none2@pci0:0:17:0:      class=0x0c0330 card=0xa11b177d chip=0xa01b177d rev=0x01 hdr=0x00
    vendor     = 'Cavium, Inc.'
    device     = 'THUNDERX xHCI USB Controller'
    class      = serial bus
    subclass   = USB
    cap 10[70] = PCI-Express 2 endpoint max data 128(128) link x0(x0)
    cap 11[b0] = MSI-X supports 4 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0xf0000]
    ecap 000e[100] = ARI 1
    ecap 000b[108] = Vendor 1 ID 160
Comment 1 Hans Petter Selasky freebsd_committer 2015-11-09 12:41:02 UTC
Hi,

This looks like a problem in the PCI subsystem for ARM?

Or the "rid" is not correct for MSI?

Referring to sys/dev/usb/controller/xhci_pci.c.

Can you boot with bootverbose on?

--HPS
Comment 2 Ed Maste freebsd_committer 2015-11-09 16:37:06 UTC
A full boot -v log is at https://people.freebsd.org/~emaste/arm64/thunderx-verbose-boot-xhci.log
Comment 3 Hans Petter Selasky freebsd_committer 2015-11-09 17:09:13 UTC
Hi,

I see the MSIX is working with the AHCI.

in xhci_pci.c, can you try to change:

(rid != 0 ? 0 : RF_SHAREABLE)

Into simply:

RF_SHAREABLE

?
Comment 4 Ed Maste freebsd_committer 2015-11-09 17:33:56 UTC
Same behaviour with that change,

xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x968000000000-0x9680001fffff,0x
968000200000-0x9680002fffff at device 16.0 on pci8
xhci0: 64 bytes context size, 64-bit DMA
xhci0: Could not allocate IRQ
device_attach: xhci0 attach returned 6
xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x969000000000-0x9690001fffff,0x
969000200000-0x9690002fffff at device 17.0 on pci8
xhci0: 64 bytes context size, 64-bit DMA
xhci0: Could not allocate IRQ
Comment 5 Ed Maste freebsd_committer 2015-11-09 18:57:21 UTC
The failing case is (pci_do_msi is 1):

        /* MSI capability present? */
        if (cfg->msi.msi_location == 0 || !pci_do_msi)
Comment 6 Ed Maste freebsd_committer 2015-11-09 19:59:02 UTC
That is, we have only MSI support in xhci, not MSI-X

Capabilities from my x86 desktop xhci:
    cap 01[70] = powerspec 2  supports D0 D3  current D0
    cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message

Capabilities from the ThunderX:
    cap 10[70] = PCI-Express 2 endpoint max data 128(128) link x0(x0)
    cap 11[b0] = MSI-X supports 4 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0xf0000]
    ecap 000e[100] = ARI 1
    ecap 000b[108] = Vendor 1 ID 160
Comment 7 Hans Petter Selasky freebsd_committer 2015-11-09 20:39:58 UTC
If you in the XHCI pci attach routine set:

        uint8_t usemsi = 0; /* was: 1 */

--HPS
Comment 8 Hans Petter Selasky freebsd_committer 2015-11-09 20:44:11 UTC
Or set:

hw.usb.xhci.msi=0

before booting the kernel.

--HPS
Comment 9 Andrew Turner freebsd_committer 2015-12-21 00:02:21 UTC
I added minimal support for MSI-X interrupts and found with them I don't get any xhci error messages on boot. It looks like we will need MSI-X support for ThunderX.
Comment 10 commit-hook freebsd_committer 2015-12-24 09:41:14 UTC
A commit references this bug:

Author: andrew
Date: Thu Dec 24 09:40:29 UTC 2015
New revision: 292683
URL: https://svnweb.freebsd.org/changeset/base/292683

Log:
  Ads support to the xhci pci attachment to use MSI-X interrupts when
  available. As with MSI interrupts these can be disabled by setting
  hw.usb.xhci.msix to 0 in the loader.

  MSI-X interrupts are needed on some hardware, for example the Cavium
  ThunderX only supports them, and with this we don't fall back to polling.

  PR:		204378
  Reviewed by:	hselasky, jhb
  MFC after:	1 week (after r292669)
  Sponsored by:	ABT Systems Ltd
  Differential Revision:	https://reviews.freebsd.org/D4698

Changes:
  head/sys/dev/usb/controller/xhci.h
  head/sys/dev/usb/controller/xhci_pci.c