ThunderX XHCI requires MSI I tried booting with: hw.usb.xhci.use_polling=0 hw.usb.xhci.debug=255 and I see: xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x868000000000-0x8680001fffff,0x868000200000-0x8680002fffff at device 16.0 on pci0^M xhci_init: ^M xhci_init: CAPLENGTH=0x20^M xhci_init: RUNTIMEOFFSET=0x440^M xhci_init: DOOROFFSET=0x480^M xhci_init: xHCI version = 0x0100^M xhci_init: HCS0 = 0x0220f665^M xhci0: 64 bytes context size, 64-bit DMA^M xhci_init: Max slots: 64^M xhci_init: HCS2=0x0c0000f1^M xhci_init: Max scratch: 1^M xhci0: Could not allocate IRQ^M xhci_halt_controller: ^M device_attach: xhci0 attach returned 6^M xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x869000000000-0x8690001fffff,0x869000200000-0x8690002fffff at device 17.0 on pci0^M xhci_init: ^M xhci_init: CAPLENGTH=0x20^M xhci_init: RUNTIMEOFFSET=0x440^M xhci_init: DOOROFFSET=0x480^M xhci_init: xHCI version = 0x0100^M xhci_init: HCS0 = 0x0220f665^M xhci0: 64 bytes context size, 64-bit DMA^M xhci_init: Max slots: 64^M xhci_init: HCS2=0x0c0000f1^M xhci_init: Max scratch: 1^M xhci0: Could not allocate IRQ^M xhci_halt_controller: ^M device_attach: xhci0 attach returned 6^M I tried booting with xhci polling: set hw.usb.xhci.use_polling=1 set hw.usb.xhci.debug=255 and I see: xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x868000000000-0x8680001fffff,0x868000200000-0x8680002fffff at device 16.0 on pci0^M xhci_init: ^M xhci_init: CAPLENGTH=0x20^M xhci_init: RUNTIMEOFFSET=0x440^M xhci_init: DOOROFFSET=0x480^M xhci_init: xHCI version = 0x0100^M xhci_init: HCS0 = 0x0220f665^M xhci0: 64 bytes context size, 64-bit DMA^M xhci_init: Max slots: 64^M xhci_init: HCS2=0x0c0000f1^M xhci_init: Max scratch: 1^M xhci0: Could not allocate IRQ^M xhci0: Interrupt polling at 1000Hz^M xhci_interrupt: real interrupt (status=0x00000001)^M xhci_interrupt: host controller halted^M xhci_halt_controller: ^M xhci_start_controller: ^M xhci_start_controller: CONFIG=0x00000000 -> 0x00000040^M xhci_start_controller: ERSTSZ=0x00000000 -> 0x00000001^M xhci_start_controller: ERDP(0)=0x0000000007600080^M xhci_start_controller: ERSTBA(0)=0x0000000007600000^M xhci_start_controller: CRCR=0x0000000007600d80^M usbus0 on xhci0^M xhci0: usbpf: Attached^M random: harvesting attach, 8 bytes (4 bits) from usbus0^M random: harvesting attach, 8 bytes (4 bits) from xhci0^M xhci1: <Cavium ThunderX USB 3.0 controller> mem 0x869000000000-0x8690001fffff,0x869000200000-0x8690002fffff at device 17.0 on pci0^M xhci_init: ^M xhci_init: CAPLENGTH=0x20^M xhci_init: RUNTIMEOFFSET=0x440^M xhci_init: DOOROFFSET=0x480^M xhci_init: xHCI version = 0x0100^M xhci_init: HCS0 = 0x0220f665^M xhci1: 64 bytes context size, 64-bit DMA^M xhci_init: Max slots: 64^M xhci_init: HCS2=0x0c0000f1^M xhci_init: Max scratch: 1^M xhci1: Could not allocate IRQ^M xhci1: Interrupt polling at 1000Hz^M xhci_interrupt: real interrupt (status=0x00000001)^M xhci_interrupt: host controller halted^M xhci_halt_controller: ^M xhci_start_controller: ^M xhci_start_controller: CONFIG=0x00000000 -> 0x00000040^M xhci_start_controller: ERSTSZ=0x00000000 -> 0x00000001^M xhci_start_controller: ERDP(0)=0x00000000066cd080^M xhci_start_controller: ERSTBA(0)=0x00000000066cd000^M xhci_start_controller: CRCR=0x00000000066cdd80^M usbus1 on xhci1^M xhci1: usbpf: Attached^M random: harvesting attach, 8 bytes (4 bits) from usbus1^M random: harvesting attach, 8 bytes (4 bits) from xhci1^M and then every few seconds: xhci_set_hw_power: xhci_set_hw_power: xhci_set_hw_power: xhci_set_hw_power: xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001 xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001 xhci_roothub_exec: UR_GET_STATUS i=1 xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001 xhci_roothub_exec: port status=0x000002a0 xhci_roothub_exec: UR_GET_STATUS i=1 xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002 xhci_roothub_exec: port status=0x000002a0 xhci_roothub_exec: UR_GET_STATUS i=2 xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002 xhci_roothub_exec: port status=0x000002a0 xhci_roothub_exec: UR_GET_STATUS i=2 xhci_roothub_exec: UR_GET_STATUS i=1 xhci_roothub_exec: port status=0x000002a0 xhci_roothub_exec: port status=0x000002a0 xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0001 xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002 xhci_roothub_exec: UR_GET_STATUS i=1 xhci_roothub_exec: UR_GET_STATUS i=2 xhci_roothub_exec: port status=0x000002a0 xhci_roothub_exec: port status=0x000002a0 xhci_roothub_exec: type=0xa3 request=0x00 wLen=0x0004 wValue=0x0000 wIndex=0x0002 xhci_roothub_exec: UR_GET_STATUS i=2 xhci_roothub_exec: port status=0x000002a0 pciconf -lvc shows: none1@pci0:0:16:0: class=0x0c0330 card=0xa11b177d chip=0xa01b177d rev=0x01 hdr=0x00 vendor = 'Cavium, Inc.' device = 'THUNDERX xHCI USB Controller' class = serial bus subclass = USB cap 10[70] = PCI-Express 2 endpoint max data 128(128) link x0(x0) cap 11[b0] = MSI-X supports 4 messages Table in map 0x20[0x0], PBA in map 0x20[0xf0000] ecap 000e[100] = ARI 1 ecap 000b[108] = Vendor 1 ID 160 none2@pci0:0:17:0: class=0x0c0330 card=0xa11b177d chip=0xa01b177d rev=0x01 hdr=0x00 vendor = 'Cavium, Inc.' device = 'THUNDERX xHCI USB Controller' class = serial bus subclass = USB cap 10[70] = PCI-Express 2 endpoint max data 128(128) link x0(x0) cap 11[b0] = MSI-X supports 4 messages Table in map 0x20[0x0], PBA in map 0x20[0xf0000] ecap 000e[100] = ARI 1 ecap 000b[108] = Vendor 1 ID 160
Hi, This looks like a problem in the PCI subsystem for ARM? Or the "rid" is not correct for MSI? Referring to sys/dev/usb/controller/xhci_pci.c. Can you boot with bootverbose on? --HPS
A full boot -v log is at https://people.freebsd.org/~emaste/arm64/thunderx-verbose-boot-xhci.log
Hi, I see the MSIX is working with the AHCI. in xhci_pci.c, can you try to change: (rid != 0 ? 0 : RF_SHAREABLE) Into simply: RF_SHAREABLE ?
Same behaviour with that change, xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x968000000000-0x9680001fffff,0x 968000200000-0x9680002fffff at device 16.0 on pci8 xhci0: 64 bytes context size, 64-bit DMA xhci0: Could not allocate IRQ device_attach: xhci0 attach returned 6 xhci0: <Cavium ThunderX USB 3.0 controller> mem 0x969000000000-0x9690001fffff,0x 969000200000-0x9690002fffff at device 17.0 on pci8 xhci0: 64 bytes context size, 64-bit DMA xhci0: Could not allocate IRQ
The failing case is (pci_do_msi is 1): /* MSI capability present? */ if (cfg->msi.msi_location == 0 || !pci_do_msi)
That is, we have only MSI support in xhci, not MSI-X Capabilities from my x86 desktop xhci: cap 01[70] = powerspec 2 supports D0 D3 current D0 cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message Capabilities from the ThunderX: cap 10[70] = PCI-Express 2 endpoint max data 128(128) link x0(x0) cap 11[b0] = MSI-X supports 4 messages Table in map 0x20[0x0], PBA in map 0x20[0xf0000] ecap 000e[100] = ARI 1 ecap 000b[108] = Vendor 1 ID 160
If you in the XHCI pci attach routine set: uint8_t usemsi = 0; /* was: 1 */ --HPS
Or set: hw.usb.xhci.msi=0 before booting the kernel. --HPS
I added minimal support for MSI-X interrupts and found with them I don't get any xhci error messages on boot. It looks like we will need MSI-X support for ThunderX.
A commit references this bug: Author: andrew Date: Thu Dec 24 09:40:29 UTC 2015 New revision: 292683 URL: https://svnweb.freebsd.org/changeset/base/292683 Log: Ads support to the xhci pci attachment to use MSI-X interrupts when available. As with MSI interrupts these can be disabled by setting hw.usb.xhci.msix to 0 in the loader. MSI-X interrupts are needed on some hardware, for example the Cavium ThunderX only supports them, and with this we don't fall back to polling. PR: 204378 Reviewed by: hselasky, jhb MFC after: 1 week (after r292669) Sponsored by: ABT Systems Ltd Differential Revision: https://reviews.freebsd.org/D4698 Changes: head/sys/dev/usb/controller/xhci.h head/sys/dev/usb/controller/xhci_pci.c