Attached a new Terra Master USB 3 array to my FreeBSD NUC running current. Attach and camcontrol work fine when in multiuser. When rebooting with DAS connected, array is not probed and XHCI controller fails to initialize devices. Subsequent attachments of any USB 3 device fails. --- Successful attach after multi-user --- Feb 12 16:57:56 alice kernel: pci5: <ACPI PCI bus> on pcib4 Feb 12 16:57:58 alice kernel: pcib7: <ACPI PCI-PCI bridge> at device 0.0 on pci5 Feb 12 16:57:58 alice kernel: pci6: <ACPI PCI bus> on pcib7 Feb 12 16:57:58 alice kernel: pcib8: <ACPI PCI-PCI bridge> at device 0.0 on pci6 Feb 12 16:57:58 alice kernel: pci7: <ACPI PCI bus> on pcib8 Feb 12 16:57:58 alice kernel: pcib9: <ACPI PCI-PCI bridge> at device 1.0 on pci6 Feb 12 16:57:58 alice kernel: pcib10: <ACPI PCI-PCI bridge> at device 2.0 on pci6 Feb 12 16:57:58 alice kernel: pci8: <ACPI PCI bus> on pcib10 Feb 12 16:57:58 alice kernel: xhci1: <XHCI (generic) USB 3.0 controller> mem 0xd9f00000-0xd9f0ffff at device 0.0 on pci8 Feb 12 16:57:58 alice kernel: xhci1: 32 bytes context size, 64-bit DMA Feb 12 16:57:58 alice kernel: usbus1 on xhci1 Feb 12 16:57:58 alice kernel: usbus1: 5.0Gbps Super Speed USB v3.0 Feb 12 16:57:58 alice kernel: ugen1.1: <0x8086 XHCI root HUB> at usbus1 Feb 12 16:57:58 alice kernel: uhub3 on usbus1 Feb 12 16:57:58 alice kernel: uhub3: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 Feb 12 16:57:58 alice kernel: uhub3: 4 ports with 4 removable, self powered Feb 12 16:58:13 alice kernel: ugen1.2: <TerraMaster TerraMaster DAS> at usbus1 Feb 12 16:58:13 alice kernel: umass0 on uhub3 Feb 12 16:58:13 alice kernel: umass0: <TerraMaster TerraMaster DAS, class 0/0, rev 3.00/3.22, addr 1> on usbus1 Feb 12 16:58:13 alice kernel: da0 at umass-sim0 bus 0 scbus3 target 0 lun 0 Feb 12 16:58:13 alice kernel: da0: <WDC WD20 03FYPS-27Y2B0 0103> Fixed Direct Access SPC-4 SCSI device Feb 12 16:58:13 alice kernel: da0: Serial Number 20200715000F Feb 12 16:58:13 alice kernel: da0: 400.000MB/s transfers Feb 12 16:58:13 alice kernel: da0: 1907729MB (3907029168 512 byte sectors) Feb 12 16:58:13 alice kernel: da0: quirks=0x2<NO_6_BYTE> Feb 12 16:58:13 alice kernel: da1 at umass-sim0 bus 0 scbus3 target 0 lun 1 Feb 12 16:58:13 alice kernel: da1: <MB2000EA MZF 0103> Fixed Direct Access SPC-4 SCSI device Feb 12 16:58:13 alice kernel: da1: Serial Number 20200715000F Feb 12 16:58:13 alice kernel: da1: 400.000MB/s transfers Feb 12 16:58:13 alice kernel: da1: 1907729MB (3907029168 512 byte sectors) Feb 12 16:58:13 alice kernel: da1: quirks=0x2<NO_6_BYTE> Feb 12 16:58:13 alice kernel: da2 at umass-sim0 bus 0 scbus3 target 0 lun 2 Feb 12 16:58:13 alice kernel: da2: <Hitachi HUA722020ALA330 0103> Fixed Direct Access SPC-4 SCSI device Feb 12 16:58:13 alice kernel: da2: Serial Number 20200715000F Feb 12 16:58:13 alice kernel: da2: 400.000MB/s transfers Feb 12 16:58:13 alice kernel: da2: 1907729MB (3907029168 512 byte sectors) Feb 12 16:58:13 alice kernel: da2: quirks=0x2<NO_6_BYTE> Feb 12 16:58:13 alice kernel: da3 at umass-sim0 bus 0 scbus3 target 0 lun 3 Feb 12 16:58:13 alice kernel: da3: <WDC WD20 03FYPS-27Y2B0 0103> Fixed Direct Access SPC-4 SCSI device Feb 12 16:58:13 alice kernel: da3: Serial Number 20200715000F Feb 12 16:58:13 alice kernel: da3: 400.000MB/s transfers Feb 12 16:58:13 alice kernel: da3: 1907729MB (3907029168 512 byte sectors) Feb 12 16:58:13 alice kernel: da3: quirks=0x2<NO_6_BYTE> Feb 12 16:58:13 alice kernel: da4 at umass-sim0 bus 0 scbus3 target 0 lun 4 Feb 12 16:58:13 alice kernel: da4: <WDC WD20 03FYPS-27Y2B0 0103> Fixed Direct Access SPC-4 SCSI device Feb 12 16:58:13 alice kernel: da4: Serial Number 20200715000F Feb 12 16:58:13 alice kernel: da4: 400.000MB/s transfers Feb 12 16:58:13 alice kernel: da4: 1907729MB (3907029168 512 byte sectors) Feb 12 16:58:13 alice kernel: da4: quirks=0x2<NO_6_BYTE> ---- Failure during boot and detach after disconnect --- Feb 12 16:49:26 alice kernel: ugen1.1: <0x8086 XHCI root HUB> at usbus1 Feb 12 16:49:26 alice kernel: ugen0.1: <0x8086 XHCI root HUB> at usbus0 Feb 12 16:49:26 alice kernel: uhub0 on usbus1 Feb 12 16:49:26 alice kernel: uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 Feb 12 16:49:26 alice kernel: uhub1 on usbus0 Feb 12 16:49:26 alice kernel: uhub1: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ... then after disconnecting the array in multiuser ... Feb 12 16:50:35 alice kernel: uhub0: detached Feb 12 16:50:35 alice kernel: ugen1.1: <0x8086 XHCI root HUB> at usbus1 (disconnected) Feb 12 16:50:35 alice kernel: unknown: at usbus1, port 1, addr 1 (disconnected) Feb 12 16:50:35 alice kernel: usbus1: detached Feb 12 16:50:36 alice kernel: xhci1: Controller reset timeout. Feb 12 16:50:36 alice kernel: xhci1: detached Feb 12 16:50:36 alice kernel: pci6: detached
Hi, Does it help to set this quirk in /boot/loader.conf: hw.usb.xhci.dcepquirk=1 --HPS
(In reply to Hans Petter Selasky from comment #1) Just tried this on 14.0-CURRENT #1 main-n253814-4f75af31a86 and I didn't see any change to behavior. If I leave the array attached, warm reboot or cold power off, fails to see any devices. Single user still works fine from the loader to attach and mount the array.
Are there any BIOS USB settings you can toggle? Could you share dmesg of failing and working case?
Created attachment 233402 [details] Full reboot with USB3 disk array connected dmesg from a verbose bootup. Shows xhci(4) starting to attach, but never completes.
Created attachment 233403 [details] Single user boot showing attach and probe success In this example, we go to single user, connect the machine and wait for probe/attach to succeed, see all devices and continue to multi-user normally.
Can you try this patch: https://cgit.freebsd.org/src/commit/?id=cda31e734925346328fd2369585ab3f6767ec225 --HPS
(In reply to Hans Petter Selasky from comment #6) Just FYI, I'm running a kernel with this patch applied and booting with a 4-disk USB3.1 enclosure running at boottime works quite well for me. In fact, it works even better than turning on the enclosure after booting, because it contains several 8TB drives which spin up very slowly, which results in almost random daX values. This, however, doesn't mean that the patch is a guaranteed fix for your case.
I updated my machine this weekend and didn't see any change to behavior. It still fails to attach when I reboot and the XHCI controller never sees the array if I disconnect and reconnect. The workaround of booting to single user, connecting the array and then continuing to multiuser still works though. % uname -v FreeBSD 14.0-CURRENT #2 main-n255024-835ee05f3c7: Sat Apr 23 10:46:21 MDT 2022 sbruno@alice:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG
There were some patches recently to speed up booting in general. Not sure if they are to blame. --HPS
sbruno@ : When the device is connected, does usbconfig say SUPER speed or HIGH speed for this device? --HPS
I can't say anything about USB devices detection side, but there are plenty of interesting things going on PCI side here. First, this port seems to be Thunderbolt, not plain USB, so when you hot-plug the USB, system actually detects PCI hot-plug of several levels of PCI hierarchy with XHCI controller at the end. A pleasant surprise is that all resource allocation succeeds in case of hot-plug, it seems BIOS through SMI or something else pre-allocated the resources before OS touches it, so everything just works. In case of boot with the USB devices already connected (contrary to what I expect) resources below the top level bridge are not allocated by BIOS and I see plenty of allocation errors. Considering the XHCI controller is detected I guess none of the errors are fatal, but I can only speculate that something may still be not right there. Sean, are there any BIOS options on the system like "fast boot" or something, that could control resource allocation for "unnecesary" devices during boot?
Does this NUC has old-style USB (not USB-C, or just non-Thunderbolt USB-C) to try?
Some thunderbolt support is here: If the USB device is thunderbolt, this is the expected behaviour. --HPS
Missing link in previous message: https://github.com/hselasky/usb4
(In reply to Hans Petter Selasky from comment #14) Oh ... I didn't understand. Should I give this patch set on the top of this tree a try?
I switched over to the github tree from Comment#14 and added "tb_load=YES" to my loader.conf ... didn't seem to resolve much, but I do see the tboltX devices being created on startup. tbolt0@pci0:4:0:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x8086 device=0x1576 subvendor=0x2222 subdevice=0x1111 vendor = 'Intel Corporation' device = 'DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]' class = bridge subclass = PCI-PCI tbolt1@pci0:5:0:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x8086 device=0x1576 subvendor=0x2222 subdevice=0x1111 vendor = 'Intel Corporation' device = 'DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]' class = bridge subclass = PCI-PCI tbolt2@pci0:5:1:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x8086 device=0x1576 subvendor=0x2222 subdevice=0x1111 vendor = 'Intel Corporation' device = 'DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]' class = bridge subclass = PCI-PCI tbolt3@pci0:5:2:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x8086 device=0x1576 subvendor=0x2222 subdevice=0x1111 vendor = 'Intel Corporation' device = 'DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]' class = bridge subclass = PCI-PCI
Created attachment 240244 [details] Probe and attach from startup Copy paste from dmesg of startup.
Created attachment 240245 [details] Minor update to -current Just my update to compile against today's current.