Bug 251503

Summary: Disconnecting/reconnecting USB-C dock makes other USB ports unusable
Product: Base System Reporter: Ali Abdallah <ali.abdallah>
Component: usbAssignee: Hans Petter Selasky <hselasky>
Status: Closed FIXED    
Severity: Affects Only Me CC: grahamperrin, hselasky, scottl
Priority: ---    
Version: CURRENT   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
T495 usbconfig
none
pciconf
none
Skip stopping disabled endpoints by default none

Description Ali Abdallah 2020-12-01 10:09:53 UTC
Created attachment 220135 [details]
T495 usbconfig

I have a T495 with the Lenovo USB-C docking station.

When the dock is disconnected/reconnected (or when it looses power for an instant), other devices USB devices that are connected to the laptop are completely lost, such as the USB webcam, USB pen drive, etc...

I can work around the problem to bring back those devices by:

# devctl disable pci0:6:0:3
# devctl enable pci0:6:0:3

But obviously any previously mounted USB disk is lost

I'm attaching 3 usbconfig files:
usbconfig-inserted: original situation booting while dock inserted
usbconfig-dock-removed: when the dock is removed
usbconfig-dock-reinserted: when the dock is reinserted back, and other USB devices on xhci0 are lost.
Comment 1 Ali Abdallah 2020-12-01 10:10:19 UTC
Created attachment 220136 [details]
pciconf
Comment 2 Hans Petter Selasky freebsd_committer freebsd_triage 2020-12-01 10:12:39 UTC
Adding Scott Long.
Comment 3 Ali Abdallah 2020-12-21 13:28:05 UTC
The attachment order of usb hubs is somehow broken when the dock is unplugged and plugged again.

When the system is first booted, on xhci0@pci0:6:0:3 I have:

xhci0
 usbus1
  uhub0
   uhub3
    uhub5
     uhub6
      uhid0
      uhid1
      uaudio0
       pcm4

But when I unplug/re-plug the dock, I get 

xhci0
 usbus1
  uhub0
   uhub3
   uhub5

When all USB devices on the dock were correctly detected, uhub5 was a child of uhub3 and not of uhub0.

On system boot:

uhub5 on uhub3
uhub5: <VIA Labs, Inc. USB2.0 Hub, class 9/0, rev 2.10/d.23, addr 3> on usbus1
uhub5: MTT enabled
uhub5: 4 ports with 3 removable, self powered

When the dock is unplugged/replugged again:

uhub5: detached
uhub5 on uhub0
uhub5: <VIA Labs, Inc. USB3.1 Hub, class 9/0, rev 3.10/d.24, addr 2> on usbus1

And no devices attached to the dock are detected. 

BTW: I tried out Dragon Fly 5.8.3 since it has similar USB implementation to FreeBSD, and the issue doesn't happen, uhub5 remains child of uhub3 and uhub6 child of uhub5, with all USB devices being correctly detected when the dock is unplugged/plugged back.

Unfortunately I know little about the USB stack, could you please provide some hints on how to proceed here?
Comment 4 Ali Abdallah 2020-12-23 09:10:55 UTC
I've enabled USB_DEBUG, I'm attaching here the relevant messages when the dock is removed and inserted again

ugen1.2: <VIA Labs, Inc. USB2.0 Hub> at usbus1 (disconnected)
uhub3: at uhub0, port 1, addr 1 (disconnected)
ugen1.3: <Realtek USB-C Dock Ethernet> at usbus1 (disconnected)
ugen1.4: <VIA Labs, Inc. USB2.0 Hub> at usbus1 (disconnected)
uhub5: at uhub3, port 3, addr 3 (disconnected)
ugen1.5: <Lenovo USB2.0 Hub> at usbus1 (disconnected)
uhub6: at uhub5, port 3, addr 4 (disconnected)
ugen1.6: <Cypress Semiconductor 40AS> at usbus1 (disconnected)
uhid0: at uhub6, port 1, addr 5 (disconnected)
uhid0: detached
ugen1.7: <Lenovo ThinkPad USB-C Dock Gen2 USB Audio> at usbus1 (disconnected)
uaudio0: at uhub6, port 2, addr 6 (disconnected)
pcm4: detached
uaudio0: detached
uhid1: at uhub6, port 2, addr 6 (disconnected)
uhid1: detached
uhub6: detached
ugen1.8: <SMI Corporation USB DISK> at usbus1 (disconnected)
umass0: at uhub5, port 4, addr 7 (disconnected)
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <INTENSO USB 1100>  s/n AA04012900007619 detached
(da0:umass-sim0:0:0:0): Periph destroyed
umass0: detached
uhub5: detached
uhub3: detached

--> Attaching

ugen1.2: <VIA Labs, Inc. USB2.0 Hub> at usbus1
uhub3 on uhub0
uhub3: <VIA Labs, Inc. USB2.0 Hub, class 9/0, rev 2.10/d.24, addr 1> on usbus1
uhub3: MTT enabled
uhub3: 4 ports with 2 removable, self powered
Could not stop endpoint 3 index 1
Could not set dequeue ptr for endpoint 3 index 1
uhub_reattach_port: port 5 U1 timeout failed, error=USB_ERR_IOERROR
uhub_reattach_port: port 5 U2 timeout failed, error=USB_ERR_IOERROR
ugen1.3: <VIA Labs, Inc. USB3.1 Hub> at usbus1
uhub5 on uhub0
uhub5: <VIA Labs, Inc. USB3.1 Hub, class 9/0, rev 3.10/d.24, addr 2> on usbus1
uhub5: 4 ports with 2 removable, self powered
Could not stop endpoint 3 index 2
Could not set dequeue ptr for endpoint 3 index 2
Could not set dequeue ptr for endpoint 1 index 1
Could not set dequeue ptr for endpoint 1 index 2
Could not set dequeue ptr for endpoint 1 index 1
Could not set dequeue ptr for endpoint 1 index 2
Could not set dequeue ptr for endpoint 1 index 1
Could not set dequeue ptr for endpoint 1 index 2

As you can, they are xhci messages of dequeue failures, and endpoint stopping failures when the dock in inserted back.

BTW, on dfly bsd the same sequence of events is causing full controller reset, which is clearly bad, as USB devices attached (to the laptop) on xhci0 will vanish.
Comment 5 Ali Abdallah 2020-12-23 09:50:20 UTC
Created attachment 220836 [details]
Skip stopping disabled endpoints by default

With the attached patch, when an endpoints is in XHCI_EPCTX_0_EPSTATE_DISABLED state, xhci_cmd_stop_ep is not called anymore (it was failing on disable endpoints anyway). Magically with that, my USB devices attached to the dock are all re-discovered when dock looses power or remove/attached back.
Comment 6 Ali Abdallah 2020-12-23 09:55:23 UTC
Also according to the xhci endpoint state diagram, when in disabled state, only address device or configure endpoint are possible.
Comment 7 Hans Petter Selasky freebsd_committer freebsd_triage 2020-12-23 10:30:04 UTC
Hi,

Your patch looks good.

I'll try to get it in.

--HPS
Comment 8 commit-hook freebsd_committer freebsd_triage 2020-12-23 11:06:14 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1622a498525b4ef0d23d30a587b9a3888c3ee0d5

commit 1622a498525b4ef0d23d30a587b9a3888c3ee0d5
Author:     Hans Petter Selasky <hselasky@FreeBSD.org>
AuthorDate: 2020-12-23 10:54:42 +0000
Commit:     Hans Petter Selasky <hselasky@FreeBSD.org>
CommitDate: 2020-12-23 11:04:51 +0000

    No need to stop XHCI endpoints in disabled state.

    Some AMD XHCI implementations apparently assert a permanent
    internal failure if this happens.

    Submitted by:   ali.abdallah@suse.com
    PR:             251503
    MFC after:      1 week
    Sponsored by:   Mellanox Technologies // NVIDIA Networking

 sys/dev/usb/controller/xhci.c | 2 ++
 1 file changed, 2 insertions(+)
Comment 9 commit-hook freebsd_committer freebsd_triage 2020-12-23 11:06:15 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ddce63fcb6d0fe03a5f62fe819e04192c74f27c0

commit ddce63fcb6d0fe03a5f62fe819e04192c74f27c0
Author:     Hans Petter Selasky <hselasky@FreeBSD.org>
AuthorDate: 2020-12-23 10:37:44 +0000
Commit:     Hans Petter Selasky <hselasky@FreeBSD.org>
CommitDate: 2020-12-23 11:04:46 +0000

    Remove not needed variable initialization.
    And switch from int to bool while at it.

    Reviewed by:    melifaro@
    Differential Revision:  https://reviews.freebsd.org/D27725
    MFC after:      1 week
    Sponsored by:   Mellanox Technologies // NVIDIA Networking

 sys/net/if_ethersubr.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Comment 10 commit-hook freebsd_committer freebsd_triage 2021-01-12 16:52:22 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=92b2a4deed657a389cdbf91d92ab20ae6996989c

commit 92b2a4deed657a389cdbf91d92ab20ae6996989c
Author:     Hans Petter Selasky <hselasky@FreeBSD.org>
AuthorDate: 2020-12-23 10:54:42 +0000
Commit:     Hans Petter Selasky <hselasky@FreeBSD.org>
CommitDate: 2021-01-12 16:44:02 +0000

    MFC 1622a498525b:
    No need to stop XHCI endpoints in disabled state.

    Some AMD XHCI implementations apparently assert a permanent
    internal failure if this happens.

    Submitted by:   ali.abdallah@suse.com
    PR:             251503
    Sponsored by:   Mellanox Technologies // NVIDIA Networking

 sys/dev/usb/controller/xhci.c | 2 ++
 1 file changed, 2 insertions(+)
Comment 11 Graham Perrin freebsd_committer freebsd_triage 2021-10-31 09:30:46 UTC
Fixed?
Comment 12 commit-hook freebsd_committer freebsd_triage 2022-04-27 19:18:09 UTC
A commit in branch stable/11 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b199e55ce6d9ec938933cb86fbad04fb064910d9

commit b199e55ce6d9ec938933cb86fbad04fb064910d9
Author:     Hans Petter Selasky <hselasky@FreeBSD.org>
AuthorDate: 2020-12-23 10:54:42 +0000
Commit:     Hans Petter Selasky <hselasky@FreeBSD.org>
CommitDate: 2022-04-27 19:17:40 +0000

    No need to stop XHCI endpoints in disabled state.

    Some AMD XHCI implementations apparently assert a permanent
    internal failure if this happens.

    Submitted by:   ali.abdallah@suse.com
    PR:             251503
    Sponsored by:   Mellanox Technologies // NVIDIA Networking

    (cherry picked from commit 1622a498525b4ef0d23d30a587b9a3888c3ee0d5)

 sys/dev/usb/controller/xhci.c | 2 ++
 1 file changed, 2 insertions(+)
Comment 13 Ali Abdallah 2022-08-26 20:18:01 UTC
Closing as fixed.