Bug 249966

Summary: USB config SX lock
Product: Base System Reporter: Bjoern A. Zeeb <bz>
Component: usbAssignee: freebsd-usb (Nobody) <usb>
Status: New ---    
Severity: Affects Only Me CC: darius, dinoex, emaste, hselasky, imp
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   

Description Bjoern A. Zeeb freebsd_committer freebsd_triage 2020-09-28 22:28:28 UTC
Given I am regularly having these "USB config SX lock" hangs I would love to track them, hence opening this PR.

Here's the first one to track (not the first one I keep seeing):

In this case a micro SD card on da3 did not show up with partitions anymore; the da1 on the same card reader still worked.  Unplugging the card reader checking on USB in order to reset it gave me this:

# usbconfig
load: 0.27  cmd: usbconfig 61812 [USB config SX lock] 1.61r 0.00u 0.00s 0% 2120k
mi_switch+0xc1 sleepq_catch_signals+0x3d1 sleepq_wait_sig+0x9 _sx_xlock_hard+0x451 usbd_enum_lock_sig+0xa1 usb_ref_device+0x221 usb_open+0x106 devfs_open+0x145 VOP_OPEN_APV+0x1c vn_open_vnode+0x1eb vn_open_cred+0x3b7 kern_openat+0x248 amd64_syscall+0x119 fast_syscall_common+0xf8

"can ^c"

xhci on a Kaby Lake system.
Neither the other USB3 not the USB-C port detect anythign new plugged in currently.  Not tried to disable enumeration yet.
Had unplugged an external mass storage on USB3/da4 yesterday and not used the still plugged-in reader since.  Was camcontrol ejected da4.
Comment 1 Daniel O'Connor 2023-04-26 05:39:18 UTC
I don't suppose you ever got to the bottom of this?
I see a similar thing with a USB HD.
Comment 2 Hans Petter Selasky freebsd_committer freebsd_triage 2023-04-26 08:06:33 UTC
This may be related to missing support for so-called USB4.

The output from:

procstat -akk | grep -i usb

May be helpful. If this is a USB disk, then it may also be a refcount leak in the CAM layer in FreeBSD.
Comment 3 Daniel O'Connor 2023-04-26 09:19:49 UTC
Yes looks like CAM is stuck:

[cain 18:48]  >sudo procstat -akk | grep -i usb

   15 100468 usb                 usbus0              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100469 usb                 usbus0              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100470 usb                 usbus0              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100471 usb                 usbus0              mi_switch+0xc2 _sleep+0x1fc cam_sim_free+0x80 umass_detach+0x13a device_detach+0x175 device_delete_child+0x15 usb_detach_device+0x16f usb_unconfigure+0x2b usb_free_device+0x11d uhub_detach+0x9b device_detach+0x175 device_delete_child+0x15 usb_detach_device+0x16f usb_unconfigure+0x2b usbd_set_config_index+0x53 usb_bus_suspend+0xa5 usb_bus_reset+0x4d usb_process+0x100
   15 100472 usb                 usbus0              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100474 usb                 usbus1              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100475 usb                 usbus1              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100476 usb                 usbus1              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100477 usb                 usbus1              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100478 usb                 usbus1              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100480 usb                 usbus2              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100481 usb                 usbus2              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100482 usb                 usbus2              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100483 usb                 usbus2              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100484 usb                 usbus2              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100488 usb                 usbus3              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100489 usb                 usbus3              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100490 usb                 usbus3              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100491 usb                 usbus3              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 100492 usb                 usbus3              mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
   15 102528 usb                 urndis0             mi_switch+0xc2 _cv_wait+0x113 usb_process+0xc1 fork_exit+0x7e fork_trampoline+0xe
Comment 4 Hans Petter Selasky freebsd_committer freebsd_triage 2023-04-26 09:29:14 UTC
Adding Warner:

Looks like there are still issues with CAM refcounts.

Daniel:

What does "uname -a" say?

--HPS
Comment 5 Daniel O'Connor 2023-04-26 09:32:58 UTC
uname -a:
FreeBSD cain.gsoft.com.au 13.1-RELEASE-p6 FreeBSD 13.1-RELEASE-p6 GENERIC amd64
Comment 6 Hans Petter Selasky freebsd_committer freebsd_triage 2023-04-26 09:56:58 UTC
Can you try a 13-stable kernel?

Is this reproducible?

--HPS
Comment 7 Warner Losh freebsd_committer freebsd_triage 2023-04-26 16:03:14 UTC
(In reply to Hans Petter Selasky from comment #4)
> Looks like there are still issues with CAM refcounts.

Not as far as I know. I've not seen any ref count leaks in CAM in a long time that weren't related to "stuttering" devices that came and went many times quickly over the course of a few seconds. So this is 

Though sleeping in cam_free_sim() suggests that somebody hasn't kept proper book somewhere...  Or that somehow we've missed a wakeup on the sim from cam_sim_release...

If this is reproducible, I have some basic debugging tools that we can use with kgdb to look at the state of CAM to see if this problem is caught by one of the checks I have.
Comment 8 Daniel O'Connor 2023-04-26 22:38:51 UTC
I don't have exact steps to reproduce it but it happens pretty reliably so I can try any debugging suggestions you have.