I'm using a bluetooth keyboard for FreeBSD 12.0-R box. I found that lot's of interrupts come from USB bluetooth dongle. $ env LC_ALL=C systat -vmstat 1 users Load 0.13 0.25 0.24 May 30 14:21 Mem usage: 42%Phy 28%Kmem Mem: KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 489720 74344 528043K 92740 4635728 count All 491424 75860 528066K 115680 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt ioflt 2072 total 151 4454 5 166 2035 53 3 cow uart0 4 zfod 2 vgapci0 16 0.2%Sys 0.2%Intr 0.0%User 0.0%Nice 99.6%Idle ozfod 13 cpu0:timer | | | | | | | | | | %ozfod 3 cpu1:timer daefr 8 cpu2:timer dtbuf 3 prcfr 14 cpu3:timer Namei Name-cache Dir-cache 212045 desvn 3 totfr 1999 xhci0 264 Calls hits % hits % 52909 numvn react 1 ahci0 265 45 45 100 41582 frevn pdwak hdac0 266 59 pdpgs 32 em0:irq0 Disks ada0 cd0 pass0 pass1 pass2 intrn KB/t 0.00 0.00 0.00 0.00 0.00 2381176 wire tps 0 0 0 1 0 153500 act MB/s 0.00 0.00 0.00 0.00 0.00 818332 inact %busy 0 0 0 0 0 252 laund 4635728 free buf When I remove the bluetooth dongle, `systat -vmstat` show as follows. 1 users Load 0.25 0.14 0.16 May 30 14:33 Mem usage: 42%Phy 28%Kmem Mem: KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 527812 75424 528176K 93956 4613172 count All 529516 76940 528199K 116896 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt ioflt 64 total 156 313 4 155 33 51 1 cow uart0 4 zfod 1 vgapci0 16 0.0%Sys 0.1%Intr 0.1%User 0.0%Nice 99.8%Idle ozfod 15 cpu0:timer | | | | | | | | | | %ozfod 5 cpu1:timer daefr 5 cpu2:timer dtbuf 8 prcfr 7 cpu3:timer Namei Name-cache Dir-cache 212045 desvn 8 totfr xhci0 264 Calls hits % hits % 52916 numvn react 1 ahci0 265 35 35 100 41591 frevn pdwak hdac0 266 67 pdpgs 30 em0:irq0 Disks ada0 cd0 pass0 pass1 pass2 intrn KB/t 0.00 0.00 0.00 0.00 0.00 2387644 wire tps 0 0 0 1 0 178324 act MB/s 0.00 0.00 0.00 0.00 0.00 810580 inact %busy 0 0 0 0 0 252 laund 4613172 free buf `dmesg` shows my USB controller as follows. xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xe1020000-0xe102ffff at dev ice 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA `usbconfig dump_device_desc` shows my bluetooth dongle as follows. ugen0.4: <vendor 0x0a12 CSR8510 A10> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x00e0 <Wireless controller> bDeviceSubClass = 0x0001 bDeviceProtocol = 0x0001 bMaxPacketSize0 = 0x0040 idVendor = 0x0a12 idProduct = 0x0001 bcdDevice = 0x8891 iManufacturer = 0x0000 <no string> iProduct = 0x0002 <CSR8510 A10> iSerialNumber = 0x0000 <no string> bNumConfigurations = 0x0001 I have tried 'sysctl hw.usb.xhci.use_polling=1', but nothing changed.
It's the same for me. I don't know if it's okay or not. But it reduces overall system performance by ~1% FreeBSD 13.0-CURRENT r367771 # dmesg -a | grep xhci xhci0: <Intel Panther Point USB 3.0 controller> mem 0xf7d00000-0xf7d0ffff irq 16 at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA xhci_interrupt: host controller halted xhci0: Port routing mask set to 0xffffffff usbus0 on xhci0 # usbconfig dump_device_desc ugen0.2: <Broadcom Corp BCM20702A0> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x00ff <Vendor specific> bDeviceSubClass = 0x0001 bDeviceProtocol = 0x0001 bMaxPacketSize0 = 0x0040 idVendor = 0x0b05 idProduct = 0x17cb bcdDevice = 0x0112 iManufacturer = 0x0001 <Broadcom Corp> iProduct = 0x0002 <BCM20702A0> # systam -vm 1 2000 xhci0 #sysctl hw.acpi.cpu.cx_lowest=C2 hw.acpi.cpu.cx_lowest: C1 -> C2 # systam -vm 1 1000 xhci0
Can you check using: usbconfig dump_stats Which USB device is responsible for the 1000 transfers per second? --HPS
(In reply to Hans Petter Selasky from comment #2) As soon as I'm plugging USB bluetooth dongle '<Broadcom Corp BCM20702A0>' (without any bluetooth-connected devices) # usbconfig dump_stats ugen0.1: <0x8086 XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA) { UE_CONTROL_OK : 0 UE_ISOCHRONOUS_OK : 0 UE_BULK_OK : 0 UE_INTERRUPT_OK : 0 UE_CONTROL_FAIL : 0 UE_ISOCHRONOUS_FAIL : 0 UE_BULK_FAIL : 0 UE_INTERRUPT_FAIL : 0 } ugen2.1: <Intel EHCI root HUB> at usbus2, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA) { UE_CONTROL_OK : 0 UE_ISOCHRONOUS_OK : 0 UE_BULK_OK : 0 UE_INTERRUPT_OK : 0 UE_CONTROL_FAIL : 0 UE_ISOCHRONOUS_FAIL : 0 UE_BULK_FAIL : 0 UE_INTERRUPT_FAIL : 0 } ugen1.1: <Intel EHCI root HUB> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA) { UE_CONTROL_OK : 0 UE_ISOCHRONOUS_OK : 0 UE_BULK_OK : 0 UE_INTERRUPT_OK : 0 UE_CONTROL_FAIL : 0 UE_ISOCHRONOUS_FAIL : 0 UE_BULK_FAIL : 0 UE_INTERRUPT_FAIL : 0 } ugen1.2: <vendor 0x8087 product 0x0024> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA) { UE_CONTROL_OK : 6084 UE_ISOCHRONOUS_OK : 0 UE_BULK_OK : 0 UE_INTERRUPT_OK : 0 UE_CONTROL_FAIL : 0 UE_ISOCHRONOUS_FAIL : 0 UE_BULK_FAIL : 0 UE_INTERRUPT_FAIL : 0 } ugen2.2: <vendor 0x8087 product 0x0024> at usbus2, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA) { UE_CONTROL_OK : 6084 UE_ISOCHRONOUS_OK : 0 UE_BULK_OK : 0 UE_INTERRUPT_OK : 0 UE_CONTROL_FAIL : 0 UE_ISOCHRONOUS_FAIL : 0 UE_BULK_FAIL : 0 UE_INTERRUPT_FAIL : 0 } ugen0.2: <Broadcom Corp BCM20702A0> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA) { UE_CONTROL_OK : 35 UE_ISOCHRONOUS_OK : 362068 UE_BULK_OK : 7384 UE_INTERRUPT_OK : 85 UE_CONTROL_FAIL : 0 UE_ISOCHRONOUS_FAIL : 8 UE_BULK_FAIL : 0 UE_INTERRUPT_FAIL : 0 } ugen0.3: <Logitech USB Receiver> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (98mA) { UE_CONTROL_OK : 23 UE_ISOCHRONOUS_OK : 0 UE_BULK_OK : 0 UE_INTERRUPT_OK : 54649 UE_CONTROL_FAIL : 2 UE_ISOCHRONOUS_FAIL : 0 UE_BULK_FAIL : 0 UE_INTERRUPT_FAIL : 0 }
It might be the isochronous part of bluetooth which is causing this. Try to capture this number every second, and see how quickly it rises: UE_ISOCHRONOUS_OK : 362068 --HPS
I suspect the XHCI controller might be interrupting every job it completes, instead of every chunk of jobs it completes. This might be related to the brand of XHCI controller you are using. --HPS
Created attachment 219809 [details] Try this patch
(In reply to Hans Petter Selasky from comment #6) UE_ISOCHRONOUS_OK = 60-62/sec, increases evenly After applying the patch still 1000 interrupts/sec by xhci0 in systat -vmstat But: without patch: hw.acpi.cpu.cx_lowest=C1 2000 hw.acpi.cpu.cx_lowest=C2 1000 with patch: hw.acpi.cpu.cx_lowest=C1 ~1050-1100 hw.acpi.cpu.cx_lowest=C2 1000
Maybe there is a bug in that particular XHCI controller, that it generate events for every ISOCHRONOUS job ... Else we are not setting the BEI bit correctly. Can you try the same device with another XHCI controller? --HPS
I see this issue reproduces on my Skylake aswell, and I investigated. The problem is that the XHCI controller doesn't support supressing completion events for LINK TRBs. Also the BEI bit appears to have some quirks. LINK TRBs are critical to the XHCI driver in FreeBSD. This issue can't be solved. You will get 1000 IRQ/s second when you use bluetooth, unless you disable the ISOCHRONOUS transfers in ubt0, which I think are mostly used for audio. --HPS
Created attachment 227066 [details] Disable_ISOCHRONOUS_transfers.patch
Just for your information. Disabling ISOCHRONOUS transfers works for me as a workaround. I feel it's OK to use a bluetooth keyboard and a mouse. See `Disable_ISOCHRONOUS_transfers.patch` for details.
Maybe we can make this into a tunable sysctl.
I wrote a patch to introduce a loader tunable. Please review my differential https://reviews.freebsd.org/D33282.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=67cbbf19595da4565d3c8603030fdb8d33ed571e commit 67cbbf19595da4565d3c8603030fdb8d33ed571e Author: Hans Petter Selasky <hselasky@FreeBSD.org> AuthorDate: 2021-12-07 10:28:21 +0000 Commit: Hans Petter Selasky <hselasky@FreeBSD.org> CommitDate: 2021-12-07 10:39:49 +0000 ng_ubt(4): Introduce net.bluetooth.usb_isoc_enable loader tunable to disable isochronous transfers. If users want to disable isochronous transfers, which cause high frequency periodic interrupts from the USB host controller, then net.bluetooth.usb_isoc_enable can be set to zero, either as a sysctl(8) or as a loader.conf(5) tunable. Differential Revision: https://reviews.freebsd.org/D33282 Submitted by: naito.yuichiro@gmail.com PR: 238235 MFC after: 1 week Sponsored by: NVIDIA Networking sys/netgraph/bluetooth/drivers/ubt/ng_ubt.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
Set net.bluetooth.usb_isoc_enable=0 in /boot/loader.conf for now. Closing this issue.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=03f0393477db4c11f8dc4166fadc2628b2c15ae9 commit 03f0393477db4c11f8dc4166fadc2628b2c15ae9 Author: Hans Petter Selasky <hselasky@FreeBSD.org> AuthorDate: 2021-12-07 10:50:38 +0000 Commit: Hans Petter Selasky <hselasky@FreeBSD.org> CommitDate: 2021-12-07 10:50:51 +0000 ng_ubt(4): Make net.bluetooth.usb_isoc_enable writable. Differential Revision: https://reviews.freebsd.org/D33282 PR: 238235 MFC after: 1 week Sponsored by: NVIDIA Networking sys/netgraph/bluetooth/drivers/ubt/ng_ubt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Created attachment 229968 [details] add hw.ubt sysctl node
hselasky@ Thanks for committing. And I'm very sorry that I made a mistake in the latest differential in https://reviews.freebsd.org/D33282 by accident. It's missing 'hw.ubt' sysctl node creation. Please apply 'add hw.ubt sysctl node' patch.
I'm confused that ng_ubt.ko can not be loaded by following message. ``` link_elf_obj: symbol sysctl___net_bluetooth undefined linker_load_file: /boot/kernel/ng_ubt.ko - unsupported file type link_elf_obj: symbol sysctl___net_bluetooth undefined linker_load_file: /boot/kernel/ng_ubt.ko - unsupported file type link_elf_obj: symbol sysctl___net_bluetooth undefined linker_load_file: /boot/kernel/ng_ubt.ko - unsupported file type link_elf_obj: symbol sysctl___net_bluetooth undefined linker_load_file: /boot/kernel/ng_ubt.ko - unsupported file type ```
I'll fix that.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8fa952937bbe44a0fdd17348adfbbfd44aef6004 commit 8fa952937bbe44a0fdd17348adfbbfd44aef6004 Author: Hans Petter Selasky <hselasky@FreeBSD.org> AuthorDate: 2021-12-08 09:17:27 +0000 Commit: Hans Petter Selasky <hselasky@FreeBSD.org> CommitDate: 2021-12-08 09:18:32 +0000 ng_ubt(4): Add missing module dependency for _net_bluetooth sysctl node. Differential Revision: https://reviews.freebsd.org/D33282 PR: 238235 MFC after: 1 week Sponsored by: NVIDIA Networking sys/netgraph/bluetooth/drivers/ubt/ng_ubt.c | 1 + 1 file changed, 1 insertion(+)
My mistake :-) I moved the sysctl node to the existing net.bluetooth.xxx .
hselasky@ I could confirm your commit solved ng_ubt loading problem. Thank you!
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7b976be508ce7bbb9652687d20a98d2e4b2d5a5b commit 7b976be508ce7bbb9652687d20a98d2e4b2d5a5b Author: Hans Petter Selasky <hselasky@FreeBSD.org> AuthorDate: 2021-12-07 10:28:21 +0000 Commit: Hans Petter Selasky <hselasky@FreeBSD.org> CommitDate: 2022-01-07 13:23:10 +0000 ng_ubt(4): Introduce net.bluetooth.usb_isoc_enable loader tunable to disable isochronous transfers. If users want to disable isochronous transfers, which cause high frequency periodic interrupts from the USB host controller, then net.bluetooth.usb_isoc_enable can be set to zero, either as a sysctl(8) or as a loader.conf(5) tunable. Differential Revision: https://reviews.freebsd.org/D33282 Submitted by: naito.yuichiro@gmail.com PR: 238235 Sponsored by: NVIDIA Networking (cherry picked from commit 67cbbf19595da4565d3c8603030fdb8d33ed571e) (cherry picked from commit 03f0393477db4c11f8dc4166fadc2628b2c15ae9) (cherry picked from commit 8fa952937bbe44a0fdd17348adfbbfd44aef6004) sys/netgraph/bluetooth/drivers/ubt/ng_ubt.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)