Bug 278726

Summary:	14.0/14.1-RELEASE kernel uses 200% of CPU interrupts+swapin
Product:	Base System	Reporter:	ota
Component:	kern	Assignee:	freebsd-arm (Nobody) <freebsd-arm>
Status:	New ---
Severity:	Affects Only Me	CC:	chris
Priority:	---
Version:	14.0-RELEASE
Hardware:	arm64
OS:	Any

Description ota 2024-05-04 04:25:44 UTC

I just installed 14.1-PRELEASE(stable/14).
This one is arm64 virtual machine and has about 4GB.
I use ZFS but as an external device and thus don't use enable_zfs.

14.1-PRELEASE booted okay.
After zpool import, kernel started using 100% of a single CPI.
The "top" shows keeps "swapin".
Memory usage is low all times.

Comment 1 ota 2024-05-09 03:57:07 UTC

I tested with amd64 from the same source code.
amd64 didn't reproduce the issue.

Comment 2 ota 2024-05-16 03:26:00 UTC

After `git bisect`, I found a range of bad commits.
There multiple versions that buildworld/buildkernel failed and couldn't identify a single version.

On the older version of kernel, kernel spins just booting.  The zfs only causes swapin spin in newer version of kernel including 14.1-BETA*.

This is the output of final bisect steps:

Bisecting: 9 revisions left to test after this (roughly 3 steps)
[ea3197e69bf3c232999bd2bdf20cc968e1a361bb] kcmp: Add a manual page
% git bisect bad
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[7dff3d1cdfd1556d2137ddd8dd1916e0945ed240] kcmp(2): implement for devfs files
% git bisect skip
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[575a909aa2430fcfe5b8ca8ae0d063a8b775fe4b] kcmp(2): implement for generic file types
% git bisect skip
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[d7c4a6205feb6ba2d44159e343ec61cae3ecffb4] file: add fo_cmp method
% git bisect good
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[c0565e7d34ac248bf3c20c66f341a64a6e33d6d9] Create namespace for the symbols added during 15-CURRENT cycle
% git bisect bad
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[b566e44b2b88eb9f1b3b658998c7418cccab9e2f] stable/14: bump __FreeBSD_version for kcmp(2) and kern_openatfp(9)
% git bisect skip
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
7dff3d1cdfd1556d2137ddd8dd1916e0945ed240
098500b79e1734f269743eabb624c854804c67ac
81cee1350ff87537f8203d2db0e8ed2acdff7aef
0b864eb03f6621d9d0f122e6b18360a4778a4647
575a909aa2430fcfe5b8ca8ae0d063a8b775fe4b
136d75ad86d1a62d673490ed2679e7819444f0a9
f8d885c08dfc3d93ba00ae216c9587e990d64a5b
add51cc280f3a48abb2f4fc2e2e4f56efc186d54
b566e44b2b88eb9f1b3b658998c7418cccab9e2f
c0565e7d34ac248bf3c20c66f341a64a6e33d6d9
We cannot bisect more!

Comment 3 ota 2024-05-27 04:00:02 UTC

Actually, kernel spinning happens two other cases more often.

1. An easy to see case is at boot time.  When kernel boots, kern spins with swapin and I also see interrupts using 100%.

2. I run aarch64 in a virtual machine.  After suspending and resuming, the kernel starts spinning with swapin while interrupts also uses 100% CPU.  It looks device detection seem to trigger these spins.

I've been bisecting but haven' yet found the commit starts this behavior.

Comment 4 ota 2024-06-07 04:42:14 UTC

I was able to reproduce in 14.0-RELEASE as well although it really happens rarely.

I had to adjust git bisect range and been still actively trying to find its root case.