Bug 290234 - sys/netinet/tcp_hpts_test.py tests cause a kernel panic in CI
Summary: sys/netinet/tcp_hpts_test.py tests cause a kernel panic in CI
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: tests (show other bugs)
Version: 16.0-CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Michael Tuexen
URL: https://ci.freebsd.org/view/Test/job/...
Keywords: crash, regression
Depends on:
Blocks:
 
Reported: 2025-10-14 18:48 UTC by Siva Mahadevan
Modified: 2025-11-12 09:52 UTC (History)
2 users (show)

See Also:


Attachments
kyua test report (2.93 KB, text/plain)
2025-10-14 18:48 UTC, Siva Mahadevan
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Siva Mahadevan freebsd_triage 2025-10-14 18:48:47 UTC
Created attachment 264584 [details]
kyua test report

The 'sys/netinet/tcp_hpts_test.py' tests cause a kernel panic in CI:

TCP HPTS started 2 ((unbounded)) swi interrupt threads
sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_concurrent_operations  ->  failed: /usr/tests/atf_python/sys/netlink/netlink.py:376: ValueError  [0.274s]
sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_cpu_assignment  ->  passed  [0.204s]
sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_deferred_requests  ->  failed: /usr/tests/atf_python/sys/netlink/netlink.py:376: ValueError  [0.258s]
sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_direct_wake_mechanism  ->  Sleeping on "-" with the following non-sleepable locks held:
exclusive sleep mutex tcp_hpts_lck (hpts) r = 0 (0xfffff8007a659600) locked @ /home/siva/src/fbsdsrcgit/sys/netinet/tcp_hpts_test.c:1467

I have reproduced this error using Bricoler with 100 runs for good measure and attached the failing test report in this bug:

$ bricoler run freebsd-src-regression-suite --param freebsd-src:url=/usr/src --param freebsd-src:branch= --param freebsd-src-regression-suite:hypervisor=bhyve --param freebsd-src-regression-suite:memory=4096 --param freebsd-src-regression-suite:ncpus=2 --param freebsd-src-regression-suite:parallelism=1 --param freebsd-src-regression-suite:count=100 --param freebsd-src-regression-suite:tests='sys/netinet/tcp_hpts_test.py'

@tuexen, this panic arises somewhere in your latest patch series between commit be1ad90e6bb6ea915d122f7abe407dc18d38a673 and commit 2241cd10d6e0eaf16d1f3090f1d438543e127f0c. I am not able to bisect further due to build issues in the intermediate commits inside the series. See the URL for the failure showing up in Jenkins CI. Please triage as necessary.
Comment 1 Michael Tuexen freebsd_committer freebsd_triage 2025-10-14 23:06:03 UTC
A patch is under review D53108.
Comment 2 commit-hook freebsd_committer freebsd_triage 2025-10-14 23:39:58 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1985940204dc01d4524c6351ab25a182bfde23ae

commit 1985940204dc01d4524c6351ab25a182bfde23ae
Author:     Michael Tuexen <tuexen@FreeBSD.org>
AuthorDate: 2025-10-14 23:36:54 +0000
Commit:     Michael Tuexen <tuexen@FreeBSD.org>
CommitDate: 2025-10-14 23:36:54 +0000

    conf: enable TCP_HPTS_KTEST on debug builds

    Enable the kernel build option TCP_HPTS_KTEST on debug builds and
    disable it on non-debug builds.

    PR:                     290234
    Reviewed by:            Nick Banks, glebius
    Sponsored by:           Netflix, Inc.
    Differential Revision:  https://reviews.freebsd.org/D53108

 sys/conf/std.debug   | 1 +
 sys/conf/std.nodebug | 1 +
 2 files changed, 2 insertions(+)
Comment 3 Siva Mahadevan freebsd_triage 2025-11-12 08:34:47 UTC
@tuexen does this also need to be MFC'd to stable/15? I see that the only failing test leftover is this: https://ci.freebsd.org/job/FreeBSD-stable-15-amd64-test/230/testReport/sys.netlink.test_netlink_message_writer/py/__test_cases_list__/. It seems that stable/15 does not have TCP_HPTS_KTEST.

After https://cgit.freebsd.org/src/commit/?id=1aad58b919d3d22f86be01b8e26a203cd020eaae (Remove std.debug from kernel configs) in stable/15, this test fails.
Comment 4 Michael Tuexen freebsd_committer freebsd_triage 2025-11-12 08:46:15 UTC
(In reply to Siva Mahadevan from comment #3)
This does not need to be MFCed to stable/15. Neither the commit mentions that nor this bug report mentions that.
Comment 5 Michael Tuexen freebsd_committer freebsd_triage 2025-11-12 08:47:31 UTC
(In reply to Michael Tuexen from comment #4)
What do you expect to happen here, since you changed the state from Closed to In Progress?
Comment 6 Siva Mahadevan freebsd_triage 2025-11-12 09:52:42 UTC
My mistake then, if TCP_HPTS_KTEST is not relevant to the failing netlink tests, I'll investigate the cause further. Changing back to closed.