Bug 268943 - virtualized kernel panics on process exit running under pmcstat
Summary: virtualized kernel panics on process exit running under pmcstat
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Mitchell Horne
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2023-01-14 00:05 UTC by John F. Carr
Modified: 2024-10-27 21:58 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description John F. Carr 2023-01-14 00:05:52 UTC
Running a recent CURRENT under bhyve, pmcstat reliably causes a panic when the monitored process exits.

# pmcstat -p instructions ls > /dev/null 2> /tmp/2
panic: [amd,770] PMC0, CPU0 "K8-0" already stopped
cpuid = 0
time = 1673651198
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c326dc20
vpanic() at vpanic+0x151/frame 0xfffffe00c326dc70
panic() at panic+0x43/frame 0xfffffe00c326dcd0
amd_stop_pmc() at amd_stop_pmc+0x12e/frame 0xfffffe00c326dcf0
pmc_process_exit() at pmc_process_exit+0x26d/frame 0xfffffe00c326dd80
exit1() at exit1+0x36e/frame 0xfffffe00c326ddf0
sys_exit() at sys_exit+0xd/frame 0xfffffe00c326de00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe00c326df30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c326df30
--- syscall (1, FreeBSD ELF64, exit), rip = 0x2ee9ac03458a, rsp = 0x2ee9a9c13488, rbp = 0x2ee9a9c134a0 ---

System is
CPU: AMD EPYC 7402P 24-Core Processor                (2794.91-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x830f10  Family=0x17  Model=0x31  Stepping=0
and the VM has 8 cores.

This does not happen on the 13.1-STABLE host or on a desktop class Zen CPU ("AMD Ryzen 5 PRO 2400GE") running CURRENT.

I am splitting this off from bug 268897.  I found it trying to reproduce that bug, but it looks different.
Comment 1 Mitchell Horne freebsd_committer freebsd_triage 2023-06-22 19:44:39 UTC
It appears that bhyve does not support virtualized performance counters, neither on AMD or Intel. Thus, the assertion is triggered because reading the counter configuration register will always return a value of zero.

The AMD hwpmc class should detect when this support is disabled and bail from the initialization routine. I will work on this change.
Comment 2 Mark Johnston freebsd_committer freebsd_triage 2024-10-27 13:50:11 UTC
This is still reproducible on main, for what it's worth:

root@freebsd:~ # pmcstat -p instructions ls
.cshrc          .panic: [amd,519] PMC0, CPU3 "K8-0" already stopped
cpuid = 3
time = 1730036956
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00b3b31b30
vpanic() at vpanic+0x136/frame 0xfffffe00b3b31c60
panic() at panic+0x43/frame 0xfffffe00b3b31cc0
amd_stop_pmc() at amd_stop_pmc+0x105/frame 0xfffffe00b3b31cf0
pmc_process_exit() at pmc_process_exit+0x281/frame 0xfffffe00b3b31d80
exit1() at exit1+0x370/frame 0xfffffe00b3b31df0
sys_exit() at sys_exit+0xd/frame 0xfffffe00b3b31e00
amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe00b3b31f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00b3b31f30
--- syscall (1, FreeBSD ELF64, exit), rip = 0x2cf749f1674a, rsp = 0x2cf7462af8f8, rbp = 0x2cf7462af910 ---
KDB: enter: panic
[ thread pid 3157 tid 100141 ]
Stopped at      kdb_enter+0x33: movq    $0,0x1056282(%rip)
Comment 3 John F. Carr 2024-10-27 21:58:38 UTC
A comment in pmc_amd_initialize says there is no CPUID flag for performance counters.  It may be necessary to do a trial wrmsr and see if the value written sticks.