Bug 268943 - virtualized kernel panics on process exit running under pmcstat
Summary: virtualized kernel panics on process exit running under pmcstat
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Mitchell Horne
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2023-01-14 00:05 UTC by John F. Carr
Modified: 2026-07-10 16:45 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description John F. Carr 2023-01-14 00:05:52 UTC
Running a recent CURRENT under bhyve, pmcstat reliably causes a panic when the monitored process exits.

# pmcstat -p instructions ls > /dev/null 2> /tmp/2
panic: [amd,770] PMC0, CPU0 "K8-0" already stopped
cpuid = 0
time = 1673651198
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c326dc20
vpanic() at vpanic+0x151/frame 0xfffffe00c326dc70
panic() at panic+0x43/frame 0xfffffe00c326dcd0
amd_stop_pmc() at amd_stop_pmc+0x12e/frame 0xfffffe00c326dcf0
pmc_process_exit() at pmc_process_exit+0x26d/frame 0xfffffe00c326dd80
exit1() at exit1+0x36e/frame 0xfffffe00c326ddf0
sys_exit() at sys_exit+0xd/frame 0xfffffe00c326de00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe00c326df30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c326df30
--- syscall (1, FreeBSD ELF64, exit), rip = 0x2ee9ac03458a, rsp = 0x2ee9a9c13488, rbp = 0x2ee9a9c134a0 ---

System is
CPU: AMD EPYC 7402P 24-Core Processor                (2794.91-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x830f10  Family=0x17  Model=0x31  Stepping=0
and the VM has 8 cores.

This does not happen on the 13.1-STABLE host or on a desktop class Zen CPU ("AMD Ryzen 5 PRO 2400GE") running CURRENT.

I am splitting this off from bug 268897.  I found it trying to reproduce that bug, but it looks different.
Comment 1 Mitchell Horne freebsd_committer freebsd_triage 2023-06-22 19:44:39 UTC
It appears that bhyve does not support virtualized performance counters, neither on AMD or Intel. Thus, the assertion is triggered because reading the counter configuration register will always return a value of zero.

The AMD hwpmc class should detect when this support is disabled and bail from the initialization routine. I will work on this change.
Comment 2 Mark Johnston freebsd_committer freebsd_triage 2024-10-27 13:50:11 UTC
This is still reproducible on main, for what it's worth:

root@freebsd:~ # pmcstat -p instructions ls
.cshrc          .panic: [amd,519] PMC0, CPU3 "K8-0" already stopped
cpuid = 3
time = 1730036956
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00b3b31b30
vpanic() at vpanic+0x136/frame 0xfffffe00b3b31c60
panic() at panic+0x43/frame 0xfffffe00b3b31cc0
amd_stop_pmc() at amd_stop_pmc+0x105/frame 0xfffffe00b3b31cf0
pmc_process_exit() at pmc_process_exit+0x281/frame 0xfffffe00b3b31d80
exit1() at exit1+0x370/frame 0xfffffe00b3b31df0
sys_exit() at sys_exit+0xd/frame 0xfffffe00b3b31e00
amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe00b3b31f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00b3b31f30
--- syscall (1, FreeBSD ELF64, exit), rip = 0x2cf749f1674a, rsp = 0x2cf7462af8f8, rbp = 0x2cf7462af910 ---
KDB: enter: panic
[ thread pid 3157 tid 100141 ]
Stopped at      kdb_enter+0x33: movq    $0,0x1056282(%rip)
Comment 3 John F. Carr 2024-10-27 21:58:38 UTC
A comment in pmc_amd_initialize says there is no CPUID flag for performance counters.  It may be necessary to do a trial wrmsr and see if the value written sticks.
Comment 4 commit-hook freebsd_committer freebsd_triage 2026-06-09 19:47:08 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=dded0ab415cc09eed506968366e383d406834823

commit dded0ab415cc09eed506968366e383d406834823
Author:     Ali Mashtizadeh <ali@mashtizadeh.com>
AuthorDate: 2026-06-05 23:48:53 +0000
Commit:     Mitchell Horne <mhorne@FreeBSD.org>
CommitDate: 2026-06-09 19:46:31 +0000

    hwpmc: Disable AMD PMCs if in an unsupported VM

    AMD does not have a CPUID bit to indicate the lack of K8 PMCs.  If all
    other PMC features are not present we should test an event selector to
    see if it stores and returns a value.  If the VM is implemented
    correctly, this should result in a #GP on the initial wrmsr_safe.  Bhyve
    and a few other VMs ignore writes, so I got one step further and test
    that it retains the OS and USR bits.

    Tested on Zen 5 native and a Zen 5 Bhyve virtual machine.  This code
    should not run on any recent hardware, except in a VM, as it checks that
    the core counter extension is missing.

    PR:             268943
    Reported by:    Sandipan Das, John F. Carr <jfc@mit.edu>
    Reviewed by:    mhorne, imp
    Sponsored by:   Netflix
    MFC after:      1 week
    Pull Request:   https://github.com/freebsd/freebsd-src/pull/2272/changes

 sys/dev/hwpmc/hwpmc_amd.c | 37 +++++++++++++++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 2 deletions(-)
Comment 5 commit-hook freebsd_committer freebsd_triage 2026-07-10 15:29:34 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=620bca5338abdb599e304e3437446f7731cb78db

commit 620bca5338abdb599e304e3437446f7731cb78db
Author:     Ali Mashtizadeh <ali@mashtizadeh.com>
AuthorDate: 2026-06-05 23:48:53 +0000
Commit:     Mitchell Horne <mhorne@FreeBSD.org>
CommitDate: 2026-07-10 15:28:08 +0000

    hwpmc: Disable AMD PMCs if in an unsupported VM

    AMD does not have a CPUID bit to indicate the lack of K8 PMCs.  If all
    other PMC features are not present we should test an event selector to
    see if it stores and returns a value.  If the VM is implemented
    correctly, this should result in a #GP on the initial wrmsr_safe.  Bhyve
    and a few other VMs ignore writes, so I got one step further and test
    that it retains the OS and USR bits.

    Tested on Zen 5 native and a Zen 5 Bhyve virtual machine.  This code
    should not run on any recent hardware, except in a VM, as it checks that
    the core counter extension is missing.

    PR:             268943
    Reported by:    Sandipan Das, John F. Carr <jfc@mit.edu>
    Reviewed by:    mhorne, imp
    Sponsored by:   Netflix
    MFC after:      1 week
    Pull Request:   https://github.com/freebsd/freebsd-src/pull/2272/changes

    (cherry picked from commit dded0ab415cc09eed506968366e383d406834823)

 sys/dev/hwpmc/hwpmc_amd.c | 37 +++++++++++++++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 2 deletions(-)