Bug 234325 - pmcstat seems to be broken in sampling mode (at least on amd hardware)
Summary: pmcstat seems to be broken in sampling mode (at least on amd hardware)
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2018-12-24 07:41 UTC by shamaz.mazum
Modified: 2020-08-19 17:26 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description shamaz.mazum 2018-12-24 07:41:50 UTC
Hello. I use FreeBSD 12.0-RELEASE on two machines (one with Ryzen 5 1600X processor and the other with FX-6300).

Recently (presumable after upgrading to 12.0-RELEASE) pmcstat stopped working in sample mode.

I run it as:
`pmcstat -P instructions -O test.out -n 65536 ./noisecpu` on FX-6300
or
`pmcstat -P ex_ret_instr -O test.out -n 65536 ./noisecpu` on Ryzen

and then I run `pmcstat -R test.out -g`

In both cases it creates directories (e.g. ex_ret_instr) which are either empty or containing only kernel.gmon

`noisecpu` is computation heavy program, calculating value noise on a large grid and works for ~8 seconds.

Conversion statistics usually looks like following:
#exec/elf 1
#samples/total 169
#samples/unclaimed 165
#callchain/dubious-frames 165

In counting mode (with -p argument) everything seems to be working. Can anybody confirm this?
Comment 1 Leandro Lupori freebsd_committer 2020-08-19 17:26:24 UTC
Hello. I'm using FreeBSD 13.0-CURRENT (r364197) on an amd64 VM (qemu with kvm) and I'm seeing similar issues. Host has a Core i7-6700K CPU.

pmcstat -S/-s/-p seems to work fine, but I've noticed several issues with pmcstat -P.

I started using sysutils/stress for the tests - that, with --cpu flag, basically calls sqrt(rand()) in a tight loop - but the only way to get it working under pmcstat -P was to build it statically, with debug symbols and modify it so it doesn't fork (I called this modified version mystress).
Then the following commands produce correct results:

pmcstat -n 500000 -d -P inst_retired.any -O sample.out /tmp/mystress.static
pmcstat -R sample.out -G sample.graph
CONVERSION STATISTICS:
 #exec/elf                                1
 #samples/total                           52068

Sample.graph correctly shows hogcpu(), rand() and random_r() as the functions that consumed most of the time.

But, when using a static version of stress, with debug symbols, but that forks, pmcstat fails to resolve all userspace callchains:

# case 1
pmcstat -n 500000 -d -P inst_retired.any -O sample.out ./stress.static -c 1 -t 3
pmcstat -R sample.out -G sample.graph
CONVERSION STATISTICS:
 #exec/elf                                1
 #samples/total                           49771
 #callchain/dubious-frames                49742

From a brief look at pmcstat code, it seems there is a missing PROCFORK event, to tell it about the forked child.

Another case in which pmcstat -P fails to resolve most symbols is when using a dynamically linked binary (mystress in this case, that doesn't fork):

# case 2
pmcstat -n 500000 -d -P inst_retired.any -O sample.out /tmp/mystress
pmcstat -R sample.out -G sample.graph
CONVERSION STATISTICS:
 #exec/elf                                1
 #samples/total                           52558
 #samples/unknown-function                5973
 #callchain/dubious-frames                38383

The main binary symbols are resolved correctly (e.g. hogcpu), but none of the shared libraries' ones are (e.g. rand).

Yet another case in which pmcstat -P fails to resolve symbols is when using it with -t to specify the process, instead of starting it on the command line:

# case 3
./mystress.static &
sleep 1
pmcstat -n 500000 -d -P inst_retired.any -O sample.out -t mystress
pmcstat -R sample.out -G sample.graph
CONVERSION STATISTICS:
 #samples/total                           4195
 #callchain/dubious-frames                4193

Sometimes no samples are collected at all.

All the issues above also occur on a PowerPC64 machine.