rctl's idea of cputime is unreasonably high with lots of process turnover. Fix: Attached patch seems to help. How-To-Repeat: # jail -c command=sh jail# while true; do id > /dev/null; done meanwhile: # dtrace -n 'rusage:add-cred/args[0]->cr_prison->pr_id != 0 && args[1] == 0/{printf("%d: jail %d cputime %d", pid, args[0]->cr_prison->pr_id, args[2])}' 5 57139 rusage:add-cred 37375: jail 5 cputime 124211 5 57139 rusage:add-cred 37375: jail 5 cputime 6330 5 57139 rusage:add-cred 37375: jail 5 cputime 51237828 5 57139 rusage:add-cred 37375: jail 5 cputime 173602 5 57139 rusage:add-cred 37375: jail 5 cputime 6834680 (...)
Sorry, please ignore my patch; I guess the problem is just that = p_prev_runtime is never initialized. I'm not sure why it exists, but = removing it makes things work as expected.=
Responsible Changed From-To: freebsd-bugs->trasz I'll take it.
Since this problem can not be fixed for a long time it may be the best solution to add one more line into 'BUGS' of rctl(8) https://man.freebsd.org/rctl/8 At the moment (end of 2017) this problem has not been fixed in any supported FreeBSD version and this problem makes it impossible to have the correct statistics on the jail and makes it dangerous for people who use a billing system based on RACCT. This problem also affects 'pcpu' metrics ( %CPU, in percents of a single CPU core ) and can be easy to reproduce on single core: 1) Run jail1 2) Try to execute ant fast/light external command (e.g. /bin/ls ) in the loop. Or compile this sample as /root/a.out in jail: --- #include <stdio.h> int main() { return 0 } --- Write execution loop and drop it into jail, e.g /root/run.sh: --- #!/bin/sh while [ 1 ]; do /root/a.out > /dev/null done --- Run inside jail this script via: cpuset -c -l 0 /bin/sh /root/run.sh After this we can see on the 'top -P': --- 182 processes: 2 running, 180 sleeping CPU 0: 34.1% user, 0.0% nice, 65.9% system, 0.0% interrupt, 0.0% idle CPU 1: 0.5% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.5% idle CPU 2: 3.1% user, 0.0% nice, 1.2% system, 0.0% interrupt, 95.7% idle CPU 3: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle CPU 4: 1.2% user, 0.0% nice, 0.8% system, 0.0% interrupt, 98.1% idle CPU 5: 0.8% user, 0.0% nice, 0.4% system, 0.0% interrupt, 98.8% idle CPU 6: 1.2% user, 0.0% nice, 0.0% system, 0.0% interrupt, 98.8% idle CPU 7: 0.4% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.2% idle ... PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 41437 root 1 76 0 11408K 2232K CPU0 0 0:07 12.79% sh ... --- Only one core is busy. However if we look at the RACCT from the hoster side, we see the following picture: freebsd:~ # rctl -u jail:jail1 | grep pcpu pcpu=25600 freebsd:~ # rctl -u jail:jail1 | grep pcpu pcpu=25600 freebsd:~ # rctl -u jail:jail1 | grep pcpu pcpu=25600 Unfortunately this is not an unlikely reproduction of the problem. Similar you can see in real life very often, for example at the configuration stage when a large number of commands are executed: Try to execute in the jail, for example: env BATCH=no make -C /usr/ports/misc/mc clean configure And you will see the problem of statistics again
As an aside, you might want to look at /usr/bin/sa for accounting.
(In reply to Allan Jude from comment #4) Yes, I know about sa(8) but there are other problems (there is no support for jail, only cpu metrics..) Ideally for each component of FreeBSD (jail, racct..) would have a active maintainer. But today FreeBSD is a hobby-OS with a catastrophically small number of developers, fixes can be expected for several years (and not see them). Therefore, if we can not fix the bug, it should be described in the man pages. PS: openfiles metrics also give abnormally high values (in a few dozen) via RACCT (compared to fstat / lsof). But without subsystem maintainer and without entering such information into the man page, I'm not sure that it makes sense to write PR for this. Unfortunately, I can only help with testing (from a practical point of view) and report for issue but do not fix it ;-) Thanks.
(In reply to olevole from comment #5) The maintainer of RACCT is still active, just busy. For the open files count, remember that every socket, pipe, and other special types of file descriptor counts as an open file. I'll try to get someone to look at this
Be aware that my problem was with the cputime measurement, which seemed to be fixed not long after this bug. Thanks!
(In reply to ben from comment #7) Indeed, there is no problem with cputime now. Probably, this PR should be closed. Nevertheless the problem with ncpu is real
(In reply to Allan Jude from comment #6) Allan, Thanks! Should I register a new PR ? The original problem on which this PR is introduced is already fixed. The problem with 'pcpu' is very similar: 'pcpu' value are normal on the static processes ( e.g install net-p2p/cpuminer and run: minerd --benchmark ) but incorrect for this cases: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171811#c3
For bugs matching the following conditions: - Status == In Progress - Assignee == "bugs@FreeBSD.org" - Last Modified Year <= 2017 Do - Set Status to "Open"
(In reply to olevole from comment #9) Please create new PR for pcpu issue so this one could be closed. Thanks
done https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235556
The original bug is fixed per comment 7. There is a patch for PR 235556 which will land shortly.