Summary: | ZFS ARC stats have wrong count | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Base System | Reporter: | Yuichiro NAITO <naito.yuichiro> | ||||||||
Component: | kern | Assignee: | freebsd-fs (Nobody) <fs> | ||||||||
Status: | Closed FIXED | ||||||||||
Severity: | Affects Many People | CC: | allanjude, cy, pi, ruben | ||||||||
Priority: | --- | ||||||||||
Version: | 11.1-RELEASE | ||||||||||
Hardware: | Any | ||||||||||
OS: | Any | ||||||||||
Attachments: |
|
Description
Yuichiro NAITO
2017-10-11 06:23:20 UTC
As discussed at BSDTW, please do the following: zfs-stats -a <your script> zfs-stats -a OK, I will do it when I return to Japan, it will be 2 days later. I’m traveling Taiwan now. I changed my script as follows. File.stat() loops 5 times, and print `zfs-stats -a` message at first and in end of each loop. ``` puts "#{Time.now} start" puts `zfs-stats -a` 5.times do |c| Dir.glob("ports-head/**/*").each do |file| File.stat(file) end puts "#{Time.now} loop count : #{c}" puts `zfs-stats -a` end ``` The result is shown in attached 'zfs-stat.log' file. Attached `top.log` shows `top` header of before and after running this script. Created attachment 188004 [details]
zfs-stat.log
Created attachment 188005 [details]
top.log
(In reply to naito.yuichiro from comment #4) This log shows an interesting pattern, where the 'Cache Hit Ratio' goes up during the first run (reaching 58.16k hits total, from 8.73k), but doesn't go up on the successive runs. I will investigate this further I have confirmed the impact you are seeing, but also confirmed that is not a major bug. If you run: zpool iostat 1 You'll see that running your ruby script will not actually result in any reads from the disk. There is a small issue with the stats accounting in ZFS, where if the metadata being read, happens to be stored in an "Embedded Block Pointer" (so instead of being stored as a data block, the data is embedded in the parent block, to save an entire sector, and to save an I/O to read that sector), then it is incorrectly counted as a miss. This is because to read the embedded block pointer, it has to create a read zio and go through most of the process of doing a read, but then ends up copying the data out of the block pointer instead of from disk. Anyway, I am investigating a quick fixes to account for the cache hit correctly, instead of as a cache miss. I am also looking to see if it would be relatively simple to optimize the case and return the data more directly in arc_read() instead of creating a zio and going the currently more complicated path. This path mostly exists because it makes it possible for other functions to not need to know about the embedded block pointer feature. (In reply to Allan Jude from comment #7) Thank you very much to your investigation. My ruby script doesn't read disks actually. I thought this means all metadata is cached. I saw no problem with performance. > Anyway, I am investigating a quick fixes to account for the cache hit > correctly, instead of as a cache miss. I hope your fix. > I am also looking to see if it would be relatively simple to optimize the case > and return the data more directly in arc_read() instead of creating a zio and > going the currently more complicated path. This path mostly exists because it makes > it possible for other functions to not need to know about the embedded block > pointer feature. I don't know about zfs source codes. But if you need to test your patch in my environment, please ask me. I will test in new boot environment on my PC. I learned the feature in your session in BSDtw! Created attachment 192787 [details]
zfs-stat-resolved.log
I checked that r332523 has fixed this problem. After upgrading r332641, I ran my script described in comment:3 again. It shows as 'zfs-stat-resolved.log'. Cache Hit Ratio increases in each loop as I expected. I will close this PR. Thank you! |