Bug 204716 - boot loader bcache is trashed by larger sequential reads from zfs/ufs
Summary: boot loader bcache is trashed by larger sequential reads from zfs/ufs
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-hackers (Nobody)
Keywords: patch
Depends on:
Reported: 2015-11-21 11:55 UTC by Toomas Soome
Modified: 2016-04-19 10:19 UTC (History)
4 users (show)

See Also:

loader-performance-boost patch (2.42 KB, patch)
2015-11-21 11:55 UTC, Toomas Soome
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Toomas Soome 2015-11-21 11:55:55 UTC
Created attachment 163375 [details]
loader-performance-boost patch

this work is direct result of findings while working on loader support on illumos; since illumos kernel needs not just kernel itself but also rather large boot_archive (especially in case of installer image or smartos), I have noticed slow read of large files.

while investigating possible causes, I noticed the block cache trashing (huge number of misses from bcachestat) after loading large files. So i did implement additional mechanism to make larger reads to bypass the bcache and it resulted huge boost in file loading, also quite visible in case of freebsd.

note the switch off condition for bcache in libstand/read.c is arbitrary (2*512B sectors) and perhaps the better solution is possible, however it seems to provide "good enough" results.  read() change is to help to boost zfs reader, for ufs I did add bcache disabler for file read call, so other reads should benefit from bcache.

Since for illumos support I had to add mechanism to recognize gzip'ped files, I did leave in the compression specific flags in attached diff, perhaps the order of the flags should be reversed to avoid including compression flags for fbsd... for now I left it as is:)
Comment 1 Andrey V. Elsukov freebsd_committer 2015-11-22 19:01:04 UTC
add hackers@ to discuss this change.
Comment 2 Andrey V. Elsukov freebsd_committer 2015-11-22 19:01:28 UTC
I reduced usage of bcache in r241053. Now it doesn't needed for caching of partition tables metadata. I think we need to measure how often and how many bytes of the metadata read UFS and ZFS. And probably remove bcache at all.
Comment 3 Toomas Soome 2015-11-22 22:05:35 UTC
well, of course it really depends on other interfaces, I actually did something more about bcache - instead of LRU, I use it as hash table, and in fact did make it a bit larger for test purposes.

zfs boot to beastie menu:
cache blocks: 8192
cache blocksz: 512
3906 ops  1533 bypasses  2281 hits  559 misses  6 flushes

few / directory listings and:
cache blocks: 8192
cache blocksz: 512
10110 ops  4473 bypasses  5754 hits  569 misses  6 flushes

same with illumos ufs:
cache blocks: 8192
cache blocksz: 512
1018 ops  297 bypasses  6636 hits  594 misses  6 flushes

few / directory listings and:
cache blocks: 8192
cache blocksz: 512
1564 ops  411 bypasses  13228 hits  914 misses  6 flushes

so, it can provide some use still - from this trivial sample, especially with ufs. but once again - this sample is with hash table and without 2sec TTL check.
Comment 4 Toomas Soome 2016-04-19 10:19:20 UTC
fix implemented as https://svnweb.freebsd.org/base?view=revision&revision=298230