Bug 159251 - [zfs] [request]: add FLETCHER4 as DEDUP hash option
Summary: [zfs] [request]: add FLETCHER4 as DEDUP hash option
Status: Closed Not Accepted
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 8.2-STABLE
Hardware: Any Any
: Normal Affects Only Me
Assignee: Xin LI
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-07-28 09:40 UTC by Slawomir Wojciech Wojtczak
Modified: 2014-10-10 08:32 UTC (History)
2 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Slawomir Wojciech Wojtczak 2011-07-28 09:40:06 UTC 
Hi,

it would be nice to have fletcher4 as a faster alternative to sha256:

# zfs set dedup=fletcher4,verify tank
cannot set property for 'tank': 'dedup' must be one of 'on | off | verify | sha256[,verify]'

FROM: http://blogs.oracle.com/bonwick/entry/zfs_dedup
----------------------------------------------------------------------
Given the ability to detect hash collisions as described above, it is
possible to use much weaker (but faster) hash functions in combination
with the 'verify' option to provide faster dedup. ZFS offers this option
for the fletcher4 checksum, which is quite fast: 

# zfs set dedup=fletcher4,verify tank 

The tradeoff is that unlike SHA256, fletcher4 is not a pseudo-random
hash function, and therefore cannot be trusted not to collide. It is
therefore only suitable for dedup when combined with the 'verify' option,
which detects and resolves hash collisions. On systems with a very high
data ingest rate of largely duplicate data, this may provide better
overall performance than a secure hash without collision verification.
----------------------------------------------------------------------

Regards,
vermaden
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2011-07-29 06:54:25 UTC 
Responsible Changed
From-To: freebsd-bugs->freebsd-fs

Over to maintainer(s).
Comment 2 Jean-François BOEUF 2014-10-08 13:12:53 UTC 
not relevant anymore : fletcher4 is the default (since r259477). Can be closed ?
Comment 3 Slawomir Wojciech Wojtczak 2014-10-09 07:41:38 UTC 
> not relevant anymore : fletcher4 is the default (since r259477). Can be closed ?

Check the message again.

Its not about:
# zfs set checksum=fletcher4

Its about:
# zfs set checksum=fletcher4,verify

... which still does not work:
# zfs set checksum=fletcher4,verify sys/TEST
cannot set property for 'sys/TEST': 'checksum' must be one of 'on | off | fletcher2 | fletcher4 | sha256'

Regards,
vermaden
Comment 4 Xin LI freebsd_committer freebsd_triage 2014-10-10 08:27:15 UTC 
This feature is intentionally removed in OpenSolaris build 128 because fletcher4 is not endianness safe and caused problems for cross-endianness.  On the other hand the collision rate is fairly high, making it not suitable as a dedup checksum.

We may add some other faster checksum algorithms in the future but it's unlikely that fletcher4 will ever be added back as a dedup checksum.
Comment 5 Slawomir Wojciech Wojtczak 2014-10-10 08:32:15 UTC 
Ok, so You can close this one.

Regards,
vermaden