DragonFly users List (threaded) for 2011-07
Re: Real World DragonFlyBSD Hammer DeDup figures from HiFX - Reclaiming more than 1/4th ( 30% ) Disk Space from an Almost Full Drive
:> >> take?
:> > I ran them one by one. at my own pace but the biggest two
:> > simultaneously did not take more than 2 hrs.
:> > So I guess 2-3 hrs would be a nice approximation :-)
:> My experiences were different on a file system containing a lot of data
:> I didn't try dedup itself but a dedup-simulate already ran for more than
:> two days (consuming a lot of memory in the process) before I finally
:> cancelled it.
: Most odd - I just tried a dedup-simulate on a 2TB filesystem with
:about 840GB used, it finished in about 30 seconds and reported a ratio of
:1.01 (dedup has been running automatically every night on this FS).
:Steve O'Hara-Smith | Directable Mirror Arrays
I think this could be a case of the more CRC collisions we have, the
more I/O dedup (or dedup-simulate) needs to issue to determine if the
collision is an actual dup or if it's just a CRC collision and the
data is different.
The memory use can be bounded with some additional work on the software,
if someone wants to have a go at it. Basically the way you limit memory
use is by dynamically limiting the CRC range that you observe in a pass.
As you reach a self-imposed memory limit you reduce the CRC range and
throw away out-of-range records. Once the pass is done you start a new
pass with the remaining range. Rinse, repeat until the whole thing is
That would make it possible to run de-dup with bounded memory. However,
the extra I/O's required to verify duplicate data cannot be avoided.