Re: Futures - HAMMER comparison testing?

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

:>     So it comes down to how much space you are willing to eat up to store
:>     the history, and what kind of granularity you will want for the history.
:
:OK - so it WILL be a 'tunable', then.
:...
:HAMMER cannot protect against all forms of human error - BUT - if it 
:inherently rebuilds more intelligently than the least-intelligent of 
:RAID1, it can greatly reduce the opportunity for that sort of 'accident' 
:to occur.

    One idea I had was to number the records as they were layed down on
    disk, and validate the file or directory by determining that no records
    were missing.  But that doesn't fly very well when things are deleted
    and replaced.

    Another idea, much easier to implement, is to have a way to guarentee that all
    the bits and pieces of the file had been found by creating a record which
    contains a CRC of the whole mess.   One could have a 'whole file' CRC, or
    even a 'whole directory tree' CRC (as-of a particular timestamp).
    Since HAMMER is record oriented associating special records with inodes
    is uttery trivial.

    For archival storage one could then 'tag' a directory tree with such a
    record and have a way of validating that the directory tree had not become
    corrupted, or was recovered properly.

    For encryption one could 'tag' a directory tree or a file with an
    encryption label.

    Not implemented yet but a definite possibility.  There are so many things
    we can do with HAMMER due to its record oriented nature.

:>     Ultimately it will be extremely efficient simply by the fact that
:>     there will be a balancer going through it and repacking it.
:> 
:"... constantly, and in the background..." (I presume)

    In the background, for sure.  Probably not constantly, but taking a piece
    at a time with a nightly cron job.  One thing I've learned over the
    years is that it is a bad idea to just go randomly accessing the disk
    at unexpected times.

    The nice thing is that the balancing can occur on a cluster-by-cluster
    basis, so one can do a bunch of clusters, then stop, then do a bunch
    more, then stop, etc.

:Is variable-length still likely to have a payback if the data records 
:were to be fixed at 512B or 1024B or integer multiples thereof?

    Not a good idea for HAMMER.  A HAMMER record is 96 bytes and a
    HAMMER B-Tree element is 56 bytes.  That's 152 bytes of overhead
    per record.  The smaller the data associated with each record,
    the larger the overhead and the less efficient the filesystem
    storage model.  Also, while accessing records is localized, you
    only reap major benefits over a linear block storage scheme
    if you can make those records reference a significant amount of
    data.

    So for large static files we definitely want to use a large
    per-record data size, and for small static files we want to use
    a small data size.  Theoretically the best-case storage for a tiny
    file would be 96 + 56 + 128 (inode data) + 64 (data), or
    344 bytes of disk space.  That's very, very good.  (In the current
    incarnation the minimum disk space use per file is 96 + 56 + 128 + 16384).

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>