Re: RAID 1 or Hammer

[Michael Neumann <mneumann@xxxxxxxx>]

Am 13.01.2009 06:02, schrieb Dmitri Nikulin:
> On Tue, Jan 13, 2009 at 2:15 PM, Matthew Dillon
> <dillon@apollo.backplane.com>  wrote:
>>     I've seen uncaught data corruption on older machines, but not in the
>>     last few years.  Ah, the days of IDE cabling problems, remembered
>>     fondly (or not).  I've seen bad data get through TCP connections
>>     uncaught!  Yes, it actually does happen, even more so now that OS's
>>     are depending more and more on CRC checking done by the ethernet 
device.
>
> I once came across a machine in which the IDE cable was plugged in
> wrong. It was plugged a whole row over, leaving the end two pins
> unconnected, and offsetting each pin that was connected. Somehow the
> machine worked fine and only under high load would it start to give
> DMA errors, drop down the DMA level (which any OS I've seen does
> automatically), and continue slower but stable. I found no evidence of
> data corruption and the machine worked at full speeds when the cable
> was moved.
>
> One of my own machines had been supplied with a bad ASUS-labelled IDE
> cable, which exhibited similar symptoms to the one above that was
> plugged in wrong. From both incidents I learned that IDE is pretty
> robust on modern chipsets.
>
> I knew I wasn't just paranoid about ethernet bugs. That's why I run
> mission-critical links over SSH or OpenVPN so that a proper checksum
> and replay protocol take care of it.
>
>>     ZFS uses its integrity check data for a lot more then simple 
validation.
>>     It passes the information down into the I/O layer and this 
allows the
>>     I/O layer (aka the software-RAID layer) to determine which 
underlying
>>     block is the correct one when presented with multiple choices. 
So, for
>>     example, if data is mirrored the ZFS I/O layer can determine 
which of
>>     the mirrored blocks is valid... A, B, both, or neither.
>
> Actually that's precisely what I like about ZFS, the self-healing with
> redundant copies. It means that as long as any corruption is healed
> before the redundant copy is similarly damaged, there will be
> basically no loss of data *or* redundancy. This is in stark contrast
> to a plain mirror in which one copy will become unusable and the data
> in question is no longer redundant, and possibly incorrect.
>
>>     People have debunked Sun's tests as pertaining to a properly 
functioning
>>     RAID system.  But ZFS also handles any Black Swan that shows up 
in the
>>     entire I/O path.  A Black Swan is an unexpected condition.  For 
example,
>>     an obscure software bug in the many layers of firmware that the data
>>     passes through.  Software is so complex these days there are 
plenty of
>>     ways the data can get lost or corrupted without necessarily causing
>>     actual corruption at the physical layer.
>
> I confess that, lacking ZFS, I have a very paranoid strategy on my
> Linux machines for doing backups (of code workspaces, etc). I archive
> the code onto a tmpfs and checksum that, and from the tmpfs distribute
> the archive and checksum to local and remote archives. This avoids the
> unthinkably unlikely worst case where an archive can be written to
> disk, dropped from cache, corrupted, and read bad wrong in time to be
> checksummed. The on-disk workspace itself has no such protection, but

But then your RAM can be faulty as well. I got a 1-bit correction
message last time I did a full buildworld on this ASUS mainboard.
Without ECC RAM I wouldn't have noticed this error.

> at least I can tell that each backup is as good as the workspace was
> when archived, which of course has to pass a complete recompile.

You could also use git for your code workspaces, which creates SHA1
checksums of every file, so that you can verify your backups all the
time.  There is even a very cool guy who uses git for doing backups of
his system [1].

Regards,

  Michael

[1]: http://eigenclass.org/hiki/gibak-backup-system-introduction