Re: OT - was Hammer or ZFS based backup, encryption

[Michael Neumann <mneumann@xxxxxxxx>]

Am Sun, 22 Feb 2009 06:33:44 -0800
schrieb Jeremy Chadwick <jdc@parodius.com>:

> On Sun, Feb 22, 2009 at 01:36:28PM +0100, Michael Neumann wrote:
> > Am Sat, 21 Feb 2009 19:17:11 -0800
> > schrieb Jeremy Chadwick <jdc@parodius.com>:
> > 
> > > On Sun, Feb 22, 2009 at 11:59:57AM +1100, Dmitri Nikulin wrote:
> > > > On Sun, Feb 22, 2009 at 10:34 AM, Bill Hacker
> > > > <wbh@conducive.org> wrote:
> > > > > Hopefully more 'good stuff' will be ported out of Solaris
> > > > > before it hits the 'too costly vs the alternatives' wall and
> > > > > is orphaned.
> > > > 
> > > > Btrfs has been merged into mainline Linux now, and although it's
> > > > pretty far behind ZFS in completeness at the moment, it
> > > > represents a far greater degree of flexibility and power. In a
> > > > couple of years when it's stable and user friendly, high-end
> > > > storage solutions will move back to Linux, after having given
> > > > Sun a lot of contracts due specifically to ZFS.
> > > 
> > > The fact that btrfs offers grow/shrink capability puts it ahead
> > > of ZFS with regards to home users who desire a NAS.  I can't
> > > stress this point enough.  ZFS's lack of this capability limits
> > > its scope.  As it stands now, if you replace a disk with a larger
> > > one, you have to go through this extremely fun process to make
> > > use of the new space available:
> > > 
> > > - Offload all of your data somewhere (read: not "zfs export");
> > > rsync is usually what people end up using -- if you have multiple
> > > ZFS filesystems, this can take some time
> > > - zpool destroy
> > > - zpool create
> > > - zfs create
> > > 
> > > And if you add a new disk to the system, it's impossible to add
> > > that disk to the existing pool -- you can, of course, create an
> > > entirely new zpool which uses that disk, but that has nothing to
> > > do with the existing zpool.  So you get to do the above dance.
> > 
> > Hm, I thought that would work easily with ZFS, and at least in
> > theory I think that should work well with ZFS. Or what is wrong
> > with:
> > 
> >   zpool add tank /dev/ad8s1
> 
> This will only work how you expect if you're using a ZFS mirror.  With
> RAIDZ, it doesn't work -- you're forced to add the new disk into a new
> zpool.  This is one of the shortcomings of ZFS (and it is documented,
> but only lightly so).
> 
> > Okay "zpool remove" doesn't seem to work as expected, but it should
> > work well at least for RAID-1 (which probably no one uses for large
> > storage systems ;-). Maybe "zfs replace" works, if you replace an
> > old disk, with a larger disk, and split it into two partitions, the
> > one equally sized to the old, and the other containing the
> > remainder of the space. Then do:
> > 
> >   zfs replace tank old_device new_device_equally_sized
> >   zfs add tank new_device_remainder
> > 
> > But you probably know more about ZFS than me ;-)
> 
> In this case, yes (that I know more about ZFS than you :-) ).  What
> you're trying to do there won't work.
> 
> The "zfs" command manages filesystems (e.g. pieces under a zpool).
> You cannot do anything with devices (disks) with "zfs".  I think you
> mean "zpool", especially since the only "replace" command is "zpool
> replace".

Oops, yep, that was of course a typo of mine ;-)
 
> What you're trying to describe won't work, for the same reason I
> described above (with your "zpool add tank ad8s1" command).  You can
> split the disk into two pieces if you want, but it's not going to
> change the fact that you cannot *grow* a zpool.  You literally have to
> destroy it and recreate it for the pool to increase in size.

Ah okay, that's probably because the filesystem and RAID system are too
tighly bundled in ZFS. So if I understand correctly, you can't grow a
ZFS RAID-5 pool or anything similar to RAID-5.
Now the ZFS filesystem probably can only use blocks from one pool, so
the result is that you can't grow a ZFS filesystem living on a RAID-5+
pool as well. A bad example of coupling...

With Hammer the situation is different. You can let vinum
manage a RAID-5 pool (don't know if this is stable, but that's not my
point) and add the storage to a Hammer FS. If you need more space you
have too choices:

  1) Replace a disk with a larger one, splitting it into two subdisks
     (as I described in the last post).

  2) simply create a new RAID-5 pool (built using some new
     disks) and add it as well to the same filesystem. If you reblock
     everything to the new RAID-5 pool you could then remove the old
     RAID-5 pool completely.

Regards,

  Michael