HEADS DOWN: dm_target_crypt, tcplay and libdm

[Alex Hornung <ahornung@xxxxxxxxx>]

Hi all,

heads down if you use master: problems (maybe) and new features are
coming your way!

Over the past few weeks I've been preparing a major series of commits. A
few weeks ago I updated opencrypto and also added support for twofish
and serpent ciphers. This was the preliminary work needed for tcplay.

tcplay is a fully featured BSD-licensed TrueCrypt implementation using
our dm_target_crypt; it is 100% compatible (with recent versions; older
versions using aes-lrw and similar are not). It supports cipher
cascades, hidden volumes, system volumes, etc. I started it as an
experiment to investigate the TrueCrypt header format, but it ended up
as a full implementation written from scratch. It now is divided into
tcplay(8), the tool itself, and libtcplay (tcplay(3)), a very simple API
to mount and unmount TrueCrypt volumes. tcplay is now fully integrated
into DragonFly, including cryptdisks(8) support and root mount support
(the realroot type is "tcplay", as documented in mkinitrd(8)).

WARNING: I've done my best in testing it and its compatibility with
TrueCrypt, but it might well be that I missed something. For now I
advise you to treat tcplay as unstable/experimental. If you find out
that some volume is not compatible but works fine in TrueCrypt, it'd be
good if you could provide me with a sample volume that behaves like that
so I can dig into it.

While testing tcplay with cipher cascades, I found a bug in
dm_target_crypt which affects interdependent/stacked volumes. If one
crypt volume was dependent on another, a situation could occur where one
volume would starve the other of memory I/O buffers provided by a shared
mpipe. To avoid this situation I've changed the mpipes to be
per-instance, to ensure that every instance always has some I/O buffers
available. It uses now at most 0.2% of physical memory for each
instance. Previously it used at most 0.5% of physical memory across all
instances. This is pretty arbitrary, if someone feels it's too
small/big, please let me know.

Since tcplay was intended to be BSD licensed but depended on
libdevmapper which is GPL-licensed, I decided on Friday to rewrite
libdevmapper under a BSD license; the result of which is libdm. It's a
fairly simple implementation but has (almost) all public features of
libdevmapper. It won't work with stuff like dmsetup and lvm since those
use all sorts of internal pools and trees that libdm doesn't have, but I
have tested it with both tcplay(8) and cryptsetup(8) and it works like a
charm. By default both tcplay(8) and cryptsetup(8) now use libdm instead
of libdevmapper. I also switched over cryptsetup after the testing as
both libtcplay and libcryptsetup are used in cryptdisks(8) and hence
both have to use the same backend dm library as the namespace collides.

WARNING: A word of warning about libdm: I wrote it in just a few hours
and I bet there are still some minor issues. I've tested the main
functionality of cryptsetup linked against libdm, but there might be
some cases left that will behave erratically. If that's the case, please
let me know so I can dig into that.

Just in case someone wonders, I am fully aware of NetBSD's libdm. I did
not use it because it has a different API that is not directly
compatible with linux libdevmapper's and would hence require rewriting
of libdevmapper consumers instead of a plug-in replacement.

While on it, I also further simplified some of dm(4). I got rid of
excessive locking in the I/O path and concurrency and performance under
heavy use should improve. Previously the I/O strategy path was under one
exclusive lock which is now gone. This also fixes some other issues that
could occur when using remove or remove_all, where remove_all would hold
the same lock as the strategy routine required, and removal would deadlock.

I plan to write some proper documentation on the whole dm story (lvm,
cryptsetup, tcplay, etc) but I still haven't found the time to do that.


Cheers,
Alex Hornung