DragonFly kernel List (threaded) for 2004-12
Re: Description of the Journaling topology
Matthew Dillon wrote:
When all is said and done the journaling mechanism is going to look
-> [ MEMORY FIFO ] -> [ worker thread ] -> STREAM
/ e.g. 16MB [ secondary spool]
VFSOP -> [journal shim]-- e.g. 16GB
(transaction id) \
-> [filesystem VFS op]
------> target (e.g. an off-site machine).
[transid acks going back]
STREAM = generic file descriptor. e.g. regular file, socket,
fifo, pipe, whatever. Half or full duplex.
This ASCII art does not at all look clear on the reader I am using....?
Eventually (not in two weeks) the journaling layer will make these acked
transaction ids available to any journal-aware VFS filesystem allowing
the filesystem to leverage the kernel's journaling layer for its own use
and/or to control the underlying filesystem's own management of commits
to physical storage. I also intend to use the journaling layer, with
suitable additional cache coherency protocols, to handle filesystem
synchronization in a clustered environment. In particular, an ability
to do high-level cache-coherent replication that would be immune to
catastrophic corruption rather then block-device-level replication which
tends to propogate corrupting events. As you can see, I have *BIG* plans
for the journaling layer over the next few years.
I think referring to Journaling is a bit misleading as these features
will not help the traditional case of recovery after power loss on a
file system (all buffered data would be lost as it is in memory). It
seems to be more of an integration of transactions into the VFS layer,
so that the kernel will use a Transaction topology when reading/writting
data. I am guessing that the actual file system/remote file system will
do any 'journaling' and indicate when transactions are committed etc.
How will non-journaling file systems work? Will this inter operate with
anything else that is not DragonFlyBSD (if it doesn't it makes remote
backup less useful).
Could do all of this from a user application (a lot of distributed file
system's work from user land I think)?