Re: Journaling layer update - any really good programmer want to start working on the userland journal scanning utility ? You need to have a LOT of time available!

[Miguel Mendez <flynn@xxxxxxxxxxxxxxxxxx>]

On Thu, 3 Mar 2005 22:06:38 -0800 (PST)
Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx> wrote:

Argh, so much interesting stuff to work on and so little time. Right
now I have some items on my TO-DO list one of them is helping Devon
with the GTK+ frontend of the installer, but I'm going to look at this
during the weekend, as I consider it a very interesting area of the
system. Maybe if somebody else joins in I could help a bit.

>     * Writing out the UNDO information.  UNDO information is what makes a
>       journal reversable, which is one of the big ticket items that I want
>       to support, but it requires writing out the prior contents of
>       data blocks, prior uid, gid, time info, and so forth.

I have a question about this. How far can the UNDO system go? Will it
be some kind of redo-log (think VxFS here) or will it allow arbitrary
undo levels? Maybe it sounds crazy, but you could have 'unlimited' undo
where unlimited amounts to available free disk space. It complicates
the free block management a bit, but it's certainly doable. Basically
the journal code should have an undo bitmap. It can allocate any free
data block to store undo information. As soon as a datablock is needed
by the sytem the oldest undo datablock gets used to store data. The
filesystem doesn't need to know about it, just the journaling code,
although balloc now needs to go through the journal code to make sure
the undo blocks bitmap gets updated as well. This bitmap could live
anywhere in the disk.

>     I am doing all the kernel work, but I am looking for someone to help
>     engineer and write the user utility that actually does something real 
>     with the generated journal!  Anyone interested in doing this?   We
>     want a utility that is capable of:
> 
>     * extracting a file subhierarchy and generating a mirror of the filesystem.
>     * extracting a file subhierarchy and generating human readable output
>       showing the audit trail of all changes made within the subhierachy.
>     * extracting a file subhierarchy and generating a new raw journal 
>       containing only that subhierarchy.
>     * extracting deleted files by name ('undelete' realized!)
>     * extracting a file subhierarchy and generating a mirror that is
>       'as of' a particular date in the past.

If someone else steps in I'll try to help here.

> 		-- Technical Journal Record Format Details -
> 
>     The journal record format is in sys/journal.h.  It's quite straightforward
>     but it IS a multi-layer recursive record format.  The first layer is a
>     virtual stream layer (needed because multiple entities may be 
>     writing out transactions to the journal simultaniously).   Virtual
>     streams are typically short-lived entities that represent transactions.
>     One transaction per virtual stream.  The virtual stream layer is
>     controlled by the journal_rawrecbeg and journal_rawrecend structure
>     (designed so a utility program can scan the journal forwards or
>     backwards). 

Sounds very well thought out, Matt. Definitely worth a look. I'll try
to be in IRC during the weekend so maybe we can come up with something.

Cheers,
-- 
Miguel Mendez <flynn@xxxxxxxxxxxxxxxxxx>
http://www.energyhq.es.eu.org
PGP Key: 0xDC8514F1