cvs commit: src/sys/sys journal.h src/sbin/jscan Makefile dump_debug.c dump_mirror.c dump_output.c dump_record.c jfile.c jscan.8 jscan.c jscan.h jstream.c subs.c

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxxxxx>]

dillon      2005/09/05 23:42:44 PDT

DragonFly src repository

  Modified files:
    sys/sys              journal.h 
    sbin/jscan           Makefile dump_debug.c dump_mirror.c 
                         jfile.c jscan.8 jscan.c jscan.h jstream.c 
                         subs.c 
  Added files:
    sbin/jscan           dump_output.c dump_record.c 
  Log:
  Rework and expand the algorithms in JSCAN, part 1/2.  Implement a new
  infrastructure in the userland jscan program to create a framework for
  supporting the following:
  
  * Pipe-throughs (without storing data in a buffering file).
  * File-based buffering
  * Streamed replication of journaled data.
  * Sequenced output files limited to a specific size (to match against
    backup media)
  * Transaction id tracking to support journal restarts and resynchronization.
    This is needed to make off-site and networked backups reliable.
  * Decoupled journal operations.  e.g. (once finished) we will be able to run
    a jscan in realtime to record journal data, and another one in batch to
    take that data and use it to maintain a mirror or to send it to another
    machine in a restartable and resynchronizable manner.
  * Infrastructure to be able to scan a journal forwards or backwards by
    an arbitrary number of raw records and maintain a mirror in both
    the forwards (redo) and backwards (undo) direction based on that ability.
  * Infrastructure that will aid in the preferred backup methodology of
    maintaining a real time mirror and UNDO/REDO journal on a 100% full
    backup partition such that one is able to free space on the partition
    simply by removing the oldest journaling file.
  * Infrastructure that will make it easier to solve the overlapping
    meta-transaction problem illustrated below:
  
        current synchronization point
                       V
    [ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ]	<= Raw records
     A     				<= meta transaction A
        B        B     B	B		<= meta transaction B
                    C              C  C	<= meta transaction C
                             D  D         <= meta transaction D
  	 E  E                           <= meta transaction E
  
    As you can see, the point at which a journal is 'broken' could
    be right smack in the middle of several meta transactions at
    once, due to the fact that meta transactions are laid out as
    their records can be generated and multiple processes could be
    modifying the filesystem at the same time.
  
    If we are attempting to mirror this data, only transactions A and E
    will have been completed and issued to the mirror.  If the journal
    is broken at the sychnronization point and then restarted, in
    order to pick up where we left off we have to scan existing
    journal records all the way back to the first 'B' in order to
    recover transactions B and C, then continue recording the journal
    from the synchronization point until those transactions are complete
    and can be issued to the mirror.  The difficulty here is that we
    have to carefully track exactly what needs to be synchronized.
    In recovering transaction 'B' and 'C' we do not want to reissue
    transaction E, for example, which had already been completed prior to
    the break.
  
    The problem exists in both the forwards and reverse direction, and
    we need to be able to go in both directions to control the 'snapshot'
    view the mirror is presenting to us.
  
  NOTE: this is just part 1/2.  This code and the new options are
  NON WORKING at the moment.
  
  Revision  Changes    Path
  1.8       +2 -0      src/sys/sys/journal.h
  1.3       +2 -1      src/sbin/jscan/Makefile
  1.4       +5 -3      src/sbin/jscan/dump_debug.c
  1.5       +2 -68     src/sbin/jscan/dump_mirror.c
  1.6       +537 -75   src/sbin/jscan/jfile.c
  1.6       +224 -11   src/sbin/jscan/jscan.8
  1.5       +327 -48   src/sbin/jscan/jscan.c
  1.6       +61 -28    src/sbin/jscan/jscan.h
  1.5       +19 -174   src/sbin/jscan/jstream.c
  1.7       +18 -0     src/sbin/jscan/subs.c


http://www.dragonflybsd.org/cvsweb/src/sys/sys/journal.h.diff?r1=1.7&r2=1.8&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/Makefile.diff?r1=1.2&r2=1.3&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/dump_debug.c.diff?r1=1.3&r2=1.4&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/dump_mirror.c.diff?r1=1.4&r2=1.5&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/jfile.c.diff?r1=1.5&r2=1.6&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/jscan.8.diff?r1=1.5&r2=1.6&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/jscan.c.diff?r1=1.4&r2=1.5&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/jscan.h.diff?r1=1.5&r2=1.6&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/jstream.c.diff?r1=1.4&r2=1.5&f=u
http://www.dragonflybsd.org/cvsweb/src/sbin/jscan/subs.c.diff?r1=1.6&r2=1.7&f=u