Re: Binary Updates for DragonFly

[Matthias Schmidt <schmidtm@xxxxxxxxxxxxxxxxxxxxxxxxx>]

Hi Matthew,

* Matthew Dillon wrote:
>     Here's my opinion on binary updates.  I considered two scenarios:
> 
>     (1) Distribute the object tree and use the normal make installkernel,
> 	installworld, and make upgrade mechanisms to update.
> 
>     (2) Distribute the binaries and write a custom installkernel and 
> 	installworld mechanism, but keep using 'make upgrade' to deal
> 	with the final clean up.
> 
>     I think option #2 is probably our best bet as for the most part everything
>     done by instalkernel or installworld can be packaged together into a 
>     single tar file that one just extracts, with only some minor renames
>     of critical library files to avoid blowing the system up.

I agree with you on that.

>     * The binary dist creates a tar file.  The name of the tar file includes
>       the rev and date... basically a unique naming convention that can be
>       used to programmatically glue binary diffs together.

Something like DragonFly-<RELEASE>-<PATCH REV> (eg DragonFly-1.10-0 for the
first tar file.  I think it would be good idea to distribute the "first/zero
tar file" together with a release (users updating manually have to
create this file themeselves, if they want to).  If we have to update files 
we can send out a binary patch relative to that first tar.

>     * Binary diffs would be relative to older tar files and use a self
>       identifying naming scheme so a script or application can verify
>       that the correct binary diffs are available and applicable to the
>       correct base file(s).

I can write a script for that task once the naming and update convention
discussion is done.

>     * The tar files can be gzipped for distribution (but diffs are based on
>       the non-gzipped version).  The gzipped tar file would remain on the
>       target host as a basis for future binary patches.  The installed
>       binaries would NOT be used as a basis for future binary patches, only
>       the tar file.

Yes, zipping the binary patch would save another amount of bandwith.

>     * A streaming patch to apply the binary diffs would be the best thing, so
>       the tar file can be left on-disk in gzipped format.  I don't know if
>       the binary diff mentioned in the thread can do that.  e.g. some sort
>       of 'gunzip < tar | binary_patch <patchfile> | tar xvpf -' sequence
>       to apply a binary patch and extract at the same time.  A similar 
>       sequence could be used to generate the new tar file.  Again, the
>       binary patch has to be against the NON gzipped tar file or it will
>       never work.

A sequence like the one above will work (slightly modified).  Generating
binary patches from tar files works flawlessly.

>     * Patches would contain the MD5 of the final result for validation
>       (ungzipped MD5).

I used SHA1 (or SHA*) regarding to the latest discussion about "MD5
considered harmfull" :)  We should send out a checksum for the original
file (the to-be-patched tar file), the binary patch itself and as you
said, for the result tar file.

>     * The tar file is installed on the machine.  Certain critical files
>       (main library files like libc.so) are named differently in the tar
>       file.  The next-to-last step after application of the tar file
>       renames them to the official names, replacing the current system
>       library files as a final step.
> 
>       e.g. in the tar libc.so would be 'libc.so.new' and then as a
>       second-to-last step after extraction libc.so.new would be renamed
>       to libc.so.

A unique naming would be also necessary here.  Eg rename all .new files
in the second-to-last step.

Regards,

	Matthias

-- 
Dipl.-Inf. Matthias Schmidt <schmidtm@mathematik.uni-marburg.de>
Dept. of Mathematics and Computer Science, Distributed Systems Group
University of Marburg, Hans-Meerwein-Strasse, 35032 Marburg, Germany
Tel: +49.6421.28 21 591, Fax: +49.6421.28 21 573, Office C4347