rsync vs. cvsup benchmarks

[Vincent Stemen <vs1@xxxxxxx>]

I have some benchmark test results comparing rsync to cvsup.  I did 12 client
side tests over the last week.  5 against TheShell.com, 3 against AllBSD.org,
and 4 against chlamydia.fs.ei.tum.de.  All tests were mirroring the DragonFly
BSD source repository.  The tests were done with various aged repositories at
different times of the day and night, some with compression on and some with it
off.  Each test was done by unpacking two identical copies of a given aged
repository, one to run the cvsup test on and one to run the rsync test on.
Then the rsync and cvsup test was run back to back.


===============
* Environment *
===============

All tests were done in the following environment:

DragonFly 1.10.1-RELEASE system with a 1.11.0-DEVELOPMENT kernel.
CPU:                Intel(R) Pentium(R) 4 CPU 1.60GHz
Inernet connection: Cox cable
rsync version:      2.6.9
rsync directories:  CVSROOT, doc, and src
rsync flags:        --archive --hard-links --delete --verbose
                  + --compress (on tests with compression on)
cvsup version:      SNAP_16_1h
cvsup supfile:      DragonFly-cvs-supfile that comes with DragonFly BSD
cvsup file collections:
    dragonfly-cvs-root
    dragonfly-cvs-src
    dragonfly-cvs-doc


===========
* Summary *
===========

On average:
===========
    cvsup took 3.76 times as long as rsync making rsync 276% faster.
    rsync consumed 13.5% of the cpu time on updates to an existing repository.
    cvsup consumed 33.7% of the cpu time on updates to an existing repository.

Minimum performance difference: 
    cvsup took from 1.34 times as long as rsync.
Maximum performance difference: 
    cvsup took 9.1 times as long as rsync

rsync's best performance was on repositories that are less than a week old,
where there was not a large number of updates.  In these cases it was usually
from 4 to 7 times faster than cvsup, except for one test from allbsd.org where
rsync was only about 40% faster.  

On a Full download of the entire repository, where the tools are not having to
check for file differences, the CPU loads were so low and similar enough that
I would consider the differences between rsync and cvsup insignificant, so
I excluded those two tests from the cpu load average calculation above.
However, rsync was still more then twice as fast at completing the entire
download.

cvsup, on average, consumed 250% of the load that rsync consumed but did not
outperform rsync on one single test.  Of course, this is client side only.
I will leave it up to somebody else to do server side load tests.

My conclusions:
===============

The results are dramatic, with rsync performing hundreds of percent faster on
average while only loading the processor on the client side a little over
a third as much as cvsup.  Either the performance claims about cvsup being
faster than rsync are based on theory without real world testing or cvsup has
gotten a lot slower or rsync has gotten a lot faster than in the past.

For those who are concerned about the validity of these results without
including server side load tests and tests under bandwidth congestion
conditions, here are my thoughts on the matter.

    No matter where a bottleneck exists in the transfer, whether it is server
    side load, client side load, or bandwidth limits, you are going to
    experience similar loss of throughput.  So, I would consider client side
    only tests reasonably valid at determining overall average performance
    differences when both tools are run back to back under the same conditions.

    If rsync is shifting more of the processing load to the server than cvsup
    or consuming more bandwidth, well, it can bog the server down or consume
    more bandwidth by at least an *extra* 150% or more before it even slows
    down enough to match cvsup's performance on the average.  And, of course,
    that is only if the bandwidth or server load is hitting saturation.


================
* Test Results *
================

repository age: about 1.5 hours
site:           TheShell.com
compression:    off
                (Probably made no difference since there were
                 apparently no updates)

date:           Thu Jan 17 20:29:40 CST 2008
rsync time:     1.34s user 3.41s system 13% cpu 34.846 total

date:           Thu Jan 17 20:17:12 CST 2008
cvsup time:     54.17s user 26.03s system 36% cpu 3:40.77 total

=================================================
cvsup took 6.33 times as long as rsync
rsync was 533% faster
cvsup consumed 2.77 times the cpu load of rsync
=================================================


repository age: about 2 days
site:           TheShell.com
compression:    on

date:           Thu Jan 17 18:50:11 CST 2008
rsync time:     8.29s user 13.31s system 17% cpu 2:03.07 total

date:           Thu Jan 17 19:13:10 CST 2008
cvsup time:     155.08s user 69.40s system 40% cpu 9:14.73 total

=================================================
cvsup took 4.5 times as long as rsync
rsync was 350% faster
cvsup consumed 2.35 times the cpu load of rsync
=================================================


repository age: about 2.5 days
site:           chlamydia.fs.ei.tum.de
compression:    on

date:           Tue Jan 29 12:06:16 CST 2008
rsync time:     2.26s user 5.44s system 15% cpu 50.987 total

date:           Tue Jan 29 12:07:12 CST 2008
cvsup time:     119.74s user 62.39s system 39% cpu 7:44.44 total

==================================================
cvsup took 9.1 times as long as rsync
rsync was 810% faster
cvsup consumed 2.6 times the cpu load of rsync
==================================================


repository age: 3 days
site:           AllBSD.org
compression:    on

date:           Tue Jan 29 20:06:24 CST 2008
rsync time:     18.09s user 25.35s system 12% cpu 5:40.84 total

date:           Tue Jan 29 20:12:10 CST 2008
cvsup time:     120.81s user 63.78s system 38% cpu 7:55.77 total

=================================================
cvsup took 1.4 times as long as rsync
rsync was 40% faster
cvsup consumed 3.17 times the cpu load of rsync
=================================================


repository age: about 5 days
site:           AllBSD.org
compression:    on

date:           Mon Jan 21 01:52:27 CST 2008
rsync time:     5.79s user 9.86s system 12% cpu 2:02.08 total

date:           Mon Jan 21 02:06:15 CST 2008
cvsup time:     158.18s user 74.17s system 24% cpu 15:37.18 total

=================================================
cvsup took 7.67 times as long as rsync
rsync was 667% faster
cvsup consumed 2 times the cpu load of rsync
=================================================


repository age: 8 days
site:           chlamydia.fs.ei.tum.de
compression:    on

date:           Sat Jan 26 22:21:30 CST 2008
rsync time:     15.45s user 20.97s system 14% cpu 4:13.69 total

date:           Sat Jan 26 22:31:03 CST 2008
cvsup time:     161.53s user 86.40s system 30% cpu 13:45.23 total

=================================================
cvsup took 3.26 times as long as rsync
rsync was 226% faster
cvsup consumed 2.14 times the cpu load of rsync
=================================================


repository age: 11.7 days
site:           AllBSD.org
compression:    on

date:           Tue Jan 29 13:26:08 CST 2008
rsync time:     9.61s user 16.59s system 13% cpu 3:09.76 total

date:           Tue Jan 29 13:29:22 CST 2008
cvsup time:     61.22s user 35.91s system 38% cpu 4:14.15 total

===============================================
cvsup took 1.34 times as long as rsync
rsync was 34% faster
cvsup consumed 2.92 times the cpu load of rsync
===============================================


repository age: 13 days
site:           chlamydia.fs.ei.tum.de
compression:    off

date:           Mon Jan 28 17:49:09 CST 2008
rsync time:     13.54s user 21.71s system 8% cpu 6:33.04 total

date:           Mon Jan 28 17:55:47 CST 2008
cvsup time:     158.97s user 86.89s system 26% cpu 15:17.05 total

=================================================
cvsup took 2.33 times as long as rsync
rsync was 133% faster
cvsup consumed 3.25 times the cpu load of rsync
=================================================


repository age: 13 days
site:           chlamydia.fs.ei.tum.de
compression:    on

date:           Mon Jan 28 18:52:29 CST 2008
rsync time:     15.48s user 21.56s system 15% cpu 3:58.37 total

date:           Mon Jan 28 18:56:33 CST 2008
cvsup time:     158.32s user 85.46s system 39% cpu 10:24.99 total

=================================================
cvsup took 2.6 times as long as rsync
rsync was 160% faster
cvsup consumed 2.6 times the cpu load of rsync
=================================================


repository age: about 6 months  (last update 2007-07-18)
site:           TheShell.com
compression:    on

date:           Wed Jan 23 03:17:57 CST 2008
rsync time:     19.36s user 23.45s system 16% cpu 4:17.03 total

date:           Wed Jan 23 03:22:19 CST 2008
cvsup time:     111.55s user 55.78s system 27% cpu 9:59.97 total

=================================================
cvsup took 2.33 times as long as rsync
rsync was 133% faster
cvsup consumed 1.69 times the cpu load of rsync
=================================================


repository age: Full download from empty repository
site:           TheShell.com
compression:    on

date:           Sun Jan 27 11:43:54 CST 2008
rsync time:     28.34s user 27.31s system 7% cpu 11:58.62 total

date:           Sun Jan 27 11:55:58 CST 2008
cvsup time:     73.28s user 79.70s system 9% cpu 25:48.36 total

=================================================
cvsup took 2.15 times as long as rsync
rsync was 115% faster
=================================================


repository age: Full download from empty repository
site:           TheShell.com
compression:    off

date:           Sun Jan 27 13:59:31 CST 2008
rsync time:     13.06s user 31.54s system 2% cpu 35:11.62 total

date:           Sun Jan 27 14:34:48 CST 2008
cvsup time:     73.56s user 95.48s system 3% cpu 1:16:49.63 total

=================================================
cvsup took 2.18 times as long as rsync
rsync was 118% faster
=================================================