Re: [issue952] system hang with rsync

[Vincent Stemen <vince.dragonfly@xxxxxxxxxxx>]

On Fri, Feb 29, 2008 at 11:20:38AM -0800, Matthew Dillon wrote:
> 
> :On Sat, Feb 23, 2008 at 01:00:05PM -0800, Matthew Dillon wrote:
> :>     If you sync every second on the server while running your test, does
> :>     it still hang?
> :
> :Sorry it is taking so long.  I will try to get on this and run this test
> :in the next day or two.
> 
>     If the sync test succeeds in preventing the crash then I have a pretty
>     good idea where the issue might be, but a crash dump would tell
>     me precisely.
> 
>     If that doesn't work out, try to describe the rsyncd server setup
>     (approximate number of files and directories being rsynced, approximate
>     size of the files, amount of memory the machine has, etc) and I will
>     try to reproduce it.

Hi Matt.

I finally ran the tests today.  Sorry it took so long to get to it.

The sync did not prevent it.  It is not actually crashing, but just
hanging.  After I suspend or kill the rsync process on the client
machine (running NetBSD), the server stays hung for several minutes and
then eventually comes back alive.  Top never shows any process load.

I also tried syncing every .1 seconds.  Same result but it gave me
a more fine grained resolution of the freeze pattern.  I printed a '.'
for every sync.  The moment I run the rsync command on the client it
freezes (no more dots).  Some of the time it will freeze for a few
seconds, print a few more dots, freeze again for much longer, sometimes
print another dot or two and then stay frozen.  I also tried nicing the
sync to -10.  Same result.

As I mentioned, it is only with the '--delete' rsync option.  The freeze
takes place while deleting large files and seems to only be when there
are more than a certain number of them to delete.  When I tested a while
back with fewer files to delete, it did not do it.  I don't know where
the threshhold is of how many files triggers it.  However, the freeze
seems to always be while deleting the first file.  After killing it and
waiting several minutes for it to come back alive, the first file in the
list does seem to be deleted.

The directory size on the DragonFly server is 45GB with most of the file
sizes ranging between 2GB and 4GB.  I can provide you with the full
directory listing of both the server and the client if you like.  Either
email it directly to you or post it to the list.  Let me know.

There are 36 files on the server.
Here is a sample listing of a few of the files.

-rw-r--r--  1 vince  wheel  -        356 Apr 20  2007 Earth Revealed: Introductory Geology [101].info
-rw-r--r--  1 vince  wheel  - 1961564096 Apr 21  2007 Earth Revealed: Introductory Geology [101].mpeg
-rw-r--r--  1 vince  wheel  -        490 May 26  2007 From the Earth to the Moon (1968) [04].info
-rw-r--r--  1 vince  wheel  - 3926833088 May 29  2007 From the Earth to the Moon (1968) [04].mpeg
-rw-r--r--  1 vince  wheel  -  915902498 Jul 15  2007 Man vs. Wild (Everglades).avi
-rw-r--r--  1 vince  wheel  -        387 Jul 13  2007 Man vs. Wild (Everglades).info

Server machine:
    kern.osrelease: 1.11.0-DEVELOPMENT
    kern.osrevision: 200708
    hw.physmem: 528859136
    hw.model: Intel(R) Pentium(R) 4 CPU 1.60GHz

    # swapinfo
    Device          1K-blocks     Used    Avail Capacity  Type
    /dev/ad0s1b        511936       40   511896     0%    Interleaved