DragonFly On-Line Manual Pages

rwpdedupe(1)                    SiLK Tool Suite                   rwpdedupe(1)

NAME
       rwpdedupe - Eliminate duplicate packets collected by several sensors

SYNOPSIS
         rwpdedupe { --first-duplicate | --random-duplicate[=SCALAR] }
               [--threshold=MILLISECONDS] FILE... > OUTPUT-FILE

         rwpdedupe --help

         rwpdedupe --version

DESCRIPTION
       Detects and eliminates duplicate records from ttccppdduummpp(1) capture files.
       Duplicate records are defined as having timestamps within a user-
       configurable time of each other.  In addition, their Ethernet (OSI
       layer 3) headers must match.  If they are not IP packets, then their
       entire Ethernet payload must match.  If they are IP packets, then their
       source and destination addresses, protocol, and IP payload must match.

OPTIONS
       Option names may be abbreviated if the abbreviation is unique or is an
       exact match for an option.  A parameter to an option may be specified
       as --arg=param or --arg param, though the first form is required for
       options that take optional parameters.

       --threshold=MILLISECONDS
           Set the maximum number of milliseconds which may elapse between two
           packets and still have those packets be detected as duplicates.
           Default 0 (exact timestamp match).  Must be a value between 0 and
           1,000,000 milliseconds.

       One and only one of the following switches is required:

       --first-duplicate
           When selecting between multiple duplicate packets, always choose
           the packet with the earliest timestamp.  Not compatible with
           --random-duplicate.

       --random-duplicate
       --random-duplicate=SCALAR
           Select a random packet from the list of duplicate packets.  SCALAR
           is a random number seed, so that multiple runs can produce
           identical results.

       --help
           Print the available options and exit.

       --version
           Print the version number and information about how SiLK was
           configured, then exit the application.

EXAMPLES
       In the following example, the dollar sign ("$") represents the shell
       prompt.  The text after the dollar sign represents the command line.
       Lines have been wrapped for improved readability, and the back slash
       ("\") is used to indicate a wrapped line.

       Given tcpdump files data1.tcp and data2.tcp, detect and eliminate
       duplicate packets which occur within one second of each other (when
       choosing which timestamp to output, pick one randomly.)  Store the
       result file in out.tcp.

        $ rwpdedupe --threshold=1000 --random-duplicate \
               data1.tcp data2.tcp > out.tcp

SEE ALSO
       mmeerrggeeccaapp(1), ttccppdduummpp(1), ppccaapp(3)

NOTES
       mmeerrggeeccaapp(1) can be used to merge two tcpdump capture files without
       eliminating duplicate packets.

SiLK 3.11.0.1                     2016-02-19                      rwpdedupe(1)