DragonFly On-Line Manual Pages
rwtotal(1) SiLK Tool Suite rwtotal(1)
NAME
rwtotal - Count how much traffic matched specific keys
SYNOPSIS
rwtotal {--sip-first-8 | --sip-first-16 | --sip-first-24 |
--sip-last-8 | --sip-last-16 | --dip-first-8 |
--dip-first-16 | --dip-first-24 | --dip-last-8 |
--dip-last-16 | --sport | --dport | --proto | --packets |
--bytes | --duration | --icmp-code}
[--summation] [--min-bytes=COUNT] [--max-bytes=COUNT]
[--min-packets=COUNT] [--max-packets=COUNT]
[--min-records=COUNT] [--max-records=COUNT] [--skip-zeroes]
[--no-titles] [--no-columns] [--column-separator=CHAR]
[--no-final-delimiter] [{--delimited | --delimited=CHAR}]
[--print-filenames] [--copy-input=PATH] [--output-path=PATH]
[--pager=PAGER_PROG] [--site-config-file=FILENAME]
{[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}
rwtotal --help
rwtotal --version
DESCRIPTION
rwtotal reads SiLK Flow records, bins those records by the user-
specified specified key, computes the volume per bin (record count and
sums of packets and bytes), and prints the bins and their volumes.
rwtotal reads SiLK Flow records from the files named on the command
line or from the standard input when no file names are specified and
--xargs is not present. To read the standard input in addition to the
named files, use "-" or "stdin" as a file name. If an input file name
ends in ".gz", the file will be uncompressed as it is read. When the
--xargs switch is provided, rwtotal will read the names of the files to
process from the named text file, or from the standard input if no file
name argument is provided to the switch. The input to --xargs must
contain one file name per line.
By default, rwtotal prints a bin for every possible key, even when the
volume for that bin is zero. Use the --skip-zeroes switch to suppress
the printing of these empty bins.
Use the --summation switch to include a row giving the volume for all
flow records.
The maximum key value that rwtotal supports is 16,777,215. When the
key field is --bytes or --packets, rwtotal will create a bin for all
unique values up to 16,777,214. The final bin (16,777,215) will
consist of all values greater than 16,777,214.
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option. A parameter to an option may be specified
as --arg=param or --arg param, though the first form is required for
options that take optional parameters.
One and only one of the following counting keys is required:
--sip-first-8
Key on the first 8 bits of the source IP address
--sip-first-16
Key on the first 16 bits of the source IP address
--sip-first-24
Key on the first 24 bits of the source IP address
--sip-last-8
Key on the last 8 bits of the source IP address
--sip-last-16
Key on the last 16 bits of the source IP address
--dip-first-8
Key on the first 8 bits of the destination IP address
--dip-first-16
Key on the first 16 bits of the destination IP address
--dip-first-24
Key on the first 24 bits of the destination IP address
--dip-last-8
Key on the last 8 bits of the destination IP address
--dip-last-16
Key on the last 16 bits of the destination IP address
--sport
Key on the source port.
--dport
Key on the destination port.
--proto
Key on the protocol.
--packets
Key on the number of packets in the record
--bytes
Key on the number of bytes in the record
--duration
Key on the duration of the record.
--icmp-code
Key on the ICMP type and code. This switch will assume that all
incoming records are ICMP.
The following options affect the output:
--summation
Print as the final row a total of the values in each column.
--min-bytes=COUNT
Disable printing of bins with fewer than COUNT bytes. By default,
all bins are printed.
--max-bytes=COUNT
Disable printing of bins with more than COUNT bytes. By default,
all bins are printed.
--min-packets=COUNT
Disable printing of bins with fewer than COUNT packets. By
default, all bins are printed.
--max-packets=COUNT
Disable printing of bins with more than COUNT packets. By default,
all bins are printed.
--min-records=COUNT
Disable printing of bins with fewer than COUNT flow records. By
default, all bins are printed.
--max-records=COUNT
Disable printing of bins with more than COUNT flow records. By
default, all bins are printed.
--skip-zeroes
Disable printing of bins with no traffic. By default, all bins are
printed.
--no-titles
Turn off column titles. By default, titles are printed.
--no-columns
Disable fixed-width columnar output.
--column-separator=C
Use specified character between columns and after the final column.
When this switch is not specified, the default of '|' is used.
--no-final-delimiter
Do not print the column separator after the final column. Normally
a delimiter is printed.
--delimited
--delimited=C
Run as if --no-columns --no-final-delimiter --column-sep=C had been
specified. That is, disable fixed-width columnar output; if
character C is provided, it is used as the delimiter between
columns instead of the default '|'.
--print-filenames
Print to the standard error the names of input files as they are
opened.
--copy-input=PATH
Copy all binary input to the specified file or named pipe. PATH
can be "stdout" to print flows to the standard output as long as
the --output-path switch has been used to redirect rwtotal's ASCII
output.
--output-path=PATH
Determine where the output of rwtotal (ASCII text) is written. If
this option is not given, output is written to the standard output.
--pager=PAGER_PROG
When output is to a terminal, invoke the program PAGER_PROG to view
the output one screen full at a time. This switch overrides the
SILK_PAGER environment variable, which in turn overrides the PAGER
variable. If the value of the pager is determined to be the empty
string, no paging will be performed and all output will be printed
to the terminal.
--site-config-file=FILENAME
Read the SiLK site configuration from the named file FILENAME.
When this switch is not provided, rwtotal searches for the site
configuration file in the locations specified in the "FILES"
section.
--xargs
--xargs=FILENAME
Causes rwtotal to read file names from FILENAME or from the
standard input if FILENAME is not provided. The input should have
one file name per line. rwtotal will open each file in turn and
read records from it, as if the files had been listed on the
command line.
--help
Print the available options and exit.
--version
Print the version number and information about how SiLK was
configured, then exit the application.
EXAMPLES
In the following examples, the dollar sign ("$") represents the shell
prompt. The text after the dollar sign represents the command line.
Lines have been wrapped for improved readability, and the back slash
("\") is used to indicate a wrapped line.
Group by the protocol
Group all incoming data for the first hour of March 1, 2003 by
protocol.
$ rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:00 \
--all-destination=stdout \
| rwtotal --proto --skip-zero
protocol| Records| Bytes| Packets|
1| 15622| 10695328| 147084|
6| 330726| 120536195111| 144254362|
17| 155528| 24500079| 155528|
To get the same result with rrwwuunniiqq(1), use:
$ rwfilter ... --pass=stdout \
| rwuniq --fields=proto --values=records,bytes,packets \
--sort-output
pro| Records| Bytes| Packets|
1| 15622| 10695328| 147084|
6| 330726| 120536195111| 144254362|
17| 155528| 24500079| 155528|
Group by the source Class A addresses
$ rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:00 \
--all-destination=stdout \
| rwtotal --sip-first-8 --skip-zero
sIP_First8| Records| Bytes| Packets|
10| 173164| 59950837766| 72201390|
172| 77764| 17553593| 77764|
192| 250948| 60602999159| 72277820|
Use rrwwnneettmmaasskk(1) and rrwwuunniiqq(1) to get a similar result:
$ rwfilter ... --pass=stdout \
| rwnetmask --4sip-prefix=8 \
| rwuniq --fields=sip --values=records,bytes,packets \
--sort-output --ipv6-policy=ignore
sIP| Records| Bytes| Packets|
10.0.0.0| 173164| 59950837766| 72201390|
172.0.0.0| 77764| 17553593| 77764|
192.0.0.0| 250948| 60602999159| 72277820|
Group by the final IPv4 octet
$ rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:00 \
--proto=6 --pass=stdout --daddress=192.168.x.x \
| rwtotal --dip-last-16 --skip-zero | head -5
dIP_Last16| Records| Bytes| Packets|
0. 38| 6| 4862678| 4016|
1. 14| 1| 32844| 452|
18.146| 1| 4226| 12|
21. 4| 6| 5462032| 4521|
One way to accomplish this with rwuniq is to create a new field using
PySiLK (see ppyyssiillkk(3)) and the PySiLK plug-in capability (see
ssiillkkppyytthhoonn(3). The invocation is:
$ rwfilter ... --pass=stdout \
| rwuniq --python=/tmp/dip16.py --fields=dip-last-16 \
--values=flows,bytes,packets --sort-output | head -5
dip-last-16| Records| Bytes| Packets|
0.0.0.38| 6| 4862678| 4016|
0.0.1.14| 1| 32844| 452|
0.0.18.146| 1| 4226| 12|
0.0.21.4| 6| 5462032| 4521|
where the definition of the "dip-last-16" field is given in the file
tmp/dip16.py:
import silk
mask = silk.IPAddr("0.0.255.255")
def mask_dip(r):
return r.dip.mask(mask)
register_ipv4_field("dip-last-16", mask_dip)
ENVIRONMENT
SILK_PAGER
When set to a non-empty string, rwtotal automatically invokes this
program to display its output a screen at a time. If set to an
empty string, rwtotal does not automatically page its output.
PAGER
When set and SILK_PAGER is not set, rwtotal automatically invokes
this program to display its output a screen at a time.
SILK_CLOBBER
The SiLK tools normally refuse to overwrite existing files.
Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_CONFIG_FILE
This environment variable is used as the value for the
--site-config-file when that switch is not provided.
SILK_DATA_ROOTDIR
This environment variable specifies the root directory of data
repository. As described in the "FILES" section, rwtotal may use
this environment variable when searching for the SiLK site
configuration file.
SILK_PATH
This environment variable gives the root of the install tree. When
searching for configuration files, rwtotal may use this environment
variable. See the "FILES" section for details.
FILES
${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/local/share/silk/silk.conf
/usr/local/share/silk.conf
Possible locations for the SiLK site configuration file which are
checked when the --site-config-file switch is not provided.
SEE ALSO
rrwwaaddddrrccoouunntt(1), rrwwnneettmmaasskk(1), rrwwssttaattss(1), rrwwuunniiqq(1), ppyyssiillkk(3),
ssiillkkppyytthhoonn(3), ssiillkk(7)
BUGS
rwtotal replicates some functionality in rrwwuunniiqq(1) (most notably when
rwuniq checks by port or protocol), but the implementations differ:
rwtotal uses an array instead of a hash-table, so access is faster, the
output is always sorted, and the output includes keys with a value of
zero. The use of an array prevents rwtotal from using the complete IP
address the way rwuniq does, but it also ensures that rwtotal will not
run out of memory.
When used in an IPv6 environment, rwtotal will process every record as
long as the IP address is not part of the key. When aggregating by the
IP address, rwtotal converts IPv6 flow records that contain addresses
in the ::ffff:0:0/96 prefix to IPv4 and processes them. IPv6 records
having addresses outside of that prefix are silently ignored. rwtotal
will not be modified to support IPv6 addresses; instead, users should
use rrwwuunniiqq(1) (maybe combined with rrwwnneettmmaasskk(1)).
rwtotal is also similar to rrwwaaddddrrccoouunntt(1) and rrwwssttaattss(1).
SiLK 3.11.0.1 2016-02-19 rwtotal(1)