DragonFly On-Line Manual Pages
rwsettool(1) SiLK Tool Suite rwsettool(1)
NAME
rwsettool - Operate on IPset files to produce a new IPset
SYNOPSIS
rwsettool { --union | --intersect | --difference
| --mask=NET_BLOCK_SIZE | --fill-blocks=NET_BLOCK_SIZE
| --sample {--size=SIZE | --ratio=RATIO} [--seed=SEED] }
[--output-path=OUTPUT_PATH] [--record-version=VERSION]
[--invocation-strip]
[--note-strip] [--note-add=TEXT] [--note-file-add=FILE]
[--compression-method=COMP_METHOD] [INPUT_SET ...]
rwsettool --help
rwsettool --version
DESCRIPTION
rwsettool performs a single operation on one or more IPset file(s) to
produce a new IPset file. The operations that rwsettool provides are
union, intersection, difference, masking, and sampling. Details are
provided in the "OPTIONS" section.
rwsettool reads the IPsets specified on the command line; when no
IPsets are listed, rwsettool attempts to read an IPset from the
standard input. The strings "stdin" or "-" can be used as the name of
an input file to force rwsettool to read from the standard input. The
resulting IPset is written to the location specified by the
--output-path switch or to the standard output if that switch is not
provided. Using the strings "stdout" or "-" as the argument to
--output-path causes rwsettool to write the IPset to the standard
output. rwsettool will exit with an error if an attempt is made to
read an IPset from the terminal or write an IPset to the terminal.
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option. A parameter to an option may be specified
as --arg=param or --arg param, though the first form is required for
options that take optional parameters.
Operation Switches
At least one of the following operation switches must be provided:
--union
Perform the set union operation: The resulting IPset will contain
an IP if that IP was present in any of the input IPsets.
--intersect
Perform the set intersection operation: The resulting IPset will
contain an IP if that IP was present in all of the input IPsets.
--difference
Perform the set difference (relative complement) operation: The
resulting IPset will contain an IP if that IP was present in first
IPset and not present in any of the subsequent IPsets.
--mask=NET_BLOCK_SIZE
Perform a (sparse) masking operation: The least significant
32-NET_BLOCK_SIZE or 128-NET_BLOCK_SIZE bits of each IP in every
input IPset is set to zero, and the resulting IPset contains the
union of these IPs. That is, the result contains one IP for each
CIDR block of size NET_BLOCK_SIZE. NET_BLOCK_SIZE should be value
between 1 and 32 for IPv4 sets, and between 1 and 128 for IPv6
sets. Contrast with --fill-blocks.
--fill-blocks=NET_BLOCK_SIZE
Perform a masking operation that produces completely full blocks:
The least significant 32-NET_BLOCK_SIZE or 128-NET_BLOCK_SIZE bits
of each IP in every input IPset is set to zero. To create the
output, each IP is modified to be a completely full NET_BLOCK_SIZE
CIDR block.
--sample
Select a random sample of IPs from the input IPsets. The size of
the subset must be specified by either the --size or --ratio
switches described below. In the case of multiple input IPsets,
the resulting IPset is the union of all IP addresses sampled from
each of the input IPsets.
Sampling Switches
These switches control how records are sampled by the --sample
operation.
--size=SIZE
Select a random sample containing SIZE randomly selected records
from each input IPset. If the input set is smaller than SIZE, all
input IPs will be selected from that IPset.
--ratio=RATIO
Select a random sample where the selection probability for each
record of each input set is RATIO, specified as a decimal number
between 0.0 and 1.0. The exact size of the subset selected from
each file will vary between different runs with the same data.
--seed=SEED
Seed the pseudo-random number generator with value SEED. By
default, the seed will vary between runs. Seeding with specific
values will produce repeatable results given the same input sets.
Output Switches
These switches control the output:
--output-path=OUTPUT_PATH
Write the resulting IPset to OUTPUT_PATH. If this switch is not
provided, rwsettool will attempt to write the IPset to the standard
output, unless it is connected to a terminal.
--record-version=VERSION
Specify the format of the IPset records that are written to the
output. Valid values are 0, 2, 3, and 4. When the switch is not
provided, the SILK_IPSET_RECORD_VERSION environment variable is
checked for a version. A VERSION of 2 creates a file compatible
with SiLK 2.x, and it can only be used for IPsets containing IPv4
addresses. A VERSION of 3 creates a file that can only be read by
SiLK 3.0 or later. A VERSION of 4 creates a file that can only be
read by SiLK 3.7 or later. Version 4 files are smaller than
version 3 files. The default VERSION is 0, which uses version 2
for IPv4 IPsets and version 3 for IPv6 IPsets.
--invocation-strip
Do not record any command line history; that is, do not copy the
invocation history from the input files to the output file, and do
not record the current command line invocation in the output.
--note-strip
Do not copy the notes (annotations) from the input files to the
output file. Normally notes from the input files are copied to the
output.
--note-add=TEXT
Add the specified TEXT to the header of the output file as an
annotation. This switch may be repeated to add multiple
annotations to a file. To view the annotations, use the
rrwwffiilleeiinnffoo(1) tool.
--note-file-add=FILENAME
Open FILENAME and add the contents of that file to the header of
the output file as an annotation. This switch may be repeated to
add multiple annotations. Currently the application makes no
effort to ensure that FILENAME contains text; be careful that you
do not attempt to add a SiLK data file as an annotation.
--compression-method=COMP_METHOD
Specify how to compress the output. When this switch is not given,
output to the standard output or to named pipes is not compressed,
and output to files is compressed using the default chosen when
SiLK was compiled. The valid values for COMP_METHOD are determined
by which external libraries were found when SiLK was compiled. To
see the available compression methods and the default method, use
the --help or --version switch. SiLK can support the following
COMP_METHOD values when the required libraries are available.
none
Do not compress the output using an external library.
zlib
Use the zzlliibb(3) library for compressing the output, and always
compress the output regardless of the destination. Using zlib
produces the smallest output files at the cost of speed.
lzo1x
Use the lzo1x algorithm from the LZO real time compression
library for compression, and always compress the output
regardless of the destination. This compression provides good
compression with less memory and CPU overhead.
best
Use lzo1x if available, otherwise use zlib. Only compress the
output when writing to a file.
--help
Print the available options and exit.
--version
Print the version number and information about how SiLK was
configured, then exit the application.
EXAMPLES
In the following examples, the dollar sign ("$") represents the shell
prompt. The text after the dollar sign represents the command line.
Assume the following IPsets:
A.set = { 1, 2, 4, 6 }
B.set = { 1, 3, 5, 7 }
C.set = { 1, 3, 6, 8 }
D.set = { } (empty set)
Then the following commands will produce the following result IPsets:
+---------------------------------+----------------------------+
| OPTIONS | RESULT |
+---------------------------------+----------------------------+
| --union A.set B.set | { 1, 2, 3, 4, 5, 6, 7 } |
| --union A.set C.set | { 1, 2, 3, 4, 6, 8 } |
| --union A.set B.set C.set | { 1, 2, 3, 4, 5, 6, 7, 8 } |
| --union C.set D.set | { 1, 3, 6, 8 } |
| --intersect A.set B.set | { 1 } |
| --intersect A.set C.set | { 1, 6 } |
| --intersect A.set B.set C.set | { 1 } |
| --intersect A.set D.set | { } |
| --difference A.set B.set | { 2, 4, 6 } |
| --difference B.set A.set | { 3, 5, 7 } |
| --difference A.set B.set C.set | { 2, 4 } |
| --difference C.set B.set A.set | { 8 } |
| --difference C.set D.set | { 1, 3, 6, 8 } |
| --difference D.set C.set | { } |
+---------------------------------+----------------------------+
Sampling yields variable results, but here some example runs:
+---------------------------------+----------------------------+
| COMMAND | RESULT |
+---------------------------------+----------------------------+
| --sample -size 2 A.set | { 1, 4 } |
| --sample -size 2 A.set | { 1, 6 } |
| --sample -size 3 A.set | { 2, 4, 6 } |
| --sample -size 2 A.set B.set | { 1, 2, 5, 7 } |
| --sample -size 2 A.set B.set | { 3, 4, 5, 6 } |
| --sample -size 2 A.set B.set | { 1, 4, 5 } |
| --sample -ratio 0.5 A.set | { 2, 6 } |
| --sample -ratio 0.5 A.set | { 4 } |
| --sample -ratio 0.5 A.set B.set | { 1 } |
| --sample -ratio 0.5 A.set B.set | { 2, 3, 5, 6, 7 } |
+---------------------------------+----------------------------+
These examples demonstrate some important points about sampling from
IPsets:
o When using --size, an exact number of items is selected from each
input set.
o When using --size with multiple input sets, the number of records
in the output set may not be (num_input_sets*size) in all cases.
o When using --ratio, the number of items sampled is not stable
between runs.
Given an IPset containing the three IPs
10.1.1.1
10.1.1.2
10.1.3.1
specifying --mask=24 will produce an IPset containing two IPs:
10.1.1.0
10.1.3.0
while specifying --fill-blocks=24 will produce an IPset containing 512
IPs:
10.1.1.0/24
10.1.3.0/24
Suppose the IPset file mixed.set contains IPv4 and IPv6 addresses. To
create an IPset file that contains only the IPv4 addresses, intersect
mixed.set with an IPset that contains ::ffff:0:0/96.
$ echo '::ffff:0:0/96' | rwsetbuild - all-v4.set
$ rwsettool --intersect mixed.set all-v4.set > subset-v4.set
To create an IPset file that contains only the IPv6 addresses, subtract
an IPset that contains ::ffff:0:0 from mixed.set:
$ rwsettool --difference mixed.set all-v4.set > subset-v6.set
ENVIRONMENT
SILK_IPSET_RECORD_VERSION
This environment variable is used as the value for the
--record-version when that switch is not provided.
SILK_CLOBBER
The SiLK tools normally refuse to overwrite existing files.
Setting SILK_CLOBBER to a non-empty value removes this restriction.
SEE ALSO
rrwwsseett(1), rrwwsseettbbuuiilldd(1), rrwwsseettccaatt(1), rrwwffiilleeiinnffoo(1), ssiillkk(7), zzlliibb(3)
NOTES
The --record-version switch was added in SiLK 3.0. Prior to SiLK 3.6,
the only supported arguments for the switch were 2 and 3, with the
default being 3. As of SiLK 3.6, the default is 0. Version 4 was
added in SiLK 3.7.
SiLK 3.11.0.1 2016-02-19 rwsettool(1)