DragonFly On-Line Manual Pages
SHMUX(8) DragonFly System Manager's Manual SHMUX(8)
NAME
shmux - Shell Multiplexor
SYNOPSIS
shmux [ -h | -V ]
shmux [ -bBdFmpqQstv ] [ -C timeout ] [ -M max ] [ -r rcmd ] [ -S mode
] [ -e list ] [ -E list ] [ -a analyzer ] [ -A condition ] [ -o dir ] [
-P timeout ] [ -T timeout ] -c command [ - | targets... ]
DESCRIPTION
shmux is program for executing the same command on many hosts in
parallel. For each target, a child process is spawned by shmux, and a
shell on the target obtained one of the supported methods: rsh, ssh, or
sh. The output produced by the children is received by shmux and
either (optionally) output in turn to the user, and/or written to files
for later processing. The list of targets may either be specified on
the command line or be read from the standard input.
The default method may be set by using the -r option, and may be
overridden for any target by prefixing the target name by the method
and a colon. Two special methods, ssh1 and ssh2 may be used to select
the SSH protocol version 1 or 2, respectively. For the sh method, it
is up to the specified command to contact the target. In order for the
executed command to know which target it is being invoked for, the
SHMUX_TARGET environment variable is set by shmux.
Before executing the specified command, shmux will optionally ping each
target to ensure that it can be reached, and/or run a dummy test
command to make sure that the target not only is alive, but that it is
possible to cleanly execute a command on it. Both these tests are
typically run with a fairly short timeout to quickly dismiss
unavailable targets rather than waiting for the standard (longer)
network timeout.
After executing the specified command, shmux will optionally process
the output or run a user specified command to determine whether the
command was successful or resulted in an error. Optionally, the
command output is suppressed for successful targets.
Finally shmux will print out a summary of the results and list targets
with errors.
OPTIONS
-h Display a brief help message.
-V Display the version information.
-c command
Specify the command to execute on targets.
-C timeout
Specify a timeout for the command being executed on targets.
This should be a number followed by a time unit. The following
are valid time units: s(econds), m(inutes), h(our), d(ays),
w(eeks). See the PROCESS MANAGEMENT section for details on how
this is handled.
-M max Defines the maximum number of spawned processes. While there is
no real (or hard coded) limitation for this, the system
resources are typically bounded and this affects shmux in turn.
The most critical resource is the maximum number of open files a
process may have. But one should also consider the load imposed
on the system by the creation of a (potentially) large number of
processes.
-r rcmd
Defines the default method used to run a shell on targets.
-S mode
This options defines the strategy used by shmux when spawning
children to run the command on targets. If the mode is set to
"all", then processes will be spawned as fast as possible. When
set to "check", processes will be spawned as long as commands
succeed. Finally, the default mode, "one", is to spawn only one
process at first, and wait for it to succeed, at which point
shmux will switch to the "check" mode. In the "one" or "check"
modes, when the command fails on a target, shmux will stop
spawning new processes.
-F When using either the "one" or "check" spawn modes (see -S),
shmux will stop spawning new processes if the command fails on a
target. By default, shmux then waits for user input if
available, but reverts to spawning processes if such input
cannot be obtained (because the interactive mode is disabled,
either via -B or lack of a terminal). Using the -F option
changes this behaviour to force shmux to quit once all existing
processes have terminated after such a failure.
-e list
Defines which command exit codes should be considered errors and
reported as such. The list should be a comma separated list of
ranges of the form "x", "x-y", "x-" or "-y" where "x" and "y"
are numbers between 0 and 255.
-E list
Defines which command exit codes should always be output to the
screen. Codes that are considered errors (see the -e option)
are always shown, but by default, other exit codes are only
shown if the -v option is specified. This allows more
flexibility.
-a analyzer
Defines how output should be analyzed by shmux after the command
completes on a target. By default, nothing is done. Valid
options are: lnregex, lnpcre, regex, pcre and run. This option
requires -o to be used as well.
-A condition
When the -a option is used, it is also necessary to configure
the chosen analyzer.
For line based regular expressions (lnregex and lnpcre), the -A
option should be specified twice: once for the standard output
and once for the standard error output. In both cases, the
argument should be the name of a file containing a list of
regular expressions, one per line. Each regular expression must
be prefixed by the character '=' or '!'. The output of the
command is analyzed line per line by checking whether it matches
any of the specified regular expressions. If the first matching
regular expression is prefixed by '=', then the command is
considered to be successful (so far). Any other result (e.g. no
match, or a match prefixed by '!') indicates an error condition.
The default is to consider standard output normal, and standard
error output indicative of an error.
For other regular expressions (regex and pcre), the -A option
should be specified twice: once for the standard output and once
for the standard error output. This option allows defining the
expression to be applied to the output. If the condition starts
with the character '=' then a match indicates that the command
completed without error. If the condition starts with the
character '!' then a failure to match indicates that the command
completed without error. The second character of the condition
should either be '=' if the regular expression follows or '<' if
the regular expression should be read from the file which name
follows. (The '=' characters may be omitted.) A condition must
be specified for the standard output, but is optional for
standard error output. The default for the latter is to
consider any output as indicative of an error.
For the run analyzer, the -A must be specified at least once
with the name of a program to run, and optionally a second time
to specify a timeout which defaults to 15 seconds if unset.
This timeout is implemented using alarm(3) and will not
otherwise be enforced by shmux. The program will be called
after a command completes on a target unless the command exit
code is considered an error (see -e) with two arguments: the
target name and the directory specified with the -o option. The
output of the program will always be shown to the user. Exit
codes other than 0 indicate that the command failed for the
particular target.
-o dir If specified, shmux will place the output and (for normal
terminations) the exit code of the executed commands in files
under this directory. For each target, the files will be named
after the target name so as to make it easy to identify. The
files must not already exist, so it is recommended that the
directory be empty. This also means that each target must be
unique. The directory will be created if it does not already
exist.
-p Ping targets to verify they are alive before doing anything.
The target names must be unique or bad things will happen.
-P timeout
Defines the initial target ping timeout in milliseconds, see
fping(8). (Implies -p.)
-t Before executing the specified command for a target, send a
simple "echo" test and verify that the output is correct. This
step goes beyond the simple ping test as it verifies that it is
possible to get a shell on the target. If this option is
specified twice (or once when -T is used), output for failed
tests will be displayed to help the user understand what went
wrong.
-T timeout
Defines the test timeout in seconds. (Implies -t.)
-m By default the output is displayed as soon as it is received.
For multi-line outputs, this will typically result in output
from several targets being mixed. This option may be used to
keep each target output separate.
-b Show the bare output from the executed commands instead of
prefixing each line by the corresponding target name.
-B Enables batch mode which forcefully disables the interactive
mode, ignoring input from the terminal. See also -s and -F.
-s Suppress the progress status line. See also -B.
-q Used once, this will suppress the output from successful targets
(as defined by the use of options options -e, -a and -A. If
using the run analyzer, the output of all targets is suppressed.
Note that using this option effectively implies -m in most cases
as failure is ultimately determined upon completion of the
command. (Only the lnregex and lnpcre analyzers are able to
determine errors while the command is running.) When specified
twice, this option allows to completely suppress output from the
command run on targets. This also changes the default spawn
stategy to "all". Using this option (once or twice) requires
using -o.
-Q Suppress the final summary of results.
-v Display internal status messages.
-D Display internal debug messages.
EXIT CODES
shmux will optionally report the exit code of the command it runs for
each target. For the sh command, this will be the same as the command
itself, but this is the exception. For other methods, the exit code
reported by shmux really is the code returned by the program used to
spawn command for that method.
In particular, rsh typically only returns either 0 or 1 while different
implementations of ssh have different behaviours which may include
returning the actual exit code of the remotely executed command or not.
PROCESS MANAGEMENT
Children processes spawned by shmux are made process group leaders
(using setpgid(2)) in order to group all of their descendants in a
single process group which allows to easily send signals to all of
these processes. This is especially useful for the sh method which can
generate many processes locally, but has no effect on remote processes.
shmux uses pipe(2) to capture the output generated by its children
(even when this output is redirected to a file with -o) and will read
from these until they are closed (by the child as well as its
descendants, if any). Once there is nothing left to read, shmux will
wait for the child and its descendants to terminate before considering
that particular target done with.
The timeout is implemented by using alarm(3) which (unless canceled or
intercepted) will terminate the child process upon receipt but is not
inherited by descendants of that process. For that reason, shmux will
send a SIGALRM signal to the child's process group when the child
itself terminates upon such signal.
After the scheduled SIGALRM time, shmux will wait an additional 5
seconds for all of the processes in the group to terminate to give the
child a chance to handle the timeout more gracefully if desired. It
will then send a SIGTERM signal to the child's process group. After
another 5 seconds, if any process is still alive in the group, a final
SIGKILL signal is sent to the process group.
Note that only commands which terminate because of an ALRM signal or
because of a TERM or KILL signal after such signal has been issued by
shmux will be reported as having timed out in the final summary report
produced by shmux upon exiting. This is not as accurate as it should
be.
INTERACTIVE MODE
By default, shmux offers a minimal "interactive mode" while running: it
reads commands from the terminal and acts upon them accordingly. This
can be disabled using the -B option. The following commands are
currently recognized:
q Prevent shmux from spawning any more children, wen for active
children to complete, and then quit.
Q Quit immediately (e.g. abort), not waiting for any active
children to terminate. This isn't recommended as children may
be left running unattended. A better alternative is to press
control-C which will be received by shmux and all running
children. These will typically abort, and shmux terminate in a
cleaner fashion.
space Initiate a pause, this prevents shmux from spawning any more
children to do its work. The following three commands may be
used to resume work.
1 This is equivalent to specifying "-S more" on the command line,
but may be used at any time.
return This is equivalent to specifying "-S check" on the command line,
but may be used at any time.
+ This is equivalent to specifying "-S all" on the command line,
but may be used at any time.
F This toggles the failure mode for the "one" and "check" spawn
modes (see -S) between "pause" and "quit" (see -F).
p Display the list of pending targets.
r Display the list of targets for which a children process is
currently running.
f Display the list of targets for which shmux failed to run the
command.
e Display the list of targets for which the command completed with
error(s).
s Display the list of targets for which the command completed
successfully.
a Display the list of all targets with their status.
k Allows sending a signal to a target.
v Toggles whether internal messages are displayed or not.
D Toggles whether debug messages are displayed or not.
ENVIRONMENT VARIABLES
shmux will use the following environment variables if set:
SHMUX_MAX
Specifies the maximum number of processes to spawn
simultaneously. If the -M option is specified, it overrides
this variable.
SHMUX_RCMD
Specifies the default command used to run a shell on targets.
If the -r option is specified, it overrides this variable.
SHMUX_SH
Specify the shell to use for targets using the sh method.
(Default is "/bin/sh".)
SHMUX_RSH
Specify an alternative command to "rsh" to use for targets using
the rsh method.
SHMUX_SSH, SHMUX_SSH1, SHMUX_SSH2
Specify an alternative command to "ssh" to use for targets using
the ssh, ssh1 and ssh2 method (respectively). The variable
SHMUX_SSH is used for all methods, but superseded by the other
specific variables.
SHMUX_SSH_OPTS, SHMUX_SSH1_OPTS, SHMUX_SSH2_OPTS
The variables may be used to specify replace the default "-xa"
options passed to "ssh", "ssh1" and "ssh2" (respectively). The
variable SHMUX_SSH_OPTS is used for all methods, but superseded
by the other specific variables.
Multiple arguments may be supplied, using white-space characters
to separate them from each other. You can use single- and/or
double-quotes to preserve white-spaces in an argument (but not
backslashes). Note that doubling a single-quote inside a
single-quoted string gives you a single-quote; likewise for
double-quotes (though you need to pay attention to which
quotes your shell is parsing and which quotes shmux is
parsing).
SHMUX_SPAWNMODE
Specifies the default spawn mode. If the -S option is
specified, it overrides this variable.
SHMUX_ERRORCODES
Specifies the default list of exit codes to be considered
errors. If the -e option is specified, it overrides this
variable.
SHMUX_SHOWCODES
Specifies the default list of exit codes to be always display.
If the -E option is specified, it overrides this variable.
EXIT STATUS
The following exit values are returned:
0 indicates that shmux was able to complete its work. However,
this does not mean that all targets successfully ran the
command, nor that shmux ran without any error.
1 indicates shmux was cleanly interrupted before it could finish
its work.
2 indicates shmux aborted upon user request, possibly leaving some
children running.
3 indicates shmux encountered a fatal error which caused it to
abort, possibly leaving some children running.
4 indicates a syntax error or a fatal error which caused shmux to
abort before it could spawn any children.
SEE ALSO
fping(8), sh(1), rsh(1), ssh(1), regex(3), pcre(3).
AVAILABILITY
The latest official release of shmux is available on the web. The home
page is http://web.taranis.org/shmux/
AUTHOR
Christophe Kalt <kalt@taranis.org>
BUGS
All rsh and some ssh implementations effectively achieve user
authentication by using privileged ports. Since there are only 1024
such ports, there is an obvious limitation on how many connections may
be open using these programs. To make matters worse such ports may be
held for four minutes (by default) as per the TCP specification. When
using shmux, one can very quickly run out of privileged ports, at which
point failures will start happening.
All rsh and ssh implementations fail to accurately return the remote
command exit code to the caller. Typically, rsh will simply not bother
to return this exit code, and ssh will try its best but is limited by
the fact it also tries to return an error code for SSH related errors.
In either case, this may pose problems when using -e.
Under Cygwin, child processes do not inherit alarm(3) settings from the
parent disabling this simple timeout enforcement system used by shmux.
Some shells will ignore SIGALRM, others die upon its receipt
(regardless of any trap). Again, this will affect the simple timeout
enforcement system used by shmux.
Send bug reports to `shmux-bugs@taranis.org'.
$Date: 2007-07-11 08:09:36 $ SHMUX(8)