DragonFly On-Line Manual Pages
DFREGRESS(8) DragonFly System Manager's Manual DFREGRESS(8)
NAME
dfregress - an automation test driver and framework
SYNOPSIS
dfregress [-o output_plist] [-t testcase_dir] [-p prepost_dir]
runlist_file
dfregress -h
DESCRIPTION
dfregress is a regression test framework and automation driver. It
executes a series of testcases as specified in the runlist_file and
collects the results.
The path to the testcase collection is specified via the testcase_dir
argument. If this argument is not specified, the default is assumed to
be the same directory as that of the runlist. For example if the used
runlist is /usr/src/test/testcases/sample.run the default testcase
directory, unless otherwise specified, will be /usr/src/test/testcases.
Similarly the path to the pre- and post commands is specified via
prepost_dir. The prepost_dir only needs to be specified if the runlist
contains custom pre or post commands.
The output is saved in plist format to the output_plist file, or if none
is specified, to a file with the same base name as the runlist, but in
the current working directory and with a .plist extension. For example
if the used runlist is /usr/src/test/testcases/sample.run the default
output, unless otherwise specified, will be ./sample.plist. Other tools,
known as frontends, can parse the plist output into an easier to read
form.
The following is a summary of the optional parameters:
-o output_plist
Specifies the file to which to write the results. The resulting
file, output_plist, will be in plist format.
-t testcase_dir
Specifies the directory in which to find the testcases specified
in the runlist.
-p prepost_dir
Specifies the directory in which to find the pre- and post
commands used in the runlist. This argument is only required
when the runlist uses custom pre- or post commands.
-h Prints a short synopsis.
RUNLIST SYNTAX
Testcases are specified one testcase per line, with whitespace separated
values. Empty lines and lines beginning with a "#" are ignored.
Runlist lines are of the following format:
testcase type options arguments ...
The testcase part needs to be a relative path from the testcase base
directory specified by the -t argument to the resulting (after building
the testcase) testcase executable. The testcase will be executed with
the arguments passed as command line arguments to it.
Valid types are userland, kernel and buildonly:
userland
A userland testcase is a normal userland executable that
returns a non-zero exit value on test failure.
kernel A kernel testcase is run with the kernel test bridge inside
the kernel.
buildonly
A buildonly test passes when the build for the given
testcase succeeds.
Valid options are defaults, interpreter, make, rc, timeout, nobuild,
runas, intpre, intpost, pre, and post:
defaults
This option does nothing. All default options are applied.
interpreter command
Uses command for running the test case instead of calling
it directly. This is useful when you have tests that you
want to call with a specific shell like sh(1), and there is
no shebang in the testcase file, there are no execution
permissions and you don't want or cannot modify the
testcase.
make make_command
Uses make_command instead of make(1) to build the testcase.
rc code
Sets the return code that will be considered successful.
By default 0 is considered success.
timeout timeout
Sets the timeout for the testcase after which the testcase
will be aborted to timeout seconds.
nobuild
If this option is set, the build stage and cleanup stage of
the test case are not run.
runas uid
Runs the testcase as the user identified by the given uid.
intpre Executes the testcase command with the argument "pre"
during the pre-run command stage.
intpost
Executes the testcase command with the argument "post"
during the post-run command stage.
pre precmd
Executes the command specified by precmd during the pre-run
command stage.
pre postcmd
Executes the command specified by postcmd during the post-
run command stage.
TESTCASE EXECUTION
1. CHDIR to the testcase directory. If it fails, the result will
be set to RESULT_PREFAIL and the sysbuf buffer will provide
further details.
2. BUILD the testcase using the command specified by the make
option or, if not specified, make(1), unless the nobuild
option was specified. If there is an internal driver error,
the result will be set to RESULT_PREFAIL, the next step to be
performed will be CLEANUP and the sysbuf buffer will provide
further details. If no internal error occurs, the buildbuf
will contain the build log.
3. RUN PRE COMMAND if intpre or pre are set. If there is an
internal driver error, the result will be set to
RESULT_PREFAIL, the next step to be performed will be CLEANUP
and the sysbuf buffer will provide further details. If the
pre command has a non-zero exit value, the result will be set
to RESULT_PREFAIL and the next step to be performed will be
CLEANUP. In this case and in the case where the command
succeeds, the precmd_buf will contain the execution log.
4. RUN TESTCASE depending on type:
buildonly
testcases will get their result set to RESULT_PASS at this
point, since the build must have succeeded already.
userland
testcases will get executed in a separate child process,
possibly as a different user, depending on whether the
runas option was specified. The result will be set to
RESULT_TIMEOUT if the timeout is reached before the
testcase finishes, RESULT_SIGNALLED if the testcase dies
because of an unhandled signal (often due to crashing),
RESULT_NOTRUN if the testcase could not be executed,
RESULT_FAIL if the testcase completes but returns failure
or RESULT_PASS if the testcase completes and returns
success.
kernel
testcases will be executed in kernel space by loading a
given module and running the testcase entry point
function. The result will be set to RESULT_NOTRUN if the
testcase could not be executed. Otherwise the result will
be set according to the kernel test case to one of
RESULT_TIMEOUT, RESULT_FAIL, or RESULT_PASS.
The output will be logged separately for stdout and stderr to
the stdout_buf and stderr_buf respectively. If the result was
RESULT_NOTRUN the sysbuf will contain more information.
5. RUN POST COMMAND if intpost or post are set. If there is an
internal driver error, the result will be set to
RESULT_POSTFAIL, the next step to be performed will be CLEANUP
and the sysbuf buffer will provide further details. If the
post command has a non-zero exit value, the result will be set
to RESULT_POSTFAIL and the next step to be performed will be
CLEANUP. In this case and in the case where the command
succeeds, the postcmd_buf will contain the execution log.
6. CLEANUP the testcase execution using the command specified by
the make option or, if not specified, make(1) with the
parameter "clean", unless the nobuild option was specified.
If there is an internal driver error the sysbuf buffer will
contain more information. If no internal error occurs, the
cleanu_pbuf will contain the cleanup log.
FRONTENDS
The output of dfregress is in the Apple plist serialized object format.
This format can be easily parsed by using proplib(3) when using C. Ruby
and Python also have parsers for the plist format.
A frontend for dfregress parses the intermediate output plist into a more
easily readable format such as plain text or websites.
By default dfregress ships only with the dfr2text(8) text-based frontend.
HOW TO WRITE A TESTCASE
A userland testcase is a simple program that prints some relevant
information to stdout and stderr, both of which are captured by the test
driver, and returns an exit value of 0 if the test passed, or any other
non-zero exit value to signal a failure. The exact exit value is
recorded by dfregress. All signals/exceptions not explicitly caught by
the testcase will abort the execution of the testcase and the result will
be set appropriately and the signal number will be recorded.
A kernel testcase is a simple kernel module that defines two methods: a
run() method as well as an optional abort() method. The run() method
will be run from a separate kernel thread. The testcase will need to
call tbridge(9) functions to record output and to notify of testcase
completion. Refer to the tbridge(9) manual page for more details.
For all testcase types the build stage is common. Every testcase should
either have the nobuild option set, or have a Makefile or similar in its
directory. By default dfregress assumes it is a standard make(1)
Makefile. If that is not the case, the build option needs to be adjusted
accordingly.
GENERAL ADVICE ON WRITING TESTCASES
Test only one thing at a time, it is not good practice to test multiple
things in the same testcase as it makes it less obvious what's going on.
Keep it short, simple and well documented on what the requirements are,
what the preconditions need to be, what exactly is being tested, ideally
with a reference to a particular bug if that exists, and most importantly
what the expected outcomes are.
Make sure your testcase doesn't depend on any other being run previously
as well as that it won't hinder any other testcase from running. This
effectively means that your testcase should make no assumptions as to
what has been run previously unless it has a registered pre-command and
that the system should be left as found before your testcase.
EXAMPLES
An example runlist can be found under test/testcases/sample.run.
Several example testcases for both userland and kernel are under
test/testcases/sample.
SEE ALSO
dfr2text(8), tbridge(9)
HISTORY
The dfregress utility appeared in DragonFly 2.13.
AUTHORS
Alex Hornung
DragonFly 5.9-DEVELOPMENT November 27, 2020 DragonFly 5.9-DEVELOPMENT