git: dfregress - An automated test driver and framework

[Alex Hornung <alexh@xxxxxxxxxxxxxxxxxxxxxxx>]

commit 6a7006f10516182dbc2d134ecce752158f3ce61b
Author: Alex Hornung <ahornung@gmail.com>
Date:   Sun Oct 30 01:21:59 2011 +0000

    dfregress - An automated test driver and framework
    
     * dfregress is a simple test framework and test automation driver. It
       supports both normal userland testcases and kernel test cases.
    
     * It's key aim is to make it simple, if not outright dead easy, to
       write test cases. A test case is a simple program using
       printf/fprintf and exit - no magic needed. A kernel test case is a
       very small module that just needs to implement a few functions and
       call a logging and result function.
    
     * Sample output of the text frontend (Frontends are explained further
       down the text): http://leaf.dragonflybsd.org/~alexh/dfregress.txt
    
     * dfregress is very UNIXy, it uses makefiles to build the testcases,
       stdout/stderr redirection from the testcases (no fancy output
       functions needed in the testcases) and evaluates the normal return
       value of the testcase (no need to call fancy functions).
    
     * For kernel testcases it is a bit different - you do have to call
       functions to log output and to log the result, but it is very simple,
       hardly any overhead.
    
     * The test driver assumes that testcases are in the testcases/
       directory, but it supports several command line options.
    
     * The tests to run, including several options, are specified in the
       runlist file. An example runlist including all the testcases is
       included as config/runlist.run. Options that can be specified are:
       - timeout: (in seconds) after which the test is aborted if it hasn't
         finished.
       - test type (userland, kernel or buildonly)
       - make: which 'make' tool to use to build the test cases. Defaults to
         'make', but could also be 'gmake', etc.
       - nobuild: doesn't build the testcase and tries to directly execute
         it.
       - pre, post: external pre-run and post-run commands, e.g. to set up a
         vn device and to tear it down. This is to avoid duplication in test
         cases, the common setup can be factored out.
       - intpre, intpost: similar to the above, but it assumes that the
         testcase, when passed the parameter 'pre', will do the pre-setup,
         and when passed 'post' will do the post-test cleanup/setup/etc.
       - any number of command line arguments that are passed to the test
         case. (See the crypto/ test examples in the runlist).
    
     * A range of sample testcases are available in
       test/dfregress/testcases/sample, including a kernel testcase sample.
    
     * Note that many of the test cases in the testcases/ directory have
       been copied from elsewhere in the main repository and are,
       temporarily at least, duplicated.
    
     * The test driver is completely separated from the frontends. The test
       driver outputs the test run results in an XML-like format (plist)
       that can easily be parsed using proplib. Python and Ruby also have
       plist modules that could be used to parse the output.
    
     * The only available frontend is a simple C program that will parse the
       intermediate format to an easy to read plain text format. Additional
       frontends can be written in any language, as long as it is possible
       to load plists. Frontends are in the fe/ directory.
    
     * XXX: the default options (currently just the timeout) are still
       hardcoded in config.c.
    
     * The NOTES file gives details on how the test execution occurs and
       what result code can be raised at which point. This document, and a look
       at the generated plist, is all you need to write a new frontend that,
       for example, generates beautiful HTML output.
       For completeness sake, a part of NOTES is reproduced under the main commit
       message - specifically the part detailing the execution of a single test
       case.
    
    ======
    Execution of a single test case:
    ======
    1) chdir to testcase directory
            - if it fails, set RESULT_PREFAIL (sysbuf is of interest), goto (6)
    
    2) build testcase (make) (unless nobuild flag is set).
            + build_buf is used for stdout/stderr
            - if there is an internal driver error (that leads to not running the
              build command), set RESULT_PREFAIL (sysbuf is of interest), goto (6)
            - if the build command has a non-zero exit value, set the result to
              BUILDFAIL, unless it's a buildonly test case, in which it is set to
              the actual result value (TIMEOUT, SIGNALLED, FAIL)
              goto (6)
    
    3) run 'pre' command if intpre or pre is set.
            + precmd_buf is used for stdout/stderr
            - if there is an internal driver error (that leads to not running the
              command), set RESULT_PREFAIL (sysbuf is of interest), goto (6)
            - if the pre command has a non-zero exit value, set RESULT_PREFAIL and
              goto (6)
    
    4) run actual testcase, depending on type
            + stdout_buf is used for stdout
            + stderr_buf is used for stderr
            - for BUILDONLY: set RESULT_PASS since the build already succeeded
            - for userland and kernel: run the testcase, possibly as a different
              user (depending on the runas option), set the result to the actual
              result value (TIMEOUT, SIGNALLED, FAIL, NOTRUN)
            - if there is an internal driver error (that leads to not running the
              command), RESULT_NOTRUN is set (sysbuf is of interest)
    
    5) run 'post' command if intpost or post is set.
            + postcmd_buf is used for stdout/stderr
            - if there is an internal driver error (that leads to not running the
              command), set RESULT_POSTFAIL (sysbuf is of interest), goto (6)
            - if the post command has a non-zero exit value, set RESULT_POSTFAIL
              and goto (6)
    
    6) clean testcase directory (make clean) (unless nobuild flag is set).
            + cleanup_buf is used for stdout/stderr and system (driver error) buffer
            - no further action.
    
    7) results are saved.

Summary of changes:
 test/dfregress/NOTES                               |   76 +++
 test/dfregress/config/defaults.conf                |    4 +
 test/dfregress/config/runlist.run                  |   53 ++
 test/dfregress/driver/Makefile                     |   13 +
 test/dfregress/driver/config.c                     |   59 ++
 test/dfregress/driver/config.h                     |   30 +
 test/dfregress/driver/kernel.c                     |  145 ++++
 test/dfregress/driver/kernel.h                     |   30 +
 test/dfregress/driver/main.c                       |  103 +++
 test/dfregress/driver/parser.c                     |  218 ++++++
 test/dfregress/driver/parser.h                     |   35 +
 test/dfregress/driver/runlist.c                    |  312 +++++++++
 test/dfregress/driver/runlist.h                    |   40 ++
 test/dfregress/driver/testcase.c                   |  695 ++++++++++++++++++++
 test/dfregress/driver/testcase.h                   |  116 ++++
 test/dfregress/driver/userland.c                   |  295 +++++++++
 test/dfregress/driver/userland.h                   |   34 +
 test/dfregress/fe/text/Makefile                    |   18 +
 test/dfregress/fe/text/fe_text.c                   |  223 +++++++
 test/dfregress/framework/dfregress.h               |   11 +
 test/dfregress/framework/tbridge.h                 |   72 ++
 test/dfregress/framework/test.txt                  |    2 +
 test/dfregress/kernel_bridge/Makefile              |    4 +
 test/dfregress/kernel_bridge/dfregress_bridge.c    |  344 ++++++++++
 test/dfregress/kernel_bridge/safe_mem.c            |  191 ++++++
 test/dfregress/testcases/Makefile                  |   10 +
 test/dfregress/testcases/compiler/Makefile         |    3 +
 test/dfregress/testcases/compiler/div128/Makefile  |    4 +
 test/dfregress/testcases/compiler/div128/div128.c  |   12 +
 test/dfregress/testcases/crypto/Makefile           |    3 +
 test/{ => dfregress/testcases}/crypto/aes/Makefile |    0
 .../{ => dfregress/testcases}/crypto/aes/aestest.c |    0
 .../testcases}/crypto/aes/ecbnk44.txt              |    0
 .../testcases}/crypto/aes/ecbnk48.txt              |    0
 .../testcases}/crypto/aes/ecbnt44.txt              |    0
 .../testcases}/crypto/aes/ecbnt48.txt              |    0
 .../testcases}/crypto/aes/ecbvk44.txt              |    0
 .../testcases}/crypto/aes/ecbvk48.txt              |    0
 .../testcases}/crypto/aes/ecbvt44.txt              |    0
 .../testcases}/crypto/aes/ecbvt48.txt              |    0
 .../testcases}/crypto/aesctr/Makefile              |    0
 .../testcases}/crypto/aesctr/aesctr.c              |    0
 .../testcases}/crypto/aesxts/Makefile              |    0
 .../testcases}/crypto/aesxts/aes_xts.c             |    0
 .../testcases}/crypto/serpent/Makefile             |    0
 .../testcases}/crypto/serpent/serpent_test.c       |    0
 .../testcases}/crypto/serpent/serpentecb_vk.txt    |    0
 .../testcases}/crypto/serpent/serpentecb_vt.txt    |    0
 .../testcases}/crypto/twofish/Makefile             |    0
 .../testcases}/crypto/twofish/twofish_test.c       |    0
 .../testcases}/crypto/twofish/twofishecb_vk.txt    |    0
 .../testcases}/crypto/twofish/twofishecb_vt.txt    |    0
 test/dfregress/testcases/io/Makefile               |    6 +
 test/dfregress/testcases/io/kqueue_1/Makefile      |    4 +
 test/dfregress/testcases/io/kqueue_1/kqueue_1.c    |   49 ++
 test/dfregress/testcases/io/kqueue_2/Makefile      |    4 +
 .../dfregress/testcases/io/kqueue_2/kqueue_2.c     |    0
 test/dfregress/testcases/io/pselect_1/Makefile     |    4 +
 test/dfregress/testcases/io/pselect_1/pselect_1.c  |  145 ++++
 test/dfregress/testcases/io/select_1/Makefile      |    4 +
 test/dfregress/testcases/io/select_1/select_1.c    |  125 ++++
 test/dfregress/testcases/io/select_2/Makefile      |    4 +
 .../dfregress/testcases/io/select_2/select_2.c     |    0
 test/dfregress/testcases/io/select_3/Makefile      |    4 +
 .../dfregress/testcases/io/select_3/select_3.c     |    0
 test/dfregress/testcases/io/select_4/Makefile      |    4 +
 .../dfregress/testcases/io/select_4/select_4.c     |    0
 test/dfregress/testcases/io/sendfd_1/Makefile      |    4 +
 test/dfregress/testcases/io/sendfd_1/sendfd_1.c    |   93 +++
 test/dfregress/testcases/mem/Makefile              |    3 +
 test/dfregress/testcases/mem/mmap_1/Makefile       |    5 +
 test/dfregress/testcases/mem/mmap_1/mmap_1.c       |   48 ++
 .../testcases/mem/mmap_madvise_1/Makefile          |    4 +
 .../testcases/mem/mmap_madvise_1/mmap_madvise_1.c  |   97 +++
 test/dfregress/testcases/misc/Makefile             |    3 +
 test/dfregress/testcases/misc/sh_1/Makefile        |    4 +
 test/dfregress/testcases/misc/sh_1/sh_1.sh         |   11 +
 test/dfregress/testcases/priv/Makefile             |    3 +
 test/dfregress/testcases/priv/setreuid/Makefile    |    4 +
 test/dfregress/testcases/priv/setreuid/setreuid.c  |   38 ++
 test/dfregress/testcases/sample/Makefile           |    5 +
 test/dfregress/testcases/sample/test1/Makefile     |    4 +
 test/dfregress/testcases/sample/test1/test1.c      |   15 +
 test/dfregress/testcases/sample/test2/Makefile     |    4 +
 test/dfregress/testcases/sample/test2/test2.c      |   13 +
 test/dfregress/testcases/sample/test3/Makefile     |    4 +
 test/dfregress/testcases/sample/test3/test3.c      |    9 +
 test/dfregress/testcases/sample/test4/Makefile     |    4 +
 test/dfregress/testcases/sample/test4/test4.c      |   20 +
 test/dfregress/testcases/sample/test5/Makefile     |    4 +
 test/dfregress/testcases/sample/test5/test5.c      |    9 +
 test/dfregress/testcases/sample/testb1/Makefile    |    4 +
 test/dfregress/testcases/sample/testb1/testb1.c    |    7 +
 test/dfregress/testcases/sample/testk1/Makefile    |    4 +
 test/dfregress/testcases/sample/testk1/testk1.c    |   25 +
 test/dfregress/testcases/sample/testk2/Makefile    |    4 +
 test/dfregress/testcases/sample/testk2/testk1.c    |   25 +
 test/dfregress/testcases/threads/Makefile          |    3 +
 .../testcases/threads/umtx_errno/Makefile          |    5 +
 .../testcases/threads/umtx_errno/umtx_errno.c      |   95 +++
 100 files changed, 4088 insertions(+), 0 deletions(-)
 create mode 100644 test/dfregress/NOTES
 create mode 100644 test/dfregress/config/defaults.conf
 create mode 100644 test/dfregress/config/runlist.run
 create mode 100644 test/dfregress/driver/Makefile
 create mode 100644 test/dfregress/driver/config.c
 create mode 100644 test/dfregress/driver/config.h
 create mode 100644 test/dfregress/driver/kernel.c
 create mode 100644 test/dfregress/driver/kernel.h
 create mode 100644 test/dfregress/driver/main.c
 create mode 100644 test/dfregress/driver/parser.c
 create mode 100644 test/dfregress/driver/parser.h
 create mode 100644 test/dfregress/driver/runlist.c
 create mode 100644 test/dfregress/driver/runlist.h
 create mode 100644 test/dfregress/driver/testcase.c
 create mode 100644 test/dfregress/driver/testcase.h
 create mode 100644 test/dfregress/driver/userland.c
 create mode 100644 test/dfregress/driver/userland.h
 create mode 100644 test/dfregress/fe/text/Makefile
 create mode 100644 test/dfregress/fe/text/fe_text.c
 create mode 100644 test/dfregress/framework/dfregress.h
 create mode 100644 test/dfregress/framework/tbridge.h
 create mode 100644 test/dfregress/framework/test.txt
 create mode 100644 test/dfregress/kernel_bridge/Makefile
 create mode 100644 test/dfregress/kernel_bridge/dfregress_bridge.c
 create mode 100644 test/dfregress/kernel_bridge/safe_mem.c
 create mode 100644 test/dfregress/testcases/Makefile
 create mode 100644 test/dfregress/testcases/compiler/Makefile
 create mode 100644 test/dfregress/testcases/compiler/div128/Makefile
 create mode 100644 test/dfregress/testcases/compiler/div128/div128.c
 create mode 100644 test/dfregress/testcases/crypto/Makefile
 copy test/{ => dfregress/testcases}/crypto/aes/Makefile (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/aestest.c (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbnk44.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbnk48.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbnt44.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbnt48.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbvk44.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbvk48.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbvt44.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aes/ecbvt48.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/aesctr/Makefile (100%)
 copy test/{ => dfregress/testcases}/crypto/aesctr/aesctr.c (100%)
 copy test/{ => dfregress/testcases}/crypto/aesxts/Makefile (100%)
 copy test/{ => dfregress/testcases}/crypto/aesxts/aes_xts.c (100%)
 copy test/{ => dfregress/testcases}/crypto/serpent/Makefile (100%)
 copy test/{ => dfregress/testcases}/crypto/serpent/serpent_test.c (100%)
 copy test/{ => dfregress/testcases}/crypto/serpent/serpentecb_vk.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/serpent/serpentecb_vt.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/twofish/Makefile (100%)
 copy test/{ => dfregress/testcases}/crypto/twofish/twofish_test.c (100%)
 copy test/{ => dfregress/testcases}/crypto/twofish/twofishecb_vk.txt (100%)
 copy test/{ => dfregress/testcases}/crypto/twofish/twofishecb_vt.txt (100%)
 create mode 100644 test/dfregress/testcases/io/Makefile
 create mode 100644 test/dfregress/testcases/io/kqueue_1/Makefile
 create mode 100644 test/dfregress/testcases/io/kqueue_1/kqueue_1.c
 create mode 100644 test/dfregress/testcases/io/kqueue_2/Makefile
 copy tools/test/kqueue/kqueue_oob.c => test/dfregress/testcases/io/kqueue_2/kqueue_2.c (100%)
 create mode 100644 test/dfregress/testcases/io/pselect_1/Makefile
 create mode 100644 test/dfregress/testcases/io/pselect_1/pselect_1.c
 create mode 100644 test/dfregress/testcases/io/select_1/Makefile
 create mode 100644 test/dfregress/testcases/io/select_1/select_1.c
 create mode 100644 test/dfregress/testcases/io/select_2/Makefile
 copy tools/test/select/select_eof.c => test/dfregress/testcases/io/select_2/select_2.c (100%)
 create mode 100644 test/dfregress/testcases/io/select_3/Makefile
 copy tools/test/select/select_many_write.c => test/dfregress/testcases/io/select_3/select_3.c (100%)
 create mode 100644 test/dfregress/testcases/io/select_4/Makefile
 copy tools/test/select/select_oob.c => test/dfregress/testcases/io/select_4/select_4.c (100%)
 create mode 100644 test/dfregress/testcases/io/sendfd_1/Makefile
 create mode 100644 test/dfregress/testcases/io/sendfd_1/sendfd_1.c
 create mode 100644 test/dfregress/testcases/mem/Makefile
 create mode 100644 test/dfregress/testcases/mem/mmap_1/Makefile
 create mode 100644 test/dfregress/testcases/mem/mmap_1/mmap_1.c
 create mode 100644 test/dfregress/testcases/mem/mmap_madvise_1/Makefile
 create mode 100644 test/dfregress/testcases/mem/mmap_madvise_1/mmap_madvise_1.c
 create mode 100644 test/dfregress/testcases/misc/Makefile
 create mode 100644 test/dfregress/testcases/misc/sh_1/Makefile
 create mode 100644 test/dfregress/testcases/misc/sh_1/sh_1.sh
 create mode 100644 test/dfregress/testcases/priv/Makefile
 create mode 100644 test/dfregress/testcases/priv/setreuid/Makefile
 create mode 100644 test/dfregress/testcases/priv/setreuid/setreuid.c
 create mode 100644 test/dfregress/testcases/sample/Makefile
 create mode 100644 test/dfregress/testcases/sample/test1/Makefile
 create mode 100644 test/dfregress/testcases/sample/test1/test1.c
 create mode 100644 test/dfregress/testcases/sample/test2/Makefile
 create mode 100644 test/dfregress/testcases/sample/test2/test2.c
 create mode 100644 test/dfregress/testcases/sample/test3/Makefile
 create mode 100644 test/dfregress/testcases/sample/test3/test3.c
 create mode 100644 test/dfregress/testcases/sample/test4/Makefile
 create mode 100644 test/dfregress/testcases/sample/test4/test4.c
 create mode 100644 test/dfregress/testcases/sample/test5/Makefile
 create mode 100644 test/dfregress/testcases/sample/test5/test5.c
 create mode 100644 test/dfregress/testcases/sample/testb1/Makefile
 create mode 100644 test/dfregress/testcases/sample/testb1/testb1.c
 create mode 100644 test/dfregress/testcases/sample/testk1/Makefile
 create mode 100644 test/dfregress/testcases/sample/testk1/testk1.c
 create mode 100644 test/dfregress/testcases/sample/testk2/Makefile
 create mode 100644 test/dfregress/testcases/sample/testk2/testk1.c
 create mode 100644 test/dfregress/testcases/threads/Makefile
 create mode 100644 test/dfregress/testcases/threads/umtx_errno/Makefile
 create mode 100644 test/dfregress/testcases/threads/umtx_errno/umtx_errno.c

http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/6a7006f10516182dbc2d134ecce752158f3ce61b


-- 
DragonFly BSD source repository