shelltestrunner

shelltestrunner tests command-line programs (or arbitrary shell commands.) It reads simple declarative tests specifying a command, some input, and the expected output, and can run them run in parallel, selectively, with a timeout, in color, and/or with differences highlighted. shelltestrunner has been tested on gnu/linux, mac and windows; projects using it include hledger, berp, cblrepo and eddie. shelltestrunner is free software released under GPLv3+.

Installing

If your machine's packaging system does not provide an up-to-date shelltestrunner, install it with cabal (and if your packaging system does not provide an up-to-date cabal, get the haskell platform):

$ cabal install shelltestrunner

You should now have the shelltest program in your path.

Release notes

See http://hub.darcs.net/simon/shelltestrunner/CHANGES.

Defining tests

Test files, typically named tests/*.test, contain one or more tests consisting of:

Example: here's example.test, a file containing two simple tests:

# 1. let's test that echo runs. Numbering your tests can be helpful.
echo
>>>= 0

# 2. and now the cat command. On windows, this one should fail.
cat
<<<
foo
>>>
foo
>>>= 0

Run it with shelltest:

$ shelltest example.test
:t.test:1: [OK]
:t.test:2: [OK]

         Test Cases  Total
 Passed  2           2
 Failed  0           0
 Total   2           2

Test format:

# optional comment
the command to test
<<<
zero or more lines of standard input
>>>
zero or more lines of expected standard output (or /REGEXP/ added to the previous line)
>>>2
zero or more lines of expected standard error output (or /REGEXP/ added to the previous line)
>>>= EXITCODE (or /REGEXP/)

Here are some real-world project tests to give you ideas.

Running tests

The shelltest program accepts one or more test file or directory paths. A directory means all files below it with the test file suffix (default: .test).

Example: run all tests defined in or below the tests directory with args in their name, up to 8 in parallel, allowing a maximum of 1 second each, showing only failures and the final summary:

$ shelltest tests -- -targs -j8 -o1 --hide

         Test Cases   Total
 Passed  2            2
 Failed  0            0
 Total   2            2

Command-line options:

$ shelltest --help
shelltest 1.2.1

shelltest [OPTIONS] [TESTFILES|TESTDIRS]

Common flags:
  -a --all              Show all failure output, even if large
  -c --color            Show colored output if your terminal supports it
  -d --diff             Show failures in diff format
  -p --precise          Show failure output precisely (good for whitespace)
  -x --exclude=STR      Exclude test files whose path contains STR
     --execdir          Run tests from within the test file's directory
     --extension=EXT    Filename suffix of test files (default: .test)
  -w --with=EXECUTABLE  Replace the first word of (unindented) test commands
     --debug            Show debug info, for troubleshooting
     --debug-parse      Show test file parsing info and stop
     --help-format      Display test format help
  -? --help             Display help message
  -V --version          Print version information

     -- TFOPTIONS       Set extra test-framework options like -j/--threads,
                        -t/--select-tests, -o/--timeout, --hide-successes.
                        Use -- --help for a list. Avoid spaces.

Contributing

The released version is on hackage. The latest code (browse, changes) is here:

$ darcs get http://hub.darcs.net/simon/shelltestrunner

Feedback, code, testing, documentation/blogging are most welcome.
Add your experience to the user survey (results).
or chat me (sm on irc.freenode.net).
For focussed support/faster development/good karma you can hire me or Donate with WePay

Credits

Simon Michael wrote and maintains shelltestrunner. John Macfarlane, Bernie Pope and Trygve Laugstøl have contributed code. It relies heavily on Max Bolingbroke's test-framework, other libraries, and of course the Glorious Haskell Compiler. It was inspired by the tests for John Wiegley's ledger.