shelltestrunner tests command-line programs (or arbitrary shell commands.) It reads simple declarative tests specifying a command, some input, and the expected output, and can run them run in parallel, selectively, with a timeout, in color, and/or with differences highlighted. shelltestrunner has been tested on gnu/linux, mac and windows; projects using it include hledger, berp, cblrepo and eddie. shelltestrunner is free software released under GPLv3+.
If your machine's packaging system does not provide an up-to-date shelltestrunner, install it with cabal (and if your packaging system does not provide an up-to-date cabal, get the haskell platform):
$ cabal install shelltestrunner
You should now have the
shelltest program in your path.
Test files, typically named
tests/*.test, contain one or more tests consisting of:
<<<), standard output (
>>>) and/or standard error output (
example.test, a file containing two simple tests:
# 1. let's test that echo runs. Numbering your tests can be helpful. echo >>>= 0 # 2. and now the cat command. On windows, this one should fail. cat <<< foo >>> foo >>>= 0
Run it with
$ shelltest example.test :t.test:1: [OK] :t.test:2: [OK] Test Cases Total Passed 2 2 Failed 0 0 Total 2 2
# optional comment the command to test <<< zero or more lines of standard input >>> zero or more lines of expected standard output (or /REGEXP/ added to the previous line) >>>2 zero or more lines of expected standard error output (or /REGEXP/ added to the previous line) >>>= EXITCODE (or /REGEXP/)
/REGEXP/pattern may be used instead of explicit data. In this case a match anywhere in the output allows the test to pass. The regular expression syntax is regex-tdfa's.
EXITCODEis a numeric exit status, eg
0for a successful exit.
EXITCODEto negate the match.
#may be used between tests.
Here are some real-world project tests to give you ideas.
shelltest program accepts one or more test file or directory paths. A directory means all files below it with the test file suffix (default:
Example: run all tests defined in or below the
tests directory with
args in their name, up to 8 in parallel, allowing a maximum of 1 second each, showing only failures and the final summary:
$ shelltest tests -- -targs -j8 -o1 --hide Test Cases Total Passed 2 2 Failed 0 0 Total 2 2
$ shelltest --help shelltest 1.2.1 shelltest [OPTIONS] [TESTFILES|TESTDIRS] Common flags: -a --all Show all failure output, even if large -c --color Show colored output if your terminal supports it -d --diff Show failures in diff format -p --precise Show failure output precisely (good for whitespace) -x --exclude=STR Exclude test files whose path contains STR --execdir Run tests from within the test file's directory --extension=EXT Filename suffix of test files (default: .test) -w --with=EXECUTABLE Replace the first word of (unindented) test commands --debug Show debug info, for troubleshooting --debug-parse Show test file parsing info and stop --help-format Display test format help -? --help Display help message -V --version Print version information -- TFOPTIONS Set extra test-framework options like -j/--threads, -t/--select-tests, -o/--timeout, --hide-successes. Use -- --help for a list. Avoid spaces.
Test commands normally run within your current directory;
--execdir makes them run within the directory where they are defined, instead.
-w/--with replaces the first word of all test commands with something else, which can be useful for testing alternate versions of a program. Commands which have been indented by one or more spaces will not be affected by this option.
The test-framework library provides additional options which you can specify after
-- (note: avoid spaces between flags and values here.) Run
shelltest -- --help for a list. Here are some useful ones:
-j NUMBER --threads=NUMBER number of threads to use to run tests -o NUMBER --timeout=NUMBER how many seconds a test should be run for before giving up, by default -t TEST-PATTERN --select-tests=TEST-PATTERN only tests that match at least one glob pattern given by an instance of this argument will be run --hide-successes hide sucessful tests, and only show failures
The released version is on hackage. The latest code (browse, changes) is here:
$ darcs get http://hub.darcs.net/simon/shelltestrunner
Feedback, code, testing, documentation/blogging are most welcome.
Add your experience to the user survey (results).
or chat me (
sm on irc.freenode.net).
For focussed support/faster development/good karma you can hire me or
support latest Diff, cmdargs, test-framework; tested with GHC 7.6.1 (Magnus Therning)
fix unicode handling on GHC >= 7.2
--preciseoptions now interact well
-a/--allflag shows all failure output without truncating
>>>=field is now required; you may need to add it to your existing tests
-i/--implicitoption has been dropped
-d/--diffoption shows test failures as a unified diff when possible, including line numbers to help locate the problem
-x/--excludeoption skips certain test files (eg platform-specific ones)
Simon Michael wrote and maintains shelltestrunner. John Macfarlane, Bernie Pope and Trygve Laugstøl have contributed code. It relies heavily on Max Bolingbroke's test-framework, other libraries, and of course the Glorious Haskell Compiler. It was inspired by the tests for John Wiegley's ledger.