Command-line testing

Human or Replicant ? If you have a testing situation such as this1, we at Joyful Corp. can help.

shelltestrunner is a command-line tool for testing other command-line programs, or general shell commands, on (eg) GNU/Linux, Mac and Windows.

It reads tests which specify a command to run, some input, and the expected stdout, stderr, and exit status output. It can run tests in parallel, selectively, with a timeout, with color output, or with expected/actual differences highlighted. shelltestrunner is free software released under GPLv3+.

1 expressed on the command line

Getting started

Debian, Ubuntu:   apt-get install shelltestrunner
Gentoo: emerge shelltestrunner
Elsewhere:


Get GHC and cabal (or the Haskell Platform),
ensure ~/.cabal/bin is in your $PATH,
cabal install shelltestrunner

Tests are kept in files with the .test suffix by default. Here's a simple test file:

# echo, given no input, prints nothing and terminates normally
echo
>>>= 0

and another, containing two more tests:

# cat copies its input to stdout, nothing appears on stderr, exit status is 0
cat
<<<
foo
>>>
foo
>>>2
>>>= 0

# cat prints an error containing "unrecognized option" if given a bad flag
cat --no-such-flag
>>>2 /unrecognized option/
>>>= !0

Run the tests:

$ shelltest echo.test cat.test
:echo.test: [OK]
:cat.test:1: [OK]
:cat.test:2: [OK]

         Test Cases  Total      
 Passed  3           3          
 Failed  0           0          
 Total   3           3          

That's it!

Test format

Test files contain one or more tests, which look like this:

# optional comment
a one-line shell command
<<<
zero or more lines of standard input
>>>
zero or more lines of expected standard output (or /REGEXP/ added to the previous line)
>>>2
zero or more lines of expected standard error output (or /REGEXP/ added to the previous line)
>>>= STATUS (or /REGEXP/)

The command and the final exit status line are required; the other parts are optional.

A /REGEXP/ pattern may be used instead of specifying the full output, in which case a match anywhere in the output allows the test to pass. The regular expression syntax is regex-tdfa's, plus you can put ! before /REGEXP/ to negate the match.

STATUS is a numeric exit status or a /REGEXP/. Again, use a ! prefix to negate the match. Eg !0 matches an unsuccessful exit.

Comment lines beginning with # may be used between tests, but not within them.

Here are some real world examples.

Usage

shelltest accepts one or more test file or directory arguments. A directory means all files below it which have the test file suffix (.test, by default).

Command-line options:

$ shelltest --help
shelltest [OPTIONS] [TESTFILES|TESTDIRS]

Common flags:
  -a --all              Show all failure output, even if large
  -c --color            Show colored output if your terminal supports it
  -d --diff             Show failures in diff format
  -p --precise          Show failure output precisely (good for whitespace)
  -x --exclude=STR      Exclude test files whose path contains STR
     --execdir          Run tests from within the test file’s directory
     --extension=EXT    Filename suffix of test files (default: .test)
  -w --with=EXECUTABLE  Replace the first word of (unindented) test commands
     --debug            Show debug info, for troubleshooting
     --debug-parse      Show test file parsing info and stop
     --help-format      Display test format help
  -? --help             Display help message
  -V --version          Print version information
     --numeric-version  Print just the version number

     -- TFOPTIONS       Set extra test-framework options like -j/--threads,
                        -t/--select-tests, -o/--timeout, --hide-successes.
                        Use -- --help for a list. Avoid spaces.

Test commands normally run within your current directory; --execdir makes them run within the directory where they are defined, instead.

-w/--with replaces the first word of all test commands with something else, which can be useful for testing alternate versions of a program. Test commands which have been indented by one or more spaces will not be affected by this option.

--exclude can be useful to avoid running certain tests, eg unix-specific tests when on windows and vice-versa.

The test-framework library provides additional options which you can specify after -- (note: avoid spaces between flags and values here.) Run shelltest -- --help for a list. Here are some useful ones:

  -j NUM  --threads=NUMBER    number of threads to use to run tests
  -o NUM  --timeout=NUMBER    how many seconds a test should be run for before giving up
  -t PAT  --select-tests=PAT  only tests that match at least one glob pattern given by
                               an instance of this argument will be run
          --hide-successes    hide sucessful tests, and only show failures

Example:

Run

$ shelltest tests -c -- -targs -j8 -o1 --hide

2 A test's name is what you see when running tests, ie the file name plus the sequence number within the file.

Contributing

The released version is on hackage (Release notes).

The latest code is on darcs hub3 (browse, changes). Clone it with:

$ darcs get http://hub.darcs.net/simon/shelltestrunner

Feedback, testing, code, documentation, packaging, blogging are most welcome. Here's the 2012 user survey. or chat me (sm on irc.freenode.net).

3 For help with darcs see here.

Credits

Simon Michael wrote shelltestrunner, inspired by John Wiegley's test system for Ledger.

Code contributors include: John Macfarlane, Bernie Pope, Trygve Laugstøl, Iustin Pop, Sergei Trofimovich, Andrés Sicard-Ramírez, John Chee.

shelltestrunner depends on several fine libraries, in particular Max Bolingbroke's test-framework, and of course on the Glorious Haskell Compiler.

The Blade Runner font is by Phil Steinschneider.