Documentation/test-suite

	Sparse test suite
	~~~~~~~~~~~~~~~~~

Sparse has a number of test cases in its validation directory. The test-suite
script aims at making automated checking of these tests possible. It works by
embedding tags in C comments in the test cases.

check-name: (mandatory)
	Name of the test.

check-description: (optional)
	A description of what the test checks.

check-command: (optional)
	There are different kinds of tests. Some can validate the sparse
	preprocessor, while others will use sparse, cgcc, or even other backends
	of the library. check-command allows you to give a custom command to
	run the test-case.
	The '$file' string is special. It will be expanded to the file name at
	run time.
	It defaults to "sparse $file".

check-exit-value: (optional)
	The expected exit value of check-command. It defaults to 0.

check-output-start / check-output-end (optional)
	The expected output (stdout and stderr) of check-command lies between
	those two tags. It defaults to no output.

check-known-to-fail (optional)
	Mark the test as being known to fail.


	Using test-suite
	~~~~~~~~~~~~~~~~

The test-suite script is called through the check target of the Makefile. It
will try to check every test case it finds (find validation -name '*.c').

It can be called to check a single test with:
$ cd validation
$ ./test-suite single preprocessor/preprocessor1.c
     TEST     Preprocessor #1 (preprocessor/preprocessor1.c)
preprocessor/preprocessor1.c passed !


	Writing a test
	~~~~~~~~~~~~~~

test-suite comes with a format command to make a test easier to write:

	test-suite format file [name [cmd]]

name:
	check-name value. If no name is provided, it defaults to the file name.
cmd:
	check-command value. If no cmd is provided, it defaults to
	"sparse $file".

The output of the test-suite format command can be redirected into the
test case to create a test-suite formated file.

$ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c
$ cat !$
cat bad-assignment.c
/*
 * check-name: bad assignment
 *
 * check-command: sparse $file
 * check-exit-value: 1
 *
 * check-output-start
bad-assignment.c:3:6: error: Expected ; at end of statement
bad-assignment.c:3:6: error: got \
 * check-output-end
 */

You can define the check-command you want to use for the test. $file will be
extended to the file name at run time.

$ ./test-suite format validation/preprocessor2.c "Preprocessor #2" \
		"sparse -E \$file" >> validation/preprocessor2.c
$ cat !$
cat validation/preprocessor2.c
/*
 * This one we happen to get right.
 *
 * It should result in a simple
 *
 *	a + b
 *
 * for a proper preprocessor.
 */
#define TWO a, b

#define UNARY(x) BINARY(x)
#define BINARY(x, y) x + y

UNARY(TWO)
/*
 * check-name: Preprocessor #2
 *
 * check-command: sparse -E $file
 * check-exit-value: 0
 *
 * check-output-start

a + b
 * check-output-end
 */