Autoconf, Automake, and Libtool: Adding a Test Suite

| ## 12.5 Adding a Test Suite Automake has very flexible support for automated test-suites within a project distribution, which are discussed more fully in the Automake manual. I have added a simple shell script based testing facility to Sic using this support -- this kind of testing mechanism is perfectly adequate for command line projects. The tests themselves simply feed prescribed input to the uninstalled `sic` interpreter and compare the actual output with what is expected. Here is one of the test scripts:
    ## -*- sh -*-
    ## incomplete.test -- Test incomplete command handling

    # Common definitions
    if test -z "$srcdir"; then
        srcdir=echo "$0" | sed s,[^/]*$,,'
        test "$srcdir" = "$0" && srcdir=.
        test -z "$srcdir" && srcdir=.
        test "${VERBOSE+set}" != set && VERBOSE=1
    fi
    . $srcdir/defs

    # this is the test script
    cat <<\EOF > in.sic
    echo "1
    2
    3"
    EOF

    # this is the output we should expect to see
    cat <<\EOF >ok
    1
    2
    3
    EOF

    cat <<\EOF >errok
    EOF

    # Run the test saving stderr to a file, and showing stdout
    # if VERBOSE == 1
    $RUNSIC in.sic  2> err | tee -i out >&2

    # Test against expected output
    if ${CMP} -s out ok; then
        :
    else
        echo "ok:" >&2
        cat ok >&2
        exit 1
    fi

    # Munge error output to remove leading directories, `lt-' or
    # trailing `.exe'
    sed -e "s,^[^:]*[lt-]*sic[.ex]*:,sic:," err >sederr && mv sederr err

    # Show stderr if doesnt match expected output if VERBOSE == 1
    if "$CMP" -s err errok; then
        :
    else
        echo "err:" >&2
        cat err >&2
        echo "errok:" >&2
        cat errok >&2
        exit 1
    fi
The tricky part of this script is the first part which discovers the location of (and loads) ``$srcdir/defs'`. It is a little convoluted because it needs to work if the user has compiled the project in a separate build tree -- in which case the ``defs'` file is in a separate source tree and not in the actual directory in which the test is executed. The ``defs'` file allows me to factor out the common definitions from each of the test files so that it can be maintained once in a single file that is read by all of the tests:
    #! /bin/sh

    # Make sure srcdir is an absolute path.  Supply the variable
    # if it does not exist.  We want to be able to run the tests
    # stand-alone!!
    #
    srcdir=${srcdir-.}
    if test ! -d $srcdir ; then
        echo "defs: installation error" 1>&2
        exit 1
    fi

    #  IF the source directory is a Unix or a DOS root directory, ...
    #
    case "$srcdir" in
        /* | [A-Za-z]:\\*) ;;
        *) srcdir=`\cd $srcdir && pwd` ;;
    esac

    case "$top_builddir" in
        /* | [A-Za-z]:\\*) ;;
        *) top_builddir=`\cd ${top_builddir-..} && pwd` ;;
    esac

    progname=`echo "$0" | sed 's,^.*/,,'`
    testname=`echo "$progname" | sed 's,-.*$,,'`
    testsubdir=${testsubdir-testSubDir}

    SIC_MODULE_PATH=$top_builddir/modules
    export SIC_MODULE_PATH

    # User can set VERBOSE to prevent output redirection
    case x$VERBOSE in
        xNO | xno | x0 | x)
            exec > /dev/null 2>&1
            ;;
    esac

    rm -rf $testsubdir > /dev/null 2>&1
    mkdir $testsubdir
    cd $testsubdir \
       || { echo "Cannot make or change into $testsubdir"; exit 1; }

    echo "=== Running test $progname"

    CMP="${CMP-cmp}"
    RUNSIC="${top_builddir}/src/sic"
Having written a few more test scripts, and made sure that they are working by running them from the command line, all that remains is to write a suitable ``Makefile.am'` so that `automake` can run the test suite automatically.
    ## Makefile.am -- Process this file with automake to produce Makefile.in

    EXTRA_DIST              = defs $(TESTS)
    MAINTAINERCLEANFILES    = Makefile.in

    testsubdir              = testSubDir

    TESTS_ENVIRONMENT       = top_builddir=$(top_builddir)

    TESTS                   =                       \
                            empty-eval.test         \
                            empty-eval-2.test       \
                            empty-eval-3.test       \
                            incomplete.test         \
                            multicmd.test

    distclean-local:
            -rm -rf $(testsubdir)
I have used the ``testsubdir'` macro to run the tests in their own subdirectory so that the directory containing the actual test scripts is not polluted with lots of fallout files generated by running the tests. For completeness I have used a hook target(26) to remove this subdirectory when the user types:
    $ make distclean
    ...
    rm -rf testSubDir
    ...
Adding more tests is accomplished by creating a new test script and adding it to the list in `noinst_SCRIPTS`. Remembering to add the new ``tests'` subdirectory to ``configure.in'` and the top-level ``Makefile.am'`, and reconfiguring the project to propagate the changes into the various generated files, I can run the whole test suite from the top directory with:
    $ make check
It is often useful run tests in isolation, either when developing new tests, or to examine more closely why a test has failed unexpectedly. Having set this test suite up as I did, individual tests can be executed with:
    $ VERBOSE=1 make check TESTS=incomplete.test
    make  check-TESTS
    make[1]: Entering directory
    /tmp/sic/tests
    === Running test incomplete.test
    1
    2
    3
    PASS: incomplete.test
    ==================
    All 1 tests passed
    ==================
    make[1]: Leaving directory /tmp/sic/tests
    $ ls testSubDir/
    err   errok   in.sic   ok   out
The ``testSubDir'` subdirectory now contains the expected and actual output from that particular test for both ``stdout'` and ``stderr'`, and the input file which generated the actual output. Had the test failed, I would be able to look at these files to decide whether there is a bug in the program or simply a bug in the test script. Being able to examine individual tests like this is invaluable, especially when the test suite becomes very large -- because you will, naturally, add tests every time you add features to a project or find and fix a bug. Another alternative to the pure shell based test mechanism I have presented here is the Autotest facility by Fran@,cois Pinard, as used in Autoconf after release 2.13. Later in 20. A Complex GNU Autotools Project, the Sic project will be revisited to take advantage of some of the more advanced features of GNU Autotools. But first these advanced features will be discussed in the next several chapters -- starting, in the next chapter, with a discussion of how GNU Autotools can help you to make a tarred distribution of your own projects.

This document was generated by Gary V. Vaughan on February, 8 2006 using texi2html