The DejaGnu Testing Framework

for DejaGnu Version 1.0

May 1993

Copyright © 1992, 1993 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

The Appendix “Tool Command Language Overview” was written by John Ousterhout, of the University of California at Berkeley. Special copyright terms apply to that Appendix; @pxref{Tcl,,Tool Command Language Overview}.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided also that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions.

1 What is DejaGnu?

DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Beyond this, DejaGnu offers several advantages for testing:

1. The flexibility and consistency of the DejaGnu framework make it easy to write tests for any program.
2. DejaGnu provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test for GDB can run (from any Unix based host) on any target architecture that DejaGnu supports. Currently DejaGnu runs tests on several single board computers, whose operating software ranges from just a boot monitor to a full-fledged, Unix-like realtime OS.
3. All tests have the same output format. This makes it easy to integrate testing into other software development processes.

DejaGnu is written in expect, which in turn uses Tcl (Tool command language).

Running tests requires two things: the testing framework, and the test suites themselves. Tests are usually written in expect using Tcl, but you can also use a Tcl script to run a test suite that is not based on expect.

1.1 Running existing tests

To run tests from an existing collection, first use configure as usual to set up the source directory containing the tests. Then try running

make check

If the check target exists, it usually saves you some trouble—for instance, it can set up any auxiliary programs or other files needed by the tests.

Once you have run ‘make check’ to build any auxiliary files, you might want to call the test driver runtest directly to repeat the tests. You may also have to call runtest directly for test collections with no check target in the ‘Makefile’.

Typically, you must use two command-line options: ‘--tool’, to specify which set of tests to run(1), and ‘--srcdir’, to specify where to find test directories.

For example, if the directory ‘gdb/testsuite’ contains a collection of DejaGnu tests for GDB, you can run them like this:

eg$ cd gdb/testsuite
eg$ runtest --tool gdb
Test output follows, ending with:

		=== gdb Summary ===

# of expected passes 508
# of expected failures 103
# of unexpected successes 0
# of unexpected failures 0
/usr/latest/bin/gdb version 4.8.3 -nx

You can use the option ‘--srcdir’ to point to some other directory containing a collection of tests:

eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite

These examples assume a native configuration, where the same computer runs both runtest and the tests themselves. When you have a cross configuration, the tests run on a different computer, controlled by the host running runtest. In this situation, you need the option ‘--name’ to specify the network address for the other computer:

eg$ runtest --tool gdb --name vx9.munist.com

If you always use the same option values, you can record them in a file called ‘site.exp’, rather than typing them each time. See section Site Configuration.

By default, runtest prints only the names of the tests it runs, the tests that fail (or succeed unexpectedly), and a summary. To include all tests in the output (those that pass, and those that are expected to fail), use the ‘--all’ flag. For more verbose output about processes being run, communication, and so on, use ‘--verbose’. To see even more output, use multiple ‘--verbose’ options. See section Using runtest, for a more detailed explanation of each runtest option.

Test output goes into two files in your current directory: summary output in ‘tool.sum’, and detailed output in ‘tool.log’. (tool refers to the collection of tests; for example, after a run with ‘--tool gdb’, look for output files ‘gdb.sum’ and ‘gdb.log’.) See section The files DejaGnu writes.

1.2 What does a DejaGnu test look like?

Each DejaGnu test is an expect script; the tests vary widely in complexity, depending on the nature of the tool and the feature tested.

Here is a very simple GDB test—one of the simplest tests shipped with DejaGnu (extracted from ‘gdb.t00/echo.exp’):(2)

# send a string to the GDB stdin:
send "echo Hello world!\n"

# inspect the GDB stdout for the correct reply,
# and determine whether the test passes or fails:
expect {
  -re "Hello world.*$prompt $"    { pass "Echo test" }
  -re "$prompt $"                 { fail "Echo test" }
  timeout                         { fail "(timeout) Echo test" }
  }

Though brief, this example is a complete test. It illustrates some of the main features of DejaGnu test scripts:

• The test case does not start the tested program (GDB in this case); all test scripts for interactive tools can assume the corresponding tool is running.
• Comments start with ‘#’.
• The main commands you use to control a tested program are send (to give it commands) and expect (to analyze its responses).
• The expect command uses a list of pairs; a pattern (regular expression if ‘-re’ specified), followed by an action to run if the pattern matches output from the program. Only the action for the first matching pattern will execute.
• Test cases use the commands pass and fail to record the test outcome.

1.3 Design goals

DejaGnu grew out of the internal needs of Cygnus Support. Cygnus maintains and enhances a variety of free programs on many different environments, and we needed a testing tool that:

• is useful to developers while fixing bugs;
• automates running many tests during the release process;
• is portable between a variety of host computers;
• supports cross-development testing;
• permits testing interactive programs, like GDB; and
• permits testing batch oriented programs, like GCC.

DejaGnu was designed to meet these goals.

Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important: for instance, it is imperative to test the GDB back end on several remote targets.

Probably the greatest challenge was testing in a cross-development environment (which can be a real nightmare). Most cross-development environments are customized by each developer. Even when buying packaged boards from vendors there are many differences. The communcation interfaces vary from a serial line to ethernet. DejaGnu was designed with a modular communication setup, so that each kind of communication can be added as required, and supported thereafter. Once a communication procedure is coded, any test can use it. Currently DejaGnu can use rsh, rlogin, telnet, tip, kermit, and mondfe(3) for remote communications.

Julia Menapace first coined the term “Deja Gnu” to describe an earlier testing framework at Cygnus Support. When we replaced it with the Expect-based framework, we could not resist using the name DejaGnu all over again…

1.4 Future directions

In the near future, there are two parallel directions for DejaGnu development. The first is to add support for more targets; part of this consists of porting Tcl and expect to new hosts as well. The second is to add more test cases, and support for more tools. It would be particularly useful to add tests for conformance to the ANSI C and C++ specifications, to POSIX, and to other standards.

Another useful extension would be to permit testing programs with a more complex interface, whether text based or GUI based. Two components already exist: a Tcl based X window toolkit, and a terminal package for expect. Both of these must be merged into DejaGnu in a way that permits testing programs that run in each environment.

1.5 Tcl and Expect

Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter Usenix conference, Tcl: An Embeddable Command Language. That paper is included in PostScript form in the ‘doc’ subdirectory of the Tcl distribution. @xref{Tcl,,Tool Command Language Overview}, for details on Tcl.

Don Libes introduced expect in his paper expect: Curing Those Uncontrollable Fits of Interaction at the 1990 Summer Usenix conference. The paper is included in PostScript form in the expect distribution (as are several other papers about expect). @xref{Expect,,Expect: Programmed Dialogue}, for details on expect.

2 Using runtest

runtest is the executable test driver for DejaGnu. You can specify two kinds of things on the runtest command line: command line options, and Tcl variables for the test scripts. The options are listed alphabetically below. You can control the defaults for all runtest options; see Site configuration.

runtest returns an exit code of 1 if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns 0 as the exit code.

runtest flags the outcome of each test as one of these four cases:

PASS

The most desirable outcome: the test succeeded, and was expected to succeed.

XPASS

A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.

FAIL

A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.

XFAIL

A test failed, but it was expected to fail. This result indicates no change in a known bug.

runtest may also display the following messages:

ERROR

Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target.

WARNING

Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.

This is the full set of command line options that expect recognizes.

runtest envvar=value…
[ --all ]  [ --connect type ]  [ --debug ]  [ --diff ]
[ --help ]  [ --mail "name1 name2 …" ]
[ --noreboot ]  [ --outdir path ] [ --objdir path ]
[ --runtests "file1 file2…" ]  [ --srcdir path ]
[ --strace n ]  [ --name name ]  --tool tool
[ --target string ] [ --host string ]
[ -v | --verbose ]  [ -V | -version ]

tclvar=value

You can define Tcl variables for use by your test scripts in the same style used with make for environment variables. For example, ‘runtest GDB=gdb.old’ defines a variable called ‘GDB’; when your scripts refer to ‘$GDB’ in this run, they use the value ‘gdb.old’.

The default Tcl variables used for most tools are defined in the main DejaGnu Makefile; their values are captured in the ‘site.exp’ file. See section Site configuration.

--all

Display all test output. By default, runtest shows only the output of tests that produce unexpected results; that is, tests with status ‘FAIL’ (unexpected failure), ‘XPASS’ (unexpected success), or ‘ERROR’ (a severe error in the test case itself). Specify ‘--all’ to see output for tests with status ‘PASS’ (success, as expected) ‘XFAIL’ (failure, as expected), or ‘WARNING’ (minor error in the test case itself).

--baud baud rate

-b

Set the default baud rate to something other than 9600. (Some serial interface programs, like tip, use a separate initialization file instead of this value.)

--connect type

Connect to a target testing environment as specified by type, if the target is not the computer running runtest. For example, use ‘--connect’ to change the program used to connect to a “bare board” boot monitor. The choices for type in the DejaGnu 1.0 distribution are ‘rlogin’, ‘telnet’, ‘rsh’, ‘tip’, ‘kermit’, and ‘mondfe’.(4)

The default for this option depends on the configured target (see section Remote targets supported).

--debug

Turns on the expect internal debugging output. Debugging output is displayed as part of the runtest output, and logged to a file called ‘dbg.log’. The output shows all attempts at matching the test output of the tool with the scripted patterns describing expected output. The output generated with ‘--strace’ also goes into this file.

--diff filename

Show on the standard output device the differences between the summary output from the current test run, and a summary from a previous run. runtest does not use the Unix utility diff; instead it uses an internal Tcl based implementation that is better suited to output logs (and is reliably portable).

Warning: The ‘--diff’ is not fully implemented in the DejaGnu 1.0 release.

--help

-he

Prints out a short summary of the runtest options, then exits (even if you also specify other options).

--mail "name1 name2 …"

--mail name

Send the summary testing output to the specified list, via email. If used with the ‘--diff’ option, only the diff file is mailed. The quotes are only needed if you supply more than one name with a single ‘--mail’ option.

--noreboot

Do not reboot the target board when runtest initializes. Usually, when running tests on a separate target board, it is safer to reboot the target to be certain of its state. However, when developing test scripts, this option can save a lot of time.

--outdir path

Write output logs in directory path. The default is ‘.’, the directory where you start runtest. This option affects only the summary and the detailed log files ‘tool.sum’ and ‘tool.log’. The DejaGnu debug log ‘dbg.log’ always appears (when requested) in the local directory.

--runtest "file1 file2 …"

--runtest filename

Specify the names of test files to run. Use this option to choose a subset of the tests available in the ‘--srcdir’ directory. You may supply just filenames with no path; the subdirectory name is not necessary.

--objdir path

Use path as the top directory containing any auxiliary compiled test code. This defaults to ‘.’. Use this option to locate pre-compiled test code. You can normally prepare any auxiliary files needed with make.

--srcdir path

Use path as the top directory for test scripts to run. runtest looks in this directory for any subdirectory whose name begins with the toolname (specified with ‘--tool’). For instance, with ‘--tool gdb’, runtest uses tests in subdirectories ‘gdb.*’ (with the usual shell-like filename expansion). If you do not use ‘--srcdir’, runtest looks for test directories under the current working directory.

--strace n

Turn on internal tracing for expect, to n levels deep. By adjusting the level, you can control the extent to which your output expands multi-level Tcl statements. This allows you to ignore some levels of case or if statements. Each procedure call or control structure counts as one “level”.

The output is recorded in the same file, ‘dbg.log’, used for output from ‘--debug’.

--target string

string is a full configuration “triple” name(5) as used by configure. Use this option to override the default setting (running native tests). This option changes the configuration runtest uses for the default tool names, and other setup information. See Using configure in Cygnus configure, for details about configure names.

--host string

string is a full configuration “triple” name as used by configure. Use this option to override the default setting for the host. This choice affects only the procedures istarget, isnative, and setup_xfail.

--name name

name is a name for the particular testing target machine (for cross testing). If the testing target has IP network support (for example, RPC or NFS), this is the network name for the target itself. (name is not the configuration string you specify as a target with configure; the ‘--name’ option names a particular target, rather than describing a class of targets.) For targets that connect in other ways, the meaning of the name string depends on the connection method. See section Remote targets supported.

--tool tool

tool specifies what set of tests to run, and what initialization module to use. tool is used only for these two purposes: it is not used to name the executable program to test. Executable tool names (and paths) are recorded in ‘site.exp’ (see section Site configuration), and you can override them by specifying Tcl variables on the command line.

For example, including ‘--tool gcc’ on the runtest command line runs tests from all test subdirectories whose names match ‘gcc.*’, and uses one of the initialization modules named ‘config/*-gcc.exp’. To specify the name of the compiler (perhaps as an alternative path to what runtest would use by default), use ‘GCC=binname’ on the runtest command line.

--verbose

-v

Turns on more output. Repeating this flag increases the amount of output displayed. Level one (‘-v’) is simply test output. Level two (‘-v -v’) shows messages on options, configuration, and process control. Verbose messages appear in the detailed (‘*.log’) log file, but not in the summary (‘*.sum’) log file.

--version

-V

Prints out the version numbers of DejaGnu, expect and Tcl.

3 Configuring DejaGnu

Once you have the DejaGnu source unpacked and available, the first requirement before you start using DejaGnu is to configure for the host (where runtest executes) and target (where the tools and their output are tested). Moreover, it is usually best to configure in a directory separate from the source tree, specifying where to find the source with the ‘--srcdir’ option to configure:

configure host-type --target=target-type --srcdir=path

To run tests on a full-fledged operating system, you can specify host and target as the same machine; we call this a native configuration. For native configurations, specifying the target explicitly is not necessary. For example, to configure for a Sun-4 running SunOS:

configure sun4 --srcdir=/usr/cygnus/src/dejagnu

When the host and target are not the same (typically when testing software tools on a “bare board” controlled from a separate general-purpose computer), we refer to a cross configuration. For example, to run tests controlled by Sun-4 on an AMD 29000 board, communicating via the AMD “UDI” protocol:

configure sun4 --target=a29k-amd-udi \
    --srcdir=/usr/cygnus/src/dejagnu

You may also want to use the configure option ‘--prefix’ to specify where you want DejaGnu and its supporting code installed. By default, installation is in subdirectories of ‘/usr/local’, but you can select any alternate directory altdir by including ‘--prefix=altdir’ on the configure command line. (This value is captured in the Makefile variables prefix and exec_prefix.)

DejaGnu uses Cygnus Configure. See What Configure Does in Cygnus Configure, for more information on its capabilities, conventions, and command-line options.

Aside from the usual effect of preparing and building the executables on your host, configuring and making DejaGnu creates a file called ‘site.exp’. This file encapsulates information from both configure and make, as default values for DejaGnu tests. See section Site Configuration.

DejaGnu has some additional requirements for target support, beyond the general-purpose provisions of Cygnus configure. runtest must actively communicate with the target, rather than simply generating or managing code for the target architecture. Therefore, each tool requires an initialization module for each target. For new targets, you must supply a few Tcl procedures to adapt DejaGnu to the target. This permits DejaGnu itself to remain target independent. See section Initialization module.

Save for a small number of example tests, the DejaGnu distribution itself does not include any test suites; these are available separately. Test suites for the GNU compiler (testing both GCC and G++) and for the GNU binary utilities are distributed in parallel with the DejaGnu distribution (but packaged as separate files). The test suite for the GNU debugger is distributed in parallel with each release of GDB itself, starting with GDB 4.9. After configuring the top-level DejaGnu directory, unpack and configure the test directories for the tools you want to test; then, in each test directory, run make to build auxiliary programs required by some of the tests.

3.1 Installing DejaGnu

To install DejaGnu in your filesystem (either in ‘/usr/local’, or as specified by your ‘--prefix’ option to configure), execute

eg$ make install

‘make install’ does these things for DejaGnu:

1. Look in the path specified for executables (‘$exec_prefix’) for directories called ‘lib’ and ‘bin’. If these directories do not exist, ‘make install’ creates them.
2. Create another directory in the ‘lib’ directory, called ‘dejagnu’.
3. Create a subdirectory of ‘lib/dejagnu’ whose name is the configuration string. For a native configuration, ‘make install’ creates two configuration-specific directories: one with the usual configuration name, and another called ‘native’.
4. Copy the runtest shell script to ‘$exec_prefix/bin’.
5. Copy all the library files (used to support the framework) into ‘$exec_prefix/lib/dejagnu’.
6. Copy ‘runtest.exp’ into ‘$exec_prefix/lib/dejagnu’. This is the main Tcl code implementing DejaGnu.
7. Copy the ‘site.exp’ file (which captures configuration values in Tcl variables) into a subdirectory named after the configuration, in the directory ‘$exec_prefix/lib/dejagnu/’. On a native configuration, ‘make install’ also copies ‘site.exp’ into ‘native/’ in the same directory.

Each test suite collection comes with simple installation instructions in a ‘README’ file; in general, the test suites are designed to be unpacked in the source directory for the corresponding tool, and extract into a directory called ‘testsuite’.

3.2 Site configuration

The site configuration file, ‘site.exp’, captures configuration-dependent values from environment variables and propagates them to the DejaGnu test environment using Tcl variables. This ties the DejaGnu test scripts into the configure and make programs.

DejaGnu supports more than one ‘site.exp’ file. First, make install for the DejaGnu distribution places a “master” ‘site.exp’ in each configuration-dependent subdirectory of the DejaGnu library directory. Second, each directory of tests usually has another ‘site.exp’, initially created by ‘make check’. The multiple instances of ‘site.exp’; are loaded in a fixed order built into DejaGnu (the more local last). You can further override the default values from this automatic process in a user-editable section of the file, or by setting variables on the runtest command line.

1. There is a “master” ‘site.exp’, capturing configuration values that apply to DejaGnu across the board, in each configuration-specific subdirectory of the DejaGnu library directory. runtest loads these values first.
2. Any directory containing a configured test suite also has a ‘site.exp’, capturing configuration values specific to the tool under test. Since runtest loads these values last, the individual test configuration can either rely on and use, or override, any of the global values from the “master” ‘site.exp’.

In either case, you can generate or update ‘site.exp’ by typing make site.exp in the appropriate configured directory.

All ‘site.exp’ files have two sections, separated by comment text. The first section starts with:

## these variables are automatically generated by make ##
# Do not edit here. If you wish to override these values
# add them to the last section

Do not edit this first section; if you do, your changes are replaced next time you run make. The first section ends with this line:

## All variables above are generated by configure. Do Not Edit ##

The first section is the part that is generated by make. It is essentially a collection of Tcl variable definitions based on ‘Makefile’ environment variables. Since they are generated by make, they contain the values as specified by configure. (You can also customize these values by using the ‘--site’ option to configure.) In particular, this section contains the ‘Makefile’ variables for host and target configuration data.

In the second section, you can override any default values (locally to DejaGnu) for all the variables. The second section can also contain your preferred defaults for all the command line options to runtest. This allows you to easily customize runtest for your preferences in each configured test-suite tree, so that you need not type options repeatedly on the command line. (The second section may also be empty, if you do not wish to override any defaults.)

It is usually more convenient to keep these “manual overrides” in the ‘site.exp’ local to each test directory, rather than in the “master” ‘site.exp’ in the DejaGnu library.

To override the values captured in the first part of ‘site.exp’ by configure and make, simply set your own value for any of these variables in the second part of ‘site.exp’. Set the variables using the Tcl set command; for example, to run tests on an experimental version of GCC, the second part of your ‘site.exp’ might include:

set CC "/working/devo/gcc/xgcc"

Here are the ‘Makefile’ variables set in ‘site.exp’. The default values depend on your configuration. The defaults shown are those built into the DejaGnu distribution, which will appear only if your configuration does not change them. (For example, in cross configurations, program names often reflect the target type.)

CC

The C compiler. The default value is "cc".

CFLAGS

The list of options passed to the C compiler. The default value is ‘-g’.

CXX

The C++ compiler. The corresponding variable for make is C++, but Tcl variable names cannot contain the ‘+’ character. The default value is "gcc".

CXXFLAGS

The list of options for the C++ compiler. The default value is "-g".

AS

The assembler. The default value is "as".

ASFLAGS

The list of options for the assembler. The default value is "".

FC

The FORTRAN compiler. The default value is "".

FCFLAGS

The list of options for the FORTRAN compiler. The default value is "".

LD

The linker. The default value is "ld".

LDFLAGS

The list of options for the linker. The default value is "".

RANLIB

The program for writing an index to object code archive files. The default value is "ranlib".

RANLIBFLAGS

The options for $RANLIB. The default value is "".

NM

The program that dumps the list of symbols from an object file. The default value is "nm".

NMFLAGS

The options for $NM. The default value is "".

SIZE

This is the program that lists the size of all the sections in the compiled file. The default value is "size".

SIZEFLAGS

The options for $SIZE. The default value is "".

STRIP

The program to strip the symbol table from compiled files to reduce size. The default value is "strip".

STRIPFLAGS

The options for $STRIP. The default value is "".

OBJDUMP

The program that dumps the parts of an object file. The default value is "objdump".

OBJDUMPFLAGS

The options for the object file dumper. The default value is "".

LEX

The program that makes lexical analyzers. The default value is "flex".

LEXFLAGS

The options for the lexical analyzer generator. The default value is "".

YACC

The “compiler-compiler”, a program to generate parsers. The default value is "byacc".

YACCFLAGS

The options for $YACC. The default value is "".

AR

The program that builds object code archives. The default value is "ar".

ARFLAGS

The options for the archiver. The default value is "".

CPP

The C compiler preprocessor. The default value is "gcc -E".

CPPFLAGS

The options for the C preprocessor. The default value is "".

GDB

The C and C++ debugger. The default value is "gdb".

GDBFLAGS

The options for the debugger. The default value is "".

DIFF

The program that finds differences in text files. The default value is "diff".

DIFFFLAGS

The options for the diff program. The default value is "".

The following variables, also in ‘site.exp’, extend the standard set of configuration variables. You should never redefine these; they permit runtest to determine its configuration at runtime.

host_os

The host operating system name, as seen by ‘config.sub’. The default depends on the value you specify with ‘configure --host’ for your configuration.

host_alias

A shorter name for the host system. The default depends on the value you specify with ‘configure --host’ for your configuration.

host_cpu

The host processor type. The default depends on the value you specify with ‘configure --host’ for your configuration.

host_vendor

The manufacturer of the host system. The default depends on the value you specify with ‘configure --host’ for your configuration.

host_triplet

This is the canonical name produced by configure. Its value is the same as ‘$host_cpu-$host_vendor-$host_os’.

target_os

The target operating system name, as seen by ‘config.sub’. The default depends on the value you specify with ‘configure --target’.

target_alias

A possibly shorter name for the target system. The default depends on the value you specify with ‘configure --target’.

target_cpu

The target processor type. The default depends on the value you specify with ‘configure --target’.

target_vendor

The manufacturer of the host system. The default depends on the value you specify with ‘configure --target’.

target_triplet

This is the canonical name produced by configure. Its valeu is the same as ‘$target_cpu-$target_vendor-$target_os’.

In the user editable second section of ‘site.exp’, you can not only override the configuration variables captured in the first section, but also specify default values for all the runtest command line options. Save for ‘--debug’, ‘--help’, and ‘--version’, each command line option has an associated Tcl variable. Use the Tcl set command to specify a new default value (as for the configuration variables). The following table describes the correspondence between command line options and variables you can set in ‘site.exp’. See section Running the Tests, for explanations of the command-line options.

3.3 Initialization module

The initialization module (or “init file”) has two purposes: to provide tool and target dependent procedures, and to start up an interactive tool to the point where it is ready to operate. The latter includes establishing communications with the target. All the tests for interactive programs assume that the tool is already running and communicating. Initialization modules for non-interactive programs may only need to supply the support functions.

Each test suite directory must contain (in its ‘config’ subdirectory) a separate initialization module for each target. The appropriate init file is identified by name: it must be named ‘$target_os.exp’, where target_os is the Tcl variable recording the configuration name for the target operating system.

If the test suite contains tests for more than one tool, it can either use a single init file for all the tools, or segregate the init files by tool using a more elaborate naming convention: ‘$target_os-$tool.exp’, where tool is the Tcl variable identifying tests for a particular tool. It is better to avoid this convention if possible, since it may lead to unacceptable constraints on tool names due to systems with low limits on file-name length. You can always segregate tests for different tools into different test-suite directories, instead.

In a test running native, writing the Tcl code for initialization is usually quite simple. For cross configurations, however, more elaborate instructions are usually needed to describe how to talk to a remote target.

There are conventionally four procedures in each initialization module. Their names begin with ‘$tool’, the string that identifies tests for a particular tool: $tool_start, $tool_load, $tool_exit, and $tool_version. For example, the start procedure for GDB is called gdb_start. (Since start procedures are used differently for batch and interactive tools, however, runtest itself never calls the start procedure. Init files for interactive tools are expected to end by running the start procedure.)

The initialization module is also a good place to call load_lib to get any collections of utility procedures meant for a family of test cases, and to set up default values for any additional Tcl variables needed for a specific set of tests.

See section Target dependent functions, for full descriptions of these procedures.

3.4 Remote targets supported

You can set the target name and the connect mode in the ‘site.exp’ file (using the Tcl variables ‘targetname’ and ‘connectmode’, respectively), or on the runtest command line (using ‘--target’ and ‘--connect’).

The DejaGnu distribution includes support for the following remote targets. If you suspect a communication problem, try running the connection interactively from expect. (There are three ways of running expect as an interactive interpreter. You can run expect with no arguments, and control it completely interactively; or you can use ‘expect -i’ together with other command-line options and arguments; or you can run the command interpreter from any expect procedure. Use return to get back to the calling procedure (if any), or return -tcl to make the calling procedure itself return to its caller; use exit or end-of-file to leave expect altogether.) Run the program whose name is recorded in ‘$connectmode’, with the arguments in ‘$targetname’, to establish a connection. You should at least be able to get a prompt from any target that is physically connected. See section Adding a target, for some discussion of writing and debugging new targets.

AMD 29000, with UDI protocol

Configure DejaGnu for target ‘a29k-amd-udi’. (Cygnus configure also recognizes the abbreviation ‘udi29k’.) Then, to run tests, use the runtest target name to specify whether you want to use a simulator, or a particular hardware board. The particular string to use with ‘--target’ will depend on your UDI setup file, ‘udi_soc’ (if ‘udi_soc’ is not in your working directory, the environment variable ‘UDICONF’ should contain a path to this file). For example, if your UDI setup file includes these lines:

iss   AF_UNIX  *   isstip -r /home/gnu/29k/src/osboot/sim/osboot
mon   AF_UNIX  *   montip -t serial -baud 9600 -com /dev/ttyb

 

use ‘--target iss’ to run tests on the simulator, and ‘--target mon’ to run tests on the 29K hardware. See the manufacturer’s manuals for more information on UDI and ‘udi_soc’.

The default connect protocol is ‘mondfe’ with either back end. mondfe is the only shell DejaGnu supports for UDI targets.

Warning: This target requires GDB version 4.7.2 (or greater). Earlier versions of GDB do not fully support the load command on this target, so DejaGnu has no way to load executable files from the debugger.

Motorola 680x0 boards, a.out or COFF object format

Configure DejaGnu for any remote target matching ‘m68k-*’.

Warning: Most ‘m68k-*’ configurations run all tests only for native testing (when the target is the same as the host). When you specify most of these targets for a cross configuration, you will only be able to use tests that run completely within the host (for example, tests of the binary utilities such as the archiver; or compiler tests that only generate code rather than running it).

To run a.out or COFF binaries on a remote M68K, you must configure DejaGnu for a particular target board. ‘m68k-abug’ is an example. (In general for an embedded environment, because it dosen’t have absolute addresses, a.out is not a good choice for output format in any case; most often S-records or Hex-32 are used instead.)

Motorola 68K MVME board running 135ABug boot monitor

Configure for ‘m68k-abug-aout’ or ‘m68k-abug-coff’ (as a target). This boot monitor can only download S-records; therefore, the DejaGnu tests for this environment require a linker command script to convert either output format to S-records, setting the default addresses for .text, .bss, and .data.

With this configuration, the default for ‘--connect’ is ‘tip’. ‘tip’ is the only communications protocol supported for connecting to ‘m68k-abug-*’ targets. ‘tip’ uses an ASCII downloader (the ~put command) to load S-records into the target board. The ‘--target’ string must be a machine name that tip understands (for example, on some tip implementations it must be an entry from the initialization file for tip; this file is sometimes called ‘/etc/remote’).

See your system documentation for information on how to create new entries in ‘/etc/remote’. (Some UNIX systems are distributed with at least one default entry with a name resembling ‘hardwire’; if your system has one, you can edit it, or make a modified copy with a new name.) When you have a working ‘/etc/remote’ entry abugtarget, you should be able to type ‘tip abugtarget’, and get the prompt ‘135ABUG>’ from the board. Use the same abugtarget string with ‘runtest --target’.

Motorola IDP board running the rom68k boot monitor

This is the same in functionality as the MVME board running the BUG boot monitor. Only the monitor commands and the addresses are different.

VxWorks (Motorola 68K or Intel 960)

Configure DejaGnu for either ‘m68k-wrs-vxworks’ (abbreviated ‘vxworks68’) or ‘i960-wrs-vxworks’ (abbreviated ‘vxworks960’). Since both targets support IP addressing, specify the network address (for example, a host name from ‘/etc/hosts’) with ‘--target’.

The default connect protocol is ‘rlogin’, but you can use any of ‘--connect rlogin’, ‘--connect telnet’, or ‘--connect rsh’.

Test scripts need no special code to load programs into these targets; since VxWorks supports NFS, all you must do is ensure test programs are on an exported filesystem.

When you compile for VxWorks, use the linker ‘-r’ option to make the linker output relocatable—at least if you want to use library routines. Many standard C routines are included in VxWorks; often no additional libraries are needed. See your VxWorks system documentation for additional details.

4 The DejaGnu Implementation

DejaGnu is entirely written in expect, which uses Tcl as a command language. expect serves as a very programmable shell; you can run any program, as with the usual Unix command shells—but once the program is started, your expect script has fully programmable control of its input and output. This does not just apply to the programs under test; expect can also run any auxiliary program, such as diff or sh, with full control over its input and output.

The most important part of DejaGnu—and the largest body of code—is the set of test suites distributed separately for each GNU tool. Future releases of GNU tools will include even more tests, developed throughout the free software community.

runtest, of course, is the glue to tie together and manage the test scripts. The runtest program is actually a simple Bourne shell script that locates a copy of the expect shell and then starts the main Tcl code, runtest.exp. runtest.exp itself has these essential functions:

1. Parse the command line options, load the library files, and load the default configuration files.
2. Locating the individual test scripts. runtest.exp locates the tests by exploiting a straightforward naming convention based on the string you specify with the ‘--tool’ option.
3. Providing an extended test environment, by defining additional Tcl procedures beyond those already in expect.
4. Locating target-dependent functions, to standardize the test environment across a wide variety of test platforms.

4.1 Conventions for using tool names

DejaGnu uses ‘$tool’, the name of the tool under test, to tie together the testing configuration in a straightforward but flexible way.

Each initialization module is tool-dependent; their names optionally use the tool name as a suffix. If the ‘testsuite/config’ directory is only for one tool, then the tool name need not be part of the init file name. See section Initialization module.

‘$tool’ is not used to invoke the tool, since sites that run multiple configurations of a particular tool often call each configuration by a different name. runtest uses the configuration-dependent variables captured in ‘site.exp’ to determine how to call each tool.

runtest also uses tool names to find directories containing tests. runtest scans the source directory (specified with --srcdir) for all directories whose names start with the tool name followed by a period. For instance, directories that start with ‘g++.’ contain G++ tests. To add a new test, just put it in any directory (including, if you wish, an entirely new directory) whose name follows this convention.

A test is any file in an appropriately named subdirectory whose name ends in ‘.exp’ (the conventional way of naming expect scripts). These simple naming conventions make it as simple as possible to install new tests: all you must do is put the test in the right directory.

runtest sorts the tests in each subdirectory by name (using the Tcl lsort command) and runs them in the resulting order.

4.2 DejaGnu procedures

DejaGnu provides these Tcl procedures for use in test scripts. You can also use any standard expect or Tcl function.

error "string"

Declares detection of a severe error in the test case itself. error writes in the log files a message beginning with ‘ERROR’, appending the argument string. (This is a Tcl built-in; for more detail, @pxref{Tcl Commands, Built-in commands}).

fail "string"

Declares a test to have failed. fail writes in the log files a message beginning with ‘FAIL’ (or XFAIL, if failure was expected), appending the argument string.

ishost "host"

Tests for a particular host environment. If the currently configured host matches the argument string, the result is 1; otherwise the result is 0. host must be a full three-part configure host name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters to specify sets of names).

istarget "target"

Tests for a particular target environment. If the currently configured target matches the argument string, the result is 1; otherwise the result is 0. target must be a full three-part configure target name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters to specify sets of names).

isnative

Tests whether the current configuration has the same host and target. When it runs in a native configuration this procedure returns a 1; otherwise it returns a 0.

load_lib "library-file"

Loads the file library-file by searching a fixed path built into runtest. If DejaGnu has been installed, it looks in a path starting with the installed library directory. If DejaGnu is not installed, it defaults to the current directory. In either case, it then looks in the current directory for a directory called lib. If there are duplicate definitions, the last one loaded takes precedence over the earlier ones.

pass "string"

Declares a test to have passed. pass writes in the log files a message beginning with ‘PASS’ (or XPASS, if failure was expected), appending the argument string.

setup_xfail "config [bugno]"

Declares that the test is expected to fail on a particular set of configurations. The config argument must be a list of full three-part configure target name; in particular, you may not use the shorter nicknames supported by configure (but you can use the common shell wildcard characters to specify sets of names). The bugno argument is optional, and used only in the logging file output; use it as a link to a bug-tracking system such as GNATS (see Overview in Tracking Bugs With GNATS).

Warning: you must call either pass or fail after using setup_xfail in a test case. Otherwise, the expected-failure declaration applies to whatever test runs next, leading to surprising results.

verbose "string" number

Test cases can use this function to issue helpful messages depending on the number of ‘--verbose’ options on the runtest command line. It prints string if the value of the variable verbose is higher than the optional number. The default value for number is 1.

warning "string"

Declares detection of a minor error in the test case itself. warning writes in the log files a message beginning with ‘WARNING’, appending the argument string. Use warning rather than error for cases (such as communication failure to be followed by a retry) where the test case can recover from the error.

4.3 Target dependent functions

Each combination of target and tool requires some target-dependent procedures. The names of these procedures have a common form: the tool name, followed by an underbar ‘_’, and finally a suffix describing the procedure’s purpose. For example, a procedure to extract the version from GDB is called ‘gdb_version’. See section Initialization Module, for a discussion of how DejaGnu arranges to find the right procedures for each target.

tool_start

Starts a particular tool. For an interactive tool, tool_start starts and initializes the tool, leaving the tool up and running for the test cases; an example is gdb_start, the start function for GDB. For a batch oriented tool, tool_start is optional; the recommended convention is to let tool_start run the tool, leaving the output in a variable called comp_output. Test scripts can then analyze ‘$comp_output’ to determine the test results. An example of this second kind of start function is gcc_start, the start function for GCC.

runtest itself does not call tool_start. The initialization module ‘tool_init.exp’ must call tool_start for interactive tools; for batch-oriented tools, each individual test script calls tool_start (or makes other arrangements to run the tool).

tool_load

Loads something into a tool. For an interactive tool, this conditions the tool for a particular test case; for example, gdb_load loads a new executable file into the debugger. For batch oriented tools, tool_load may do nothing—though, for example, the GCC support uses gcc_load to load and run a binary on the target environment. Conventionally, tool_load leaves the output of any program it runs in a variable called ‘exec_output’. Writing tool_load can be the most complex part of extending DejaGnu to a new tool or a new target, if it requires much communication coding or file downloading.

Test scripts call tool_load.

tool_exit

Cleans up (if necessary) before runtest exits. For interactive tools, this usually ends the interactive session. You can also use tool_exit to remove any temporary files left over from the tests.

runtest calls tool_exit.

tool_version

Prints the version label and number for tool. This is called by the DejaGnu procedure that prints the final summary report. The output should consist of the full path name used for the tested tool, and its version number.

runtest calls tool_version.

The usual convention for return codes from any of these procedures (although it is not required by runtest) is to return 0 if the procedure succeeded, 1 if it failed, and -1 if there was a communication error.

4.4 The files DejaGnu reads

There are a few files used by DejaGnu for input. The most important one is ‘site.exp’, which contains the configuration data. ‘runtest.exp’ implements the main test framework; the runtest program used to invoke DejaGnu is a short shell script generated by make during the configuration process.

The only other general-purpose input files are collections of expect utilities kept in the ‘lib’ subdirectory. These files are not used by all tests, but are used (via the Tcl command source) often enough to warrant a common definition.

‘lib/remote.exp’ defines these functions, for establishing and managing communications:

Procedures to establish a connection: Each of these procedures tries to establish the connection up to three times before returning. Warnings (if retries will continue) or errors (if the attempt is abandoned) report on communication failures. The result for any of these procedures is either -1, when the connection cannot be established, or the spawn ID returned by the expect command spawn.

telnet hostname

rlogin hostname

rsh hostname

IP network procedures. hostname refers to the IP address or name (for example, an entry in ‘/etc/hosts’) for this target. The procedure names reflect the Unix utility used to establish a connection.

tip port

Serial line procedure. Connect using the Unix utility tip. port must be a name from the tip configuration file ‘/etc/remote’. Often, this is called ‘hardwire’, or something like ‘hardwire-a’. This file holds all the configuration data for the serial port.

kermit port bps

Serial line procedure. Connect using the program kermit. port is the device name, e.g. ‘/dev/ttyb’. bps is the line speed to use (in bits per second) for the connection.

Procedures to manage a connection:

tip_download spawnid file

Download ‘file’ to the process spawnid (the value returned when the connection was established), using the ~put command under tip. Most often used for single board computers that require downloading programs in ASCII S-records. Returns 1 if an error occurs, 0 otherwise.

exit_remote_shell spawnid

Exits a remote process started by any of the connection procedures. spawnid is the result of the connection procedure that started the remote process.

‘lib/utils.exp’ defines these utility procedures:

getdirs path

getdirs path pattern

Returns a list of all the directories in path that match pattern. If you do not specify pattern, getdirs assumes ‘*’. You may use the common shell wildcard characters in pattern.

find path pattern

Search for files whose names match pattern (using shell wildcard characters for filename expansion). Search subdirectories recursively, starting at path. The result is the list of files whose names match; if no files match, the result is empty. Filenames in the result include all intervening subdirectory names.

which binary

Searches the execution path for an executable file binary, like the the BSD which utility. This procedure uses the shell environment variable ‘PATH’. It returns 0 if the binary is not in the path, or if there is no ‘PATH’ environment variable. If binary is in the path, it returns the full path to binary.

grep filename regexp

grep filename regexp line

Search the file called filename (a fully specified path) for lines that contain a match for regular expression regexp. The result is a list of all the lines that match. If no lines match, the result is an empty string. Specify regexp using the standard regular expression style used by the Unix utility program grep.

Use the optional third argument ‘line’ to start lines in the result with the line number in filename.

prune list pattern

Remove elements of the Tcl list list. Elements are fields delimited by spaces. The result is a copy of list, without any elements that match pattern. You can use the common shell wildcard characters to specify pattern.

Most tools also have a library file associated with the tool, containing procedures found useful by other test developers. These are especially helpful when the tests span several subdirectories.

4.5 The files DejaGnu writes

runtest always writes two kinds of output files: summary logs and detailed logs. The contents of both of these are determined by your tests.

For troubleshooting, a third kind of output file is useful: use ‘--debug’ to request an output file showing details of what expect is doing internally.

4.5.1 Summary log

runtest always produces a summary output file ‘tool.sum’. This summary shows the names of all test files run; for each test file, one line of output from each pass command (showing status ‘PASS’ or ‘XPASS’) or fail command (status ‘FAIL’ or ‘XFAIL’) (or from error or warning commands); trailing summary statistics that count passing and failing tests (expected and unexpected); and the full pathname and version number of the tool tested. (All four possible outcomes, and all errors, are always reflected in the summary output file, regardless of whether or not you specify ‘--all’.)

For example, after ‘runtest --tool binutils’, look for a summary log in ‘binutils.sum’. Normally, runtest writes this file in your current working directory; use the ‘--output’ option to select a different directory.

Here is a short sample summary log:

Test Run By rob on Mon May 25 21:40:57 PDT 1992
                === gdb tests ===
Running ./gdb.t00/echo.exp ...
PASS:   Echo test
Running ./gdb.all/help.exp ...
PASS:   help add-symbol-file
PASS:   help aliases
PASS:   help breakpoint "bre" abbreviation
FAIL:   help run "r" abbreviation
Running ./gdb.t10/crossload.exp ...
PASS:   m68k-elf (elf-big) explicit format; loaded
XFAIL:  mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed
C types 
                === gdb Summary ===
# of expected passes 5
# of expected failures 1
# of unexpected successes 0
# of unexpected failures 1
/usr/latest/bin/gdb version 4.6.5 -q

4.5.2 Detailed log

runtest also saves a detailed log file ‘tool.log’, showing any output generated by tests as well as the summary output. For example, after ‘runtest --tool binutils’, look for a detailed log in ‘binutils.log’. Normally, runtest writes this file in your current working directory; use the ‘--output’ option to select a different directory.

Here is a brief example showing a detailed log for G++ tests:

Test Run By rob on Mon May 25 21:40:43 PDT 1992

                === g++ tests ===

--- Running ./g++.other/t01-1.exp ---
        PASS:   operate delete

--- Running ./g++.other/t01-2.exp ---
        FAIL:   i960 bug EOF
p0000646.C: In function `int  warn_return_1 ()':
p0000646.C:109: warning: control reaches end of non-void function
p0000646.C: In function `int  warn_return_arg (int)':
p0000646.C:117: warning: control reaches end of non-void function
p0000646.C: In function `int  warn_return_sum (int, int)':
p0000646.C:125: warning: control reaches end of non-void function
p0000646.C: In function `struct foo warn_return_foo ()':
p0000646.C:132: warning: control reaches end of non-void function

--- Running ./g++.other/t01-4.exp ---
        FAIL:   abort
900403_04.C:8: zero width for bit-field `foo'
--- Running ./g++.other/t01-3.exp ---
        FAIL:   segment violation
900519_12.C:9: parse error before `;'
900519_12.C:12: Segmentation violation
/usr/latest/bin/gcc: Internal compiler error: program cc1plus got
fatal signal

                === g++ Summary ===

# of expected passes 1
# of expected failures 3
# of unexpected successes 0
# of unexpected failures 0
/usr/ps/bin/g++ version cygnus-2.0.1

4.5.3 Logging expect internal actions

With the ‘--debug’ option, you can request a log file showing the output from expect itself, running in debugging mode. This file (‘dbg.log’, in the directory where you start runtest) shows each pattern expect considers in analyzing test output.

This file reflects each send command, showing the string sent as input to the tool under test; and each expect command, showing each pattern it compares with the tool output.

The log messages for expect begin with a message of the form

expect: does {tool output} (spawn_id n) match pattern
{expected pattern}?

For every unsuccessful match, expect issues a ‘no’ after this message; if other patterns are specified for the same expect command, they are reflected also, but without the first part of the message (‘expect…match pattern’).

When expect finds a match, the log for the successful match ends with ‘yes’, followed by a record of the expect variables set to describe a successful match. Here is an excerpt from the debugging log for a GDB test:

send: sent {break gdbme.c:34\n} to spawn id 6
expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file
 gdbme.c, line 34.*\(gdb\) $}? no
{.*\(gdb\) $}? no
expect: does {} (spawn_id 0) match pattern {<return>}? no
{\(y or n\) }? no
{buffer_full}? no
{virtual}? no
{memory}? no
{exhausted}? no
{Undefined}? no
{command}? no
break gdbme.c:34
Breakpoint 8 at 0x23d8: file gdbme.c, line 34.
(gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: file
gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern
{Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes
expect: set expect_out(0,start) {18}
expect: set expect_out(0,end) {71}
expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file
gdbme.c, line 34.\r\n(gdb) }
expect: set expect_out(spawn_id) {6}
expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8 at
0x23d8: file gdbme.c, line 34.\r\n(gdb) }
        PASS:   70      0       breakpoint line number in file

This example exhibits three properties of expect and DejaGnu that might be surprising at first glance:

• Empty output for the first attempted match. The first set of attempted matches shown ran against the output ‘{}’—that is, no output. expect begins attempting to match the patterns supplied immediately; often, the first pass is against incomplete output (or completely before all output, as in this case).
• Interspersed tool output. The beginning of the log entry for the second attempted match may be hard to spot: this is because the prompt ‘(gdb) ’ appears on the same line, just before the ‘expect:’ that marks the beginning of the log entry.
• Default patterns. Most of the patterns shown are fail-safe patterns provided by DejaGnu itself.

runtest first tries the patterns specified explicitly in each test case; however, if none of these patterns matches, runtest tries a default list of patterns designed to try to catch common errors or oversights in writing test scripts.

The default patterns (like the debugging log itself) are primarily useful while developing test scripts. The actions for these default patterns produce messages starting with ‘ERROR’ on the runtest standard output, and in the detailed log file. Here is a list of these default patterns:

‘{<return>}’

Checks for an unexpected <RET> prompt. This may happen, for example, if the tested tool can filter output through a pager.

‘{\(y or n\) }’

A common pattern in prompts requiring user input. This will trap some unexpected prompts from a program.

‘{buffer_full}’

Detects overflow of the internal expect buffer for output from the tested tool. The default buffer size is 2000 characters. You can specify a larger buffer, if some of your tests need one, with the match_max Tcl function provided by expect.

‘{virtual} {memory} {exhausted}’

The message ‘virtual memory exhausted’ from GDB (and perhaps other tested tools) may indicate a problem with the test script itself. Sometimes a pattern with poor closure (see section Hints on Writing a Test Case) fills up available memory, leaving very little for the process that was spawned.

‘{Undefined} {command}’

The message ‘Undefined command’ from GDB (and perhaps other tested tools) may be a response to a test script that sends a nonexistent command. Sometimes commands disappear as updates are released; this may also arise from a simple typing mistake in a test script.

5 How To Write a Test Case

5.1 Writing a test case

The easiest way to prepare a new test case is to base it on an existing one for a similar situation. There are two major categories of tests: batch or interactive. Batch oriented tests are usually easier to write.

The GCC tests are a good example of batch oriented tests. All GCC tests consist primarily of a call to a single common procedure, since all the tests either have no output, or only have a few warning messages when successfully compiled. Any non-warning output is a test failure. All the C code needed is kept in the test directory. The test driver, written in expect, need only get a listing of all the C files in the directory, and compile them all using a generic procedure. This procedure and a few others supporting for these tests are kept in the library module ‘lib/c-torture.exp’. Most tests of this kind use very few expect features, and are coded almost purely in Tcl.

Writing the complete suite of C tests, then, consisted of these steps:

1. Copying all the C code into the test directory. These tests were based on the C-torture test created by Torbjorn Granlund (on behalf of the Free Software Foundation) for GCC development.
2. Writing (and debugging) the shared expect library code.
3. Writing the simple test driver: its main task is to search the directory (using the Tcl procedure glob for filename expansion with wildcards) and call a Tcl procedure with each filename. It also checks for a few errors from the testing procedure.

Testing interactive programs is intrinsically more complex. Tests for most interactive programs require some trial and error before they are complete.

However, some interactive programs can be tested in a simple fashion reminiscent of batch tests. For example, the GDB distribution itself includes a testing procedure that covers most of the DejaGnu GDB tests. This procedure is very robust, and has already undergone much more debugging and error checking than many other DejaGnu test cases. Accordingly, the best approach was simply to encapsulate the existing GDB tests, for reporting purposes, in DejaGnu tests that use common library code.

‘gdb.t10/crossload.exp’ is a good example of an interactive test.

5.2 Debugging a test case

These are the kinds of debugging information available from DejaGnu:

1. Output controlled by test scripts themselves, explicitly allowed for by the test author. This kind of debugging output appears in the logs recorded in the detail directory. To do the same for new tests, use the ‘verbose’ variable to control how much output to generate; this will make it easier for other people running the test to debug it if necessary. Whenever possible, if ‘$verbose’ is 0, there should be no output other than the output from pass, fail, error, and warning. Then, to whatever extent is appropriate for the particular test, allow successively higher values of ‘verbose’ to generate more information. Be kind to other programmers who use your tests: provide for a lot of debugging information.
2. Output from the internal debugging functions of Tcl and expect. There is a command line options for each; both forms of debugging output are recorded in the file dbg.log in the current directory.

   Use ‘--debug’ for information from the expect level; it generates displays of the expect attempts to match the tool output with the patterns specified (see section Debug Log). This output can be very helpful while developing test scripts, since it shows precisely the characters received. Iterating between the latest attempt at a new test script and the corresponding ‘dbg.log’ can allow you to create the final patterns by “cut and paste”. This is sometimes the best way to write a test case.

   Use ‘--strace’ to see more detail at the Tcl level; this shows how Tcl procedure definitions expand, as they execute. The associated number controls the depth of definitions expanded; see the discussion of ‘--strace’ in Running the Tests.

3. Finally, if the value of ‘verbose’ is 3 or greater, runtest turns on the expect command log_user. This command prints all expect actions to the expect standard output, to the detailed log file, and (if ‘--debug’ is on) to ‘dbg.log’.

5.3 Adding a test case to DejaGnu

There are two slightly different ways to add a test case. One is to add the test case to an existing directory. The other is to create a new directory to hold your test. The existing test directories represent several styles of testing, all of which are slightly different; examine the directories for the tool of interest to see which (if any) is most suitable.

Adding a GCC test can be very simple: just add the C code to any directory beginning with ‘gcc.’ and it runs on the next ‘runtest --tool gcc’.

To add a test to GDB, first add any source code you will need to the test directory. Then you can either create a new expect file, or add your test to an existing one (any file with a ‘.exp’ suffix). Creating a new ‘.exp’ file is probably a better idea if the test is significantly different from existing tests. Adding it as a separate file also makes upgrading easier. If the C code has to be already compiled before the test will run, then you’ll have to add it to the ‘Makefile.in’ file for that test directory, then run configure and make.

Adding a test by creating a new directory is very similar:

1. Create the new directory. All subdirectory names begin with the name of the tool to test; e.g. G++ tests might be in a directory called ‘g++.other’. There can be multiple test directories that start with the same tool name (such as ‘g++’).
2. Add the new directory name to the ‘configdirs’ definition in the ‘configure.in’ file for the testsuite directory. This way when make and configure next run, they include the new directory.
3. Add the new test code to the directory.
4. To add support in the new directory for configure and make, you must also create a Makefile.in and a configure.in. See What Configure Does in configure.info.

5.4 Hints on writing a test case

There may be useful existing procedures already written for your test in the ‘lib’ directory of the DejaGnu distribution. See section The files DejaGnu reads.

It is safest to write patterns that match all the output generated by the tested program; this is called closure. If a pattern does not match the entire output, any output that remains will be examined by the next expect command. In this situation, the precise boundary that determines which expect command sees what is very sensitive to timing between the expect task and the task running the tested tool; as a result, the test may sometimes appear to work, but is likely to have unpredictable results. The best way to ensure closure is to use the ‘-re’ option for the expect command to write the pattern as a full regular expressions; then you can match the end of output using a ‘$’. It is also a good idea to write patterns that match all available output by using ‘.*\’ after the text of interest; this will also match any intervening blank lines. Sometimes an alternative is to match end of line using ‘\r’ or ‘\n’, but this is usually too dependent on terminal settings.

Always escape punctuation, such as ‘(’ or ‘"’, in your patterns; for example, write ‘\(’. If you forget to escape punctuation, you will usually see an error message like ‘extra characters after close-quote’.

If you have trouble understanding why a pattern does not match the program output, try using the ‘--debug’ option to runtest, and examine the debug log carefully. See section Debug Log.

Be careful not to neglect output generated by setup rather than by the interesting parts of a test case. For example, while testing GDB, I issue a send ‘set height 0\n’ command. The purpose is simply to make sure GDB never calls a paging program. The ‘set height’ command in GDB does not generate any output; but running any command makes GDB issue a new ‘(gdb) ’ prompt. If there were no expect command to match this prompt, the output ‘(gdb) ’ begins the text seen by the next expect command—which might make that pattern fail to match.

To preserve basic sanity, I also recommended that no test ever pass if there was any kind of problem in the test case. To take an extreme case, tests that pass even when the tool will not spawn are misleading. Ideally, a test in this sort of situation should not fail either. Instead, print an error message by calling one of the DejaGnu procedures error or warning.

5.5 Special variables used by test cases

Your test cases can use these variables, with conventional meanings (as well as the variables saved in ‘site.exp’ see section Site configuration):

These variables are available to all test cases.

prms_id

The tracking system (e.g. GNATS) number identifying a corresponding bugreport. (‘0’ if you do not specify it in the test script.)

bug_id

An optional bug id; may reflect a bug identification from another organization. (‘0’ if you do not specify it.)

subdir

The subdirectory for the current test case.

These variables should never be changed. They appear in most tests.

expect_out(buffer)

The output from the last command. This is an internal variable set by expect.

exec_output

This is the output from a tool_load command. This only applies to tools like GCC and GAS which produce an object file that must in turn be executed to complete a test.

comp_output

This is the output from a tool_start command. This is conventionally used for batch oriented programs, like GCC and GAS, that may produce interesting output (warnings, errors) without further interaction.

6 New Tests, Targets, or Hosts

The most common ways to extend the DejaGnu framework are: adding a suite of tests for a new tool to be tested; adding support for testing on a new target; and porting runtest to a new host.

6.1 Writing Tests for a New Tool

In general, the best way to learn how to write (code or even prose) is to read something similar. This principle applies to test cases and to test suites. Unfortunately, well-established test suites have a way of developing their own conventions: as test writers become more experienced with DejaGnu and with Tcl, they accumulate more utilities, and take advantage of more and more features of expect and Tcl in general.

Inspecting such established test suites may make the prospect of creating an entirely new test suite appear overwhelming. Nevertheless, it is quite straightforward to get a new test suite going.

There is one test suite that is guaranteed not to grow more elaborate over time: both it and the tool it tests were created expressly to illustrate what it takes to get started with DejaGnu. The ‘example/’ directory of the DejaGnu distribution contains both an interactive tool called calc, and a test suite for it. Reading this test suite, and experimenting with it, is a good way to supplement the information in this section. (Thanks to Robert Lupton for creating calc and its test suite—and also the first version of this section of the manual!)

To help orient you further in this task, here is an outline of the steps to begin building a test suite for a program example.

1. Create or select a directory to contain your new collection of tests. Change to that directory (shown here as testsuite):

   eg$ cd testsuite/
   

2. Create a ‘configure.in’ file in this directory, to control configuration-dependent choices for your tests. So far as DejaGnu is concerned, the important thing is to set a value for the variable target_abbrev; this value is the link to the init file you will write soon. (For simplicity, we assume the target environment is Unix, and use ‘unix’ as the value.)

   What else is needed in ‘configure.in’ depends on the requirements of your tool, your intended test environments, and which configure system you use. This example is a minimal configure.in for use with Cygnus Configure. (For an alternative based on the FSF autoconf system, see the calc example distributed with DejaGnu.) Replace example with the name of your program:

   # This file is a shell script fragment # for use with Cygnus configure. srctrigger="example.0" srcname="The DejaGnu example tests" # per-host: # per-target: # everything defaults to unix for a target target_abbrev=unix # post-target:

3. Create ‘Makefile.in’, the source file used by configure to build your ‘Makefile’. Its leading section should as usual contain the values that configure may override:

   srcdir = . prefix = /usr/local exec_prefix = $(prefix) bindir = $(exec_prefix)/bin libdir = $(exec_prefix)/lib tooldir = $(libdir)/$(target_alias) datadir = $(exec_prefix)/lib/deja-gnu RUNTEST = runtest RUNTESTFLAGS = FLAGS_TO_PASS = #### host, target, and site specific Makefile frags come in here.

   This should be followed by the standard targets at your site. To begin with, they need not do anything—for example, these definitions will do:

   all: info: install-info: install: uninstall: clean: -rm -f *~ core *.info*

   It is also a good idea to make sure your ‘Makefile’ can rebuild itself if ‘Makefile.in’ changes, with a target like this (which works for either Cygnus or FSF Configure):

   Makefile : $(srcdir)/Makefile.in $(host_makefile_frag) \ $(target_makefile_frag) $(SHELL) ./config.status

   You also need to include two targets important to DejaGnu: check, to run the tests, and site.exp, to set up the Tcl copies of configuration-dependent values. The check target must run ‘runtest --tool example’:

   check: site.exp all $(RUNTEST) $(RUNTESTFLAGS) $(FLAGS_TO_PASS) \ --tool example --srcdir $(srcdir) \ --target $(target_canonical)

   The site.exp target should usually set up (among other things!) a Tcl variable for the name of your program:

   site.exp: ./config.status Makefile @echo "Making a new config file..." -@rm -f ./tmp? @touch site.exp -@mv site.exp site.bak @echo "## these variables are automatically\ generated by make ##" > ./tmp0 @echo "# Do not edit here. If you wish to\ override these values" >> ./tmp0 @echo "# add them to the last section" >> ./tmp0 @echo "set host_os ${host_os}" >> ./tmp0 @echo "set host_alias ${host_alias}" >> ./tmp0 @echo "set host_cpu ${host_cpu}" >> ./tmp0 @echo "set host_vendor ${host_vendor}" >> ./tmp0 @echo "set target_os ${target_os}" >> ./tmp0 @echo "set target_alias ${target_alias}" >> ./tmp0 @echo "set target_cpu ${target_cpu}" >> ./tmp0 @echo "set target_vendor ${target_vendor}" >> ./tmp0 @echo "set host_triplet ${host_canonical}" >> ./tmp0 @echo "set target_triplet ${target_canonical}" >> ./tmp0 @echo "set tool binutils" >> ./tmp0 @echo "set srcdir ${srcdir}" >> ./tmp0 @echo "set objdir `pwd`" >> ./tmp0 @echo "set examplename example" >> ./tmp0 @echo "## All variables above are generated by\ configure. Do Not Edit ##" >> ./tmp0 @cat ./tmp0 > site.exp @cat site.bak | sed \ -e '1,/^## All variables above are.*##/ d' \ >> site.exp -@rm -f ./tmp?

4. Create a directory (in ‘testsuite/’) called ‘config/’:

   eg$ mkdir config
   

5. Make an init file in this directory; its name must start with the target_abbrev value, so call it ‘config/unix.exp’. This is the file that contains the target-dependent procedures; fortunately, most of them do not have to do very much in order for runtest to run.

   If example is not interactive, you can get away with this minimal ‘unix.exp’ to begin with:

   proc foo_exit {} {} proc foo_version {} {}

   If example is interactive, however, you might as well define a start routine and invoke it by using an init file like this:

   proc foo_exit {} {} proc foo_version {} {} proc foo_start {} { global examplename spawn $examplename expect { -re "" {} } } foo_start

6. Create a directory whose name begins with your tool’s name, to contain tests:

   eg$ mkdir example.0
   

7. Create a sample test file in ‘example.0’. Its name must end with ‘.exp’; you can use ‘first-try.exp’ To begin with, just write there a line of Tcl code to issue a message:

   send_user "Testing: one, two...\n"

8. Back in the ‘testsuite/’ (top level) directory, run

   eg$ configure
   

   (You may have to specify more of a path, if a suitable configure is not available in your execution path.)

9. You are now ready to triumphantly type ‘make check’ or ‘runtest --tool example’. You should see something like this:

   Test Run By rhl on Fri Jan 29 16:25:44 EST 1993 === example tests === Running ./example.0/first-try.exp ... Testing: one, two... === example Summary === # of expected passes 0 # of expected failures 0 # of unexpected successes 0 # of unexpected failures 0

10. Begin writing some real tests. For an interactive tool, you should probably write a real exit routine in fairly short order; in any case, you should also write a real version routine soon.

6.2 Adding a target

Supporting a new target requires one thing: an initialization module for each tool you wish to test there. This module contains all the operating system dependent code. See section Initialization Module, for a discussion of naming conventions that will enable DejaGnu to use your new init files.

Most of the work is in getting the communications right. Communications code (for several situations involving IP networks or serial lines) is available in a DejaGnu library file, ‘lib/remote.exp’. See section The files DejaGnu reads.

Usually the best way to write a new initialization module is to edit an existing initialization module; some trial and error will be required. If necessary, remember you can use the ‘--debug’ option to see what is really going on.

When you code an initialization module, be generous in using print statements controlled by the level in ‘$verbose’.

6.3 Porting to a new host

The task of porting DejaGnu is basically that of porting Tcl and expect. Tcl and expect, as distributed with DejaGnu, both use autoconf; they should port automatically to most Unix systems. Currently Tcl and expect have run on these environments: Cray running Unicos; HP700 (PA-RISC) running HP-UX; HP300 running HP-UX; an RS600 running AIX; SGI running IRIX; Sun3 and Sun4 running SunOS; Sun4 running Solaris 2.x; DECstation running Ultrix; 386BSD; and Linux.

Once Tcl and expect are ported, DejaGnu should run. Most system dependencies are taken care of by using expect as the main command shell.

Index

Footnotes

(1)

‘--tool’ selects a particular suite of tests, not the name of the executable program to test. See section Site configuration, for information on the variables that you can use to specify the names of programs to run.

(2)

This is also one of the first DejaGnu tests created for GDB. More recent tests use the ‘gdb_test’ procedure, which encapsulates the common pattern of sending a command to GDB, looking for a particular return text, and testing for common failure conditions. Using that procedure, you can write a test equivalent to ‘echo.exp’ simply as ‘ gdb_test "echo Hello world!" "Hello world!" ’

(3)

mondfe is an AMD specific monitor program freely available from AMD.

(4)

mondfe is specific to AMD targets using the UDI protocol; it will not work for other targets.

(5)

Configuration triples have the form ‘cpu-vendor-os’.

Table of Contents

About This Document

This document was generated on January 16, 2023 using texi2html 5.0.

The buttons in the navigation panels have the following meaning:

where the Example assumes that the current position is at Subsubsection One-Two-Three of a document of the following structure:

• 1. Section One
  • 1.1 Subsection One-One
    • ...
  • 1.2 Subsection One-Two
    • 1.2.1 Subsubsection One-Two-One
    • 1.2.2 Subsubsection One-Two-Two
    • 1.2.3 Subsubsection One-Two-Three     <== Current Position
    • 1.2.4 Subsubsection One-Two-Four
  • 1.3 Subsection One-Three
    • ...
  • 1.4 Subsection One-Four

This document was generated on January 16, 2023 using texi2html 5.0.