To avoid confusion, here is a quick list of definitions:

This is the first thing determined by TextTest on being called, and is where in your file system it will start to look for tests. All test files are placed in subdirectories of this directory. It is determined as follows:

For normal usage, you should set TEXTTEST_HOME to an appropriate value. It is always set internally by TextTest, however its value is initially determined, so its value may be used in the configuration files that are described below.

To test a system with TextTest, the first thing to do is to choose a unique identifier to be used as an extension for all files relevant to that application. It does not matter what this is. For the next few paragraphs this variable is indicated by <app>. On Windows, you will probably want to associate this extension with a text editor like notepad or wordpad. The use of file extensions for this purpose is historical: TextTest grew up on UNIX where file extensions do not have the meaning that they do on Windows.

Basic information about an application and how to run it appears in a file called config.<app>, generally referred to as the “config file”. TextTest will look for files with names of this format to determine which applications it will run. It will start looking at the root directory and look in that directory and one level down in the directory structure: this is so that tests for related applications can easily be grouped together in subdirectories of the root directory. When TextTest is started, it will by default look for and use all config files it can find. To tell it to look for just one particular application, specify "-a <app>" on the command line.

This file basically consists of key, value pairs, where the keys are “properties” with names predefined by TextTest. The most important of these is the entry "binary", which defines the path to the SUT and without which nothing much will happen. This should be an absolute path, although environment variables may be included. It can be any executable program, not just a binary. It can also be the name of a Java class that will be found on the Java class path, provided you set “interpreter” to “java” (below).

The entry “interpreter” allows you to specify a program to use as interpreter for the SUT, in the case that it is a script rather than a binary. To some extent TextTest will try to infer this from the file extension (e.g. set it to “python” if the file ends in “.py”, “java” if it ends in “.jar”), but it is sometimes necessary to specify it explicitly.

The file itself has a specific format, called Standard Dictionary Format. It is worth familiarising yourself with this, and often the best way is to look at the many examples in TextTest's tests for itself (that come with the download).

It is also possible to have a personalised config file which accepts all the same settings as the normal config file, and will override anything provided there. This is particularly useful for setting things like GUI preferences. On UNIX, provide a file called “.texttest” in your home directory. On Windows, put a file called “.texttest” somewhere, and point the environment variable TEXTTEST_PERSONAL_CONFIG at that location.

Sometimes it can be very useful to share configuration settings between several related applications. In that case you can use the “import_config_file” entry to identify files of the same format whose settings should be included. The file should be found under TEXTTEST_HOME, in the same way as described in the above paragraph for the config files themselves. By setting the entry “extra_config_directory” it can also be placed in the arbitrary location indicated. Such a file doesn't need to have a particular name.

A Test Suite is a recursive collection of test cases arranged in a particular order. It is defined for a Test Application by a directory in the file system containing a local file called testsuite.<app>. This lists subdirectories in the order in which they should be considered. These subdirectories may correspond to test cases or may themselves be test suites. The file is in Standard List Format. Having found a test application by finding config.<app>, will then look at testsuite.<app> in the same directory to determine what the full test suite consists of. It will then look, in the order given, at all the subdirectories specified, and where they are themselves test suites, will repeat this process recursively until all specified test cases have been found. Each test suite directory, apart from the top level one, will have the same name as the test suite itself.

Note that testsuite.<app> files are generated and edited automatically when using the static GUI to create test cases or test suites. However, it is also useful to view and edit them by hand: the static GUI will automatically refresh if the elements in the test suite file are re-ordered, and the contents of the Description fields are added as comments to this file.

If you find that managing an explicit order of tests is too much effort, you can set the “auto_sort_test_suites” config file setting to “1”. The order in the testsuite files will then be ignored and all test suites and test cases presented in alphabetical order.

A test case is represented in TextTest by a particular directory in the file system, and the name of the test case is always the same as the name of the directory. Many test applications may share the same test case if desired. To define a test case for a test application, at least one test definition file must be present. Definition files tell TextTest how to run the SUT for this test case. The following files will be taken as test definition files:

The expected output files from the SUT are also stored in this directory: these will be compared with the actual result for each test run. By default, the standard output of the system under test is redirected to output.<app>, while its standard error is redirected to errors.<app>. Other textual output files can also be collected, and the collection of these can be disabled: look here for details of how.

TextTest assumes that high-level information of interest to it will be logged to one particular result file. This file is indicated by the “log_file” config file entry and defaults to “output” (i.e. the standard output of the SUT). Many different features of TextTest will look here for information to extract of one sort or another.

All of these files are in whatever format is expected or produced by the SUT: TextTest does not itself look at their contents.

Any test suite or test case can tell TextTest to set environment variables by providing an environment file. This is a file called environment.<app> or just environment (it has been found that applications often need to share environment variables). This file is in Standard Dictionary Format , with the environment variable names as keys and their values as entries. If the environment file is provided in a test case these variables will be set just for that test case. If in a test suite it will be set for all test suites and test cases containing in that test suite, operating recursively.

When exiting the test suite, attempts will be made to unset the environment variables, however be aware that not all versions of Python/operating systems support this. Therefore you may need to set dummy values in other test suites to prevent unintended effects.

The values of the variables may themselves contain environment variables: if so, this should be done UNIX-style using $<var_name>.

When a large test suite has been created, you often want to gather information from it, or even update its contents in a predictable way. It is very useful to be able to re-use TextTest's ability to parse and understand the structure while writing your own script in Python to analyse or update in terms of the applications, test suites and test cases.

There is thus a mechanism to plug in arbitrary scripts, which are run with “-s <module_name>.<class_name>”, where <module_name> is some Python module and <class_name> is the name of the script. In order to pass arguments to such scripts, a form with <option>=<value> is used. For example, to call the default.ReplaceText script with appropriate arguments, you would call:

texttest.py -s “default.ReplaceText file=errors old=bad new=good”

TextTest includes several such scripts which have been found to be generally useful, which are listed here. To see how to write your own such scripts, consult the guide to writing your own configuration.

The above example (default.ReplaceText) is a particularly useful way to update lots of results in a predictable way. It is basically a search-and-replace mechanism with the advantage that you can select tests in the normal ways and the files relevant to the testsuite will be chosen for you. The above example will naturally replace all instances of “bad” with “good” in all “errors” result files.

TextTest reads two file formats - Standard List Format (for testsuite files) and Standard Dictionary Format (for config and environment files). These are designed to be as human readable as possible.

Both will filter out blank lines and lines beginning with "#", the latter being interpreted as comments. It's good practice to use the latter feature to document things about your application and its tests in the TextTest files themselves.

Both will also expand environment variables, indicated by "$".

Standard List Format is simple: each entry is a complete line. So a Standard List File is simply interpreted as an ordered list of the lines in it which are not blank and do not start with "#".

Standard Dictionary Format has entries in the form <key>:<value>, where <key> indicates an environment variable to be set to <value> in the case of the environment file, and a variable understood by TextTest in the case of the config file.

In the case of the config file, it can be useful to have the value be a list itself. This is achieved by adding several entries for the same <key>. So if we want to set the key “my_key” to a list containing A and B,, this is done by
my_key:A
my_key:B

Providing the entry
my_key:{CLEAR LIST}

allows overriding files to remove entries added in a parent version file.

It can also be useful to have the value be a dictionary. This is achieved by the "section header" format, i.e.
[my_dictionary]
first_key:first_value
second_key:second_value

Entries within section headers can also use the list format described above.

In a few cases the value is essentially a dictionary, but with two keys (a “composite dictionary entry”). This is used for the batch mode and performance-related settings. The format does not look very different to the above, but because anything can appear in the dictionary rather than predefined key names, it is necessary to have a mechanism to share values. This is achieved by the special predefined key name “default”.

[my_composite_dictionary]
first_key:first_value
default:default_value

Here a lookup of first_key in my_composite_dictionary will produce “first_value”. A lookup of “second_key” (or anything else) will produce “default_value”.

If you only wish to override the default value of such a dictionary, it is acceptable to use the format for non-dictionary entries, i.e. simply

my_composite_dictionary:default_value

The TextTest download includes a test suite for itself. It is recommended that you look around this (or any other example you can find) to get an idea of how it works. There is also a quick guide document included to help you find the simple “target application” test suites which are used for testing itself. These are simple in order to provide minimal tests, so function as demonstration examples also.

When writing your own tests it is often best to start with working files for another application and edit them suitably. This reduces the risk of typing things wrongly, particularly in the config file.