Documentation for 3.9.1
Understanding TextTest Test Suites
A Guide to the Files and
To avoid confusion, here is a quick list of definitions:
- System Under Test (SUT)
– The program that you wish to test. Assumed to be
available as an executable file.
- Test Application (or just
Application) – A set of TextTest files corresponding
to testing a particular SUT.
- Test Case –
defines a particular way to run the SUT, and the textual output
to be expected.
- Test Suite – a group of Test Cases or
(recursively) Test Suites which are in some way related.
This is the first thing determined by TextTest on being
called, and is where in your file system it will start to look
for tests. All test files are placed in subdirectories of this
directory. It is determined as follows:
- If the command line option
"-d" has been set, use the value of that.
- If not, but the environment
variable "TEXTTEST_HOME" is set, use that.
- If neither of the above, use the current working
normal usage, you should set TEXTTEST_HOME to an appropriate
value. It is always set internally by TextTest, however its
value is initially determined, so its value may be used in the
configuration files that are described below.
To test a system with TextTest, the first thing to do is to
choose a unique identifier to be used as an extension for all
files relevant to that application. It does not matter what this
is. For the next few paragraphs this variable is indicated by
<app>. On Windows, you will probably want to associate
this extension with a text editor like notepad or wordpad. The
use of file extensions for this purpose is historical: TextTest
grew up on UNIX where file extensions do not have the meaning
that they do on Windows.
Basic information about an application and how to run it
appears in a file called config.<app>,
generally referred to as the “config file”. TextTest
will look for files with names of this format to determine which
applications it will run. It will start looking at the root
directory and look in that directory and one level down in the
directory structure: this is so that tests for related
applications can easily be grouped together in subdirectories of
the root directory. When TextTest is started, it will by default
look for and use all config files it can find. To tell it to
look for just one particular application, specify "-a
<app>" on the command line.
This file basically consists of key, value pairs, where the
keys are “properties” with names predefined by
TextTest. The most important of these is the entry "binary",
which defines the path to the SUT and without which nothing much
will happen. This should be an absolute path, although
environment variables may be included. It can be any executable
program, not just a binary. It can also be the name of a Java
class that will be found on the Java class path, provided you
set “interpreter” to “java” (below).
The entry “interpreter” allows you to specify a
program to use as interpreter for the SUT, in the case that it
is a script rather than a binary. To some extent TextTest will
try to infer this from the file extension (e.g. set it to
“python” if the file ends in “.py”,
“java” if it ends in “.jar”), but it is
sometimes necessary to specify it explicitly.
The file itself has a specific format, called Standard
. It is worth familiarising yourself with
this, and often the best way is to look at the many examples in
TextTest's tests for itself (that come with the download).
It is also possible to have a personalised config file which
accepts all the same settings as the normal config file, and
will override anything provided there. This is particularly
useful for setting things like GUI
. On UNIX, provide a file called “.texttest”
in your home directory. On Windows, put a file called
“.texttest” somewhere, and point the environment
variable TEXTTEST_PERSONAL_CONFIG at that location.
Sometimes it can be very useful to share configuration
settings between several related applications. In that case you
can use the “import_config_file” entry to identify
files of the same format whose settings should be included. The
file should be found under TEXTTEST_HOME, in the same way as
described in the above paragraph for the config files
themselves. Such a file doesn't need to have a particular name.
A Test Suite is a recursive collection of test cases arranged
in a particular order. It is defined for a Test Application by a
directory in the file system containing a local file called
. This lists subdirectories
in the order in which they should be considered. These
subdirectories may correspond to test cases or may themselves be
test suites. The file is in Standard
. Having found a test application by finding
, will then look at
in the same directory to
determine what the full test suite consists of. It will then
look, in the order given, at all the subdirectories specified,
and where they are themselves test suites, will repeat this
process recursively until all specified test cases have been
found. Each test suite directory, apart from the top level one,
will have the same name as the test suite itself.
Note that testsuite.<app> files are
generated and edited automatically when using the static GUI to
create test cases or test suites. However, it is also useful to
view and edit them by hand: the static GUI will automatically
refresh if the elements in the test suite file are re-ordered,
and the contents of the Description fields are added as comments
to this file.
If you find that managing an explicit order of tests is too
much effort, you can set the “auto_sort_test_suites”
config file setting to “1”. The order in the
testsuite files will then be ignored and all test suites and
test cases presented in alphabetical order.
A test case is represented in TextTest by a particular
directory in the file system, and the name of the test case is
always the same as the name of the directory. Many test
applications may share the same test case if desired. To define
a test case for a test application, at least one test definition
file must be present. Definition files tell TextTest how to run
the SUT for this test case. The following files will be taken as
test definition files:
- This will be interpreted as command line options to be given
to the system under test.
- This will be redirected to the system under test as standard
- The use case recorder will be configured to replay the system
under test from this file.
The expected output files from the SUT are also stored in
this directory: these will be compared with the actual result
for each test run. By default, the standard output of the system
under test is redirected to output.<app>
while its standard error is redirected to errors.<app>
Other textual output files can also be collected, and the
collection of these can be disabled: see the guide
to configuring the evaluation of test results
TextTest assumes that high-level information of interest to
it will be logged to one particular result file. This file is
indicated by the “log_file” config file entry and
defaults to “output” (i.e. the standard output of
the SUT). Many different features of TextTest will look here for
information to extract of one sort or another.
All of these files are in whatever format is expected or
produced by the SUT: TextTest does not itself look at their
Any test suite or test case can tell TextTest to set
environment variables by providing an environment file. This is
a file called environment.<app>
(it has been found that applications
often need to share environment variables). This file is in
, with the environment variable names as keys and
their values as entries. If the environment file is provided in
a test case these variables will be set just for that test case.
If in a test suite it will be set for all test suites and test
cases containing in that test suite, operating recursively.
When exiting the test suite, attempts will be made to unset
the environment variables, however be aware that not all
versions of Python/operating systems support this. Therefore you
may need to set dummy values in other test suites to prevent
The values of the variables may themselves contain
environment variables: if so, this should be done UNIX-style
It is often needed to define different versions of an
application which may not be quite the same in all respects, but
which want to share some portion of the test suite structure of
the parent application. To specify a version to run, fill in the
“run this version” field under Running/What to Run
tabs on the static GUI, or use -v <version> on the command
line. Note that there is a similar field in the Recording tab in
case you are testing GUIs.
For each file type described so far, the framework will then
look also for files called <root>.<app>.<version>
where <root> is “environment”, “config”,
“testsuite” etc. in each case. If such a file does
not exist, <root>.<app> is then always used. If the
file does exist, it is used instead of <root>.<app>
in the case of files in test cases (which are interpreted by the
SUT) and Standard List Format files (i.e. the testsuite.<app>
files). In the case of Standard Dictionary Format (i.e. the
config and environment files), however, they are interpreted as
overriding particular entries in the dictionary, so that entries
not present in <root>.<app>.<version> are read
It is possible to save test results with a version
identifier, so that they will be used for comparison next time
that version is run.
Several versions can be "aggregated" and used at
the same time. This is done by specifying -v
<version1>.<version2> on the command line, and can
be applied to any number of versions.
Note that such aggregation is not order-specific. Running
with “-v a.b” will be exactly equivalent to running
“-v b.a” : no preference is indicated by the
ordering. This can be a problem when there are settings for
version a and version b which are in conflict – it is
essentially not defined which will be preferred to which.
To clearly define which versions should be preferred to which
other versions, use the “version_priority” config
file setting. This takes the form of a dictionary, where the
keys are version names and the values are numbers, where a low
number implies that the settings for that version should be used
in preference to another version. The default priority is 99 for
You can also say that a version "inherits" settings
from another by adding the "base_version" entry to the
config file. Thus if config.<app>.v1 contains the line
"base_version:v2" then all the files for v1 are also
read as well as those for v2.
Note that the syntax -v <version1>,<version2> is
similar to -a <app1>,<app2>: i.e. it will first run
everything with version1 and then run everything with version2.
It can be useful to start another version all the time when
running tests, so that -v v1 behaves like - v v1,v2 at all times
(or no option behaves like -v ,v2). This is achieved by adding
the line "extra_version:v2" to the relevant config
Such versioned results are easy to create but tend to be hard
to remove, you can end up with a lot of identical files with
different version IDs. To help solve this, the plugin
identify such redundancy and remove the versioned files. It will
also warn where versions are equivalent but not redundant. For
example, if the files output.myapp and output.myapp.2 are
identical, then output.myapp.2 will be removed. If
output.myapp.2 and output.myapp.3 are the same, then only a
warning is printed.
All of the files and directories discussed here can amount to
a substantial structure once you have a few tests. These will
clearly change over time along with the code that they test.
When you have multiple developers it is hence nearly always a
good idea to version-control the test suite files so that
developers making changes to the test suite do not disturb each
other. Using the history provided by version-control software
can also be very useful to track the behaviour of your
application over time.
TextTest does not integrate directly with any version control
software right now, but it does have a concept of a “checkout”
which aids in using it. In a version-controlled environment, you
want to be able to specify the path to the SUT as a relative
path, so that different developers can test their own code in
their own user space, and they can also painlessly run each
others code or maintain several checkouts of the system..
It is expected that TextTest test suites will want to be
version-controlled, and hence an easy means of switching between
different "checkouts" of the version-control system is
needed. A checkout is different to a version in that all
checkouts are expected to produce the same results, and making
sure that last night's central checkout does the same as a
developer's local code is an essential part of verifying
TextTest will export the environment variable
TEXTTEST_CHECKOUT. Any setting in the config or
environment files can be made to depend on this variable:
you can insert it as you would with other environment
variables.. A very common usage is the compulsory “binary”
setting in the config file.
How does TextTest detemine its value? A default value is
provided via the config file, but it can also be configured per
run. Checkouts are identified by short checkout names, and/or by
the full path which they correspond to.
There are two entries in the config file, "checkout_location"
and "default_checkout". “default_checkout”
identifies what the short name of the checkout is by default.
“checkout_location” is a dictionary entry mapping
these short names to one or more full paths, so several
different variants can be provided depending on how different
users have named the directories in their space.
TEXTTEST_CHECKOUT will be set to the first existing path found
in this way.
The “short name” can be used in these paths as
$TEXTTEST_CHECKOUT_NAME. In previous versions of TextTest, the
rule has been to concatenate the “short name” with
the path given by “checkout_location”, which for
backwards compatibility is still the behaviour if the
checkout_location doesn't depend on $TEXTTEST_CHECKOUT_NAME.
To change the checkout on the command line, use the -c
option, or from the static GUI, fill in the “Use checkout”
text box under the “What to Run” tab. If the value
provided is a relative path, it will be used as the “short
name” and combined with the corresponding value of
"checkout_location" as described above. If it is an
absolute path, it will be used as is and the config file
When a large test suite has been created, you often want to
gather information from it, or even update its contents in a
predictable way. It is very useful to be able to re-use
TextTest's ability to parse and understand the structure while
writing your own script in Python to analyse or update in terms
of the applications, test suites and test cases.
There is thus a mechanism to plug in arbitrary scripts, which
are run with “-s <module_name>.<class_name>”,
where <module_name> is some Python module and <class_name>
is the name of the script. In order to pass arguments to such
scripts, a form with <option>=<value> is used. For
example, to call the default.ReplaceText script with appropriate
arguments, you would call:
texttest.py -s “default.ReplaceText file=errors old=bad new=good”
TextTest includes several such scripts which have been found to
be generally useful, which are listed
. To see how to write your own such scripts, consult the
guide to writing your own
The above example (default.ReplaceText) is a particularly
useful way to update lots of results in a predictable way. It is
basically a search-and-replace mechanism with the advantage that
you can select tests in the normal ways and the files relevant
to the testsuite will be chosen for you. The above example will
naturally replace all instances of “bad” with “good”
in all “errors” result files.
TextTest reads two file formats - Standard List Format (for
testsuite files) and Standard Dictionary Format (for
config and environment files). These are designed
to be as human readable as possible.
Both will filter out blank lines and lines beginning with
"#", the latter being interpreted as comments. It's
good practice to use the latter feature to document things about
your application and its tests in the TextTest files themselves.
Both will also expand environment variables, indicated by
Standard List Format is simple: each entry is a complete
line. So a Standard List File is simply interpreted as an
ordered list of the lines in it which are not blank and do not
start with "#".
Standard Dictionary Format has entries in the form
<key>:<value>, where <key> indicates an
environment variable to be set to <value> in the case of
the environment file, and a variable understood by
TextTest in the case of the config file.
In the case of the config file, it can be useful to
have the value be a list itself. This is achieved by adding
several entries for the same <key>. So if we want to set
the key “my_key” to a list containing A and B,, this
is done by
Providing the entry
allows overriding files to remove entries added in a parent
It can also be useful to have the value be a dictionary. This
is achieved by the "section header" format,
Entries within section headers can also use the list format
In a few cases the value is essentially a dictionary, but
with two keys (a “composite dictionary entry”). This
is used for the batch mode and performance-related settings. The
format does not look very different to the above, but because
anything can appear in the dictionary rather than predefined key
names, it is necessary to have a mechanism to share values. This
is achieved by the special predefined key name “default”.
Here a lookup of first_key in my_composite_dictionary will
produce “first_value”. A lookup of “second_key”
(or anything else) will produce “default_value”.
If you only wish to override the default value of such a
dictionary, it is acceptable to use the format for
non-dictionary entries, i.e. simply
The TextTest download includes a test suite for itself. It is
recommended that you look around this (or any other example you
can find) to get an idea of how it works. There is also a quick
guide document included to help you find the simple “target
application” test suites which are used for testing
itself. These are simple in order to provide minimal tests, so
function as demonstration examples also.
When writing your own tests it is often best to start with
working files for another application and edit them suitably.
This reduces the risk of typing things wrongly, particularly in
the config file.