Documentation for 3.10
The TextTest Sandbox
handling test data and avoiding global side effects
When TextTest tests are run, it will try to write the output
of the system under test to a temporary directory structure
created specifically for this purpose. This is known as the
“TextTest sandbox”: its purpose is to provide a
totally separate environment where your system can create,edit
and delete as much as it wants without doing anything permanent,
and also an environment where test data can be provided in an
easily accessed way.
There is one created for each run of each test. It is created
under the directory indicated by the environment variable
TEXTTEST_TMP. It defaults to ~/texttesttmp on UNIX systems and
the value of the TEMP environment variable on Windows. This is
hereafter referred to as the “root temporary directory”.
Each time TextTest is started, it is assigned a unique
identifier based on the version and the time stamp at which the
run was submitted (the string “static_gui” is
prepended in the case of the static GUI). A subdirectory of the
root temporary directory is then created with this name. All
temporary files and directories created by this run will then be
created under this directory.
In the case of the dynamic GUI or the console interface,
tests are actually being run. This means that, for every test
being run, a temporary directory structure is created which
essentially mirrors the permanent directories which represent
the tests (see the guide to TextTest
), so that each test is assigned a unique
temporary directory. All temporary files corresponding to
particular tests are then written to these directories. When the
tests are run, each test starts the system under test with the
corresponding temporary directory as current working directory,
with its standard output and standard error redirected to local
It is imperative that you ensure any other files created by
the system under test are created relative to this temporary
directory, to avoid global side effects and to aid TextTest in
finding them. This should be possible by always specifying
relative paths in your test configuration files, which will be
interpreted relative to this directory for each run.
(In the temporary directory for each test case, TextTest
creates a subdirectory called “framework_tmp”. It
uses this to write its own temporary files, such as filtered
versions of the output, performance data etc.)
In the case of the static GUI, the temporary directory will
contain logs of each dynamic GUI run that is started from it.
These will write files in subdirectories labelled
dynamic_run<n>, with the numbers increasing for each run
that is started. When the dynamic GUI is closed, the contents of
whatever it wrote on standard error will be displayed in a
message box by the static GUI, as well as in a file in this
Sometimes the system under test needs to read some file
relative to the current working directory. TextTest allows you
to place such files in the permanent
test directory structure
. You should then specify the
“link_test_path” config file entry as the (local)
file name of the file you want to provide. You can then refer to
a local file of the appropriate name in your options
in that test case, for example.
TextTest will look for the file name you specify, first in
the permanent directory of the test case, then in the parent
test suite and so on up to the root. If it finds such a file (or
directory), it will create a symbolic link to it from the
temporary directory (UNIX) or copy it (Windows). If it doesn't,
it will silently continue, as it is regaded as a normal
situation to need test data files for some tests but not others.
The files can be given any name at all (unless the system
under test requires a particular name), but beware! For those
who have got used to TextTest's naming scheme by now, it can be
tempting to add .<app> endings. However, the naming scheme
used by files interpreted or compared by TextTest doesn't apply
to data files, and such naming isn't generally a good idea as it
will convince TextTest that these are files that should be
This search path can also be extended as desired, use the
“test_data_searchpath” config file entry. This is a
dictionary whose keys are the names given in (for example)
“link_test_path” and whose values are directories to
look further for test data. The “default” key can be
specified to make the same search path apply to all test data.
Sometimes the system under test will itself edit existing
files. In this case, you will want to copy to the temporary
directory the file or directory structure which it plans to
edit, so that test runs are repeatable and do not have global
side effects. You can do this using the “copy_test_path”
config file entry, which will find files or directories to copy
in the same way as link_test_path, and indeed is equivalent to
link_test_path on Windows.
On UNIX, there is a third option. Sometimes an application
may need to read from a very large directory structure, and
potentially edit some files in it. Copying the whole structure
for each test run is possible but time consuming. It's better to
be able to copy just the parts that will be changed and link the
rest. This is done with the “partial_copy_test_path”
config file entry, in conjunction with the catalogue
(“create_catalogues” in the
config file). The first time the test is run, all the files are
copied, and the catalogue records which files are created,
edited and deleted. The next time, the structure will be copied
and linked as determined by what is in the catalogue file.
If any use is made of symbolic links to the master data, it
is generally recommended to make the entire “master copy”
of the data readonly, in case bugs in the application would
cause it to corrupt the test data. It is possible to tell
TextTest to ignore the catalogue file and copy everything again
if the file-changing properties of the test change : check the
“Ignore catalogue when isolating data” box
(-ignorecat on the command line)
Applications will often reference their test data structures
via environment variables. When these structures are isolated by
TextTest as described above, it can be helpful to update the
variables accordingly. There are two ways to do this, which are
fairly similar in effect.
The first is to simply refer to the environment variable
using one of the test data config file settings described above
(link_test_path, copy_test_path or partial_copy_test_path). For
example, you could write
This would take the value of the environment variable MY_ENV_VAR
as determined by the environment files and the external
environment, identify if it refers to an existing file or
directory, and if so, copy that as test data. The environment
variable will also be updated to point at the absolute path of
the copied location.
Alternatively, you can associate environment variables with
test data found via the normal mechanism. This is done via the
“test_data_environment” config file setting, which
is a dictionary. For example
For each name identified by link_test_path, copy_test_path or
partial_copy_test_path, you can provide an entry which will be
the name of an environment variable to set to the isolated
version of the data.
Note! In both of
these cases the environment variables will be set even if no
data is found. The assumption is that the system under test
might in that case want to create such data in an equivalent
If you specify a directory as test data, via any of the three
ways described above, it will be treated as test data
recursively in its entirety. Sometimes, however, some parts of
it are not really part of the data and should not be displayed
as such, either in the GUIs or in the catalogue files. If it is
version controlled via CVS it is for example likely to contain
CVS directories which we will want to ignore.
To achieve this, use the setting “test_data_ignore”.
The keys are names identified by “link_test_path”,
“copy_test_path” or “partial_copy_test_path”.
The values should be names (or regular expressions) of files or
directories under the relevant structure which should be
ignored. If a directory is ignored, so are its contents.
This has several effects. These files/directories will not be
shown in the static GUI (which will behave as if they didn't
exist) and changes in them will not appear in the catalogue
file, if there is one. If you are using
“partial_copy_:test_path”, that means they will also
not be copied, ever: changes there are regarded as
uninteresting. It is thus a very bad idea to use this setting in
conjunction with partially copied structures, if there is any
chance of a write-conflict between several tests, or if files
will be created (and not deleted) by test runs in that place. In
the last case the directory would grow without limit.
When you quit the GUI (or the console interface terminates),
the temporary directory associated with the run is by default
automatically removed. It is thus important to save any test
results that you wish to use again, either as the default result
or as a version.
It is also possible to run TextTest in “keeptmp”
mode. This means that the temporary directory structure of the
run is not removed when texttest exits. However, beware!
In this mode it will clean up the run directories of previous
runs that it finds which had the same application and version.
This means that you cannot run more than one instance of
TextTest simultaneously on the same application with the same
version in “keeptmp” mode, or they will delete each
Running in batch mode
selects “keeptmp” mode for the temporary
directories. It may also be requested explicitly using the
“-keeptmp” option on the command line, or checking
the “keep temporary write directories” box in the
“side effects” tab from the static GUI.