Documentation for 3.14
Using TextTest Versions and Checkouts
Handling different versions and instances of the system under test
It is often needed to define different versions of an
application which may not be quite the same in all respects, but
which want to share some portion of the test suite structure of
the parent application. For example different releases of the same system
may not be able to pass all tests, and may produce slightly different log
schemas, but are similar enough to want to share most of the test suite.
A version is simply identified by a name. For each TextTest file type
guide to TextTest test suites
), the framework will then
look also for files called <root>.<app>.<version>
where <root> is “environment”, “config”,
“testsuite” etc. in each case. If such a file does
not exist, <root>.<app> is then always used. If the
file does exist, it is used instead of <root>.<app>
in the case of files in test cases (which are interpreted by the
SUT) and Standard List Format files (i.e. the testsuite.<app>
files). In the case of Standard Dictionary Format (i.e. the
config and environment files), however, they are interpreted as
overriding particular entries in the dictionary, so that entries
not present in <root>.<app>.<version> are read
The presence of versioned files also affect the
mechanism for finding
and prioritising files in the hierarchy
. In this case the search path is essentially
processed in order once per version present. This means that a data file or environment file
in a test suite which has a relevant version suffix will be preferred to a file that
has no such suffix but is found in the test case. (In TextTest 3.10 and earlier
test-specific files were preferred to version-specific files).
It is possible to save
test results with a version identifier, so that they will be used for
comparison next time that version is run. So particular tests can
have version-specific results while others are shared between all versions.
To specify a version to run, fill in the “run this version” field
under the Running/Basic tabs on the static GUI, or use -v <version> on the command
line. Note that there is a similar field in the Recording tab in
case you are testing GUIs.
Several versions can be "aggregated" and used at
the same time. This is done by specifying -v
<version1>.<version2> on the command line, and can
be applied to any number of versions.
Note that such aggregation is not order-specific. Running
with “-v a.b” will be exactly equivalent to running
“-v b.a” : no preference is indicated by the
ordering. This can be a problem when there are settings for
version a and version b which are in conflict – it is
essentially not defined which will be preferred to which.
To clearly define which versions should be preferred to which
other versions, use the “version_priority” config
file setting. This takes the form of a dictionary, where the
keys are version names and the values are numbers, where a low
number implies that the settings for that version should be used
in preference to another version. The default priority is 99 for
You can also say that a version "inherits" settings
from another by adding the "base_version" entry to the
config file. Thus if config.<app>.v1 contains the line
"base_version:v2" then all the files for v1 are also
read as well as those for v2.
Note that the syntax -v <version1>,<version2> is
similar to -a <app1>,<app2>: i.e. it will first run
everything with version1 and then run everything with version2.
It can be useful to start another version all the time when
running tests, so that -v v1 behaves like -v v1,v2 at all times
(or no option behaves like -v ,v2). This is achieved by adding
the line "extra_version:v2" to the relevant config
When multiple versions are given to the static GUI in this way, only one line will appear
in the test tree view for each test, irrespective of how many versions it is present for.
A separate count is then kept of the number of tests and the number of distinct tests: the
latter is a count of how many rows are being shown. Actions such as copying and pasting
will apply to all versions of the test. In the dynamic GUI however a separate tree will be produced
for each version so that the results can be viewed separately.
It can also be useful to specify that certain additional versions should only be run in particular
batch sessions, if they are likely to consume a lot of resources. This is done via the config file
entry "batch_extra_version:v2", which is keyed on batch session like most other batch settings.
For more details on batch mode see
Sometimes you may want to run multiple copies of the same test, for example in order to try
and track down a race condition or other indeterminism. Here, you can use the "-cp"
option from the command line ("Times to Run" entry on the Running/Basic tab in the static GUI).
This is essentially just syntactic sugar for running multiple versions whose names are generated on the fly:
i.e. -cp 3 is the same thing as -v ,copy_1,copy_2.
Such versioned results are easy to create but tend to be hard
to remove, you can end up with a lot of identical files with
different version IDs. To help solve this, the plugin
identify such redundancy for you so that you can remove the files. It will
also warn where versions are equivalent but not redundant. For
example, if the files output.myapp and output.myapp.2 are
identical, then output.myapp.2 will be suggested for removal. If
output.myapp.2 and output.myapp.3 are the same, then only a
warning is printed. Previous to TextTest 3.10 this script was called RemoveObsoleteVersions,
assumed the tests were in CVS and actually performed removals itself. The new version
is better tested, more generic and less easy to misuse hopefully.
All of the files and directories discussed here can amount to
a substantial structure once you have a few tests. These will
clearly change over time along with the code that they test.
When you have multiple developers it is hence nearly always a
good idea to version-control the test suite files so that
developers making changes to the test suite do not disturb each
other. Using the history provided by version-control software
can also be very useful to track the behaviour of your
application over time.
TextTest does not integrate directly with any version control
software right now, but it does have a concept of a “checkout”
which aids in using it. In a version-controlled environment, you
want to be able to specify the path to the SUT as a relative
path, so that different developers can test their own code in
their own user space, and they can also painlessly run each
others code or maintain several checkouts of the system..
It is expected that TextTest test suites will want to be
version-controlled, and hence an easy means of switching between
different "checkouts" of the version-control system is
needed. A checkout is different to a version in that all
checkouts are expected to produce the same results, and making
sure that last night's central checkout does the same as a
developer's local code is an essential part of verifying
TextTest will export the environment variable
TEXTTEST_CHECKOUT. Any setting in the config or
environment files can be made to depend on this variable:
you can insert it as you would with other environment
variables.. A very common usage is the compulsory “executable”
setting in the config file.
How does TextTest detemine its value? A default value is
provided via the config file, but it can also be configured per
run. Checkouts are identified by short checkout names, and/or by
the full path which they correspond to.
There are two entries in the config file, "checkout_location"
and "default_checkout". “default_checkout”
identifies what the short name of the checkout is by default.
“checkout_location” is a dictionary entry mapping
these short names to one or more full paths, so several
different variants can be provided depending on how different
users have named the directories in their space.
TEXTTEST_CHECKOUT will be set to the first existing path found
in this way.
The “short name” can be used in these paths as
$TEXTTEST_CHECKOUT_NAME. In previous versions of TextTest, the
rule has been to concatenate the “short name” with
the path given by “checkout_location”, which for
backwards compatibility is still the behaviour if the
checkout_location doesn't depend on $TEXTTEST_CHECKOUT_NAME.
To change the checkout on the command line, use the -c
option, or from the static GUI, fill in the “Use checkout”
text box under the “Running/Basic” tabs. If the value
provided is a relative path, it will be used as the “short
name” and combined with the corresponding value of
"checkout_location" as described above. If it is an
absolute path, it will be used as is and the config file
settings ignored. You can also provide a comma-separated list of checkouts,
which will run several system instances against the same tests, much
as if they were separate versions.
If you use CVS, Bazaar or Mercurial to version-control your tests,
you can view log, diff, status and annotation information directly
from the TextTest GUI. An appropriate menu and drop-down menus will just appear,
you don't need to do anything to enable it (which you did up to version 3.12).
Hopefully more support for version control systems will follow.
Additionally, normal bookkeeping of tests is now sensitive
to the version control system. So renaming, moving and removing of tests that
are in CVS,Bazaar or Mercurial will also do so in the appropriate version control system.
It is also possible to add tests or individual files to version control from
the appropriate menu: this is a separate action to actually adding the test. All
new or copied tests will not be added to version control automatically.