It is often needed to define different versions of an application which may not be quite the same in all respects, but which want to share some portion of the test suite structure of the parent application. For example different releases of the same system may not be able to pass all tests, and may produce slightly different log schemas, but are similar enough to want to share most of the test suite.

A version is simply identified by a name. For each TextTest file type (see the guide to TextTest test suites), the framework will then look also for files called <root>.<app>.<version> where <root> is “environment”, “config”, “testsuite” etc. in each case. If such a file does not exist, <root>.<app> is then always used. If the file does exist, it is used instead of <root>.<app> in the case of files in test cases (which are interpreted by the SUT) and Standard List Format files (i.e. the testsuite.<app> files). In the case of Standard Dictionary Format (i.e. the config and environment files), however, they are interpreted as overriding particular entries in the dictionary, so that entries not present in <root>.<app>.<version> are read from <root>.<app>.

The presence of versioned files also affect the mechanism for finding and prioritising files in the hierarchy. In this case the search path is essentially processed in order once per version present. This means that a data file or environment file in a test suite which has a relevant version suffix will be preferred to a file that has no such suffix but is found in the test case. (In TextTest 3.10 and earlier test-specific files were preferred to version-specific files).

It is possible to save test results with a version identifier, so that they will be used for comparison next time that version is run. So particular tests can have version-specific results while others are shared between all versions.

To specify a version to run, fill in the “run this version” field under the Running/Basic tabs on the static GUI, or use -v <version> on the command line. Note that there is a similar field in the Recording tab in case you are testing GUIs.

Several versions can be "aggregated" and used at the same time. This is done by specifying -v <version1>.<version2> on the command line, and can be applied to any number of versions.

Note that such aggregation is not order-specific. Running with “-v a.b” will be exactly equivalent to running “-v b.a” : no preference is indicated by the ordering. This can be a problem when there are settings for version a and version b which are in conflict – it is essentially not defined which will be preferred to which.

To clearly define which versions should be preferred to which other versions, use the “version_priority” config file setting. This takes the form of a dictionary, where the keys are version names and the values are numbers, where a low number implies that the settings for that version should be used in preference to another version. The default priority is 99 for all files.

You can also say that a version "inherits" settings from another by adding the "base_version" entry to the config file. Thus if config.<app>.v1 contains the line "base_version:v2" then all the files for v1 are also read as well as those for v2.

Note that the syntax -v <version1>,<version2> is similar to -a <app1>,<app2>: i.e. it will first run everything with version1 and then run everything with version2. It can be useful to start another version all the time when running tests, so that -v v1 behaves like -v v1,v2 at all times (or no option behaves like -v ,v2). This is achieved by adding the line "extra_version:v2" to the relevant config file.

When multiple versions are given to the static GUI in this way, only one line will appear in the test tree view for each test, irrespective of how many versions it is present for. A separate count is then kept of the number of tests and the number of distinct tests: the latter is a count of how many rows are being shown. Actions such as copying and pasting will apply to all versions of the test. In the dynamic GUI however a separate tree will be produced for each version so that the results can be viewed separately.

It can also be useful to specify that certain additional versions should only be run in particular batch sessions, if they are likely to consume a lot of resources. This is done via the config file entry "batch_extra_version:v2", which is keyed on batch session like most other batch settings. For more details on batch mode see plugin script

Sometimes you may want to run multiple copies of the same test, for example in order to try and track down a race condition or other indeterminism. Here, you can use the "-cp" option from the command line ("Times to Run" entry on the Running/Basic tab in the static GUI). This is essentially just syntactic sugar for running multiple versions whose names are generated on the fly: i.e. -cp 3 is the same thing as -v ,copy_1,copy_2.

Such versioned results are easy to create but tend to be hard to remove, you can end up with a lot of identical files with different version IDs. To help solve this, the plugin script “comparetest.PrintObsoleteVersions” will identify such redundancy for you so that you can remove the files. It will also warn where versions are equivalent but not redundant. For example, if the files stdout.myapp and stdout.myapp.2 are identical, then stdout.myapp.2 will be suggested for removal. If stdout.myapp.2 and stdout.myapp.3 are the same, then only a warning is printed. Previous to TextTest 3.10 this script was called RemoveObsoleteVersions, assumed the tests were in CVS and actually performed removals itself. The new version is better tested, more generic and less easy to misuse hopefully.

All of the files and directories discussed here can amount to a substantial structure once you have a few tests. These will clearly change over time along with the code that they test. When you have multiple developers it is hence nearly always a good idea to version-control the test suite files so that developers making changes to the test suite do not disturb each other. Using the history provided by version-control software can also be very useful to track the behaviour of your application over time.

TextTest does not integrate directly with any version control software right now, but it does have a concept of a “checkout” which aids in using it. In a version-controlled environment, you want to be able to specify the path to the SUT as a relative path, so that different developers can test their own code in their own user space, and they can also painlessly run each others code or maintain several checkouts of the system..

It is expected that TextTest test suites will want to be version-controlled, and hence an easy means of switching between different "checkouts" of the version-control system is needed. A checkout is different to a version in that all checkouts are expected to produce the same results, and making sure that last night's central checkout does the same as a developer's local code is an essential part of verifying development work.

TextTest will export the environment variable TEXTTEST_CHECKOUT. Any setting in the config or environment files can be made to depend on this variable: you can insert it as you would with other environment variables.. A very common usage is the compulsory “executable” setting in the config file.

How does TextTest detemine its value? A default value is provided via the config file, but it can also be configured per run. Checkouts are identified by short checkout names, and/or by the full path which they correspond to.

There are two entries in the config file, "checkout_location" and "default_checkout". “default_checkout” identifies what the short name of the checkout is by default. “checkout_location” is a dictionary entry mapping these short names to one or more full paths, so several different variants can be provided depending on how different users have named the directories in their space. TEXTTEST_CHECKOUT will be set to the first existing path found in this way.

The “short name” can be used in these paths as $TEXTTEST_CHECKOUT_NAME. In previous versions of TextTest, the rule has been to concatenate the “short name” with the path given by “checkout_location”, which for backwards compatibility is still the behaviour if the checkout_location doesn't depend on $TEXTTEST_CHECKOUT_NAME.

To change the checkout on the command line, use the -c option, or from the static GUI, fill in the “Use checkout” text box under the “Running/Basic” tabs. If the value provided is a relative path, it will be used as the “short name” and combined with the corresponding value of "checkout_location" as described above. If it is an absolute path, it will be used as is and the config file settings ignored. You can also provide a comma-separated list of checkouts, which will run several system instances against the same tests, much as if they were separate versions.

If you use CVS, Bazaar or Mercurial to version-control your tests, you can view log, diff, status and annotation information directly from the TextTest GUI. An appropriate menu and drop-down menus will just appear, you don't need to do anything to enable it (which you did up to version 3.12). Hopefully more support for version control systems will follow.

Additionally, normal bookkeeping of tests is now sensitive to the version control system. So renaming, moving and removing of tests that are in CVS,Bazaar or Mercurial will also do so in the appropriate version control system. It is also possible to add tests or individual files to version control from the appropriate menu: this is a separate action to actually adding the test. All new or copied tests will not be added to version control automatically.