Tests in TextTest are evaluated on the basis of comparing plain text output produced by an application to a previously accepted version of that text. By default, these will be stored in two files which will be compared exactly, and any different at all will be reported as failure.

In practice the system under test may well write out the current date or the process ID, which will of course be different next time the test is run. Naturally we don't want to have tests which just fail all the time so TextTest needs to be told about such "run-dependent text", so it can compare a filtered version of the SUT output which is deterministic. This document describes the various filtering operations that can be performed on your application's output before it is compared with the "Standard Result".

This is controlled primarily by the config file dictionary entry 'run_dependent_text', whose keys corresponding to the TextTest name of the file : i.e. the stem of the file name. This could be "stdout" or "stderr" for the standard output and error of the application (alternatively "errors" and "output" using the classic naming scheme), or it could be the name of a collated file (See also the file format documentation for more details of this). The values are lists of strings or regular expressions to search for. For example:

[run_dependent_text]
stdout:Process ID
stdout:[0-9][0-9]:[0-9][0-9]
my_file:Machine name

This will cause all lines that contain the string “Process ID” or match the given regular expression on the second line to be filtered out from the standard output. Likewise, the collated file "my_file" will be stripped of lines containing the string "Machine name".

The patterns provided may contain regular expressions. Any line in the file which matches the expression, or contains the text provided, will be filtered out in its entirety from the comparison. The regular expressions used are those used by Python, and their format is documented here.

TextTest tries to judge whether a regular expression is intended or not, and it does this by looking for the presence of any of the characters "^$[]{}\*?|+". If one is found, it will attempt to compile the regular expression, and failure will result in using the string as a literal string to search for. If none is found the string will be interpreted this way to start with.

Therefore a regular expression containing only "." characters, such as "..." to mean any three characters, will not be recognised by TextTest and will be interpreted as three literal dots.

Various extensions are available, using a special syntax specific to this entry. This is defined as follows:

For example:

[run_dependent_text]
stdout:Process ID{WORD 3}
stdout:[0-9][0-9]:[0-9][0-9]{LINES 3}
stderr:{LINE 1}
my_file:Machine name{->}End of Machines Section
my_file:{INTERNAL writedir}{REPLACE <texttest write dir}

There is also the config file dictionary entry “unordered_text” which works in a similar way and supports a similar syntax to “run_dependent_text”. In this case the matching text is not removed, but assumed to appear in random order. It is therefore sorted alphabetically and placed in a section of its own at the end of the filtered file, so that the contents are asserted to be as before but the order in which things occur is not important.

Sometimes you want to create a test suite that will run on multiple operating systems. TextTest's test suite for itself is one such example. The main problem you run into is that different operating systems use different characters for the end of lines, so that a “standard result” file from UNIX and a generated file from a test run on Windows will compare as different.

To fix this, set the “home_operating_system” config file entry. This string should be one of the strings Python uses to identify operating systems, i.e. “posix” or “nt” for the platforms supported by TextTest. It defaults to “any” which means don't worry about it.

If the entry is set and different from the running operating system, TextTest will perform the filter operation on all result files even if no filters are defined for them. This makes sure that all files are generated for the target platform and avoids false failures on line endings.

If the output contains floating point values, setting up the tests can be messy because the “exact” values may change depending on compiler, operating system, optimization level, and so on. To cope with this, there is the config file dictionary entry “floating_point_tolerance” and “relative_float_tolerance” which work similar to “run_dependent_text” and try to filter these changes by detecting floating point data in your input and only report them as a difference if they exceed the tolerance specified, e.g.

[floating_point_tolerance]
stdout:0.0101

will report differences only if two floating point values in the standard output differ by more than 0.0101, thus 6.00 will be “equal” to 6.01. The following setting

[relative_float_tolerance]
stdout:0.01

will allow for deviations up to 1%, which means 6.00 is still “equal” to 6.01 but 0.51 against 0.52 does not match (which would be tolerated with the absolute tolerance setting). Both filtering mechanisms can be applied on the same file. The former is often useful to cope with rounding errors, the latter to check for deficiencies coming from different floatng point precisions.

This filtering step is applied after the run_dependent_text step. Please keep in mind that this filtering might not always work as expected, because it operates based on a textual diff. This means if the numbers you want to be compared are not at the same location in the file this filtering will not work. Furthermore the detection of floating point values is not very elaborate, it supports scientific notation though. If you happen to see unexpected bevavior concerning this option, try to compare the files using texttest/lib/default/fpdiff.py as a command line tool and report the results back to the texttest mailing list.