Documentation for 3.14
Other versions: trunk  3.27  3.26  3.25  3.24  3.23  3.22  3.21  3.20  3.19  3.18  Older...

Filtering the Output from the System Under Test
Making the tests determinstic by removing run-dependent text
Tests in TextTest are evaluated on the basis of comparing plain text output produced by an application to a previously accepted version of that text. By default, these will be stored in two files which will be compared exactly, and any different at all will be reported as failure.
In practice the system under test may well write out the current date or the process ID, which will of course be different next time the test is run. Naturally we don't want to have tests which just fail all the time so TextTest needs to be told about such "run-dependent text", so it can compare a filtered version of the SUT output which is deterministic. This document describes the various filtering operations that can be performed on your application's output before it is compared with the "Standard Result".
Filtering out Run-dependent Text
This is controlled primarily by the config file dictionary entry 'run_dependent_text', whose keys corresponding to the TextTest name of the file : i.e. the stem of the file name. This could be "output" or "errors" for the standard output and error of the application, or it could be the name of a collated file (See also the file format documentation for more details of this). It should take this form:
The patterns provided may contain regular expressions. Any line in the file which matches the expression, or contains the text provided, will be filtered out in its entirety from the comparison. The regular expressions used are those used by Python, and their format is documented here.
TextTest tries to judge whether a regular expression is intended or not, and it does this by looking for the presence of any of the characters "^$[]{}\*?|+". If one is found, it will attempt to compile the regular expression, and failure will result in using the string as a literal string to search for. If none is found the string will be interpreted this way to start with.
Therefore a regular expression containing only "." characters, such as "..." to mean any three characters, will not be recognised by TextTest and will be interpreted as three literal dots.
For example:
output:Process ID
my_file:Machine name
This will cause all lines that contain the string “Process ID” or match the given regular expression to be filtered out from the standard output. Likewise, the collated file my_file will be stripped of lines containing the string “Machine name”.
Filtering out multiple lines and parts of lines
Various extensions are available, using a special syntax specific to this entry. This is defined as follows:
  • <pattern>{LINES <number_of_lines>} - On encountering a match with <pattern>, instead of removing the single line containing the pattern, remove <number_of_lines>. The count includes the line matched, so {LINES 1} has no effect.
  • {LINE <line_number>} - Remove the entire line <line_number>, counting from the top of the file.
  • <pattern>{WORD <word_number>} - On encountering a match with the pattern, do not remove the whole line but only the word numbered <word_number> from the start. Use negative numbers to count from the end of the line: i.e. {WORD -2} will remove the second-to-last word. You can also specify to remove every word after a certain number, for example {WORD 4+} will remove word 4 and the rest of the line after this.
  • <pattern1>{->}<pattern2> - On encountering a match with <pattern1>, all lines are filtered out until <pattern2> is matched. Neither the line matching <pattern1> nor the line matching <pattern2> are themselves filtered.
  • <pattern>{REPLACE <text>}- On encountering a match with the pattern, instead of removing the whole line, replace just the part of the line which matched the pattern with the text indicated by <text>. This can be combined with {WORD...} which will replace just the indicated word rather than the text matched. If <pattern> is a regular expression, it can include groups (indicated by parantheses) which can then be referred to in the REPLACE text as \1, \2 etc.
  • {INTERNAL writedir} – This is a special pattern that will match TextTest's own temporary paths for the test. Sometimes your application will write out absolute paths, which will naturally be relative to the temporary directory where the test is run. These will then produce different text every time. This syntax is mostly to save you the bother of producing an exact regular expression to match these paths.
For example:
output:Process ID{WORD 3}
output:[0-9][0-9]:[0-9][0-9]{LINES 3}
errors:{LINE 1}
my_file:Machine name{->}End of Machines Section
my_file:{INTERNAL writedir}{REPLACE <texttest write dir}
Filtering the order of certain lines
There is also the config file dictionary entry “unordered_text” which works in a similar way and supports a similar syntax to “run_dependent_text”. In this case the matching text is not removed, but assumed to appear in random order. It is therefore sorted alphabetically and placed in a section of its own at the end of the filtered file, so that the contents are asserted to be as before but the order in which things occur is not important.
Multiple-OS testing: forcing the filtering to occur
Sometimes you want to create a test suite that will run on multiple operating systems. TextTest's test suite for itself is one such example. The main problem you run into is that different operating systems use different characters for the end of lines, so that a “standard result” file from UNIX and a generated file from a test run on Windows will compare as different.
To fix this, set the “home_operating_system” config file entry. This string should be one of the strings Python uses to identify operating systems, i.e. “posix” or “nt” for the platforms supported by TextTest. It defaults to “any” which means don't worry about it.
If the entry is set and different from the running operating system, TextTest will perform the filter operation on all result files even if no filters are defined for them. This makes sure that all files are generated for the target platform and avoids false failures on line endings.
Filtering floating point differences
If the output contains floating point values setting up the tests can be messy because the “exact” values change depending on compiler, os, optimization level, ... Thus there is the config file dictionary entry “floating_point_tolerance” which works in a similar way and tries to filter these changes by detecting floating point data in your input and only report them as a difference if they exceed the tolerance specified, e.g.
will report differences only if the two floating point values to compare differ by more than 0.0101, thus 6.00 will be “equal” to 6.01. This filtering step is applied after the run_dependent_text step. Please keep in mind that this filtering might not always work as expected, because it operates based on a textual diff. This means if the numbers you want to be compared are not at the same location in the file this filtering will not work. Furthermore the detection of floating point values is not very elaborate, it supports scientific notation though. If you happen to see unexpected bevavior concerning this option, try to compare the files using texttest/lib/default/ as a command line tool and report the results back to the texttest mailing list.

Last updated: 05 October 2012