Documentation for 3.10
Other versions: trunk  3.27  3.26  3.25  3.24  3.23  3.22  3.21  3.20  3.19  3.18  Older...

Filtering the Output from the System Under Test
Making the tests determinstic by removing run-dependent text
Introduction
Tests in TextTest are evaluated on the basis of comparing plain text output produced by an application to a previously accepted version of that text. By default, these will be stored in two files which will be compared exactly, and any different at all will be reported as failure.
In practice the system under test may well write out the current date or the process ID, which will of course be different next time the test is run. Naturally we don't want to have tests which just fail all the time so TextTest needs to be told about such "run-dependent text", so it can compare a filtered version of the SUT output which is deterministic. This document describes the various filtering operations that can be performed on your application's output before it is compared with the "Standard Result".
Filtering out Run-dependent Text
This is controlled primarily by the config file dictionary entry 'run_dependent_text', whose keys corresponding to the TextTest name of the file : i.e. the stem of the file name. This could be "output" or "errors" for the standard output and error of the application, or it could be the name of a collated file (See also the file format documentation for more details of this). It should take this form:
[run_dependent_text]
<texttest_name>:<run_dependent_pattern1>
<texttest_name>:<run_dependent_pattern2>
The patterns provided may contain regular expressions. Any line in the file which matches the expression, or contains the text provided, will be filtered out in its entirety from the comparison. For example:
[run_dependent_text]
output:Process ID
output:[0-9][0-9]:[0-9][0-9]
my_file:Machine name
This will cause all lines that contain the string “Process ID” or match the given regular expression to be filtered out from the standard output. Likewise, the collated file my_file will be stripped of lines containing the string “Machine name”.
Filtering out multiple lines and parts of lines
Various extensions are available, using a special syntax specific to this entry. This is defined as follows:
  • <pattern>{LINES <number_of_lines>} - On encountering a match with <pattern>, instead of removing the single line containing the pattern, remove <number_of_lines>. The count includes the line matched, so {LINES 1} has no effect.
  • {LINE <line_number>} - Remove the entire line <line_number>, counting from the top of the file.
  • <pattern>{WORD <word_number>} - On encountering a match with the pattern, do not remove the whole line but only the word numbered <word_number> from the start. Use negative numbers to count from the end of the line: i.e. {WORD -2} will remove the second-to-last word. You can also specify to remove every word after a certain number, for example {WORD 4+} will remove word 4 and the rest of the line after this.
  • <pattern1>{->}<pattern2> - On encountering a match with <pattern1>, all lines are filtered out until <pattern2> is matched. Neither the line matching <pattern1> nor the line matching <pattern2> are themselves filtered.
  • <pattern>{REPLACE <text>}- On encountering a match with the pattern, instead of removing the whole line, replace just the part of the line which matched the pattern with the text indicated by <text>. This can be combined with {WORD...} which will replace just the indicated word rather than the text matched. If <pattern> is a regular expression, it can include groups (indicated by parantheses) which can then be referred to in the REPLACE text as \1, \2 etc.
  • {INTERNAL writedir} – This is a special pattern that will match TextTest's own temporary paths for the test. Sometimes your application will write out absolute paths, which will naturally be relative to the temporary directory where the test is run. These will then produce different text every time. This syntax is mostly to save you the bother of producing an exact regular expression to match these paths.
For example:
[run_dependent_text]
output:Process ID{WORD 3}
output:[0-9][0-9]:[0-9][0-9]{LINES 3}
errors:{LINE 1}
my_file:Machine name{->}End of Machines Section
my_file:{INTERNAL writedir}{REPLACE <texttest write dir}
Filtering the order of certain lines
There is also the config file dictionary entry “unordered_text” which works in a similar way and supports a similar syntax to “run_dependent_text”. In this case the matching text is not removed, but assumed to appear in random order. It is therefore sorted alphabetically and placed in a section of its own at the end of the filtered file, so that the contents are asserted to be as before but the order in which things occur is not important.
Multiple-OS testing: forcing the filtering to occur
Sometimes you want to create a test suite that will run on multiple operating systems. TextTest's test suite for itself is one such example. The main problem you run into is that different operating systems use different characters for the end of lines, so that a “standard result” file from UNIX and a generated file from a test run on Windows will compare as different.
To fix this, set the “home_operating_system” config file entry. This string should be one of the strings Python uses to identify operating systems, i.e. “posix” or “nt” for the platforms supported by TextTest. It defaults to “any” which means don't worry about it.
If the entry is set and different from the running operating system, TextTest will perform the filter operation on all result files even if no filters are defined for them. This makes sure that all files are generated for the target platform and avoids false failures on line endings.


Last updated: 05 October 2012