|
Documentation for 3.14
Filtering the Output from the System Under Test
Making the tests determinstic by removing run-dependent text
Tests in TextTest are evaluated on the basis of comparing
plain text output produced by an application to a previously accepted
version of that text. By default, these will be stored in two files which will
be compared exactly, and any different at all will be reported as failure.
In practice the system under test may well write out the current date or the process ID,
which will of course be different next time the test is run. Naturally we don't want
to have tests which just fail all the time so TextTest needs to be told about
such "run-dependent text", so it can compare a filtered version of the SUT output
which is deterministic. This document describes the various filtering operations
that can be performed on your application's output before it is compared with the
"Standard Result".
This is controlled primarily by the config file dictionary
entry 'run_dependent_text', whose keys corresponding to the
TextTest name of the file : i.e. the stem of the file name. This could be "output" or
"errors" for the standard output and error of the application, or it could be the
name of a collated file (See also the file format documentation
for more details of this). It should take this form:
[run_dependent_text]
<texttest_name>:<run_dependent_pattern1>
<texttest_name>:<run_dependent_pattern2>
The patterns provided may contain regular expressions. Any line
in the file which matches the expression, or contains the text
provided, will be filtered out in its entirety from the
comparison. The regular expressions used are those used by Python,
and their format is documented here.
TextTest tries to judge whether a regular expression is intended or not, and it does this by looking for the presence of
any of the characters "^$[]{}\*?|+". If one is found, it will attempt to compile the regular expression, and failure will
result in using the string as a literal string to search for. If none is found the string will be interpreted this way to
start with.
Therefore a regular expression containing only "." characters,
such as "..." to mean any three characters, will not be recognised by TextTest and will be interpreted
as three literal dots.
For example:
[run_dependent_text]
output:Process ID
output:[0-9][0-9]:[0-9][0-9]
my_file:Machine name
This will cause all lines that contain the string “Process
ID” or match the given regular expression to be filtered
out from the standard output. Likewise, the collated file
my_file will be stripped of lines containing the string “Machine
name”.
Various extensions are available, using a special syntax
specific to this entry. This is defined as follows:
- <pattern>{LINES
<number_of_lines>} - On encountering a
match with <pattern>, instead of removing the single line
containing the pattern, remove <number_of_lines>. The
count includes the line matched, so {LINES 1} has no effect.
- {LINE <line_number>} - Remove the
entire line <line_number>, counting from the top of the
file.
- <pattern>{WORD <word_number>}
- On encountering a match with the pattern, do not remove the
whole line but only the word numbered <word_number> from
the start. Use negative numbers to count from the end of the
line: i.e. {WORD -2} will remove the second-to-last word. You
can also specify to remove every word after a certain number,
for example {WORD 4+} will remove word 4 and the rest of the
line after this.
- <pattern1>{->}<pattern2>
- On encountering a match with <pattern1>, all lines are
filtered out until <pattern2> is matched. Neither the
line matching <pattern1> nor the line matching <pattern2>
are themselves filtered.
- <pattern>{REPLACE
<text>}- On encountering a match with the
pattern, instead of removing the whole line, replace just the
part of the line which matched the pattern with the text
indicated by <text>. This can be combined with {WORD...}
which will replace just the indicated word rather than the text
matched. If <pattern> is a regular expression, it can
include groups (indicated by parantheses) which can then be
referred to in the REPLACE text as \1, \2 etc.
- {INTERNAL writedir} – This is a
special pattern that will match TextTest's own temporary paths
for the test. Sometimes your application will write out
absolute paths, which will naturally be relative to the
temporary directory where the test is run. These will then
produce different text every time. This syntax is mostly to
save you the bother of producing an exact regular expression to
match these paths.
For example:
[run_dependent_text]
output:Process ID{WORD 3}
output:[0-9][0-9]:[0-9][0-9]{LINES 3}
errors:{LINE 1}
my_file:Machine name{->}End of Machines Section
my_file:{INTERNAL writedir}{REPLACE <texttest write dir}
There is also the config file dictionary entry
“unordered_text” which works in a similar way and
supports a similar syntax to “run_dependent_text”.
In this case the matching text is not removed, but assumed to
appear in random order. It is therefore sorted alphabetically
and placed in a section of its own at the end of the filtered
file, so that the contents are asserted to be as before but the
order in which things occur is not important.
Sometimes you want to create a test suite that will run on
multiple operating systems. TextTest's test suite for itself is
one such example. The main problem you run into is that
different operating systems use different characters for the end
of lines, so that a “standard result” file from UNIX
and a generated file from a test run on Windows will compare as
different.
To fix this, set the “home_operating_system”
config file entry. This string should be one of the strings
Python uses to identify operating systems, i.e. “posix”
or “nt” for the platforms supported by TextTest. It
defaults to “any” which means don't worry about it.
If the entry is set and different from the running operating
system, TextTest will perform the filter operation on all result
files even if no filters are defined for them. This makes sure
that all files are generated for the target platform and avoids
false failures on line endings.
If the output contains floating point values
setting up the tests can be messy because the “exact” values
change depending on compiler, os, optimization level, ...
Thus there is the config file dictionary entry “floating_point_tolerance”
which works in a similar way and tries to filter these changes by detecting
floating point data in your input and only report them as a difference if
they exceed the tolerance specified, e.g.
[floating_point_tolerance]
output:0.0101
will report differences only if the two floating point values to compare differ
by more than 0.0101, thus 6.00 will be “equal” to 6.01.
This filtering step is applied after the run_dependent_text
step. Please keep in mind that this filtering might not always work as expected,
because it operates based on a textual diff. This means if the numbers
you want to be compared are not at the same location in the file this filtering
will not work. Furthermore the detection of floating point values is not very
elaborate, it supports scientific notation though.
If you happen to see unexpected bevavior concerning this option, try to compare
the files using texttest/lib/default/fpdiff.py as a command line tool
and report the results back to the texttest mailing list.
|