Documentation for 3.9.1
Other versions: trunk  3.27  3.26  3.25  3.24  3.23  3.22  3.21  3.20  3.19  3.18  Older...

File Collation and Text Filtering
Configuring the Test Result Evaluation

Introduction
Tests in TextTest are evaluated on the basis of comparing plain text output by an application to a previously accepted version of that text. This document aims to discuss this “comparison” operation and the various ways it can be configured.
The Standard Procedure and what can be configured in it
By default, the standard output of the system under test will be collected to a file called output.<app> and the standard error will be collected to a file called errors.<app> (see the guide to files and directories). These will then be compared exactly with any previously saved “standard results” and any difference at all will be reported as failure.
If standard results have not already been collected for a particular file produced when a test is run, the file is reported under “New Results” and should be checked carefully by hand and saved if correct. New files appearing is also sufficient reason to fail a test, so every test should fail the first time unless the expected results are imported externally.
In the same way, if files are not produced that are present and expected in the standard results, these will be reported under “Missing Results” and the test will fail. Saving such a result will cause the missing files to be removed from the standard results.
There are four types of things that can be configured here:
  1. Any file output by the SUT can be collected and compared, provided you tell TextTest how to find it (and potentially how to convert it to a convenient format for comparison). Files that are collected by default can also be discarded.
  2. The comparison operation itself can be configured by inserting filtering operations before the comparison is done. This means that certain uninteresting differences can be suppressed and prevented from causing correctly behaving tests from being shown as failing. There are two types of filtering : for handling the cases where the content is run-dependent, or the order of that content is run-dependent, but the content itself constant.
  3. Result files can be declared to have different “severity”, so that TextTest can help you spot more serious errors quickly. A difference in the final file output by the system is probably more serious than a difference in a file listing the memory consumption, for example.
Telling TextTest to collect additional files
This can be done by specifying the config file entry “collate_file”. This entry is a dictionary and so takes the following form :
[collate_file]
<texttest_name>:<source_file_path>

where <source_file_path> is some file your application writes and <texttest_name> is what you want it to be called by TextTest.
If you plan to do this, make sure you read the document describing how the TextTest temporary directory works first. <source_file_path> here should in principle never be an absolute path : it should be relative (implicitly to the temporary directory described above). This is because your tests will otherwise have global side effects – making them harder to understand and more prone to occasional failure, particularly if run more than once simultaneously.
Note that this ordering can seem counter-intuitive, in effect you are asking TextTest to copy the text file located at <source_file_path> to <texttest_name>.<app> in the temporary directory of that test, where it will be picked up and compared. You might expect the source to be named before the target, but many different config dictionary entries use these TextTest names for result files as keys so this one works the same for consistency.
Standard UNIX file pattern matching is allowed in both <texttest_name> and <source_file_path>. Where this is used in the path to the source file it simply means that the exact name of the file that will be produced may vary, but whatever file matches the pattern will be copied and given the same name each time by TextTest, provided it was created or modified by the test run (unchanged files will not be collected in this way).
If comparison of a collected file is not desired for any reason, it can be added to the config file list entry “discard_file”. The most common usage of this is to disable the collection of standard output and/or standard error (i.e. by adding “errors” or “output” to the list)..
If the file is not plain-text or needs to be pre-processed before it can easily be compared, you can tell TextTest to run an arbitrary script on the file. This script should take a single argument (the file name to read) and should write its output to the standard output. You do this by specifying the composite dictionary entry “collate_script”, which has the same form as “collate_file” except the value should be the name of the script to run. The script should be placed somewhere on your PATH or identified via an absolute path.
Collecting multiple related files at the same time (advanced)
When patterns are used in the TextTest name it means that all previously saved files that match this “target pattern” and all files written by the test that match the “source pattern” become collated files. E.g. suppose we have the following entry in the config file:
[collate_file]
data*:data*.dump
Suppose also that an earlier saved run had produced data1.<app> and data2.<app> and the latest run produced data1.dump and data3.dump. Then the list of collated files becomes: data1, data2, data3. This means that the latest run's data1 will be compared against the file saved in data1.<app>, data2 will be flagged as missing and data3 flagged as new result.
Some care is required in writing collate patterns. Completely general patters like “*:* “ would cause confusion since anything could relate to anything, in theory. The current implementation assumes that files have a common stem, i.e.: it can handle stems like the example above, but not unrelated stems like “*good* : *bad*”
Generating a catalogue of file/process changes
Sometimes a system will potentially create and remove a great many files in a directory structure (TextTest itself is one example!) Collecting and comparing every single file might be overkill. Instead, you have the possibility to create a catalogue file, which will essentially compare which files (under the test's temporary directory) are present before and after the test has run, and which files and directories that were present before have been edited during the test run.
It will then report what has been created, what has been removed and what has been edited. This is done by setting the config file entry “create_catalogues” to true. It will generate result files called catalogue.<app>. If no differences are found, this is noted briefly at the top of the file : catalogue files are always created from version 3.6 and onwards.
Note that this feature can be used to aid test data isolation also.
In addition, you can request that the catalogue functionality checks for processes that were created (leaked!) by the test. If such processes are found to exist, they will be reported to the catalogue file and automatically terminated by TextTest. This is done by getting the SUT to log when it creates a process in a predictable way. The text identifying the process created should be provided in the “catalogue_process_string” config file entry. TextTest will then search the result file indicated by “log_file” for matches with this string, and assume the process ID immediately follows it. If the process is found to be running, it will be reported to the catalogue file and terminated.
Filtering the Result Files before Comparison
Evaluation of test results consists by default of comparing exactly all files that have been “collated” along with the standard output and error files generated by default. Often, though, this will produce “false failures” because incidental things are reported by the files: for example: process IDs, execution machine names, today's date. These will be different every time the test is run and you want to tell TextTest about them so that it can filter them out before comparing the files.
This is controlled primarily by the config file dictionary entry 'run_dependent_text', whose keys corresponding to the TextTest name of the file : i.e. the stem of the file name. (See the file format documentation for more details of this). It should take this form:
[run_dependent_text]
<texttest_name>:<run_dependent_pattern1>
<texttest_name>:<run_dependent_pattern2>
The patterns provided may contain regular expressions. Any line in the file which matches the expression, or contains the text provided, will be filtered out in its entirety from the comparison. For example:
[run_dependent_text]
output:Process ID
output:[0-9][0-9]:[0-9][0-9]
my_file:Machine name
This will cause all lines that contain the string “Process ID” or match the given regular expression to be filtered out from the standard output. Likewise, the collated file my_file will be stripped of lines containing the string “Machine name”.
Filtering out multiple lines and parts of lines
Various extensions are available, using a special syntax specific to this entry. This is defined as follows:
  • <pattern>{LINES <number_of_lines>} - On encountering a match with <pattern>, instead of removing the single line containing the pattern, remove <number_of_lines>. The count includes the line matched, so {LINES 1} has no effect.
  • {LINE <line_number>} - Remove the entire line <line_number>, counting from the top of the file.
  • <pattern>{WORD <word_number>} - On encountering a match with the pattern, do not remove the whole line but only the word numbered <word_number> from the start. Use negative numbers to count from the end of the line: i.e. {WORD -2} will remove the second-to-last word. You can also specify to remove every word after a certain number, for example {WORD 4+} will remove word 4 and the rest of the line after this.
  • <pattern1>{->}<pattern2> - On encountering a match with <pattern1>, all lines are filtered out until <pattern2> is matched. Neither the line matching <pattern1> nor the line matching <pattern2> are themselves filtered.
  • <pattern>{REPLACE <text>}- On encountering a match with the pattern, instead of removing the whole line, replace just the part of the line which matched the pattern with the text indicated by <text>. This can be combined with {WORD...} which will replace just the indicated word rather than the text matched. If <pattern> is a regular expression, it can include groups (indicated by parantheses) which can then be referred to in the REPLACE text as \1, \2 etc.
  • {INTERNAL writedir} – This is a special pattern that will match TextTest's own temporary paths for the test. Sometimes your application will write out absolute paths, which will naturally be relative to the temporary directory where the test is run. These will then produce different text every time. This syntax is mostly to save you the bother of producing an exact regular expression to match these paths.
For example:
[run_dependent_text]
output:Process ID{WORD 3}
output:[0-9][0-9]:[0-9][0-9]{LINES 3}
errors:{LINE 1}
my_file:Machine name{->}End of Machines Section
my_file:{INTERNAL writedir}{REPLACE <texttest write dir}
Filtering the order of certain lines
There is also the config file dictionary entry “unordered_text” which works in a similar way and supports a similar syntax to “run_dependent_text”. In this case the matching text is not removed, but assumed to appear in random order. It is therefore sorted alphabetically and placed in a section of its own at the end of the filtered file, so that the contents are asserted to be as before but the order in which things occur is not important.
Multiple-OS testing: forcing the filtering to occur
Sometimes you want to create a test suite that will run on multiple operating systems. TextTest's test suite for itself is one such example. The main problem you run into is that different operating systems use different characters for the end of lines, so that a “standard result” file from UNIX and a generated file from a test run on Windows will compare as different.
To fix this, set the “home_operating_system” config file entry. This string should be one of the strings Python uses to identify operating systems, i.e. “posix” or “nt” for the platforms supported by TextTest. It defaults to “any” which means don't worry about it.
If the entry is set and different from the running operating system, TextTest will perform the filter operation on all result files even if no filters are defined for them. This makes sure that all files are generated for the target platform and avoids false failures on line endings.
The Severity of Differences in particular Files
This is controlled by the dictionary entry “failure_severity”, and takes the form:
[failure_severity]
<texttest_name>:<severity>
<severity> here is a number, where 1 is the most severe and increasing the number means decreasing the severity. If the entry is not present, both “output” and “errors” files will be given severity 1, while everything else will have severity 99.
The severity has two effects on how TextTest behaves:
  1. When multiple files are found to be different, the difference is always reported in the dynamic GUI “details” column as a difference in the most “severe” file found to be different.
  2. If a severity 1 file is found to be different, the whole line will turn red, otherwise only the “details” column will turn red.
As an example, the test below has failed in “performance”, which is a severity 2 file. If the output had also been different, the whole line on the left would be red and the details would report “output different(+)”.




Last updated: 05 October 2012