Documentation for 3.11.1
Guide to the TextTest Static GUI
Examining and maintaining the test suites
The static GUI is started by default unless you specified
otherwise in your config file. If you did, it can be started via the command line option "-gx".
As can be seen, the structure of the static GUI is similar to
that of the dynamic GUI, and tests can be viewed in much the
same way. On the left is a colour-coded tree view of the whole
test suite.
The above example shows the same test suite and the same test
as previously being viewed in the static GUI. The files can be
double clicked in a similar way, here they will invariably be
viewed with “view_program” (emacs/notepad by
default). Note this setting can also be configured per result
type as described here.
In addition, you can use this view to create new files of either of the three types,
use "Create File" from the popup menu and select the type of file you wish to create.
The static GUI can be used to creating new tests or test
suites. By right-clicking a test suite, and selecting "Add Test" or "Add Suite" from
the popup menu, you get a dialog that can be filled in and will result in a new test
under the chosen suite.
In order to run tests, you first need to select some tests to
run. You can do this by single-clicking tests in the test window
on the left. Use ctrl + left click to build up a multiple
selection one test at a time, shift + left click to select
everything between what is currently selected and the line
clicked, ctrl + A to select everything. Alternatively, you can
select tests according to search criteria using the “Select”
button and “Selection” tab on the right (see below
for details of what can be done).
At the top right is a tab called “Running” which
will have three sub-tabs. The tabs “Basic” and
“Advanced” can be used to configure a multitude of
things about how the tests will be run. At the start the default
options should be sufficient. (Note that the tabs are
essentially a graphical representation of all command line
options that can be given to the dynamic GUI)
Once you are happy with these, press “Run” (on
the toolbar or in one of the above tabs). This will start the
dynamic GUI on the selected tests.
On the right there is a “Selection” tab(it should be
visible when you start TextTest). This provides a simple search mechanism for
finding tests, useful when the test suite grows too large to
always select comfortably which tests you want to run via the
test tree view alone. When the “Select” button is pressed, all
tests will be selected which fulfil all of the criteria
specified by the text boxes in the “Select” tab. It
follows that if no filters are provided and “Select”
pressed, all tests will be selected.
Using the same criteria on the same tab, it is also
possible to filter the tests, so that instead of selecting tests that match
the criteria, TextTest will hide those that do not match the criteria. This
is often useful when you want to work with a subset of the test suite for some time.
In addition, the View menu contains actions to turn a selection into a filtering
by hiding all tests that aren't currently selected.
The respective frames for selection and filtering
contain various “modes” represented by radio buttons. These operate
independently of each other. “Discard”, the default, will ignore the current
selection or filtering. “Extend”
will keep the current selection or filtering and add to it. “Refine”
will match only tests that were already selected or shown and match the
search criteria, while “Exclude” will match only
test that were not already selected (and isn't currently implemented for filtering).
Note that the number of selected tests (and the total number
of tests, and the number of hidden tests) is displayed in the column header of the test view at
all times. The various selection criteria can also be tried out
from the command line, using the plugin
script “default.CountTest”.
The simplest filters focus on matching the names of tests and
the test suites they are in. The “Test Names Containing”
field (-t on the command line) will select all test cases which
have the indicated text as a substring of their names. If
instead a standard regular expression is used, all tests whose
name matches that expression will be selected..
In a similar way, the “Suite Names Containing”
field (-ts on the command line) provides a way to select entire
test suites based on a similar substring/regular expression
search. Note that the string matched is the whole path
of the test suite : test suites can contain other test suites.
Likewise it is possible to match on the name of the application,
which can be useful if several different ones are loaded into the GUI at the same time,
or also on the description of the test, using a similar substring/regular expression matching
to that described above. This can also be done on the command line via "-a" and "-desc" respectively.
Sometimes test suites contain different tests depending on
the version identifier. In
this case, fill in the “Tests for Version” filter to
select the tests applicable to a particular version. This is
filled automatically if the static GUI is itself started with a
version identifier. It is not generally useful to do this on the
command line - simply running with a version will have the same
effect.
You can also search for certain logged text in the result
files. This is done via the “Test-files containing”
filter (-grep on the command line). By default, this will search
in the file identified by the “log_file” config file
entry. If the “Test-file to search” filter is also
provided (-grepfile on the command line), that file will be
searched instead. This allows selecting all tests that exercise
a certain part of the system's functionality, for example.
If system resource usage testing
is enabled for CPU time, you can select tests based on how
much CPU time they are expected to consume. This is done via the
“Execution Time” filter (-r on the command line). A
single number will be interpreted as a maximum CPU time to
select. Two comma-separated numbers will be interpreted as a
minimum and a maximum. All times are in minutes. In addition,
you can use the format mm:ss, rather than needing to convert
times into a fraction of a minute, and can also use the
operators <,>,<= and >= to specify ranges of times
to include.
Sometimes it may be useful to define such a subselection of
the tests that you may wish to reuse. To do this, select “Save
Selection” from the “File” menu, which brings
up a file chooser dialog so you can choose a file to save in.
Note it has two different options, allowing you to specify that
either the exact tests currently selected are to be saved, or
the criteria which were used to select them. Whichever, a new
“filter file” is created, which can be selected
again via “Load Selection” in the same menu, and
also via the “Tests listed in file” tab under
“Selection”.
The differences between the two variants become apparent when
somebody tries to load this file. Loading an explicit list of
tests will probably be faster than re-selecting them according
to some criteria, but if new tests are added since the selection
was saved, it will naturally not pick up these tests.
By default, the static GUI files will be saved in a directory
called “filter_files” under the directory where your
config file is. The dynamic GUI will save them in a temporary
location which is removed when the static GUI is closed. These
locations are used to generate the drop-down list for the “Tests
listed in file” option, and are also those searched if -f
is provided on the command line. These locations can be extended
or replaced by defining the config file entry
“test_list_files_directory”.
It is also possible to define which tests
to run by default based on such filter files. The config file setting
"default_filter_file" will make sure that only the tests that match
the criteria in the given file (found via the path mechanism described above)
are included in the run. This is primarily useful for defining a
version
of the test suite. A closely related concept is also available in
batch mode via the setting "batch_filter_file".
It's fairly common that the files in a test suite
get changed outside of the static GUI, for example if they are version-controlled
and you update them (which cannot be done from the GUI - yet...). As they
are all plain text it is also easy to edit them in a text editor independently
of the GUI. There is therefore a "refresh" action in the Actions menu, which
will re-read the whole test suite, including the config file settings, from the files.
It doesn't take account of which tests are selected so it isn't possible to refresh
just individual tests right now.
The order of the test suites is primarily defined by the
testsuite.<app> files, unless automatic sorting is enabled
(see the guide
to files and directories) However, there are some quick ways
to sort the tests after the fact. By simply clicking on the
column header they can be sorted “transiently” (i.e.
nothing is saved in any files and the sort is gone if you
restart). You can also sort more permanently by selecting the
various options from the Edit menu, which also contains various
options for manual sorting by moving the selected tests up and
down. These options are also avaiable via the “Re-order”
submenu in the popup menu for the test window.
Note that by default, sorting a test suite does so
recursively (i.e. all contained test suites will also be
sorted). To disable this behaviour, set the config file entry
“sort_test_suites_recursively” to 0.
Note that you can also simply edit the testsuite file via the file view, and
re-ordering of the tests performed this way
will show up in the GUI without needing to restart it.
Anything that is written on standard error by a dynamic GUI run will
be placed in a popup dialog in the static GUI when the dynamic GUI is closed down. These usually
indicate a bug in TextTest but they can also indicate environmental problems, for example
GTK issues or shell startup problems. Hopefully these can usually be fixed but occasionally you
end up with environmental issues that cannot easily be fixed.
These popups can therefore be suppressed if desired. Simply
add a config file entry "suppress_stderr_popup" followed by a substring or regular expression
that matches the line that is being repeatedly printed. All lines that match these entries
will be filtered out before displaying a dialog, and of course if no lines are left no dialog
will be displayed.
|