Documentation for 3.9.1
Running tests in parallel on a grid
When you have more than one machine at your disposal for
testing purposes, it is very beneficial to be able to utilise
all of them from the same test run. “Grid Engine”
software allows you to do this, so that tests can run in
parallel across a network. This greatly speeds up testing,
naturally, and means far more tests (or longer tests) can be run
with somebody waiting on the results.
TextTest's queuesystem configuration is enabled by setting
the config file entry “config_module” to
“queuesystem”. It operates against an abstract grid
engine, which is basically a Python API. Two implementations of
this are provided, for the free open source grid engine SGE
(which sadly doesn't work on Windows) and Platform
(which does, but costs money). You choose between these by
setting the config file entry “queue_system_module”
to “SGE” or “LSF”: it defaults to “SGE”.
By default, it will submit all tests to the grid engine. It
is still possible to run tests locally as with the default
configuration, you need to select the option “Run Tests
Locally” (-l on the command line). Internally, TextTest in
fact submits itself to the grid engine and runs a slave process
remotely, which runs the test in question and communicates the
result back to the master process via a socket.
As soon as each test finishes, the test will go green or red,
and results will be presented. Unlike the default configuration,
the tests will not naturally finish in order. Here is a sample
screenshot, using SGE:
tests have finished and gone green, but others are still running
and hence yellow. TextTest reports their state in the grid
engine (“RUN”) followed by the machine each is
running on in brackets.
As this functionality works with a different configuration
module, additional config file entries, running options and
support scripts are available, over and above those provided by
A grid engine will be configured to have a number of queues,
which will hold jobs (tests in our case) until a CPU becomes
available somewhere, and then dispatch them to that machine.
These queues also handle job priorities, it is generally
possible to set up several queues so that jobs from one will
cause jobs from the other to be suspended. In the case of
testing, it is often useful to prioritise jobs by how long they
are expected to take, so that a one-hour test can be suspended
to allow a five-second test to run. To find out more, read the
documentation of the grid engine of your choice.
As far as TextTest is concerned,
it must decide which queue to submit each test to. The procedure
is as follows:
- If the option “Request
SGE/LSF queue” has been filled in (-q on the command
line), that queue will be used.
- If the config file setting
“default_queue” has been set to something other
than its default value (“texttest_default”), that
queue will be used.
- If neither, the default queue
of the queue system will be used, supposing it has one.
It is often useful to write a
to modify this logic, for example to
introduce some mechanism to select queues based on expected time
Resources are used to specify properties of machines
where you wish your job to run. For example, you might want to
request a machine running a particular flavour of linux, or you
might want a machine with at least 2GB of memory. Such requests
are implemented by resources, in both grid engines. Naturally,
more details can again be had from the documentation of the
queue systems. LSF requests resources via “bsub -R”
on the command line, while SGE uses “qsub -l”.
TextTest must choose which
resources to request on the command line. The procedure here is
to request all of the resources as specified below:
- The value of the option
“Request SGE/LSF resource”, if it has been filled
in (-R on the command line)
- The value of the environment
variable QUEUE_SYSTEM_RESOURCE, if it has been set in an
- All resources as generated by
the performance_test_resource functionality, if enabled (see
TextTest will transfer the environment it has read from its
environment files to the slave processes it starts. This can
cause a problem if the remote machine is running on a different
platform. In this case it is necessary to ensure a full correct
login remotely. It has been found useful to support the start of
a subshell remotely to make this happen: correct logins are
often ensured in shell start-up scripts.
To this end, the “login_shell” entry is provided.
This defaults to “sh” (Bourne Shell) but can be
configured to be whichever shell startup contains your
The queuesystem configuration also provides some improvements
in default configuration's
functionality for comparing system resource usage
This is essentially in the area of the concept of a performance
. In the default configuration, tests are run
locally, so all we can do is see if our current machine is
enabled for performance testing. With a grid engine at our
disposal, we can actually request a performance machine for
The simplest way to do this is to check the “run on
performance machines only” box from the static GUI
(“-perf” on the command line). That will make sure
the grid engine requests that the test only run on such
It is also possible to say (for CPU time testing only) that
once tests take a certain amount of time they should always be
run on performance machines only (it is assumed that the
performance of the longest tests is generally the most
interesting.) This can be done via the setting
There is also an additional mechanism for specifying the
performance machines, which on SGE has to be used instead. The
config file setting 'performance_test_resource' allows you to
identify your performance machines via a grid engine resource,
for example to say “test performance on all Opteron250
machines”. This is generally easier than writing out a
long list of machines, and is compulsory with SGE for more than
one machine. With LSF, you can write out the machines as for the
default configuration if you want to.
The queuesystem configuration also provides an improvement to
the batch mode
unattended runs. This basically involves adding a horizon when
all remaining tests are killed off and reported as unfinished.
This will be done if it receives the signal SIGUSR2 on UNIX.
Both grid engines can be set up to send this signal at a
particular time themselves, which involves submitting the
TextTest batch run itself to the grid engine. In LSF, use “bsub
-t 8:00” to send SIGUSR2 to the job at 8am the next
morning. In SGE, use “qsub -notify”, and then call
“qdel” for the job at the allotted time: this will
also cause the signal to be sent.
Both LSF and SGE have mechanisms to send signals to jobs when
they exceed a certain time limit. It is possible to configure
the queues such that they send SIGXCPU if more than a certain
amount of CPU time has been consumed, or SIGUSR1/SIGUSR2 if too
much wallclock time is consumed.
TextTest will assume this meaning for these signals and
report accordingly. SIGXCPU is always assumed to mean a CPU
limit has been reached. SIGUSR2 is interpret to mean a kill
notification in SGE and a maximum wallclock time in LSF, while
SIGUSR1 is used the other way round in the two grid engines.
To configure SGE to play nicely with this, it's useful to set
a notify period of about 60 seconds when jobs are killed
(TextTest submits all jobs with the -notify flag). By default,
SIGUSR1 is also used to be a suspension notification in SGE,
which TextTest does not expect or handle. It's therefore
important to disable the NOTIFY_SUSP parameter in SGE if you
aren't going to get tests spuriously failing with RUNLIMIT
whenever they would be suspended:
Both grid engines have functionality for testing systems
which are themselves parallel, setting aside several CPUs for
the same test. TextTest integrates with this functionality also.
This is basically done via the environment variable
QUEUE_SYSTEM_PROCESSES, which says how many CPUs will be needed
for each test under that point in the test suite. In LSF this
basically translates to the “-n” option to “bsub”.
In SGE, you need to use an SGE parallel environment (read the
SGE docs!), this is specified via the config file entry
“parallel environment name”,
The performance machine functionality described above still
works here. In this case TextTest will ask the queue system for
all machines that have been used, and only if they are all
performance machines will performance be compared.