|
Documentation for 3.11.1
Faking it with TextTest
How to avoid running your whole system for real
TextTest is about testing at the system level, and it is
expected that you are trying to run your system in as similar a
way as possible to how it is used in practice. It is, however,
more important to have maintainable, repeatable tests that are
free of global side effects. Sometimes it is necessary to
disable some part of the system, or fake some behaviour that
cannot easily be triggered for real in a repeatable way. This is
referred to as “mocking out” a subsystem for testing
purposes.
(All the following mechanisms involve the test
data mechanism : it's probably a good idea to read that
document first if you aren't familiar with how it works) The problem can here be broken down into 4 sub-categories, of
which TextTest offers direct help with 3:
- System calls Somewhere your
program makes a system call to a “third-party”
command-line program. (You might have written it too, but it is
external to what you're trying to test). For whatever reason
the behaviour of this program is hard to control in a
repeatable way. One a example would be a build script which
updated the current source from your version control system.
Clearly you don't want to do this for real every time you run
the test. There are two methods available here: one of which is
based on writing your own version of the
program to do something predictable and another is based on
TextTest observing and reproducing
what the program actually does once, and then re-creating
those conditions for you.
- Built-in modules. Somewhere
your program uses a standard utility whose behaviour is hard to
control or simulatae. For example, say your program should
print a warning if the GUI library has too old a version. You
don't want to have to install an old version of in order to
test this. If you've written your program in Python or Java
help is at hand: you can simply write
your own version of the module (Python) or class (Java)
- Databases. These are good at global side effects
and changes in them are hard to reverse. The recommended
approach here is to find and use a file-based database which
can quickly be started from a file and shut down when the test
is done. In this way you can avoid having to try and shut
things down by hand. Recommended are Firebird
and MySQL for example,
while Oracle and
PostgreSQL are less
good at this. Once you have the file defining the database you
can simply define it as editable
test data and TextTest doesn't need to know that a database
is present.
- Distributed systems with
plain-text messaging. If you have a distributed
system with many components, it can be impractical to start and
stop the whole thing every time you run the tests. It can be
useful to have a way to define component tests from system
tests: i.e. to create tests running the whole system that can
nevertheless be run with the other parts of the system mocked
out.
When TextTest creates the t emporary
write directory on running the test, and populates it with
test data, it also makes sure to insert that directory at the
start of the PATH, PYTHONPATH and CLASSPATH variables. That
makes it possible to provide executable programs that will be
run instead of their real versions via the normal t est
data mechanism (“link_test_path”). Likewise, you
can provide a Python module or a Java class, which will then be
imported instead of the real version at the appropriate moment.
Then you just need to place such executable/importable test data
in the appropriate place in the permanent test structure, just
as you would with more passive test data. The above mechanism for executable programs is powerful but
can be overkill. It's also not very easy to steer in the case
where a program is called many times and needs to behave
differently on each occasion. It can be better and easier to
simply ask TextTest to take over and “record” what
the program does. This is known as the command-line traffic
interception mechanism.
To enable this, add the name of the program concerned to the
config file list entry “collect_traffic”. For each
test for which you want to mock out this program, go to the
Advanced tab under “Running” and check the “(Re-)
record command-line traffic” box. TextTest will then
create its own fake version of the program and place it in the
temporary write directory as if it were test data, as above.
When called, this program will send the command line it was
given back to TextTest via a socket. TextTest will then execute
it, and record to a new definition file called “traffic.<app>”
what the command line given was, what was returned on standard
output and standard error and what the exit status was.
Additionally,
TextTest will scan the command line for arguments that appear to be files or directories (it will
take anything that is an absolute path or a pre-existing relative path) and will store any
changes to those files made while the process runs. If the test is saved these will then
appear in the test file view under "Externally Edited Files".
When the test is next run without the above box checked, this
traffic file will be used instead of the real program. As
before, the command line is captured and sent to TextTest via a
socket. This time, however, it will look up the given command in
the traffic file, and for the closest matching command line
recorded, will return the standard output, standard error and
exit status via the socket, which in turn will be relayed back
to your system as if the real program had run. Any file edits that were
stored will also be reproduced at this point.
It's then easy to fake certain conditions by simply editing
this traffic file by hand, if desired
Sometimes environment variables need to be provided for such
programs. As they are run directly from TextTest they don't
automatically inherit any environment your program may have set
up. In this case you should set the
“collect_traffic_environment” dictionary config
setting, with the keys being the program names as provided for
“collect_traffic” and the values being the names of
the environment variables. In this case these environment
variables with their values will also be sent to TextTest and
will be part of the information recorded.
The “traffic interception” mechanism can also be
used for this purpose. Here it is less a matter of configuring
TextTest and more of configuring your own application. As above,
TextTest considers itself to be recording such traffic when the
“(Re-) record traffic” switch is enabled, and
replaying such traffic when a “traffic.<app>”
file already exists. In these circumstances it sets up its own
“traffic server” and sets the environment variable
TEXTTEST_MIM_SERVER (MIM stands for “Man in the Middle”)
to <host:port> for this place.
On testing a client-server system you probably need to write
a wrapper script that can for example start server, start client
connecing to server, close client, shutdown server. To use this
traffic-recording mechanism you should modify this script such
that the client will instead connect to the host and port given
by TEXTTEST_MIM_SERVER, instead of that given by its own server.
You should also modify it so that on discovering the host and
port where the real server is running, this information should
be sent to TextTest's MIM server in the format <some_message>
<host:port>. TextTest will then parse this and know where
the server is.
When your client sends a request it will go to TextTest
instead. TextTest will record it as a client request in the
traffic file, and forward it to the server. The server will then
reply, which will be recorded as a server reply, and forwarded
back to the client. In this way a complete log of the
communication can be built up.
You can then replay this either as client or server. When
running the server with a traffic file already in place,
TextTest will read the file in order. It will itself send the
client traffic in the order it is written down, and each time a
server reply is present it will suspend and wait for such a
reply. Replaying the client is much the same, except that
TextTest suspends initially and waits for contact from the
client before doing anything.
This may be easier to understand by example. There is a nice
little fake client-server system tested in this way as part of
the self-tests, under
TestSelf/TrafficInterception/ClientServer/TargetApp.
|