TextTest is about testing at the system level, and it is expected that you are trying to run your system in as similar a way as possible to how it is used in practice. It is, however, more important to have maintainable, repeatable tests that are free of global side effects. Sometimes it is necessary to disable some part of the system, or fake some behaviour that cannot easily be triggered for real in a repeatable way. This is referred to as “mocking out” a subsystem for testing purposes.

(All the following mechanisms involve the test data mechanism : it's probably a good idea to read that document first if you aren't familiar with how it works)

The problem can here be broken down into 4 sub-categories, of which TextTest offers direct help with 3:

When TextTest creates the sandbox directory on running the test, and populates it with test data, it also makes sure to insert that directory at the start of the PATH, PYTHONPATH and CLASSPATH variables. That makes it possible to provide executable programs that will be run instead of their real versions via the normal test data mechanism (“link_test_path”). Then you just need to place such executable test data in the appropriate place in the permanent test structure, just as you would with more passive test data.

Likewise, you can provide a Python module or a Java class, which will then be imported instead of the real version at the appropriate moment.

For Python programs, though, there is an alternative, which is usually preferable. This is to provide a file called "testcustomize.py". The idea of this is to be similar to Python's "sitecustomize.py", i.e. to provide some Python code that will be called on interpreter startup just for that test or test suites (it can be provided anywhere in the hierarchy, like anything else). The advantages of this is that it's easy to change the behaviour of several modules from a single location, it's easier to "monkey patch" individual functions rather than entire modules, and because it isn't dependent on PYTHONPATH, it works to manipulate also the behaviour of builtin modules. (But note also the Python interception mechanism, see below)

The above mechanism for executable programs is powerful but can be overkill. It's also not very easy to steer in the case where a program is called many times and needs to behave differently on each occasion. It can be better and easier to simply ask TextTest to take over and “record” what the program does. This is known as the command-line traffic interception mechanism.

To enable this, add the name of the program concerned to the config file entry “collect_traffic”. For each test for which you want to mock out this program, go to the “Running” tab and select "Record All" for the "Traffic Files" radio button. TextTest will then create its own fake version of the program and place it in the temporary write directory as if it were test data, as above.

When called, this program will send the command line it was given back to TextTest via a socket. TextTest will then execute it, and record to a new definition file called “traffic.<app>” what the command line given was, what was returned on standard output and standard error and what the exit status was.

When the test is next run without the above box checked, this traffic file will be used instead of the real program. As before, the command line is captured and sent to TextTest via a socket. This time, however, it will look up the given command in the traffic file, and for the closest matching command line recorded, will return the standard output, standard error and exit status via the socket, which in turn will be relayed back to your system as if the real program had run.

It's then easy to fake certain conditions by simply editing this traffic file by hand, if desired

For example, suppose you wish to test a Linux program that can print documents. Clearly, the last thing you want is for paper to be spewing out of the printer every time the test is run. So, we set the test up so that it will capture the print command line, run it for real once and see that the paper looks right, and ever afterwards trust that so long as we continue to submit that same print command, the system is behaving correctly. Subsequent runs of the test will not talk to the printer for real but will merely verify the command.

So, on Linux we are running the "lpr" program to do this. We therefore tell TextTest to intercept this program:

collect_traffic:lpr

We then run the test with the record flag checked (or -rectraffic on the command line) as described above. This will produce paper for real which we can check looks right, and a new file in the test called "traffic.<app>" This has a standard format defined by TextTest and will look something like this

<-CMD:lpr -PthePrinter graph.ps
->OUT:Printing with args -PthePrinter graph.ps

The first two characters of each line indicate the direction of the communication. "<-" says that this was created by the program: "->" that it was received from the external system in response. The next three letters are a code for the type of communication. Here "CMD" signifies a system call via the command line, while "OUT" signifies a response on standard output. For command-line responses, you can also get "ERR" for responses on standard error and "EXC" for a non-zero exit code. After the colon you get the exact command line plus arguments, or the full response as appropriate.

Assuming we like what we saw, we can save this file along with the others. When we then rerun the test without the record flag checked, a new traffic file will be generated from the "lpr" commands that the system creates this time, with the responses generated by looking in the saved file and providing the appropriate match, or a best-fit match if the exact command cannot be found in the file. In this way we can control changes to the generated command lines without needing to call "lpr" for real or write any test code ourselves.

Clearly there is also plenty of scope for creating tests for error conditions such as the printer being out of paper. We only need to create such conditions once and then capture them. If we are confident of how "lpr" will behave under such circumstances we can also just edit the traffic file responses by hand, for example to return an error message and an exit code, which saves us the trouble of trying to simulate it at all.

Additionally, TextTest will scan the command line for arguments that appear to be files or directories (it will take anything that is an absolute path or a pre-existing relative path) and will store any changes to those files made while the process runs. If the test is saved these will then appear in the test file view under "Externally Edited Files". When it is run without the "record" flag these file edits will then be reproduced as if the real program had run.

By default it will only monitor these files while the program in question is running: however you can tell TextTest that it may start background processes that will edit the files, and in this case it will check the files before and after every subsequent received command. To do this, use the "asynchronous" key when adding it to "collect_traffic", which is in fact a dictionary although you don't need to know that unless using this feature.

For example, suppose our system under test updates some files via a version-control system, say CVS. We put the ones that we want for the test in the repository, and tell TextTest to capture interaction with CVS:

collect_traffic:cvs

As before, we run with the record flag and get a traffic file generated. It looks something like this:

<-CMD:cvs update -dP /path/to/my/checkout
->FIL:checkout
->OUT:U subdir/myfile.txt
->ERR:cvs update: Updating .
cvs update: Updating subdir

Note that besides the OUT and ERR response mentioned before, we also have a FIL response, which indicates an edit of a file or directory with the given name. When we save this, the file "subdir/myfile.txt" which CVS updated for us will be stored and can be viewed (or edited) under "Externally Edited Files". We can then even run this test on a machine without CVS installed, as its role will be played by TextTest producing the response from this file and the program itself will be none the wiser that the real CVS hasn't actually been called.

Subsequent edits to the same file or directory will also be handled, in this case they will be referred to for example as

->FIL:checkout.edit_2

Sometimes environment variables need to be provided for such programs. As they are run directly from TextTest's "traffic server" they don't automatically inherit any environment your program may have set up. In this case you should set the “collect_traffic_environment” dictionary config setting, with the keys being the program names as provided for “collect_traffic” and the values being the names of the environment variables. In this case these environment variables with their values will also be sent to TextTest and will be part of the information recorded.

For example, if our system sets CVSROOT (which CVS uses to find its central repository) before calling CVS we will need to tell TextTest this. We do this via

[collect_traffic_environment]
cvs:CVSROOT
[end]

which will also affect the traffic file. It will now instead start with the line

<-CMD:env 'CVSROOT=/path/to/cvs' cvs update -dP /path/to/my/checkout

providing a means to make sure our program is providing it to the CVS call correctly. If environment variables were unset by the SUT this would also be recorded via "--unset" options to "env".

(Note that this command isn't actually what TextTest executes, which of course would not work on Windows, it is just a representation of what it does which coincides with a legal UNIX command line. This mechanism is portable.)

In a similar way, if your program changes the current working directory for the program it calls, this will be captured and recorded by TextTest. In this case you don't need to do anything to configure it. For example your program might call CVS in a different but equivalent way to do the update, and the above call could equally end up looking like this:

<-CMD:cd /path/to/my/checkout; env 'CVSROOT=/path/to/cvs' cvs update -dP

Again this isn't what is executed internally: it is a representation only to allow easy comparison with future calls.

If your system under test is written in Python you can make use of a variation of this mechanism to intercept and replay the interaction with particular modules in a similar way to that described above. Examples would be things like "smtplib", "xmlrpclib" or "urllib" which may refer to resources that aren't always available where you want to run the tests, or which may cause undesirable global side effects.

To enable this, add the name of the module concerned to the config file entry “collect_traffic_python”. For each test for which you want to mock out this program, go to the “Running” tab and select "Record All" for the "Traffic Files" radio button as above. TextTest will then create its own "sitecustomize.py" file which will be loaded when your program starts, and manipulate "sys.meta_path" so that its own version of the modules given will be loaded instead of the real ones. These will then communicate with the traffic server in a similar way to the "collect_traffic" mechanism. (Note that this works internally a bit differently to how it did in versions prior to 3.20 : and it now works for builtin modules as it isn't based on changing PYTHONPATH)

For example, suppose our Python script is designed to send an email under certain circumstances. Clearly we don't want emails to be sent for real every time the test is run. In a similar way to the printing example, we can check it is sent for real once, capture the interaction, and then monitor future changes to that interaction.

We start by intercepting the Python module for sending email in our config file:

collect_traffic_python:smtplib

We then run with the "record" flag set and check that an appopriate-looking email arrives. We also get a traffic file as before, looking something like this:

<-PYT:import smtplib
<-PYT:smtplib.SMTP()
->RET:Instance('SMTP', 'smtp1')
<-PYT:smtp1.connect('machine.site.com')
->RET:(220, 'machine.site.com ESMTP Sendmail; Tue, 9 Feb 2010 14:32:54 +0100')
<-PYT:smtp1.sendmail('me@localhost', ['tom', 'dick', 'harry'], '''From: me@localhost
To: tom,dick,harry
Subject: Hi Guys!

I love you all!
''')
->RET:{}
<-PYT:smtp1.quit()

This provides the full email interaction and contents. The "PYT" communications represent calls made to the module by the system, while the "RET" ones are the responses provided. When a basic Python object, like a string or a list, is returned, it is referred to via its textual representation, i.e. via "repr". When an object is returned, as in when we construct a smtplib.SMTP object here, it is assigned a numeric name ("smtp1" here) and is referred to in the response as "Instance('SMTP', 'smtp1')". All future interaction with such an object is also intercepted as shown here.

Naturally we can then run this test and just verify that the smtplib interaction is the same, or make judgements on differences in the contents of the email, without needing to actually send emails for real every time. As before, it is also easy to simulate conditions by editing the file by hand. An added bonus here is that it is of course not very difficult to transform this file into a valid Python script, which can be very useful for extracting simple example code from your own code when you are unsure of how you are supposed to call a third-party library correctly.

Exceptions are also handled seamlessly. For example, if the SMTP server above could not be found, we will simply get something like

<-PYT:import smtplib
<-PYT:smtplib.SMTP()
->RET:Instance('SMTP', 'smtp1')
<-PYT:smtp1.connect('no_such_server@nowhere')
->RET:raise socket.gaierror("(-2, 'Name or service not known')")

If the exception is itself defined in the intercepted module, it will be referred in a similar way to the SMTP object above, i.e.

->RET:raise Instance('MyException', 'myexception1')

In additional to intercepting entire Python modules, you can also intercept and replay individual function calls. A good example is the current date (datetime.date.today() in Python) so that you can test code that depends on it without needing to write any code to fake what it does. You do this in the same way as above, i.e. using "collect_traffic_python". To intercept this call you would therefore do as follows. (Note that this format is changed since TextTest 3.19, when it was originally introduced. Also note that it now works on Windows, which it didn't in 3.19)

collect_traffic_python:datetime.date.today

This would produce a traffic file that looked something like

<-PYT:datetime.date.today()
->RET:datetime.date(2010, 5, 12)

and all other usage of the datetime module would not be intercepted. It would create you a test that behaved as though "today" was always 12th May 2010, saving you the trouble of figuring out how to fake it or how to manage test data that needed to refer to dates within a certain timeframe of it.

Note that any usage of the "datetime" module other than calls to "datetime.date.today" would just behave as normal and not be intercepted.

Note also that it does not currently work to provide the name for a bound method here: it must be a module-level function or attribute, static method or class name that is intercepted. Bound methods will hopefully be supported in future.

If you intercept something quite low-level, such as "os.getpid" you run into the issue that this may well be called by the standard library itself, or by development tools like PyUseCase or coverage.py. This obviously leads to far more calls being intercepted and recorded than you actually want.

TextTest therefore makes sure that any calls made by the standard library or by its own interceptor programs are not intercepted and recorded. This does mean that you need to intercept exactly the right thing. For example it will not work to intercept "subprocess.Popen" if your program actually calls "subprocess.call", even though the latter calls the former internally.

To tell it to block calls made from other places, you use the config file setting "collect_traffic_python_ignore_callers", which is a list of modules or directories from where calls should not be intercepted. For example, this will intercept "os.getpid" except when it is called from any module in a directory called "coverage" or from the "usecase" module, assuming our tests might use PyUseCase and coverage.py to extract information.

collect_traffic_python:os.getpid

collect_traffic_python_ignore_callers:coverage
collect_traffic_python_ignore_callers:usecase

By default each such request will be handled by a separate thread internally in TextTest's "traffic server" so that concurrent calls can be handled correctly without deadlocking. Some modules (e.g. Windows COM, GUI-related stuff such as tkMessageBox in Tkinter) however do not appreciate successive calls being made from threads that aren't the main thread, or from threads which are not the thread where the previous call was made.

In this case it's necessary to enforce a single-threaded mode for TextTest's "traffic server". This can be done by setting "collect_traffic_use_threads:false" in your config file which will force the traffic server to operate everything on a single thread. With any moderately complex Python module interception it's usually a good idea to set this.

The “traffic interception” mechanism can also be used for this purpose. This is enabled by setting

collect_traffic_client_server:true

in your config file. Though in this case you'll also need to make some changes to your system under test.

As above, TextTest considers itself to be recording such traffic when the "Traffic Files" switch is set to "Record", and replaying such traffic when a “traffic.<app>” file already exists. In these circumstances it sets up its own “traffic server” and sets the environment variable TEXTTEST_MIM_SERVER (MIM stands for “Man in the Middle”) to <host:port> for this place.

On testing a client-server system you probably need to write a wrapper script that can for example start server, start client connecting to server, close client, shutdown server. To use this traffic-recording mechanism you should modify this script such that the client will instead connect to the host and port given by TEXTTEST_MIM_SERVER, instead of that given by its own server. You should also modify it so that on discovering the host and port where the real server is running, this information should be sent to TextTest's MIM server in the format <some_message> <host:port>. TextTest will then parse this and know where the server is.

When your client sends a request it will go to TextTest instead. TextTest will record it as a client request in the traffic file, and forward it to the server. The server will then reply, which will be recorded as a server reply, and forwarded back to the client. In this way a complete log of the communication can be built up. It will look something like this:

<-CLI:Here is a string
->SRV:Length was 17
<-CLI:Here is a longer string
->SRV:Length was 24
<-CLI:terminate
->SRV:Exiting...

The format is similar to before, with "CLI" referring to client requests and "SRV" to server responses. This example is a "string-length" server where we send it strings and it tells us how long they are.

You can then replay this either as client or server. When running the server with a traffic file already in place, TextTest will read the file in order. It will itself send the client traffic in the order it is written down, and each time a server reply is present it will suspend and wait for such a reply. Replaying the client is much the same, except that TextTest suspends initially and waits for contact from the client before doing anything.

This may be easier to understand by example. There is a nice little fake client-server system tested in this way as part of the self-tests, under TestSelf/TrafficInterception/ClientServer/TargetApp. It is based around the string-length server shown above.