|
Introducing TextTest and Acceptance Testing
There are hundreds of tools out there that aim to do
automated testing at a more or less high level. This is only a
brief survey of what we consider to be the most interesting ones
(other than our own) and how they compare to using the tools
available here.
It seems simplest to break the problem down into application
types, which have different considerations and in some cases
different tools.
Batch programs. Here I mean a tool
which essentially takes some input, performs some calculation
and produces some output without any interaction from a user
other than starting it. Such tools may not have a GUI, or the
GUI may be entirely focussed on setting up the input and
displaying the output, and as such testing the GUI can be
considered as a separate problem.
Rich-client GUIs.Here I
include anything using a standard GUI toolkit, and also
interactive console programs controlled via standard input
responses. Acceptance Testing them is viewed as testing their
“use cases” - modelling a series of GUI actions
from a user (workflow).
Web applications. Anything run over HTTP in a
standard web browser.
Here our candidate tool is TextTest
alone, with no need for simulating user input. The main
alternative I have come across is Fit
using ColumnFixture (or FitLibrary
using CalculateFixture, but these are not very different as far
as I can see). Here is an example: This is a very compact and user-friendly format for
specifying lots of tests for a small calculator. The first three
columns specify numeric inputs and the final column the expected
output. There is a limit to how big such a table can get though
before it ceases to be a nice expression format, both in terms
of numbers of relevant input and output values and their size
(very long strings will probably not read well in this format).
For a larger “calculator”, it is better to separate
each test and then TextTest seems a better candidate –
it's easy to keep all the details of the data out of the way
unless you really want to look at it.
While it may be relatively unusual that an entire system is
of this form, it is much more common that parts of it are. It
can be very useful to identify them and test them separately in
this way.
Here is a table comparing
the tools. Here our candidate tool is TextTest
with StoryText or JUseCase. Suddenly there are more
alternatives, both that try to make use of the GUI and that
bypass it and call into the system via an API instead. I have
picked out four that I regard as especially interesting.
MarathonMan
– a record/playback tool for Java Swing GUIs that was
originally built by two guys at Thoughtworks aiming to make
record/playback a bit more agile. Tests are recorded in Python
in terms of the GUI mechanics: rather than user intent. Useful
mainly as a comparison with other modern record/playback tools.
Naturally there are many other tools of this type, of which
Abbot is probably
the most popular.
Fit with ActionFixture –
uses HTML tables to define tests against an API (i.e. bypasses
the GUI). ActionFixture is essentially a simple “test
language” with four GUI-like keywords: (check, press,
select and enter) which calls into the system under test via an
API. Included mainly because it's popular and acts as a way to
introduce the following two rather more interesting
contributions.
Fit
with FitLibrary/DoFixture – FitLibrary is an
extension for Fit that aims to support testing in a domain
language (using a meta-language called DoFixture) written by
Rick Mugridge. While you could see ActionFixture (above) as a
meta-language, it is very clumsy compared to DoFixture and
consequently hard to believe that was its intention. FitLibrary
contains many other more minor improvements and extensions to
the core Fit.
Exactor –
another API-driven tool focussed entirely on workflow testing.
Written by Sean Hanly and Brian Swan at Exoftware. Also
primarily focussed on the creation of a domain language for
tests. Tests expressed in simple plain text files rather than
tables: tables are great for calculations but seem to just get
in the way for workflow. Additionally provides “standard
commands” that define a language for driving the GUI
directly. Some other interesting features lacked by both
variants of Fit, notably a means of refactoring the tests at
the customer level, pretty similar to our own GUI
shortcuts
There is also a table
comparing the features of these tools for workflow testing of
GUIs. It's now several years since we tried to test a web application
and it's become very clear that the state of the art has moved on hugely since then.
The previous advice, and our proposed approach with WebUseCase, have now been removed.
It might be possible to involve TextTest in web-testing somehow, in combination with
one of today's web simulators, but as far as I know nobody has attempted to do this.
Anyone trying to test a web application today needs to take a long
hard look at Selenium which seems to have
evolved into the tool of choice. Its javascript engine was a major leap forward in allowing
web-testing to actually make use of real browsers rather than bypass them. The biggest drawback
is the back-end, as there is no support for creation of a domain language for use-case style
scripts, which was our biggest aim with WebUseCase - although that was built on HtmlUnit. It seems like there is now a tool with
a similar aim, STIQ, built
as a domain language layer on top of Selenium.
|
