The aim of the workshop was to discuss different aspects of acceptance testing in a fairly high-level and tool-independent way. Running the workshop were myself, Brian Swan (author of Exactor) and Rick Mugridge (author of FitLibrary and a newly published book on acceptance testing). What unites TextTest+StoryText, Exactor and Fit+FitLibrary is the aim to support writing tests in the language of the business domain : something most of the myriad other tools out there are not concentrating on. The following text is something that all three presenters agreed was a fair representation of the discussion.

There was a growing realisation and consensus on the importance of acceptance tests as a vehicle for establishing communication between business types (“customers”) and the development team about requirements. That is, while producing maintainable executable tests is a major point of acceptance testing, it isn't the only one.

For business types to be involved at this level, it becomes a major advantage for the tests to be written in their language: the language of the business domain. This is in preference to the alternatives: tests being written in a programming language, or a pre-packaged language defined by the test tool. It was perhaps not a surprising conclusion given the common thread of the tools developed by the organisers! A further point that emerged, though, is that tools should focus on constraining this language as little as possible to avoid strait-jacketing the communication : few constraints will be useful in all domains. We didn't examine the actual constraints imposed by actual tools, though.

Highlighted by several participants was the fact that many in management seem to want to regard testing and development as entirely separate enterprises, and some tools encourage this view by suggesting that effective tests can be written without needing to touch the code or “disturb” the developers at all. In fact, such compartmentalisation was seen as dangerous and can lead to important conversations being skipped.

Another problem for XP teams with long experience of unit testing is the notion of “test first”. You write a fully executable unit test, see the bar go red, write the code and see the bar go green. Some expect a similar process with acceptance testing : that the customer will write a fully executable acceptance test, “throw it over the wall” and the developers will write the code to make it pass. The discussion emphatically rejected this view: writing an acceptance test is an iterative procedure between customer and developers, where the two “sides” understand each other more and more, with the final result being an executable test.

Many practitioners report that getting business types involved in writing acceptance tests is difficult: they do not see it as their job. Unrealistic expectations in terms of the tools placed in their hands and expectations they will do the whole thing themselves have probably contributed to such problems, as discussed above.

A few points were raised about how to mitigate this:

There seemed to be a consensus that TextTest's approach of using logging to verify behaviour was particularly suitable for legacy systems, because it is much less intrusive to the system under test. Log statements can be added to a legacy codebase in an exploratory manner without any risk of breaking it. Introducing any tool that aims to access the system under test via an API (i.e. nearly anything else) involves performing (maybe substantial) refactoring before tests can be put in place.

It can be tempting to focus on testing the code. In fact, large tracts of the code may be entirely unused. Far more useful is to study the current usage patterns of the system and concentrate on writing tests for those.

A related point is that it is essential to find out what is now understood to be correct behaviour of the legacy system – which is very likely to differ from its original specification. The users may be so used to the bugs in it that they want to preserve them! The focus at first needs to be on “behaviour preservation” rather than “correctness assertion”.

Once such things are in place, the next step depends on the aim of the enterprise. Is it to basically fix bugs or is major new functionality needed? In the first case, behaviour preserving tests may well be all you need. If major development is needed, a serious refactoring effort will probably be needed, and unit tests can be added along with that effort.

When creating unit tests, much effort is (or can be) invested in isolating units that can be tested alone. This generally involves creating “mocks”, “fakes” or “stubs” to replace parts of the system that are not interesting for the test currently under consideration. Similar considerations arise in acceptance testing, though the “requirement” rather than “code” focus seems to make the distinctions between “mocks”, “fakes” and “stubs” uninteresting. The assembled company chose to borrow the work “mock” as it sounds best, somehow.

So, the question became : what should you mock out in acceptance tests? What is a valid motivation for doing so? The answers were pretty varied and a consensus was not really reached. In general Brian and I seemed to answer that as little should be mocked out as was practically feasible, while Rick and some others were happy mocking a bit more.

Naturally, creating mock versions of things creates the potential for unfound errors. Exactly the reason, in fact, why unit tests can miss important problems. The more what you test differs from what you ship, the more potential there is for this problem.

Of course, there are situations in which mocking is unavoidable:

Then there are situations where the “liberals” might mock things and the “fundmentalists” might not...