TextTest was originally built to test C, C++ and Python programs, and it isn't always obvious how to use it to test programs written in Java. This article hopes to give some pointers.

You have a java program, that is, a class with a main method, and you want to test it with TextTest. You run this on the command line like this:

java com.mycompany.MyClass

So you start TextTest with the --new option so you can start creating a new test suite for it. TextTest then lets you name the application, say "myapp", and decide where to store the tests. There is an additional field for a Java class name. Enter "com.mycompany.MyClass" in this field, which will grey out the executable so you don't actually need to choose a file.

If your program is a jar file, just locate it using the "executable" file chooser and leave the "Java Class name" blank.

If you program is of any size larger than "Hello World", you're going to also want to set the classpath. The easiest way to fix this is to put the correct classpath into the $CLASSPATH environment variable. TextTest can help you with this as described here , it will look for a file "environment.myapp" and put these values into the environment of your system under test. So it might look a bit like this:

# environment.myapp
CLASSPATH:c:\workspace\myproject\build\eclipse;c:\workspace\myproject\dep-lib\some_jar.jar

So now hopefully TextTest will be able to run your java program, and you can use all the usual mechanisms to give command line arguments to your program, copy input files into the sandbox directory, and collect output files for comparison.

Sometimes you want to pass arguments to java itself rather than your program. These can be System Properties that your application needs, or things like initial and maximum heap size. For example, you want to call your program like this:

java -Xms256m -Xmx1024m -DmailServerPort=1642 com.mycompany.MyClass

The way to do this is using an "interpreter_options.myapp" file containing

-Xms256m -Xmx1024m -DmailServerPort=1642

These can then be varied throughout the test suite structure as described in the test suite guide.

In general Java properties in arbitrary properties files can be set according to the properties file mechanism.

Maintaining a classpath by hand in the "environment.myapp" file, and keeping it in sync with your IDE becomes a hassle. This is where you probably want to use Ant to generate it.

If you're using Ant to build your Java program, you probably have an entry a bit like this to define the classpath to use:

  <path id="full-classpath">
    <pathelement path="${build.path}/eclipse" />
    <pathelement path="${build.classpath}" />
    <fileset dir="dep-lib">
      <include name="**/*.jar" />
    </fileset>
  </path>

In other words, the classpath includes your eclipse build directory, the folder where ant has just generated class files, and all the jars in your "dep-lib" folder.

The trick is to get ant to generate the classpath.myapp file and put it where texttest expects it. So to this end, create a template classpath file like this:

# environment.myapp.template
CLASSPATH:@CLASSPATH@

Then in your ant build.xml you can write something like:

    <property name="runclasspath" refid="full-classpath" />
    <copy file="${basedir}/texttest/${texttest.appname}/environment.${texttest.appname}.template"
        toFile="${basedir}/texttest/${texttest.appname}/environment.${texttest.appname}"
        overwrite="true">
      <filterset>
        <filter token="CLASSPATH" value="${runclasspath}" />
      </filterset>
    </copy>

(${basedir}/texttest is my $TEXTTEST_HOME in this case). Then Ant will generate the environment.myapp file for you with the correct classpath.

The next thing you're going to want to do with ant is to get it to run your TextTest tests unattended. If you're using a Continuous Integration server like Hudson, you will want it to run your tests there. There is currently no actual ant task for TextTest (again, there is an enhancement request). For the moment, you can still do it, it is just a little verbose in your ant build.xml file.

In order to run TextTest in batch mode, you need some specific settings in your config file. You may not want these settings there the rest of the time, and you probably want Ant to control what they contain. This makes it useful to have Ant generate an extra config file, with extra settings in. We can then tell TextTest to use this as our "personal configuration file" by setting the environment variable "$TEXTTEST_PERSONAL_CONFIG" as described here. To do this, we create a template config file, perhaps like this:

# config.hudson.template
[batch_junit_format]
hudson:true

[batch_junit_folder]
hudson:@TEST_OUTPUT_FOLDER@

TextTest will use this config to let it run a batch run called "hudson". We use ant to fill in the @TEST_OUTPUT_FOLDER@ so that the junitreport task will later be able to pick up the results.

In our build.xml we end up with two targets, something like this:

# build.properties
texttest.appname=myapp
test-output.dir=test-output # where texttest should write junit format results
test.classes.path=classes # where ant has put class files for this app
python.install.dir=C:/python # where python is installed
texttest.install.dir=C:/texttest # where texttest is installed
test-report.dir=test-report # where final junit report should be written
# build.xml
  <target name="runtests" description="Run tests">
    <!-- Delete results of previous test run -->
    <delete failonerror="false">
        <fileset dir="${test-output.dir}/${texttest.appname}">
          <include name="*.xml" />
        </fileset>
    </delete>
	<!-- use template config file to create extra config file for batch run -->
    <mkdir dir="${basedir}/texttest_rundir" />
    <copy file="${basedir}/texttest.config.template"
        toFile="${test.classes.path}/config"
        overwrite="true">
      <filterset>
        <filter token="TEST_OUTPUT_FOLDER" value="${test-output.dir}" />
      </filterset>
    </copy>
    <!-- execute texttest batch run and set exit code property to 1 if there are failing tests -->
    <exec executable="${python.install.dir}/python" resultproperty="texttest_exitcode">
    	<arg line="${texttest.install.dir}/bin/texttest.py -b hudson -a ${texttest.appname}"/>
	    <env key="TEXTTEST_HOME" value="${basedir}/test/texttest" />
	    <env key="TEXTTEST_PERSONAL_CONFIG" value="${test.classes.path}" />
	    <env key="TEXTTEST_TMP" value="${basedir}/texttest_rundir" />
    </exec>
  </target>  
  
  <target name="generate-test-report">
    <mkdir dir="${test-report.dir}" />
    <junitreport todir="${test-report.dir}">
      <fileset dir="${test-output.dir}">
        <include name="${texttest.appname}/*.xml" />
      </fileset>
      <report format="noframes" todir="${test-report.dir}" />
    </junitreport>
    
    <fail message="TextTest reported failed tests">
      <condition>
        <isfailure code="${texttest_exitcode}"/>
      </condition>
    </fail>
    
  </target>

When you put this project onto Hudson, it's just a matter of pointing out to Hudson the folder where the junit report is being written (${test-report.dir}), and it will display pretty graphs of the results and links to summaries of failures. I assume other CI servers have similar mechanisms.

Sometimes it is useful to have TextTest run all of your tests, including unit tests. Sometimes you want to write half of the test in JUnit and have TextTest check the files produced. Anyway, it is not hard to get TextTest to run JUnit tests. Start by creating a new test suite with "texttest.py --new", and then write "org.junit.runner.JUnitCore" as the Java class name. Then have your command line options in each TextTest test contain the name of the JUnit test case:

com.mycompany.MyTest

Hopefully that gives you some pointers as to what is possible when using TextTest to test a java program. For questions and comments, please join the TextTest mailing list.