x Issues in Testing Java Applets

Issues in Testing Java Applets

Jan Newmarch
Information Science and Engineering
University of Canberra
jan@ise.canberra.edu.au

Abstract

Testing Java applets introduces a new level of variation in test environments, in that the applet must run within a variety of browsers. This paper looks at some of the issues arising from this, and reports on a test framework under development to answer these issues.

Introduction

Testing is an important part of software development, to detect bugs, improve software design and reduce usability problems. There are a variety of test techniques, and a large number of tools support testing in different circumstances.

Java [Java] is a relatively new language that can be used to build standalone applications. In addition, it can be used to build applets. Java applets are distributed programs that execute on a variety of platforms, usually embedded within a Web browser. This environment gives rise to a number of unique problems. This paper details some of these issues, and describes work in progress to build a tool to aid in testing applets.

Variable Environment

Traditional applications are built for a limited number of environments. A large number of errors are independent of their environment, such as falling off the end of an array, but a number depend on external factors, such as the precise behaviour of a Operating System call. To catch the environment-independent errors involves running a test suite on any one platform. Catching the environment-dependent errors involves building the application on every environment and running a test suite. It is quite common - at least in Unix, where there are innumerable small differences betwen versions of Unix - to have a build procedure that includes an explicit test phase for installation on each platform:
  ./Configure
  make
  make test
  make install

The Java language and core libraries are designed for a wide set of environments. Strictly, they are designed for one environment: the Java Virtual Machine [JVM], but this has been implemented on many platforms, in many different ways. The slogan is ``write once, run anywhere''. While true, there is often an addendum: ``write once, test everywhere''. There are sufficient differences between the Virtual Machine implementations and in the libraries to make this essential.

The so-called Java Pure initiative [PureJava] attempts to eliminate many programming practises that can result in variant behaviour (probably incorrect) on different platforms. However, it can only guarantee ``purity'', and this is not as strong as portability, let alone correctness.

Java applets run within browsers. In addition to the wide set of platforms and different Java Virtual Machines, the browser itself will add further variety to the environment. Some of the differences are

This widens the set of situations for which tests must be performed. In addition to ``test everywhere'' is ``test on every browser, in various user set configurations''.

Test Harness

Components of an application can be tested by building a custom test harness. For example, to test a stack a simple program can be written to exercise the class.

Applications have fewer entry points to test against. A common test method is to simulate user interaction in some way. For example, for applications with a graphical user interface, events can be simulated and placed on the event queue. Sophisticated techniques may be available for this: for example, the replayXt system of the author [replayXt] uses action's available in any Xt widget to drive simulated user events. Non-windows applications may be driven by systems such as expect, which will supply a pseudo-terminal and feed user actions into this, for consumption by the application [expect].

To test an applet, a suitable harness must be present. This must be able to load an applet and run a test script against it. The tool JavaStar runs a Java program around the applet, which is generated for that applet [JavaStar]. This Java program can be run across all platforms supporting the Java VM, but does not run within a browser. Therefore, it fails to test under different browser conditions.

The test harness needs to run within each browser.Therefore it needs to be an applet itself. This applet needs to load the applet to be tested and run it within the same environment.

Test Language

Many test tools use a scripting language in which the tests are written. For example, expect and replayXt scripts are written in tcl [tcl], whereas JavaStar scripts are written in Java. For standalone applications the choice of language is under the control of the tester since they tend to be more in charge of the environment. For applets, one can envision a more open testing situation, where any arbitrary user can run the test suite.

This places limitations on the scripting language. For example, the tcl test suite is written in tcl since it is known that it will be available. JavaScript uses Java since the Java VM must be available to test Java applications or applets. However, this is not the only possibility: any scripting language written in Java can be downloaded to run within any applet. Jacl [Jacl] is a tcl interpreter written in Java, and a tcl interpreter can be run from within any applet. In particular, it can be run from an applet implementing a test harness for other applets.

Location of Test Scripts

Applications that only live on one platform need only be tested on that platform. For applications that may be run on multiple platforms, it is common to include a test suite along with the distribution. For example, most GNU and tcl applications include a test suite.

Applets are loaded across the network into browsers, where they run. Test scripts could potentially exist on the browser side, but this would involve replication on every test site. Explicit replication of test code through mechanisms such as tar files would be fairly inflexible. Test code should exist on the same Web site as the applet. This can then either be loaded up to a browser either as a parameter to the test harness (which is an applet), or the test harness can upload a script file itself.

Locating tests on the server with the applet they are testing is a logical choice.

Security

Applets run within a ``sandbox''. This is a security model that severely restricts the capabilities of applets to access the local system. Excluded are means of reading from or writing to the local disk, or forking arbitrary local processes.

The exclusions are in fact more severe than this. If an applet harness needs to load an applet, then the tested applet can only come from the same site as the harness. The harness will need to load the code using a ClassLoader, and access to these is restricted. This is to prevent an applet using a class loader that breaks the sandbox model. In addition, a harness that generates user events and adds then to the tested applet's event queue will need access to that queue, and this is currently restricted. This is because there is presently only a single event queue for any application/applet, and if a browser such as HotJava allowed access to its event queue then buttons such as the ``File Save As..'' could be pressed - leading to access to the local file system.

Working within the security restrictions is an unavoidable issue. While many of these restrictions are fairly easy to manage, the issue of a single versus multiple event queues is not. In JDK 1.2 multiple event queues are promised, but have not appeared in the beta versions available so far. The capabilities security model introduced in Netscape 4 and refined in HotJava may provide an alternative mechanism, but at a security risk.

GUI Elements

Testing a graphical user application involves generating user events and placing them in the event queue. They may exist some more internal method, such as setting the selection item on a list widget. For example, there are systems such as XTest which can generate X events and inject them into the event queue for any X application. On the other hand, the Xt layer above X puts events through a complex translation mechanism which can be tapped into at various points. This is used by replayXt to call semantic-level actions for Xt widgets.

The original set of GUI elements for Java was the set of AWT objects. These are built above native code GUI objects. For example, the Java Button is built on top of the Motif XmPushButton on Unix platforms. These objects are virtually impossible to drive in any sensible way. Events are divided into two classes: input events (such as mouse click and keypress) and semantic events. Input events may be generated by an application, but are discarded before they reach the native objects [Newmarch]. Semantic events may be generated, and they may affect Java GUI objects such as Lists, but they never reach the underlying object either. There is no standardised mechanism for interacting with Java objects at the semantic level.

The AWT is being replaced with the Swing objects [Swing], written entirely in Java. Events generated and placed into the event queue do what one would hope - they affect internal state and external appearance of the GUI objects. This is all good stuff. It is not too hard to do this - provided you have access to the event queue.

Generating events for Swing objects has the same problems that it does in all window systems: (x,y) co-ordinates often have to be specified, and this locks you down to a particular geometry. In some situations this is not a problem: for example, a Button can be queried for its width and height and then co-ordinates can be calculated that place the event in the centre of the Button. However, some widgets, such as FileChooser have internal structure which is not made visible. FileChooser often has an OK button. The location of this button is dependant on a geometry manager that will place it in varying positions depending on the space it has available. There is currently no means of querying the widget for the location of any of its components, so this is unreliable.

There is another problem with event generation in Java. The Java event is not translated into a native event, and to some extent these still underly part of the GUI libraries. So it is possible to generate a Java event that corresponds to mouse moved, but the actual mouse will not be moved. This will be an issue for demonstration systems that rely on an observer tracking the motion of the mouse throughout the demonstration.

Some Swing objects have semantic methods such as doClick() on the Swing JButton. This redraws the object (several times) and calls all the action listeners. However, not all objects have a complete set of methods. For example, FileChooser has no method to click on the OK button. This particular object is not yet finalised in Swing 1.0, and we may hope for a complete set of semantic methods for all objects.

An Applet Test Harness

The points listed above have led to the following: The replayJava framework already addresses some of these issues [replayJava]. It is designed as a general framework to run Java applications and applets through a scripting language. The scripting language is tcl, and this is accessible to browsers because there is an interpreter written in Java. The framework can drive applications through the event queue or through semantic methods.

Driving applets by the event queue is dependant on a proper implementation of getEventQueue() for applets, that doesn't just attempt to return the the system event queue (which will often fail for security reasons). Using events on structured objects such as FileChooser requires knowledge of the location of internal components and methods to do this are not yet available. Driving applets by semantic methods requires a complete set of methods for all GUI objects.

Conclusion

Testing Java applets involves extra issues above testing standalone applications. These involve a wider set of environments, more careful choice of test harness and security. In addition, current deficiencies within the libraries mean that some objects are hard to drive in a robust way.

A tool is under development that can be used to test applications and applets. Deficiences in the Java libraries that are exposed by this are being reported to JavaSoft, hopefully to be fixed in later releases.

References

[expect]
Expect home page, http://expect.nist.gov
[Jacl]
Jacl, sunscript.sun.com
[Java]
Java home page, http://java.sun.com
[JavaStar]
SunTest home page, http://suntest.sun.com
[JVM]
T. Lindholm and F. Yellin, ``The JavaTM Virtual Machine Specification'' http://java.sun.com/docs/books/vmspec/html/VMSpecTOC.doc.html
[Newmarch]
J. Newmarch, The New Event Model in Maximum Java, SAMS, 1997
[PureJava]
PureJava home page, http://java.sun.com/100percent
[replayJava]
J. Newmarch.
[replayXt]
J. Newmarch, http://pandonia.canberra.edu.au/SW#replayXt
[Swing]
Swing Connection http://java.sun.com/products/jfc/swingdoc-current
[tcl]
Tcl home page, http://sunscript.sun.com

Jan Newmarch (http://pandonia.canberra.edu.au)
jan@ise.canberra.edu.au
Last modified: Wed Apr 8 22:31:08 EST 1998
Copyright ©Jan Newmarch