Replay System for Java Swing Objects

sleep msecs

frame frameName

The purpose of this command is to find the toplevel windows of an application. From there, any other GUI Component of the application can be found.

command component ?args? 

• actionPerformed component [actionCommand command]
• adjustmentValueChanged component [-value n] [-type type]
  where the type is one of unitIncrement, unitDecrement, blockIncrement, blockDecrement
• componentHidden component
• componentMoved component
• componentResized component
• componentShown component
• focusGained component [-temporary boolean]
• focusLost component [-temporary boolean]
• itemStateChanged component
• keyPressed component [-keyChar char [-modifiers modifiers]
  where the modifiers is a tcl list with possible elements Alt, Ctrl, Meta, Shift
• keyReleased component [-keyChar char] [-modifiers modifiers]
• keyTyped component [-keyChar char] [-modifiers modifiers]
• mouseClicked component [-x x] [-y y] [-clickCount count] [-modifiers modifiers]
• mousePressed component [-x x] [-y y] [-clickCount count] [-modifiers modifiers]
• mouseReleased component [-x x] [-y y] [-clickCount count] [-modifiers modifiers]
• mouseDragged component
• mouseMoved component
• windowClosed component
• windowClosing component
• windowDeiconified component
• windowIconified component
• windowOpened component

This sequence can be performed within some SwingComponents by methods which perform semantic actions. These include

• AbstractButton.doClick()
  This draws the button pressed and then released, generates an ActionEvent and delivers it. This works for all subclasses of AbstractButton, such as JButton, JCheckBox, JRadioButton, JToggleButton, JMenuItem.
• JScrollBar.setValue(int)
  This sets the value and moves the ScrollBar.
• JList.setSelectedIndex(int)
  This sets the selection and highlights it.

Care must be taken when generating events: for example, a MouseEvent needs to have the x, y coordinates specified. If the component is a different size to what is expected, then these coordinates may be outside of the component. This can often happen when a recording is made of an application and a replay is made under different circumstances.

The Swing event delivery model ensures that an event delivered to a Container will be forwarded to a child if the child is at the right coordinates for the event. So events can be sent directly to the component that will receive them, or to a parent container

import java.awt.*;
import java.awt.event.*;
import com.sun.java.swing.*;

public class SwingButton extends JFrame 
                         implements ActionListener{

    public static void main(String argv[]) {
	new SwingButton().setVisible(true);
    }

    public SwingButton() {
	JButton btn = new JButton("Press me");
	btn.addActionListener(this);
	getContentPane().add("Center", btn);
	pack();
    }

    public void actionPerformed(ActionEvent e) {
	System.out.println("Button pressed: " + e.toString());
    }
}

The following tcl script will press and release the button in various ways:

java::load jnewmarch.replay.PlayTcl

set args [java::new {java.lang.String[]} {0} {}]
java::call SwingButton main $args

sleep 3000

# get the toplevel JFrame, frame0
set frame [frame frame0]

# find the contentsPane container
set pane [$frame getContentPane]

# get the Components it contains
set components [$pane getComponents]

# find the first object, which should be the JButton
set btn [$components get 0]

# the above steps could have been collapsed into
# set btn [[[[frame frame0] getContentsPane] getComponents] get 0]

# press and release the button with delays between
# press and release, by creating events
mousePressed $btn -x 47 -y 38
sleep 1000
mouseReleased $btn -x 47 -y 38
mouseClicked $btn -x 47 -y 38

sleep 3000
# a more robust way ensures that we are
# in middle of the button
set width [$btn getWidth]
set midWidth [expr $width / 2]
set height [$btn getHeight]
set midHeight [expr $height / 2]
mousePressed $btn -x $midWidth -y $midHeight
sleep 1000
mouseReleased $btn -x $midWidth -y $midHeight
mouseClicked $btn -x $midWidth -y $midHeight

sleep 3000
# now do the same action semantically
$btn doClick

import java.awt.*;
import java.awt.event.*;
import com.sun.java.swing.*;
import com.sun.java.swing.event.*;

public class SwingList extends JFrame 
                       implements ListSelectionListener{

    public static void main(String argv[]) {
	new SwingList().setVisible(true);
    }

    public SwingList() {
	String[] elmts = {"one", "two", "three", "four",
			  "five", "six", "seven", "eight",
	                  "nine", "ten", "eleven", "twelve",
	                  "thirteen", "fourteen", "fifteen",
	                  "sixteen", "seventeen", "eighteen",
	                  "nineteen", "twenty"};
	JScrollPane scrollPane = new JScrollPane();
	JList list = new JList(elmts);
	scrollPane.getViewport().setView(list);
	list.addListSelectionListener(this);
	getContentPane().add(scrollPane, "Center");
	setSize(100, 200);
    }

    public void valueChanged(ListSelectionEvent e) {
	System.out.println("Item selected: " + e.toString());
    }
}

Using input events to select the first item, click the scrollbar to page down and then select the last item can be done by

java::load jnewmarch.replay.PlayTcl

set args [java::new {java.lang.String[]} {0} {}]
java::call SwingList main $args

# get the toplevel JFrame
set frame [frame frame0]

# select the first item
mousePressed $frame -x 18 -y 34
sleep 1000
mouseReleased $frame -x 18 -y 34
mouseClicked $frame -x 18 -y 34

sleep 3000
# move down one block in the scrollbar
mouseMoved $frame -x 87 -y 179
mousePressed $frame -x 87 -y 179
sleep 1000
mouseReleased $frame -x 87 -y 179
mouseClicked $frame -x 87 -y 179

sleep 3000
# select the last item
mouseMoved $frame -x 44 -y 187
mousePressed $frame -x 44 -y 187
sleep 1000
mouseReleased $frame -x 44 -y 187
mouseClicked $frame -x 44 -y 187

This can also be driven using semantic methods. Not all of the desired methods are part of the Swing set, but in this case we can write tcl code to fill the gap.

# find the contentsPane container
set pane [$frame getContentPane]

# get the Components it contains
set components [$pane getComponents]

# find the first object, which should be the ScrollPane
set scrollPane [$components get 0]

# find the JList
set list [[$scrollPane getViewport] getView]

# find the JScrollBar
set scrollBar [$scrollPane getVerticalScrollBar]


# first let's move back up a block increment
# there isn't a blockIncrement method, so we define 
# one in tcl

proc blockDecrement {scrollBar} {
    set currValue [$scrollBar getValue]

    # get the value "1" for vertical vs horizontal
    # can't ignore direction since this method is
    # overridden in ScrollPane
    set direction [java::field com.sun.java.swing.SwingConstants VERTICAL]
    set blockIncr [$scrollBar getBlockIncrement $direction]

    set newValue [expr $currValue - $blockIncr]
    set minValue [$scrollBar getMinimum]
    if {$newValue < $minValue} {
	set newValue $minValue
    }
    $scrollBar setValue $newValue
}

blockDecrement $scrollBar
sleep 1000

# choose the sixth item
$list setSelectedIndex 5

import java.awt.*;
import com.sun.java.swing.*;

public class SwingText extends JFrame {

    public static void main(String argv[]) {
	new SwingText().setVisible(true);
    }

    public SwingText() {
	JTextArea text = new JTextArea();
	getContentPane().add("Center", text);
	setSize(300, 100);
    }
}

Using input events we can enter text using keyTyped and select text using mouseDragged

java::load jnewmarch.replay.PlayTcl

set args [java::new {java.lang.String[]} {0} {}]
java::call SwingText main $args

set frame [frame frame0]

set pane [$frame getContentPane]

# get the Components it contains
set components [$pane getComponents]

# find the first object, which should be the JTextArea
set text [$components get 0]

sleep 3000
# set focus to the text
mouseMoved $text -x 20 -y 40
mousePressed $text -x 20 -y 40
mouseReleased $text -x 20 -y 40
mouseClicked $text -x 20 -y 40

# add some chars "abcDEF"
sleep 2000
keyTyped $text -keyChar a
sleep 500
keyTyped $text -keyChar b
sleep 500
keyTyped $text -keyChar c

# we don't need to set Shift for keyTyped
sleep 500
keyTyped $text -keyChar D
sleep 500
keyTyped $text -keyChar E
sleep 500
keyTyped $text -keyChar F

sleep 1000
# select a piece of text - slowly
mousePressed $text -x 8 -y 5
mouseDragged $text -x 14 -y 5
sleep 100
mouseDragged $text -x 20 -y 5
sleep 100
mouseDragged $text -x 25 -y 5
mouseReleased $text -x 25 -y 5

# and replace it
sleep 500
keyTyped $text -keyChar x
sleep 500
keyTyped $text -keyChar y
sleep 500
keyTyped $text -keyChar z