.

Editor: James Bach, ST Labs, Sterling Plaza, Third Floor, 3535 128th Ave. SE, Bellevue, WA 98006; [email protected]; http://www.stlabs.com

Software
Realities
Pitfalls and
• Automating only easy-to-run tests.
Early in testing, a program might be
unable to pass complex test cases.
Later, though, simple automated

Strategies in
tests are weak in comparison to the
increasingly harsh testing done by a
skilled manual tester.

Automated
Problems with maintainability
Maintenance requirements don’t go
away just because your friendly auto-
mated tool vendor forgot to mention

Testing
them. Two recurring issues focused our
discussion at the February LAWST:

• When the program’s UI changes,

how much work do you have to do
to update the test scripts so they
Cem Kaner, kaner.com accurately reflect and test the pro-
gram?
• When the UI language changes (such
ccording to popular mythology, Problems with the basic paradigm as English to French), how hard is it

A people with little programming

experience can use GUI-level
regression test tools to quickly
and competently create exten-
sive black box test suites that are easy to
maintain. Development managers, the
story goes, can save time, money, and
Here is the basic paradigm:

1. Design a test case, then run it.

2. If the program fails the test, report a
bug. Start over after the bug is fixed.
3. If the program passes the test, auto-
mate it: Run the test again (either
to revise the scripts so that they accu-
rately reflect and test the program?

We need strategies we can count on to

deal with these issues. Capture-based
test-case creation and test scripting done
in a tester’s “spare time” are ad hoc
aggravation by replacing pesky testers from a script or with the aid of a cap- approaches, not working strategies.
with these tools. These myths are spread ture utility). Capture the screen out-
by tool vendors, by executives who don’t put at the end of the test. Save the test Beware of capture tools
understand testing, and even by testers case and the output. In your first programming course, you
and test managers who should (and 4. Next time, run the test case and com- probably learned not to code
sometimes do) know better. pare its output to the saved output. If
Though some efforts to use these the outputs match, pass the program. SET A = 2
tools have been successful, several have SET B = 3
failed miserably. This was the focus of a • Expense. This approach is expen- PRINT A+B
two-day meeting at which 13 experi- sive: It can take 10 times as long (or
enced testers discussed patterns of longer) to create, verify, and mini- Embedding constants in code is obvi-
success and failure in GUI-based mally document the automated test ously foolish. But when we use capture
automation. The Los Altos Workshop as it takes to create and run the test utilities, we create a script by capturing
on Software Testing (LAWST), the first once, by hand. Don’t automate tests an exact sequence of keystrokes, mouse
of its kind, was run by seasoned facili- you will rarely run. movements, or commands. These are
tator Brian Lawrence. (See the “LAWST • Delayed bugs. Because it takes so constants, just like 2 and 3. A slight
approach and participants” sidebar for much longer, this approach can also change to the program’s UI will invali-
more information.) Here I integrate increase the cost of finding and fix- date the script. Maintenance costs asso-
highlights of that meeting with my other ing bugs. If you spend time early in ciated with captured test cases are
testing experiences—look for more the testing process writing test unacceptable.
detail at this year’s Software Quality scripts instead of testing, you won’t Capture utilities can help you script
Week. find bugs until later, when they are tests by showing you how the test tool
more expensive to fix. interprets a manual test case. They are
PITFALLS • Low power tests. Will you find bugs dangerous but not useless.
Here are a few pitfalls in automating by automating only tests that the pro-
testing; James Bach lists plenty of others gram has passed? I’ve heard estimates Beware of spare-time scripting
in “Test Automation Snake Oil” (Win- that thorough automation yields only Test groups often try to automate in
dows Tech Journal, Oct. 1995). 6 to 30 percent of total bugs found. their spare time without a unifying plan.

114 Computer
 .

They design and code each test case inde- • understand the requirements, ify the title, column 2 the title typeface,
pendently. Different scripts often repeat • adopt an architecture that helps us column 3 the type size, and so on. A test
exact sequences of commands. This efficiently develop, integrate, and case specifies a combination of values of
approach is as fragile as capture/replay. maintain features and data, the many parameters. Create the matrix
• adopt and live with rules (stan- using a spreadsheet and execute test
STRATEGIES dards) shared by all developers, and cases using a script that reads one row
Automated regression tests can be very • be disciplined. (test case) at a time and runs a series of
effective. Here are some suggestions for miniature scripts, one per column, to set
developing an automated regression test Of all people, testers must realize how the value of each parameter.
strategy that works. important it is to follow a disciplined Suppose the type size specification
approach to software development. moves to a different dialog. With a test
Reset management expectations Without it, we should be prepared to fail matrix you’ll only have to change a few
Despite the belief that test automation as miserably as so many of the applica- lines of code, in the one miniature script
yields rapid payback, every LAWST par- tions we have tested. that handles the size. You will only have
ticipant agreed: Most benefits of GUI- to change these lines once; this change will
level regression automation in release N Data-driven architecture carry over to every test case in the spread-
will not be realized until the development Imagine testing a program that lets a sheet. This separation of code from data
of release N + 1. However, it is possible user create and print tables, manipulat- is tremendously efficient compared to
to realize benefits in release N, such as: ing, among other things modifying the script for each test case.
There are several other ways to set up
• There’s a big payoff in automating a • the face, size, and orientation (hor- a data-driven approach. For example,
suite of acceptance-into-testing izontal or vertical) of the table title; Brett Pettichord fills his spreadsheet with
(“smoke”) tests. You might run • the position of the title relative to lists of commands (“Success with Test
these 50 or 100 times during devel- the table (above, below, or beside); Automation,” Proc. 9th Int’l Software
opment of release N, dwarfing the • the use of graphics (of any size, Quality Week, 1996). Other testers use
cost of developing and maintaining either bitmap or vector) and their sequences of simple test cases or of
the tests. position relative to the table title; machine states.
• You can save time, reduce human • the thickness of the line used as the Another way to drive testing with data
error, and obtain good tracking by table’s bounding box; uses previously created documents.
automating configuration and com- • the number and sizes of rows and Imagine testing a word processor by feed-
patibility tests that you run against columns; ing it a thousand documents. For each
many devices or environments. • the font, face, size of the text in each document, the script makes the program
table cell; load the document and perform a sequence
Take advantage of opportunities for • the use of graphics in each cell; and of simple actions (such as printing).
near-term payback from automation, but • the paper size (letter, legal, A4) and A well-designed data-driven approach
be cautious when looking for short-term orientation of the printout (land- can make it easier for nonprogramming
gains. Cost-justify the automation of scape or portrait). test planners to specify their test cases,
additional test cases. because they can simply write them into
For longer term gains, develop scaf- These parameters are related because the matrix. It can also provide a set of
folding in release N to improve testing they operate on the same page at the tables that concisely show what test cases
breadth and efficiency in release N+1. same time. If the rows are too big, there’s are being run by the automation tool.
no room for graphics. If too many type-
Test automation is faces are used, the program might run Framework-based architecture
software development out of memory. This example cries out A framework isolates the application
Automation of software testing is just for testing the variables in combination, under test from test scripts by providing
like all of the other automation efforts but there are millions of combinations. a set of functions in a shared function
that software developers engage in— Imagine writing 100 scripts to test just library. Test script writers treat these
except that this time, the testers are writ- 100 potential combinations. If one ele- functions as commands of their test
ing the automation code. It is code, even ment of the interface should change—if tool’s programming language.
if the programming language is funky. type size specification moves from one For example, a framework might
Every test case is a really a feature. And dialog box to another—you might have include the function openfile(p),
every aspect of the underlying applica- to revise each script. which opens file p. It might operate by
tion (the one under test) is data. Now imagine working from a test pulling down the file menu, selecting the
As we’ve learned on so many other matrix, in which each row specifies a test Open command, copying the file name
software development projects, develop- case and each column a parameter set- to the file name field, and selecting the
ers must ting. For example, column 1 might spec- OK button to execute the operation. The

April 1997 115

.

LAWST Approach and Participants

LAWST is a two-day meeting that focuses on a difficult testing problem. Many
labs have partial solutions to these difficult problems. By pooling experience, we
try to make significant progress in a short time. Meetings are kept small and are
structured to encourage participation by each attendee. We will hold another
LAWST in June. As the organizer and cohost of LAWST, I’ll gladly share details have enough staff. Several automation
about the meeting structure if you want to set up workshops of your own. Contact projects have failed because testers tried
me at [email protected]. The participants of the first LAWST were Chris Agruss to create the ultimate, gotta-have-every-
(Autodesk), Tom Arnold (ST Labs), James Bach (ST Labs), Jim Brooks (Adobe), thing programming library. Management
Doug Hoffman (Software Quality Methods), Cem Kaner (kaner.com), Brian support (and some peoples’ jobs) ran out
Lawrence (Coyote Valley Software Consulting), Tom Lindemuth (Adobe), Brian before the framework was ready to use.
Marick (Testing Foundations), Noel Nyman (Microsoft), Brett Pettichord You must prioritize. You have to build
(Unison), Drew Pritsker (Pritsker Consulting), and Melora Svoboda (Electric your library over time.
Communities). Organizational affiliations are given for identification purposes
only. Participants’ views are their own, not the views of the companies listed. Staffing realities
Many excellent black box testers lack
programming experience. They provide
function might be richer than this, check- Define commands or features of the subject matter expertise or other experi-
ing whether file p was actually opened or tool’s language. Test-automation tools ence that few programmers can provide.
log the attempt to open the file. The func- come with a scripting language. You can They are indispensable to the testing
tion might pull up the File Open dialog add a layer of indirection by putting a effort. But you can’t expect them to write
by using a command shortcut instead of wrapper around each command (T. automation code competently. You need
navigating through the menu. Or maybe Arnold, Software Testing with Visual a strategy that doesn’t require everyone
the function can pass the file name and Test 4.0, IDG Books, 1996). A wrapper to write test code.
path to a single command from the pro- is a routine created around another func- Nonprogrammers can be well served
gram’s application programming inter- tion that might do nothing more than by data driven approaches that let them
face. The function’s definition might calling the wrapped function. You can develop test cases simply by entering test
change from week to week. The script- modify the wrapper to add or replace planning ideas into a spreadsheet.
writer doesn’t care, as long as open- functionality, to avoid a bug in the test Be cautious about using contractors to
file(x) opens file x. tool, or to take advantage of an update to implement automation. Develop these
Frameworks include several types of the scripting language. LAWST attendees skills in-house, using contractors as
functions. Here are some basic types. who had used this approach said it had trainers or to do the more routine work.
Define every feature of the applica- repeatedly paid for itself.
tion. You can write functions to select a Define small, conceptually unified tasks Other types of automation
menu choice, pull up a dialog, set a value done frequently. This is straightforward LAWST focused on GUI-level regres-
for a variable, or issue a command. If the code reuse, which is just as desirable in sion tools, so I focused on them in this
UI changes how one of these works, you test automation as in any other software article. However, we all felt that many
change how the function works. Any development. of our biggest successes involved
script that was written using the function Define larger, complex test cases used approaches driven by extensive collabo-
changes automatically when you compile in several tests. It may be desirable to ration with the programmers who were
or relink. encapsulate larger command sequences. writing the application.
Frameworks are essential for dealing However, there are risks in this, espe-
with owner-draw controls. An owner- cially if you overdo it. A very complex here is too much hype and wishful
draw control uses programmer-supplied
graphics commands to draw a dialog.
The test-automation tool does not know
sequence probably won’t be reused in
many test scripts, so it might not be
worth the labor required to generalize it,
T thinking surrounding automated
testing. These tools can create a dan-
gerously false illusion of testing cover-
what’s inside this window. How can the document it, and insert the error-check- age, create staff turnover, and focus your
test tool press a button or select an item ing code into it that you would expect of most skilled staff on test cases that yield
from a listbox when it doesn’t know they a competently written library function. relatively few bugs. While they can be
are there? Think about dealing consis- Also, the more complex the sequence, the very useful, they require investment,
tently with invisible UI elements when more likely it is to need maintenance planning, discipline, and a well-trained
you change video resolution. when the UI changes. Rarely used com- staff. ❖
At LAWST, we talked of kludges upon plex commands might dominate your
kludges to deal with issues like these. library’s maintenance costs.
Some people spend half of their automa- Define utility functions. For example,
tion development time working around create a function to log test results to disk Cem Kaner is an attorney, software
problems created by owner-draw con- in a standardized way. Each tool provides development consultant, and senior
trols. If you must contend with owner- a set of prebuilt utility functions. You author of Testing Computer Software.
draw controls, hiding kludges inside might not need many additional functions. He can be reached at PO Box 1200,
functions is probably your most urgent You can’t add all these commands to Santa Clara, CA 95052; kaner@kaner.
task in building a framework. your library at the same time. You don’t com

116 Computer