Exploratory Testing and the Planning Myth

James Bach, Satisfice, Inc.

[email protected]

www.satisfice.com

(540) 631-0600

I grant permission to make digital or hard copies of this work for personal or
classroom use, provided that (a) Copies are not made or distributed for profit
or commercial advantage, (b) Copies bear this notice and full citation on the
first page, and if you distribute the work in portions, the notice and citation
must appear on the first page of each portion. Abstracting with credit is
permitted. The proper citation for this work is Rapid Software Testing (course
notes, Fall 2002), www.testing-education.org, (c) Each page that you use from
this work must bear the notice "Copyright (c) James Bach,
[email protected]” or, if you modify the page, "Modified slide, originally
from James Bach", and (d) If a substantial portion of a course that you teach is
derived from these notes, advertisements of that course must include the
statement, "Partially based on materials provided by James Bach." To copy
otherwise, to republish or post on servers, or to distribute to lists requires
prior specific permission and a fee. Request permission to republish from
James Bach, [email protected].
Exploratory Testing and the Planning Myth
by James Bach, http://www.satisfice.com
This first appeared on http://www.StickyMinds.com as a column feature

My essay “What is Exploratory Testing?” provoked some interesting reactions. One came
from Yogita Sahoo, who agreed that ET can be useful, then asked “But can we get others
to agree on such an unplanned process?” Ms. Sahoo’s question raises some very
important issues, such as:

• What is “planning”?
• Is exploratory testing necessarily unplanned?
• Why might we need to do things that aren’t rigorously planned?
• How do we coordinate and support each other, if not with a plan?

Here’s my short answer: Exploratory testing, as I practice it, usually proceeds according
to a conscious plan. But not a rigorous plan. Rigor requires certainty and implies
completeness, but I perform exploratory testing precisely because there’s so much I don’t
know about the product and I know my testing can never be fully complete. Exploratory
testers coordinate with each other and gain support in the same way executives run large
companies: by talking with each other and building credibility over time. I want to say
it’s not rocket science, except that rocket scientists and all scientists use exploratory
tactics in their work.

Exploratory testing may well be planned…

The American Heritage Dictionary offers us four definitions of plan:

1) A scheme, program, or method worked out beforehand for the accomplishment of an

objective: a plan of attack.
2) A proposed or tentative project or course of action: had no plans for the evening.
3) A systematic arrangement of important parts; an outline or a sketch: the plan of a story.
4) A drawing or diagram made to scale showing the structure or arrangement of something.

Notice that a plan can be a plan without necessarily being a specific, rigorous, complete,
or recorded plan. Thus, one answer to the Ms. Sahoo’s question might be that good
exploratory testing is a planned activity, even though it’s not scripted in detail. I find it
useful to define a test plan as “the set of ideas that guide the test process”. This definition
is consistent with at least the first two American Heritage definitions. In order to guide
testing, the test plan must exist prior to the testing, but how long prior? What if it comes
into existence only moments before the testing? No problem. Nothing in the nature and
notion of planning prohibits simultaneous planning and execution. In football, when a
quarterback finds his receivers are too well covered, he may quickly make a new plan to
run with the ball himself. An instant later, he executes his plan. Thinking fast is no crime
against plankind.

…But, learning comes before the plan…

Still, I may be missing the point. Another meaning of the planning question could be
“How can we get others to agree on a process with such an open-ended and unpredictable
plan?” To address that issue it helps to imagine a very rigorous, repeatable activity,
perhaps playing a flawless piano concerto, or executing a computer program. Now ask
yourself, is there a plan in those activities? I would have to say yes. Now ask yourself,
what happened before there was a plan? I would have to say some kind of composing and
practicing process. If so, then was that prior process guided by the same rigorous sort of
plan? I suspect not. In fact, if we keep peeling back the layers of how things came to be,
we soon get to processes that obviously were not rigorously planned (at least not by
humans): our education, society, childhood, life, the creation of the universe, etc.

Anything explicitly planned must have arisen, at some point in history, from a less
explicitly planned activity. Exploratory testing is exactly the kind of thing that comes
before rigor and makes rigor possible. ET is primarily creative. It has value in terms of
the problems we find while doing it, but it also has value because it is a way to create
other kinds of tests. It’s a great way to learn about a product. And if I did want to make a
rigorous test plan, I’d better start exploring the thing I intend to rigorously test. That
gives us the answer to the planning question: an open-ended plan may be the only plan
available. If you already know for sure what the next test is that would be worth
executing, by all means follow that plan. Otherwise, you’ll have to figure that out for
yourself: and that’s an exploratory process.

You get someone to agree to such a process the same way you get someone to agree to
allow a house to be built before they try to live in it. If they understand the difference
between a house and an empty lot, and they want a house, and they see that there is no
house, they may well agree that someone ought to build one.

…and dealing with a complex world requires skill, above all.

Still another way to think about Ms. Sahoo’s question is that it’s a query about
coordinating people on a project and gaining support from management to do your work.
How can a relatively lightly planned activity be organized and managed? This is where
the myth of planning truly rears its head. The truth is, we organize and manage all kinds
of things, all over the place, all through our lives, without detailed plans. In the absence
of pre-scripted moves, what do we do? We pay attention. We react to the situation at
hand. We anticipate problems. Because we don’t know the right answers in advance, we
find ways to come up with the answers on the fly. Think about this the next time you’re
merging onto a busy freeway.

In The Effective Executive, Peter Drucker defines an executive as someone who manages
the value of his own time. Testers are like executives. We can train testers in a manner
similar to training executives: start small, provide coaching, learn from failure and build
on successes. I don’t send a tester on a complex exploratory mission until that tester has
gained credibility on the easier missions. The tester must show an ability to design a good
test strategy, execute good tests, find important problems, and report them in a
compelling way. Furthermore, the tester must be able to explain his work. Only under
those conditions do I come to trust that tester’s work.

Remember it’s the skill of the tester, the social system of the project team, and the testing
situation at any given moment that enables excellent exploratory testing. Don’t dwell too
much on planning. If you’re challenged to defend exploratory testing on the basis that it’s
not planned, reply that when the developers tell you where all the bugs are, precisely and
accurately, you will happily write a precise and accurate script that visits each one. Until
then you must spend some of your time in an unscripted search.