1

Exploratory
Testing: A Myth or a Blessing?
STC 2013 Crossing the Chasm from Assurance to Confirmation

By:

Niti Chaturvedi and Pramod Jadhav ([email protected], [email protected])

ADP Pvt Ltd, Building # 3, S.P Infocity, Hadpasar, Pune

 2

Abstract
When I started my career of Software Testing, All I had was a product and a Timeline.
Moreover my first defect was very interesting scenario and unusual (but not unrealistic)
usage sequences. Did I go through any checklists? Did I follow any step by step Test
Cases? Did I have the steps documented? And many more questions. Flooded with
questions in my mind, Answers to each of the question were “NO”. It was just a user
perspective. Later over a period of time this became a practice. A methodology termed as
"Exploratory Testing”. Going by the name EXPLORE. It is an effective and efficient method
of testing.

Exploratory testing develops concurrent test design, test execution, test logging and
learning within time-boxes and consists of test objectives .It is one of the ideal ways that
combined the experience of testers with a controlled approach to testing where
specifications or documentation are either missing or inadequate and where there is severe
time pressure. In this way exploratory testing maximizes the amount of testing that can be
achieved within a limited time frame, using test objectives to maintain focus on the most
important areas. And thus its one of the powerful approach can be used in agile
methodology.

In this paper we will be sharing how exploratory testing is effective, creative and
productive. It will be best suited for maintenance or legacy product. Exploratory testing
could be easily incorporated into agile methodology.

1. Introduction

The Real Game is not on the field plan, not on the practice ground. The real play starts
when the score board displays (‘Score - 00:00’). Tighten the knots of the shoes, eyes on the
field. The Half time is about to start, deep breath to recall the strategies. Team A and Team
B warm up time.

The strategies for a Football game are broadly classified into three: Defensive Strategy:
Man-to-man, Coverage shells (cover 2, cover 3, etc.), Zone blitz, Offensive Strategy, Team
Strategy. Each of these has a definitive approach. The Strategy is set. Whistle loud and
clear all around the ground!! I am standing at the peak point of the triangle formation
 3

chasing the ball and my brain is full of thoughts. I need to be smart with an ‘ART’ to pass
the ball on the right time at the right site. Fractions of second delay in passing the ball can
cause disaster at the score board. But Rather than holding the ball until the goal post and
trying to escape from the attackers, it is better to ‘Pass it on’. A lot of controlled, patient
and alert mindset is expected during the game. Players may lose temper which can directly
impact the team when the refry shoots a red card. It results into a disaster when a team of
11 turns 10 and risk would be even high if the forward and goal keeper are at gun point.
Strategies and or Approaches are to be executed during the game at the ground. Likewise
the procedure and guidelines are determined but during the execution of the
application/module the real goal comes in the picture. At the game play the new risks and
thoughts flow in the mind of players, they don’t just go by the set plan; Plans ought to
change with the changing risks, time, and position. What is worth noting is the fact though
the entire game is played per a plan; both the teams are constantly exploring ways to
achieve the end Goal.

Let’s see with simple and hypothetical example:

User Story: As user I want to register the candidate information so I can enrol his
information.

Solution Approach: Candidate Registration form with ‘Insert’, ‘Update’ and ‘delete’
Functionality, Text fields and captions of the fields, Mandatory fields ,Field length, Data
types etc.

Acceptance Criteria: User should be able to insert, update and delete candidate’s
information.

Test Case Design: Test case design and documented as per the IEEE 829 standards

When Enhancement delivered and client started using this feature, encountered a defect.

Problem statement: When a user click on Insert button, the Delete button also gets enabled
and if user clicks on the delete button, the page displayed undefined error.

Defects escaped the from software development cycle phase and retrospect can confront
the requirements, design and testing, But to my mind it wasn’t about any one. This is just
 4

because of client did exploration on feature provide by Software product development team
and he found unexpected behaviour. How the client did exploration? Or what was the
question on his mind during exploration? Or is there any extraordinary thoughts came on
his mind after looking into feature? Did client have better understanding about the feature
than software Product team?

Yes! Read it Right. Certainty and uncertainty is not our cup of Coffee. Only Assurance is the
real taste maker, the real beans which will add flavour to the results. I mean we are human,
born with brains and supporting organs to stimulate. We cannot create a bridge in the brain
to calculate the amount of testing. Tester’s mind should have only one expected result i.e.
Open to all the possibilities.

Assumption is like that application used by the skilled users with specific input and output.
This frames the problem like application is used by variety of users so called humans. And
the human has more way to interact with application.

To explore means not to tell other people what to do. To explore means to replace
commenting with observation. To explore means to watch, to listen, and to look.

In exploring we discover. In discovering we see. With seeing we are aware. With awareness
there is this potential to act without causing disorder.

Allow us to introduce to you the term you have been reading and ignoring most of the time
i.e. Exploratory Testing

2. What is Exploratory Testing

Exploratory thought term used in psychology which has goal to seeking truth on the basis of
considering multiple point of view and tries to anticipate all possible objection to, or flaws in
particular position.

This term was coined by social psychologist Jennifer Lerner and psychology professor Philip
Tetlock in the 2002 book Emerging Perspectives in Judgment and Decision Making.

The term coined by Cem Kaner in 1983. As he suggest the Exploratory Software Testing is
style of software testing that emphasizes the personal freedom and responsibility of the
individual tester to continually optimize the value of work by treating Test related learning,
Test Design, Test Execution and Test result interpretation as mutually supportive activities
that run in parallel throughout the project.

According to Cem Kaner & James Marcus Bach, exploratory testing is more a mindset or
"Way of thinking about testing" than a methodology. They also say that it crosses a
continuum from slightly exploratory to highly exploratory.

Myth: ‘Exploratory Testing’ is considered as an approach that comes into implementation

with a good amount of experience. Unless they are well verse with the product and domain
the Exploratory way remains dark. Or also the approach is considered to be a medium to
 5

omit the documentation part from the Testing. Exploratory Testing is just banging on the
keyboard.

Exploratory Testing is known as refined and more structured version of ad hoc testing. Ad
hoc testing is performed without a plan of action and any actions taken are not typically
documented. Ad hoc testing is an informal and improvisational approach to assessing the
viability of a product. It is non-methodical; ad hoc testing can miss flaws that would be
found in a more structured testing system. Testers may not have detailed knowledge of
product requirements. Ad hoc testing is also referred to as random testing and monkey
testing.

Exploratory Testing is an approach of mapping the testing thoughts, Ideas or questions on

the product, business requirement with simultaneous execution of these thoughts or ideas
so that we can verify the functionality of a product.

As soon as we think about the test; execute them immediately. Observe the result and log
it. It is like you Learn, think, observe and execute the Tests on the product. Exploratory
Testing differs from other style of testing approach. It involves the Critical observation that
helps a tester to understand the expected and unexpected part of software. The practitioner
of the Impromptus Testing approach strongly believes in learning and enhancing the
product, domain expertise. In brief exploratory testing imbibes these parameters to the
core: Learning and Execution. It is an interactive test process

Exploratory is one of the approaches to finding or seeking the defects in the application that
could tell us what the quality of the software is. Also we get to know how the software
works.

In Reality actual Software testing follows the combination of exploratory and

scripted testing. All the Testing technique can be performed in exploratory testing
approach.

Some group of people says exploratory testing approach is nothing but a quick testing.

Exploratory Testing is not a testing technique; it’s an approach toward the software testing.
Testing technique will provide the guideline to creating specific tests. Some of the testing
technique mention as below

a. Function testing
b. High volume automated testing
c. Security testing
d. Printer compatibility testing
e. User testing
f. Data flow testing
g. Build verification testing

We shall not cease from exploration, and the end of all our exploring will be to arrive where
we started and know the place for the first time By T.S. Eliot
 6

3. Proposed Procedure –

Suppose Test manager ask to document the test cases for any new enhancement of the
product. Test Manager has given this responsibility to different testers having the same
experience on product as well as on software testing. Test cases designed and documented
by both testers covered slightly different aspect of the product. People are always having
different view on product. Usability Imagination command on product is varying from person
to person.

The Procedure is mainly focused on approach used for exploratory testing. Now-a-days agile
terminology is relatively spreading speedily over in Software development industry. As Agile
reduced the junk of documentation traditionally required for software development process.
Exploratory Testing also attract to similar highway about reducing the documentation.
Reducing the documentation does not mean that Exploratory Testing has unstructured
format. This Procedure will guide you how to fit the exploratory testing in agile
methodology. Basic concept of being ‘Agile’ in a project’s scope is broken down into small
iterations of development; customers should get happier and happier as the feature set
becomes more complete. Customer interaction and feedback is fundamental to agile
software development.

Iterations: You can minimize risk by developing the software in short iterations. Each
iteration passes though full software development cycle. Iteration passes though a mini-
software development cycle in a fairly short amount of time (typically between two to four
weeks) executing the Plan-Do-Study-Act cycle.

Open collaboration: Also known as being customer-centric, you’ll want to get feedback
from the customer and, based on their feedback, move forward with iteration. Agile
methodology relies on customer direction and feedback; customers should get progressively
happier as the project progresses.

If possible, the customer should also reside in that work setting or at least be readily
available for feedback and guidance.

In each sprint the user story (divided part of an enhancement) come over with the feature
to be delivered. The ‘I’ in the Idea needs to test it. The defined Test Plan now needs to
prove itself by covering all parts of the module to be tested. It requires a real focused and
pin point thought processing at the given time to ensure that the software reaches its goal
i.e. to the Client. Just like the above mentioned example the module needs to pass on from
various stages simultaneously. Tester geared with an eye of a client can better take the
module to the goal and stand with the objective of the project and the company.

Best Practice suggest us that Every Agile Project must have customer representative.

Following are the key elements of Exploratory Testing approach that could be implemented
in “agile methodology”.

A. Internal Client Team (mostly Internal Software testing Team)

Every software development project that follows the ‘Agile’ methodology has a team which
serves as a dummy client to accept the deliverable as per the External Client requisites. The
 7

dummy client team is a shadow of the Clients. But the role of client team comes at the end
of the entire software development cycle in the development methodologies.

In our proposition we propose the Testers having the responsibility of an ‘Internal Client’
coordinating with the Business analyst or Client representative to get better on the
understanding of the requirement/ improvement in the Business knowledge thus resulting in
an evolution of a tester with the product having improved business ideas coming upfront in
the early stages of the development cycle. The Test idea/Ideas will play a defining role.
Thus we are proposing that every agile project must have at least one internal dummy
client that will represent the client. This is the first step toward exploratory testing.

What is the test Idea?

This topic will be covered in point C of this Paper.

B.QA Cook Book

First of all it is important for us to understand that ‘Exploratory testing’ doesn’t stop us from
documenting anything at any given level. This approach is to give us a new view towards
testing where learning and implementation go hand-in-hand.

We call the Procedure as “QA Cook Book”

The “QA Cook Book” plays a vital role in testing phases using exploratory approach. During
the phase of testing, software Tester prepares and executes the test ideas with the test
data. All the recording will happen on QA cook book simultaneously. The Beauty of the QA
Cookbook is that there are no specific rules or process to maintaining this document. It just
depends on the team to decide how they want to maintain the recording of Exploratory
testing in order to understand the progress and utilize the same for future references.

This is a recipe of testing that could yield the yummy tummy app (bug free). The more
your hands on to the kitchen the better you evolve as cook. Our USB of Software Testing
comes with an exploratory way!

C. Test Idea

Test idea is a brief statement about the ‘Test’ that needs to be tested. Test ideas are easily
understandable and support powerful test. It is a small independent entity of testing that
could be converted into an executable test with exact descriptions of inputs and expected
results. To generate a good Test idea that will fit into software /application / product testing
is a most difficult task. As End user operates the product in much different way so testing
should have be done with different test idea execution. Execution of these Test Ideas will be
recorded and maintained in QA cook book. Mostly test idea should be one liner.

There are so many ways to generate the good test idea; clicks of few are mentioned below:

How to generate the good test Ideas?

1. Brainstorm around the new enhancement or the application

2. Understand the purpose of the overall product.
3. Always consider the business compliance, penalties, or any other legal obligation.
 8

4. Map the Business knowledge / requirement to Product.

5. Most important thing that software can fail in many ways; only condition is that
tester needs to ask the question to product “what if” that will expose the behaviour
of internal, external, anticipated, unexpected, realistic failure.
6. Asking useful or relevant question to product so that you will get the formation that
you are seeking.
i. What
ii. Who
iii. When
iv. Why
v. Where
vi. How

E.g. Like in above mention example if the Tester ask the question to product like
“Why delete button is getting enable when I click on Insert Button? Or what will
happen when I directly click on delete Button?

7. Understand how end user will use the software at operation side?

D. The Flip Way

It is a twisted way of looking towards the software at test. Consider that the sprint is
supposed to be completed in 10days. And the tester A is currently testing the Module1 of
the planned sprint and Tester B is working on the Module2. The flip way internally divides
the sprint into two half times after the end of first half the Module1 will be tested by the
Tester A and the Module B by the Tester B. The trained exploratory testers at the end of the
sprint deliver the modules 1 & 2 with coverage of 100 % Client perspective.

4. Take Away ( Exploratory Testing is indeed a blessing )

Sea Divers dive into the sea to EXPLORE Precious stones, extinct species, and hidden
beauty and the world of possibilities .In similar fashion Exploratory testing practitioner’s
imagination fly and defect lying underneath are thrown side by when imagination slide on
the ship of mind. Exploratory testing gives you real-time control of risk and coverage. Agile
exploratory testing is valuable addition to the normal testing practices. Exploratory testing
practitioner will find more defect that developer or testing tool cannot. It is powerful
approach of testing could be used in maintenance or legacy product testing.
 9

5. References:

 How to Reduce the Cost of Software Testing

 By Matthew Heusser and Govind Kulkarni (eds)

 Handbook of Usability Testing: How to Plan, Design, and Conduct EffectiveTests,

Second Edition
 By Jeff Rubin and Dana Chisnell

 Agile Testing: How to Succeed in an Extreme Testing Environment

 By John Watkins

 Software Testing: An ISEB Foundation, Revised

 By Brian Hambling (ed) and et al.

 Software Testing: A Craftsman's Approach, Third Edition

 By Paul C. Jorgensen

 The Black Swan by Nassim Nicholas Taleb, Random House, 2007

 An Introduction to Scenario Testing by Cem Kaner

 Scenarios: The Art of Strategic Conversation by Kees van der Heijden, Royal
Dutch/Shell’s former

 head of scenario planning, Wiley, 1996

 Value Of Checklists by Cem Kaner, CAST 2008 presentation

 A tutorial in Exploratory Testing by Cem Kaner

 The Nature of Exploratory Testing by Cem Kaner

 Rapid software testing by James Bach

 Session Based Test Management by James Bach, STAR West 2000

 Exploratory Testing: Design, Execute, and Learn by Michael Bolton, QUEST 2008
Testing vs. Checking by Michael Bolton, Agile 2009
 Juha Itkonen, Mika V. Mantyla and Casper Lassenius, Defect Detection Efficiency:
Test Case Based.vs. Exploratory Testing, First Intl. Symposium on Empirical software
Eng. and Meas. IEEE Comp.Soc., 2007
 Lisa Crispin and Janet Gregory, Agile Testing: A Practical Guide for Testers and Agile
Teams, Addison-Wesley Professional, 2009
 James Bach, To repeat tests or not to repeat,
http://www.satisfice.com/blog/archives/24
 Cem Kaner, Exploratory Testing, Keynote address at QAI conference, Nov, 2006
 http://epf.eclipse.org/wikis/xp/xp/guidances/concepts/test-
ideas_list_9B7EE6E1.html
 10

About the Authors

Niti Chaturvedi is a passionate QA Analyst. She is working in an agile environment and having
experience of 2.5 yrs. She is being part of Testing SIG (Special Interest Group) and interested in learning
emerging concepts in testing and has qualified CSTE & ISTQB certifications. It is the first experience of a
paper presentation and would like to take this opportunity to share how the Exploratory Testing in Agile
environment is a key to deliver a quality Product.

Pramod Jadhav is a Consultant with Automatic Data Processing (ADP Pune). He has 8 years of
experience in IT industry. He has an affinity towards the Analytical and problem solving skill and believes
in continuous practice, improvement in process, learning an up-coming testing concept. Currently
following the agile Methodology and like to put into practice a new emerging testing concept. Primarily
focus on delivered the Quality product within budget and time of frame.