Exploratory Testing

Rediscover the art of exploratory testing

Ingo Philipp
www.tricentis.com
E2E Test Automation Load Testing Test Management Process Automation
@IngoPhilipp ‹› Glasses don't do the seeing, tools don't do the testing.

Just as glasses do not see, tools do not test.

Glasses help us to see, tools help us to test

James Bach
@IngoPhilipp ‹› Testing is not test cases just as recipes is not cooking.

Testing is no more about test cases

than astronomy is about telescopes

Ingo Philipp

∷ Inspired by Edsger Wybe Dijkstra & Michael Bolton

 ‹› Automation accelerates, intensifies, amplifies, often simplifies, and sometimes even enables testing.

If we’re testing badly, then automation can

help us to test badly at an accelerated rate

Michael Bolton

@vertaontest ‹› If you automate chaos, all you get is faster chaos.

 ‹› Confusing testing with automation is a dangerous thing.

Testing? There's only automation!

We automate everything Oops

Damn!
OK, there's testing, and testing
is probably more than just automation.
We're just not convinced that we really need it

∷ Inspired by Brendan Leonard

Test Case
 Passing
Test Case
 Passing
Test Case

@iamdevloper
© BBC
 Testing is exactly like
washing a pig. Because
it’s messy. It has no rules. No
clear beginning, middle, or end.
It’s kind of a pain in the ass,
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.

∷ Inspired by Luke Sullivan © BBC

 ‹› We actually wash a performing pig.

Testing is exactly like

washing a pig. Because
it’s messy. It has no rules. No
clear beginning, middle, or end.
It’s kind of a pain in the ass,
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.

∷ Inspired by Luke Sullivan

 ‹› We actually wash a performing pig.
That’s what we can know

Testing is exactly like

washing a pig. Because
it’s messy. It has no rules. No
clear beginning, middle, or end.
It’s kind of a pain in the ass,
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.

∷ Inspired by Luke Sullivan

 © Richard Bradshaw That’s what we can know

Problem

Testing is exactly like

washing a pig. Because That’s what we know

it’s messy. It has no rules. No

clear beginning, middle, or end. That’s
It’s kind of a pain in the ass, what we
check
and when you’re done you’re
not sure if the pig is really clean
or even why you were washing
a pig in the first place.
Knowledge Gap

∷ Inspired by Luke Sullivan

 © Richard Bradshaw That’s what we can know

Problem
The purpose of testing is to
close the knowledge gap That’s what we know

That’s
The goal is information, what we
not gratuitous automation check

Testing is and always will be

a search for information
Knowledge Gap

∷ Cem Kaner
 © www.wikipedia.org – How to use toilet paper?

Under Over

Confirmation Exploration
« Demonstrate your depth of knowledge » « Demonstrate your breadth of knowledge »
 Change Detector Problem Detector

Low Information Value High Information Value

Here we check what we have already learned Here we learn something new about software

Monitor Known Risks Analyze Potential Risks

Here we confirm what we already know Here we focus on the things we don’t know

Mechanical Testing Creative Testing

Here we process predesigned data in predefined steps Here we create new test ideas

Confirmation Exploration
« Demonstrate your depth of knowledge » « Demonstrate your breadth of knowledge »

∷ Bolton ‹› Kaner ‹› Bach

 Change Detector Problem Detector
Does this assertion pass or fail?

High Information Value

Here we learn something new about software

Analyze Potential Risks

Here we focus on the things we don’t know

Evaluate a product by applying algorithmic

Creative Testing
Here we create new test ideas
decision rules to specific observations of a product

Checking Exploration
« Requires Processing » « Demonstrate your breadth of knowledge »

∷ Bolton ‹› Kaner ‹› Bach

 Change Detector Problem Detector
Does this assertion pass or fail? Is there a problem here?

Evaluate a product by applying algorithmic Evaluate a product by learning about it

decision rules to specific observations of a product through exploration and experimentation

Checking Exploring
« Requires Processing » « Requires Thinking »

∷ Bolton ‹› Kaner ‹› Bach

 Change Detectors ‹› Monitor Known Risks
Problem Detector
Evaluate through Pay attention to Is there a problem here?
Instructions Deviations

Create Follow
Test Cases Procedure

Examine Factory
Requirements Process
Evaluate a product by learning about it
through exploration and experimentation

Checking Exploring
« Requires Processing » « Requires Thinking »

∷ Rich Rogers
 Change Detectors ‹› Monitor Known Risks Problem Detectors ‹› Analyze Potential Risks

Evaluate through Pay attention to Investigate through Pay attention to

Instructions Deviations Experiments Oracles

Create Follow Create Follow

Test Cases Procedure Test Ideas Clues

Examine Factory Examine Adaptive

Requirements Process Risks Investigation

Checking Exploring
« Requires Processing » « Requires Thinking »

∷ Rich Rogers
 Change Detectors ‹› Monitor Known Risks Problem Detectors ‹› Analyze Potential Risks

Creative Process
Evaluate through Pay attention to Investigate through Pay attention to
Instructions Deviations Experiments Oracles
Mechanical Process

Create Follow Create Follow

Test Cases Procedure Test Ideas Clues

Examine Factory Examine Adaptive

Requirements Process Risks Investigation

Checking Exploring
« Requires Processing » « Requires Thinking »

∷ Rich Rogers Robots Investigators

 ‹› If your test case is good-for-nothing, you are probably good-for-nothing.

Behind a test case, there's a test. Behind a test, there's a

test idea. Behind a test idea, there's a human tester

Ingo Philipp

∷ Just as you don't blame gravity for falling flat on your face, you shouldn't blame your tools for your poor test cases.
‹› Exploration isn’t something special in testing, it is central to testing, and so all true testing is exploratory.

Checking is a guest in the house of testing,

exploration is a permanent resident

Michael Bolton
 ‹› The coin of testing has two sides.

Checked
Machine & Human Checking

+
Agile Explored
Human Exploration
Testing Equation
=
Tested
Thorough Testing

∷ Elisabeth Hendrickson
Machine
Checking
Human
Checking
Checked
Machine & Human Checking

+
Exploratory
Testing Explored
Human Exploration

=
Manual
Testing Tested
Thorough Testing
‹› Think of AI-assisted testing, not AI-based testing.

∷ Picture by Shelly Palmer

 ‹› The beautiful thing about learning exploratory testing is that no one can take it away from you.

Exploratory testing is not a talent,

it’s a set of skills that can be learnt

Ingo Philipp

∷ Inspired by Edward de Bono

Technique Approach
Provides Systematic Procedure Provides Orientation
 1
2
3

Technique 4
Provides Systematic Procedure 5
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

2
3

Technique 4
Provides Systematic Procedure 5
∷ Jonathan Bach
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

2
3
4
5
Straightjacketed Imagination
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

Chartered
2
Uninterrupted
Reviewable 3
Session 4
5
∷ Jonathan Bach
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

2
3
Timebox 4
5
∷ Jonathan Bach
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

2
3
Charter 4
5
∷ Jonathan Bach
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

Requirements-Based Testing
Limit the scope to make it manageable

3
Scope 4
5
 Validation
Does our software meets the user's needs?
Session-Based Testing
Structure exploratory testing to allow large-scale implementations

Actual
Product Requirements-Based Testing
Limit the scope to make it manageable

3
You

Product
Description
Product
Idea 4

Verification
5
Does our software meet the specification?
 ‹› Use BDD to create a shared understanding.

Session-Based Testing
Structure exploratory testing to allow large-scale implementations

Requirements-Based Testing
Limit the scope to make it manageable
Nobody (e.g. product owner)
can make you (e.g. tester) think
BDD is the art of using 3
& feel as deeply as they do about
examples in conversations software. Be aware of this fact
to illustrate behavior to explore this gap.
4
Liz Keogh

5
∷ Inspired by Daniel Saint
 ‹› BDD is about understanding, not test case automation.

Session-Based Testing
Structure exploratory testing to allow large-scale implementations
Having
Conversations Requirements-Based Testing
Limit the scope to make it manageable
is more important than Nobody (e.g. product owner)
can make you (e.g. tester) think
Capturing 3
& feel as deeply as they do about
Conversations software. Be aware of this fact
to explore this gap.
is more important than
4
Automating
Conversations 5
∷ Liz Keogh ‹› CAST 2018 ∷ Inspired by Daniel Saint
 ‹› We don't just explore the unknown in our software.

Session-Based Testing
What Structure exploratory testing to allow large-scale implementations
product
owners
think Requirements-Based Testing
Limit the scope to make it manageable
What they
Nobody (e.g. product owner)
can put into can make you (e.g. tester) think
words
3
& feel as deeply as they do about
software. Be aware of this fact
What they
to explore this gap.
say to their
teams 4
What you usually
understand
5
∷ arcfly.blogspot.com ∷ Inspired by Daniel Saint
 R » Recent Session-Based Testing
What parts of the product changed recently? Structure exploratory testing to allow large-scale implementations

C » Core
What critical parts of the product must continue to work? Requirements-Based Testing
Limit the scope to make it manageable

R » Risky
What parts of the product are inherently risky?

3
C » Configuration
What parts of the product depend on environment settings?

R » Repaired 4
What parts of the product changed to address defects?

C » Chronic
What parts of the product chronically break?
5
∷ Karen Johnson
 S » Structure Session-Based Testing
Test what the product is made of. Structure exploratory testing to allow large-scale implementations

F » Function
Test what the product does. Requirements-Based Testing
Limit the scope to make it manageable

D » Data
Test what the product processes.

3
P » Platform
Test what the product depends upon.

O » Operations 4
Test how the product is used.

T » Time
Test how the product is affected by time.
5
∷ James Bach
 Supermodel
Tour
Session-Based Testing
Structure exploratory testing to allow large-scale implementations
Saboteur Museum
Tour Tour
Requirements-Based Testing
Limit the scope to make it manageable

Tour-Based Testing
Set concrete goals to provide a clear focus

Goals
FedEx
4
Tour
Money
Tour 5
Couch Potato
Tour
∷ James Whittaker
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

Requirements-Based Testing
Limit the scope to make it manageable

Tour-Based Testing
Set concrete goals to provide a clear focus

@speed

Quality 4

Quality is value to some person 5

∷ Jerry Weinberg
 Session-Based Testing
Quality is inherently subjective Structure exploratory testing to allow large-scale implementations

Different stakeholders Requirements-Based Testing

Limit the scope to make it manageable
will perceive the same product as
having different levels of quality
Tour-Based Testing
Set concrete goals to provide a clear focus

We must look for different

things for different stakeholders 4
We must diversify testing 5
∷ Jerry Weinberg
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

Requirements-Based Testing
Limit the scope to make it manageable

Tour-Based Testing
Set concrete goals to provide a clear focus

Polychrome Testing
Explore the product from different viewpoints to diversify testing

5
 Creative
Session-Based Testing
Structure exploratory testing to allow large-scale implementations
Control Critical

Requirements-Based Testing
Limit the scope to make it manageable

Tour-Based Testing
Set concrete goals to provide a clear focus

Polychrome Testing
Diversity Explore the product from different viewpoints to diversify testing

Positive Factual
5
Emotional
© Edward De Bono
 Session-Based Testing
Structure exploratory testing to allow large-scale implementations

Requirements-Based Testing
Limit the scope to make it manageable

Tour-Based Testing
Set concrete goals to provide a clear focus

Polychrome Testing
Ideas Explore the product from different viewpoints to diversify testing

Scenario-Based Testing
Capture each test idea to make it reviewable
‹› Testing is enactment of critical thinking about software to help people make better decisions.

Testing is not so much a thing you do,

it’s far more a way you think

Michael Bolton
 ?
Questions
The show is over. It’s your turn.