Combinatorial Black-Box Testing with Classification Trees

W14
Session
10/26/2016 3:00:00 PM
Combinatorial Black-Box Testing with
Classification Trees
Presented by:
Peter Kruse
Berner & Mattner Systemtechnik GmbH
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-‐268-‐8770 ·∙ 904-‐278-‐0524 - info@techwell.com - http://www.starcanada.techwell.com/

Peter Kruse
Berner & Mattner Systemtechnik GmbH
Software engineer and test consultant Peter Kruse works in the domain of
testing, including evolutionary testing and the classification tree method. He is an
experienced software developer and tester in the German automotive industry.
Peter's project experience includes hardware-in-the-loop testing, model driven-
development, ISO 26262, and evolutionary structural and functional testing.
Apart from combinatorial testing, Peter's focuses on improving the costs and
benefits in any test process and related test activity, striving to balance emerging
trends and ideas with approved methods.

COMBINATORIAL BLACK-BOX TESTING WITH
CLASSIFICATION TREES
PETER M. KRUSE

Test Process and Test Activities
Testmanagement, Testdokumentation
Test
Planning
Test
Design
Test
Evaluation
Test
Execution
Test
Implementation
Test Management / Test Documentation
e.g. Module Test, Integration Test, System Test
Specification Tested (Sub-)
System
Motivation: How to obtain and structure test goals and efforts –
in a reliable, standard-conform, reproducible way?

Test Process and Test Activities
Testmanagement, Testdokumentation
Test
Planning
Test
Design
Test
Evaluation
Test
Execution
Test
Implementation
Test Management / Test Documentation
Specification Tested (Sub-)
System
e.g. Module Test, Integration Test, System Test
Solution: Put emphasis here.
Motivation: How to obtain and structure test goals and efforts –
in a reliable, standard-conform, reproducible way?

Classification Tree Method
• Test Design Technique
• Systematic Process
• Graphical Representation of Test Problem
• Independent of Test Domain, Test Level and Test Object
• Input: (Functional) Specification of System under Test
• Output: Set of Tests to be Implemented
• Tool Support: TESTONA
• Recommended for Certified Tester
• Part of ISO 29119-4

Classification Tree Method Example
• Test object: Database Management System
Select a test object: decompose

Input domain
Determine input data space

Privileges Operation Access Method
Input domain
Identify relevant aspects (e.g.
from specification)
Determine input data space

Input domain
Classify the input data
space into classes

1
2
3
Input domain
Combine classes
into test cases
space into classes

Classification Tree Method – Steps
This is how TESTONA structures test goals:
reliable, standard-conform, and reproducible.
Combine classes into
test cases
Identify relevant
aspects (e.g. from
specification)
Determine input data
space
Select a test object:
decompose
space into classes
Privileges
Operation
Access Method

Test Generation with TESTONA
• Equivalence Class Testing
• Boundary Value Analysis
• Combinatorial Testing
• Constraints support (dependency rules)
• Minimal Combination (test each input value at least once)
• All-Pairs test (Pairwise)
• Mixed-Strength Generation
• Full Combination (Cartesian product of all input values)
• Prioritized and Statistical Testing
This is how TESTONA obtains test efforts:
reliable, standard-conform, and reproducible.
Combine classes
into test cases
Automate test
generation
TESTONA

TESTONA
• Graphical editor for test design (e.g. classification tree method, boundary value
analysis, equivalence classes, etc.)
• Automatic verification of test cases and dependency rules
• Automatic test case generation
• Tag approach for annotation of any
information and meta-data
• Generic and customized
tool coupling
Used by

Test Oracle
William E. Howden (1978):
non-trivial challenge of
deciding whether a test case
has passed or failed
Non-automated Oracles
• Trivial case: Manual Assignment
Implicit Oracles
• Indirect evaluation (e.g. no exception)
Derived Oracles
• A posteriori (e.g. Regression Test)
Specified Oracles
• Based on (formal) Specification
Categorization by Barr, Harman, McMinn,
Shahbaz, and Yoo (2015)

Non-automated Oracle Example
• Manual assignment for each test case

Implicit Oracle Example
• Monitor system
No exceptions at
runtime,
CPU load, …

Derived Oracle Example
• Run once, record results, inspect, assign to tests

Specified Oracle Example
(New AND Success ↔ more) AND (Delete AND Success ↔ less)
(empty AND NOT New) OR (full AND New) ↔ Fail

TEST IMPLEMENTATION
(TEST SCRIPT GENERATION)

Code Generation: Motivation
Classification
Tree Test Cases
Specification
(Functional Requirements,
Agile Board, etc.)

Code Generation: Motivation
Classification
Tree Test Cases
TESTONA
generic
code
export
• Template-based export of test cases to JUnit, XML, text, ...
Specification
(Functional Requirements,
Agile Board, etc.)

Technical Details / Demonstration: JUnit
setDB("%CLASS%");
int oldCount = getCount();
setUser("%CLASS%");
perform("%CLASS%");
assertTrue(getCount() %CLASS% oldCount);
assertEquals(isSuccess(), %CLASS%);
Include these per mark in test case
@Test
public void %TEST% throws Exception {
}
Include this per test case

setDB("%CLASS%");
int oldCount = getCount();
setUser("%CLASS%");
perform("%CLASS%");
assertTrue(getCount() %CLASS% oldCount);
assertEquals(isSuccess(), %CLASS%);
@Test
public void %TEST% throws Exception {
}
import org.junit.Assert.*;
public class Example {
}
Include this once at start

Technical Details / Demonstration: XML
<database>%CLASS%</database>
<user>%CLASS%</user>
<action>%CLASS%</action>
<expect val="count">%CLASS%</expect>
<expect val="ok">%CLASS%</expect>
<testcase>
</testcase>
<testspec>
</testspec>
Include this once at start

Technical Details / Demonstration: XML

•Expected result prediction
•Test Script Generation
Conclusion
•Classification Tree Method to
structure test goals
•TESTONA to obtain test efforts
Peter M. Kruse @pmk99
peter.kruse@berner-mattner.com
www.testona.net

Combinatorial Black-Box Testing with Classification Trees

Combinatorial Black-Box Testing with Classification Trees

Recommended

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Combinatorial Black-Box Testing with Classification Trees (20)

More from TechWell (20)

Recently uploaded (20)

Combinatorial Black-Box Testing with Classification Trees