Research and Technology

TACOS‘04, Barcelona, March 2004

Test Case Design by means of
CTM and CTE
Joachim Wegener
DaimlerChrysler AG, Research and Technology
[email protected]

Introduction and Motivation

State of the Art
Classification-Tree Method
Classification-Tree Editor CTE
Conclusion and Future Work

Wegener, Software Technology Research Dept., Slide 1

Introduction and Motivation

Research and Technology

Software Development Requirements at DC

 Software as differentiating
 Increasing amount of software factor for competitive
in products in almost all areas advantage

 Cost-effective  Increasing
development and complexity of
time-to-market systems
crucial for
competitiveness

 Increasing significance
of software in safety
 High consequential relevant areas
costs due to faults
up to 60 communicating embedded controllers
up to  High requirements
1,000,000 for software
lines of software code
based systems
various buses, hundredsdueoftomessages
norms,
standards and legal regulations
> 2 km wire

Wegener, Software Technology Research Dept., Slide 2

Introduction and Motivation
Research and Technology

Applications at DC

Wegener, Software Technology Research Dept., Slide 3

Introduction and Motivation

Research and Technology

Applications at DC

Wegener, Software Technology Research Dept., Slide 4

Introduction and Motivation
Research and Technology

System Development at DC

DaimlerChrysler Acceptance Test

SPICE Vehicle Tests

Pre-Development Assessments
Vehicle Integration
Prototyping
Code Quality
Specification System Test Assessments

Integration Test
Component Development
Integration

Implementation Unit Test

Suppliers

Wegener, Software Technology Research Dept., Slide 5

Introduction and Motivation

Research and Technology

Testing in Practice
• testing is an immense cost factor in the course of system development

average distribution of costs for embedded systems

system and
acceptance 20
testing
50 other development activities
30
module and
integration
testing

• high cost
Test automation
• testing is too unsystematic
• low error detection rate Test methods

Wegener, Software Technology Research Dept., Slide 6

Introduction and Motivation
Research and Technology

Objectives of Testing
Through executing a system with selected test data the test is aiming at
z detecting errors in the system under test and
z gaining confidence in the correct functioning of the test object.

Strong Features
z taking into consideration the real environment (e.g. target computer, compiler) and
z testing the dynamic system behavior (e.g. run time behavior, memory space requirement)

Weak Features
z an exhaustive test is usually impossible

test data selection by certain criteria

Wegener, Software Technology Research Dept., Slide 7

State of the Art

Research and Technology

Test Activities

Test Planning

Specification Program
Test Documentation
Test Organization

Test Case Design

Test Execution Monitoring

Test Evaluation

• most important for test quality

• methods and tools needed

Wegener, Software Technology Research Dept., Slide 8

State of the Art
Research and Technology

Testing Methods
Functional test:
• test case design performed on the basis of the specification
• most important test approach
• common methods show insufficient systematics when partitioning
the input domain, difficult procedure, or no efficient tool support
• test size is hard to quantify (coverage of requirements, function points, ...)
• difficult to automate
• examples: equivalence partitioning, boundary value test, category-partition method

Structural test:
• test case design performed on the basis of the program structure
• common test approach (included in many standards)
• not possible to check whether all requirements have been implemented
• difficult to automate (limits of symbolic execution)
• examples: statement test, branch test, path test

Wegener, Software Technology Research Dept., Slide 9

State of the Art

Research and Technology

Testing Methods
Statistical Test:
• test data generation in accordance with operational profile
• operational profile hard to determine, especially for new systems
• extensive test evaluation, if no test oracle available

Random Test:
• generation of random test data
• specific test-relevant value combinations are produced only with little probability
• extensive test evaluation, if no test oracle available

Evolutionary Test:
• generation of test data by means of search techniques
• test objectives have to be transformed into an optimization problem

Mutation Test:
• test cases are build in order to kill mutants of original program
• no procedure how to define the test cases

Wegener, Software Technology Research Dept., Slide 10

Classification-Tree Method
Research and Technology

Functional Testing
• lack of functional testing methods which are
• efficient
• easy to use in practice
• suited for automation
• lack of testing tools for black-box test case design

Classification-Tree Method
efficient functional test method CTM:
• systematic stepwise procedure which is easy to
understand
• graphical notation with compact representation
of the over-all test
• good test documentation
• tool support with the CTE XL
• proven in practical use

Wegener, Software Technology Research Dept., Slide 11

Classification-Tree Method CTM

Research and Technology

Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks

large

Aspects
Size Color Shape Size Color Shape

small large red green blue circle triangle square

Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain

Wegener, Software Technology Research Dept., Slide 12

Classification-Tree Method CTM
Research and Technology

Classification-Tree Method

Size Color Shape

small large red green blue circle triangle square

Shape of
triangle
equi-
lateral isosceles scalene

Wegener, Software Technology Research Dept., Slide 13

Classification-Tree Method CTM

Research and Technology

Classification-Tree Method

Size Color Shape

small large red green blue circle triangle square

Shape of
triangle
equi-
lateral isosceles scalene
1
2
3

Wegener, Software Technology Research Dept., Slide 14

Classification-Tree Method CTM
Research and Technology

Classification-Tree Method

Size Color Shape

small large red green blue circle triangle square

Shape of
triangle
equi-
lateral isosceles scalene
1
2
3

Wegener, Software Technology Research Dept., Slide 15

Classification-Tree Method CTM

Research and Technology

Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks
classification
large
class

Aspects
Size Color Shape Size Color Shape

small large red green blue circle triangle square

Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain

Wegener, Software Technology Research Dept., Slide 16

Classification-Tree Method CTM
Research and Technology

Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks
Minimality criterion: 5 testcases
large five disjoint classes for
shape

Aspects
Size Color Shape Size Color Shape

small large red green blue circle triangle square

Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain

Wegener, Software Technology Research Dept., Slide 17

Classification-Tree Method CTM

Research and Technology

Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks
Maximality criterion: 30 testcases
large 2*3*5

Aspects
Size Color Shape Size Color Shape

small large red green blue circle triangle square

Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain

Wegener, Software Technology Research Dept., Slide 18

Classification-Tree Editor CTE
Research and Technology

Classification-Tree Editor CTE

• syntax directed
• context sensitive
• graphical
editor to support the systematic test case design by means of the classification-tree method

Supporting
• hierarchies to structure large classification trees and
• hierarchies to structure sets of test cases
• description of logical dependencies between classes of the classification-tree
• combination rules for the generation of test cases according to logical dependencies and
combination rules
• sorting of test cases
• comprehensive export functionalities
• open API and architecture supporting the implementation of tool integrations such as with
• DOORS
• Test Director
• Tessy
• Time Partition Testing
• Win Runner

Wegener, Software Technology Research Dept., Slide 19

Hierarchies for large trees

Research and Technology

2. Test cases can be hierachically grouped:

1. Subtrees can be folded/unfolded at will:

folded
subtree

groups of
test cases

Wegener, Software Technology Research Dept., Slide 20

Implicit (Syntactical) Rules
Research and Technology

1. Classes belonging to the same classification 2. If a class is marked all parent tree nodes
are disjoint (i.e. exclusive). are marked, too.
e.g., red, green, and blue e.g., equilateral implies triangle

‘triangle’
is implicitly
marked
disjoint
classes

Wegener, Software Technology Research Dept., Slide 21

Usage of Logical Dependencies

Research and Technology

1. Specification of impossible or required

2. Verification of dependencies
class combinations e.g., circle => NOT red
e.g., circle => NOT red

verification
of logical
dependencies

specification
of logical
dependencies

red circle

Wegener, Software Technology Research Dept., Slide 22

Combination Rules
Research and Technology

• express the desired combinatorics between classes and classifications

• usage of combinatorial expressions

specification
of combination
rules

Wegener, Software Technology Research Dept., Slide 23

Automated Test Case Generation

Research and Technology

• based on combination rules and logical dependencies

• algorithm based on heuristic combinatorial search

Wegener, Software Technology Research Dept., Slide 24

Classification-Tree Editor CTE
Research and Technology

Real-World Example: Positioning

Wegener, Software Technology Research Dept., Slide 25

API Supporting Tool Integrations

Research and Technology

Examples Tool Integration

Acceptance Test

Vehicle Tests

Vehicle Integration
dSPACE
Specification DOORS TPT MSC TTCN3
Auto-
mation
Win
System Test Runner
Desk

System Test

Integration Test
Integration
MTest
Unit Test Tessy

Wegener, Software Technology Research Dept., Slide 26

Conclusion and Future Work
Research and Technology

Summary
• Classification-Tree Method enables systematic test case design for functional testing

• successfully applied in practice to nearly all application domains

• Efficient tool support by means of the CTE XL

• available on www.systematic-testing.com
• you can start using it

• Many tool integrations available. Further integrations under development.

Future Work

• Extensions for test management and product line approaches

• Description of expected behavior
• Generation rules for sub-trees
• Metrics to assess test case design quality

Wegener, Software Technology Research Dept., Slide 27