Seminar on Software Testing

Software Testing
As software quality control

Beat Fluri
Zurich SoftSummer 2009
Summer School on Complex Software Systems

University of Zurich
Department of Informatics software evolution & architecture lab

Me, myself, and I

Beat Fluri
Senior research associate (UZH)

ETH Zurich UZH
Studies of Computerscience Doctorate in area of Software
Evolution Analysis

Lecturer @ UZH
Software Testing

ﬂuri.computerscience.ch
Material of this seminar

Dr. Beat Fluri © 2009 2

Agenda
Introduction into Software Testing

Continuous Integration

Unit Testing

Integration Testing

System and GUI Testing

Regression Testing

Tools you may like in you CI environment


Resources

John F. Smart’s blog

http://easytesting.org

http://java-source.net/open-source/testing-tools

What is software testing?

Design and implementation of a special
kind of software system; one that
exercises another software system with
the intent to ﬁnding bugs.
Rober V. Binder, TOOSMPT


Test design
Identify, model, and analyze the responsibilities of the system
under test
On different level of abstraction: methods, classes, packages, subsystems, system

Design test cases based on this external perspective
Functional testing

Add test cases based on code analysis, suspicions, and heuristics
Structural testing

Develop expected results for each test case or choose an approach
to evaluate the pass or no pass status of each test case


Test execution
Implementation under test is minimally operational
Bottom-up to reduce test driver and test double overhead

Test suite execution: result is evaluated as pass or no pass

Coverage tool to instrument the implementation under test
Deﬁne goal: 95% statements, 80% branches, 70% conditions, etc.

If necessary, develop additional tests to exercise uncovered code

Stop: coverage goal is met and all tests pass


Why is software testing so hard?
Input/state space
public Line(int x0, int y0, int x1, int y1)

Input per int: 232 different values: 232 x 232 x 232 x 232 = 2128
1000 lines per second: 1028 years

Execution sequences

for (int i = 0; i < n; i++) {
if (a.get(i) == b.get(i)) {
x[i] += 100;
2n + 1 paths with n iterations
} else {
x[i] /= 2;
}
}


Coincidental correctness

public class Account {
protected long lastTxDate;
public long daysSinceLastTx() {
long today = Calendar.getInstance().getTimeInMillis();
return 1 + (today - lastTxDate) / (24 * 3600 * 1000);
}
}

public class DepositAccount extends Account {
public long daysSinceLastTx() {
long today = Calendar.getInstance().getTimeInMillis();
return (today - lastTxDate) / (24 * 3600 * 1000);
}
}

Undecidability problem (halting problem)
There isn’t any program that can decide whether the execution of another program
will halt or not
For simple program it is decidable but not in general


Why software testing?
Find bugs early
Airbag does not work: ﬁnd out before or after releasing the car?

Part of quality assurance
Would you trust the airbag software if it wasn’t tested?

Supports changing a software system
Verify component integrity after a change


Why is testing neglected?
Expensive (software testing is hard)
Test case design and selection
Building test environment
Integration environment
Few methodologies for test automation

Not part of development process
Hard to include for speciﬁc projects (customer won’t pay extra effort)
Not part of the software engineering culture
Missing testing environment


Short glossary
Test case
Set of inputs, execution conditions, and expected result (pass and no pass criterion)

Test suite
Set of test cases, usually related to a testing goal

Test driver
Class or utility program that applies test cases

Double (stub, mock)
Partial, temporary implementation of a component

Test harness
Substitution for parts of the deployment environment, i.e., a system of test drivers
and other tools to support test execution

Short glossary
Oracle
A mechanism to evaluate the actual result of a test case as pass or no pass

Equivalence class
Set of input values: if one value is processed correctly/incorrectly, all other values will
also be processed correctly/incorrectly

Coverage
Percentage of elements required by a test strategy that have been exercised by a
given test suite


Short glossary
Fault model
Identifies relationships and components that are most likely to have faults

Test model
Testable representation of the relationships among elements of an implementation
based on a fault model

Test strategy
Algorithm or heuristic to create test cases from an implementation or test model

Test design
Produces a test suite using a test strategy: identification of interesting test inputs,
placing these test inputs into a sequence, and defining the expected results


Short glossary
Fault-directed testing
Testing that seeks to reveal implementation faults

Conformance-directed testing
Testing that seeks to establish conformance to requirements or speciﬁcations


V-Model
Speciﬁcation Implementation
Acceptance Test
User
Delivery
needs

System Test

System Spec System

Integration Test

Subsystem
Subsystem
Design/Spec

Unit Test

Unit
Unit
Design/Spec


V-Model

User
needs Validation Delivery

System Spec System

Subsystem
Design/Spec Veriﬁcation Subsystem

Unit
Unit
Design/Spec


Continuous Integration - What

... is a software development practice where members
of a team integrate their work frequently, usually each
person integrates at least daily - leading to multiple
integrations per day. Each integration is verified by an
automated build (including test) to detect integration
errors as quickly as possible. Many teams find that this
approach leads to significantly reduced integration
problems and allows a team to develop cohesive
software more rapidly.
Martin Fowler, Article

Source: http://martinfowler.com/articles/continuousIntegration.html


Continuous Integration - How

commit multi-stage
changes builds
commit
changes
unit tests

commit triggers
changes build integration tests

quality assurance
deployment


Continuous Integration - How
Version control repository: CVS, Subversion, Git, Mercurial, etc.

Build tools: Ant, Maven, Make, Gant, Grails, Rake, etc.

Continuous integration environment (server): CruiseControl,
Continuum, Hudson

Our experience
Subversion (http://subversion.tigris.org/)
Apache Maven (http://maven.apache.org/)
Hudson (https://hudson.dev.java.net/)


Continuous Integration - Why
Reduce risks
Defects are detected when they are introduced
Measure the health of a software
Environment always the same and build starts clean => no assumptions

Reduce repetitive manual processes (safe time and costs)
Process runs the same every time
Ordered process: compile, unit tests, integration tests, qa, etc.

Generate deployable software at any time and at any place
Going back in the build history to deploy older, maybe stable, builds

Source: Paul M. Duval. Continuous Integration. Pearson Education, Inc., 2007


Continuous Integration - Why
Enable better project visibility
Decisions on quality improvements and tests are shown immediately
Ability to notice trends in various quality metrics (# bugs; # checkstyle, findbugs, pmd
violations; code coverage of test cases)

Establish greater confidence in the software product from the
development team
Making progress is visible and encourages developers
Confidence increases if the increase of overall product quality is visible

Same code basis for every developer: reduces communication
overhead
Go to build #x, do you see...

Source: Paul M. Duval. Continuous Integration. Pearson Education, Inc., 2007


uDoo - Todo Application

udoo.model udoo.controller


uDoo - Model

<<interface>> <<interface>> <<interface>>
IEventProvider ITodoListModel IObservable

Todo
* <<interface>>
TodoList Dispatcher * IObserver
1..*

Project

EventID * Event


uDoo - Controller
<<interface>>
ITodoListController

<<interface>>
ITodoCommand
TodoListController * execute()
undo()

AddProject DueDateChange PriorityChange RemoveProject Tod

doToProject DescriptionChange MoveTodo ProjectTitleChange RemoveTodo

*
Todo

2
Project

<<interface>>
ITodoListModel

Demo
uDoo with Maven and Hudson

Unit Testing

User Test units in isolation Delivery
needs Concentrate on possible bugs within
smallest possible units

System Spec System

Subsystem
Subsystem
Design/Spec

Unit Test

Unit
Unit
Design/Spec


Unit Testing

public void doSomething(int number)
1
Method scope
Constraints on input

3

Method & class scope
Inheritance
private List<Todo> fTodos; Polymorphism
2
Class scope
Encapsulation
State dependant behavior

1

Method Scope Test
Constraints on Input

Method Scope - General Approach
Decompose the speciﬁcation into independently testable features

Identify representative values (domain analysis)
Identify constraints for all input variables (boundaries)
Select test values for each variable in each boundary
Select test values for variables not given in the boundary

Generate test case speciﬁcations
Determine expected results for these inputs

Generate test cases and instantiate tests


Domain Analysis - On, Off, In Points
On point
A value that lies on a boundary

Off point
A value that lies not on the boundary

In point
A value that lies not on the boundary and satisﬁes all boundary conditions


Open boundary: x > 10
On point: 10

Off point: 11
On point makes the boundary condition false
Off point makes the boundary condition true

Closed boundary: x <= 100
On point: 100

Off point: 101
On point makes the boundary condition true
Off point makes the boundary condition false


List
empty: List.size() == 0

loaded: List.size() > 0 && List.size() < List.MAX_CAPACITY

full: List.size() == List.MAX_CAPACITY

Abstract state on point
Smallest possible change in any state variable would produce a state change
empty: empty list
loaded:
list with one element

list with (List.MAX_CAPACITY - 1) elements
full: full list


Abstract state off point
empty: list with one element
loaded: empty list

list with List.MAX_CAPACITY elements

full: list with (List.MAX_CAPACITY - 1) elements

Abstract state in point
empty: empty list

loaded:
list with (0 < x < List.MAX_CAPACITY) elements

list with (0 < x < List.MAX_CAPACITY) elements
full: full list


One-by-One (1x1) Selection Criteria
One on point and one off point for each relational condition
x > 10: on point = 10; off point = 11

One on point and two off points for each strict equality condition
x == 10: on point = 10; off point1 = 9; off point2 = 11

One on point and one off point for each non-scalar type
x == “Testing”: on point = “Testing”; off point = “Testing!”


One-by-One (1x1) Selection Criteria
One on point and at least one off point for each abstract state
invariant
!list.isEmpty()
on point: list with one element
off point: empty list

Don’t repeat identical tests for adjacent subdomains
Adjacent subdomains may share boundaries

State 1: x >= 10, on point = 10, off point = 9

State 2: x < 10, on point = 10, off point = 11

Test points: 9, 10, 11


Domain Test Matrix
Boundary Test Cases

Variable Condition Type 1 2 3 4 5 6

On 10
> 10
Off 11

x On 100
<= 100
Off 101

Typical In 15 90

On false
!isEmpty()
aList Off true

Typical In true true true true

Expected result Reject Accept Accept Reject Reject Accept


Category-Partition Test Pattern
Design method scope test suites based on input/output analysis

Method to test:
TodoList.addTodo(Todo todo, Project project)

Procedure
1. Identify the functions of the method under test

2. Identify the input and output parameters of each function

Identify representative values
3. Identify categories for each input parameter

4. Partition each category into choices

5. Identify constraints on choices

6. Generate test cases by enumerating all choice combinations

7. Develop expected results for each test case

1. Identify the functions of the method under test
Adds the given Todo to the given Project and returns true

Does nothing if the given Project does not exist and returns false

2. Identify the input/output parameters of each testable function
Input: Given Project
Input: Given Todo
Input: Content of the todo-list

Output: true or false
Output: Given Todo is in given Project or not


3. Identify categories for each input parameter
Parameter Category
Todo value
Project value
Todo-list special case

4. Partition each category into choices
Parameter Category Choices

Todo

value

null

not null

Project

value

null

not null
Todo-list

special case

contains Project

does not contain Project


5. Identify constraint on choices
If todo-list throws an exception for (todo != null) and (project == null) it will
also throw one for (todo == null) and (project == null)

If either (todo == null) or (project == null) the state of the todo-list does not
care (DC)

6. Generate test cases by enumerating all choice combinations
2 x 2 x 2 choices: 8 test cases
Remove cases identiﬁed by Step 5: 4 test cases



Parameter/Choices Expected Result

TC Todo Project Todo-list Returned Exception Todo-list

1 DC null DC NullPointerException !todo

2 DC not null DC NullPointerException !todo

3 not null not null project true todo

4 not null not null !project false !todo

Legend
project todo-list contains project
!project todo-list does not contain project
todo project contains todo
!todo project does not contain todo


2

Class Scope Test
Encapsulation

Encapsulation
Don’t test private methods

Pollute public API
Private -> Package
Delegators for private methods

Inner test class
Deployment mess

Reﬂection


2

Class Scope Test
State dependent behavior

State Dependent Behavior
Non-modal classes
Does not impose any constraints on the sequence of message accepted, i.e., does not
have a state

Quasi-modal classes
Imposes sequential constraints on message acceptance that change with the content
of the object (Collections)

Modal classes
Places both message and domain constraints on the acceptable sequence of a
message (Account)


Invariant Boundaries Test Pattern
Select test-efﬁcient test value combinations for classes, interfaces,
and components composed of complex and primitive data types

Procedure
1. Deﬁne the class invariants

Identify representative values
2. Develop on and off points for each condition using 1x1 selection criteria

3. Complete test suite by developing in points for variables not referenced in a condition

4. Represent all test cases in a domain test matrix


Invariant Boundaries Test Pattern
Elevator Example

Elevator can only run if the following invariants are met
0 < weight <= MAX_WEIGHT
LOWEST_FLOOR <= selectedFloor <= HIGHEST_FLOOR
selectedFloor != currentFloor
door.isClosed
!inEmergency


Elevator - Domain Test Matrix
Boundary Test Cases
Variable Condition Type 1 2 3 4 5 6 7
On 0
> 0
Off 1
weight On max
<= MAX_WEIGHT
Off max+1
Typical In 23 50 66
On low
>= LOWEST_FLOOR
Off low-1
On high
<= HEIGHEST_FLOOR
Off
selectedFloor
On
!= currentFloor Off
In 1 2 0 -1 2 6 1
Typical In 0 3 5 6
On
isClosed()
door Off
Typical In true true true true true true true
On
false
inEmergency Off
Typical In false false false false false false false
Expected result Reject Accept Accept Reject Accept Reject Accept

Elevator - Domain Test Matrix
Boundary Test Cases
Variable Condition Type 8 9 10 11 12 13 14 15
On
> 0
Off
weight On
<= MAX_WEIGHT
Off
Typical In 23 55 56 54 10 11 18 14
On
>= LOWEST_FLOOR
Off
On
<= HEIGHEST_FLOOR
Off high+1
selectedFloor
On 0 -1
!= currentFloor Off 0
In 3 -1 4 0 3
Typical In -1 0 0 1 5 4 2
On true
isClosed()
door Off false
Typical In true true true true true true
On false
false
inEmergency Off true
Typical In false false false false false false
Expected result Reject Accept Accept Reject Accept Reject Accept Reject

Modal Class Test Pattern

public class Account {
private int balance;
private Date lastActivity;
public void open() {}
public int balance() {}
public void credit(int amount) {}
public void debit(int amount) {}
public void freeze() {}
public void unfreeze() {}
public void close() {}
}


Constraints on message sequences
Account: debit is not accepted if the balance is <= 0
Account: freeze is not accepted if account is closed or already frozen

Modal class test
1. Method scope test
2. Alpha-omega cycle
3. Generate state model and transition tree
4. Elaborate transition tree with a full expansion of conditional transition variants
5. Tabulate events and actions along each path to form test cases
6. Develop test data for each path by using Invariant Boundaries
7. Develop sneak path test cases


State model of Account
balance

credit [balance >= 0]
debit [balance < 0]
Overdrawn
!
credit

balance balance

Open freeze Frozen
unfreeze

debit
[today - lastActivity > 5y] balance
credit

Inactive

[balance == 0] close

[balance == 0] close Closed "


Transition tree of Account balance
Open

debit
Open

credit
Open
balance
Overdrawn

debit [balance < 0] credit
Overdrawn Overdrawn

credit [balance >= 0]
Open
Open

balance
freeze Frozen
Frozen
unfreeze
Open

balance
[today - lastActivity > 5y] Inactive
Inactive
close
Closed

Closed

balance
Conditional Transitions Open

credit
Open

debit [balance > 0]
Open
balance
debit [balance == 0] Overdrawn
Open
credit [balance < 0]
Overdrawn
debit [balance < 0]
Overdrawn credit [balance > 0]
Open

credit [balance == 0]
Open Open

balance
freeze Frozen
Frozen
unfreeze
Open

balance
[today - lastActivity > 5y] Inactive
Inactive
Closed
[balance != 0] close/error
Open [balance != 0] close/error
Inactive
Closed

Sneak Path
Bug that allows an illegal message to be accepted, resulting in an illegal transition

Test sneak path
Send illegal messages

Example: Account in state overdrawn
Send message close

Expected responses to sneak paths
Message should be rejected
State of the object should be unchanged


Test case specification
For each path in the final transition tree specify one test case

Test Case/Event Path Expected

TC Level 1 Level 2 Level 3 Terminal State

1 New Open

2 New balance Open

3 New debit [balance < 0] balance Overdrawn

Test case specification
For each sneak path specify one test case
Test Case/Event Path Expected Result

TC Test State Test Event State Exception

1 Inactive freeze Inactive IllegalEventException

2 Overdrawn close Overdrawn IllegalEventException

3 Frozen debit Frozen IllegalEventException

3

Method & Class Scope Test
Inheritance and polymorphism

Inheritance-Related Bugs
Incorrect Initialization
Superclass initialization is omitted or incorrect

Inadvertent Bindings
Privates are invisible
Non-private are invisible if subclass has member with the same name

Missing Override
A subclass-speciﬁc implementation of a superclass method is omitted
Java: use abstract classes


Naked Access
A superclass instance variable is visible in a subclass and subclass methods update
variables directly

Square Peg in a Round Hole
A subclass is incorrectly located in a hierarchy

For example: Square subclass of Rectangular

Naughty Children
A subclass either does not accept all messages that the superclass accepts or leaves
the object in a state that is illegal in the superclass


Worm Holes
A subclass computes values that are not consistent with the superclass invariant or
superclass state invariants

Gnarly Hierarchies
A subclass creates new holes, boundaries, or intersections that are inconsistent with
the superclass domain
For example, strengthen precondition (in Eiffel, e.g., not possible)

Weird Hierarchies
Inheritance is abused as a kind of code-sharing macro to support hacks without
regard to the resulting semantics

Fat Interface
A subclass inherits methods that are inappropriate or irrelevant


Polymorphic Message Test Pattern
Develop tests for a client of a polymorphic server that exercise all
client bindings to the server

Class hierarchy that overrides methods of the superclass
Client may fail to meet all preconditions for all possible bindings
The implementation of a server class is changed or extended

Approach (different than in Binder)
Determine all possible dynamic types
Call method with each possible dynamic type


Polymorphic Message Test Pattern
Movie 1 Price
title: String
getCharge(days: int) getCharge(days: int)
getFrequentRenterPoints(days: int) getFrequentRenterPoints(days: int)

ChildrensPrice RegularPrice

getCharge(days: int) getCharge(days: int)
@Test getFrequentRenterPoints(days: int)
public void getChargeTest() {
NewReleasePrice
Movie m = new Movie(new ChildrensPrice());
Assert.assertThat(m.getCharge(1), is(2)); getCharge(days: int)
getFrequentRenterPoints(days: int)
m = new Movie(new NewReleasePrice());
Assert.assertThat(m.getCharge(1), is(4));
m = new Movie(new RegularPrice());
Assert.assertThat(m.getCharge(1), is(3));
}

@Test
public void getFrequentRenterPointsTest() {
// similar to getChargeTest() for ChildrensPrice and NewReleasePrice
} Adapted from: Martin Fowler. Refactoring. Addison-Wesley Professional, 1999.


Polymorphic Server Test Pattern
Design a test suite at class scope that veriﬁes LSP compliance for
polymorphic server hierarchy

LSP: Liskov substitution principle
If for each object o1 of type S there is an object o2 of type T such that for all programs
P deﬁned in terms of T, the behavior of P is unchanged when o1 is substituted for o2,
then S is a subtype of T.
Barbara Liskov, Data Abstraction and Hierarchy, Keynote OOPSLA 1987

Hierarchy under test is nonmodal
No constraints on the sequence of messages accepted

Goal: exercise overridden methods in a polymorphic hierarchy


Polymorphic Server Test Pattern
Test each method in its deﬁning class and all subclasses that
override it

Reuse of the test suite for an overridden superclass method
It could, but check whether additional test cases are necessary to test functionality
and to satisfy coverage criteria

Overriding a superclass method used by a superclass method
Additionally test subclass inherited methods within the context of the subclass

Check for conformance with LSP
Make a client stub (simulate a client in test suite)
Clients of a polymorphic server can substitute any subtype without causing a failure


Polymorphic Server Test
class ASTNode { class ASTNodeTest {
final void accept(ASTVisitor v) { @Test void accept0Test() {/*..*/}
v.preVisit(this); @Test void acceptTest() {/*..*/}
accept0(v); }
v.postVisit(this); class IfStatementTest {
} @Test void accept0Test() {/*..*/}
void accept0(ASTVisitor v) {} @Test
} void acceptTest() {
class IfStatement extends ASTNode { ASTNode a = new IfStatement();
void accept0(ASTVisitor v) { a.accept(new VisitorStub());
boolean children = v.visit(this); // ...
if (children) {
acceptChild(v, getExpression()); }
acceptChild(v, getThenStatement()); }
acceptChild(v, getElseStatement()); class WhileStatementTest {
} @Test void accept0Test() {/*..*/}
v.endVisit(this); @Test
} void acceptTest() {
} ASTNode a = new WhileStatement();
class WhileStatement extends ASTNode { a.accept(new VisitorStub());
void accept0(ASTVisitor v) { // ...
// ... }
} }
}

Structural Testing
Find structural problems in source code

Structural Testing
Test cases that satisfy coverage criteria

Simple, but effective, coverage criteria
Statement coverage
Branch coverage
A

Condition coverage

B C

D

F
E X

G
I

J H

K


Statement coverage
All statements in a method have been executed at least once

Why is statement coverage is not enough
Statement coverage can be achieved without exercising all branches
Single loop iteration can cover all statements but may not reveal loop control bugs
Statement coverage can be achieved without exercising all true-false combinations of
conditions


Branch Coverage
Every path from a node is executed at least once

Why is branch coverage not enough
Branch coverage can be achieved without
exercising all true-false combinations of conditions
Branch coverage does not ensure that all entry-
exit paths are executed

int foo(int x) {
if (a == b) ++x;
if (x == y) --x;
return x;
}


Condition Coverage
Condition coverage
Each basic condition to be evaluated as true and false at lease once
It is not required to test all branches
if (x > 0 || x <= 10) {}
(true || false) and (false || true)

Not possible: Short-circuit evaluation

Branch/condition coverage
Condition coverage + each branch taken at least once
Still not all true-false combinations of conditions exercised

(true || false) and (false || true) and (false || false)


Who does the coverage calculation?
Yes, you can!
But you shouldn’t...

Use Cobertura
integrates nicely into Maven and Hudson


What about Testing in Isolation?
Always possible; but dependencies must be simulated
Stubs or mocks

Stubs
Provide answers to calls made during tests. Re-implemented interface of the original
class

Mocks
Stubs + expectations which form a speciﬁcation of the calls they are expected to
receive


When to Use Mocks
Classical way: Use real objects if possible
Mocks if using real objects is awkward or if real objects cannot be used
Focus on result of behavior
Cost/beneﬁt balance

Mockist way: Always use a mock for any object
Problem: strong coupling to the implementation
Mocking model objects is overkill
Focus on how a unit/component is implemented to write expectations

Source: http://martinfowler.com/articles/mocksArentStubs.html


Tools for Mocking
EasyMock
http://www.easymock.org
based on reflection
mocking of static, final, or private methods; constructors not possible

JMock
http://www.jmock.org
based on reflection
mocking of static, final, or private methods; constructors not possible


Tools for Mocking
Amock
http://amock.blogspot.com/
based on aspect oriented programming
mocking of static, ﬁnal, or private methods; constructors possible
AspectJ must be used

JMockit (0.95)
https://jmockit.dev.java.net/
based on instrumentation in Java 1.5
(VM arg: -javaagent:<path>/jmockit.jar=junit4)
mocking of static, ﬁnal, or private methods and constructors possible
no other tool necessary


uDoo
<<interface>> <<interface>>
ITodoListModel IObservable

Todo
*
TodoList Dispatcher
1..*

Project

Goal: test TodoList in isolation

Problems
Project and Todo objects are necessary
Dispatcher object is necessary to test notiﬁcation


JMockit in uDoo
Classical way: Use real objects if possible

Project and Todo objects are normally used (container objects)

Since Dispatcher can be replaced; test against IObserable
Use expectations to test whether or not IObservable notiﬁes observers
Use Hamcrest-Matchers to test whether or not the observers are notiﬁed with the
correct Event

Use mock for Project to test corrupted Project (i.e.,does not store
Todos)


V-Model

User
Delivery
needs

System Spec System

Integration Test

Subsystem
Subsystem
Design/Spec

Unit Test

Unit
Unit
Design/Spec


Integration Testing
Integration testing is a search for components faults that cause
intercomponent failures

Breaking up component isolation step-by-step

Integration strategy
Which components are the focus of the integration test?
In what sequence should component interfaces be exercised?

Integration testing in object-oriented development begins early
Within a class (integrating methods)
Within a class hierarchy (integrating superclasses)
Between a client and its server
Within a package (integrating classes)
...

Dependency Analysis
Integration strategy follows sequences in which components have
to be tested

Dependency analysis among components is used to deﬁne the
sequences

Dependency analysis: approach
Take component diagram, e.g., UML class diagram
Make topology (i.e., directed graph) of the dependencies


Dependency Analysis

P

*
CC C O * M

* *
A CP S


Dependency Analysis
CC

P C

O A

M CP

S
depends on

Big Bang Integration
Demonstrate stability by attempting to exercise an entire system
with a few test runs

Integrate all components at once
Monolithic system
Components tightly coupled

No doubles are necessary

Problems
Fault locations are difﬁcult to ﬁnd
Big bang test suites are usually to small to reveal all interface faults

Try to avoid big bang integration!

Big Bang Integration
Driver

CC

P C

O A

M CP

S


Bottom-up Integration
Demonstrate stability by adding components to the system under
test in uses-dependency order, beginning with components having
the fewest dependencies

Approach
Test leaf components in dependency graph with drivers for each leaf. Use doubles
when necessary
Repeat: test components on the next level; until the entire system is tested


CC

P C

O A

M CP

S Driver


CC

P C

O A
Driver

M CP Driver

S


Advantages
Early integration of leaf components
Parallel development on components in subtrees of dependency graph
Reduces the number of doubles (only for cycles)

Disadvantages
Many drivers have to be developed
Interaction testing is limited to collaborations implemented in the component (root
of subtree)
Postpones checking critical control interfaces and collaborations until the end (root
of dependency graph)


Top-down Integration
Demonstrate stability by adding components to the system under
test in control hierarchy order, beginning with the top level
control objects

Approach
Test component(s) at the highest level ﬁrst. Use doubles for lower level components
Continue in a breath-ﬁrst order at each level in the dependency graph until all
components are integrated


Driver

CC

P C

O A

M CP

S


Driver

CC
Driver

P C

O A

M CP

S


Advantages
Early demonstration of end-to-end functionality
Fewer drivers necessary than with bottom-up integration
Test cases can be reused to drive lower-level tests

Disadvantages
Development and maintenance of doubles
Test cases in high-level components may not be sufﬁcient for lower-level components


Collaboration Integration
Demonstrate stability by adding sets of components to the system
under test that are required to support a particular collaboration

Approach
Map collaborations onto the dependency graph
Choose a sequence in which to apply testing on collaborations
Test collaboration in the chosen sequence


CC

P C

O A

M CP

S


CC

P C

Driver O A

M CP

S


Driver CC

P C

O A

M CP

S


Advantages
Few drivers and doubles necessary
Focus on end-to-end functionality; useful for system scope testing
Reuse of collaboration integration test suite for system test

Disadvantages
Inter-collaboration may not be tested thoroughly
Participants in a collaboration are not exercised separately (big bang)
Probably many doubles for testing initial collaborations


Backbone Integration
Combination of top-down integration, bottom-up integration,
and big bang integration to verify interoperability among tightly
coupled subsystems

Used, for instance, for embedded system applications
Backbone provides services that are essential for running tests and the application
Creating doubles for the entire backbone would be impractical

Approach
Test each component in backbone in isolation and do a bottom-up integration or big
bang integration of the backbone
Do a top-down integration on the application control components



Backbone


Advantages
Mitigates the disadvantages of top-down and bottom-up integration

Disadvantages
Doubles and drivers are necessary


Integration Testing and Continuous Integration

Multi-Stage Builds
Special test classes for integration testing
john ferguson smart

Say Maven to run integration tests only for goal “integration-test”
mvn integration-test

Make a separate job in Hudson for integration testing
Rapid feedback on unit testing level is still satisﬁed
Integration may and are allowed to run longer than unit tests


Integration Testing for uDoo
<<interface>>
ITodoListController
Integrating controller with
model <<interface>>
ITodoCommand
* execute()
TodoListController
Integrating UI with controller and undo()

model comes later

AddTodo AddProject DueDateChange PriorityChange RemoveProject

Bottom-up
AddTodoToProject DescriptionChange MoveTodo ProjectTitleChange RemoveT
ITodoCommands with TodoList
TodoListController with
*
ITodoCommands and TodoList Todo

2
Project

Top-down <<interface>>
ITodoListModel

TodoListController with TodoList (including ITodoCommands, i.e., no mocks for
commands)

Decision: top-down
The two steps with bottom-up are more or less the same

Demo
Integration Testing for uDoo

V-Model

User
Delivery
needs

System Test

System Spec System

Integration Test

Subsystem
Subsystem
Design/Spec

Unit Test

Unit
Unit
Design/Spec


System and GUI Testing
Testing at application level

Why testing at system level scope?
Individual veriﬁcation of components cannot guarantee a correctly functioning
system.
Demonstrate that the system implements all required capabilities.
Answer the question: “Can we ship it yet?”

Objectives for testing at system level scope
Requirement documents (formal)
Natural language documents (informal)
Prototypes
Use cases


Two-step approach
UI Integration test
Verify if UI interaction works as intended with the subsystem

User story or use case testing with UI
Validate if system works as intended when simulating user stories or use cases with UI
interaction


GUI Integration Testing
Manual GUI testing
Testing script that is executed by a user
Hard to repeat (e.g., in a CI environment)

Semi-automated GUI testing
Capture and replay
Have to be adapted on changing UI

Automated GUI testing with a framework
FEST-swing: http://fest.easytesting.org
Abbot: http://abbot.sourceforge.net/doc/overview.shtml


FEST-Swing
FEST-Swing features
Simulation of user interaction with a GUI (e.g., mouse and keyboard input)
Reliable GUI component lookup (by type, by name, or custom search criteria)
Support for all Swing components included in the JDK
Compact and powerful API for creation and maintenance of functional GUI tests
Supports Applet testing
Ability to embed screenshots of failed GUI tests in HTML test reports
Can be used with either TestNG or JUnit
Supports testing violations of Swing's threading rules

Source: FEST-swing: http://fest.easytesting.org/swing/


uDoo and FEST-Swing
uDoo UI integration with controller component

For each functionality in the TodoController
Trigger it via the UI
Verify that the model is as expected (trigger via UI does not have side-effects in the
model)
Verify that the model is consistently displayed in the UI (trigger via UI does not have
side-effects in the UI)
Verify that the entire model is still accessible via UI


User Story Testing
Develop a system test suite by simulating a user story

Difference to integration testing
User stories cover a chain of actions
Integration testing starts over for each test case

Approach
Write down a story that covers all features of the application
Specify a test case that runs the story as a system test including UI interaction
Verify consistency after each feature invocation

Use coverage tool to verify that each method was called at least
once


uDoo User Story Testing
Specify a user story that uses each feature of uDoo

After executing each feature verify consistency
Create a new todo and change its title: verify
Move a todo and undo the action: verify
etc.

Implement the user story as a FEST-Swing test case


Demo
FEST-swing in uDoo
UI Integration Testing
User Story Testing

Regression Testing
A new or modiﬁed component can fail when used with
unchanged components
Failures by generating side-effects or feature interactions

When a new or modiﬁed component causes a system failure, the
system is said to regress

A regression test case is a test case that the baseline version of a
component has passed and which is expected to pass when run on
a delta version


Regression Testing
Regression test suite is a set of existing test cases

Does not contain tests for new or changed functionality

Plays an important role in revealing bugs
The software failure in the Ariane 5 rocket controller was in large part due to an
assumption that previously tested code would work when it was reused, obviating
the need for regression testing.


Regression Testing - When and How
When a new subclass has been developed
Rerun the superclass test cases on the subclass, beside new test cases for the new
subclass

When a superclass is changed
Rerun the superclass test cases on the superclass and on its subclasses. Rerun the
subclass test cases

When a server class is changed
Rerun test cases for the clients of the changed class plus testing the changed class

When a bug ﬁx has been completed
Rerun test cases on any part of the software that depend on the changed code to
reveal unintended side effects of the bug ﬁx


Regression Faults
Two components B and D pass individually their test suite, but fail
when used together

D has a side effect on B
D changes an entity used by B

D is a client of B
D sends a msg that violates B’s invariant or a precondition

D is a server of B
B sends a msg to D. D’s postconditions have changed or are buggy. The return value
may cause a failure of B


Regression Faults
D is a new subclass of a polymorphic hierarchy that is not LSP-
compliant
B is a client of D and expects the new subtype to be compatible

D is incompatible with B
Ariane 5 disaster resulted from an incorrect ﬂoating-point conversion. The
requirements the algorithm , and the code did not change

An undesirable feature interaction occurs between B and D


Reducing a Test Suite
Full regression test can be too time consuming (setup or run)

Reduced regression test: Select a set of test cases for a speciﬁc test
scenario
In other words, which test cases can be skipped?

Safe reduction (likelihood of ﬁnding regression faults is not
reduced)


Safe Reduction
Inclusiveness
Percentage of baseline test cases that may show regression faults; a safe regression
test suite is 100 percent inclusive

Precision
Percentage of baseline test cases in a reduced test suite that cannot reveal regression
faults and that are not selected for the reduced test suite

Efﬁciency
Cost of identifying a reduced regression test suite

Generality
The range of application for the selection strategy


Regression Test Patterns
Patterns describe which baseline test cases should be selected

Each pattern has a “safe reduction” evaluation with the
predicates inclusiveness, precision, efﬁciency, and generality

Patterns
Retest All
Retest Risky Use Cases
Retest Changed Code


Rest All
Rerun the entire baseline test suite on a delta version

Just run all existing tests

Safe reduction evaluation
Inclusiveness: All baseline tests are selected; => is safe
Precision: No tests that could be skipped are skipped; => the least precise
Efﬁciency: Lowest cost of analysis and setup. Highest run cost
Generality: Can always be applied. Easy to set up, no analysis or special development
is necessary


Use risk-based heuristics to select a partial regression test suite.
Run this subset on a delta version

Suspicious use cases: Select use cases that depend on components
that
are individually unstable or unproven
have not been shown to work together before
implement complex business rules
have a complex implementation
were subject to high churn during development (many changes spread all over the
system)


Critical use cases: Select tests for use cases that are necessary for
safe, effective operation

Inclusiveness: Unsafe; test cases are not selected by analysis of dependencies
Precision: Some tests that could be skipped may be repeated
Efﬁciency: Setup cost are low: no code analysis is necessary; run cost may be high
Generality: Can always be applied. Easy to set up, no analysis or special development
is necessary


Retest Changed Code
Use code change analysis to select a partial regression test suite.
Run this subset on a delta version

Procedure
1. Use Cobertura to check which test case cover which source code parts

2. Use CVS/SVN diff information to see which source code parts changed
3. Rerun test cases that cover deleted or changed source code parts


Retest Changed Code
Inclusiveness: All baseline test cases that can produce a different result are selected;
=> is safe
Precision: Most precise of the white box regression test strategies
Efﬁciency: If test case selection can be automated, the setup cost is low; in worst case
(all baseline test case have to be selected) the total cost of this pattern can be higher
than Retest All pattern
Generality: Class scope; => analysis knowledge and special development is necessary


Regression Testing and CI
Recommendation: make a separate Maven module and Hudson
job

Retest all
Copy all existing tests to the regression-testing module

Retest risky use cases
Create a test suite with the risky use cases and conﬁgure maven

Retest changed code
Write a script...


Hudson Plugins
Besides automated build and test Hudson offers a wealth of
plugins
Source code management, build triggers, build notiﬁers, build tools, build wrappers,
build reports, Maven, etc.
http://wiki.hudson-ci.org/display/HUDSON/Plugins

Build report plugins for QA
Checkstyle
Cobertura (already shown)
FindBugs
PMD


Checkstyle
“Checkstyle is a development tool to help programmers write Java
code that adheres to a coding standard. It automates the process
of checking Java code to spare humans of this boring (but
important) task. This makes it ideal for projects that want to
enforce a coding standard.”

http://checkstyle.sourceforge.net/

Maven plugin
http://maven.apache.org/plugins/maven-checkstyle-plugin/

Conﬁgure via XML or wizard in Eclipse
http://eclipse-cs.sourceforge.net/


FindBugs
“[FindBugs] a program which uses static analysis to look for bugs
in Java code.”

http://findbugs.sourceforge.net/

Over 350 bug patterns
http://findbugs.sourceforge.net/bugDescriptions.html

Maven plugin
http://mojo.codehaus.org/findbugs-maven-plugin/2.0.1/


PMD
PMD scans Java source code and looks for potential problems like:
Possible bugs - empty try/catch/ﬁnally/switch statements
Dead code - unused local variables, parameters and private methods
Suboptimal code - wasteful String/StringBuffer usage
Overcomplicated expressions - unnecessary if statements, for loops that could be
while loops
Duplicate code - copied/pasted code means copied/pasted bugs

http://pmd.sourceforge.net/

Over 280 rules
http://pmd.sourceforge.net/rules/index.html

Maven plugin
http://maven.apache.org/plugins/maven-pmd-plugin/

Seminar on Software Testing

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Seminar on Software Testing (20)

Recently uploaded (20)

Seminar on Software Testing