Software testing

““ Testing
Testing isis the
the process
process of
of
executing
executing aa program
program withwith the
the
intent
intent of
of finding
finding errors.”
errors.”
Glen
GlenMyers
Myers

12/08/21 CMPE 310 Winter 2001 1

 Limits of software testing
 Testing can only prove the presence of
bugs - never their absence

WHY???

12/08/21 CMPE 310 Winter 2001 2

 Exhaustive testing
Assume that there are
520 possible different
execution flows. If
we execute one test
per millisecond, it
would take 3.170
years to test this
program!!
loop < 20x

12/08/21 CMPE 310 Winter 2001 3

 The Peculiarity of Testing
 Testing is the process of executing a program
with the intent of finding an error
 A good test case has a high probability of
finding an as-yet undiscovered error

A successful test is one that uncovers
an as-yet undiscovered error

12/08/21 CMPE 310 Winter 2001 4

 Who tests the software better??

developer independent tester

Understands the system Must learn about the system,
but, will test “gently” but, will attempt to break it
and, is driven by “delivery” and, is driven by quality

12/08/21 CMPE 310 Winter 2001 5

 Testing Principles
 All testing should be traceable to customer
requirements
 Tests should be planned long before testing begins
 The Pareto principle applies to software testing as
well
 Testing should begin in the small and progress
toward testing in the large
 To be most effective testing should be performed by
a third party

12/08/21 CMPE 310 Winter 2001 6

 Testability = developing a
program be tested easily
 Operability - “The better it works, the more efficiently it can be
tested”
 Observability - the results are easy to see, distinct output is
generated for each input, incorrect output is easily identified
 Controllability - processing can be controlled, tests can be
automated & reproduced
 Decomposability - software modules can be tested
independently
 Simplicity - no complex architecture and logic
 Stability - few changes are requested during testing
 Understandability - program is easy to understand

12/08/21 CMPE 310 Winter 2001 7

 Test case design
“Bugs lurk in corners
and congregate at
boundaries…”
Boris Beizer

OBJECTIVE to uncover errors

CRITERIA in a complete manner
CONSTRAINT with a minimum of effort and time
12/08/21 CMPE 310 Winter 2001 8
 Approaches to tests
requirements

output

input

events

Black Box Testing White Box Testing

12/08/21 CMPE 310 Winter 2001 9

 White box testing

Also called glass

box testing

CANNOT BE
EXHAUSTIVE!!!

12/08/21 CMPE 310 Winter 2001 10

 Approaches to White Box
Testing
 Exercise all independent paths in a
program at least once
 Exercise all logical decisions on their true
and false sides
 Execute all loops at their boundaries and
within their operational bounds
 Exercise internal data structures to assure
their validity
12/08/21 CMPE 310 Winter 2001 11
 Why bother with white box
testing?
Black box testing: (1)Requirements fulfilled and (2)
Interfaces available and working
 Question: Why white box testing?

• Logical errors and incorrect assumptions are

inversely proportional to the probability that a
program path will be executed
• We often believe that a logical path is not likely to
be executed when, in fact, it may be executed on
a regular basis
• Typographical errors are random
12/08/21 CMPE 310 Winter 2001 12
 Exhaustive testing
If-then-else There are 520=1014 (approx.)
possible paths!

loop < 20x

Selective Testing
12/08/21 CMPE 310 Winter 2001 13
 Selective testing
a selected path Selective:

Basis Path testing

Condition testing
Dataflow testing
Loop testing

12/08/21 CMPE 310 Winter 2001 14

 Basis set
 Basis set = a set of paths that will execute all
statements and all conditions in a program at
least once (can have redundancies)
 The basis set is not unique
 Cyclomatic complexity defines the number of
independent paths in the basis set
 Goal: Define test cases for basis set

12/08/21 CMPE 310 Winter 2001 15

 Basis path testing
Rationale:
Rationale I want to test all
statements of a program
at least once.

Compute the cyclomatic complexity of the

program P - V(P)
Use it to determine how many test you have to
do:
V(P) provides an upper bound of tests that
must be executed to guarantee coverage of all
program statements
12/08/21 CMPE 310 Winter 2001 16
 Basis path testing
1. We start with computing V(P):
1 Since V(P) = 4, there are four
paths
2
Path 1: 1,2,3,6,7,8
4 3
Path 2: 1,2,3,5,7,8
5 6
Path 3: 1,2,4,7,8
Path 4: 1,2,4,7,2,4…7,8
7
2. We derive test cases to exercise
8 these paths.
12/08/21 CMPE 310 Winter 2001 17
 Documenting test cases
 Name
 Number
 Values for the inputs
 Expected outputs
 Short description (if needed)

12/08/21 CMPE 310 Winter 2001 18

 Basis path testing notes
 you don’t need a flow chart,
but the picture will help when
you trace program paths
 count each simple logical test,
compound tests count as 2 or
more
 basis path testing should be
applied to critical modules

12/08/21 CMPE 310 Winter 2001 19

 Example
Program
void
void f()
f() {{
if(count
if(count >> 10){
10){
fixed_count
fixed_count == fixed_count
fixed_count ++ count;
count;
done
done == 1;
1;
}} else
else if(count
if(count >5){
>5){
fixed_count
fixed_count --;
--;
}} else
else {{
fixed_count
fixed_count == count
count ** 4;
4;
}}
}}

12/08/21 CMPE 310 Winter 2001 20

 Flow Graph
void f()
11

if(count > 10)

22
if(count >5)
44 33
fixed_count = fixed_count + count;
fixed_count = count * 4; 55 66 done = 1;
77

88 fixed_count --;

return;

12/08/21 CMPE 310 Winter 2001 21

 Derivation of the test cases
 V(P) = 3 (2 conditions + 1)
11
 Independent paths:
(a) 1, 2, 3, 8 22

(b) 1, 2, 4, 5, 7, 8 44 33

(c) 1, 2, 4, 6, 7, 8 55 66
 Test cases: 77

(a) count  13 88

(b) count  8
(c) count  2

12/08/21 CMPE 310 Winter 2001 22

 Comment
 We have not consider all possible cases
 The number of possible cases also in
this simple example is infinite …

Why?
 Suppose you embed your program in a
HW with a documented bug each time
44 is incremented to 45 ...
12/08/21 CMPE 310 Winter 2001 23
 Not required for exams

Condition Testing
 Condition testing is a test case design method
that is based on testing all possible conditions
in a program
 A condition in a program may contain:
 Boolean operators and variables
 Parenthesis
All of them can fail!!!
 Relational operators
 Arithmetic expressions
Condition testing focus on testing
 …. each condition in the program
12/08/21 CMPE 310 Winter 2001 24
 Kinds of condition testing
 Branch testing: for each condition C, the true and the
false branches resulting from it and every simple condition in C
need to be executed at least once
 Domain testing: it uses information on the domain;
 given a relational condition in the program such as a<b I have to
test it with with values of a and b a<b, a=b, a>b and a and be
VERY close
 given a boolean condition with n variables, I have to test it for each
combination of the values (that is 2n cases)
 There are other approaches, such as Branch and Relational
operator testing

12/08/21 CMPE 310 Winter 2001 25

 Not required for exams

Data Flow Testing

 Data flow testing identifies paths in the
program that go from the assignment of
a value to a variable to the use of such
variable, to make sure that the var is
properly used.
 The DU (Definition - Usage) testing
strategy requires that each DU chain is
covered at least once
12/08/21 CMPE 310 Winter 2001 26
 Data Flow Testing
Considering x, there are two DU
[1]
[1] void
void k()
k() {{ paths:
[2]
[2] xx == 11;
11; (a) [2]-[4]
[3]
[3] if
if (p(cond1))
(p(cond1)) {{
[4] yy == xx ++ 1;
(b) [2]-[6]
[4] 1;
[5]
[5] }} else
else if if (q(cond2))
(q(cond2)) {{ We need therefore to derive test
[6]
[6] ww == xx ++ 3;
3; cases to match the following
[7]
[7] }} else
else {{
[8] ww == yy ++ 1;
conditions:
[8] 1;
[9]
[9] }} (a) k() is executed and
[10]}
[10]} p(cond1) is true
(b) k() is executed and
p(cond1) is false and
q(cond2) is true
12/08/21 CMPE 310 Winter 2001 27
 Loops
 Cornerstone of every program
 Loops can lead to non-terminating programs
 Error in indexes of loops are very easy to
occur

Loop Testing is a white-box testing technique

that focuses only on the validity of loop constructs

12/08/21 CMPE 310 Winter 2001 28

 Loop testing
simple loop

nested concatenated unstructured

loops loops loops
12/08/21 CMPE 310 Winter 2001 29
 Loop testing: simple loops
 Minimum conditions - simple loops
 1. skip the loop entirely
 2. only one pass through the loop
 3. two passes through the loop
 4. m passes through the loop m < n
 5. (n-1), n, and (n+1) passes through the loop

where n is the maximum number of allowable

passes

12/08/21 CMPE 310 Winter 2001 30

 Nested loops
 Just extending simple loop testing: number
of tests increases geometrically
 Reduce the number of tests:
 start at the innermost loop; set all other loops to minimum
values
 conduct simple loop test; add out of range or excluded
values
 work outwards while keeping inner nested loops to
typical values
 continue until all loops have been tested

12/08/21 CMPE 310 Winter 2001 31

 Concatenated loops
 Same strategies as simple loops if the loops
counters are independent
 Same strategies as nested loops if the loops
counters depend on each other:

int k;
for (k=0;k<10;k++)
k=0;k<10;k++
{
w();
if p(m) break;
}
for (;k<10;k++)
;k<10;k++ {
r();
}
12/08/21 CMPE 310 Winter 2001 32
 How do we select the
strategy?
 The requirements of reliability of the
system
 The resources available
 The kind of programs
 functional languages do not have loops…
 The tools available to support the
testing process

12/08/21 CMPE 310 Winter 2001 33

 Black box testing

requirements

output

input

events

12/08/21 CMPE 310 Winter 2001 34

 Categories of errors in
black box testing
 incorrect of missing functions
 interface errors
 errors in data structures or external
database access
 performance errors
 initialization and termination errors

12/08/21 CMPE 310 Winter 2001 35

 Example
abs(integer x)
Return x if x >= 0
Return -x if x<0
 The concrete value of x does not

matter!

12/08/21 CMPE 310 Winter 2001 36

 Issues on black-box testing
 How is functional validity tested?
 What classes of input will make good test cases?
 Is the system particularly sensitive to certain input
values?
 How are the boundaries of a data class isolated?
 What data rates and data volume can the system
tolerate?
 What effect will specific combinations of data have
on system operations?

12/08/21 CMPE 310 Winter 2001 37

 Techniques for black box testing
 Graph-Based Testing Methods
 Equivalence Partitioning
 Boundary Value Analysis
 Comparison Testing

12/08/21 CMPE 310 Winter 2001 38

 Not required for exams

Graph-Based Testing
 Complex Systems are hard to understand
 They can be modeled as interacting objects
 Test can be performed to ensure that all the
required relations are in place

Pressman, p. 472

12/08/21 CMPE 310 Winter 2001 39

 Equivalence partitioning
Partitioning is based mouse picks on menu
on input conditions
output
user format
queries requests
responses
to prompts
numerical
data

command key input

12/08/21 CMPE 310 Winter 2001 40
 Equivalence classes (1)
Kinds of Valid data
 user supplied commands
 responses to system prompts
 file names
 computational data
 physical parameters

 bounding values

 initiation values

 output data formatting commands

 responses to error messages
 graphical data (e.g., mouse picks)

12/08/21 CMPE 310 Winter 2001 41

 Equivalence classes (2)
Kinds of Invalid data
 data outside bounds of the program
 physically impossible data
 proper value supplied in wrong place

12/08/21 CMPE 310 Winter 2001 42

 Defining equivalence classes
 Input condition is a range: one valid
and two invalid classes are defined
 Input condition requires specific value:
one valid and two invalid classes are
defined
 Input condition is boolean: one valid
and one invalid class are defined

12/08/21 CMPE 310 Winter 2001 43

 Boundary value analysis
mouse picks on menu

output
user format
queries requests
output responses
domain to prompts
numerical
data

command key input

12/08/21 CMPE 310 Winter 2001 44
 Boundary value testing
 Range a..b
 test cases: a, b, just above a, just below b
 Number of values:
 test cases: max, min, just below min, just
above max
 Output bounds should be checked
 Boundaries of externally visible data
structures shall be checked (e.g. arrays)
12/08/21 CMPE 310 Winter 2001 45
 Comparison testing
 Safety critical systems
 Develop several implementations for a specification
 Run tests with all and compare results
if not identical: investigate in
detail and/or go on a majority
criteria
 Problems:
 expensive
 errors in specification
 all implementations return the same but incorrect result

12/08/21 CMPE 310 Winter 2001 46

 Automating software testing
 Manual software testing is time
consuming
 Software testing has to be repeated
after every change (regression testing)
 Write test drivers that can run
automatically and produce a test report

12/08/21 CMPE 310 Winter 2001 47