1

Software Quality Assurance

Đảm bảo chất lượng phần mềm
Lecture 5: White Box Approach

2
Contents
• Control Flow Testing
• Predicate Testing
• Data Flow Testing
• Unit-level Testing
• Mutation Testing

3
5.1. White Box Testing

4
 Effective Test Case Design
• Exhaustive testing (use of all possible inputs and conditions) is
impractical
• Must use a subset of all possible test cases
• Must have high probability of detecting faults
• Need processes that help us selecting test cases
• Effective testing - detect more faults
• Focus attention on specific types of faults
• Know you are testing the right thing
• Efficient testing - detect faults with less effort
• Avoid duplication
• Systematic techniques are measurable and repeatable

5
Basic Testing Strategies
Basic Testing Strategies
• Motivation:
• Effective Test
Case Design
• Compare 2
approaches
• Black Box
• White Box
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

6
Black Box vs. White Box
• Black Box: functional
Black-Box vs. White-Box
testing
• Unit test System
• Integration test
Specification
• System test Implementation

• Programmers & Test

Engineers & Quality
Assurance Engineers Missing functionality:
Cannot be revealed by white-
Unexpected functionality:
Cannot be revealed by black-box
box techniques techniques
• White Box: structural
testing Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

• Unit test
• Integration test
• Programmers & Test
Engineers

7
5.2. Control Flow Testing
Kiểm thử luồng điều khiển

IT4501 - Software Engineering Department -

9/27/21 8
SoICT/HUST
Basic terms
• Program unit: entry point to exit point
• Procedures, Functions, Modules, Components
• 2 kinds of statement
• Assignment statement
• Conditional statement
• Program path
• A sequence of statements from entry point to exit point of a
program unit
• A program unit may contains many program paths
• An execution instance of the unit
• A given set of input data, the unit executes a different
path

9
Control Flow Graph
Đồ thị luồng điều khiển
CFG symbols
• Represent the 4.3 CONTROL FLOW

graphical structure of
a program unit
True False
• A sequence of Computation Decision

statements from
entry point to exit
point of the unit
Sequential computation Decision point Merge point
depicted using graph
Figure 4.2 Symbols in a CFG.
notions
computation. A maximal sequential computation can be represented
Lund Universitysingle rectangle
/ Faculty of or by many
Engineering/ Department rectangles,
of Computer eachEngineering
Science / Software corresponding
Research Groupto one stat
source code.
We label each computation and decision box with a unique inte
branches of a decision box are labeled with T and F to represent the t
evaluations, respectively, of the condition within the box.10We will not l
node, because one can easily identify the paths in a CFG even witho
 Control Flow Testing
White-Box Testing (Ch 4, 5)
• Main idea: select a few paths Exhaustive Testing
in a program unit and observe
whether or not the selected If-then-else There are many possible paths!
paths produce the expected 520 (~1014 ) different paths
outcome
• Executing a few paths while
loop < 20x
trying to assess the behavior
of the entire program unit
Selective Testing
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

10

11
 Select as few paths as possible
Selective White-box Testing
• Selecting relevant paths
based on some criteria Selective:
a selected path
• All paths
• Statement Coverage
üControl flow testing
• Branch Coverage
üData flow testing
• Predicate Coverage üLoop testing
• Most applicable to new üFault-based testing
software for unit test
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

12
 Outline of Control Flow Testing
90 CHAPTER 4 CONTROL FLOW TESTING

• Inputs Path
selection
• Source code of unit Work criteria

• Path selection criteria

process
• Generate CFG: draw CFG Program
unit
Draw a
control flow Control
flow graph
Select
paths
Selected
paths
graph
from source code of the unit
Inputs
• Selection of paths: selected Generate
paths to satisfy path test input
data
selection criteria
• Generation of test input data No
Are
the selected
paths
feasible?

Yes

Process of generating test input data

Fig 4.1 Output Test input

data
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

Figure 4.1 Process of generating test input data for control flow testing.

Selection of Paths: Paths are selected from the CFG to satisfy the path selec-
13
tion criteria, and it is done by considering the structure of the CFG.
Generation of Test Input Data: A path can be executed if and only if a
 Path selection criteria

• Example: Life Insurance Example

Age, Gender 1

• Given the source 2

bool AccClient(agetype T if(female) F
code of the function age; gndrtype gender) T 3 F
AccClient bool accept
Age <85
T 4
Age <80
F

if(gender=female)
• Draw the CFG accept := age < 85;
Accept = true 5 Accept = false 6
else
accept := age < 80;
Return accept 7
return accept

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

14
 Path selection criteria
• Example:
• Given the source Life Insurance Example
Age, Gender 1

code of the 2
bool AccClient(agetype T if(female) F
function AccClient age; gndrtype gender) T 3 F
Age <85

• Draw the CFG bool accept

if(gender=female)
T 4
Age <80
F

accept := age < 85;

Accept = true 5 Accept = false 6
else
accept := age < 80;
Return accept 7
return accept

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

15
All path coverage criterion
Tiêu chí lựa chọn tất cả các đường dẫn
• Objective: Design all All paths
Female Age < 85 Age < 80
possible test cases so Yes Yes Yes Age, Gender 1

Yes Yes No
that all paths of the Yes No Yes T
2
if(female) F

program are executed Yes

No
No
Yes
No
Yes
T 3
Age <85
F
4 F
T
No Yes No Age <80

• 4 test cases satisfy the No No Yes

No No No
all path coverage <Yes,Yes,*> 1-2(T)-3(T)-5-7
Accept = true 5 Accept = false 6

criterion <Yes,No,No> 1-2(T)-3(F)-6-7 Return accept 7

<No,Yes,Yes> 1-2(F)-4(T)-5-7
<No,*,No> 1-2(F)-4(F)-6-7
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

16
Statement coverage criterion
Tiêu chí bao phủ hết tập câu lệnh
Statement Coverage
• Main idea: Execute bool AccClient(agetype
each statement at age; gndrtype gender) Age, Gender 1

bool accept
least once if(gender=female)
T if(female)
2
F

• A possible concern accept := age < 85;

else
T
Age <85
F
T
Age <80
4
F

may be: accept := age < 80;

6
return accept 5

• dead code Accept = true Accept = false

AccClient(83, female)->accept Return accept

7

AccClient(83, male) ->reject

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

17
Disadvantages of statement coverage
• Statement coverage is the weakest, indicating the fewest
number of test cases
• Bugs can easily occur in the cases that statement coverage
cannot see
• The most significantly shortcoming of statement coverage
is that it fails to measure whether you test simple If
statements with a false decision outcome.

18
 Branch coverage criterion
Tiêu chí bao phủ nhánh
• Also called Decision Coverage
• A branch is an outgoing edge from a node
• A rectangle node has at most one out going branch
• All diamond nodes have 2 outgoint branches
• A decision element in a program may be one of
• If – then
• Switch – case
• Loop
• Main idea: selecting paths such that every branch is included
in at least one path

19
Example

AccClient(83,
Branch Coverage /1 female)->accept

• We test the path bool AccClient(agetype Age, Gender 1

age; gndrtype gender)

• 1-2(T)-3(T)-5-7 bool accept
T
2
if(female) F

if(gender=female) T 3
Age <85
F
4 F
accept := age < 85; T
Age <80
true
else
accept := age < 80; Accept = true 5 Accept = false6
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

19

20
Example
AccClient(83, male)
Branch Coverage /2 ->reject

• We test the path bool AccClient(agetype Age, Gender 1

age; gndrtype gender)

• 1-2(F)-4(F)-6-7 bool accept
T if(female) 2 F

if(gender=female) T 3
Age <85
F
T 4 F
accept := age < 85; Age <80

else
accept := age < 80; Accept = true 5 Accept = false 6
false
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

21
Example

AccClient(78, male)-
Branch Coverage /3 >accept

• We test the path bool AccClient(agetype Age, Gender 1

• 1-2(F)-4(T)-5-7 age; gndrtype gender)

bool accept
T if(female) 2 F

if(gender=female) T 3
Age <85
F
4 F
accept := age < 85; T
Age <80
true
else
accept := age < 80;
Accept = true 5 Accept = false 6
true
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

22
Example
AccClient(88,
Branch Coverage /4 female) ->reject

• We test the path bool AccClient(agetype Age, Gender 1

age; gndrtype gender)

• 1-2(T)-3(F)-6-7 bool accept T
2
if(female) F

if(gender=female) T 3 F
Age <85
accept := age < 85; T 4
Age <80
F
false
else
accept := age < 80;
false Accept = true 5 Accept = false 6
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

23
Comparing 3 criteria
• (1) All path coverage: assure 100% paths executed
• (2) Statement coverage: pick enough paths to assure that
every source statement is executed at least once
• (3) Branch coverage: assure that every branch has been
exercised at least once under some test
• (1) implies (3), (3) implies (2)
• These 3 criteria are also called as Path Testing Techniques

24
Example of statement and branch coverage

25
Example of statement and branch coverage
• Question 1:
• Does every decisions have a T and F values?
• Yes è Implies branch coverage
• Question 2:
• Is every link covered at least once?
• Yes è Implies statement coverage

Decisions Process Link

Path
4 6 7 9 a b c d e f g h i j k l m
abcde T T x x x x x
abhkgde F T F x x x x x x x
abhlibcde F T T x x x x x x x x
abcdfjgde T F T x x x x x x x x
abcdfmibcde T F F x x x x x x x

26
Example: Exponential Function

27
Example: Exponential Function

28
Example: Bubble sort

29
Example: Bubble sort

30
Limitations of path testing techniques
• Path Testing is applicable to new unit
• Limitations
• Interface mismatches and mistakes are not taken
• Not all initialization mistakes are caught by path testing
• Specification mistakes are not caught

31
5.3. Predicate Coverage
Tiêu chí bao phủ điều kiện

IT4501 - Software Engineering Department -

9/27/21 32
SoICT/HUST
 Basic concepts
• Predicate: are expresions that can be evaluated to a boolean
value (true or false)
• Predicate may contain:
• Boolean variables
• Non-boolean variables that are compared with the relational operators {≥, ≤, =, ≠
, <, >}
• Boolean function calls
• The internal structure is created by logical operators:
• ¬, →, ↔,∧,∨, ⨁
• A clause: is a predicate that does not contain any of the logic operator.
Example: (a=b) ∨ C ∧ p(x) contains 3 clauses

33
How to create a Path Predicate Expression?

• Write down the predicates

for the decisions you meet
along a path
• The result is a set of path
predicate expressions
• All of these expressions must
be satisfied to achieve a
selected path

34
Predicate Faults
Các lỗi điều kiện có thể xảy ra
• An incorrect Boolean operator is used
• An incorrect Boolean variable is used
• Missing or extra Boolean variables
• An incorrect relational operator is used
• Parentheses are used incorrectly

35
 Predicate Coverage

• To ensure all predicates in the

source code are implemented
correctly
• Main idea:
• For each predicate p, test
cases must assure that p
evaluates to true at least
once and p evaluates to
false at least once

36
Example

37
 Clause Coverage
• Main idea: For each
individual condition
(clause) c, c should
evaluate to true and c
should evaluate to false
at least once.

38
Example

39
 Predicate vs. Clause Coverage
• Does Predicate coverage
subsume clause coverage?
• Does clause coverage
subsume predicate coverage?
• Naturally, we want to test
both the predicate and
individual clauses

40
Predicate and Clause Coverage
• PC does not fully exercise all the clauses
• For example: p = a v b. In most programming languages, at run
time, when a evaluates to True, p evaluates to True and b does not
exercise.
• CC does not always ensure PC
• This is, we can satisfy CC without causing the predicate to
be both true and false
• This is definitively not what we want
• We need to come up with another approach
• Inversely, with PC, it is not always that all individual clauses
evaluate to both true and false

41
 Combinatorial Coverage
Bao phủ kết hợp
• Main idea: For each
predicate p, test cases
should ensure that the
clauses in p should
evaluate to each possible
combination of truth
values

42
Example 1
• Write all the clauses and the Combination of Clauses (CoC)
of the given predicate P = ((a >b) v C) ∧ p(x)

43
Example 1 - Answer

44
What is the problem with combinatorial
coverage?
• Combinatorial coverage is very expensive if we have
multiple clauses in the predicate
• For each decision point with N conditions, we need 2^N
test cases

45
Motivation
• Predicate should evaluate to both true and false in test
cases
• Each individual condition (clause) is tested independently
of each other
• Each individual clause should affect the predicate
• i.e., changing the value of an individual clause, the value of
predicate is also changed

46
Active clause
47
• Major clause: the
clause which is being
focused upon
• Minor clause: all other
clauses in the predicate
• Determination: clause
c in p determines p

47
Example: P = a ⋀ (b ⋁ c)
• Identify, when considering a as active clause, which values
should be assigned to b and c?
• Answer:
• a is active clause, the values of b and c are those such that
changing the value of a changes the value of P.
• So, (b v c) = 1, the value of P is determined by the value of a. We
can assign b = 1, c = 1
• Test case T = {(a=T, b=T), (a=F,b=F)} satisfy GACC, but not
PC, all both cases, P = T

48
Active Clause Coverage (ACC)
• For each predicate p and each major clause c of p, choose
minor clauses so that c determines p. 2 test requirements
for assuring ACC: for each c: c evaluates to true and c
evaluates to false
• With a decision point consists of N conditions, only N+1
test cases are required. Why?

49
Example

50
Example - Answer

51
Resolving the Ambiguity problem
Giải quyết vấn đề nhập nhằng
• With the example p =
a v (b ∧ c), a is active
clause
• How we can choose
the value of b and c? c
should be different
from b or not?

52
General Active Clause Coverage (GACC)
• For each clause c, choose minor
clauses’ values such that c
determines p
• Clause c has to evaluate to both
true and false
• Minor clauses don’t need to be
the same when clause c is true as
well as when c is false.
• cj(ci=true)=cj (ci = false) for all cj
OR cj(ci=true)!=cj (ci = false) for all
cj

Does GACC subsume predicate coverage? NOT always, ex: P = A <-> 53

9/27/21 B

53
General Active Clause Coverage (GACC)

a and (b or c) 1 0

a is active 1. . 0 . .

b is active . 1 . . 0 .

c is active . . 1 . . 0

All active values are in diagonal

54
General Active Clause Coverage (GACC)

a and (b or c) 1 0
a is active 11 0 0 0 1
Values of minor clauses are different
when active clause evaluates to true b is active 11 0 1 0 0
and false
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 0 1, 1 0 0

55
 General Active Clause Coverage (GACC)

a and (b or c) 1 0
a is active 11 0 0 1 0
Values of minor clauses are the same
when active clause evaluates to true b is active 11 0 1 0 0
and false
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 1 0, 1 0 0

56
Example

57
GACC and subsumption of CC/PC
• By definition: GACC subsumes CC, that is, satisfying GACC
leads to satisfy CC, but not PC
• Example: P = a <->b.
• Test case T = {(a=T, b=T), (a=F,b=F)} satisfy GACC, but not PC, all both
cases, P = T
• Test case T = {(a=T, b=F), (a=F,b=T)} satisfy GACC too, but not PC either, all
both cases, P = F

58
Correlated Active Clause Coverage (CACC)
• For each clause ci in P, choose minor clauses cj such that ci
determines the predicate P
• Clause ci has to evaluate to true or false in test cases
• The values chosen for the minor clauses cj must cause P to
be true for one value of the major clause ci and false for the
other value of ci
• P(ci = true) != P(ci = false)
• Minor clauses don’t need to be the same. That is:
• cj(ci=true)=cj (ci = false) for all cj OR cj(ci=true)!=cj (ci = false) for all cj

59
 Example
• When a is active
clause, which rows
can be chosen to
satisfy CACC?
• When a is active, values
of b and c are chosen
such that P(a=true) !=
P(a=false)
• Any of rows 1, 2, 3 and
any of row 5,6,7 - one
of total nine pairs
satisfies CACC

60
Test cases for CACC

a and (b or c) 1 0
a is active 11 0 0 1 0
b is active 11 0 1 0 0
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 1 0, 1 0 0

61
CACC and subsumption of GACC/CC/PC
• By definition, CACC subsumes GACC thus CC and also PC
• Example: P = a<->b
• Which test cases satisfy PC?
• Which test cases satisfy CACC?

62
Example: P = a ⋀ (b ↔ c)
• Which test cases
satisfy GACC?
• Which test cases
satisfy CACC?

63
Restricted Active Clause Coverage
(RACC)
• For each predicate P and each active clause ci in P, choose
minor clause cj so that ci determines P. ci evaluates to both
true and false in test cases
• Predicate P evaluates to true in one case of major clause
and false in the other (that is CACC)
• The values choosen for minor clauses cj must be the same
when ci evaluates to true as when ci evaluates to false.
• That is, cj(ci=true)=cj (ci = false) for all cj
• This is the stricter version of CACC

64
 RACC Example
• When a is active clause,
which rows can be
chosen to satisfy RACC?
• Rows 1 and 8 are not
chosen because a is
not active in these
two cases
• b and c should have
the same values as
when a is true as well
as when a is false
• One of three pairs:
{1,5}, {2, 6} and {3, 7}
satisfy RACC

65
Test cases for RACC

a and (b or c) 1 0
a is active 11 0 0 1 0
b is active 11 0 1 0 0
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 1 0, 1 0 0

66
RACC and subsumptions of
CACC/GACC/PC/CC
• By definition, RACC subsumes CACC, thus subsumes GACC, CC
and PC
• RACC often leads to infeasible test requirements
• Example: Consider the predicate P = ((a > b) && (b < c)) || (c>a) =
(X && Y)|| Z
• When Z is active clause, we choose X = 1, Y = 0. Infeasible test case: Z
= 0, X = 1, Y = 0

67
Logical Operators in Source Code
• & | ~ ^ correspond also to
logical operators.
• What is the difference
between (a&&b) and
(a&b)? (a||b) and (a|b)?

68
Code transformation

if if (a){
(a&&b) if (b)
S1; S1;
else else
S2; S2;
} else
S2;

69
 Exercise 1

• Identify test cases for

• PC
• CC
• CoC
• GACC
• CACC
• RACC

70
Exercise 2
Example: predicates/clauses in source code

• Determine predicates public static int daysInMonth(int m, int y) {!

if (m <= 0 || m > 12)!
and clauses in the source throw new IllegalArgumentException("Invalid month: "
if (m == 2) {!
+ m);!

code if (y % 400 == 0 || (y % 4 == 0 && y % 100 != 0)) !

return 29;!
• Identify reachable else!
return 28;!
predicates of the source }!
if (m <= 7) {!
code if (m % 2 == 1)!
return 31;!
Predicates and clauses
p1: c1 || c2 ; c1: m <= 0, c2: m > 12!
• Identify test cases that }!
return 30;!
p2: c3; c3: m == 2; !
satisfy PC, CC and CoC if (m % 2 == 0)!
return 31;! p3: c4 || (c5 && c6) ; c4: y % 400 == 0,
c5: y % 4 == 0, c6: y % 100 != 0; !
return 30;!
• Are there infeasible } p4: c7; c7: m <= 7; !

requirements ? p5: c8; c8: m % 2 == 1; !

p6: c9; c9: m % 2 == 0

71
 Exercise 2
• Determine predicates
and clauses in the Exercise 3
source code public static int daysInMonth(int m, int y) {!
if (m <= 0 || m > 12) // p1!

• Identify reachable
throw new IllegalArgumentException("Invalid month: " + m);!
if (m == 2) { // p2!
predicates of the if (y % 400 == 0 || (y % 4 == 0 && y % 100 != 0)) // p3!
return 29;!
source code else!
return 28;!
Reachability predicates - r(p)

• Identify test cases }!

if (m <= 7) { // p4!
r(p1) = true (always reached)!

that satisfy PC, CC and if (m % 2 == 1) // p5!

return 31;!
r(p2) = ¬ p1 = m >= 0 ⋀ m < 12!

r(p3) = r(p2) ⋀ p2 = (m > 0 ⋀ m < 12 ⋀ m == 2)

CoC }!
return 30;! = (m == 2)!

• Are there infeasible if (m % 2 == 0) // p6!

return 31;!
r(p4) = r(p2) ⋀ ¬ p2!

r(p5) = r(p4) ⋀ p4 = (m > 0 ⋀ m < 12 ⋀ m != 2)

requirements ? }
return 30;! ⋀ m <= 7!

r(p6) = r(p4) ⋀ ¬ p4 = (m > 0 ⋀ m < 12 ⋀ m !=

2) ⋀ m > 7

Identify test cases that satisfy PC, CC, CoC (in this order).
13 Are there infeasible requirements?

72
PC Test Cases

Test Input
# PC m y
1 p1 0 -
2 ¬p1 1 -
3 p2 2 -
4 ¬p2 1 -
5 p3 2 2000
6 ¬p3 2 2017
7 p4 3 -
8 ¬p4 8 -
9 p5 3 -
10 ¬p5 4 -
11 p6 8 -
12 ¬p6 9 -

73
 CC Test Cases

# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
CC c1 ¬c1 c2 ¬c2 c3 ¬c3 c4 ¬c4 c5 ¬c5 c6 ¬c6 c7 ¬c7 c8 ¬c8 c9 ¬c9

m 0 1 13 1 2 1 2 2 2 2 2 2 3 8 3 4 8 9
y - - - - - - 2000 2017 2000 2017 2017 2000 - - - - - -

74
CoC Test Case

# c1 c2 c3 c4 c5 c6 c7 c8 c9
1 T T infeasible
2 T F m=0
3 F T m=13
4 F F m=1
5 T m=2
6 F m=1
7 m=3 T
8 m=8 F
9 m=3 T
10 m=4 F
11 m=8 T
12 m=9 F

75
CoC Test Case

# c1 c2 c3 c4 c5 c6 c7 c8 c9

13 infeasible T T T

14 m=2, y=2000 T T F

15 infeasible T F T

16 infeasible T F F

17 m=2, y=2016 F T T

18 m=2, y=2100 F T F

19 m=2, y=2017 F F T

20 infeasible F F F

76
Exercise 3 Exercise 4
public static TClass triangleType(int a, int b, int c) {!
if (a <= 0 || b <= 0 || c <= 0) /* p1 */!
return INVALID;!
if (a >= b + c || b >= a+c || c >= a+b) /* p2 */!
return INVALID;!
int count = 0;!
if (a == b) /* p3 */! Identify:
count++;!
if (a == c) /* p4 */! 1) the reachability predicates;
count++;!
if (b == c) /* p5 */! 2) TR(CC) and TR(PC)
count++;!
if (count == 0) /* p6 */! 3) test cases that satisfy a) PC b)
return SCALENE;! CC
if (count == 1) /* p7 */!
return ISOSCELES;! 4) determination predicates for
return EQUILATERAL; ! the clauses of p1 and p2
}
5) TR(CACC)
6) test cases that satisfy a)
CACC b) RACC [are there
infeasible requirements?]
28

77
5.4. Data Flow Testing
Kiểm thử luồng dữ liệu

IT4501 - Software Engineering Department -

9/27/21 78
SoICT/HUST
Motivation
• S2: define x, assign to
x a value
• S6: use x to assign to
other variable
• If Branch Coverage
• 1-2-4-5-7
• 1-3-4-6-7
• We cannot explore
simultaneously the
relationship between
definition of x in
statement 2 and the
use of x in statement 6

79
Motivation
• GOAL: Try to ensure that values are computed and used
correctly
• Consider: how data gets accessed and modified in the system
and how it can get corrupted
• Common access-related bugs:
• Using an undefined or uninitialized variable
• Deallocating or reinitializing a variable before it is constructed,
initialized or used
• Deleting a collection object leaving its members
unaccessible

80
Variable Definition
Định nghĩa biến
• A program variable is DEFINED whenever its value is modified
• on the left hand side of an assigment statement, e.g., y = 20
• in an input statement, e.g., read(y)
• as an call-by-reference parameter in a subroutine call, e.g.,
update(x, &y)

81
Variable Use
Sử dụng biến
• A program variable is USED whenever its value is read:
• On the right hand side of an assignment statement, e.g., y = x +
17, x is used, y is defined
• as an call-by-value parameter in a subroutine or function call,
e.g., y = sqrt(x)
• in the predicate of a branch statement, e.g., if (x > 0){...}

82
 Variable use: p-use and c-use
• Use in the predicate of a branch statement is a predicate-use or p-
use
• Any other use is a computation-use or c-use
• For example, in the code below, there is a p-use of x and a c-use of y

if (x > 0) {
print(y);
}

83
Variable Use
• A variable can also be used and then re-defined in a single
statement when it appears
• on both sides of an assignment statement, e.g., y = y + x
• as an call-by-reference parameter in a subroutine call, e.g.,
increment(&y)

84
 Terminology
Các thuật ngữ khác

• Definition-Clear-Path
• A path is definition-clear (“def-
clear”) with respect to a
variable v if it has no variable
re-definition of v on this path
• p1 is def-clear path of v while
p2 is not

85
 Terminology
Các thuật ngữ khác

• Complete-Path
• A path is complete if the initial
node of this path is a entry
node and the final node of this
path is an exit node
• p1, p2 are not complete while
p3 is

86
 Definition-Use Pair (DU-pair)
Cặp định nghĩa – sử dụng

• A definition-use pair (du-pair) with respect

to a variable v is a pair (d, u) such that
• d is a node defining v
• u is a node or edge using v
• When it is a p-use of v, u is an out-
going edge of the predicate
statement
• There is a def-clear path with respect
to v from d to u

87
Example

1. input(A,B)
if (B>1){
2. A = A + 7
}
3. if (A > 10){
4. B = A + B
}
5. output(A,B)

88
Example: Identifying DU-Pairs of
Variable A

89
Example: Identifying DU-Pairs of
Variable A

90
Example: Identifying DU-Pairs of
Variable A

91
Example: Identifying DU-Pairs of
Variable B

92
Data Flow Test Coverage Criteria
Các tiêu chí kiểm thử bao phủ luồng dữ liệu

• All-defs coverage – Bao phủ tất cả các điểm định nghĩa dữ liệu trên đồ thị
• All-uses coverage – Bao phủ tất cả các điểm sử dụng dữ liệu trên đồ thị
• All-DU-Paths coverage – Bao phủ tất cả các đường dẫn DU trên đồ thị
• All-p-uses/Some-c-uses coverage – Bao phủ tất cả các điểm sử dụng dữ
liệu trong các câu lệnh rẽ nhánh và một vài điểm sử dụng dữ liệu
• All-c-uses/Some-p-uses coverage – Bao phủ tất cả các điểm sử dụng dữ
liệu và một vài điểm sử dụng dữ liệu trong các câu lệnh rẽ nhánh
• All-p-uses coverage – Bao phủ tất cả các loại sử dụng dữ liệu trong điều
kiện
• All-c-uses coverage – Bao phủ tất cả các loại sử dụng thông thường của
dữ liệu

93
All-Defs Coverage
Bao phủ tất cả các điểm định nghĩa
• For every program variable v, at least one def-clear path from
every definition of v to at least one c-use or one p-use of v must
be covered

94
 Example 1

• All Defs requires

• d0(x) to a use
• Satisfactory Path:
• <0-1-2-4-6>

95
 Example 2

• Identify all test cases to satisfy All-

defs
• Two variables: A and B
• Consider a test case executing
path
• t1: <1,2,3,4,5>
• t1 satisfy All-defs or not?

96
Example – Considering Variable A

97
Example – Considering Variable B

98
Answer
• Since <1,2,3,4,5> covers at least one def-clear path
from every definition of A or B to at least one c-use
or p-use of A or B è At least one du-pair from every
definition of A (n1 and n2) and of B (n1, n4) are
covered!
• All-Defs coverage is achieved

99
All-Uses Coverage
Bao phủ tất cả các điểm sử dụng
• For every program variable v, at least one def-clear path
from every definition of v to every c-use and every p-use
(including all outgoing edges of the predicate statement) of
v must be covered
• Requires that all du-pairs covered

100
Example 1
• All-Uses requires
• a: d0(x) to a u2(x)
• b: d0(x) to u3(x)
• c: d0(x) to u5(x)
• Satisfactory Path:
• <0-1-2-4-5-6>
satisfies a, c
• <0-1-3-4-6>
satisfies b
Example 2

• Consider two executing paths:

• t2: <1,3,4,5>
• t3: <1,2,3,5>
• Do all three test cases t1, t2,
t3 provide All-Uses coverage?

102
Example: Def-clear paths by t2 for
Variable A

103
Example: Def-clear paths by t2 for
Variable B

104
Example: Def-clear paths by t3 for
Variable A

105
Example: Def-clear paths by t3 for
Variable B

106
Answer
• Since all the du-pairs for variables A and B are
covered by 3 test cases è All-Uses coverage is
achieved
• Note: all du-pairs means: at least one def-clear path
from every definition to every c-use/p-use of the
variable

107
Example 3

108
Du-Pairs with respect to Variable X

109
Du-Pairs with respect to Variable X

110
Du-Pairs with respect to Variable X

111
Du-Pairs with respect to Variable X

112
Du-Pairs with respect to Variable X

113
Du-Pairs with respect to Variable X

114
More Dataflow Terms and Definitions
• A path (either partial or complete) is simple if all edges
within the path are distinct, i.e., different
• A path is loop-free if all nodes within the path are
distinct, i.e., different

115
Example: Simple and Loop-Free Paths

116
DU-Path
• A path <n1, n2, ... , nj, nk> is a du-path with respect to a
variable v, if v is defined at node n1 and either:
• there is a c-use of v at node nk and <n1, n2, ... , nj, nk> is a def-
clear simple path, or
• there is a p-use of v at edge <nj, nk> and <n1, n2, ... , nj> is a def-
clear loop-free path

117
Example 3: Identify du-path

118
 All DU-Paths Coverage

For every program

variable v, every du-
path from every
definition of v to
every c-use and
every p-use of v
must be covered

119
Example 2: All DU-
paths coverage

• Identify all DU-Paths

for variable A and B

120
Example 3: All DU-
paths coverage

• Identify all DU-

Paths for variable X
and Y

121
Data Flow Testing
Summary

• Relationship between
Data Flow Testing and
Path Testing
• Path testing with
Branch Coverage and
Data-flow testing with
All-uses is a very good
combination in
practice
Exercise 1

• Draw a data flow

graph for the
binsearch function
• Find a set of complete
paths satisfying all-
defs criterion with
respect to variable mid
• Do the same for
variable high
Exercise 2

• Using All-DU paths to

test the pow function
• Using All-Uses
combining with
Branch Coverage to
test this function
5.5. Mutation Testing
Kiểm thử đột biến

125
Mutation Testing
• Mutation testing (Kiểm thử đột biến) is a structural
testing method
• i.e., We have to use the structure of code to guide the test
process
• We cover the following aspects of Mutation Testing
• What is a mutation?
• What is mutation testing?
• When should we use mutation testing?
• Mutations
• Mutation Testing tools

126
What is a mutation?
• A mutation is a small change in the program
• Such small changes are intended to model low level
defects that arise in the process of coding systems
• Ideally, mutations should model low-level defect
creation

127
When should we use mutation
testing?
• Structural test suites are dỉrected at identifying defects
in the code
• One goal of mutation testing: To assess or improve
the efficacy of test suites in discovering defects
• Defects remaining in the code is measured through
Residual Defect Density (RDD)
• RDD is measured in defects per thousand lines of code
• Given the program under test P
• P satisfies all the tests in a test suite T
• Mutation testing on P provide an estimate of RDD on P

128
Using Mutation Testing to estimate RDD

• Given program P, test suite T

• Suppose we have 𝑟 , an estimate of RDD on P – could
be based on the number of faults our tests have
already detected
• Generate n mutants of P
• Test each mutant with the test suite T
• Find the number, k, of mutants that are killed by T
• RDD = r x (n-k)/k
• k/n is a measure of the adequacy of T in finding defects
in P

129
An approach to Mutation
• Consider mutation operators in the form of rules that
match a context and create some systematic mutation
of the context to create a mutant
• Simple approach: to consider all possible mutants but
this may create a very large number of mutants

130
Kinds of mutation
• Value Mutation
• Changing the values of constants or parameters (by
adding or subtracting a value)
• e.g., changing loop bounds
• Decision Mutation
• Modifying conditions to reflect potential slips and errors
in the coding of conditions in program
• e.g., replacing < by >
• Statement Mutation
• Deleting certain lines to reflect omissions in coding
• Swapping the order of lines of code
• Changing operations in arithmetic expressions
• ...

131
Mutations for Inter-class Testing

132
Value Mutation (đột biến giá trị)
• Typical examples:
• Changing values to one larger or smaller
• Swapping values in initializations
• The commonest approach is to change constants by
one in an attempt to generate a one-off error
• Coverage criterion: Perturb all constants in the
program or unit at least once or twice

133
Value Mutation - Example

134
Decision Mutation
(Đột biến điều kiện)
• Typical examples:
• Modelling “one-off” error by changing < to <= or vice versa
(common in checking loop bounds)
• Modelling confusion about larger and smaller by changing < to
> or vice versa
• Getting parenthesisation wrong in logical expressions:
mistaking precedence between && and ||
• Coverage Criterion: consider one mutation for each
condition in the program

135
Decision Mutation
(Đột biến điều kiện)

136
Statement Mutation
(Đột biến câu lệnh)
• Typical examples:
• Deleting a line of code
• Duplicating a line of code
• Permuting the order of statements
• Coverage Criterion:
• Applying this procedure to each statement in the program

137
Statement Mutation
(Đột biến câu lệnh)

138
Mutation Testing Tools
• Stryker JS: mutation testing for javascript
• Mutant: automated code reviews via mutation testing
• Infection: PHP mutation testing library
• Pitest: State of the art mutation testing for the JVM
• Cosmic Ray: mutation testing for Python
• Muter: automated mutation testing for Swift
• ...

139
 Thank you
for your
attention!!!

140