Unit 3

Software Testing Strategies

White Box Approaches
Based on the following textbooks:
D. Galin, Software Quality Assurance: From Theory to Implementation, 1st ed. Addision-
Wesley, 2004

Ilene Burnstein, Practical Software Testing: A Process-Oriented Approach, Springer, 2003.

Outline
• Introduction to White Box Testing
• Test Adequacy Criteria
• Coverage and Control Flow Graphs
• Path Coverage
• The McCabe’s Cyclomatic Complexity Metric
• Additional White Box Test Design Approaches
• Data Flow
• Loop Testing
• Mutation Testing
Introduction to White Box Testing
• White box testing is a complementary approach to black box test case design in
which the tester has knowledge of the internal structure of the SUT.
• The knowledge needed for white box testing often becomes available to the
tester in the later phases of the software life cycle, specifically during the detailed
design and coding phases of development. Therefore, white box testing follows
black box testing.
• Black box testing approaches can be used for both small and large software
components, whereas white box testing is most useful when testing small
components.
Test Adequacy Criteria
• The goal for white box testing is to ensure that the internal components of a
program are working properly.
• A common focus is on structural elements such as statements and branches.
• The tester develops test cases that exercise (execute) these structural elements to determine
if defects exist in the program structure.
• By exercising all of the selected structural elements the tester improves the chances for
detecting defects.
• Testers need a framework for deciding which structural elements to select and for
deciding when the testing efforts are adequate enough.
• Such a framework exists in the form of test adequacy criteria.
Test Adequacy Criteria
• Definition: a test adequacy criterion is a stopping rule.
• A program is said to be adequately tested with respect to a given criterion if all of
the target structural elements have been exercised according to the selected
criterion.
• An Example: A test set T is statement adequate for program P if it causes all the statements in
P to be executed.
• A test adequacy criterion can also be expressed as a percent (which is called the
degree of coverage).
• An Example: A test set T satisfies 50% statement adequate for program P if it causes 50% of
the statements in P to be executed.
Test Adequacy Criteria
• The planned degree of coverage is specified in the test plan and then measured
when the tests are actually executed.
• The planned degree of coverage is usually specified as 100%, however, under
some circumstances the planned degree of coverage may be less than 100% due
to the following:
• The nature of the unit.
• The lack of resources
• Other project-related issues such as timing, scheduling, and marketing constraints.
Coverage and Control Flow Graphs
• The application of coverage analysis is typically associated with one of the
following program structural elements:
• Program statements
• Decisions/branches
• Conditions
• Combinations of decisions and conditions
• Paths
Coverage and Control Flow Graphs
• All structured programs can be built from three basic primes:
• Sequential (e.g., assignment statements)
• Decision (e.g., if/then/else statements)
• Iteration (e.g., while and for loops)
• A program prime is an atomic programming unit.
Coverage and Control Flow Graphs
• Using the concept of a prime and the ability to use combinations of primes to
develop structured code, a control flow graph for the SUT can be developed.
• The flow graph can be used by the tester to develop white box test cases.
• It can clearly show the program elements needed to design the test cases using a given
coverage criterion.
• There are commercial tools that will generate control flow graphs from code and
in some cases from pseudo code.
Coverage and Control Flow Graphs
Statement Adequacy/Coverage Criterion
• The tester should develop a set of test cases so that all of the statements in the
module are executed at least once.
• In terms of a flow graph model of the code, satisfying this criterion requires that all of the
nodes in the graph are exercised at least once by the test cases.
Decision (Branch) Adequacy/Coverage
Criterion
• The tester should develop a set of test cases so that each decision element in the
code (if-then, case, loop) executes with all possible outcomes at least once..
• In terms of a flow graph model of the code, satisfying this criterion requires that all of the
edges in the graph are exercised at least once by the test cases.
• Complete decision coverage is stronger than statement coverage.
• Complete decision coverage implies complete statement coverage.
Decision Coverage Criterion Example
Condition Coverage Criterion
• Definition: A condition is an expression that evaluates to true/false and does not
contain any other true/false-valued sub-expressions (e.g., a>3).
• A decision statement may contain multiple conditions:
If(x < MIN and y > MAX and (not INT Z))
• Condition coverage requires that the tester insure that each individual condition
in a compound predicate takes on all possible values at least once during
execution of the test cases.
Condition Coverage Criterion - Example
• If(age < 65 and married == true)
do X
do Y
else
do Z
Decision/Condition Coverage Criterion
• Decision/condition coverage requires that every condition will be set to all
possible outcomes and the decision as a whole will be set to all possible
outcomes.
• Decision/Condition coverage is stronger than decision coverage.
Multiple Condition Coverage Criterion
• Decision/condition coverage does not require the coverage of all the possible
combinations of conditions.
• The multiple condition coverage criterion requires covering all possible
combinations of condition outcomes in each decision.
• Multiple condition coverage is stronger than decision/condition coverage.
Notes on Coverage Criteria
• The stronger the coverage criterion, the larger the number of test cases that must
be developed to insure complete coverage.
• For code with multiple decisions and conditions the complexity of test case
design increases with the strength of the coverage criterion.
• The tester must decide, based on the type of code, reliability requirements, and
resources available which criterion to select.
Path Coverage Criteria
• Line coverage
• The test cases are designed to cover all the program code lines.
• Path coverage
• The test cases are designed to cover all the possible paths.
Path Coverage Criteria

Program flow chart of a module The corresponding Program flow graph

Path Coverage Criteria

The full list of paths (there are 24 paths required for full path coverage)
Path Coverage Criteria

Full line coverage can be achieved by using three paths only

Path Coverage Criteria
• Full path coverage is the strongest program-based criterion.
• However, this may not be a practical goal for a tester:
• The number of paths can be huge.
• Some paths in a program may be unachievable.
Path Coverage Criteria
• The McCabe’s Cyclomatic Complexity Metric
• It determines the maximum number of independent paths needed to achieve full line coverage.
• The measure is based on graph theory and is thus calculated according to the program
characteristics as captured by its program flow graph.
• An independent path is defined with reference to the succession of independent paths
accumulated.
• Any path on the program flow graph that includes at least one edge that is not included in any of the
former independent paths,
• It is expressed in three different ways:
• V(G) = R, where R is the number of regions in the program flow graph.
• V(G) = E – N + 2, where E is the number of edges and N is the number of nodes in the program flow
graph.
• V(G) = P + 1, where P is the number of decisions contained in the graph, represented by nodes having
more than one leaving edge.
Path Coverage Criteria
• The McCabe’s cyclomatic complexity metric determines the maximum number of
independent paths that can be indicated in the program flow graph.
• Equivalently it provides an upper bound on the number of test cases needed to achieve full
branch/decision coverage.
Path Coverage Criteria-
Example

A maximal set of independent paths (there are 6 paths)

Path Coverage Criteria
• Using the McCabe’s cyclomatic complexity metric can lead to an overestimation
of the number of test cases needed for decision/branch coverage.

This set of paths achieve full decision/branch coverage

Path Coverage Criteria –
Example
Data-Flow Based White Box Testing
Techniques
• These techniques generate test data based on data flow information, rather than
control-flow information present in software logic and control structures.
• Test adequacy criteria are defined based on the data flow in a given program.
Data-Flow Based White Box Testing
Techniques
• We say that a variable is defined in a statement when its value is assigned or
changed.
• In data flow notation this is indicated as a def for the variable Y.
Data-Flow Based White Box Testing
Techniques
• We say that a variable is used in a statement when its value is utilized in a
statement. The value of the variable is not changed.
• A variable can be used in a predicate or can be part of a computation.
• The former case is denoted as a p-use and the latter case is denoted as a c-use.

X, c-use X, p-use
Data-Flow Based White Box Testing
Techniques
• Date-flow based test adequacy criteria can be defined based on coverage goals in
terms of variable uses:
• All def
• All p-uses
• All c-uses/some p-uses
• All p-uses/some c-uses
• All uses
• All def-use paths
• The strongest criterion is all def-use paths.
• We refer to a path from a variable definition to a use as a def-use path.
Data-Flow Based White Box Testing
Techniques
• To satisfy all def-use criterion, the tester must identify and classify occurrences
of all the variables in the SUT. A tabular summary is useful.
• Then for each variable, test data is generated so that all definitions and all uses
for all of the variables are exercised during test.
Data-Flow Based White Box Testing
Techniques
Example
Data-Flow Based White Box Testing
Techniques
Example
Data-Flow Based White Box Testing
Techniques
Example
Data-Flow Based White Box Testing
Techniques
Example
• The following set of test inputs cover all the def-use pairs for the variables:
Data-Flow Based White Box Testing
Techniques
• As with most white box testing methods, the data flow approach is most effective
at the unit level of testing.
• Data-flow analysis can become time consuming when code becomes more
complex and there are more variables to consider.
• Tool support for data-flow based testing is currently limited.
Loop Testing
• Loops are among the most frequently used control structures.
• Many defects are associated with loop constructs.
• Loops can be classified into four categories:
Simple Nested Concatenated Unstructured
Loop Testing
• A strategy for testing a simple loop that can have a range of zero to n iterations:
• Develop test cases so that there are:
Loop Testing
• A strategy for testing nested loops:
• Set the outer loop control variable(s) to the minimum value(s).
• Test the innermost loop using the strategy for testing simple loops.
• Perform test for the next loop and work outward.
• Continue until the outmost loop has been tested.
 Loop Testing
• A strategy for testing concatenated loops : 1

• If the two loops are independent of each other then they are individually tested using simple
loops.
• However, if the loop counter for one loop is used as the initial value for the second loop, then
these loops should not be considered as independent loops.
• In this case, these loops should be tested as nested loops.

1
https://www.guru99.com/loop-testing.html
Loop Testing
• Unstructured loops should be redesigned to reflect the use of the structured
programming constructs.
Loop Testing
• A loop count adequacy criterion can be defined in terms of the number of
iterations executed in the case of a loop having a maximum of n iterations.
• Loop testing usually suffers from the large number of tests needed.
Mutation Testing
• The approaches discussed so far focus on code behavior and code structure.
• Mutation testing is another approach that requires knowledge of code structure
but is classified as a fault-based testing approach.
• It considers the possible faults that could occur in a SUT as the basis for test data generation
and evaluation of testing effectiveness.
Mutation Testing
• Mutation testing starts with a code component, its associated test cases, and the
test results.
• The original code component is modified in a simple way to provide a set of
similar components that are called mutants.
• Each mutant contains a fault as a result of the modification.
• The original test cases are run with the mutants:
• If a test case reveals the fault in the mutant by producing a different output as a result of
execution, then the mutant is said to be killed.
• If the test cases do not produce outputs that differ from the original, then the test set is not
capable of revealing such defects. In this case, the tests cannot distinguish the original from
the mutant. The tester then must develop additional test cases to reveal the fault and kill the
mutant.
Mutation Testing
• A test adequacy criteria for mutation testing is the following:
Mutation Testing
• A mutation is a simple change in the original code component, for example:
constant replacement, arithmetic operator replacement, data statement
alteration, statement deletion, and logical operator replacement.
Consider the following simple changes:
- In Line 7, change i= i + 1 to i = i + 2
If we re-run the tests used for branch
coverage, this mutant would be killed.
- In Line 5, change if a[i]>0 to if a[i]<0
If we re-run the tests used for branch
coverage, this mutant would be killed.
- In Line 5, change if a[i]>0 to if a[i]>=0
This mutant would not be killed by our original
test data. We need to augment the test cases
with a new case that includes a zero in a[0].
However, even this new case would not cause
the mutant to be killed!
Mutation Testing
• Sometimes it is not possible to kill a mutant.
• In this case, the mutant is said to be equivalent to the original program.
• To measure the mutation adequacy of a test set T for a program P, we can use the
mutation score metric (MS):
Mutation Testing
• Mutation Testing is useful in that it can show that certain faults as represented in
the mutants are not likely to be present in the original code since they would
have been revealed by the test cases.
• It also helps the tester to generate hypotheses about the different types of
possible faults in the code and to develop test cases to reveal them.
• There are tools to support developers and testers with producing mutants.
• However, running the tests, analyzing the results, and developing additional tests to kill the
mutants are all time consuming.
• For this reason, mutation testing is usually applied at the unit level.
• However, recent research in an area called interface mutation (the application of mutation
testing to evaluate how well unit interfaces have been tested) has suggested that it can be
applied effectively at the integration test level as well.