Unit 3

Software Testing Strategies

White Box Approaches
Based on the following textbooks:
D. Galin, Software Quality Assurance: From Theory to Implementation, 1st ed. Addision-
Wesley, 2004

Ilene Burnstein, Practical Software Testing: A Process-Oriented Approach, Springer, 2013.

Outline
• Introduction to White Box Testing
• Test Adequacy Criteria
• Coverage and Control Flow Graphs
• Path Coverage
• The McCabe’s Cyclomatic Complexity Metric
• Additional White Box Test Design Approaches
• Data Flow
• Loop Testing
• Mutation Testing
Introduction to White Box Testing
• White box testing is a complementary approach to black box test case design in
which the tester has knowledge of the internal structure of the SUT.
• The knowledge needed for white box testing often becomes available to the
tester in the later phases of the software life cycle, specifically during the detailed
design and coding phases of development. Therefore, white box testing follows
black box testing.
• Black box testing approaches can be used for both small and large software
components, whereas white box testing is most useful when testing small
components.
Test Adequacy Criteria
• The goal for white box testing is to ensure that the internal components of a
program are working properly.
• A common focus is on structural elements such as statements and branches.
• The tester develops test cases that exercise (execute) these structural elements to determine
if defects exist in the program structure.
• By exercising all of the selected structural elements the tester improves the chances for
detecting defects.
• Testers need a framework for deciding which structural elements to select and for
deciding when the testing efforts are adequate enough.
• Such a framework exists in the form of test adequacy criteria.
Test Adequacy Criteria
• Definition: a test adequacy criterion is a stopping rule.
• A program is said to be adequately tested with respect to a given criterion if all of
the target structural elements have been exercised according to the selected
criterion.
• An Example: A test set T is statement adequate for program P if it causes all the statements in
P to be executed.
• A test adequacy criterion can also be expressed as a percent (which is called the
degree of coverage).
• An Example: A test set T satisfies 50% statement adequate for program P if it causes 50% of
the statements in P to be executed.
Test Adequacy Criteria
• The planned degree of coverage is specified in the test plan and then measured
when the tests are actually executed.
• The planned degree of coverage is usually specified as 100%, however, under
some circumstances the planned degree of coverage may be less than 100% due
to the following:
• The nature of the unit.
• The lack of resources
• Other project-related issues such as timing, scheduling, and marketing constraints.
Coverage and Control Flow Graphs
• The application of coverage analysis is typically associated with one of the
following program structural elements:
• Program statements
• Decisions/branches
• Conditions
• Combinations of decisions and conditions
• Paths
Coverage and Control Flow Graphs
• All structured programs can be built from three basic primes:
• Sequential (e.g., assignment statements)
• Decision (e.g., if/then/else statements)
• Iteration (e.g., while and for loops)
• A program prime is an atomic programming unit.
Coverage and Control Flow Graphs
• Using the concept of a prime and the ability to use combinations of primes to
develop structured code, a control flow graph for the SUT can be developed.
• The flow graph can be used by the tester to develop white box test cases.
• It can clearly show the program elements needed to design the test cases using a given
coverage criterion.
• There are commercial tools that will generate control flow graphs from code and
in some cases from pseudo code.
Coverage and Control Flow Graphs
Statement Adequacy/Coverage Criterion
• The tester should develop a set of test cases so that all of the statements in the
module are executed at least once.
• In terms of a flow graph model of the code, satisfying this criterion requires that all of the
nodes in the graph are exercised at least once by the test cases.
Decision (Branch) Adequacy/Coverage
Criterion
• The tester should develop a set of test cases so that each decision element in the
code (if-then, case, loop) executes with all possible outcomes at least once..
• In terms of a flow graph model of the code, satisfying this criterion requires that all of the
edges in the graph are exercised at least once by the test cases.
• Complete decision coverage is stronger than statement coverage.
• Complete decision coverage implies complete statement coverage.
Decision Coverage Criterion Example
Condition Coverage Criterion
• Definition: A condition is an expression that evaluates to true/false and does not
contain any other true/false-valued sub-expressions (e.g., a>3).
• A decision statement may contain multiple conditions:
If(x < MIN and y > MAX and (not INT Z))
• Condition coverage requires that the tester insure that each individual condition
in a compound predicate takes on all possible values at least once during
execution of the test cases.
Condition Coverage Criterion - Example
• If(age < 65 and married == true)
do X
do Y
else
do Z
Decision/Condition Coverage Criterion
• Decision/condition coverage requires that every condition will be set to all
possible outcomes and the decision as a whole will be set to all possible
outcomes.
• Decision/Condition coverage is stronger than decision coverage.
Multiple Condition Coverage Criterion
• Decision/condition coverage does not require the coverage of all the possible
combinations of conditions.
• The multiple condition coverage criterion requires covering all possible
combinations of condition outcomes in each decision.
• Multiple condition coverage is stronger than decision/condition coverage.
Notes on Coverage Criteria
• The stronger the coverage criterion, the larger the number of test cases that must
be developed to insure complete coverage.
• For code with multiple decisions and conditions the complexity of test case
design increases with the strength of the coverage criterion.
• The tester must decide, based on the type of code, reliability requirements, and
resources available which criterion to select.
Path Coverage Criteria
• Line coverage
• The test cases are designed to cover all the program code lines.
• Path coverage
• The test cases are designed to cover all the possible paths.
Path Coverage Criteria

Program flow chart of a module The corresponding Program flow graph

Path Coverage Criteria

The full list of paths (there are 24 paths required for full path coverage)
Path Coverage Criteria

Full line coverage can be achieved by using three paths only

Path Coverage Criteria
• Full path coverage is the strongest program-based criterion.
• However, this may not be a practical goal for a tester:
• The number of paths can be huge.
• Some paths in a program may be unachievable.
Path Coverage Criteria
• The McCabe’s Cyclomatic Complexity Metric
• It determines the maximum number of paths needed to achieve full branch/decision coverage.
• The measure is based on graph theory and is thus calculated according to the program
characteristics as captured by its program flow graph.
• An independent path is defined with reference to the succession of independent paths
accumulated.
• Any path on the program flow graph that includes at least one edge that is not included in any of the
former independent paths,
• It is expressed in three different ways:
• V(G) = R, where R is the number of regions in the program flow graph.
• V(G) = E – N + 2, where E is the number of edges and N is the number of nodes in the program flow
graph.
• V(G) = P + 1, where P is the number of decisions contained in the graph, represented by nodes having
more than one leaving edge.
Path Coverage Criteria
• The McCabe’s cyclomatic complexity metric determines the maximum number of
independent paths that can be indicated in the program flow graph.
• Equivalently it gives the maximum number of test cases needed to achieve full
branch/decision coverage.
Path Coverage Criteria-
Example

A maximal set of independent paths (there are 6 paths)

Path Coverage Criteria
• Using the McCabe’s cyclomatic complexity metric can lead to an overestimation
of the number of test cases needed for decision/branch coverage.

This set of paths achieve full decision/branch coverage

Path Coverage Criteria –
Example
Data-Flow Based White Box Testing
Techniques
• These techniques generate test data based on data flow information, rather than
control-flow information present in software logic and control structures.
• Test adequacy criteria are defined based on the data flow in a given program.
Data-Flow Based White Box Testing
Techniques
• We say that a variable is defined in a statement when its value is assigned or
changed.
• In data flow notation this is indicated as a def for the variable Y.
Data-Flow Based White Box Testing
Techniques
• We say that a variable is used in a statement when its value is utilized in a
statement. The value of the variable is not changed.
• A variable can be used in a predicate or can be part of a computation.
• The former case is denoted as a p-use and the latter case is denoted as a c-use.

X, c-use X, p-use
Data-Flow Based White Box Testing
Techniques
• Date-flow based test adequacy criteria can be defined based on coverage goals in
terms of variable uses:
• All def
• All p-uses
• All c-uses/some p-uses
• All p-uses/some c-uses
• All uses
• All def-use paths
• The strongest criterion is all def-use paths.
• We refer to a path from a variable definition to a use as a def-use path.
Data-Flow Based White Box Testing
Techniques
• To satisfy all def-use criterion, the tester must identify and classify occurrences
of all the variables in the SUT. A tabular summary is useful.
• Then for each variable, test data is generated so that all definitions and all uses
for all of the variables are exercised during test.
Data-Flow Based White Box Testing
Techniques
Example
• The following set of test inputs cover all the def-use pairs for the variables:
Data-Flow Based White Box Testing
Techniques
• As with most white box testing methods, the data flow approach is most effective
at the unit level of testing.
• Data-flow analysis can become time consuming when code becomes more
complex and there are more variables to consider.
• Tool support for data-flow based testing is currently limited.