Software Testing: White box testing

SE 110

SE 110 Spring 2012

White-box testing
White box testing is a way of testing the external functionality of the code by examining and testing the program code that realizes the external functionality. Synonymous terms: Clear box testing, glass box testing, open box testing. White box testing takes into account the program code, structure and internal design flow.
SE 110 Spring 2012

White-box testing: Classification

SE 110 Spring 2012

Static testing
Involves only the source code and not the executables or binaries. Does not involve executing the program on a machine but rather humans going through it or the usage of specialized tools. Some of the properties tested by static testing: Whether the code works according to the functional requirement Whether the code has been written in accordance with the design developed earlier in the project life cycle Whether the code follows all applicable standards Whether the code for any functionality has been missed out Whether the code handles errors properly
SE 110 Spring 2012

Static testing by humans

Humans can find errors that computers cant. Multiple humans can provide multiple perspectives. A human evaluation can compare the code against the specifications more thoroughly. It can detect multiple defects at one go. It reduces downstream, inline pressure.
SE 110 Spring 2012

Desk checking
Author informally checks the code against the specifications and corrects defects found. No structured method or formalism is required to ensure completeness. No log or checklist is maintained. It relies only on the thoroughness of the author. It may suffice for obvious programming errors but may not be effective for incomplete / misunderstood requirements.
SE 110 Spring 2012

Desk checking: Advantages and disadvantages

Advantages The programmer knows the code and programming language well and hence is best suited to read the program. Less scheduling and logistics overheads. Reduces delay in defect detection and correction. Disadvantages Person dependent, not scalable or reproducible. Tunnel visioning of developers. Developers prefer writing new code and dont like testing.
SE 110 Spring 2012

Code walkthrough
Group oriented (as against desk checking). Brings in multiple perspectives
A set of people look at the program and raise questions for the author. The author explains the logic of the code and answers the questions. Unanswered questions during walkthrough are answered later by the author.

Very similar to inspections, but considered less formal.

SE 110 Spring 2012

Fagan Inspection
Group-oriented activity Highly formal and structured Has specific roles Requires thorough preparation

SE 110 Spring 2012

Roles in formal inspection

Author Author of the work product Makes available the required material to the reviewers Fixes defects that are reported Moderator Controls the meeting(s) Inspectors (reviewers) Prepare by reading the required documents Take part in the meeting(s) and report defects Scribe Takes down notes during the meeting Assigned in advance by turns Can participate to review to the extent possible Documents the minutes and circulates them to participants
SE 110 Spring 2012

Fagan inspection: Documents

Typical documents circulated: Program code Design / program specifications SRS (if needed) Any applicable standards (e.g., coding standards) Any necessary checklists (e.g., code review checklist)
SE 110 Spring 2012

Fagan inspection: Meetings

Preliminary meeting (optional) Author explains his / her perspective Makes available the necessary documents Highlights concern areas, if any, for which review comments are sought Defect Logging Meeting All come prepared! Moderator goes through the code sequentially Each reviewer comes up with comments Comments / defects categorized as defect / observation, major / minor, systemic / mis-execution Scribe documents all the findings and circulates them Follow-up meeting (optional) Author fixes defects If required, a follow-up meeting is called to verify completeness of fixes.

SE 110 Spring 2012

Fagan inspection: Advantages and disadvantages

Advantages Thorough, when prepared well Brings in multiple perspectives Has been found to be very effective Disadvantages Logistically difficult Time consuming May not be possible to exhaustively go through the entire code
SE 110 Spring 2012

Combining various methods

Prioritizing various programs High priority / important codes subject to formal inspection, medium ones for walkthrough and low ones for desk checking. Deciding between 100% coverage and partial coverage.

SE 110 Spring 2012

Static analysis tools

Help identify / are used to:
Whether there are unreachable codes (usage of goto statements sometimes creates this situation; there could be other reasons too). Variables declared but not used. Mismatch in definition and assignment of values to variables. Illegal or error-prone type-casting of variables. Use of non-portable or architecture-dependent programming constructs. Memory allocated but not having corresponding statements for freeing up memory. For calculation of cyclomatic complexity (to be discussed later).

SE 110 Spring 2012

Code review checklist

While following static analysis techniques, it is useful to have a code review checklist. The code review checklist aids in organizational learning by indicating what to look for. In a multi-product organization, the checklist may be at two levels:
An organization-wide checklist that will include issues such as organization coding standards, documentation standards etc. A product or project-specific checklist that addresses issues specific to the product or project.
SE 110 Spring 2012

White-box testing: Classification

SE 110 Spring 2012

Structural testing
Done by running the executable on the machine. Entails running the product against some predefined test cases and comparing the results against the expected results. Designing test cases and test data to exercise certain portions of code. Types of structural testing Unit / code functional testing Code coverage testing Code complexity testing
SE 110 Spring 2012

Unit/code functional testing

Initial quick checks by developer. Removes obvious errors. Done before more expensive checks. Building debug versions. Running through an IDE.

SE 110 Spring 2012

Code coverage testing

Exercises different parts of code. Maps parts of code to required functionality. Finds out percentage of code covered by instrumentation.
Instrumentation rebuilds the product, linking it with libraries.

Instrumented code can monitor what parts of code is covered. Can help identify critical portions of code, executed often.
SE 110 Spring 2012

Types of coverage
Statement coverage Path coverage Condition coverage Function coverage

SE 110 Spring 2012

Statement coverage

Statement coverage refers to writing test cases that execute each of the program statements. Typical program constructs constituting program statements:
Sequential instructions. Two-way decision statements (if then else). Multi-way decision statements (switch statements). Loops, like while/do, for, repeat/until, etc. Additional constructs for OO-languages.

Statement coverage = (Total statements exercised/Total number of executable statements in program)*100.

SE 110 Spring 2012

Statement coverage: Sequential statements

Each circle represents one or more non-branching source code statements

Generate test data to make the program enter the sequential block, to make it go through the entire block.
SE 110 Spring 2012

Statement coverage: If-then-else

Have test cases to test then part Have test cases to test else part

SE 110 Spring 2012

Statement coverage: Switch

Switch statement can be reduced to multiple two way if-then-else statements. Thus, there would be multiple test cases to cover all possible cases of switch statement.
SE 110 Spring 2012

Statement coverage: loops

while

until

Typically loops fail in boundary conditions. Test cases to achieve statement coverage for statements within a loop should be such that
They skip the loop completely, so that the situation of the termination condition being true before starting the loop is tested. They should exercise the loop between once and the maximum number of times, to check normal operations of the loop. They should try to cover the loop, around the boundary of the loop, i.e. with values just below the boundary, the boundary value and values just above the boundary.
SE 110 Spring 2012

Example: GPA calculation

float findGPA( Collection courses ) { int totalCreditHours = 0; int totalGradePoints = 0; int gradePoints = 0; float gradePointAverage = 0; Iterator it=courses.iterator(); while( it.hasNext() ) { Course theCourse = (Course) it.next( ); if( theCourse.grade == 'A' ) gradePoints = 4*theCourse.creditHours; else if( theCourse.grade == 'B' ) gradePoints = 3*theCourse.creditHours; else if( theCourse.grade == 'C' ) gradePoints = 2*theCourse.creditHours; else if( theCourse.grade == 'D' ) gradePoints = 1*theCourse.creditHours; else if( theCourse.grade == 'F' ) gradePoints = 0*theCourse.creditHours; totalGradePoints += gradePoints; totalCreditHours += theCourse.creditHours; } gradePointAverage = totalGradePoints / totalCreditHours; return gradePointAverage ;

Course courseName creditHours grade

SE 110 Spring 2012

Statement coverage: Example

Function: findGPA Test Case Id courseName IT-100 creditHours grade

4 A

IT-101
F01-UT-01

4 B
4 A 4 D 4 F

IT-102 IT-103 IT-104

Is the statement coverage 100%?

SE 110 Spring 2012

Statement coverage: Example

Function: findGPA
Test Case Id courseName IT-100 IT-101
F01-UT-02

creditHours

grade

4 A 4 B 4 C 4 D 4 F

IT-102 IT-103 IT-104

Statement coverage is 100%.

SE 110 Spring 2012

Loop coverage: Example

Test Case Id.: IT-100 IT-101 IT-102 IT-103 IT-104 Loop count > 1 F01-UT-02 4 A 4 B 4 A 4 D 4 F Test Case Id.: IT-100 Loop count = 1 F01-UT-03 4 A courseName creditHours grade courseName creditHours grade

Test Case Id.:

F01-UT-04

Empty collection

OOPS! (Divide by zero)

Conclusion

Loop count = 0

100% Statement coverage does not guarantee bug-free code

SE 110 Spring 2012

Branch coverage: Example

Function: findGP Test Case Id courseName IT-100 IT-101
F01-UT-06

creditHours

grade 4 A 4 B 4 C 4 D 4 F

IT-102 IT-103 IT-104

How much is the branch coverage?

SE 110 Spring 2012

Path coverage
Statement Coverage may not indicate true coverage. Path coverage splits the program into a number of paths, a test can start from the beginning and take any of the paths to its completion. Path coverage = (Total paths exercised/Total number of paths in program)*100.
SE 110 Spring 2012

Path coverage: Example

Flow chart for date validation routine

Some paths:
A B-D-G B-D-H B-C-E-H B-C-F-G

SE 110 Spring 2012

Condition coverage
Further refinement of path coverage. Makes sure each constituent condition in a Boolean expression is covered. Protects against compiler optimizations. Condition coverage = Total conditions exercised / total number of conditions in the program)*100. Condition coverage is much stronger criteria than path coverage which in turn is a stronger criteria than statement coverage.
SE 110 Spring 2012

Function coverage
Finds how many functions are covered by test cases. Higher level of abstraction; hence possible to achieve 100% coverage. More logical than the other types of coverage. Can be derived from RTM. Easy to prioritize as they are based on requirements. Leads more naturally to black box tests. Can also be a natural predecessor to performance testing. Function coverage = (Total functions exercised/Total number of functions in program)*100.
SE 110 Spring 2012

Code coverage techniques: Other uses

Performance analysis and optimization. Resource usage analysis. Checking of critical sections. Identifying memory leaks. Dynamically generated code (e.g., dynamic generation of URLs and security checking).
SE 110 Spring 2012

Code complexity testing

Which of the paths are independent? (If paths are not independent, we may be able to minimize the number of tests.) Is there an upper bound on the number of tests to be executed to ensure that all the statements have been executed at least once? Cyclomatic complexity is a metric that quantifies the complexity of the program and provides answers to some of these questions.
SE 110 Spring 2012

Cyclomatic complexity: Intuitive meaning Provides an indication of how many independent paths are there in a program What gives birth to new paths? How can one count the new paths that are born?

SE 110 Spring 2012

Steps to compute cyclomatic complexity

Start with a conventional flow chart.

Identify the decision points.

Convert complex predicates to simple predicates (see next slide). If there are loops, convert the loop conditions to simple predicates. Combine all the sequential statements into a single node in the flow graph. When a set of sequential statements are followed by a simple predicate, combine all the sequential statements and the predicate check into one node and have two edges emanating out of this node. Nodes with two emanating edges are called predicate nodes.

Make sure that all edges terminate in some node, adding a node to represent all the statements at the end of the program.

SE 110 Spring 2012

Converting complex predicates into simple predicates

SE 110 Spring 2012

Steps: Sample flow chart illustration

Conventional flow chart Flow diagram for calculating Cyclomatic complexity

SE 110 Spring 2012

Cyclomatic complexity: definition

Formal definitions: CC = # of predicate nodes + 1 CC = E N + 2 (E = edges, N = nodes) Intuitive definition: When a condition is introduced, it adds a new path See the next two slides
SE 110 Spring 2012

Example #1
Hypothetical diagram with no decision nodes

# of independent paths = 1 # of nodes, N = 2; # of edges, E = 1; Cyclomatic complexity = E N + 2 = 1 # of predicate nodes, P = 0; Cyclomatic complexity = P + 1 = 1

End

SE 110 Spring 2012

Example #2
Hypothetical diagram with one decision node

# of independent paths = 2 # of nodes, N = 4; # of edges, E = 4; Cyclomatic complexity = E N + 2 = 2 # of predicate nodes, P = 1; Cyclomatic complexity = P + 1 = 2

End

SE 110 Spring 2012

Independent paths
An independent path is a path in the flow graph that has not been traversed before in other paths. A set of independent paths that cover all the edges is called a basis set. Find the basis set and write the test cases to execute all the paths in the basis set.

SE 110 Spring 2012

Cyclomatic complexity: Interpretation

Complexity 1-10 What it means

Well-written code, testability is high, cost / effort to maintain is low

Moderately complex, testability is medium, cost / effort to maintain is medium Very complex, testability is low, cost / effort to maintain is high

10-20

20-40

>40

Not testable, any amount of money / effort to maintain may not be SE 110 Spring 2012 enough

White-box testing: Challenges

Developers not finding time for testing They develop blind spots for their own defects Fully covered code may not correspond to realistic scenarios

SE 110 Spring 2012