Specification-Based Regression Test Management with Risk Analysis: A New Regression Test Selection Approach

Yanping Chen Robert L. Probert Paul Sims

Agenda
Introduction and Background Classification of Regression Test Cases Regression Test Selection with Risk Analysis
Part One: Targeted Test Selection Part Two: Safety Test Selection

Case Study and Results Evaluation Conclusions and Future work

[email protected] 2

01/10/02

Regression Testing
To ensure that new or modified features do not cause current release to regress after incorporating fixes into product -- ensure customers business wont be at risk Essential to ensure software quality In software maintenance: validate modified software In O-O software development
Ensure quality of successive increments Assess quality of re-used components
[email protected] 3

Typical Regression Test Selection

Test Suite T Program P

T: New test cases

Program P T: Regression T: New test cases selected from T test cases Changed Features New Features

Program P

Regression Test Suite T

[email protected] 4

Current State of the Art

Most techniques based on code (white-box test design) Only few techniques based on program specifications (black-box test design) Code-based regression techniques
Not scalable: good for regression analysis at unit level, but not suitable for testing big components (difficult to manage all the code info. and to create corresponding management matrices) Too time-consuming and error-prone: requires that tester access and understand code Language-dependent

Focus on specification-based regression techniques!

[email protected] 5

Main Problem with Specificationbased Regression Test Selection

Too many personal decisions involved For testers: selection criteria is subjective For managers: no efficient way to measure selected regression test suite effectiveness Our Solution Define a new regression test selection method
Make regression test selection criteria more objective Evaluate regression test suite
[email protected] 6

Classification of Regression Tests

Two goals of regression testing
Assure system stability: rerun test cases for modified or possibly affected requirement attributes Achieve adequate confidence in software quality: set appropriate pre-defined coverage target

Two categories of regression test cases

Targeted Tests
Test cases that exercise important affected requirement attributes Regression analysis similar to firewall approach

Safety Tests
Coverage-oriented and risk-oriented test cases Goal: achieve good enough quality

[email protected]

Risk-based Safety Test Selection

Risk-based testing similar to Air-based living Risk: event that has some probability of happening, and if occurs, will result in some loss All testing is motivated by risk
Testers job is finding high-priority problems to avoid risk

Testers have always used risk-based testing, but in ad hoc fashion based on personal judgment [4] Risk-based testing
Focuses testing and spends more time on key functions and critical functions Achieves highest confidence in limited time

Uses risk to measure the quality of test suite

[email protected] 8

Part One Targeted Test Selection

[email protected]

Activity Diagram
UML notation to model computations and workflows in specification, based on requirements Represents sequences in which activity may occur Grey-box: contains more information than requirement Elements of Activity Diagram
Nodes
Activity state Synchronization bar Decision diamond Start and stop marker

Edges
Solid arrow: control flow Dashed arrow: message flow
[email protected] 10

A Sample Activity Diagram

a b

Start Activity state

Display Session Time- l out Message [Invalid]

Check Session Validity Order

c d e f

[Valid]

Decision diamond

Send Get_Quote Request to Banks

m
[Time Out]

Calculate Best Quote

[ n Consecutive Failure]

j
Send Get_Quote Request to Time-out Banks

g h

Display Best Quote Update Session Info.

Stop

[email protected]

11

Activity Diagram-based Regression Tests Selection

1. Trace test cases to the activity diagram 2. Locate affected nodes and edges
Corrective maintenance(does not affect activity diagram):
Locate changed nodes and edges Traverse like paths to find differences Create set of affected edges

Progressive maintenance(affect activity diagram)

3. Choose test cases for all affected nodes and edges

[email protected] 12

Traceability Matrix
Node Test Case t1, t2, t3, t4 t1, t2, t3, t4 t1, t2, t3, t4 t1, t2, t3 t1, t2, t3 t1, t2, t3 t1, t2, t3 t1, t2, t3 t1, t2, t3 t2, t3 t2, t3 t4 t4 Edge (a, b) (b, c) (c, d) (c, l) (d, e) (e, f) (e, j) (f, g) (g, h) (h, i) (j, f) (j, k) (k, e) (l, m)
[email protected]

Test Case t1, t2, t3, t4 t1, t2, t3, t4 t1, t2, t3 t4 t1, t2, t3 t1, t2 t2, t3 t1, t2, t3 t1, t2, t3 t1, t2, t3 t3 t2, t3 t2, t3 t4
13

Test Case Path t1 a, b, c, d, e, f, g, h, i t2 a, b, c, d, e, j, k, e, f, g, h, i t3 a, b, c, d, e, j, k, e, j, f, g, h, i t4 a, b, c, l, m

a b c d e f g h i j k l m

Changes in Activity Diagram

a b
Check Session Validity Order [Invalid] [Valid] Send Get_Quote Request to Banks Display Session Time- l out Message

c d e f

Bugs m
[Time Out]

Calculate Best Quote

[ n Consecutive Failure]

j
Send Get_Quote Request to Time-out Banks

X
k
14

g h i

Display Best Quote Update Session Info.

[email protected]

 Corrective maintenance: node d

t1, t2, t3 t2, t3

Progressive maintenance: edge (k, e)

Targeted Test: {t1, t2, t3}

[email protected]

15

Part Two Safety Test Selection

[email protected]

16

Why Safety Test?

Targeted Tests selection are entirely specification-based: some defects in code might be missed Useful in two situations
Time and resources are available after running Targeted Tests: gain more confidence in modified system No time left for testing due to development schedule overrun: provide some assurance that remaining defects will not cause serious failures
[email protected] 17

A Simple Risk Model

Two elements of Risk Exposure (REf):

[4]

Probability of fault Cost (consequence or impact) of fault in corresponding function if it occurs in production

REf = Pf Cf
REf: Risk Exposure of function f (E means exposure) Pf: probability of fault occurring in function f
In our model, we consider severity of defects to assess probability Note: Pf is extended to severity probability

Cf: cost if fault occurs (in production) in function f

[email protected] 18

Risk-based Regression Testing Approach

Model-based Safety Tests Selection Method:

Step 1. Assess cost Ct for each test case Step 2. Derive severity probability Pt for each test case Step 3. Calculate Risk Exposure REt for each test case Step 4. Select test cases with top REt as Safety Tests
[email protected] 19

Assess Cost Ct
Two kinds of costs
Ct (c): Consequences of fault as seen by customer, i.e., losing market place Ct (v): Consequences of fault as seen by vendor, i.e., high software maintenance cost

Ct is categorized on 1~5 scale (1- low, 5 high)

Weight Ct (c) and Ct (v) equally Ct = (Ct (c) + Ct (v))/2
[email protected] 20

Derive Severity Probability Pt

Summarize number of defects opened for each test case after full testing Calculate average severity of defects for each test case Use result of Number of Defects (Nt) times Average Severity (St) Nt St to assess severity probability Pt fall into 1~5 range (1 - low, 5 - high)
Test cases with no defect in full testing, Pt =1 (still possible for defects to exist) Test cases with top 25% estimate Nt St, Pt = 5 Test cases with bottom 25% estimate Nt St, Pt = 2
[email protected]

21

Step 1:

Calculate Risk Exposure REt

Step 2:
Ct 5 2 3 Test Case t0010 t0020 Nt 3 1 .. 0 St 2 1 0 Pt 5 4 1 t0010 t0020 tn

Test Case

Test Case
t0010 t0020 tn

=
Ct 5 2 3

tn

Pt 5 4 1

REt = Pt Ct 25 8 3
22

[email protected]

Select Test Cases with Top REt

Choose test cases with highest value of REt Reach pre-defined coverage target (e.g., 30% of full test suite)
Test Case Full Test Suite Regression Test Suite (30%)

t0010 t0020 t0030 t0040 t0050 t0060 t0070

[email protected]

1 1 1 1 1 1 1

1 1 0 0 0 1 0
23

Case Study with historical data of IBM WebSphere

Three components of IBM WebSphere with different characters
Component One: Focus on functionality Component Two: Focus on data Component Three: Both functionality and data are important

Each component was owned by one experienced test engineer 306 test cases in total
[email protected] 24

Results
High Risk Exposure coverage and average Risk Exposure Acceptable specification coverage not our focus Powerful in Selecting Effective Test Cases and Finding Defects
Caught all defects Omitted fewer test cases that revealed defects

Defects (%) Defect-revealing Test Cases (%)

Selected Test Suite 100% 93.9%

[email protected]

Manual Test Suite 84.1% 83.1%

25

Evaluation
Effectiveness: more effective in finding defects Cost-efficiency: can be implemented into tools Sensitivity to Risk
Risk-based test selection that choose critical test cases Use risk metrics to quantitatively measure the safety of test suite

Can be applied to State Diagrams and other graphlike diagrams Powerful for function test Systematic not subjective ! Only requires straightforward calculation
[email protected] 26

Automation
Targeted Test
Manually in Excel Can be automated using more powerful test tracking tools

Safety Test
Automated Straightforward calculation that can be easily implemented
[email protected] 27

Conclusions and Future Work

Specification-based, risk-based, and activity diagram-based regression test selection technique Future Work
Choose more components for empirical case studies Develop metrics (for example, specification coverage) to help decide when to stop Implement the technique in production test environment (under discussion) Use statistical analysis tools to determine costeffectiveness of our technique in practice
[email protected] 28

Reference
[1] WebSphere Commerce Suite FVT Team Regression Selection Guidelines, ECD, IBM Toronto Laboratory, 2000 [2] Stale Amland, Risk Based Testing and Metrics: Risk analysis fundamentals and metrics for software testing including a financial application case study, 5th International Conference EuroSTAR99, November, 1999, Barcelona, Spain [3] ECD Software Development Process Function Verification Test Process, ECD, IBM Toronto Laboratory, 2001 [4] Gregg Rothermel and Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, V.22, no. 8, August 1996, pages 529 551 [5] Robert V. Binder, Testing Object-Oriented Systems, Addison-Wesley Publishing company, 2000 [6] Yanping Chen, Specification-based Regression Testing Measurement with Risk Analysis, Masters Thesis, University of Ottawa, Canada, 2002
[email protected] 29