Regression Testing Techniques
Regression Testing Techniques
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 100
Abstract --- The most technical and typical phase in the software development life cycle is maintenance phase, in which the
development team is supposed to maintain the software which is delivered to the clients by them on proper time. Software
maintenance results for the reasons like error corrections, enhancement of capabilities, modifications of outdated modules and
optimization. Now the changed or modified software needs testing known as regression testing. In this paper we have presented
the various types of regression testing techniques their classifications presented by various researchers ,explaining selective and
prioritizing test cases for regression testing in detail .
Index Terms: Regression testing, test case prioritization, software maintenance
—————————— ——————————
1. INTRODUCTION
Software maintenance is an activity which includes for P. Regression testing consists of reusing T on P′,
enhancements, error corrections, optimization and and determining where the new test cases are
deletion of obsolete capabilities. These needed to effectively test code or functionality
modifications in the software may cause the added to or changed in producing P’. There are
software to work incorrectly and may also affect various regression testing techniques (1) Retest all;
the other parts of the software, so to prevent this (2) Regression Test Selection; (3) Test Case
Regression testing is performed. Regression testing Prioritization; (4) Hybrid Approach. Figure 1
is used to revalidate the modifications of the shows various regression testing techniques.
software. Regression testing is an expensive
Regression Testing
process in which test suites are executed ensuring
that no new errors have been introduced into
previously tested code.
In section 2 of this paper we have broadly shown
various types of regression testing techniques and
further discussed classifications of these types Retest Regression Test Case Hybrid
given by various authors, then moving into the All Test Prioritization Approach
details of selective and prioritizing test cases for Selection
regression testing, discussing search algorithms for
test case prioritization. In section 3 we have Figure1. Regression Testing Techniques
discussed the approaches which may be used to
2.1. Retest All
compare various regression testing techniques and
Retest all method is one of the conventional
challenges faced by these approaches.
methods for regression testing in which all the tests
in the existing test suite are rerunned. So the retest
2. REGRESSION TESTING
all technique [3] is very expensive as compared to
Regression testing is defined [1] as “the process of
techniques which will be discussed further as
retesting the modified parts of the software and
regression test suites are costly to execute in full as
ensuring that no new errors have been introduced
it require more time and budget.
into previously tested code”. Let P be a program
[2], let P’ be a modified version of P, and let T be a
2.2. Regression Test Selection (RTS)
test suite
Due to expensive nature of “retest all” technique,
Regression Test Selection is performed. In this
————————————————
F.A. Rashmi Pandey is with the ITM-University,gwalior,
technique instead of rerunning the whole test suite
CS1807 we select a part of test suite to rerun if the cost of
selecting a part of test suite is less than the cost of
running the tests that RTS allows us to omit. RTS
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151‐9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 101
divides the existing test suite into (1) Reusable 3) Coverage focused Minimization technique: uses
test cases; (2) Retestable test cases; (3) the suite reduction technique of Gupta, Harrold,
Obsolete test cases. In addition to this and Soffa [7] to find a subset of test suite that is
classification RTS may create new test cases that minimal in covering all functions in program.
test the program for areas which are not covered Specific methods given by various authors in
by the existing test cases. RTS techniques are literature which come under the above mentioned
broadly classified into three categories [1]. techniques are:
1) Coverage techniques: they take the test coverage 1) Simulating Annealing (SA) algorithm: Mansour
criteria into account. They find coverable program and El‐Faikh[8] have proposed using an
parts that have been modified and select test cases optimization formulation of regression testing
that work on these parts. selection problem.
2) Minimization techniques: they are similar to 2) Reduction Methodology (RED): Harrold, Gupta,
coverage techniques except that they select and Soffa [9] have proposed a method for reducing
minimum set of test cases. the number of selected test cases.
3) Safe techniques: they do not focus on criteria of 3) Slicing Algorithms (SLI): Agrawal, Horgan, and
coverage, in contrast they select all those test cases Krauser [10] have proposed it which selects test
that produce different output with a modified cases whose output may be affected by change in
program as compared to its original version. the program.
Rothermel [4] identified the various categories in 4) Bahsoon and Mansour regression methods: have
which Regression Test Selection Technique can be proposed [11] three methods
evaluated and compared. These categories are: (a) a) Modification Based Reduction version 1( MBR1)
Inclusiveness; (b) Precision; (c) Efficiency; (d) which is an improvement to RED reduces the
Generality. number of selected regression tests removing tests
a) Inclusiveness is the measure of extent upto that cover requirements impacted by the change
which a technique chooses the test cases which will and those that are redundant.
cause the changed program to produce different b) Modification Based Reduction)
output than the original program, resulting in version 2( MBR2) improves MBR1 by removing
exposure of faults due to modifications. tests that cover a requirement characterized as
b) Precision is the measure of ability of technique potentially and do not show any modifications. c)
to prevent choosing test cases that will not make PreciseReduction (PR) is an improvement to SLI
the changed program to produce different output algorithms which uses relevant slices and
than the original program. modification information .PR also eliminates all
c) Efficiency measures the practicality redundant tests.
(computational cost) of a technique. 5) McCabe‐ based Regression Test Coverage: have
d) Generality is the measure of ability of a got two techniques [12]
technique to handle complex modifications, a) Reachability regression Test selection McCabe‐
realistic language constructs and realistic testing based metric (RTM) provides an upper bound of
applications. Various techniques of Regression Test the number of regression tests selected that assures
Selection as given by various researchers are: the coverage of requirements affected by the
1) Modified non core function technique: defined modification atleast once.
in [5] selects test cases that exercise functions in b) Data flow Slices regression Test McCabe‐based
program that have been changed or deleted in metric (STM) in which we extend McCabe
producing changed program, or that exercise complexity to deal with variable/data
functions using variables or structures that have modifications. It provides upper and lower bounds
been deleted or changed in producing changed to cover the affected definition‐use pairs created by
program data modifications.
2) Modification focused Minimization technique:
uses Fischer’s approach [6], which seeks a subset of 2.3 TEST CASE PRIORITIZATION
test suite that is minimal in covering all functions This technique of regression testing prioritize
in program identified as changed. the test cases so as to increase a test suite‘s
rate of fault detection that is how quickly a
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151‐9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 102
test suite detects faults in the modified program number of statements by sorting them in order of
to increase reliability. This is of two types:(1) coverage achieved. If test cases are having same
General prioritization[13] which attempts to number of statements they can be ordered pseudo
select an order of the test case that will be randomly.
effective on average subsequent versions of P4: Additional statement coverage
software .(2)Version Specific prioritization which Prioritization: which is similar to total coverage
is concerned with particular version of the
prioritization, but depends upon feedback about
software.
coverage attained to focus on statements not yet
2.3.1. Test Case Prioritization problem covered. This technique greedily selects a test case
that has the greatest statement coverage and then
Rothermel et al. [13] define the test case iterates until all statements are covered by at least
prioritization problem as follows: one test case. The moment all statements are
Given:T, a test suite; PT, the set of permutations of covered the remaining test cases undergo
T; f, a function from PT to the real numbers. Additional statement coverage prioritization by
Problem: Find T′ PT such that ( T″) (T″ resetting all statements to “not covered”.
PT)(T″≠T′) [f(T′)≥f(T″)].Here, PT represents the set P5:Total FEP prioritization:
of all possibleprioritizations (orderings) of T and f In which prioritization is done on the probability
is a function that, applied to any such ordering, of exposing faults by test cases. Mutation analysis
yields an award value for that ordering. is used to approximate the Fault‐Exposing‐
Potential (FEP) of a test case. The cost of
2.3.2. Test Case Prioritization Techniques calculating FEP using mutation analysis is quite
high which motivates the search of cost effective
There are 18 different test case prioritizations approximations of FEP.
techniques [14] numbered P1‐P18 which are P6:Additional FEP prioritization:
divided into three groups as shown in figure 2. The total FEP prioritization is extended to
Comparator techniques: Additional FEP prioritization as the total statement
P1: Random ordering: in which the test cases in coverage prioritization is extended to Additional
test suite are randomly prioritized. statement coverage prioritization.
P2: Optimal ordering: in which the test cases Function level techniques: (Coarse Granularity)
areprioritized to optimize rate of fault detection.
P7: Total function coverage prioritization:
As faults are determined by respective test cases It is similar to total statement coverage but instead
and we have programs with known faults, so test of using statements it uses functions. As it has got
cases can be prioritized optimally. coarse granularity so the process of collecting
function level traces is cheaper than the process of
collecting statement level traces in total statement
coverage.
P8:Additional function coverage prioritization:
It is similar to Additional statement coverage
prioritization with only difference that instead of
statements, it is considering function level
coverage.
P9: Total FEP prioritization (function level):
It is analogous to Total FEP prioritization with only
difference that instead of using statements it is
Figure2. Classification of test case prioritization using functions.
P10: Additional FEP prioritization (function level):
Statement level techniques: (Fine Granularity) This technique is similar to Additional FEP
P3: Total statement coverage prioritization: prioritization with only difference that instead of
In which test cases are prioritized in terms of total using statements it is using functions.
P11: Total Fault Index (FI) prioritization:
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151‐9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 103
Fault proneness is a measurable software attribute P18: Additional Diff with FEP prioritization: Total
which Diff with FEP prioritization is extended to
Additional Diff with FEP prioritization in a similar
is used for this technique. Some functions are likely way as Total function coverage prioritization is
to contain more faults than others, so the fault extended to Additional function coverage
index is generated using following steps: (1) a set prioritization
of measurable attributes [15] for each function.(2)
the metrics are standardized.(3) principal 2.3.3. Search Algorithms for Test Case
component analysis [16] which reduces the set of Prioritization
standardized metrics.(4) finally they are combined There are many search techniques for test case
to a linear function to obtain one fault index per prioritization which are being developed and
function. unfolded by various researchers in the field.
Now for each test case all the fault indexes for 1) Greedy algorithm: works on the next best search
every function are added to get total fault index for philosophy. It [18] minimizes the estimated cost to
each test case. Then sort the test cases in decreasing reach a particular goal. Its advantage is that it is
order of these sums to get the result for Total Fault cheap in both execution time and implementation.
Index (FI) prioritization. The cost of this prioritization is O (mn) for
P12: Additional Fault Index (FI) prioritization: as program containing m statements and test suite
Total function coverage prioritization is extended containing n test cases.
to Additional function coverage prioritization 2) Additional Greedy algorithm: this algorithm [18]
similarly the Total Fault Index (FI) prioritization is uses the feedback from previous selections. It
extended to Additional Fault Index (FI) selects the maximum weight element from the part
prioritization. that is not already consumed by previously
P13: Total FI with FEP coverage prioritization: this selected elements. Once the complete coverage is
technique combines both Total FI and FEP achieved the remaining test cases are prioritized by
coverage prioritization to achieve a better rate of reapplying the Additional Greedy algorithm. The
fault detection. cost of this prioritization is O (mn2) for program
P14: Additional FI with FEP coverage containing m statements and test suite containing n
prioritization: as Total function coverage test cases.
prioritization is extended to Additional function 3) 2‐Optimal algorithm: Traveling Salesman
coverage prioritization similarly Total FI with FEP Problem (TSP) is defined as “find the cycle of
coverage prioritization is extended to Additional FI minimum cost that visits each of the vertices of a
with FEP coverage prioritization. weighted graph G at least once” is solved by K‐
P15: Total Diff prioritization: this technique is optimal approach [19]. In case of 2‐Optimal
similar to Total Fault Index (FI) prioritization with algorithm [18] the value of k=2. The cost of this
the difference that Total FI prioritization require prioritization is O (mn3) for program containing m
collection of metrics whereas Total Diff statements and test suite containing n test cases.
prioritization require only the calculation of 4) Hill Climbing: it is one of the popular local
syntactic differences between the program and the search algorithms with two variations; steepest
modified program. Diff means that only syntactic ascent and next best ascent. It is very easy and
differences are given consideration. inexpensive to execute. However, this has got cons
P16: Additional Diff prioritization: Total Diff of dividing O (n2) neighbors and is unlikely to
prioritization is extended to Additional Diff scale. Steps of algorithm are explained in [18].
prioritization in a similar way as Total function 5) Genetic Algorithms (GAs): is a search technique
coverage prioritization is extended to Additional [18] based on the Darwin’s theory of survival of
function coverage prioritization. the fittest. The population is a set of randomly
P17: Total Diff with FEP prioritization: is exactly generated individuals. Each individual is
similar to Total FI with FEP coverage represented by variables/parameters called genes
prioritization, except that it is dependent upon or chromosomes. The basic steps of Genetic
changed data derived from diff. Algorithm are (1) Encoding (2) Selection (3) Cross
over (4) Mutation.
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151‐9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 104
6) PORT version1.1 (Prioritization of Requirements Thus, each approach –Case Studies and Controlled
for Testing) for Test Case Prioritization proposed Experiments have different disadvantages and
by Hema Srikant 17] which is Requirement Based advantages and comparison and analysis of all the
Test Case Prioritization technique. The technique techniques discussed in this paper requires both.
uses three prioritization factors (1) .Customer The two major Challenges are: (1) finding objects
assigned priority on requirements (2).Requirement of study (programs, releases of these programs,
Complexity (3).Requirement volatility. You can test suites, fault data) (2) selecting the appropriate
find details in [17]. approach to answer research questions.
2.4. HYBRID APPROACH 4. CONCLUSION
The fourth regression technique is the Hybrid Regression testing is done in the maintenance
Approach of both Regression Test Selection and phase of the software development life cycle to
Test Case Prioritization. There are number of retest the software for the modifications it has
researchers working on this approach and they undergone. Approximately 50% [1] of the software
have proposed many algorithms for it. For cost is involved in the maintenance phase so
example, researchers are working hard to come up with best
1) Test Selection Algorithm: proposed by results by developing new Regression Testing
Aggarwal et al. Implementation of algorithm [20]: techniques so that the expenditure made in this
(a) Input (b) Test Selection algorithm: Adjust phase can be reduced to some extent. Many
module and Reduce module (c) output. foundations like US National Science Foundation
2) Hybrid technique proposed by Wong et al (NSF), Galileo Research Group at Oregon State
which combines minimization, modification and University, Mapstext Group at University of
prioritization based selection using test history Nebraska Lincoln, Aristotle Research Group at
[21]. Georgia Institute of Technology, NCSU Software
3) Hybrid technique proposed by Yogesh Singh et Engineering Realsearch Group, National Natural
al is based on Regression Test Selection and Test Science Foundation of China are few of the many
Case Prioritization. The proposed algorithm in names who are working on development of new
detail can be studied in [22]. regression testing techniques and improving
3. Approaches and Challenges existing techniques on different aspects.
The possible approaches [14] which can be Through this paper we have discussed Regression
performed so that all the techniques discussed Testing techniques and have further classified each
above can be compared and analyzed are: (1) one of them respectively as explained by various
Controlled Experiments (2) Case Studies. authors, explaining Regression Test Selection and
1). Controlled Experiments: are performed on Test Case Prioritization in detail with Search
objects drawn from the field and further created Algorithms for Test Case Prioritization. Through
and manipulated in controlled environment. The this paper we have tried to, explain the complete
advantage of controlled experiment is that structure of Regression Testing, enlighten up the
independent variables (eg. test suite, modification possible areas of Regression Testing to make
patterns, and fault types) can be changed to researchers understand its importance and scope
determine their effect on dependent variables. The and motivate new researchers who are planning to
disadvantage of this approach is the threat to start their research “to work on it”.
external validity shaped by the “manufacturing” of
faults, test cases and modifications. REFERENCES
2). Case Studies: are performed on existing [1] K.K.Aggarwal & Yogesh Singh, “Software Engineering
programs, taken “from the field”, that have fault Programs Documentation, Operating Procedures,” New Age
International Publishers, Revised Second Edition – 2005.
data, several versions and existing test suites. The
[2] Sebastian Elbaum, Praveen Kallakuri, Alexey G.
advantage of this approach is it is “real” and Malishevsky, Gregg Rothermel, Satya Kanduri,
reduces the cost required to be artificially created “Understanding the Effects of Changes on the Cost‐
in Controlled Experiments. The disadvantage of Effectiveness of Regression Testing Techniques,” Journal of
this approach is that certain factors are not Software Testing, Verification, and Reliability, 13(2)
pages:65‐83, June 2003.
controlled, which makes replication difficult.
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151‐9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 105
[3] H. Leung and L. White, “Insights into regression testing,” [19] S Lin, “Computer Solutions of the Travelling Salesman
In Proceedings of the Conference on Software Maintenance, Problem,” Bell System Technical J., vol. 44, pages2245‐2269,
pages 60‐69, Oct. 1989. 1965.
[4] Rothermel R., “Efficient Effective Regression Testing [20] K. K. Aggrawal, Yogesh Singh, A. Kaur, “ Code
Using Safe Test Selection Techniques,” Ph.D Thesis, coverage based technique for prioritizing test cases for
Clemson University, May, 1996. regression testing,” ACM SIGSOFT Software Engineering
Notes, vol 29 Issue 5 September 2004.
[5] Y. Chen, D. Rosenblum, and K. Vo. TestTube, “A system [21] W. E.Wong, J. R. Horgan, S. London and H.Agrawal, “A
for selective regression testing,” In Proceedings of the 16th study of effective regression testing in practice,” In
International Conference on Software Engineering, pages Proceedings of the 8th IEEE International Symposium on
211‐220, May 1994. Software Reliability Engineering (ISSREʹ 97), pages 264‐274,
[6] K. Fischer, F. Raji, and A. Chruscicki, “A methodology for November 1997.
retesting modifed software,” In Proceedings of the National [22] Yogesh Singh, Arvinder Kaur, Bharti Suri, “A new
Telecommunications Conference B‐6‐3, pages 1‐6, Nov. 1981. technique for version‐specific test case selection and
[7] R. Gupta, M. J. Harrold, and M. Soffa, “An approach to prioritization for regression testing,” Journal of the CSI ,Vol.
regression testing using slicing,” In Proceedings of the 36 No.4, pages 23‐32, October‐December 2006.
Conference on Software Maintenance, pages 299‐308, Nov.
1992.
[8] N.Mansour, and K. El‐Faikh, “Simulating annealing and
genetic algorithms for optimal regression testing,” Journal of
Software Maintenance, Vol. 11, pages 19‐34, 1999.
[9] M.J.Harrold, R.Gupta, and M.L. Soffa,“ A methodology
for controlling the size of the test suite, ” ACM Transaction
on Software Engineering and Methodology, pages 270‐285,
July 1993.
[10] H.Agrawal,J.R. Horgan, and E.W. , Krauser,
“Incremental regression testing,” In: Proc. Conference on
Software Maintenance, pages 348‐357,1993.
[11] Bahsoon, R. Mansour, N, “Methods and metrics for
selective regression testing,” In Computer Systems and
Applications, ACS/IEEE International Conference, pages
463‐465, 2001.
[12] T. McCabe, “A complexity measure,” IEEE Trans. On
Software Engineering, 2(3), pages 308‐319, 1976.
[13] G. Rothermel, R.H. Untch, C. Chu, and M.J. Harrold,
“Prioritizing Test Cases for Regression Testing,” IEEE Trans.
Software Eng., vol. 27, no. 10, pages 929‐948, Oct. 2001.
[14] Sebastian Elbaum, Member, IEEE, Alexey G.
Malishevsky, Student Member, IEEE, and Gregg Rothermel,
Member, IEEE, “Test case prioritization: A family of
empirical studies,” IEEE Transactions on Software
Engineering, vol. 28, NO.2, pages 159‐182, Feb.2002.
[15] S.G. Elbaum and J.C. Munson, “A Standard for the
Measurement of C Complexity Attributes,” Technical Report
TR‐CS‐98‐02, Univ. of Idaho, Feb. 1998.
[16] R.A. Johnson and D.W. Wichorn, “Applied Multivariate
Analysis,” third ed. Englewood Cliffs, N.J.: Prentice Hall,
1992.
[17] Hema Srikanth, Laurie Williams, Jason Osborne,
“System test case prioritization of new and regression test
cases,” Proceedings of the seventh international workshop
on Economics
driven software engineering research, pages 64‐73, May
2005.
[18] Zheng Li, Mark Harman, and Robert M. Hierons,
“Search algorithms for regression test case prioritization,”
IEEE Trans. On Software Engineering, vol 33, no.4, April
2007.