See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220069502

Data Mining Static Code Attributes to Learn Defect Predictors.

Article · January 2007

DOI: 10.1109/TSE.2007.10 · Source: DBLP

CITATIONS READS
607 164

3 authors, including:

Tim Menzies Jeremy Greenwald

North Carolina State University 5 PUBLICATIONS   1,443 CITATIONS   
578 PUBLICATIONS   9,141 CITATIONS   
SEE PROFILE
SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Predicting Software Defects Across Project View project

Turning lead into gold. Hunting the Snark. Re-animating corpses. Sort of View project

All content following this page was uploaded by Tim Menzies on 13 June 2016.

The user has requested enhancement of the downloaded file.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 33, NO. 9, SEPTEMBER 2007 635

Comments __________________________________________________________________________________________________________

Comments on For evaluating the performance of a prediction model, another

set of accuracy measures is Recall and Precision, which is widely
“Data Mining Static Code Attributes
used in the used in the Information Retrieval area [2], [3]:
to Learn Defect Predictors”
TP TP
Recall ¼ ; Precision ¼ ;
Hongyu Zhang, Member, IEEE, and Xiuzhen Zhang TP þ FN TP þ FP
ð2Þ
2  Recall  Precision
F -measure ¼ :
Recall þ Precision
Abstract—In this correspondence, we point out a discrepancy in a recent paper,
“Data Mining Static Code Attributes to Learn Defect Predictors,” that was Recall is actually the same as pd, which defines the ratio of
published in this journal. Because of the small percentage of defective modules, detected true defective modules in comparison to the total number
using Probability of Detection (pd) and Probability of False Alarm (pf) as accuracy
of defective modules. Precision defines the ratio of correctly
measures may lead to impractical prediction models.
detected modules. A good prediction model should achieve high
Recall and high Precision. A single measure, the F-measure, is used
Index Terms—Defect prediction, accuracy measures, static code attributes,
empirical. to combine Recall and Precision. It is defined as the harmonic
mean of Precision and Recall (a more general form of F-measure is
Ç defined as a weighted harmonic mean of Precision and Recall). The
values of Recall, Precision, and F-measure are between 0 and 1; the
1 INTRODUCTION
higher, the better.
IN the January 2007 issue of this journal, a paper titled “Data Based on (1) and (2), we know that
Mining Static Code Attributes to Learn Defect Predictors” [1] was
published. In that paper, Probability of Detection (pd) and TP 1 1
Precision ¼ ¼ ¼ : ð3Þ
Probability of False Alarm (pf) are used to measure the accuracy T P þ F P 1 þ FT PP 1 þ NEGpf
P OSpd
of a defect prediction model. Their models generate average results
of pd ¼ 71% and pf ¼ 25%. The authors of [1] consider these results Using the data given in Fig. 3 and Fig. 12 of [1], we calculate the
satisfactory and draw their conclusions based on them. This Precision values based on (3). The results are shown in Table 2 (the
correspondence points out the limitation of using pd and pf as values in italic are the original data from [1]).
accuracy measures in imbalanced classification. Using the Recall/ We notice that the Precision and F-measure values are very low
Precision measures, we show that the models built in [1] are not for all data sets (except KC4). For example, for the CM1 data set,
satisfactory for practical use and should be improved. the Precision is 20.64 percent, which means that, if a module is
predicted as defective, the probability of it actually being defective
is only 20.64 percent. For the PC2 data set, if the prediction model
2 THE EVALUATION OF DEFECT PREDICTION MODELS
claims that a module is defective, the probability of it actually
Prediction of defective modules can be cast as a classification being defective is only 2.02 percent. These results are considered
problem in machine learning: Given training samples of modules unsatisfactory, although the pd values are high. Therefore, defect
with labels as defective (Positive) or nondefective (Negative), a prediction through such models would not be very useful in
classification model can be learned from the training data. The practice. Applying such models would defeat the very purpose of
model is then used to classify unknown modules. A prediction
defect prediction, which is about allocation of limited QA
model has four results: true positives (TP), false positives (FP), true
resources more efficiently (so that efforts can be concentrated on
negatives (TN), and false negatives (FN), as shown in Table 1. The
the potentially problematic modules).
total number of actual defective modules is denoted POS and the
The models with high pd and low pf do not necessarily lead to
total number of actual nondefective modules is denoted NEG.
accurate models with high precision. The reason is that the
To evaluate the accuracy of predictions, [1] uses the receiver
distribution of classes (defective or nondefective) is highly
operating characteristic (ROC) curves, which consist of pd and pf.
imbalanced. The number of nondefective modules is much greater
A single measure balance is also used to balance between pd and pf.
These measures are defined as follows: than the number of defective modules. As shown in Table 2, the
percentage of defective modules in each data set (except for the
TP FP KC4 data set) is very low (ranging from 0.4 percent to 12 percent).
pd ¼ ; pf ¼ ;
TP þ FN FP þ TN From (3), we can see that the Precision could be low if the NEG/
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð1Þ POS ratio is high. The only exception, the KC4 data set, has the
ð0  pfÞ2 þ ð1  pdÞ2
balance ¼ 1  pffiffiffi : NEG/POS ratio 1.04; therefore, a high pd=pf ratio leads to a high
2 Precision. For all other data sets, the number of nondefective
modules are 7-249 times more than the defective modules;
therefore, their Precision is low even though their pd is high and
. H. Zhang is with the School of Software, Tsinghua University, Beijing pf is low.
100084, China. E-mail: [email protected].
. X. Zhang is with the School of Computer Science and Information
Technology, RMIT University, Melbourne 3001, Australia.
TABLE 1
E-mail: [email protected].
The Results of a Prediction Model
Manuscript received 15 Jan. 2007; accepted 4 June 2007; published online
13 June 2007.
Recommended for acceptance by M. Harman.
For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference IEEECS Log Number TSE-0302-1206.
Digital Object Identifier no. 10.1109/TSE.2007.70706.

0098-5589/07/$25.00 ß 2007 IEEE Published by the IEEE Computer Society

Authorized licensed use limited to: RMIT University. Downloaded on November 23, 2008 at 22:39 from IEEE Xplore. Restrictions apply.
636 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 33, NO. 9, SEPTEMBER 2007

TABLE 2
The Prediction Results

Table 2 also gives the expected pf values for each data set if the REFERENCES
Precision reaches 60 percent. We can see that, in order to achieve [1] T. Menzies, J. Greenwald, and A. Frank, “Data Mining Static Code
Precision of 60 percent, the original pf values shown in Fig. 12 of Attributes to Learn Defect Predictors,” IEEE Trans. Software Eng., vol. 33,
[1] should be further improved. no. 1, Jan. 2007.
[2] R. Baeza-Yates and B. Ribeiro-Neto, Modern Information Retrieval. ACM
To verify our results, we have also repeated the study described Press, 1999.
in [1], using the same NASA data sets, the naive Bayes (with log- [3] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and
transforms) learner, and the WEKA tool. The results confirm that Techniques with Java Implementations. Morgan Kaufmann, 1999.
the prediction models proposed in [1] are impractical for software
defect prediction due to the low precisions.
. For more information on this or any other computing topic, please visit our
Digital Library at www.computer.org/publications/dlib.
3 CONCLUSION
In this comment, we have shown that the models built in [1] are
not satisfactory for practical use. We suggest using Recall/
Precision, instead of pd=pf, to measure the accuracy of a software
defect prediction model.

Authorized
View publication stats licensed use limited to: RMIT University. Downloaded on November 23, 2008 at 22:39 from IEEE Xplore. Restrictions apply.