Simple Statistical Methods for Software Engineering Data and Patterns 1st Edition C. Ravindranath Pandian - The full ebook version is just one click away
Simple Statistical Methods for Software Engineering Data and Patterns 1st Edition C. Ravindranath Pandian - The full ebook version is just one click away
https://ebookfinal.com/download/statistical-methods-for-fuzzy-
data-1st-edition-reinhard-viertl/
https://ebookfinal.com/download/statistical-methods-for-spatial-data-
analysis-1st-edition-oliver-schabenberger/
https://ebookfinal.com/download/statistical-methods-for-microarray-
data-analysis-methods-and-protocols-1st-edition-andrei-y-yakovlev/
https://ebookfinal.com/download/statistical-mechanics-made-simple-a-
guide-for-students-and-researchers-daniel-c-mattis/
Nonparametric Statistical Methods For Complete and
Censored Data 1st Edition M.M. Desu (Author)
https://ebookfinal.com/download/nonparametric-statistical-methods-for-
complete-and-censored-data-1st-edition-m-m-desu-author/
https://ebookfinal.com/download/statistical-methods-for-psychology-
seventh-edition-david-c-howell/
https://ebookfinal.com/download/object-oriented-software-engineering-
using-uml-patterns-and-java-3rd-intern-edition-bruegge/
https://ebookfinal.com/download/statistical-methods-for-the-analysis-
of-biomedical-data-second-edition-robert-f-woolson/
C. Ravindranath Pandian
Murali Kumar S K
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2015 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and
publishers have attempted to trace the copyright holders of all material reproduced in this publication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information stor-
age or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copy-
right.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222
Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that pro-
vides licenses and registration for a variety of users. For organizations that have been granted a photo-
copy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Preface..........................................................................................................xvii
Acknowledgment...........................................................................................xix
Introduction..................................................................................................xxi
Authors....................................................................................................... xxiii
Section I Data
1 Data, Data Quality, and Descriptive Statistics.......................................3
The Challenge That Persists..........................................................................3
Bringing Data to the Table Requires Motivation..........................................5
Data Quality.................................................................................................5
On Scales.............................................................................................5
Error....................................................................................................6
Data Stratification...............................................................................7
Visual Summary...........................................................................................7
Numerical Descriptive Statistics (Numerical Summary of Data)................10
Special Statistics................................................................................12
Three Categories of Descriptive Statistics...........................................12
Case Study: Interpretation of Effort Variance Descriptive Statistics............12
Application Notes.......................................................................................14
Concluding Remarks..................................................................................14
Review Questions.......................................................................................15
Exercises.....................................................................................................16
Appendix 1.1: Definition of Descriptive Statistics.......................................16
Number of Data Points......................................................................16
Sum...................................................................................................16
Variance.............................................................................................16
Standard Deviation............................................................................16
Maximum..........................................................................................17
Minimum..........................................................................................17
Range................................................................................................17
Mode.................................................................................................17
Median..............................................................................................17
Mean.................................................................................................18
Kurtosis (Flatness of Distribution).....................................................18
Skewness (Skew of Distribution)........................................................18
References...................................................................................................18
Suggested Readings....................................................................................19
2 Truth and Central Tendency.................................................................21
Mean..........................................................................................................21
Uncertainty in Mean: Standard Error................................................22
Median.......................................................................................................23
Mode..........................................................................................................25
Geometric Mean.........................................................................................25
Harmonic Mean.........................................................................................27
Interconnected Estimates............................................................................27
Weighted Mean..........................................................................................28
Robust Means.............................................................................................29
Trimmed Mean.................................................................................29
Winsorized Mean..............................................................................30
Midhinge...........................................................................................30
Midrange...........................................................................................30
Tukey’s Trimean................................................................................30
Mean Derived from Geometrical Perspectives...................................30
Two Categories...........................................................................................30
Category 1.........................................................................................30
Category 2.........................................................................................31
Truth..........................................................................................................31
Application Notes.......................................................................................32
Managing Software Projects Using Central Tendency Values............32
Making Predictions...........................................................................32
Case Study: Shifting the Mean...................................................................33
Review Questions...................................................................................... 34
Exercises.................................................................................................... 34
References.................................................................................................. 34
Suggested Reading..................................................................................... 34
3 Data Dispersion.....................................................................................35
Range-Based Empirical Representation......................................................35
Dispersion as Deviation from Center..........................................................38
Average Deviation..............................................................................39
Average Absolute Deviation...............................................................39
Median Absolute Deviation...............................................................39
Defect Density.................................................................................. 90
Defect Classification..........................................................................91
Reliability..........................................................................................91
Examples of Process Metrics.......................................................................91
Review Effectiveness..........................................................................91
Test Effectiveness...............................................................................91
Test Coverage....................................................................................92
Subprocess Metrics.....................................................................................92
Converting Metrics into Business Information...........................................93
Project Dashboard.............................................................................93
Product Health Report......................................................................95
Statistical Process Control Charts......................................................96
Case Study: Early Size Measurements.........................................................96
Project Progress Using Earned Value Metrics..............................................97
Tracking Progress..............................................................................97
Tracking Project Cost........................................................................98
Review Questions.....................................................................................100
Exercises...................................................................................................101
References.................................................................................................101
Suggested Readings..................................................................................102
7 Maintenance Metrics...........................................................................103
Fusion of Frameworks in Software Maintenance......................................103
Maintainability Index......................................................................105
Change Requests Count..................................................................105
Customer Satisfaction Index............................................................106
Resource Utilization........................................................................107
Service-Level Agreement Compliances............................................107
Percentage of On-Time Delivery......................................................109
Enhancement Size...........................................................................109
Bug Complexity............................................................................... 110
Effort Variance (EV)........................................................................ 111
Schedule Variance (SV)................................................................... 111
Quality............................................................................................ 111
Quality of Enhancement........................................................ 111
Quality of Bug Fix.................................................................. 111
Productivity.....................................................................................112
Time to Repair (TTR)....................................................................112
Backlog Index..................................................................................113
Bug Classification............................................................................113
Fix Quality......................................................................................113
Refactoring Metrics.........................................................................113
Reliability........................................................................................113
Metric-Based Dashboards.........................................................................113
Review Questions..................................................................................... 115
Exercises................................................................................................... 115
References................................................................................................. 115
Suggested Readings.................................................................................. 116
8 Software Test Metrics.......................................................................... 117
Project Metrics.......................................................................................... 117
Schedule Variance............................................................................ 117
Effort Variance................................................................................ 117
Cost................................................................................................. 118
Human Productivity........................................................................ 118
Requirement Stability...................................................................... 118
Resource Utilization........................................................................ 118
Customer Satisfaction...................................................................... 119
Test Effectiveness............................................................................. 119
Process Metrics......................................................................................... 119
Defect Removal Efficiency...............................................................120
Test Cases Count.............................................................................120
Test Coverage..................................................................................121
Functionality Coverage..........................................................121
Code Coverage.......................................................................121
Percentage of Bad Fix......................................................................122
Product Metrics........................................................................................122
Defect Counts.................................................................................122
Defect Arrival Rate.................................................................122
Defect Closure Rate...............................................................122
Component Defect Count......................................................123
Component Defect Density.............................................................123
Defect Classification........................................................................124
Testing Size: Test Case Point....................................................................124
Risk Metric...............................................................................................125
Predicting Quality....................................................................................126
Metrics for Test Automation.....................................................................126
Return on Investment......................................................................126
Percentage Automatable...................................................................126
Automation Progress........................................................................127
Case Study: Defect Age Data....................................................................127
Review Questions.....................................................................................128
Exercises...................................................................................................129
References.................................................................................................129
Suggested Readings..................................................................................129
9 Agile Metrics.......................................................................................131
Classic Metrics: Unpopular Science..........................................................132
Two Sides of Classic Metrics.....................................................................133
Metrics for Agile: Humanization..............................................................134
The Price of Humanization.......................................................................134
Common Agile Metrics............................................................................135
Velocity............................................................................................135
Story Point.......................................................................................135
Technical Debt................................................................................136
Tests................................................................................................136
Level of Automation........................................................................136
Earned Business Value (EBV)..........................................................136
Burn-Down Chart...........................................................................136
Burn-Up Chart................................................................................136
Burn Up with Scope Line................................................................137
Adding More Agile Metrics......................................................................139
Case Study: Earned Value Management in the Agile World.....................140
Review Questions..................................................................................... 141
Exercise..................................................................................................... 141
References................................................................................................. 141
Suggested Reading....................................................................................142
Exercises...................................................................................................229
References.................................................................................................229
Suggested Readings..................................................................................230
14 Law of Compliance: Uniform Distribution.........................................231
Bounded Distribution...............................................................................233
Random Number Generators....................................................................233
Shuttle Time............................................................................................ 234
Parkinson’s Law....................................................................................... 234
Censored Process......................................................................................235
Perfect Departure.....................................................................................235
Estimating Calibration Uncertainty with Minimal Information..............235
Review Questions.....................................................................................237
Exercises...................................................................................................237
References.................................................................................................237
Suggested Readings..................................................................................237
15 Law for Estimation: Triangular Distribution.....................................239
Bell Curve Morphs into a Triangle...........................................................240
Mental Model for Estimation...................................................................241
Mean........................................................................................................242
Median.....................................................................................................243
Other Statistics.........................................................................................245
Skew.........................................................................................................245
Three-Point Schedule Estimation..............................................................247
Beta Option..............................................................................................247
Triangular Risk Estimation......................................................................248
Parameter Extraction................................................................................249
Review Questions.....................................................................................250
Exercises...................................................................................................250
References.................................................................................................250
16 The Law of Life: Pareto Distribution—80/20 Aphorism....................253
Structure of Pareto....................................................................................254
An Example..............................................................................................257
The 80/20 Law: Vital Few and Trivial Many............................................257
Generalized Pareto Distribution.............................................................. 260
Duane’s Model..........................................................................................261
Tailing a Body..........................................................................................261
Review Questions.....................................................................................262
Exercises...................................................................................................262
References.................................................................................................262
This book is a tribute to great Statisticians, scholars, and teachers whose ideas are
quoted throughout this book in various contexts. These pearls of wisdom have
helped us to connect our book with the evolution of science, knowledge and engi-
neering. Eventhough there are many books on statistics, there are few dedicated
to the application of statistical methods to software engineering. Pure textbooks
provide scholarly treatment, whereas practitioners need basic understanding and
application knowledge. Very few statistical books provide application knowledge to
software engineers. We have been working toward bridging this gap for about two
decades and have come out with the current book.
Statistical methods are often discussed in the context of six sigma, Capability
Maturity Model Integrated (CMMI), establishing capability baselines, and con-
structing process performance models. Driven by CMMI auditors, such practices
have become rituals that rely heavily on automated statistical packages, which are
rarely well understood. We have been promoting excel-based solution to statistics
and have presented practical solutions, such as those achieved in this book.
Karl Pearson
We also realize that sophisticated statistics is not the ideal approach to solve
problems. Simpler techniques provide easy solutions that connect with the intu-
ition of problem solvers. Although sophisticated techniques sound impressive but
merely academic, simpler techniques are flexible and can easily penetrate to the root
of the problem. In this book, we have consciously selected simpler tools. We have
also simplified several standard techniques.
The techniques presented in this book appear to us as a minimum set of intel-
lectual tools for software engineers and managers. True software engineering
xvii
can happen only when data are collected and these statistical methods are used.
Moreover, the statistical management of processes is possible only when managers
master these techniques.
Learning these techniques in the context of software engineering will certainly
help budding engineers and fresh recruits. The examples provided in this book will
provide a deep insight into software engineering and management.
This book can be used extensively as a guidebook for training software engi-
neers and managers at different levels. It will be a very valuable asset in the hands
of quality professionals who collect data and create models.
This book also exposes practical software engineering problems and solutions to
aspiring engineering graduates and make them industry ready.
Generally, this book is a guide for professionals to think objectively with data.
It will help them to mine data and extract meanings. Some of the techniques pro-
vided in the book are excellent prediction tools, which would give foresight to those
who apply them.
This book would not have become a reality without fruitful feedback from several
software professionals, quality managers, and project managers who have taken
our training and consultancy services. We also acknowledge Software Process
Improvement Network (SPIN) for presenting some of these concepts through vari-
ous SPIN chapters in India in an attempt to propagate these methods. All the SPIN
coordinators we interacted with have provided excellent application suggestions.
We thank those organizations who have shared their metric problems with us
for analysis and resolution. They eventually provided us research opportunities that
helped us gain deeper knowledge. We also thank many research scholars who have
interacted with us and taken our research support in the context of data mining
and artificial neural network.
We thank the professors and correspondents of many colleges in India for help-
ing us interact with students. We also thank Project Management Institute (PMI)
chapters and project management institutes who gave us opportunities to present
quantitative techniques to managers.
Rathna and Samuel helped by offering a wonderful review and criticism of
Chapter 8. Swaminathan contributed to Chapter 21 by reviewing the chapter and
making valuable suggestions. Shanti Harry helped us with references and suggested
readings. We thank all these well wishers.
Finally, we thank Mr. John Wyzalek who provided moral support and editorial
help. He made serious decisions about the scope of this book and helped us make a
tough decision to leave some chapters for the future and focus on the few we have
selected for this publication.
xix
Soneto.
Quien libre está, no viva
descuidado,
que en un instante puede
estar captivo,
y el corazón helado y más
esquivo
tema de estar en llamas
abrasado.
Con la alma del soberbio y
elevado
tan áspero es Amor y
vengativo,
que quien sin él presume de
estar vivo,
por él con muerte queda
atormentado.
Amor, que á ser captivo me
condenas,
Amor que enciendes fuegos
tan mortales,
tú que mi vida afliges y
maltratas:
Maldigo dende agora tus
cadenas,
tus llamas y tus flechas, con
las cuales
me prendes, me consumes
y me matas.
Rimas provenzales.
ALCIDA
Mientras el sol sus rayos muy
ardientes
con tal furia y rigor al mundo
envía,
que de Nymphas la casta
compañía
por los sombríos mora y por
las fuentes.
Y la cigarra el canto
replicando,
se está quejando,
pastora canta,
con gracia tanta,
que enternescido
de haberte oído,
el poderoso cielo de su
grado
fresco licor envíe al seco
prado.
DIANA
Mientras está el mayor de los
planetas
en medio del oriente y del
ocaso,
y al labrador en descubierto
raso
más rigurosas tira sus
saetas.
Al dulce murmurar de la
corriente
de aquesta fuente,
mueve tal canto,
que cause espanto,
y de contentos
los bravos vientos,
el ímpetu furioso
refrenando,
vengan con manso espíritu
soplando.
ALCIDA
Corrientes aguas, puras,
cristalinas,
que haciendo todo el año
primavera,
hermoseáis la próspera
ribera
con lirios y trepadas
clavellinas,
el bravo ardor de Phebo no
escaliente
tan fresca fuente,
ni de ganado
sea enturbiado
licor tan claro,
sabroso y raro,
ni del amante triste el lloro
infame
sobre tan lindas aguas se
derrame.
DIANA
Verde y florido prado, en do
natura
mostró la variedad de sus
colores
con los matices de árboles y
flores,
que hacen en ti
hermosíssima pintura.
En ti los verdes ramos sean
essentos
de bravos vientos;
medres, crezcas
en hierbas frescas,
nunca abrasadas
con las heladas,
ni dañe á tan hermoso y
fértil suelo
el gran furor del iracundo
cielo.
ALCIDA
Aquí de los bullicios y
tempesta
de las soberbias cortes
apartados,
los corazones viven
reposados,
en sosegada paz y alegre
fiesta,
á veces recostados al
sombrío
á par del río,
do dan las aves
cantos suaves,
las tiernas flores
finos olores,
y siempre con un orden
soberano
se ríe el prado, el bosque, el
monte, el llano.
DIANA
Aquí el ruido que hace el
manso viento,
en los floridos ramos
sacudiendo,
deleita más que el popular
estruendo
de un numeroso y grande
ayuntamiento,
adonde las superbas
majestades
son vanidades:
las grandes fiestas,
grandes tempestas;
los pundonores,
ciegos errores,
y es el hablar contrario y
diferente
de lo que el corazón y el
alma siente.
ALCIDA
No tiende aquí ambición lazos
y redes,
ni la avaricia va tras los
ducados,
no aspira aquí la gente á los
estados,
ni hambrea las privanzas y
mercedes:
libres están de trampas y
passiones
los corazones;
todo es llaneza,
bondad, simpleza,
poca malicia,
cierta justicia;
y hacer vivir la gente en
alegría
concorde paz y honesta
medianía.
DIANA
No va por nuevo mundo y
nuevos mares
el simple pastorcillo
navegando,
ni en apartadas Indias va
contando
de leguas y monedas mil
millares.
El pobre tan contento al
campo viene
con lo que tiene,
como el que cuenta
sobrada renta,
y en vida escasa
alegre passa,
como el que en montes ha
gruesas manadas,
y ara de fértil campo mil
yugadas.
Soneto.
No puede darme Amor mayor
tormento,
ni la Fortuna hacer mayor
mudanza;
no hay alma con tan poca
confianza,
ni corazón en penas tan
contento.
Hácelo Amor, que esfuerza el
flaco aliento,
porque baste á sufrir mi
malandanza,
y no deja morir con la
esperanza
la vida, la aflicción ni el
sufrimiento.
¡Ay, vano corazón! ¡Ay, ojos
tristes!
¿por qué en tan largo
tiempo y tanta pena
nunca se acaba el llanto ni
la vida?
¡Ay, lástimas! ¿no os basta lo
que hecistes?
Amor ¿por qué no aflojas mi
cadena,
si en tanta libertad dejaste
Alcida?.
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
ebookfinal.com