Software Quality Metrics

Introduction
 Software metrics can be classified into three
categories
• Product metrics
• Process metrics
• Project metrics
Product metrics describe the characteristics of the product
such as size, complexity, design features, performance, and
quality level
 Process metrics can be used to improve software development
and maintenance.
Examples include the effectiveness of defect removal during
development, the pattern of testing defect arrival, and the
response time of the fix process.

Project metrics describe the project characteristics and execution.

Examples include the number of software developers, the
staffing pattern over the life cycle of the software, cost,
schedule, and productivity.
 Software quality metrics
• Software quality metrics are a subset of software metrics that
focus on the quality aspects of the product, process, and
project.
• These are more closely associated with process and product
metrics than with project metrics.
Software quality metrics can be further divided into three categories

• Product quality metrics

• In-process quality metrics
• Maintenance quality metrics
 Product Quality Metrics
• Definition of software quality consists of two levels: intrinsic
product quality and customer satisfaction.
This metrics include the following :-
• Mean time to failure
• Defect density
• Customer problems
• Customer satisfaction
• Mean Time to Failure:- It is the time between failures. This
metric is mostly used with safety critical systems such as the
airline traffic control systems, avionics, and weapons.

• Defect Density:-It measures the defects relative to the

software size expressed as lines of code or function point, etc.
i.e., it measures code quality per unit. This metric is used in
many commercial software systems.
• Customer Problems:-It measures the problems that customers
encounter when using the product. It contains the customer’s
perspective towards the problem space of the software, which
includes the non-defect oriented problems together with the
defect problems.
• The problems metric is usually expressed in terms
of Problems per User-Month (PUM).
• PUM = Total Problems that customers reported (true defect
and non-defect oriented problems) for a time period + Total
number of license months of the software during the period
Customer Problems Cont..

• Where,
Number of license-month of the software = Number of install
license of the software × Number of months in the calculation
period
• PUM is usually calculated for each month after the software is
released to the market, and also for monthly averages by year.
Customer Satisfaction

• Customer satisfaction is often measured by customer survey

data through the five-point scale −
• Very satisfied
• Satisfied
• Neutral
• Dissatisfied
• Very dissatisfied
Difference Between Errors, Defects, Faults, and Failures
(IEEE/ANSI)
• An error is a human mistake that results in incorrect
software.
• The resulting fault is an accidental condition that causes
a unit of the system to fail to function as required.
• A defect is an anomaly in a product.
• A failure occurs when a functional unit of a software-
related system can no longer perform its required
function or cannot perform it within specified limits
 The Defect Density Metric
• This metric is the number of defects over the opportunities for
error (OPE) during some specified time frame.
• We can use the number of unique causes of observed failures
(failures are just defects materialized) to approximate the
number of defects.
• The size of the software in either lines of code or function
points is used to approximate OPE.
Lines of Code 

• A line of code is any line of program text that is not a

comment or blank line, regardless of the number of statements
or fragments of statements on the line.
• This specifically includes all lines containing program
headers, declarations, and executable and non-executable
statements.
Lines of Code Possible variations Count only
executable lines
• Count executable lines plus data definitions
• Count executable lines, data definitions, and comments
• Count executable lines, data definitions, comments, and job
control language
• Count lines as physical lines on an input screen
• Count lines as terminated by logical delimiters
 Defect Rate for New and Changed Lines of Code
• Depends on the availability on having LOC counts for both the
entire produce as well as the new and changed code
• Depends on tracking defects to the release origin (the portion
of code that contains the defects) and at what release that code
was added, changed, or enhanced
Function Points

• A function can be defined as a collection of executable

statements that performs a certain task, together with
declarations of the formal parameters and local variables
manipulated by those statements.
• In practice functions are measured indirectly.
• Many of the problems associated with LOC counts are
addressed.
Measuring Function Points

• The number of function points is a weighted total of five

major components that comprise an application.
• Number of external inputs x 4
• Number of external outputs x 5
• Number of logical internal files x10
• Number of external interface files x 7
• Number of external inquiries x 4
Measuring Function Points (Cont’d)

• These are the average weighting factors. There are also low and
high weighting factors, depending on the complexity assessment
of the application in terms of the five components:-
• External input: low complexity, 3; high complexity, 6
• External output: low complexity, 4; high complexity, 7
• Logical internal file: low complexity, 7; high complexity, 15
• External interface file: low complexity, 5; high complexity, 10
• External inquiry: low complexity, 3; high complexity, 6
Measuring Function Points (Cont’d)
• With the weighting factors, the first step is to calculate the
function counts (FCs) based on the following formula:

• where wij are the weighting factors of the five components

by complexity level (low, average, high) and xij are the
numbers of each component in the application.
Measuring Function Points (Cont’d)
• The second step involves a scale from 0 to 5 to
assess the impact of 14 general system
characteristics in terms of their likely effect on
the application.
The 14 System Characteristics

• Data Communications
• Distributed functions
• Performance
• Heavily used configuration
• Transaction rate
• Online data entry
• End-user efficiency
 The 14 System Characteristics (Cont’d)

• Online update
• Complex processing
• Reusability
• Installation ease
• Operational ease
• Multiple sites
• Facilitation of change
The 14 System Characteristics (Cont’d)

• VAF is the sum of these 14 characteristics divided by 100 plus

0.65.
• Notice that if an average rating is given each of the 14 factors,
their sum is 35 and therefore VAF =1
• The final function point total is then the function count
multiplied by VAFFP = FC x VAF
Customer Problems Metric
• Customer problems are all the difficulties customers encounter when using the
product.
• They include:
• Valid defects
• Usability problems
• Unclear documentation or information
• Duplicates of valid defects (problems already fixed but not known to customer)User
errors
• The problem metric is usually expressed in terms of problems per user month
(PUM)
Customer Problems Metric (Cont’d)

• PUM = Total problems that customers reported for a time

period <divided by> Total number of license-months of the
software during the period
where
• Number of license-months = Number of the install licenses of
the software x Number of months in the calculation period
Approaches to Achieving a Low PUM

• Improve the development process and reduce the product

defects.
• Reduce the non-defect-oriented problems by improving all
aspects of the products (e.g., usability, documentation),
customer education, and support.
• Increase the sale (number of installed licenses) of the product.
Defect Rate and Customer Problems Metrics
• Problems per User-Month (PUM)
• Numerator
• Valid and unique product defects
• All customer problems (defects and non defects, first time and repeated)
• Denominator Size of product (KLOC or function point)
• Customer usage of the product (user-months)
• Measurement perspective
• Producer—software development organization
• Customer
• Scope
• Intrinsic product quality
• Intrinsic product quality plus other factors
Customer Satisfaction Metrics
• Issues
• Customer Problems
• Defects
• Customer satisfaction is often measured by customer survey data via the five-point
scale:
– Very satisfied
– Satisfied
– Neutral
– Dissatisfied
– Very dissatisfied
Examples Metrics for Customer Satisfaction

• Percent of completely satisfied customers

• Percent of satisfied customers (satisfied and completely
satisfied)
• Percent of dissatisfied customers (dissatisfied and completely
dissatisfied)
• Percent of non satisfied customers (neutral, dissatisfied, and
completely dissatisfied)
In-Process Quality Metrics

• Defect density during machine testing

• Defect arrival pattern during machine testing
• Phase-based defect removal pattern
• Defect removal effectiveness
Defect Density During Machine Testing

• Defect rate during formal machine testing (testing after code is

integrated into the system library) is usually positively
correlated with the defect rate in the field.
• The simple metric of defects per KLOC or function point is a
good indicator of quality while the product is being tested.
Defect Density During Machine Testing (Cont’d)

• Scenarios for judging release quality:

• If the defect rate during testing is the same or lower than that
of the previous release, then ask: Does the testing for the
current release deteriorate?
• If the answer is no, the quality perspective is positive.
• If the answer is yes, you need to do extra testing.
Defect Density During Machine Testing (Cont’d)

• Scenarios for judging release quality (cont’d):

• If the defect rate during testing is substantially higher than that
of the previous release, then ask: Did we plan for and actually
improve testing effectiveness?
• If the answer is no, the quality perspective is negative.
• If the answer is yes, then the quality perspective is the same or
positive.
Defect Arrival Pattern During Machine Testing

• The pattern of defect arrivals gives more information than

defect density during testing.
• The objective is to look for defect arrivals that stabilize at a
very low level, or times between failures that are far apart
before ending the testing effort and releasing the software.
Phase-Based Defect Removal Pattern

• This is an extension of the test defect density metric.

• It requires tracking defects in all phases of the development
cycle.
• The pattern of phase-based defect removal reflects the overall
defect removal ability of the development process.
Defect Removal Effectiveness

• DRE = (Defects removed during a development phase <divided

by> Defects latent in the product) x 100%The denominator can
only be approximated.
• It is usually estimated as:
• Defects removed during the phase +Defects found later.
• When done for the front end of the process (before code
integration), it is called early defect removal effectiveness.
• When done for a specific phase, it is called phase effectiveness.
 Metrics for Software Maintenance
• The goal during maintenance is to fix the defects as soon as
possible with excellent fix quality
• The following metrics are important:
• Fix backlog and backlog management index
• Fix response time and fix responsiveness
• Percent delinquent fixes
• Fix quality
 Fix Backlog

• Fix backlog is a workload statement for software maintenance.

• It is related to both the rate of defect arrivals and the rate at
which fixes for reported problems become available.
• It is a simple count of reported problems that remain at the end
of each time period (week, month, etc.)
Backlog Management Index (BMI)

• BMI = (Number of problems closed during the month

<divided by> Number of problem arrivals during the month) x
100%.
• If BMI is larger than 100, it means the backlog is reduced.
• If BMI is less than 100, then the backlog is increased.
Fix Response Time and Fix Responsiveness
• The fix response time metric is usually calculated as:
• Mean time of all problems from open to closed
• Metric may be used for different defect severity levels.
• Fix response time relates to customer satisfaction.
• But meeting agreed-to fix time is more than just achieving a
short fix time.
• A possible metric is the percentage of delivered fixes meeting
committed dates to customers.
Percent Delinquent Fixes

• The mean response time metric is a central tendency measure.

• A more sensitive metric is the percentage of delinquent fixes
(for each fix, if the turnaround time greatly exceeds the
required response time, it is classified as delinquent).
• Percent delinquent fixes = (Number of fixes that exceeded the
response time criteria by severity level <divided by> Number
of fixes delivered in a specified time) x 100%
 Percent Delinquent Fixes (Cont’d)

• This is not a real-time metric because it is for closed problems

only.
• For a real-time metric we must factor in problems that are still
open.
• We can use the following metric
• Real-Time Delinquency Index = 100 x Delinquent / (Backlog
+ Arrivals)
Fix Quality

• The number of defective fixes is another quality metric for

maintenance.
• The metric of percent defective fixes is simply the percentage
of all fixes in a time interval that are defective.
• Recording both the time the defective fix was discovered and
the time the fix was made to be able to calculate the latent
period of the defective fix.