UNIT III

SOFTWARE QUALITY METRICS

Software Measurement and Metrics
Measurement Theory

Measurement is defined as the process of assigning symbols, usually numbers, to represent an

attribute of the entity of interest, by rule. Characteristics of a good measurement are
Reliability - The outcome of the measurement process is reproducible. (Similar results are gotten
over time and across situations.)
Validity - The measurement process actually measures what it purports to measure.
Sensitivity - The measurement process shows variability in responses when it exists in the
stimulus or situation.
With respect to software, Measurement is the continuous process of defining, collecting, and
analyzing data on the software development process and its products in order to understand and
control the process and its products, and to supply meaningful information to improve that
process and its products.
Measurement is essential to achieving the basic management objectives of prediction,
progress, and process improvement. Software is usually measured to serve following purposes.
Characterization, i.e., the gathering of information about some characteristic of software
processes and products, with the goal of acquiring a better idea of ``what's going on."
Evaluation, i.e., judging some characteristic of a software process or product, for instance based
on historical data in the same development environment or data available from external sources.
Improvement, i.e., using a cause-effect relationship to identify parts of the process or product
that can be changed to obtain positive effects on some characteristic of interest, and collecting
data after the changes have been made to confirm or disconfirm whether the effect was positive
and assess its extent.
Prediction, i.e., identifying a cause-effect relationship among product and process
characteristics.
Tracking, i.e., the (possibly constant and regular) acquisition of information on some
characteristic of software processes and products over time, to understand if those characteristics
are under control in on-going projects.
Validation, i.e., validating the identified best practices experimentally.
Software Measurement process
The software measurement process must be an objective, orderly method for quantifying,
assessing, adjusting, and ultimately improving the development process. Data are collected based
on known, or anticipated, development issues, concerns, and questions. They are then analyzed
with respect to the software development process and products.
The measurement process is used to assess quality, progress, and performance throughout
all life cycle phases. The key components of an effective measurement process are:
Clearly defined software development issues and the measure (data elements) needed to provide
insight into those issues;
Processing of collected data into graphical or tabular reports (indicators) to aid in issue analysis;
Analysis of indicators to provide insight into development issues; and,
Use of analysis results to implement process improvements and identify new issues and
problems.

Figure shows the software measurement process. The activities required to design a
measurement process using this architecture are:
Developing a measurement process to be made available as part of the organization's standard
software
process;
Planning the process on projects and documenting procedures by tailoring and adapting the
process asset;
Implementing the process on projects by executing the plans and procedures; and
Improving the process by evolving plans and procedures as the projects mature and their
measurement needs change
.

Software Measurement Scales

Five measurement levels are generally used in Measurement Theory which are applicable
to software as well. Usually, the narrower the arrangement of acceptable changes, the smaller the
number of scales, and the more informative the scale. Measurement scales are hierarchal, and
each level scale possesses all the properties of lower scales as shown below. We can convert
higher level scales into lower level ones (i.e., ratio to interval or ordinal or nominal; or interval to
ordinal or nominal; or ordinal to nominal). Powerful analysis can be applied to data with more
informative scale of measurement.
Nominal Scale
This is the most primitive form of measurement. A scale is a nominal one if it divides the
set of entities into categories, with no particular ordering among them.
E.g. IEEE 802.1, 802.2, 802.3802.11.
Ordinal Scale
A scale is an ordinal one if it divides the set of entities into categories that are ordered
according to some order. There is no quantitative comparison.
E.g. programmer skill (low, medium, high).
Interval Scale
This scale captures information about the size of the intervals that separate classes. In an
interval scale, the exact difference between two values is the basis for meaningful statements.
E.g. programmer capability between: 60th and 80th percentile of population.

Ratio Scale

A measurement mapping that the preserves ordering, the size of intervals between
entities, and ratios between entities In a ratio scale, contrary to interval scale, there is an absolute
and non arbitrary zero point.
E.g. project A took twice as long as project B.
Absolute Scale
Absolute scale is the most informative in the measurement scale hierarchy [10]. In
absolute scale, measurement is done by counting method.
E.g. number of failures observed during integration testing can be measured only by counting
the number of failures observed.
Challenges with Software Measurement
Unambiguous measurement is vital in software development life cycle. There should be some
standardized measures which can
Be used as a common baseline for measurement;
Offer a point of reference for software measurers to verify their measurement results and
their ability to measure the same reference material;
Allow measurers to use the related reference concept, and thus to speak at the same level.
However measurement of software is challenging in nature Because of following reasons:
Software is an intangible product. It is an atypical product when compared to other
industrial products, in that it varies greatly in terms of size, complexity, design
techniques, test methods, applicability, etc.
There is little consensus on specific measures of software attributes, as illustrated by the
scarcity of international standard measures for software attributes, such as software
complexity and quality.
Software development is so complex that all models are weak approximations as they are
hard to validate.
Measurements are not common to all projects, organizations. Measures that work for one
project may not be applicable to another one.
Software Metrics

project.

Productivity
Attributes of Effective Software Metrics

Software quality metrics

Measurement enables the Organization to improve the software process; assist in
planning, tracking and controlling the software project and assess the quality of the software thus
produced. It is the measure of such specific attributes of the process, project and product that are
used to compute the software metrics. Metrics are analyzed and they provide a dashboard to the
management on the overall health of the process, project and product. Generally, the validation
of the metrics is a continuous process spanning multiple projects. The kind of metrics employed
generally account for whether the quality requirements have been achieved or are likely to be

achieved during the software development process. As a quality assurance process, a metric is
needed to be revalidated every time it is used. Two leading firms namely, IBM and HewlettPackard have placed a great deal of importance on software quality. The IBM measures the user
satisfaction and softwareacceptability in eight dimensions which are capability or functionality,
usability, performance, reliability, ability to be installed, maintainability, documentation, and
availability. For the Software Quality Metrics the Hewlett-Packard normally follows the five
Juran quality parameters namely the functionality,the usability, the reliability, the performance
and the serviceability. In general, for most software quality assurance systems the common
software metrics that are checked for improvement are the Source lines of code, cyclical
complexity of the code, Function point analysis, bugs per line of code, code coverage, number of
classes and interfaces, cohesion and coupling between the modules etc.
Common software metrics include:
Bugs per line of code
Code coverage
Cohesion
Coupling
Cyclomatic complexity
Function point analysis
Number of classes and interfaces
Number of lines of customer requirements
Order of growth
Source lines of code
Robert Cecil Martins software package metrics
Software Quality Metrics focus on the process, project and product. By analyzing the
metrics the organization the organization can take corrective action to fix those areas in the
process, project or product which are the cause of the software defects.
The de-facto definition of software quality consists of the two major attributes based on
intrinsic product quality and the user acceptability. The software quality metric encapsulates the
above two attributes, addressing the mean time to failure and defect density within the software
components. Finally it assesses user requirements and acceptability of the software. The intrinsic
quality of a software product is generally measured by the number of functional defects in the
software, often referred to as bugs, or by testing the software in run time mode for inherent
vulnerability to determine the software "crash" scenarios.In operational terms, the two metrics
are often described by terms namely the defect density (rate) and mean time to failure (MTTF).
Although there are many measures of software quality, correctness, maintainability, integrity and
usability provide useful insight.
Correctness
A program must operate correctly. Correctness is the degree to which the software
performs the required functions accurately. One of the most common measures is Defects per
KLOC. KLOC means thousands (Kilo) Of Lines of Code.) KLOC is a way of measuring the size
of a computer program by counting the number of lines of source code a program has.
Maintainability
Maintainability is the ease with which a program can be correct if an error occurs. Since
there is no direct way of measuring this an indirect way has been used to measure this. MTTC
(Mean time to change) is one such measure. It measures when a error is found, how much time it
takes to analyze the change, design the modification, implement it and test it.

Integrity
This measure the systems ability to with stand attacks to its security. In order to measure
integrity two additional parameters are threatand security need to be defined. Threat
probability that an attack of certain type will happen over a period of time. Security
probability that an attack of certain type will be removed over a period of time. Integrity =
Summation [(1 - threat) X (1 - security)]
Usability
How usable is your software application? This important characteristic of your application is
measured in terms of the following characteristics:
-Physical / Intellectual skill required to learn the system
-time required to become moderately efficient in the system.
-the net increase in productivity by use of the new system.
-subjective assessment(usually in the form of questionnaire on the new system)
Standard for the Software Evaluation
In context of the Software Quality Metrics, one of the popular standards that addresses
the quality model, external metrics, internal metrics and the quality in use metrics for the
software development process is ISO 9126.
Defect Removal Efficiency
Defect Removal Efficiency (DRE) is a measure of the efficacy of your SQA activities..
For eg. If the DRE is low during analysis and design, it means you should spend time improving
the way you conduct formal technical reviews.
DRE = E / ( E + D )
Where E = No. of Errors found before delivery of the software and D = No. of Errors found after
delivery of the software.
Ideal value of DRE should be 1 which means no defects found. If you score low on DRE it
means to say you need to re-look at your existing process. In essence DRE is a indicator of the
filtering ability of quality control and quality assurance activity . It encourages the team to
find as many defects before they are passed to the next activity stage. Some of the Metrics are
listed out here:
Test Coverage = Number of units (KLOC/FP) tested / total size of the system Number of tests
per unit size = Number of test cases per KLOC/FP Defects per size = Defects detected / system
size Cost to locate defect = Cost of testing / the number of defects located Defects detected in
testing = Defects detected in testing / total system defects Defects detected in production =
Defects detected in production/system size Quality of Testing = No. of defects found during
Testing/(No. of defects found during testing + No of acceptance defects found after delivery)
*100 System complaints = Number of third party complaints / number of transactions processed
Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for
Design and Documentation Test Execution Productivity = No of Test cycles executed / Actual
Effort for testing Test efficiency= (number of tests required / the number of system errors)
Measure

Metrics

Number of system enhancement requests per year Number of

maintenance fix requests per year User friendliness: call volume to
1.
Customer
customer service hotline User friendliness: training time per new user
satisfaction index
Number of product recalls or fix releases (software vendors) Number of
production re-runs (in-house information systems groups)

Normalized per function point (or per LOC) At product delivery (first 3
months or first year of operation) Ongoing (per year of operation) By
2. Delivered defect
level of severity By category or cause, e.g.: requirements defect, design
quantities
defect, code defect, documentation/on-line help defect, defect introduced
by fixes, etc.
3.
Responsiveness Turnaround time for defect fixes, by level of severity Time for minor vs.
(turnaround time) to major enhancements; actual vs. planned elapsed time (by customers) in
users
the first year after product delivery
McCabe's cyclomatic complexity counts across the system Halsteads
7. Complexity of
measure Card's design complexity measures Predicted defects and
delivered product
maintenance costs, based on complexity measures
8. Test coverage

Breadth of functional coverage Percentage of paths, branches or

conditions that were actually tested Percentage by criticality level:
perceived level of risk of paths The ratio of the number of detected faults
to the number of predicted faults.

9. Cost of defects

Business losses per defect that occurs during operation Business

interruption costs; costs of work-arounds Lost sales and lost goodwill
Litigation costs resulting from defects Annual maintenance cost (per
function point) Annual operating cost (per function point) Measurable
damage to your boss's career

Costs of reviews, inspections and preventive measures Costs of test

planning and preparation Costs of test execution, defect tracking, version
and change control Costs of diagnostics, debugging and fixing Costs of
10. Costs of quality
tools and tool support Costs of tools and tool support Costs of test case
activities
library maintenance Costs of testing & QA education associated with the
product Costs of monitoring and oversight by the QA organization (if
separate from the development and test organizations)
11. Re-work

Re-work effort (hours, as a percentage of the original coding hours) Reworked LOC (source lines of code, as a percentage of the total delivered
LOC) Re-worked software components (as a percentage of the total
delivered components)

12. Reliability

Availability (percentage of time a system is available, versus the time the

system is needed to be available) Mean time between failure (MTBF)
Mean time to repair (MTTR) Reliability ratio (MTBF / MTTR) Number
of product recalls or fix releases Number of production re-runs as a ratio
of production runs

Product Quality Metrics

The de facto definition of software quality consists of two levels: intrinsic product quality and
customer satisfaction. The metrics we discuss here cover both levels:
Mean time to failure
Defect density
Customer problems

 Customer satisfaction.

1.
2.
3.
4.

1.
2.

1.
2.
3.

The MTTF metric is most often used with safety critical systems such as airline traffic control
systems, avionics and weapons.
An error is a human mistake that results in incorrect software.
The Resulting fault is an accidental condition that causes a unit of the system to fail to function
as required.
A defect is an anomaly in a product.
A failure occurs when a functional unit of a software-related system can no longer perform its
required function or cannot perform it with in specified limits.
From the date gathering perspective, time between failures data is much more expensive. It
requires the failure occurrence time for each software failure recorded. To be useful, time
between failures data also requires a high degree of accuracy. This is perhaps the reason the
MTTF metric is not used widely by commercial developers.
2. Defect density metric : To compare the defect rates of a software product various issues are to
be addressed.
To calculate defect rate for the new and changed code, the following must be done:
LOC count: The entire software product as well as the new and changed code must be counted
for each release of the product.
Defect Tracking: Defects must be tracked to the release origin the portion of the code that
contains the defects and at what release the portion was added, changed or enhanced. When
calculating the defect rate of the entire product, all defects are used; when calculating the defect
rate for the new and changed code, only defects of the release origin of the new and changed
code are included.
3. Customers Perspective: The defect rate metric measures code quality on a per unit basis.
Good practice is Software Quality Engineering, however, also needs to consider customers
perspective. From the customers point of view, the defect rate is not as relevant as the total no.
of defects he/she is going to encounter. Therefore, it becomes necessary to reduce the number of
defects from release to release irrespective of the change in size.
4. Customer Problems Metric : Another product metric that is used by major developers in
the software industry measures the problems customers encounter when using the product.
Per User Month (PUM) = Total problems that customers reported for time period/Total no. of
license- months of the software during the period.
Where
Number of license months = Number of install licenses of the software * No. of months in
the calculation period.
To achieve a low PUM, approaches that can be taken are:
Improve the development process and reduce the product defects
Reduce the non-defect-oriented problems by improving all aspects of the products(such as
usability, documentation), customer education, and support
Increase the sale (the number of installed licenses) of the product.

1. Customer satisfaction index

(Quality ultimately is measured in terms of customer satisfaction.)
Surveyed before product delivery and after product delivery
(and on-going on a periodic basis, using standard questionnaires)
Number of system enhancement requests per year
Number of maintenance fix requests per year
User friendliness: call volume to customer service hotline
User friendliness: training time per new user
Number of product recalls or fix releases (software vendors)
Number of production re-runs (in-house information systems groups)
2. Delivered defect quantities
Normalized per function point (or per LOC)
At product delivery (first 3 months or first year of operation)
Ongoing (per year of operation)
By level of severity
By category or cause, e.g.: requirements defect, design defect, code defect,
documentation/on-line help defect, defect introduced by fixes, etc.
3. Responsiveness (turnaround time) to users
Turnaround time for defect fixes, by level of severity
Time for minor vs. major enhancements; actual vs. planned elapsed time
4. Product volatility
Ratio of maintenance fixes (to repair the system & bring it into
compliance with specifications), vs. enhancement requests

(requests by users to enhance or change functionality)

5. Defect ratios
Defects found after product delivery per function point
Defects found after product delivery per LOC
Pre-delivery defects: annual post-delivery defects
Defects per function point of the system modifications
6. Defect removal efficiency
Number of post-release defects (found by clients in field operation),
categorized by level of severity
Ratio of defects found internally prior to release (via inspections and testing),
as a percentage of all defects
All defects include defects found internally plus externally (by
customers) in the first year after product delivery
7. Complexity of delivered product
McCabe's cyclomatic complexity counts across the system
Halsteads measure
Card's design complexity measures
Predicted defects and maintenance costs, based on complexity measures
8. Test coverage
Breadth of functional coverage
Percentage of paths, branches or conditions that were actually tested
Percentage by criticality level: perceived level of risk of paths
The ratio of the number of detected faults to the number of predicted faults.
9. Cost of defects
Business losses per defect that occurs during operation
Business interruption costs; costs of work-arounds
Lost sales and lost goodwill
Litigation costs resulting from defects
Annual maintenance cost (per function point)
Annual operating cost (per function point)
Measurable damage to your boss's career
10. Costs of quality activities
Costs of reviews, inspections and preventive measures
Costs of test planning and preparation
Costs of test execution, defect tracking, version and change control
Costs of diagnostics, debugging and fixing
Costs of tools and tool support
Costs of test case library maintenance
Costs of testing & QA education associated with the product
Costs of monitoring and oversight by the QA organization
(if separate from the development and test organizations)
11. Re-work
Re-work effort (hours, as a percentage of the original coding hours)
Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
Re-worked software components (as a percentage of the total delivered components)

12. Reliability
Availability (percentage of time a system is available, versus the time
the system is needed to be available)
Mean time between failure (MTBF)
Mean time to repair (MTTR)
Reliability ratio (MTBF / MTTR)
Number of product recalls or fix releases
Number of production re-runs as a ratio of production runs

Collecting Software Engineering Data

The challenge of collecting software engineering data is to make sure that the collected
data can provide useful information for project, process, and quality management and, at the
same time, that the data collection process will not be a burden on development teams.
Therefore, it is important to consider carefully what data to collect. The data must be based on
well-defined metrics and models, which are used to drive improvements. Therefore, the goals of
the data collection should be established and the questions of interest should be defined before
any data is collected. Data classification schemes to be used and the level of precision must be
carefully specified. The collection form or template and data fields should be pretested. The
amount of data to be collected and the number of metrics to be used need not be overwhelming.
It is more important that the information extracted from the data be focused, accurate, and useful
than that it be plentiful. Without being metrics driven, over-collection of data could be wasteful.
Over collection of data is quite common when people start to measure software without an a
priori specification of purpose, objectives, profound versus trivial issues, and metrics and
models.
Gathering software engineering data can be expensive, especially if it is done as part of a
research program, For example, the NASA Software Engineering Laboratory spent about 15% of
their development costs on gathering and processing data on hundreds of metrics for a number of
projects (Shooman, 1983). For large commercial development organizations, the relative cost of
data gathering and processing should be much lower because of economy of scale and fewer
metrics. However, the cost of data collection will never be insignificant. Nonetheless, data
collection and analysis, which yields intelligence about the project and the development process,
is vital for business success. Indeed, in many organizations, a tracking and data collection system
is often an integral part of the software configuration or the project management system, without
which the chance of success of large and complex projects will be reduced.
Basili and Weiss (1984) propose a data collection methodology that could be applicable
anywhere. The schema consists of six steps with considerable feedback and iteration occurring at
several places:
1.

Establish the goal of the data collection.

2.

Develop a list of questions of interest.

3.

Establish data categories.

4.

Design and test data collection forms.

5.

Collect and validate data.

6.

Analyze data.
The importance of the validation element of a data collection system or a development tracking
system cannot be overemphasized.
In their study of NASA's Software Engineering Laboratory projects, Basili and Weiss
(1984) found that software data are error-prone and that special validation provisions are
generally needed. Validation should be performed concurrently with software development and
data collection, based on interviews with those people supplying the data. In cases where data
collection is part of the configuration control process and automated tools are available, data
validation routines (e.g., consistency check, range limits, conditional entries, etc.) should be an
integral part of the tools. Furthermore, training, clear guidelines and instructions, and an
understanding of how the data are used by people who enter or collect the data enhance data
accuracy significantly.
The actual collection process can take several basic formats such as reporting forms,
interviews, and automatic collection using the computer system. For data collection to be
efficient and effective, it should be merged with the configuration management or change control
system. This is the case in most large development organizations. For example, at IBM
Rochester the change control system covers the entire development process, and online tools are
used for plan change control, development items and changes, integration, and change control
after integration (defect fixes). The tools capture data pertinent to schedule, resource, and project
status, as well as quality indicators. In general, change control is more prevalent after the code is
integrated. This is one of the reasons that in many organizations defect data are usually available
for the testing phases but not for the design and coding phases.
With regard to defect data, testing defects are generally more reliable than inspection
defects. During testing, a "bug" exists when a test case cannot execute or when the test results
deviate from the expected outcome. During inspections, the determination of a defect is based on
the judgment of the inspectors. Therefore, it is important to have a clear definition of an
inspection defect. The following is an example of such a definition:
Inspection defect: A problem found during the inspection process which, if not fixed, would
cause one or more of the following to occur:

A defect condition in a later inspection phase

A defect condition during testing

A field defect

Nonconformance to requirements and specifications

Nonconformance to established standards such as performance, national language

translation, and usability

For example, misspelled words are not counted as defects, but would be if they were found on a
screen that customers use. Using nested IF-THEN-ELSE structures instead of a SELECT
statement would not be counted as a defect unless some standard or performance reason dictated
otherwise.
Figure 4.7 is an example of an inspection summary form. The form records the total number of
inspection defects and the LOC estimate for each part (module), as well as defect data classified
by defect origin and defect type. The following guideline pertains to the defect type classification
by development phase:
Interface defect: An interface defect is a defect in the way two separate pieces of logic
communicate. These are errors in communication between:
Components
Products
Modules and subroutines of a component
User interface (e.g., messages, panels)
Examples of interface
development phase follow.

defects

per

FIGURE 4.7 An Inspection Summary

Form

High-Level Design (I0)

Use of wrong parameter
Inconsistent use of function keys on user
interface (e.g., screen)
Incorrect message used Presentation of information on screen not usable

Low-Level Design (I1)

Missing required parameters (e.g., missing parameter on module)
Wrong parameters (e.g., specified incorrect parameter on module)

Intermodule interfaces: input not there, input in wrong order

Intramodule interfaces: passing values/data to subroutines
Incorrect use of common data structures
Misusing data passed to code

Code (I2)
Passing wrong values for parameters on macros, application program interfaces (APIs), modules
Setting up a common control block/area used by another piece of code incorrectly Not issuing
correct exception to caller of code

Logic defect: A logic defect is one that would cause incorrect results in the function to be
performed by the logic. High-level categories of this type of defect are as follows:
Function: capability not implemented or implemented incorrectly
Assignment: initialization
Checking: validate data/values before use
Timing: management of shared/real-time resources
Data Structures: static and dynamic definition of data
Examples of logic defects per development phase follow.

High-Level Design (I0)

Invalid or incorrect screen flow
High-level flow through component missing or incorrect in the review package
Function missing from macros you are implementing
Using a wrong macro to do a function that will not work (e.g., using XXXMSG to receive a
message from a program message queue, instead of YYYMSG).
Missing requirements

Missing parameter/field on command/in database structure/on screen you are implementing

Wrong value on keyword (e.g., macro, command)
Wrong keyword (e.g., macro, command)

Low-Level Design (I1)

Logic does not implement I0 design
Missing or excessive function
Values in common structure not set
Propagation of authority and adoption of authority (lack of or too much)
Lack of code page conversion
Incorrect initialization
Not handling abnormal termination (conditions, cleanup, exit routines)
Lack of normal termination cleanup
Performance: too much processing in loop that could be moved outside of loop

Code (I2)
Code does not implement I1 design
Lack of initialization
Variables initialized incorrectly
Missing exception monitors
Exception monitors in wrong order
Exception monitors not active
Exception monitors active at the wrong time
Exception monitors set up wrong
Truncating of double-byte character set data incorrectly (e.g., truncating before shift in character)

Incorrect code page conversion

Lack of code page conversion
Not handling exceptions/return codes correctly

Documentation defect: A documentation defect is a defect in the description of the function (e.g.,
prologue of macro) that causes someone to do something wrong based on this information. For
example, if a macro prologue contained an incorrect description of a parameter that caused the
user of this macro to use the parameter incorrectly, this would be a documentation defect against
the macro.

Examples of documentation defects per development phase follow.

High-Level Design (I0)

Incorrect information in prologue (e.g., macro)
Misspelling on user interface (e.g., screen)
Wrong wording (e.g., messages, command prompt text)
Using restricted words on user interface
Wording in messages, definition of command parameters is technically incorrect

Low-Level Design (I1)

Wrong information in prologue (e.g., macros, program, etc.)
Missing definition of inputs and outputs of module, subroutines, etc.
Insufficient documentation of logic (comments tell what but not why)

Code (I2)
Information in prologue not correct or missing
Wrong wording in messages

Second-level text of message technically incorrect

Insufficient documentation of logic (comments tell what but not why)
Incorrect documentation of logic

UNIT IV

SOFTWARE QUALITY ASSURANCE

Test Maturity Model

Test Maturity Model is based on the Capability Maturity Model (CMM), and it was first
developed by the Illinois Institute of Technology. It is a detailed model for test process

improvement. It can be complemented with any process improvement model or can be used as a
STAND ALONE model.
TMM has major two components
1.
2.

A set of 5 levels that define testing capability

An Assessment Model

Different Levels of Maturity Model

The five levels of the TMM help the organization to determine the maturity of its process and to
identify the next improvement steps that are essential to achieving a higher level of test maturity.

TMM Levels

Goals

Level 1:

Software should run

successfully

Initial

Objective of TMM levels

Level 2:
Defined

Develop testing and

debugging goals and
policies

At this level, no process areas are

identified
Objective of testing is to ensure that
software is working fine
This level lacks resources, tools, and
trained staff
No quality assurance checks before
software delivery
This level distinguish testing from
debugging & they are considered distinct
activities
Testing phase comes after coding
Primary goal of testing is to show
software meets specification
Basic testing methods and techniques

are in place

Level 3:

Integrate testing into the

software life cycle

Integrated

Level 4:

Establish a test
measurement program

Management and
Measurement

Level 5:

Test process optimization

Optimized

Testing gets integrated into entire life

cycle
Based on requirements test objectives
are defined
Test organization exists
Testing recognized as a professional
activity
Testing is a measured and quantified
process
Review at all development phases are
recognized as tests
For reuse and regression testing, test
cases are gathered and recorded in a test
database
Defects are logged and given severity
levels
Testing is managed and defined
Testing effectiveness and costs can be
monitored
Testing can be fine-tuned and
continuously improved
Quality control and defect prevention
are practiced
Process reuse is practiced
Test related metrics also have tool
support
Tools provide support for test case
design and defect collection

Difference between CMM & TMM

CMM

TMM

CMM or Capability Maturity Model

is for judging the maturity of the software
processes of an organization

TMM or Test Maturity Model describes the

process of testing and is related to monitoring
the quality of software testing model