Sir Syed University of Engineering & Technology, Karachi

CS-470
Software Quality Assurance & Testing
Week 15
Session 2

Batch - 2017 Department of Computer Science 1

2
… Testing Maturity Model (TMM) …

3
 …Testing Maturity Model (TMM) …
• The TMM gives guidance concerning how to improve a test process.
• Each stage is characterized by the concepts of maturity goals, supporting maturity
goals, and activities, tasks, and responsibilities (ATRs), as explained in the
following:
• Maturity Goals:
– Each maturity level, except level 1, contains certain maturity goals.
– For an organization to achieve a certain level of maturity, the corresponding maturity
goals must be met by the organization.
– The maturity goals are specified in terms of testing improvement goals.
• Maturity Sub-goals:
– Maturity goals are supported by maturity sub-goals.
– To achieve a maturity goal, it may be necessary to meet several, fine-grained Sub-goals.
• Activities, Tasks, and Responsibilities:
– Maturity sub-goals are achieved by means of ATRs that address issues concerning
implementation of activities and tasks.
– ATRs also address how an organization can adapt its practices so that it can move in-line
with the TMM model, that is, move from one level to the next. ATRs are further refined
into “views,” known as critical views, from the perspectives of three different groups of
people:
• managers, developers and test engineers, and customers (users/clients).

4
 Testing Maturity Model (TMM) – Level 1
The maturity goals associated with the five levels of the TMM model will be explained
in the following.
• Level 1. Initial:
– No maturity goals are specified at this level.
– The TMM level 1 is called the initial level.
– For an organization to be at level 1, nothing special needs to be done.
– Level 1 represents a scenario where testing is not performed in a planned
manner.
– Testing begins after code is written.
– At this level, an organization often performs testing to demonstrate that the
system works.
– No serious effort is made to track the progress of testing and the quality level
of a product.
– Test cases are designed and executed in an ad hoc manner.
– Testing resources, such as trained testers, testing tools, and test environments,
are not available.
– In summary, testing is not viewed as a critical, distinct phase in software
development.

5
 Testing Maturity Model (TMM) – Level 2 …
The maturity goals associated with the five levels of the TMM model will be explained
in the following.
• Level 2. phase definition:
– At level 2, the maturity goals are as follows:
• Develop testing and debugging goals.
• Initiate a test planning process.
• Institutionalize basic testing techniques and methods.

6
… Testing Maturity Model (TMM) – Level 2 …
• Develop testing and debugging goals – Separation of testing from
debugging is an important growth in the maturity of a testing process.
However, their separation becomes more evident when we consider
higher levels of testing.
• Some concrete maturity sub-goals that can support this goal are as
follows:
– Organizations form committees on testing and debugging, and those
committees are supported with necessary funding.
– The committees develop and document testing goals.
– One example of a testing goal is to track a defect from its initial state of
discovery to its final state of being fixed and verified. The committees develop
and document debugging goals.
– The documented testing and debugging goals are widely used in the
organization from managers to developers and test engineers. All the people
in the organization are aware of the testing and debugging goals and strive to
achieve those goals.

7
… Testing Maturity Model (TMM) – Level 2 …
• Initiate a test planning process - Planning is an indication of a certain level
of maturity of any process. The exact extent of maturity depends upon the
scope and execution of a plan.
• Test planning addresses the following:
– Identify Test Objectives: Identifying the objective of a test process is central
to its success.
– Analyse Risks: Factors that may adversely affect testing must be identified.
– Devise Strategies: Different kinds of strategies are involved in testing.
– Develop Test Specifications: Individual test cases must be judiciously
designed and documented.
– Allocate Resources: Resource allocation is an important task in any planned
activity.
• The following concrete maturity sub-goals can support the above goal at
level 2:
– The organization assigns the task of test planning to a defined committee. The
committee develops a framework for test planning.
– The committee develops a test plan template, and the template is well
documented and made widely available.
– Proper tools are used to create and manage test plans.
– Provisions are put in place so that customer needs constitute a part of the test
plan. In other words, customers are involved in the testing process.
8
… Testing Maturity Model (TMM) – Level 2
• Institutionalize basic testing techniques and methods - A number of basic
testing techniques and methods are widely known in the industry.
• The above maturity goal can be supported by the following sub-goals:
– An expert group is formed to study, evaluate, and recommend a set
of basic testing techniques and methods. The group also
recommends a few tools to support those testing techniques and
methods.
– Management must establish policies to ensure that the
recommended techniques and methods are practiced and tools are
used throughout the organization in a consistent manner.

9
 Testing Maturity Model (TMM) – Level 3…
The maturity goals associated with the five levels of the TMM model will be explained
in the following.
• Level 3. integration:
– At level 3, as the name suggests, testing is fully integrated with the
development process right from the project planning.
– The maturity goals at level 3 are as follows:
• Establish a software test group.
• Establish a technical training program.
• Integrate testing into the software life cycle.
• Control and monitor the testing process.

10
… Testing Maturity Model (TMM) – Level 3…
• Establish a software test group – Software testing is a highly specialized
task at par with capturing requirements and designing a system.
Therefore, an independent test group must exist to carry out testing
without any influence from the developers.
– Maturity sub-goals which support the above goals are as follows:
» An organization wide test group is formed with leadership,
strong support, and funding from the management.
» The test group must be vested with authority to influence the
quality of software products so that formation of the test group
is not rendered as a mere exercise.
» The test group must be involved in all stages of the software
development, and roles and responsibilities must be assigned to
individual group members in appropriate phases of software
development.
» Trained and motivated test engineers are assigned to the test
group.
» The test group in an organization must communicate with the
customers to get a feel for their needs so that testing can be
carried out to meet the expectations of the customers.
11
… Testing Maturity Model (TMM) – Level 3…
• Establish a technical training program - A technical training program is
essential to maintaining a group of qualified and skilled staff for testing.
– Organizations tend to hire not so skilled staff for testing jobs because
of general unavailability of sufficiently qualified test personnel in the
industry.
– Therefore, organizations must make a sustained effort to train test
personnel in the concept of quality, test planning, testing methods
and techniques, standards, and testing tools.
– Test engineers learn the concepts and processes of various kinds of
reviews, such as design review and code review,

12
… Testing Maturity Model (TMM) – Level 3…
• Integrate testing into the software life cycle - In contrast to level 2, where
testing is initiated as a distinct activity after coding is over, at level 3
testing is fully integrated with a software development life cycle at the
start of project planning. Therefore, test planning is initiated very early in
the lifetime of a product.
– Maturity sub-goals to support the above goal are as follows:
» The test phase is partitioned into several activities, such as test planning,
unit testing, integration testing, system testing, and acceptance testing,
so that the distinct activities are easily performed at the appropriate
times in the life cycle of a software product. Different kinds of reviews,
such as requirements review, design review, and code review, are
identified. All those testing and review activities are integrated into the
V-model of testing.
» All the testing and review activities are integrated into an
institutionalized V-model of testing, and the organization must ensure
that the model is followed.
» The organization supports an effective mechanism for the test group to
communicate with developers, customers, and the marketing group. As
a result, test engineers learn about the requirements, design details,
customer expectations, and marketing philosophy of the organization.
The more the test group is knowledgeable about the product, the more
it can influence product quality. The upper management of an
organization must facilitate such communication. 13
… Testing Maturity Model (TMM) – Level 3…
• Control and monitor the testing process - Monitoring and control are
important aspects of planning. Without monitoring it will not be known if
a project is on course, and, consequently, no corrective measures can be
taken to control the project.
– Progress of a test project can be measured as follows:
» Amount of Testing effort: This includes the number of test cases to be
designed and the number of test cases to be executed.
» Cost of Testing: This represents all the cost in carrying out test-related
activities.
» Schedule: This concerns the start and finish times of the test activities.
– The following maturity sub-goals are required to support the above goal:
» The organization develops policies and mechanisms to monitor and
control test projects.
» A set of metrics related to the employed test process must be defined,
recorded, and made available to all concerned test engineers.
» A set of potential corrective actions and contingency plans must be
defined and documented. It may be used when the actual progress of a
testing project significantly differs from the plan.

14
 Testing Maturity Model (TMM) – Level 4…
The maturity goals associated with the five levels of the TMM model will be explained
in the following.
• Level 4. management and measurement - At level 4 , testing acquires a much
larger scope and is not just another phase in a software development life cycle.
– The following are the maturity goals at level 4:
• Establish an organization wide review program.
• Establish a test measurement program.
• Evaluate software quality.

15
… Testing Maturity Model (TMM) – Level 4…
• Establish an organization wide review program - To augment the
execution based testing, a review program is established to detect defects
early in the life cycle of product development and at a lower cost.
– Maturity sub-goals that support this goal are as follows:
» Management should develop review policies and ensure that
those policies are rigorously followed.
» The test group should develop goals, plans, procedures, and
recording mechanisms for carrying out reviews.
» Objectives must be clearly defined.
» Members of the test group must be trained so that their
effective participation in review processes is ensured.

16
… Testing Maturity Model (TMM) – Level 4…
• Establish a test measurement program - A test measurement program is
established to measure productivity, progress, and quality of work
associated with testing.
– Maturity sub-goals that support this goal are as follows:
» Test metrics should be identified along with their goals.
» A test measurement plan should be developed with data
collection and analysis.
» An action plan should be developed to achieve process
improvement by considering the measured data.

17
… Testing Maturity Model (TMM) – Level 4…
• Evaluate software quality - An important goal of a software development
process is to produce software products of the highest possible quality.
– The maturity sub-goals that support this goal are as follows:
» The organization should define quality attributes and quality
goals for software products.
» Management should develop policies and mechanisms to collect
test metrics to support the quality goals.

18
 Testing Maturity Model (TMM) – Level 5
• Level 5. optimization, defect prevention, and quality control :
– optimization means spending less resources to achieve higher quality
products,
– and defect prevention means taking measures throughout the development
process so that products are largely defect free.
– At level 5, the maturity goals are as follows:
• Application of process data for defect prevention
• Statistical quality control
• Test process optimization

19
…Testing Maturity Model (TMM) – Level 5 …
• Application of process data for defect prevention - organizations make a
sustained effort to learn from experience to reduce the number of defects
in a system.
– Maturity sub-goals that support this goal are as follows:
» Management should establish a defect prevention team.
» Defects identified and removed are documented in each phase of the
development.
» Each defect is analysed to get to its root cause.
» Managers, developers, and the test group should interact to develop an
» action plan to eliminate recurrence of commonly occurring defects.
» Management should put in place a mechanism to implement and track
the action plan.

20
…Testing Maturity Model (TMM) – Level 5…
• Statistical quality control - organizations further improve the quality of
software products by driving the testing process with statistical sampling,
measurement of confidence levels, trustworthiness, and software
reliability goals.
– Sub-goals that support statistical quality control are as follows:
» The test group establishes high-level measurable quality goals, such as
test case execution rate, defects arrival rate, and total number of defects
that can be found during testing.
» Managers ensure that the new quality goals form a part of the test plan.
» The test group is trained in statistical testing and analysis methods:
• Pareto analysis, cause-and-effect diagram, flow chart, trend chart,
histogram, scatter diagram, and control chart.
» User inputs are gathered for usage modelling.

21
…Testing Maturity Model (TMM) – Level 5
• Test process optimization - refers to all those activities which result in
continuous improvement of the testing process.
– Test optimization involves the following activities:
» Identify the testing practices that can be improved.
» Define a mechanism to improve an identified practice.
» Put a mechanism in place to track the practice improvement
mechanism.
» Continuously evaluate new test-related technologies and tools.
» Develop management support for technology transfer.
– Maturity sub-goals that support test process optimization are as follows:
» Establish a test process improvement group to monitor the testing
process and identify areas for improvement.
» Evaluate new technologies and tools to improve the capability of the
testing process.
» There must be management support to establish policies and
mechanisms for this purpose.
» A mechanism is put in place for continual evaluation of the effectiveness
of the testing process.
» Test stopping criteria are based on quality goals

22