PMI Virtual Library

© 2011 Krishna V.R. Muppavarapu

Innovative Quality Measurement

System—Ideas for a Project Manager
By Krishna V. R. Muppavarapu, eGramIT Services Pvt. Ltd.

Only a well-researched, scientific approach can help design an optimal and objective
system of measurement and one that will be acceptable to all the critical stakeholders.
”
Q
uality management is an integral and key component designing it, he or she often faces some cost-quality tradeoffs.
of project management. Often, quality measurement Only a well-researched, scientific approach can help design
systems manifest themselves in the success or failure such an optimal and objective system of measurement and
of projects and, subsequently, of operations. A well-designed one that will be acceptable to all the critical stakeholders.
quality measurement system is characterized by an ability to This paper presents the efforts of the project management
detect critical defects throughout the project life cycle and team involved in the design of a “universal quality system”
beyond. for business process outsourcing projects. This unique quality
Traditional quality measurement systems have certain system applies certain statistically fundamental principles
inherent deficiencies, some of which are: to provide an objective view of the project status across
• Static sample-size calculations stakeholders and has the following unique characteristics:
• General views of all the agents (people and machines) • N-phase audit mechanism
involved in production • A complex algorithm to ensure a progressive adjustment
• Non-discrimination between errors of varying criticalities of the number of transactions to be audited based on
history (dynamic sample sizes)
These deficiencies lead to problems like increased cost • Empirically proved for a power of test (beta) of more than
of quality measurement, reduced customer satisfaction 80% and a low type-1 error (alpha)
due to poor quality output, improper defect analysis, poor • Flexible enough to allow for dynamically changing
training management, and so forth. The project management quality and productivity baselines
community needs a more dynamic and advanced quality • Incorporates Six Sigma best practices
measurement system. • Standardization through the unique concept of “defect
The project manager is in charge of most of the project equivalents” to ensure a normalized view of employee
or operational base-lining effort, so designing a good quality performance within a project for a zero-bias performance
measurement system is his or her responsibility; however, in appraisal system
Introduction leaders in various forums. Businesses have realized that
Project management as a philosophy puts much emphasis aligning to the business requirements of the client is the
on the importance of quality. A Guide to the Project only way to survive and experience growth. This model is
Management Body of Knowledge (PMBOK ® Guide), which applicable to the entire gamut of economy—agriculture,
is the benchmark standard for the project management manufacturing, services, information technology, the
discipline, has articulated the importance of balancing various government, and so forth. It is the “voice of the customer”
project constraints. One of the key constraints is “quality.” that matters the most, and effective quality management
This is a clear evolution from the classic “triple constraints” constitutes a properly interpreted and codified voice of the
of scope, time, and cost articulated in earlier versions of the customer.
PMBOK® Guide. As much as this evolution speaks of the The project management discipline has been cognizant
versatility of the PMBOK® Guide as being one of the most of this constantly evolving trend, as discussed in the previous
up-to-date standards, it also highlights the importance of section, and has reiterated its commitment to quality by
quality as a key project constraint. Moreover, the fact that the having “customer satisfaction” as one of its key focus areas.
PMBOK ® Guide lists Project Quality Management as one In fact, the PMBOK®® Guide has maintained that modern
of the Knowledge Areas itself is testimony to the fact that quality management complements project management.2
quality management is quite integral to project management. Projects are important aspects of organizations and,
Figure 1 depicts the growing importance of quality in project in general, organizations with strong project management
management. capabilities are the ones that succeed in maintaining a
competitive advantage; this applies to organizations,
The Emerging Business Model irrespective of their strategic orientation toward being
The emerging business model in the reset economy1 is that sales driven or operations driven. Hence, sound project
of “extreme customer-centricity,” as evidenced by business management practices are extremely critical for organizations
in order to thrive in the reset economy. As a corollary,
1
This is a generally accepted term for the economy in the aftermath of the organizations need to focus heavily on good project
economic recession that started in 2008. The reset economy is characterized
by a severe scarcity of capital, hyper-competitiveness, aggressive expectations management and quality to ensure that they stay in business
from customers about the quality of deliverable, mounting cost pressures, and grow, and even more so in the new economy. Investments
tremendous importance for business/technological innovation, etc; the
general agreement being that the economy can never reach the same levels 2
PMBOK® Guide – Fourth Edition Chapter 8, Project Quality
of growth that existed before the beginning of recessionary trends. Management

Scope

Scope
Risk Quality

Evolution of the
Project
PMBOK® Guide
constraints
Classic
triple
constraints Resources Budget

Cost Time
Schedule

Quality is not part of the project constraints Quality is an important part of the project constraints
(PMBOK® Guide, Third Edition) (PMBOK® Guide, Fourth Edition)

Figure 1: Evolution of the PMBOK® Guide to include “quality” as a key project constraint.

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

2
in project management capabilities and delivering good The Traditional Quality Measurement System
quality to the customer are sure to yield extremely high A lot of research has gone into designing quality measurement
returns in terms of an increased top-line, improved systems and many theories have been postulated. One of
bottom-line, value to investors, and employee satisfaction. the key objectives of any measurement system is to reduce
Organizations oblivious to this important aspect will either the statistically important type 1 and type 2 errors. Table 1
face extinction or get amalgamated into larger entities that are depicts these possible errors.
characterized by sound project management and innovative
quality management practices.
Accept Null Reject Null
The Project Manager’s Role in Quality Management Hypothesis (H0) Hypothesis (H0)
Every business transaction or process starts off as part of a
Null Desirable outcome, Type 1 error, the
project and can be mapped to some project, even at the most Hypothesis probability of which probability of which
operational level. (H0) is true is called confidence is denoted by α
Good management of business transactions or processes is level, 1-α
determined to a significant extent by the way its measurement Null Type 2 error, the Desirable state, the
system is designed. Most often, the design of the transaction/ Hypothesis probability of which is probability of which
(H0) is false denoted by β is equal to 1-β, often
process and of the measurement system is done as part of called “power of
some project even before the transaction/process enters the test”
operations mode.
This paper leaves the scope of “transaction/process Table 1: Depictions of type 1 and type 2 errors.
design” for a later deliberation and deals exclusively with the
design of a quality measurement system as part of the Project Depending on the application, one type of error can be
Quality Management knowledge area. The author takes a very more significant than the other. With the null hypothesis
firm stand that it is the responsibility of the project manager being that the accuracy of the output is greater than the
to initiate, plan, execute, and institutionalize an efficient desired, a high type 1 error in a typical business set up
quality measurement system as part of the project quality would mean that there is unnecessary rework due to holding
management, plan quality process.3 back valid outputs, which generally means a higher cost
It is often observed that systems designed in the early to the organization. This often means a stringent quality
stages of a project tend to have a lasting impact on operations. inspection, by which the customer would get a high-quality
One of the main reasons for this is that the stakeholders’ output at the cost of the supplier organization. In other
enthusiasm for adopting and learning the systems is at its terms, a high type 2 error means that there is a higher
peak during the outset. Any subsequent changes to the system probability of defective output not being detected by the
are costly both from the re-engineering and the change measurement system and ending up being supplied to the
management points of view. customer. This, in turn, would lead to many warranty claims
This makes the role of the project manager extremely and a tarnished image for the supplying organization in the
crucial, because a lot depends on the project manager’s marketplace.
innovativeness in building a good quality measurement Project managers generally build quality measurement
system early on in the project, so that the subsequent systems that are designed for a certain probability of type 1
handover to operations teams is healthy, seamless, and error (α). Type 1 error reflects the organization’s tolerance
fulfilling. for deviations in quality. A very low acceptable probability
In this paper, the author explains and builds on his of type 1 error means a less stringent quality measurement
experience as a project manager in developing an innovative system but this would mean that the probability of accepting
quality measurement system for business process outsourcing defective outputs (β) would go up. In other words, the power
(BPO) engagements. Nonetheless, the concept is universal of the quality measurement system (1- β) goes down. Figure 2
and can be customized to many industrial scenarios that illustrates the relationship between the error types, although not
involve high-volume, business-rule driven transactions. exactly linear in real life.
3
PMBOK® Guide – Fourth Edition, Chapter 8, Section 8.1.3.2 Quality
Metrics

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

3
 of the customer receiving defective outputs is minimized.
Most often, the operations/delivery teams stick to the same
sample size that was proposed in the project stage and once
High
the project team disengages, it becomes even more difficult
Power of test
Probability of type 2 error and

to really understand the logic behind such a sample size and

make changes to it, depending on the dynamically changing
power of test

business requirements.
Type 2 error One more crucial aspect that traditional quality
Type 1 error systems overlook is the fact that the behavior of people
or the machines involved in the production of output is
not uniform. Different sets of agents (people or machines)
Low Probability of type 1 error High
require varying degrees of monitoring and measurement.
The traditional quality measurement systems often prescribe
Figure 2: Relationship between type 1 and type 2 errors. a “blanket” sample size in the form of “percentage output to
be audited” or number of samples to be audited for different
The entire consideration of type 1 and type 2 errors agents. This often leads to a lack of the much required
arises out of a practical limitation on the number of outputs discrimination between various agents as far as sample sizes
that can be “economically measured” for accuracy. The are concerned. For example, auditing a high-performing
only way to reduce the probabilities of type 1 and type 2 individual at the same rate as a low-performing individual
errors is to increase the number of the samples that will be can lead to a waste of time and effort. Conversely, a low-
inspected. Theoretically, a calibrated measurement system performing individual may require a higher degree of auditing
applied consistently over 100% of output would have zero compared with a high-performing one to ensure that the
probabilities of type 1 and type 2 errors. This, however, customer does not get defective output.
is definitely not economically feasible from a business Apart from the deficiencies in the sample size
perspective. For a given sample size, type 1 and type 2 errors considerations, a major problem with the traditional
will always be correlated as shown in Figure 2. quality measurement systems is that all defects observed
This brings us to the most important question: What are generally considered at par with each other. This often
should be the sample size to achieve a maximum power of leads to skewed analysis, which leads to penalizing less
test for a given tolerance for type 1 error? Much data have critical defects at the same rate as the more critical ones or,
been tabulated to assist quality managers in fixing a sample conversely, not penalizing an identified critical defect as much
size based on type 1 error, but these tables have a limitation as it should be. The defect rate measured could be made
in that they assume normal distribution of the attributes, up of all business non-critical errors, all extremely critical
whereas in practical terms this is often not the case. The errors, or a combination of both. For example, if one page
general formulas for calculating sample sizes also have of an automotive manual has ninety-eight words and two
some limitations, such as the population variance required illustrations, there are effectively one hundred opportunities
for calculations not generally available or very difficult to for error4 on that page. A defect in the illustration could be
estimate. extremely critical, because the driver stranded on a highway
So, it is general practice to estimate the sample size based due to a breakdown may actually get into a serious problem
on experience. Even if the project manager or the quality because the part he should be repairing is not well explained
specialist in the project team uses statistical tables or formulas in the owner’s manual. At the same time, an incorrectly
to calculate the sample size required, most often the relevance capitalized letter or a missing punctuation mark is not
of the logic behind arriving at the sample size gets lost in as critical as an incorrect illustration. Traditional quality
the long run. In fact, the main shortcoming of the sample measurement systems tend to measure these defects “at par”
size calculations is that they do not consider dynamically and this is obviously not the most appropriate way to measure.
changing requirements and scenarios. For example, when the In addition, the “degree of quality requirement” that
customer revises the accuracy requirement from the earlier varies from project to project is often overlooked, which leads
96% to 98%, the sample size that needs to be considered for 4
Opportunities or Opportunities for error (OFE) is a Six Sigma concept
quality inspection is also likely to go up to ensure that the that measures the number of components of an entity or output that “can”
power of the test (1-β) is increased so that the probability go wrong.

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

4
to the same sampling methodology in all projects. Moreover, The PMBOK ® Guide does provide for proprietary quality
traditional systems do not consider the fact that different management methodologies, keeping in mind the complexity
projects and operations have different benchmarks as far as of the desired outputs of the quality management plan. With
quality requirements are concerned. For example, in the same this as the background, this paper proposes a customized and
department, there can be two projects—one with an accuracy proprietary quality measurement system that makes up for
requirement of 95% and the other with a requirement of the deficiencies of the traditional systems. The unique aspect
99%. The traditional quality system would prescribe the same is that this system considers agent-wise work allocation as the
sampling rate for both projects rather than basing it on the basis for audit sample size; hence, productivity considerations
absolute accuracy requirement. are inherent in this system. One of the ultimate outputs of
this system is a composite employee performance index made
Dynamic Quality Measurement System up of productivity and quality scores for each individual.
A major area of involvement for a project team and the The proposed quality measurement system inherently
project manager is the Plan Quality process of Project Quality incorporates the “central limit theorem”7 by progressively and
Management, as mentioned in the PMBOK ® Guide. Statistical dynamically adjusting the sample size based on the output of
sampling5 is a very important tool in the Plan Quality process. the previous sampling, thereby obviating the need for the data
As mentioned in previous sections, many project managers to be normally distributed for any kind of statistical analysis
find it difficult to come up with a reliable sampling frequency and making it fit into the Six Sigma paradigm. This is likely to
because there is always bound to be a tradeoff between the cost increase the overall sample size, with an increase in the quality
of sampling and the effectiveness6 of the sampling plan. effort, but the benefits in terms of better customer satisfaction
Also, quality metrics and quality checklists are critical far outweigh the costs associated with the increased quality
inputs to Perform Quality Assurance and Perform Quality effort and it also reduces the rates of type 1 and type 2 errors
Control processes of the Project Quality Management process as seen empirically. One of the main advantages of this
group. Identifying the right Quality metrics and checklists system is that for a particular “class”8 of projects, whatever
is very important; hence, project managers should spend the starting audit percentage, ultimately the overall audit
considerable time in identifying and formulating a good
7
Central limit theorem states that, “As sample size increases, the sampling
defect tracking system that “normalizes” various defects
distribution of sample means approaches that of a normal distribution with
against a standard criticality. Table 2 is a summary of the a mean the same as the population and a standard deviation equal to the
shortcomings of the traditional quality systems as explained in standard deviation of the population divided by the square root of n (the
the above section. sample size)”
8
Although projects are unique, there would still be some similarities be-
5
PMBOK® Guide – Fourth Edition, Chapter 8, Section 8.1.2.6 Statistical tween some of them. By “class” the author intends to convey projects/trans-
Sampling actions that have certain inherent similarities, however different they appear
6
Effectiveness of the sampling plan indicates the degrees to which type 1 to be. For example, quality inspection for customer form processing is quite
and type 2 errors are minimized when a particular sampling plan is used similar to the quality inspection of a manufactured automobile tire.

Reason Effect Results in…

Static sample size Decreased power of test when the customer’s Poor customer satisfaction and
calculations quality requirements go up or increased cost deterioration of the organization’s
of auditing when the quality requirements are image and business
relaxed
Blanket view of all agents A common “average” sample size selection Poor quality output reaching the
(people and machines) of without discriminating between high- customer, increased cost of quality
production performing and low-performing individuals or measurement
agents, leading to either too much or too little
auditing effort
Non-discrimination between An average “count” of errors without a proper Improper training needs, improper
errors normalization based on criticality leading to analysis of defects/reworks, and
skewed analysis poor corrective action

Table 2: Summary of the shortcomings of traditional quality measurement systems.

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

5
percentage over various phases of the audit will converge into was derived through a series of experiments involving three
a band of percentage values that will give statistically accurate phases of audit (for each lot), which seemed to be a practical
results. This saves the project team valuable time and effort upper limit. A one-phase audit is nothing but the traditional
that would otherwise go into determining and designing a quality measurement system in which a lot is either passed or
good quality measurement system as part of the Plan Quality rejected based on the quality of a single sample. Nevertheless,
process. This quality measurement system can be used as there may be practical scenarios that might warrant more than
a “plug-and-play” system that covers all the deficiencies three phases of audit per lot; for example, if the volumes of
mentioned in Table 2. production are very high.
The proposed system is driven by a complex algorithm The required audit percentage is represented
that was empirically derived, using heuristics. The main mathematically as per Equation 1.
feature of this system is that the entire quality audit is divided Equations 1, 2, and 3 are applicable to each employee or
into “phases.” The system can be configured as an N-phase agent separately, which allows for an agent-level analysis of
audit mechanism, depending on the “perceived” requirement the quality. Schematically, the model is represented as in
of the power of test. Figure 3. Figure 3 applies to the output from a single agent
We will explain this concept through a practical example, and each agent would have a similar scheme of measurement.
because much of the insights have come from practical The concept of defect equivalents remains to be
project management experiences; before that, however, a explained. This component of the framework is a very
mathematical representation is attempted to give it a generic important one and needs special consideration because it is a
flavor that can be replicated in a variety of scenarios through marked deviation from the traditional method of measuring
proper customization. For example, the logic for this system all defects “at par.”

aij = f (Q, aminij, qi, j-1), for i = 1 to L and j = 2 to N

= f (Q, amini1, qavgi-1) for i = 1 to L and j = 1
= a random audit percentage > 5% for i = 1, j = 1 (represented as a 11, which is
the minimum first phase audit percentage for the first lot) ………. equation (1)
where
a= system generated audit percentage (except for a11, which is the starting audit percentage for the first phase of
the first lot)
i= lot number ranging from a minimum of 1 to a maximum of L
j= phase number of the audit ranging from a minimum of 1 to a maximum of N
Q= benchmark quality as a percentage of defects in all opportunities for error (customer requirement)
q= detected quality percentage at each phase calculated by the number of defect equivalents and the number of
opportunities for error. It can be represented by equation 2

qij = f (deij, ofeij) for i = 1 to L and j = 1 to N (definitions of notations maintained as in

equation 1) ……………………… equation (2)
where
de = number of defect equivalents observed during the audit of a lot in a phase
ofe = the number of calculated or counted opportunities for error

amin = minimum audit percentage for each phase of audit, which is > 1%
qavg = weighted average of detected quality across all phases for a lot, with the number of samples at each phase as
the weights. This is system generated and can be represented mathematically as in equation 3.

qavgi = f (qij) for a given i and for j = 1 to N (definitions of notations maintained as in

equation 1) ………………. equation (3)

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

6
 a11 = Minimum q11 q12 q1(N-1)
Lot 1 audit percentage
Phase 1 Phase 2 Phase ‘N’
qavg1
(discretionary) quality audit quality audit quality audit

q21 q22 q2(N-1)

Lot 2 Phase 1 Phase 2 Phase ‘N’ qavg2
quality audit quality audit quality audit

qL1 qL2 qL(N-1)

Lot ‘L’ Phase 1 Phase 2 Phase ‘N’
quality audit quality audit quality audit qavgL

Figure 3: Schematic of an N-phase audit mechanism.

Every output or entity has a finite number of The failure modes, in turn, can be classified as fatal, critical,
opportunities for error (OFE). For example, in an automobile and non-critical based on their impacts on the deliverable.
tire there are components like the tread, the sidewall, the The failure modes are given scores, with fatal and critical ones
bead, and the ply, which represent the opportunities for error/ having the highest values. The non-critical failure modes are
defect. Each opportunity for error can have a certain number given scores that are “normalized” against the fatal or critical
of known modes of failure. Each mode of failure has its ones. Although fatal and critical failure modes technically have
criticality and this is where the proposed quality measurement the same scores for a particular opportunity for error, a clear
system borrows from the concept of FMEA (failure mode and distinction is made to facilitate a differential action in case a
effects analysis)9. Figure 4 is a mathematical representation of fatal error is encountered during quality audit.10
these ideas, illustrated through mapping diagrams. Because we are hypothesizing over a finite set of
opportunities for error and a finite set of corresponding failure
9
FMEA is a quality and project management concept that gives an objec- modes, all failure modes (irrespective of their relationship
tive format to the criticality of the failure modes of various components
through an index called the risk priority number (RPN) that is a composite 10
In many industrial scenarios, including manufacturing and call centers,
of the severity, occurrence and detect-ability ratings of a particular failure the detection of fatal errors during quality audits is treated seriously and
mode. Often the ratings are arrived at through techniques like the Delphi, often leads to the rejection of the entire lot of output (in case of product
brainstorming, benchmarking, and so forth. output) or 100% monitoring (in case of services)

OFE space (for a typical Failure mode space

project)

 FM11
 OFE1  FM12
. .
.  FM1m
.
.
.  OFE  opportunity for error
 OFEn  FMn1  FM failure mode
 FMn2  m, n, p  maximum numbers
. corresponding to OFEs and FMs
 FMnp

Figure 4: Mapping of error opportunities and failure modes.

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

7
with the OFEs) are put on a normalized scale, with the fatal of the project team. The project team the author was part
and critical errors having the highest scores, but subject to a of actually created a software application using the available
set of constraints to ensure that, logically, the total score of all software and put it to use across three projects of varying scopes.
failure modes is equal to the total number of opportunities for By using the software application, the project team recorded the
error. The total score of the failure modes observed will be the performance of a cross section of employees (agents).
“total critical defect equivalents,” also referred to in this paper Table 3 gives a sample summary of the findings and the
as “defect equivalents.” The same logic is applicable to both effectiveness of the three-phase quality audit system applied
homogeneous and heterogeneous opportunities for error.11 to one of the projects. The table summarizes the results
One of the limitations of this logic is that it fails to measured for one employee. The cells of the table shaded in
normalize projects. For example, with the proposed system, grey indicate the results using the proposed three-phase audit
an accuracy of 98% for a project with lower complexity and mechanism. The project team then measured the accuracy
an accuracy of 98% for a project with higher complexity levels by inspecting 100% output of each lot. The values
would be viewed as being equivalent. One way to overcome appear in the cells shaded blue. A comparison of the values
this limitation is to expand the mapping mentioned in Figure in grey and blue cells indicates that the three-phase audit
4 to include all projects, their corresponding opportunities mechanism is as good as the inspection of 100% output.
for error, and then normalize all failure modes across projects. Table 4 presents a broader summary of the three-phase
This is not only logically challenging but also practically quality measurement system applied to a data entry project,
infeasible but it does serve as a theoretical possibility, which an invoice reconciliation project, and a voice-based customer
probably can be evolved into practical terms over a period of feedback management project. The “overall sampling %” is
time as businesses gain maturity Until that time, however, a system-generated value that takes into consideration the
this limitation can only be addressed by attaching an overall accuracy levels detected across the three phases of each lot.
complexity index for each project and discriminating by way It is empirically evident from Table 4 that the three-
of policy intervention, like differential salaries, complexity- phase audit mechanism has very low incidences of type 1 and
based incentives to employees, and so forth. type 2 errors, thereby leading to lower rework costs (due to
low type 1 error) and lower risk of passing on bad output to
Practical Example Of Three-Phase Audit Mechanism the customer (due to low type 2 error). In fact, the power
Although the proposed quality measurement system looks of test is as high as 100%, and the type 1 error is as low as
mathematically complex, it can be easily automated using 0%.12 We observe that there is a pattern in that a higher
computer software. The logic mentioned in the above section overall sampling requirement might mean that the process
can be codified using the information technology component is not stable from an accuracy perspective, owing to lack of
standardization, uncontrollable external factors, the need for
11
For example, in an automobile tire, we have heterogeneous opportuni-
ties for error in the form of the tread, the ply, the bead, and so forth, with closer scrutiny of the particular agent, and so forth.
each opportunity being associated with different sets of failure modes;
whereas in a plainly typed paper, all the words form a homogeneous set of
12
The figures for type 1 error and power of test are observed for limited
opportunities for error with exactly similar failure modes. sample sizes.

Project: Invoice Required Starting audit Overall sampling

1
reconciliation accuracy = 99% percentage = 5% percentage = 24%
Lot Average accuracy Pass / Actual quality found Pass / Difference between Type 1 Type 2
number found using a reject the by auditing all the reject the actual accuracy and error? error?
three-phase audit lot records in the lot (b) lot measured accuracy (|a- (Yes / No) (Yes /
mechanism (a) b|) No)
1. 99.83% Pass 99.80% Pass 0.03% No No
2. 99.90% Pass 99.80% Pass 0.10% No No
3. 99.28% Pass Compare
99.80% Pass 0.52% No No

20. 99.90% Pass 99.86% Pass 0.04% No No

Table 3: Sample summary of the results of three-phase quality measurement system.

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

8
 Sample output #1 of Project: Invoice Required accuracy = Total number of
three-phase audit reconciliation 99% outputs = 750
mechanism
Start audit Effective sample size for Overall sampling % Number of type 1 errors Number of type 2
percentage 20 lots observed errors observed
5% 182 24% 0 0
10% 220 29% 0 0
15% 261 35% 0 0

Sample output #2 of Project: Data entry Required accuracy = Total number of

three-phase audit 99% outputs = 750
mechanism
Start audit Effective sample size for Overall sampling % Number of type 1 errors Number of type 2
percentage 20 lots observed errors observed
5% 181 24% 0 0
10% 223 30% 1 0
15% 288 38% 1 0

Sample output #3 of Project: Voice-based Required accuracy = Total number of

three-phase audit feedback management 90% outputs = 450
mechanism
Start audit Effective sample size for Overall sampling % Number of type 1 errors Number of type 2
percentage 20 lots observed errors observed
5% 258 57% 0 0
10% 277 62% 0 0
15% 277 62% 0 0

Table 4: Summary of the results of three-phase quality measurement system applied across various projects.

Because accuracy is measured as a proportion of “critical A word of warning here: As in any measurement system,
defect equivalents,” the N-phase quality measurement system the N-phase quality measurement system also needs to be
allows for comparing the performances of all employees periodically calibrated to ensure its relevance in dynamically
working for a particular project on a common scale. An changing business scenarios. For example, if a certain defect
example of how defect equivalents are measured is shown becomes irrelevant due to some technological intervention,
in Figure 5. The failure modes represented in bold font are that particular defect needs to be removed from the list, and
the critical defects and all other defects are measured as a the weights need to be reconstructed in such a way that the
percentage or a fraction of the critical defects to arrive at the original constraint (of matching the sum of defect scores
total defect equivalents as mentioned earlier. For example, a to the number of failure modes) is not violated. Another
simple error like a formatting error can be treated as being example is that a defect that is considered critical today may
only 10% as serious as a critical defect, much like a wrong actually become non-critical in the future.
currency being mentioned in payment terms. In other In fact, the example mentioned above has further
words, if wrong currency is given a weight of 1, a formatting integration with the employee productivity tracking system.
error would have a weight of 0.1. The project team that The work done by each employee is logged into a productivity
implemented the N phase quality measurement system tracker similar to the traditional “timesheets.” The quality
actually brainstormed and arrived at the weights for various measurement system in turn picks these efforts and throws up
defects against the critical defects. Of course, the software the number to be audited at each phase by multiplying the
application would adjust the final weights in such a way that employee’s effort with the audit percentage calculated as per
the sum of all defect weights would be equal to the total Equation 1. A multiplication of an employee’s efforts (e.g.,
number of opportunities for error. the number of invoices an employee processes or the number

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

9
 OFE for an editing project Failure modes
(Representative components of the
edited document)

 Spelling
 Product  Grammar
description  Syntax
 Format
 Specifications  Relevance
 Correctness of
the product  OFE  opportunity for
 Payment terms
error

 Spelling
 Format
 Unit of
measurement
 Values

 Currency
 Spelling
 Format
 Amount
 Mode
 Payment
options

Figure 5: Practical example of measuring defect equivalents.

of automobile tires the employee builds) in a particular audit with the wisdom embedded in the PMBOK ® Guide and
duration and the measured accuracy would give the “incentive the power of their own experiences are the best people to
index” for that particular employee. This system will ensure drive such innovative research-driven concepts. This paper
that both productivity and accuracy are considered while is an attempt to reiterate the commitment of the project
comparing employee performance. This will ensure that the management discipline to quality. It is hoped that scientific
performance of one parameter (productivity or accuracy) does project management, as envisioned in the PMBOK ® Guide
not come at the cost of the other, because the system tends will lead the way to recovery in the reset economy, and that
to instill an operational discipline among employees owing research in project management will start gaining more
to its objectivity and non-biased treatment of performance prominence in the near future.
parameters.
Acknowledgments
Conclusion The author wishes to acknowledge the following colleagues:
The proposed N-phase quality measurement system can 1. Mr. Srinivas Varma, chief executive officer of eGramIT
be used as a very effective tool to improving customer Services, for his commitment to quality and for setting
satisfaction. It is an intelligent and self-evolving system that the vision for quality. His personal reviews have helped
can give project managers very good insights into the aspects enrich the algorithm; if not for his encouragement and
of project quality management. Project managers, equipped sponsorship, this concept would have remained abstract.

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

10
2. Mr. Pierre D’Silva, head of projects and operations Project Management Institute (PMI) (2008). A guide
at eGramIT Services, whose original idea of having a to the project management body of knowledge (PMBOK ®
performance management system formed the entire Guide)—Fourth Edition. Newtown Square, PA: Author.
basis for the N-phase audit mechanism. His insights,
which cover both the project management and About the Author
operations management disciplines, reflect prominently Krishna V. R. Muppavarapu is an active member of Project
in this paper. Management Institute (PMI)® and a PMP®, is a Certified
3. Mr. Gautam Lunawat, whose intellectual brilliance Six Sigma Black Belt from the American Society for Quality
has paved the way for formulation of the statistics (ASQ),and also holds a mechanical engineering degree from
based algorithm. He was instrumental in creating this BITS, Pilani, India’s premier technology institution and a
framework in two months as a summer intern with master’s degree in management from SP Jain Institute of
eGramIT. He deserves most of the credit for giving a Management and Research (SPJIMR), Mumbai, a leading
tangible shape to what started off as a vague idea and for business school in India. Mr. Muppavarapu has worked
having operationalized it by applying his business analysis for close to 10 years in the manufacturing, information
skills to develop the software. technology, and business process outsourcing industries. At
4. Mr. Vijay Devu, whose extraordinary commitment to the time of this writing, he is working as the head of quality
building the software code made things quite interesting and process management at eGramIT Services Pvt. Ltd., a
and encouraging. His team of developers and business company that has made groundbreaking efforts in delivering
analysts has been a key driver in this initiative. business process outsourcing services. The company website is
www.egramit.com,and Mr. Muppavarapu can be contacted
References at [email protected].
Israel, G. D. (1992). Determining sample size. In Sampling
the evidence of extension program impact. Program Evaluation Footnotes
1 Accuracy is defined and measured as the “number of critical defect
Organizational Development, IFAS, University of Florida, equivalents divided by the total number of opportunities for error”
PEOD-5, October.

Disclaimer: The views and the algorithm expressed in this paper are the sole property of eGramIT Services Pvt. Ltd (www.egramit.com). The algorithm depicted in the
paper has been tested for effectiveness at eGramIT Services. The author and eGramIT Services are not responsible for any consequences occurring from this algorithm
being used elsewhere.

PMI Virtual Library | www.PMI.org | © 2011 Krishna V.R. Muppavarapu

11