Chapter I

Cost Benefit Analysis

1.1. Definition and Purpose

It is a technique that can be used to evaluate government projects and programs. It
encompasses an appraisal of a policy based on the costs and benefits of the project

A CBA should be performed for each investment alternative to enable the evaluation and
comparison of alternatives. However, some mandatory systems will not provide net
benefits to the government. In such cases, the lowest cost alternative should be selected.
If functions are to be added to a mandatory system, though, the additional functions
should provide benefits to the government.

1.2. Measurement in the Cost Benefit Analysis

Three measures are used in the cost benefit analysis to indicate the outcome of each
project in an economic sense:

1. Net Present Value

The net present value of a project is the sum of the present value of each cost and other
effects of a
project. A positive NPV indicates that benefits outweigh the costs of a project. Generally
speaking, a
project with a positive NPV should be implemented and if two options are being
compared, the option
with the highest NPV should be selected.

2. Benefit Cost Ratio

The BCR is the ratio obtained by dividing of the sum of the present value of the benefits
by the sum of
the present value of the costs. A BCR greater than 1 indicates that the benefits outweigh
the cost of a
project.

3. Net Present Value per unit of Investment

The NPVI is the net present value divided by the present value of a project’s
capital cost. This ratio
provides an indication of the effectiveness of a given level of investment.
1.3. PROCESS

The CBA process can be broken down into the following steps:

1. Determine/Define Objectives
The CBA should include a problem definition; pertinent background information
such as staffing, system history, and customer satisfaction data; and a list of investment
objectives that identify how the system will improve the work process and support the
mission.

2. Document Current Process

The current process should be thoroughly documented and address these areas:

• Customer Service —Each customer’s role and services required should be clearly
documented and quantified, if possible (e.g., in an average month, a customer inputs
two megabytes (MB) of data and spends 10 hours on database maintenance).
• System Capabilities—Resources required for peak demand should be listed. For
example, 100 MBs of disk storage space and Help Desk personnel to support 50
users.
• System Architecture —The hardware, software, and physical facilities required
should be documented, including information necessary for determining system
costs, expected future utility of items, and the item owner/lessor (i.e., government
or contractor).

Table 1. Cost Elements for Systems

Cost Category Cost Elements

Equipment, Supercomputers, mainframes, minicomputers, microcomputers, disk
Leased or Purchased drives, tape drives, printers, telecommunications, voice and data
networks, terminals, modems, data encryption devices, and facsimile
equipment.
Software, Operating systems, utility programs, diagnostic programs, application
Leased or Purchased programs, and commercial-off-the-shelf (COTS) software.
Commercial Services Commercially-provided services, such as teleprocessing, local batch
processing, on-line processing, Internet access, electronic mail, voice
mail, centrex, cellular telephone, facsimile, and packet switching.
Support services Commercially-provided services to support equipment, software, or
(Contractor services, such as maintenance, source data entry, training, planning,
Personnel) studies, facilities management, software development, system analysis
and design, computer performance evaluation, and capacity
management.
Supplies Any consumable item designed specifically for use with equipment,
software, services, or support services identified above.
 Cost Category Cost Elements
Personnel Includes the salary (compensation) and benefits for government
(compensation and personnel who perform IT functions 51percent or more of their time.
benefits) Functions include but are not limited to program management, policy,
IT management, systems development, operations,
telecommunications, computer security, contracting, and secretarial
support. Personnel who simply use IT assets incidental to the
performance of their primary functions are not included.
Intra-governmental All IT services within agencies, and between executive branch
services agencies, judicial and legislative branches, and State and local
governments.

3. Estimate Future Requirements

Two items to consider are:

• Lifecycle Time—Determine the system lifecycle, or when the system is terminated

and replaced by a system with significant changes in processing, operational
capabilities, resource requirements, or system outputs. Large, complex systems
should have a lifecycle of at least five years, and no more than ten to 12 years.
• Lifecycle Demands—Identify the most appropriate demand measures and use the
measures to determine previous year’ demands, calculate the change in demand
from year to year, average the demand change, and use the average to make
predictions. In a complex situation, more sophisticated tools, such as time-series
and regression analysis, may be needed to forecast the future.

4. Collect Cost Data

Data can be collected, from the following sources, to estimate the costs of each
investment alternative:

• Historical Organization Data—If contracts were used to provide system support in

the past, they can provide the estimated future cost of leasing and purchasing
hardware and hourly rates for contractor personnel. Contracts for other system
support services can provide comparable cost data for the development and operation
of a new system.
• Current System Costs—Current system costs can be used to price similar
alternatives.
• Market Research—Quotes from multiple sources, such as vendors, Gartner Group,
IDC Government, and government-wide agency contracts (GWACS), can provide an
average, realistic price.
• Publications—Trade journals usually conduct annual surveys that provide general
cost data for IT personnel. Government cost sources include the General Services
 Administration (GSA) pricing schedule and the OMB Circular A-76, “Performance of
Commercial Activities” supplemental listing of inflation and tax rates.
• Analyst Judgment—If data is not available to provide an adequate cost estimate, the
CBA team members can use judgment and experience to estimate costs. To provide a
check against the estimates, discuss estimated costs with other IT professionals.
• Special Studies—Special studies can be conducted to collect cost data for large
IT investments. For example, the Federal Aviation Administration (FAA) used
three different in-house studies to provide costs for software conversion, internal
operations, and potential benefits. These data sources became the foundation for a
CBA.

5. Choose at Least Three Alternatives

A CBA should present at least three viable alternatives. “Do nothing” or
“Continue current operations” should not be considered as an alternative. Each
viable technical approach should be included as an alternative. However, the
number of technical approaches may be limited if only one or two are compatible
with the architecture or if some approaches are not feasible for reasons other than
costs and benefits.

6. Document CBA Assumptions

It is important to document all assumptions and, if possible, justify them on the
basis of prior experiences or actual data. This can be an opportunity to explain why some
alternatives are not included. If an alternative is eliminated because it is not feasible, the
assumption should be clearly explained and justified.

7. Estimate Costs
Many factors should be considered during the process of estimating costs for
alternatives. Full lifecycle costs for each competing alternative should be included, and
the following factors should be addressed:

• Activities and Resources —Identify and estimate the costs associated with the
initiation, design, development, operation, and maintenance of the IT system.
• Cost Categories—Identify costs in a way that relates to the budget and accounting
processes. The cost categories should follow current USDA object class codes.
• Personnel Costs—Personnel costs are based on the guidance in OMB Circular A-76,
“Supplemental Handbook, PART II—Preparing the Cost Comparison Estimates.”
Government personnel costs include current salary by location and grade, fringe
benefit factors, indirect or overhead costs, and General and Administrative costs.
• Depreciation—The cost of each tangible capital asset should be spread over the
asset’s useful life (i.e., the number of years it will function as designed). OMB prefers
that straight-line depreciation be used for capital assets.
• Annual Costs—All cost elements should be identified and estimated for each
year of the system lifecycle.
Table 2. Sample Cost Estimates for an Investment Initiation Activity

Work Process Evaluation

DefinitionRequirements

MeasuresPerformance

AnalysisCost-Benefit
DefinitionProblem
Activties/Cost

Security Plan
Categories

Total
Hardware
Software
Services
Support Services 10,000 4,000 1,000 6,000 3,000 24,000
Supplies 100 100 0 100 100 400
Personnel 5,000 10,000 6,000 500 5,000 8,000 34,500
Inter-Agency
Services
Total 5,000 20,100 10,100 1,500 11,100 11,100 58,900
Table 3. Sample System Lifecycle Cost Estimates

Year Startu Acquisitio Developmen Operatio Maintenanc Total

p n t n e
1 100,00 100,000 200,000
0
2 800,000 800,000
3 200,000 80,000 280,000
4 200,000 60,000 260,000
5 50,000 200,000 50,000 300,000
6 50,000 200,000 50,000 300,000
7 200,000 40,000 240,000
8 200,000 30,000 230,000
9 200,000 30,000 230,000
10 200,000 30,000 230,000
Tota 100,00 200,000 800,000 1,600,000 370,000 3,070,00
l 0 0

8. Estimate Benefits
The following six activities are completed to identify and estimate the value of
benefits:

8.1. Define Benefits—Benefits are the services, capabilities, and qualities of each
alternative, and can be viewed as the return from an investment. The following questions
will help define benefits for IT systems and enable alternative comparisons:

• Accuracy—Will the system improve accuracy by reducing data entry errors?

• Availability—How long will it take to develop and implement the system?
• Compatibility—How compatible is the proposed alternative with existing procedures?
• Efficiency—Will one alternative provide faster or more accurate processing?
• Maintainability—Will one alternative have lower maintenance costs?
• Modularity—Will one alternative have more modular software components?
• Reliability—Does one alternative provide greater hardware or software reliability?
• Security—Does one alternative provide better security to prevent fraud, waste, or
abuse?

8.2. Identify Benefits—Every proposed IT system should have identifiable benefits for
both the organization and its customers. Organizational benefits could include flexibility,
organizational strategy, risk management and control, organizational changes, and
staffing impacts. Customer benefits could include improvements to the current IT
services and the addition of new services. Customers should help identify and determine
how to measure and evaluate the benefits.

8.3. Establish Measurement Criteria—Establishing measurement criteria for benefits is

crucial because the Government Performance and Results Act (GPRA) and the Clinger-
Cohen Act (CCA) emphasize tangible measures of success (benefits) related to the
organization’s overall mission and goals.

8.4. Classify Benefits—Benefits that are “capable of being appraised at an actual or

approximate value” are called tangible benefits. Benefits that cannot be assigned a dollar
value are called intangible benefits.

8.5. Estimate Tangible Benefits—The dollar value of benefits can be estimated by

determining the fair market value of the benefits. An important economic principle used
in estimating public benefits is the market value concept. Market value is the price that a
private sector organization would pay to purchase a product or service

Quantify Intangible Benefits—Intangible benefits can be quantified using a subjective,

qualitative rating system. A qualitative rating system might evaluate potential benefits
against the following:

9. Discount Costs and Benefits

After costs and benefits for each system lifecycle year have been identified,
convert them to a common measurement unit by discounting future dollar values and
transforming future benefits and costs to their “present value.”
Table 4. Sample Weighted Benefits Score

Benefi Alternative Alternative Weighting Alternative 1 Alternative 2

t 1 Raw 2 Raw Factor Weighted Weighted
Score Score Score Score
A 4 2 10 40 20
B 3 2 9 27 18
C 4 3 8 32 24
D 2 3 6 12 18
E 3 4 5 15 20
Total 16 14 126 100

Table 4—shows annual costs and benefits for a system lifecycle, along with the discount
factor, the discounted costs and benefits (present values), and the discounted net present
value [NPV]. The discounted costs and benefits are computed by multiplying costs and
benefits by the discount factor. The net benefit without discounting is $380,000
($3,200,000 minus $2,820,000) while the discounted NPV is less than $60,000 because
the biggest costs are incurred in the first two years, while the benefits are not accrued
until the third year. When evaluating costs and benefits, be cautious of returns that accrue
late in the investment’s lifecycle. Due to discounting, benefits that accrue in later years do
not offset costs as much as earlier-year benefits. Also, these later-year benefits are less
certain. Both the business and IT environments may experience significant changes
before these later-year benefits are realized.

Table 5. Sample Discounted Lifecycle Costs and Benefits

Year Annual Annual Discount Discounted Discounted Discounted

Cost Benefit Factor Cost (DC) Benefit Net
(AC) (AB) (DF) ACxDF (DB) DB - DC
ABxDF
1 150,000 0.9667 145,005 (145,005)
2 600,000 0.9035 542,100 (542,100)
3 280,000 400,000 0.8444 236,432 337,760 101,328
4 260,000 400,000 0.7891 205,166 315,640 110,474
5 300,000 400,000 0.7375 221,250 295,000 73,750
6 300,000 400,000 0.6893 206,790 275,720 68,930
 7 240,000 400,000 0.6442 154,608 257,680 103,072
8 230,000 400,000 0.6020 138,460 240,800 102,340
9 230,000 400,000 0.5626 129,398 225,040 95,642
10 230,000 400,000 0.5258 120,934 210,320 89,386
Total 2,820,00 3,200,000 2,100,143 2,157,960 57,817
0

10. Evaluate Alternatives

Many benefits cannot be quantified in dollar terms. As a result, evaluating
alternatives cannot always be done using present values, but valid evaluations can be
made using a combination of dollar values and quantified relative values (values that are
numeric, but do not represent dollar values).

Evaluate All Dollar Values—Once all the costs and benefits for each competing
alternative have been assigned dollar values and discounted, the NPV of the alternatives
should be compared and ranked.

Table 6. Sample Investment Comparison

(Lowest Cost System Provides Highest Benefit)

Alternativ Discounte Discounted Discounted Benefit-Cost

e d Cost Benefit Net (DB - Ratio
(DC) (DB) DC) (DB/DC)
1 1,800,000 2,200,000 400,000 1.22
2 1,850,000 1,750,000 (-100,000) 0.95
3 2,000,000 2,000,000 0 1.00
4 2,200,000 2,100,000 (-100,000) 0.95

Discounted Net—There will probably be very few cases where the alternative with the
lowest discounted cost provides the highest discounted benefit. The next number to
consider is the Discounted Net (Discounted Benefit minus Discounted Cost). If one
alternative clearly has the highest Discounted Net, it is considered the best alternative;
however, it is usually advisable to look at other factors.

Benefit-Cost Ratio—When the alternative with the highest discounted net is not a clear
winner, the benefit-cost ratio or BCR (discounted benefit divided by discounted cost)
may be used to differentiate between alternatives with very similar or equal Discounted
Nets. In Table E-9— Alternative 4 would be the winner because it has a higher BCR than
Alternative 5. Alternatives 4 and 5 are clearly superior to other alternatives because they
have the highest discounted net.

Evaluate With Intangible Benefits—When all the benefits are intangible, evaluation
will be based on quantifying relative benefits.

11. Perform Sensitivity Analysis

Sensitivity analysis tests the sensitivity of input parameters and the reliability of
the CBA result. Sensitivity analysis should assure reviewers the CBA provides a sound
basis for decisions. The sensitivity analysis process requires the following:

Identify Input Parameters—The assumptions documented earlier in the CBA are used
to identify the model inputs to test for sensitivity. Good inputs to test are those that have
significant (large) cost factors and a wide range of maximum and minimum estimated
values. Some common parameters include:

• System requirement definition costs

• System development costs
• System operation costs
• Transition costs, especially software conversion
• System lifecycle
• Peak system demands.

Repeat the Cost Analysis—For each parameter identified, determine the minimum and
maximum values. Then, choose either the minimum or maximum value as the new
parameter value (the number selected should be the one that most differs from the value
used in the original analysis). Repeat the CBA with the new parameter value and
document the results. Prepare a table like Table E-10—to summarize the different
outcomes and enable the results to be quickly evaluated.

Table 7. Sample Sensitivity Analysis

Parameter Parameter Best

Alternativ
Value e
Development 1,500,000 A
Cost ($) 2,000,000 A
2,500,000 B
Transition 100,000 A
Costs ($) 200,000 A
 System 5 A
Lifecycle 10 B
(Years) 15 C
Benefits ($) 1,500,000 A
2,250,000 A
3,000,000 B

Evaluate Results—Compare the original set of inputs and the resulting outcomes to the
outcomes obtained by varying the input parameters. In the previous table, the original
values are the first value listed for each parameter. Sensitivity is measured by how much
change in a parameter is required to change the alternative selected in the original
analysis. The sensitivity guidelines include the following:

• A parameter is not considered sensitive if it requires a decrease of 50 percent or an

increase of 100 percent to cause a change in the selected alternative.
• A parameter is considered sensitive if a change between 10 and 50 percent causes a
change in the selected alternative.
• A parameter is considered very sensitive if a change of 10 percent or less causes a
change in the selected alternative.

In the previous example, the analysis would appear to be somewhat sensitive to the
development costs, but not sensitive to the transition costs and benefits.

12. Compare Investments

Even if the CBA shows that benefits will outweigh costs, using Payback Period
and Return on Investment (ROI) analysis help demonstrate an investment is a better
utilization of funds than other proposed investments.
References

• Blanchard, Benjamin S, Logistic Engineering and Management, Pearson

Education International, 2004
• https://wwwagse.informatik.uni-kl.de/
teaching/seminar/ws2005/Vorbesprechung.pdf –
• www.ifi.unizh.ch/ikm/Vorlesungen/ IM3/WS0203/IM3_files/5-
projektmanagement.pdf –
• www.aib.ws.tum.de/lehre/ws0203/innotech/ Veranstaltung%204%202001-
2002%20Folien.pdf
• www.wm.tu-berlin.de/~mig/files/2003.teaching.ss/ GSF/2003-06-
25_Programmevaluationen-
• 30www.ira.uka.de/teaching/ coursedocuments/3/antsched-sosp.pdf
• www.ifu-kybernetik.de/downloads/ NOWS_Technique_Gothenburg_eng.pdf
• www.lhconsulting.com/fileadmin/downloads/company_information/Company_Se
rvices.pdf
• www.hwwa.de/Projects/Res_Programmes/
RP/Klimapolitik/Papers%20Workshop/Panigrahi.pdf
• www2.sjsu.edu/faculty/watkins/cba.htm - 27k
• www.waisman.wisc.edu/cls/HS2002D.PDF
• www.win.tue.nl/~mchaudro/cbse2004/Reuse%20Economics.pdf
• www.itcinfotech.com/binaryfiles/ Portfolio%20Analysis.pdf
• https://webportal.saalt.army.mil/ asb/studies/2003-forceprotection-exec-brf.pdf
• www.ext.vt.edu/pubs/nutrition/490-403/490-403.html - 202k
• www.smdm.org/Repository.html - 18k
• www.ocio.usda.gov/cpic/doc/ Appendix_E_COST_BENEFIT_ANALYSIS.doc
 Chapter 2
Cost Effectiveness Analysis (CEA)

2.1 Introduction

Large employers face a challenging future in managing health care benefits. Managers
have many program and coverage options, but are limited by budget constraints and data
availability. Traditionally, decision-makers have used return on investment calculations to
help guide their investment choices, but they can also consider another tool — cost-
effectiveness analysis.

Cost-effectiveness analysis refers to the consideration of decision alternatives in which

both their costs and consequences are taken into account in a systematic way. It is a
decision oriented tool, in that it is designed to ascertain which means of attaining
particular educational goals are most efficient. For example, there are many alternative
approaches for pursuing such goals as raising reading or mathematics achievements.
Developed in the military, CEA was first applied to health care in the mid-1960s and was
introduced with enthusiasm to clinicians by Weinstein and Stason in 1977.

2.2 Basic of Cost Effectiveness Analysis

Cost Effectiveness Analysis is a technique for comparing the relative value of various
clinical strategies. In its most common form, a new strategy is compared with current
practice (the "low-cost alternative") in the calculation of the cost-effectiveness ratio.
Cost-effectiveness Analysis, or CEA, is a comparison tool to help evaluate choices. It will
not always indicate a clear choice, but it will evaluate options quantitatively based on a
defined model. For managers, CEA provides peer-reviewed evidence for decision
support.
Cost-effectiveness analysis is closely related to cost-benefit analysis in that both represent
economic evaluations of alternative resource use and measure costs in the same way (see
Cost Benefit Analysis). However, cost-benefit analysis is used to address only those types
of alternatives where the outcomes can be measured in terms of their monetary values.
For example, educational alternatives that are designed to raise productivity and- income,
such as vocational education, have outcomes that can be assessed in monetary terms and
can be evaluated according to cost-benefit analysis. However, most educational
alternatives are dedicated to improving achievement or some other educational outcome
that cannot be easily converted into monetary terms. In these cases, one must limit the
comparison of alternatives to those that have similar goals by comparing them through
cost-effectiveness analysis.
2.3 Measuring Cost Effectiveness
The basic technique has been to derive results for educational effectiveness of each
alternative by using standard evaluation procedures or studies (Rossi and Freeman 1985)
and to combine such information with cost data that are derived from the ingredients
approach. The ingredients approach was developed to provide a systematic way for
evaluators to estimate the costs of social interventions (Levin 1983). It has been applied
not only to cost-effectiveness problems, but also to determining the costs of different
educational programs for state and local planning (Hartman 1981).

2.3.1 Assessing Effectiveness

Before starting the cost analysis, it is necessary to know what the decision problem is,
how to measure effectiveness, which alternatives are being considered and what their
effects are. If a problem has risen on the policy agenda that requires a response, a careful
understanding of the problem is crucial to addressing its solution
Once the problem has been formulated, it will be necessary to consider how to assess the
effectiveness of alternatives. For this purpose, clear dimensions and measures of
effectiveness will be needed. Table I shows examples of effectiveness measures that
respond to particular program objectives

Program Objective Measure of Effectiveness

Program Completions Number of Students complete program
Reducing Dropouts Number of potential dropouts who graduate
Employment of graduates Number of graduates placed in appropriate
jobs
Student learning Test scores in appropriate domains utilizing
appropriate test instruments

. Table I.

Given the problem and criteria for assessing the effectiveness of proposed solutions, it is
necessary to formulate alternative programs or interventions, The search for such
interventions should be as wide-ranging and creative as possible. This procedure sets the
stage for the evaluation of effectiveness of the alternatives, a process which is akin to the
standard use of evaluation methods (e.g., Rossi and Freeman 1985). Estimates of
effectiveness can be derived from previous evaluations or from tailored evaluations for
the present purpose. It is important to emphasize that the evaluation of effectiveness is
separable from the evaluation of costs. Most standard evaluation designs for assessing the
effectiveness of an intervention are also suitable for incorporation into cost-effectiveness
studies.

2.3.2 Cost Estimation

The costs of an intervention are defined as the value of the resources that are given up by
society to effect the intervention. These are referred to as the ingredients of the
intervention, and it is the social value of those ingredients that constitute its overall cost.
At a later stage the distribution of these costs among the decision-making agency and
other entities can be assessed. Accordingly, the method sets out systematically to identify
and ascertain the value of the ingredients that are required for each alternative that is
under consideration.
The ingredients approach to cost estimation entails three distinct phases:
(a) identification of ingredients;
(b) determination of the value or cost of the ingredients and the overall costs of an
intervention; and
(c) an analysis of the costs in an appropriate decision-oriented framework.

The first step is to ascertain which ingredients are required for an intervention . Most
educational interventions are labor-intensive, so an initial concern is to account for the
number and characteristics of personnel. It is important to stipulate whether personnel are
part-time or full-time and the types of skills or qualifications that they need. Beyond this
it is necessary to identify the facilities, equipment, materials, and other ingredients or
resources which are required for the intervention.
Identification of ingredients requires a level of detail that is adequate to ensure that all
resources are included and are described adequately to place cost values on them. For this
reason, the search for ingredients must be systematic rather than casual.
The primary sources for such data are written reports, observations, and interviews.
Written reports, usually contain at least a brief history and description of the intervention.
Other sources of information must be used to corroborate and supplement data on
ingredients from evaluations and descriptive reports. If the intervention is present at a
nearby site, it may be possible to visit and gather additional data on ingredients through
observation. A third valuable source is that of interviews, where present or former
personnel are asked to identify resources from among a number of different
classifications. The three principal types of information reports, observations, and
interviews-can be used to assure the accuracy of the data by comparing the findings from
each source and reconciling differences, the process of triangulation.
Once the ingredients have been identified and stipulated, it is necessary to ascertain their
costs. In doing this, all ingredients are assumed to have a cost, including donated or
volunteer' resources. That is, they have a cost to someone, even if the sponsoring agency
did not pay for them in a particular situation. At a later stage the costs will be distributed
among the constituencies who paid them, but at this stage the need is to ascertain the total
costs of the intervention. Ingredients can be divided into those that are purchased in
reasonably competitive markets, and those that are obtained through other types of
transactions. In general, the value of an ingredient for costing purposes is its market
value. In the case of personnel, market value may be ascertained by determining what the
costs would be for hiring a particular type .of person. Such costs must include not only
salary, but also fringe benefits ' and other employment costs that are paid by the
employer. Many of the other inputs can also be cost by using their market prices. These
include the costs of equipment, materials, utilities, and so on. Clearly the cost of leased
facilities can also be ascertained in this way. Although the market prices of some
ingredients such as personnel can often be obtained from accounting data for educational
enterprises, such data are not reliable sources for ascertaining overall program costs. The
accounting systems that are used by schools were designed for ensuring consistent
reporting to state agencies rather than for providing accurate and consistent cost data on
educational interventions. For example, they omit completely or understate the cost of
volunteers and other donated resources. Capital improvements are charged to such
budgets and accounts during the year of their purchase, even when the improvements
have a life of 20-30 years. Normal cost accounting practices would ascertain the annual
costs of such improvements by spreading them over their useful lives through an
appropriate method. Thus, data from accounting and budgetary reports must be used
selectively and appropriately and cannot be relied upon for all ingredients. There exist a
variety of techniques for ascertaining the value of ingredients that are not purchased in
competitive markets. For example, the method for ascertaining the value of volunteers
and other contributed ingredients is to determine the mark-et value of such resources if
they had to be purchased. The value of facilities can be determined by estimating their
lease value. The annual value of facilities and equipment can be estimated through a
relatively simple approach that takes account of depreciation and interest foregone by the
remaining capital investment.

2.4 Common Application

2.4.1 Evaluating Program Options

In the case of health screening, it is often difficult to determine the most cost-effective
frequency. Too frequent screening has high cost and possibly limited health benefits,
while too infrequent screening has low cost, but poor health outcomes. Determining
appropriate screening frequencies is a useful application of cost-effectiveness analysis.
The following table taken from an analysis on cervical cancer screening shows that life
years are saved at a relatively low cost in the first comparison (screening versus no
screening), but at a very high cost in the second comparison (the marginal cost and
benefit of decreasing the interval between screenings). Typically, an intervention that
costs less than $30,000/life year gained is considered cost-effective medicine. Based on
this analysis, cervical cancer screening every four years is a relatively cost-effective
benefit to cover. It is certainly more cost-effective than screening every three years.

Screen every four Screen every three

years vs No years vs Screen
Screening every four years
Life expectancy increase, days 93.8 1.6
Life expectancy increase, days (discounted 9.5 0.2
5%)
Cost Increases, dollars $ 264 $ 91
Cost per Life years gained $ 10,101 $ 184,528

Table 2. Example Data from an Analysis of Cervical Cancer Screening Frequency

2.4.2 Justifying Program Implementation
When building a case to justify the use of funds, strong data is often compelling
evidence.
Cost-effectiveness analyses can be used to support qualitative arguments for health
interventions. The following table examines a depression treatment improvement
program. Treatment facilities in the study were offered training for practice leaders and
nurses, enhanced educational and assessment resources, and trained psychotherapists for
patient follow-ups. Not only was the intervention relatively cost-effective, but it also
increased attendance in workers suffering from depression.

Quality Improvement Program vs Usual

Care
Quality-adjusted life increase, years 0.0226
Cost Increase $ 485
Cost per quality-adjusted life $ 21,460
Days of Employment Increase 20,9

Table 3. Example Data from an Analysis of a Depression Treatment Improvement

Program
References

Carnoy M, Levin H M 1975 Evaluation of educational media: Some issues Instr. Sci.
4(3/4): 385-406
Cook T D, Campbell D T 1979 Quasi-Experimentation. Houghton Mifflin, Boston,
Massachusetts
Jamison D, Klees S, Wells S 1978 The Costs of Educational Media: Guidiness for
Planning and Evaluation. Sage, Beverly Hills, California
Levin H M 1970 Cost-effectiveness analysis of teacher selection. J. Hum. Resources 5(l):
24-33
Levin H M 1983 Cost-Effectiveness: A Primer. Sage Beverly Hills, California
Levin H M, Glass G V, Meister G 1987 A Cost-effectiveness analysis of computer-
assisted instruction. Eval. Rev. 11(l): 50-72
Mayo J, McAnany E, Klees S 1975 The Mexican telesecundaria: A cost-effectiveness
analysis. Instr. Sci. 4(3/4): 193-236
Quinn B, VanMondfrans A, Worthen B R 1984 Cost-effectiveness of two math programs
as moderated by pupil SES. Educ. Eval. Policy Anal. 6(l): 39-52
Rossi P H, Freeman H E 1985 Evaluation: A Systematic I Approach, 3rd edn. Sage,
Beverly Hills, California
Tatto M T, Nielsen D, Cummings W, Kularatna N G, Dharmadasa K H 1991 Comparing
the Effects and Costs of Different Approaches for Educating Primary School Teachers:
The Case of Sri Lanka. Bridges Project, Harvard Institute for International Development,
Cambridge.
Gold M. R., Siegel J. E., Russell L. B., Weinstein M.C. Cost-Effectiveness in Health and
Medicine. New York: Oxford University Press, 1996.
Neumann P.J. “Why Don’t Americans Use Cost-Effectiveness Analysis?” American
Journal of Managed Care 2004; 10: 308-312.
 Chapter 3

Utility Analysis

3.1. Introduction

Utility analysis is a quantitative method that estimates the dollar value of benefits
generated by an intervention based on the improvement it produces in worker
productivity. Utility analysis provides managers information they can use to evaluate the
financial impact of an intervention, including computing a return on their investment in
implementing it.

The concept of utility was originally introduced by Brogden (1949) and Brogden
and Taylor (1950) and further developed by Cronbach & Gleser (1965). The concept has
been researched and extended by Cascio (1982); Schmidt, Hunter, and Pearlman (1982);
and Reilly and Smither (1983), among others. It was introduced as a method for
evaluating the organizational benefits of using systematic procedures (e.g., proficiency
tests) to improve the selection of personnel but extends naturally to evaluating any
intervention that attempts to improve human performance.

3.1.1.Basic Assumptions

The first assumption of utility analysis is that human performers generate results
that have monetary value to the organizations that employ them. This assumption is also
the basis on which people claim compensation for the work they do.

The second assumption of utility analysis is that human performers differ in the
degree to which they produce results even when they hold the same position and operate
within like circumstances. Thus, salespersons selling the same product line at the same
store on the same shift will show a variation in success over time with a few doing
extraordinarily well, a few doing unusually poorly, and most selling around the average
amount for all salespersons. This assumption is broadly supported in common experience
and in research. It is, for example, the basis on which some performers demand and
receive premium compensation.

The direct implication of these assumptions is that the level of results produced by
performers in their jobs have different monetary consequences for the organizations that
employ them. Performers are differentially productive and the productivity of performers
tends to be distributed normally (Exhibit 1).
3.1.2.How Utility Analysis Builds on These Assumptions

The approach of utility analysis asserts that the utility of any intervention can be valued
by determining how far up the productivity distribution the intervention moves the
performer. The distance the performer is moved is translated into a productivity gain and
the dollar value of that productivity gain is what is termed the utility (U$) of the
intervention.

3.1.3.What Is Needed to Complete a Utility Analysis

In completing the analysis, the performer needs to generate the following:

• A method for measuring role productivity,
• A way to assign monetary value to role productivity,
• The distribution of productivity among performers of the role,
• The dollar value of a one standard deviation difference in role productivity (SD$),
and
• A method to measure the intervention's impact on role productivity.

With these elements of information, the analyst can compute the utility of the
intervention in dollars.

To accomplish the analysis, the analyst must be skilled in the methods of quantitative
analysis in general and utility analysis in specific. This person needs to be aware of the
variety of ways one can measure human productivity, determine its monetary value, and
gauge the affects of interventions on participant performance.
Given that there are a variety of methods for computing utility, the exact resources
needed for the task will depend on the method the analyst selects. The least set of
resources anyone will need are:
• Access to the people who will be using the results of the study to make decisions;
• The identity of the intervention whose utility we will measure;
• A subject matter expert who is knowledgeable of the intervention;
• A description of each affected role including its duties, outputs, and success
criteria;
• The compensation scale for each affected role; and
• A subject matter expert who is knowledgeable of the role(s) affected by the
intervention.

3.2. Method

3.2.1.Getting Ready for the Analysis

3.2.1.1. Understand the people whose decision-making the study will support.
Tip: We need to meet the people who will use our study's findings in order to
understand what information they are seeking, what decisions they will use the
information to make, and any issues or concerns they may have about the study. We
should also alert them to our ongoing need for their feedback on the methods we will
propose for accomplishing the study. Assure them that we will guarantee that the methods
we propose satisfy the professional criteria, but that their feedback is needed to ensure
that the methods are also credible in their eyes and the eyes of anyone with whom they
will share the results.

3.2.1.2. Learn about the intervention we will assess.

Tip: Identify the intervention whose utility we will measure and contact the
subject matter expert who is knowledgeable about it. Learn about the intervention's
purpose, target population, content, operations, cost, and any metrics used to measure its
implementation and effects. Also, uncover what the thinking is about how the
intervention affects the productivity of the performers it targets. With these facts, we can
determine what information needed for the analysis exists and what information we will
need to develop.

3.2.1.3. Learn about the role(s) whose productivity is affected by the intervention.
Tip: Obtain a description of each affected role. Contact the subject matter expert
who is knowledgeable of each role. Learn each role's purpose, duties, outputs, and
success criteria. We also need to understand how the role is valued from a compensation
perspective. For example, is compensation linked to output or is it paid as a salary? We
will want to understand, as well, how the company values the output of each job. If the
output is sold, is it valued by cost or price? And we need to uncover how much
responsibility each role has for the outputs its performers produce. Finally, for each job
that is salaried, obtain its compensation scale and the average salary paid to its
incumbents. If salaries are not normally distributed, we may need to obtain either the
median or modal salary instead of the mean.

3.2.1.4. Determine how to measure the productivity of the performers of each role.
Tip: We will need to develop a productivity measure and a method for
determining the status of each role incumbent on the measure. We will need to use our
understanding of each affected role and the assistance of its subject matter expert. The
subject matter expert will have to approve the method of measurement we devise,
otherwise our approach to measuring productivity will not have credibility in the
workplace.
In devising the productivity measure, it is preferable to base the measure on
production of correct outputs for example, the total amount of sales generated less returns
or the number of welds made per unit of time less the number of welds that fail
inspection. Outputs are the tangible contributions a role makes to an enterprise and
measuring the quantity, quality, and complexity of outputs generated by performers is
usually a measure of productivity that is readily accepted.
Sometimes, however, a workplace will not accept a measure of productivity that
is tied to outputs. In these situations, we still need a way to measure how well the role is
performed. Sometimes supervisor ratings of successful performance are used or
multirater approaches that use rating of supervisors, peers, and subordinates (when
appropriate).
If the workplace will not agree that different performers achieve different levels
of success or that the level of a performer's success in performing the role can be
measured, then the utility analysis cannot be done.
Once we have devised a measure of productivity, plan how we will gather
information about the status of role incumbents on the measure. Our method must be
feasible meaning that its cost must be reasonable, its result credible, and its burden on
participants acceptable.

3.2.1.5. Determine how to value role productivity in dollars.

Tip: The method we choose will be determined by how we measure productivity.
If we use a method that calibrates outputs produced, then we will assign monetary value
based on the dollar value of the outputs. If the job produces an interim output, some
component of a larger final product, then determine the component's contribution to the
total product and determine the value of the role's output by adjusting the value of the
final output. Material outputs can be valued based on cost or sales price. Service outputs
that are used in-house (e.g., a marketing plan, a processed personnel action) can be
valued using market pricing—that is, what it would cost to purchase the service from
external sources.
If we are not using a measure of productivity that is tied to output, then we can use the
typical salary paid for the job (i.e., mean, median, or mode). Salary is acknowledged as
reflecting the value a role contributes to a company.

3.2.1.6. Decide how to measure the affect of the intervention on role productivity.
Tip: Basically, we need to find a mathematical bridge that relates participation in
the intervention and change in role productivity. There are very many ways to accomplish
this. One way is to use a control group comparison. Here, we identify two sets of people
who are comparable in all important ways except that one set went through the
intervention and the other did not. We compare the differences in productivity of these
two sets of people. If the intervention was effective, the people who went through it will
have higher productivity scores and the difference between the groups will represent the
intervention's impact on productivity. Another way is to use correlational methods to
associate some indicator of participation or benefit from the intervention with scores on
role productivity. Be sure that the information with which we are working satisfies the
requirements of the statistical method we use and that our approach makes sense to the
people who will use the results of the analysis. Our solution needs to satisfy both
professional standards and credibility to provide benefit.

3.2.1.7. Create a plan for the utility analysis.

Tip: Be sure our plan documents how we will produce each of the information
elements needed to accomplish the utility analysis. Include in it any decision rules we
will apply in making judgments. For example, if we are also computing a return on
investment ratio, what rule will we apply to decide if the ratio is positive? Will 1.0 be
sufficient? Will the ratio need to be 2.0 or higher? In a professionally conducted analysis,
all decision rules must be documented prior to the study.

3.2.2.Doing the Analysis

3.2.2.1. Determine the productivity of performers.

Tip. Execute our plan for measuring the productivity of current role incumbents.

3.2.2.2. Determine the dollar value of a one standard deviation difference in role
productivity (SD$).
Tip. Distribute the productivity scores we gather. Confirm the distribution is
essentially normal and compute its mean and standard deviation. If the distribution is not
normal, use a transformation method (e.g., z-transformation) to normalize it. Apply our
method for valuing role productivity. Derive the dollar value of productivity achieved by
average performers and the dollar value of a one standard deviation difference in
productivity (SD$).

3.2.2.3. Compute the effects on performer productivity associated with the

performer's participation in the intervention being evaluated.
Tip. Apply our method for measuring the affect of the intervention on
productivity. Determine how many standard deviations of change in worker productivity
the intervention produces (SD).

3.2.2.4. Compute the dollar value of productivity improvements generated by the

intervention.
Tip. The dollar value of productivity improvements generated by the intervention
is the intervention's utility (U$). To compute utility, multiply the number of standard
deviations of change the intervention produces in worker productivity (SD) and the dollar
value of a one standard deviation difference in productivity (SD$) (SD x SD$ = U$).

3.2.3. Following Up the Analysis

3.2.3.1. Add context to the findings.

Tip. Statistical methods are systematic and, when properly applied, produce
reliable results. They do not, however, guarantee meaningful results. Sometimes
quantitative relationships are found for which there is no reasonable explanation. One
reason this occurs is that we rarely can control all the possible factors that may influence
whether a found relationship is valid. Sometimes what our data says is causing the effect
is not, rather some other intervening factor which we have not identified or controlled
may create the appearance of a relationship that does not actually exist.
One way to eliminate this possibility is through the use of controls during the
process of relating participation in the intervention and changes in productivity. Another
way is to explore whether the content of the intervention or the experiences of its
participants suggest a meaningful mechanism for the effects our analysis detects. Study
the intervention to see if any aspects of it suggest such a mechanism. Gather or review
existing accounts of the actual experience of people participating in the intervention.
Their experiences will help provide qualitative information that may either
suggest a lack of reasonableness to the quantitative findings or provide a bases for
making the findings understandable. In analyzing the intervention itself, we look for
content or activities that previous research supports as reliably affecting human
performance. With respect to gathering participant experiences, our typical approach is to
use focus groups in which we gather the participants' observations of what happened,
whether in their viewpoint it affected their performance, and how it affected their
performance.

3.2.3.2. Report the results of the analysis.

Tip. Be sure to describe the methods we used to generate our findings and the
rationale for each. After we draft our report, obtain feedback on it from the subject matter
experts of the role(s) affected by the intervention and the intervention itself. This
feedback may identify issues we need to address and assist we in improving the
communication of our results.
3.3. Example of a Utility Analysis

We were asked to evaluate a contracts management course offered on a fee-for-

service basis by the human resource department of a government agency. The course
trained contract officer's technical representatives (COTRs) in how to specify
requirements, build a request for quote or proposals, evaluate bidders, select and contract
with the best supplier, manage contract performance, and ensure the delivery of the
needed products or services on time, at cost, and to specifications. One of the questions
being asked was whether the course returned a monetary value greater than its cost. We
proposed a utility analysis as the means to assess the monetary benefits produced by the
course and a return on investment analysis to determine the ratio of benefits received to
the cost expended. Prior to these evaluations, we determined that the content offered by
the course was relevant to the COTR role and that the course participants did demonstrate
increased proficiency in their performance as a result of completing the course.

3.3.1.Measuring Productivity and Determining Its Monetary Value

With the role identified, we studied the job it accomplished by reviewing its tasks,
outputs, and performance expectations. No measure of productivity existed—yet the
means for deriving a measure appeared evident. First, the COTR role had a defined
output and criterion for judging success. COTRs were responsible for successfully
satisfying a product or service need within their agency through contracting. Successful
satisfaction of the need meant the timely delivery of products and services that met
technical specifications and the accomplishment of these ends at the cost specified.
Second, there was a monetary value associated with the output. The dollar value of every
contract a COTR managed was systematically determined. Third, there was a logical way
to relate the monetary value of the role's output and its success criterion. A COTR
realized the value of a contract to the degree that the contract was concluded on time, at
cost, and to specifications. Conversely, to the degree it was not concluded on time, at
cost, and to specifications, monetary value was lost.

While the basic logic was sound, conversations with incumbents and supervisors
quickly revealed that while the COTR was responsible for the contract, sometimes he or
she was not free to exercise complete control over its contents or the decision-making
associated with it. Therefore, some amount of the value of the contract was outside the
control of the COTR and its realization or loss should not be credited to the performer.
We also learned that contracts sometimes yielded benefits greater than their face value
and that this could be the result of the COTR's forward thinking, selection of the means
for accomplishing the contract, speed of execution, and other factors.

To measure role productivity, we developed and tested the COTR Productivity

Rating Form. This form measured the degree to which each COTR brings in his or her
assigned contracts at cost, on time, and to specification. It also calibrated the importance
of each of these factors for the contracts managed and the typical degree of control over
each contract the COTR has. The different component measures were converted into an
overall productivity score. This score was a percentage that represented the degree to
which each COTR realized the value of the contracts he or she manages on a yearly basis.

The COTR Productivity Rating Form was sent to the 266 supervisors of COTRs
randomly selected so that the ratings would reflect the status of COTRs in the general
population. One hundred and thirty (130) responses were received (48.9% response rate).
The response level provided estimates of productivity that were accurate to +/- 5% at a
95% level of confidence.

3.3.2.Establishing the Dollar Value of Productivity

 We completed three steps to determine the dollar value of improved productivity.
First, we distributed the productivity scores achieved by COTRs to determine their
average level of productivity and the productivity levels of performers at the 15th and
85th percentiles. Exhibit 2 depicts the distribution of COTR productivity. The average
performer is 81.65% successful in extracting the controllable value from the contracts he
or she manages. In contrast, the exemplary COTR realized 94.04% of the value from the
contracts he or she manages and the poor performing COTR extracted only 69.26%.

To calibrate the value of productivity in dollars, the study used the median face
value of contracts managed by COTRs during one year as modified by the control the
COTR has over the outcome of the contracts. The degree to which a COTR brings in his
or her assigned acquisitions at cost, on time, and to specifications determines how much
of the controllable dollar value of those contracts is realized. The median face value of
contracts fulfilled per year by COTRs was $500,000. Corrected for the degree of control
COTRs have over outcomes, as perceived by their supervisors, the median potential
single year benefit a 100% productive COTR produces is $397,525. By multiplying the
average actual productivity of COTRs (81.65%) against the controllable dollar value of
the contracts a COTR manages on a yearly basis ($397,525), the study estimated the
dollar benefits generated by the average performing COTR at $324,583.13. Poor
performing COTRs— that is, performers achieving at or below the 15th percentile of all
COTRs—generated only $274,344.10 of value each year. Exemplary performing
COTRs, defined as incumbents whose productivity was at or above the 85th percentile of
all COTRs, generated $373,895.44 of value.

3.3.3.Establishing the Dollar Value of Productivity Improvement

 To determine the monetary value of improvement in productivity, the study
computed the dollar value of one standard deviation in change (SD$) in role productivity.
The SD$ for the current distribution of performers is $49,239.04. This means that if some
intervention advanced the productivity of a COTR by one standard deviation, that COTR
would generate $49,239.04 in additional benefits to the agency each year.

3.3.4.Measuring the Course's Affect on COTR Productivity

The correlation between COTR's job proficiency and productivity ratings served
as the mathematical bridge for estimating the course's impact on performer productivity.
The elements required to use this bridge were the amount of proficiency change produced
by the course, the regression coefficient (beta) relating job proficiency scores to
productivity ratings, and the standard deviation of productivity scores. Applying these
elements, the course advances COTRs upward in productivity by .1547 standard
deviations (Exhibit 3).

Exhibit 3. Measuring the Course's Impact on COTR Productivity

Change in Regression Change in Standard Change in
Proficiency Coefficient Productivity Deviation (SD) Productivity
Produced by (Beta) Produced by Difference in Expressed in
the Course Increased Productivity Standard
Proficiency Scores Deviation Units

1.02 1.81 1.84% 11.916% 0.1547

3.3.5.Determining the Course's Utility

As stated above, utility is the dollar value of the increased productivity of a single
COTR that is generated by the course. To determine the utility of the course, the study
translated the distance the course advanced COTRs along the productivity continuum into
dollars. As reported, the course advanced COTRs .1547 standard deviations up the
productivity continuum. We previously determined that one standard deviation change in
productivity has a monetary value of $49,239.04. Multiplying this amount by the .1547
provides us the course's utility (.1547 x $49,239.04 = $7,617.27). This figure ($7,616.18)
is the dollar value of the improvement in productivity evidenced by each COTR as a
result of training (Exhibit 4).

Exhibit 4. Computing the Utility (U$) of Contracts Management Training

Change in Productivity Dollar Value of a 1 Standard Dollar Value of Productivity

Expressed in Standard Deviation Change (SD$) in Improvement Produced by
Deviation Units Productivity COTR Course

0.1547 $49,239.04 $7,616.18

3.3.6.Assessing Return on Investment

The return on investment (ROI) was computed using the conventional method of
dividing the dollar value of the productivity benefits generated by the course by the cost
of participating in the course. In this study a desirable ROI was defined as any value
greater than 1. The study determined the per student cost for completing the COTR
course. It added the fee charged to departments for each COTR taking the course with the
cost of lost opportunity associated with the COTRs not performing their regular job
during the 10-day period of the instruction. This fee ($700) included all expenses
associated with the course. The cost of lost opportunity was computed by dividing the
salary of the typical COTR who participated in the course (GS-14, Step 1) by the number
of hours that define full time employment in the Government (2,087). This per hour cost
is then multiplied by the 80 hours that the COTR is off the job. The opportunity cost per
student was $2,385.63. The total cost for participating in the course was computed as
$3,005.63 per COTR.

The ROI for the course was 2.53 ($7,616,18/$3,005.63) for one year of COTR
performance following completion of the course. This means that for every dollar
invested in completing the course, the sponsoring department receives $2.53 in benefits
the first year. Any reasonable assessment of return should recognize that the benefits of
the course extended forward. Given the general stability of the content the course teaches,
a three year period for return on investment was considered conservative. Within 3 years,
the total productivity improvement benefit is $22,848.54 and the ROI is 7.60—
meaning, for every dollar spent, $7.60 in agency benefits is generated (Exhibit 4).
3.3.7.How Productivity Was Improved

The completion of the two focus group discussions with COTRs who completed
the contracts management course provided insight into the course's mechanism of impact.
Participants uniformly confirmed their experience of benefit from the course. They listed
17 ways their performance was improved by what they learned. One major element they
emphasized was that the course provided a cognitive map of the contracting process that
allowed them to see ahead, to plan and prepare, and feel more confident in the conduct of
their role. As well, the course equipped them to produce the products required by the role
and to know how to judge the adequacy of each product. Also stressed was the learning
about the various players in the contracting process, their responsibilities, the importance
of communicating with them, and the importance of creating a teamed effort. Equally
important, the course participants felt they grasped the principles that ensured the
integrity of the contracting process and that they were able to see how these principles
apply in different contracting situations. Finally, participants also reported the training
coursebook provided with the course served as a continuing learning resource that they
turned to as they encountered new contracting experiences.

3.4. References

Bernstein, Allen L. (1966) A handbook of statistical solutions for the behavioral sciences.
New York: Holt, Rinehart and Winston.

Brogden, H.E. & Taylor, E.K. (1950) The dollar criterion: applying cost accounting
concepts to criterion selection. Personnel Psychology, 3, 133-154.

Burke, Michael J. & Frederick, James T. (1986) A comparison of economic utility

estimates for alternatives SDy estimation procedures. Journal of Applied Psychology, 71,
334-339.
Byron, James & Vitalo, Raphael L. (1991) Quality improvement through exemplar-based
productivity analysis. American Productivity & Quality Center, Brief #82, March 1991.

Cascio,W. (1982) Applied Psychology in personnel management. Reston, VA: Reston

Publishing Company (see Chapter 7, Utility: the concept and its measurement).

Cronbach, L.J. & Gleser, G.C. Psychological tests and personnel decisions. (2nd ed.),
Urbana: University of Illinois, 1965.

McDaniel, Michael; Schmidt, Frank L.; & Hunter, John E. (1987) Job experience as a
determinant of job performance. Paper presented at the 95th Annual Convention of the
American Psychological Association, August 1987.

Myers, Jerome L. (1966) Fundamentals of experimental design. Boston: Allyn and

Bacon, Inc.

Reilly, Richard R. & Smither, James W. (1983) An examination of two alternative

techniques to estimate the standard deviation of job performance in dollars. Journal of
Applied Psychology, 70, 651-661.

Rossi, Peter H.; Freeman, Howard E.; & Wright, Sonia R. (1979) Evaluation: a
systematic approach. Beverly Hills: Sage Publications.

Schmidt, F.L.; Hunter, J.E.; & Pearlman, K. (1982) Assessing the economic impact of
personnel programs on workforce productivity. Personnel Psychology, 35, 333-347.

U.S. Department of Health Education and Welfare (1975) A practical guide to measuring
project impact on student achievement. Washington, DC: U.S. Government Printing
Office (Stock Number 017-080--1400-2).
 Chapter 4

Balanced Scorecard

4.1. Introduction

4.1.1. What is a Balanced Scorecard?

A new approach to strategic management was developed in the early 1990's by

Drs. Robert Kaplan (Harvard Business School) and David Norton. They named this
system the 'balanced scorecard'. Recognizing some of the weaknesses and vagueness of
previous management approaches, the balanced scorecard approach provides a clear
prescription as to what companies should measure in order to 'balance' the financial
perspective. it is a method for measuring a company's activities in terms of its vision and
strategies. It gives managers a comprehensive view of the performance of a business.

The balanced scorecard is a management system (not only a measurement system)

that enables organizations to clarify their vision and strategy and translate them into
action. It provides feedback around both the internal business processes and external
outcomes in order to continuously improve strategic performance and results. When fully
deployed, the balanced scorecard transforms strategic planning from an academic
exercise into the nerve center of an enterprise.

Balanced Scorecard as a measurement tool:

o It allows the organization to assess progress in implementing strategy
o It allows the organization to benchmark its progress against great practices outside
and inside the organization, and over time
o It provides a diagnostic framework for tracking relationships and impact among
strategies BSC is a measurement tool
o It allows the organization to assess progress in implementing strategy
o It allows the organization to benchmark its progress against great practices outside
and inside the organization, and over time
o It provides a diagnostic framework for tracking relationships and impact among
strategies

Balanced Scorecard as a communication tool :

o It allows an organization to articulate its chain of value creation within the
organization and to the world at large
o It provides a platform for planning and communication of planning BSC is a
communication tool
o It allows an organization to articulate its chain of value creation within the
organization and to the world at large
o It provides a platform for planning and communication of planning activities

Kaplan and Norton describe the innovation of the balanced scorecard as follows:
"The balanced scorecard retains traditional financial measures. But financial measures
tell the story of past events, an adequate story for industrial age companies for which
investments in long-term capabilities and customer relationships were not critical for
success. These financial measures are inadequate, however, for guiding and evaluating
the journey that information age companies must make to create future value through
investment in customers, suppliers, employees, processes, technology, and innovation."

Three keys to make implementation of Balanced Scorecard successful:

• Get Buy-In at All Levels: They know you've got those at the top signed on. Here's an
easy way to get everyone else on board: Tie Scorecard objectives to individual
compensation. Works like a charm – and gets employees to think strategically about
their jobs.
• Follow Through: Don't let the Scorecard metrics languish. Update them as major
company goals shift with the times.
• Use the Right Tools: You can buy or build an application that will help you track
metrics. The Balanced Scorecard Collaborative certifies vendor offerings, which you
can check out on their website.

What is needed is a system that provides real insight into an organization’s

operations, balances the historical accuracy of financial numbers with the drivers of
future performance, and assists us in implementing strategy. The Balanced Scorecard is
the tool that answers all these challenges. In the remainder of the chapter we will begin
our exploration of the Balanced Scorecard by discussing its origins, reviewing its
conceptual model, and considering what separates it from other systems.

4.1.2 Why We Use Balanced Scorecard?

 Kaplan and Norton found that companies are using the scorecard to:
• Clarify and update strategy
• Communicate strategy throughout the company
• Align unit and individual goals with strategy
• Link strategic objectives to long term targets and annual budgets
• Identify and align strategic initiatives
• Conduct periodic performance reviews to learn about and improve strategy

Companies that use the Balanced Scorecard methodology get a more accurate,
comprehensive view of their business performance. Balanced Scorecard approach relies
on the monitoring of critical business-strategy-oriented metrics, such as quality, customer
satisfaction, innovation, and market share—measurements that can often reflect a
company’s economic conditions and growth prospects better than its reported earnings.

Balanced Scorecard benefits according to Kaplan & Norton, 1992:

Make strategy operational by translating strategy into performance measures and targets.
Helps focus entire organization on what must be done to create breakthrough
performance. Integrates and acts as an umbrella for a variety of often disconnected
corporate programs, such as quality, re-engineering, process redesign, and customer
service. Breaks down corporate level measures so local managers, operators, and
employees can see what they must do well in order to improve organizational
effectiveness.

4.1.3 Who Are Using Balanced Scorecard?

The Balanced Scorecard, introduced in 1993 by Kaplan and Norton, has served as
the foundation for the Performance Management systems of many fortune 1000
companies and government organizations. A well designed scorecard bridges the gap
between long-term strategies and day-to-day action by aligning performance measures
with the critical perspectives of the organization.

Although by the end of 2001 about 36% of global companies are working with the
balanced scorecard (according to Bain), much of the information in the commercial
sector is proprietary, because it relates to the strategies of specific companies. Public-
sector (government) organizations are usually not concerned with proprietary
information, but also they do not usually have a mandate (or much funding) to post their
management information on web sites.

Below are organizations who implemented Balanced Scorecard as measuring method :

Organization Sector Country

Bank of Tokyo-Mitsubishi Banking Japan
BMW Financial Services Financial Services Germany
DaimlerChrysler Manufacturing Germany
ExxonMobil Corp. Energy USA
Hilton Hotels Corp. Hospitality USA
IBM Information Technology USA
Philips Electronics Manufacturing Netherlands
Sears Roebuck & Company Retail USA
Siemens AG Manufacturing Germany
Southern Gardens Citrus Processing
Corp. Food Processing USA
St. Michael's Hospital Health Care Canada
UK Ministry of Defence Government UK
Unicco Service Co. Industrial Services USA
United Way of Southeastern New
England Humanitarian USA
University of California, Los Angeles Higher Education USA
United
UPS Shipping States
US West Telecommunications USA
Walt Disney World Company Entertainment USA

www.balancedscorecard.org

4.2. Balanced Scorecard as Management System

Balanced scorecard is a management system (not only a measurement system)

that enables organization to clarify their vision, strategy and translate them into action.
Balanced scorecard is cascaded from the company to divisional / unit / functional
objectives, this generally involves all employees contributing to reaching the objectives.

4.2.1 Four Fundamental Perspectives

In organization decision making is dominated by financial parameters. There are

two difficulties with this :
• Financial parameters reflect past decisions or factors or trends, which had created
value in the past. These factors may or may not add value in future.
• Financial measures motivate short term behavior at the expense of long term health of
the organization.
In order to address the difficulties, balanced scorecard proposes four perspective which
enables management to define key perspectives that will drive the business to success.
Financial perspective

How do we perform
according to our
shareholders?

Customer perspective Vision Business proses

and perspective
How do our customer Strategy
see us? What must we excell at?

Learning and growth

perspective

Can we continue to
improve & create value?
www.balancedscorecard.org

By viewing the company from all four perspectives, the balanced scorecard provides a
more comprehensive understanding of current performance. to higher education.

Four perspectives of Balanced scorecards are :

• Learning and Growth Perspective
Learning and Growth Perspective is to achieve our vision, how we will sustain our
ability to change and improve.
 What infrastructure must we build to create long term growth, improvement and
break through performance?

• Business Proceess Perspective

Business Process Perspective is to satisfy our shareholders and customers, and to
know what business processes we must excel at.
 Decide what processes and competencies we must excel at and specify measures
 What are the critical internal operations that enable us to meet our customers’
needs?
 What factors affect cycle time, quality, employee skills and productivity?
 What core competencies and critical technologies are needed to ensure we are
Best in class?

• Customer Perspective
Customer Perspective is to achieve our vision and to know we should appears to our
customers
  How effectively and efficiently do we satisfy the needs of our customers?
 How would you define customer satisfaction?
 What customer service factors really matter to your customers?

• Financial Perspective
Financial Perspective is to succeed financially and to know how we perform
according to our shareholders.
 How will our strategy, implementation and execution contribute to our financial
improvement?
 What are our financial themes?

4.2.2 Double Loop Feedback in Balanced Scorecard

In order to shield the customer from receiving poor quality products, aggressive
efforts were focused on inspection and testing at the end of the production line. To
establish such a process, feedback data should be examined by managers to determine the
causes of variation, what are the processes with significant problems, and then they can
focus attention on fixing that subset of processes. This creates a double loop feedback
process in the Balanced Scorecard.

Metrics must be developed based on the priorities of the strategic plan, which
provides the key business drivers and criteria for metrics that managers most desire to
watch. Processes are then designed to collect information relevant to these metrics and
reduce it to numerical form for storage, display, and analysis. Decision makers examine
the outcomes of various measured processes and strategies and track the results to guide
the company and provide feedback.

The value metrics is in ability to provide a factual basis for defining :

• Strategic feedback to show the present status of the organization from many
perspectives for decision makers
• Diagnostic feedback into various processes to guide improvements on a continuous
basis
• Trends in performance over time as the metrics are tracked
• Feedback around the measurement methods themselves, and which metrics should be
tracked
• Quantitative inputs to forecasting methods and models for decision support systems

4.2.3 Revolution in Strategic Planning

Balanced scorecard is creating a revolution in the profession and practice of

strategic planning. We can compare the some differences between the balanced scorecard
to traditional strategic planning in the table below :
No. TRADITIONAL STRATEGIC PLANNING BALANCED SCORECARD
The agile planning process : shorter than 3
The standard planning cycle : typically 5 years, and strategies can be revised as
1
years, or as short as 3 years necessary without being tied to any annual
cycle
Innovation emerges from across functional
Innovation and major change is not likely to
2 teams composed of employees at all
occur
levelsworking level
The strategic plan contains a large number of Strategic plan is focused on a 3 or 4 strategic
3
goals, objectives, actions items, etc themes
Managers tend to focus on short term Managers have a balanced view of the major
4
financial goals perspectives of performance
5 Planning is goal or project oriented Planning is results oriented
Planning is mostly aimed at improving
Planning is aimed at crafting and improving
6 processes and operations ("doing things
strategies ("doing the right things")
right"), not strategies
The budgeting process is disconnected from Budgets are guided by performance
7
performance measures measurements
Performance measures are collected
Collection of performance measures is
8 systematically and continuously throughout
reactive and ad hoc, driven by data calls
the organization
Performance data is widely reported via a
9 No performance reporting mechanism
distributed software system
10 Not transparent Transparent

4.2.4 Deployment of the Balanced Scorecard

There are two sets of more or less continuous data flows required in the Balanced
scorecard system :

• Downward information flow

Line managers desired outcomes, initiatives, metrics, targets, and schedules. The
goals, metrics, targets, and schedules are aligned with those specified in the top level
strategic plan and balanced scorecard performance plan.

• Upward information flow

Define collection methods for each of the balanced scorecard metrics. This is the
most expensive, labor intensive, aspect of the balanced scorecard system, and has the
most impact on the rank and file employees.
In figure above, the top strategic goals, metrics, and targets are in red, and they flow
downward from the headquarters strategic planning office. The balanced scorecard
measurements, in blue, are collected starting at the branch level and flow upward.

At each level of the organizational hierarchy, data are aggregated across the lower
levels. Aggregation serves to reduce information overload. Periodically, measurements
are collected, aggregated and analyzed at each management level. Performance
evaluations are not only for the top level managers but also in each level. Each level has
its own responsibility.

4.2.5 Advantage and Disadvantages of Balanced Scorecard

Advantages of applying balanced scorecard :

• A framework to focus on key perspectives that will lead to success and provides a
framework to constantly assess performance against targets
• Helps align key performance measures with strategy at all levels of organization
• Provides management with a comprehensive picture of business goals and strategies
at all levels of an organization
• Gives a new way to executives of a company to assess how well their organization is
functioning, how to predict future performance, how to align the organization toward
new strategies to achieve breakthrough performance
• Transforms the strategic plan from an attractive but passive document into the
marching orders for the organization on a daily basis
• Enables executives to truly execute their strategies

Disadvantages and problems of using balanced scorecard :

• Lack of time for the decision makers to focus on strategy
• Confusion between operational efficiency and strategy
• Difficult in creating well defined metrics and connecting them to deliverables
• Cascading the objectives down to the staff that can deliver the results
• Difficult and time consuming to implement a comprehensive balanced scorecard
system in a large organization
• Require sustained top level support and commitment to ramp up and put the system in
place

4.3. Implementing Balanced Scorecard

4.3.1 How to Implement the Balanced Scorecard

To implement balanced scorecard, it has been developed an effective nine steps

processes for building and implementing the balanced scorecard. These steps are :
• Organizational assessment
• Identify strategic themes
• Define perspectives and strategic objectives
• Develop a strategy map
• Derive performance metrics
• Craft and prioritize strategic initiatives
• Automate and communicate
• Cascade the balanced scorecard through the organization collect data, evaluate, and
revise

4.3.2 Cost to Implement the Balanced Scorecard

To implement balanced scorecard, the total cost may be estimated as follows :

• Number of team members x time on team activities
• Facilitator cost
• Request for proposal and software evaluation labor cost
• Software licensing cost
• Installation and testing cost
• Annual maintenance and upgrade cost (typically 20% of initial cost)

4.3.3 Example Balanced Scorecard

Here is the example of Balanced scorecard : Regional Airline

Mission : Dedication to the highest quality of customer service delivered with a sense of
warmth, friendliness, individual pride, and company spirit.

Vision : Continue building on our unique position - the only short haul, low fare, high
frequency, point to point carier in America.
 www.balancedscorecard.org

4.4. Conclusion

• Defines the strategic linkages to integrate performance across organizations.

• Communicates objectives and measures to a business unit, joint venture, or shared
service.
• Aligns everyone within an organization so that all employees understand how what
they do supports the strategy
• Provides a basis for compensation provides feedback to senior management if the
strategy is working
• The Balanced Scorecard is part of a performance management system to enable
organizations to achieve their goals.
• Translates vision and strategy

4.5. References

http://www.balancedscorecard.org

http://www.balancedscorecardsurvival.com

http://www.qpr.com
Chow, Chee W., Kamal M. Haddad, and James E. Williamson, "Applying the Balanced
Scorecard to Small Companies", Management Accounting, August 1997, p. 21.

Curtis, Carey C. and Lynn W. Ellis, "Balanced Scorecard For New Product
Development", Journal of Cost Management, May/June 1997 Vol. 11, No. 3

Drucker, Peter F., "The Theory of Business", Harvard Business Review, Sep.-Oct. 1994,
p. 95.

Gaiss, Michael, "Enterprise Performance Management", Management Accounting,

December 1998, p. 44.

Hoffecker, John & Charles Goldenberg, "Using Balanced Scorecard to Develop

Companywide Performance Measures", Journal Cost Management, Warren, Gorham, &
Lamont, Vol 8, No. 3, Fall, 1994

Kaplan, Robert S. and David Norton, Translating Strategy Into Action The Balanced
Scorecard (Boston, MA: HBS Press 1997)

Maisel, Larry , "Performance Measurement: The Balanced Scorecard Approach", Journal

Of Cost Management, Warren, Gorham, & Lamont, Volume 6, No. 2, Summer, 1992, p.
47.

Porter, Michael E., "What is Strategy"What is Strategy", Harvard Business Review, Nov.-
Dec. 1996, p. 61

Schneiderman, Arthur, "Why Balanced Scorecards Fail", Strategic Performance

Measurement, January 1999, Special Edition

Silk, Scott, "Automating the Balanced Scorecard", Management Accounting, May 1998,
p. 38.

Thomson, Jeff and Steve Varley, "Developing A Balanced Scorecard at AT&T", Journal
of Strategic Performance Measurement, Aug/Sep 1997 Vol. 1, No. 4, p. 14.