Cost-Benefit / Cost-

Effectiveness Analysis in the

Context of Impact Evaluations
Presented by: Sergio Bautista

MC Evaluation Workshop and Operations Meeting

January 18-23, 2010
Johannesburg, South Africa
Impact Evaluation and
Efficiency Analysis
Topics

• Adding a costing component to

our impact evaluation
• Defining and measuring
efficiency
• Cost-effectiveness, cost-benefit
and Technical Efficiency

2
Impact Evaluation and
Efficiency Analysis
Topics

• Adding a costing component to

our impact evaluation
• Defining and measuring
efficiency
• Cost-effectiveness, cost-benefit
and Technical Efficiency

3
Why evaluate?
• Are we doing the right thing?
 Are the results we see due to our
intervention?
 What would have happened in the
absence of our intervention?

• Are we doing it right?

 Can we do things more effectively and
efficiently?
 What are the constraints to efficient
production?
4
Focus on outcomes and impact
• What is effect of a specific program on
specific outcomes?

• How much better off are beneficiaries

because of the intervention?

• How would outcomes differ under

alternative program designs?

• Does the program affect different people

differently?

5
Efficiency?
• What are the costs associated with the
program / alternative program designs?

• Is this “technology” the most efficient

alternative we have to achieve the desire
results?

6


Why evaluate?

From Betcherman’s From WDR review of

youth labour review youth HIV evaluations
(14 of 289) (6 of 300+)
7
Basic ingredients
• To start:
 a clear goal,
 a diagnosis of the problem, and
 a solid theory of change.
• Then use whatever tools necessary
to solve the fundamental evaluation
problem:
How can we know that differences in
outcome are due to our intervention?
8
Measuring costs appropriately
• Adding a costing component to the
evaluation does not represent a significant
additional burden
 Most of the information already exists
 Additional information needed can be collected
through the instruments used for the impact
evaluation (community, facility, household)
• Adding a costing component allows to
discuss efficiency issues, in addition to
effectiveness issues

9
Impact Evaluation and
Efficiency Analysis
Topics

• Adding a cost component to our

impact evaluation
• Defining and measuring
efficiency
• Cost-effectiveness, cost-benefit
and Technical Efficiency

10
Production efficiency
• If an economy (public sector) is not
efficient in producing a given service
(outcome), it can produce more without
reducing the production of other services
• It’s about producing using available
resources the best way possible
• In the context of public services, it means
reaching more people

11
Production frontier

y=f(x)

x
Technical efficiency

y=f(x)
B

x
Distance from perfect efficiency

y=f(x)
B
output
distance A

input x
distance
What do we need to do efficiency
analysis? Measuring efficiency
• We have a few options:
 CEA
 CBA
 Technical Efficiency Analysis (production/cost
functions)
• All of them imply different approaches to
deal with the question of efficiency
• All of them need basically the same type of
information:
 Production of outputs / outcomes
 Costs
15
Why should we measure efficiency?
• Resources are limited:
• How do we know it is worthwhile to spend
scarce tax resources on a particular
strategy or program?
• How can we choose among alternatives?
• Is any intervention always better than
none?
• Strong equity and ethical implications…

16
Impact Evaluation and
Efficiency Analysis
Topics

• Adding a cost component to our

impact evaluation
• Defining and measuring
efficiency
• Cost-effectiveness, cost-benefit
and Technical Efficiency

17
What is the difference between them?
• CBA: What is the social benefit achieved
with intervention/program A?
• CEA: What is the cost per unit of
output/impact that can be bought by
intervention A?
 Cost per youth trained
 Cost per life saved
 Cost per children without nutrition problems
 Cost per DALY / QALY

18
How?
• CBA:
 Measure impact
 Value impact in monetary terms
 Measure costs
 Take difference
• CEA:
 Measure incremental impact(s)
 Measure incremental costs
 Take CE ratio: costs/impact
Costs
• Which costs should you measure?
 Everything the program purchases
 Any other resources the program uses,

even things the program gets for free:

• Volunteer labour
• Donated equipment and supplies

• Why measure the cost of free stuff?

Isn’t it free?
NO. FREE INPUTS ARE NOT FREE.
20
Costs
• Why are free things not really free?
• Because they have an opportunity
cost:
 Each free thing can be used for a
different purpose
 Free things make resources available
that would otherwise have been used to
obtain them
• free labour means lower wage payments;
• free medicines means lower expenditure on
medicines 21
Costs
• So how can you measure the cost
of free things?
 The financial or accounting cost method:
Get estimates of similar inputs.
• What does input X cost, on average, in this
place?
• What does someone with these skills and
experience usually earn?

22
Costs
• So how can you measure the cost
of free things?
 The economic or shadow cost method:
Estimate a production function

Y    X   Z   

• where Y = output, and  and are the
returns to paid and unpaid inputs (X and Z),
respectively, in terms of physical output.
• These estimates can be denominated in
value terms.
23
Costs
• What difference does it make
which method I use?
 The accounting method gives
estimates of the average cost.
 The economic method gives
estimates of the marginal cost.

• Marginal cost estimates are

preferred for programming.

24
Costs
• Marginal costs vs Average costs:
 What is the average cost of the input, or what does
an average amount of expenditure (dollar, euro)
buy?
 What will be the cost of the next unit of output, or
how much of the output can I get for the next
amount I spend?

unit
costs rising

rising or falling,
depending on scale
constant (avg costs)

falling

A B
scale
 How much will it cost to scale up from A to B? 25
Discounting in principle
• Why discount?
 Acknowledge that resources have a cost (in real terms of
borrowing and in terms of alternative investments)
• Which discount rate should you use?
 Whichever accurately reflects the cost of capital. What
would it cost if you had to borrow money to finance your
project?
• In general:
 Higher discount rates give greater value to costs and
benefits that occur more quickly.
 Projects that have delayed benefits relative to costs will
have lower net returns.
 Consider both long-term and short-term costs and
benefits.
26
Discounting example
Project costs over time
50

40

30

20

10
benefit (cost)

-10

-20

-30

-40

-50

-60
1 2 3 4 5 6 7 8 9 10
year

27
Discounting example
Project costs and benefits over time
50

40

30

20

10
benefit (cost)

-10

-20
At 0% (nominal):
-30
PV costs = -175
-40
PV benefits = 200
-50
Net PV = 25
-60
1 2 3 4 5 6 7 8 9 10
year

28
Discounting example
Project costs and benefits over time
50

40

30

20

10
benefit (cost)

-10

-20
At 3% discount rate:
-30
PV costs = -161.28
-40
PV benefits = 175.34
-50
Net PV = 14.06
-60
1 2 3 4 5 6 7 8 9 10
year

29
Discounting example
Project costs and benefits over time
50

40

30

20

10
benefit (cost)

-10

-20
At 10% discount rate:
-30
PV costs = -137.79
-40
PV benefits = 132.18
-50
Net PV = -5.61
-60
1 2 3 4 5 6 7 8 9 10
year

30
Cost-effectiveness analysis
• What outcomes or impact do you get for
each dollar’s worth of program
expenditure?
 The number of HIV cases averted.
 The number of young people employed.
 The number of disability-adjusted life-years
(DALYs) saved.

• Can be used to compare alternative

methods of achieving a specific objective.

31
Cost-effectiveness analysis
• Natural units of output/impact
 Lives saved
 Vaccinated children
 Trained teenagers
• Composite units of impact
 QUALYs and DALYs
 Useful to compare with other interventions
 Useful to aggregate different outcomes

32
Cost-effectiveness example

33
(Knowles and Behrman 2005, Table 11)
Cost benefit example
• De-worming in rural primary schools in
Kenya.
 Randomized the timing of the introduction of de-
worming treatment across 75 schools.
 The treatment significantly reduced disease
prevalence, and increased school attendance by
about seven percent.
 The study estimated discounted lifetime benefits
over US$30 per treated child, primarily from gains
in lifetime income.
 Benefits are more than 60 times greater than
costs, based on an estimated cost of US$0.49 per
treated pupil.

34
(Miguel and Kremer 2004)
Efficiency within a program:
technical efficiency analysis

35
The Public Sector…

• Firms are not profit-maximizing

• Firms may not be cost-minimizing
• Firms produce more than one output
 Aggregation, valuing outputs
 Jointness in production, economies of scope
• Firms may not know how much demand
there is for their services
Production Function and Allocative
Efficiency
Y

Y2
Intervention 2

Y1
Intervention 1

Benefit

X1 Investment X
Allocative Efficiency

• Cost-effectiveness analyses
typically assume:
 Results are reproducible in
different contexts and scales
 Interventions are implemented
at their efficiency frontier
• A “cost-effective” intervention can
become very “cost-ineffective” if
implemented inefficiently
Production Function and Technical
Efficiency
Y
Y3

Y2

Y1
Benefit

X1
Investment X
 Scale and Average Unit Cost of VCT programs in 5
countries
1,000
US$ Average Unit Costs

100

10

1
1 10 100 1,000 10,000 100,000

Annual clients receiving VCT

Mexico Uganda Russia India South Africa
40
Source: Preliminary analysis of PANCEA data. Unpublished data. 2006
 Scale and Average Unit Cost of VCT programs in 5 countries
1,000
US$ Average Unit Costs

100

10

1
1 10 100 1,000 10,000 100,000

Annual clients receiving VCT

Mexico
41
Source: Preliminary analysis of PANCEA data. Unpublished data. 2006
 To conclude –
• Why measure efficiency?
 Budget allocation decisions
 Fiscal responsibility
 Facility performance monitoring and management
• How can you measure it?
 Add a costing component to your IE study
 Choose an appropriate method – depends on the
question you want to answer
• Evaluate technical efficiency
 Public sector / NGOs are not like for-profit firms
 Robust methods preferable – DEA or SFA
 Try to understand why facilities differ