Scribd
Upload
18 views

Chapter Six - Project Impact Evaluation

Uploaded by

ephrem
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
18 views

Chapter Six - Project Impact Evaluation

Uploaded by

ephrem
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 28

Chapter Six

Project Impact Evaluation


Abrha M (PhD in Economics)
Mekelle University
My question is: Are we making an impact?
Materials
• Impact evaluation methods
• Impact evaluation practicalities: IE and the
project cycle

• Use rural project examples


Outline - methods
• Monitoring and impact evaluation
• Why do impact evaluation
• Why we need a comparison group
• Methods for constructing the comparison
group
• When to do an impact evaluation
Monitoring and IE
IMPACT Effect on living standards
- infant and child mortality,
- prevalence of specific disease

OUTCOMES Access, usage and satisfaction of users


- number of children vaccinated,
- percentage within 5 km of health center

OUTPUTS Goods and services generated


- number of nurses
- availability of medicine

INPUTS Financial and physical resources


- spending in primary health care
Monitoring and IE

IMPACTS
Program impacts
confounded by local,
national, global effects
OUTCOMES difficulty
Users meet of
service showing
delivery causality
OUTPUTS
Gov’t/program
production
function INPUTS
Impact evaluation
• Many names (e.g. Rossi et al call this
impact assessment) so need to know the
concept.
• Impact is the difference between
outcomes with the program and without it
• The goal of impact evaluation is to
measure this difference in a way that can
attribute the difference to the program, and
only the program
Why it matters
• We want to know if the program had an
impact and the average size of that impact
– Understand if policies work
• Justification for program (big $$)
• Scale up or not – did it work?
• Meta-analyses – learning from others
– (with cost data) understand the net benefits of
the program
– Understand the distribution of gains and
losses
What we need
 The difference in outcomes with the
program versus without the program – for
the same unit of analysis (e.g. individual)
• Problem: individuals only have one
existence
• Hence, we have a problem of a missing
counter-factual, a problem of missing data
Thinking about the counterfactual
• Why not compare individuals before and
after (the reflexive)?
– The rest of the world moves on and you are
not sure what was caused by the program
and what by the rest of the world
• We need a control/comparison group that
will allow us to attribute any change in the
“treatment” group to the program
(causality)
comparison group issues
• Two central problems:
– Programs are targeted
 Program areas will differ in observable and unobservable
ways precisely because the program intended this
– Individual participation is (usually) voluntary
Participants will differ from non-participants in observable
and unobservable ways
• Hence, a comparison of participants and an
arbitrary group of non-participants can lead to
heavily biased results
Example: providing fertilizer to
farmers
• The intervention: provide fertilizer to farmers in a
poor region of a country (call it region A)
– Program targets poor areas
– Farmers have to enroll at the local extension office to
receive the fertilizer
– Starts in 2002, ends in 2004, we have data on yields
for farmers in the poor region and another region
(region B) for both years
• We observe that the farmers we provide fertilizer
to have a decrease in yields from 2002 to 2004
Did the program not work?

• Further study reveals there was a national


drought, and everyone’s yields went down
(failure of the reflexive comparison)
• We compare the farmers in the program region
to those in another region. We find that our
“treatment” farmers have a larger decline than
those in region B. Did the program have a
negative impact?
– Not necessarily (program placement)
• Farmers in region B have better quality soil (unobservable)
• Farmers in the other region have more irrigation, which is
key in this drought year (observable)
OK, so let’s compare the farmers in
region A
• We compare “treatment” farmers with their neighbors.
We think the soil is roughly the same.
• Let’s say we observe that treatment farmers’ yields
decline by less than comparison farmers. Did the
program work?
– Not necessarily. Farmers who went to register with the program
may have more ability, and thus could manage the drought
better than their neighbors, but the fertilizer was irrelevant.
(individual unobservables)
• Let’s say we observe no difference between the two
groups. Did the program not work?
– Not necessarily. What little rain there was caused the fertilizer to
run off onto the neighbors’ fields. (spillover/contamination)
The comparison group
• In the end, with these naïve comparisons,
we cannot tell if the program had an
impact
 We need a comparison group that is as
identical in observable and unobservable
dimensions as possible, to those receiving
the program, and a comparison group that
will not receive spillover benefits.
How to construct a comparison
group – building the counterfactual
1. Randomization
2. Matching
3. Difference-in-Difference
4. Instrumental variables
5. Regression discontinuity
1. Randomization
• Individuals/communities/firms are randomly assigned
into participation
• Counterfactual: randomized-out group
• Advantages:
– Often addressed to as the “gold standard”: by design: selection
bias is zero on average and mean impact is revealed
– Perceived as a fair process of allocation with limited resources

• Disadvantages:
– Ethical issues, political constraints
– Internal validity (exogeneity): people might not comply with the
assignment (selective non-compliance)
– Unable to estimate entry effect
– External validity (generalizability): usually run controlled
experiment on a pilot, small scale. Difficult to extrapolate the
results to a larger population.
Randomization in our example…
• Simple answer: randomize farmers within
a community to receive fertilizer...
• Potential problems?
– Run-off (contamination) so control for this
– Take-up (what question are we answering)
2. Matching
• Match participants with non-participants from a
larger survey
• Counterfactual: matched comparison group

• Each program participant is paired with one or more non-


participant that are similar based on observable
characteristics

• Assumes that, conditional on the set of observables, there


is no selection bias based on unobserved heterogeneity

• When the set of variables to match is large, often match


on a summary statistics: the probability of participation as
a function of the observables (the propensity score)
2. Matching
• Advantages:
– Does not require randomization, nor baseline (pre-
intervention data)
• Disadvantages:
– Strong identification assumptions
– Requires very good quality data: need to control for all
factors that influence program placement
– Requires significantly large sample size to generate
comparison group
Matching in our example…
• Using statistical techniques, we match a
group of non-participants with participants
using variables like gender, household
size, education, experience, land size
(rainfall to control for drought), irrigation
(as many observable charachteristics not
affected by fertilizer)
Matching in our example…
2 scenarios
– Scenario 1: We show up afterwards, we can only
match (within region) those who got fertilizer with
those who did not. Problem?
• Problem: select on expected gains and/or ability
(unobservable)
– Scenario 2: The program is allocated based on
historical crop choice and land size. We show up
afterwards and match those eligible in region A with
those in region B. Problem?
• Problems: same issues of individual unobservables, but
lessened because we compare eligible to potential eligible
• now unobservables across regions
An extension of matching:
pipeline comparisons
• Idea: compare those just about to get an
intervention with those getting it now
• Assumption: the stopping point of the
intervention does not separate two
fundamentally different populations
• example: extending irrigation networks
3. Difference-in-difference
• Observations over time: compare observed
changes in the outcomes for a sample of
participants and non-participants
• Identification assumption: the selection bias is time-
invariant (‘parallel trends’ in the absence of the program)
• Counter-factual: changes over time for the non-
participants

Constraint: Requires at least two cross-sections of data, pre-


program and post-program on participants and non-
participants
– Need to think about the evaluation ex-ante, before the program
• Can be in principle combined with matching to adjust for
pre-treatment differences that affect the growth rate
Implementing differences in
differences in our example…
• Some arbitrary comparison group
• Matched diff in diff
• Randomized diff in diff

• These are in order of more problems 


less problems, think about this as we look
at this graphically
As long as the bias is additive and time-
invariant, diff-in-diff will work ….
Y1
Impact
*
Y1

Y0

t=0 t=1 time


What if the observed changes over time
are affected?
Y1
Impact?
*
Y1

Y0

t=0 t=1 time


DID

You might also like