Scribd
Upload
10 views

Session 5 - 2223

An implementation evaluation seeks to determine if a public intervention is being carried out as intended. It involves three key steps: 1) describing the intended intervention theory, 2) describing the reality of how the intervention is being implemented, and 3) comparing the implementation to the original design. The evaluation examines both the service utilization plan for reaching the target population and the organizational plan for providing the services. It identifies any gaps between how the intervention was meant to operate versus how it is actually operating in practice. The goal is to understand issues that may be invalidating the public intervention's implementation and impact.

Uploaded by

Lucia Basabe
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
10 views

Session 5 - 2223

An implementation evaluation seeks to determine if a public intervention is being carried out as intended. It involves three key steps: 1) describing the intended intervention theory, 2) describing the reality of how the intervention is being implemented, and 3) comparing the implementation to the original design. The evaluation examines both the service utilization plan for reaching the target population and the organizational plan for providing the services. It identifies any gaps between how the intervention was meant to operate versus how it is actually operating in practice. The goal is to understand issues that may be invalidating the public intervention's implementation and impact.

Uploaded by

Lucia Basabe
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 25

PUBLIC MANAGEMENT

SESSION 5
Guillem Ripoll

07/02/2023
IMPLEMENTATION EVALUATION

2
IMPLEMENTING PLANS

“Now that this bill is the law of the land, let's hope we can get our government to carry it out"
3
POSSIBLE PROBLEMS OF PUBLIC INTERVENTIONS:
DESIGN AND IMPLEMENTATION

The public intervention is (/not) well designed The public intervention is (/not) well implemented
4
WHAT DOES AN IMPLEMENTATION
EVALUATION LOOK FOR?

 Implementation evaluations seek to know… If the public intervention is doing


what it should
 Does the target population receives the intended services?

‘External’ influences compromise well-


intentioned public interventions
(e.g. dependency law in Spain)

5
HOW TO DO IT

6
TIMING

Find problems

Ex-ante / plan. In-itinere / imp. Ex-post / account.


Needs
Design
Implementation
Impact
Economic

Corrective actions
7
STEPS

Identify our interest (evaluation questions)

Describe the design

Describe the reality

Overall comparison of design and reality

Detailed answer of the selected questions


8
EVALUATION QUESTIONS

 Is the service-utilization plan correctly implemented? For example…


 How many persons are receiving the proposed services?
 Are those receiving services the intended targets?
 Are the members of the target population aware of the program?
 Is the organization plan correctly implemented? For example…
 Is staffing sufficient in numbers for the functions that must be performed?
 Is the staff motivated to perform the asked functions?
 Are facilities adequate to support program’s functions?

9
THE THREE ESSENTIAL STEPS IN
IMPLEMENTATION EVALUATION

Conformity
Articulate public of the public
Describe reality
intervention theory intervention
to its’ design

10
OVERALL FRAMEWORK AND
IMPLEMENTATION EVALUATIONS

11
MAIN ELEMENTS OF THE
IMPLEMENTATION EVALUATION

 Element 1: Service utilization plan


 Provide the needed services to the target population (reach them)
 How and why the intended recipients of a public intervention will actually become engaged
and follow it to the point of receiving sufficient services to intiate the change process
represented in the program impact theory
 Element 2: Organizational plan
 Functions and activities the program is expected to perform and the human, financial and
physical resources required for that performance
 How the services will be provided?
 If we have this budget, this staff, this infrastructure, and if we manage and organize these elements in
this way, and if we do these activities … then the services will be properly provided
 Who does what? How is everybody coordinated?

12
OVERALL FRAMEWORK AND THE THREE STEPS
IN IMPLEMENTATION EVALUATIONS
Enrol 50
Describe intervention students

Describe reality 20 students are


1
enrolled
Intervention vs. reality 30 students are
missing

INVALID
IMPLEMENTATION Contract two
Describe intervention teachers
OF 1 or 2
→ Contract one
INVALID PUBLIC 2 Describe reality
teacher
INTERVENTION
Intervention vs. reality One teacher is
13
missing
THE OVERALL FRAMEWORK AND THE BASIC
STRUCTURE
Situation Situation
A B
Resources Activities Outputs Outcomes

Link to program’s service utilization and organizational plans

14
BASIC STRUCTURE AND SAMPLE
QUESTIONS (I)

Program’s
organizational
plan

15
BASIC STRUCTURE AND SAMPLE
QUESTIONS (II)

Program’s
organizational
plan

16
BASIC STRUCTURE AND SAMPLE
QUESTIONS (III)

Program’s
organizational
plan

Program’s
service-utilization
plan

17
COMMON ISSUES IN
IMPLEMENTATION EVALUATION
That need to be checked…

18
PROGRAM’S SERVICE UTILIZATION PLAN
AND COVERAGE

 To what extent the intended targets actually receive program services?


 Usually resources are not high enough in order to address the needs of all the
target population. Therefore, the target population needs to be more specifically
defined:
 Profiles of access and use (e.g. covid-19 vaccine)
 Intensity of service (e.g. university grants)
 Two key issues:
 Does the target population receives the services? And use them?
 Are services enough in terms of quantity and quality?

19
PROGRAM’S SERVICE UTILIZATION PLAN
AND BIAS

 To what extent some subgroups  Comprehension or information deficit


participate in greater/smaller (e.g. taxi city-centre – Pamplona airport)
proportions than others?  Program actions
 Over-inclusion problem  Discretion (e.g. ‘creaming effect’ in some
 Under-inclusion problem US schools)
 Procedures
 Participation derives from:  Learning, compliance and psychological
 Self-selection: costs (e.g. food stamp benefits)
 Voluntary-driven  Unforeseen influences
 Positive perception of the service (e.g. bus)  Tangible and intangible barriers (e.g.
 Negative perception of the service (e.g. public location of public office, commuting,
health) culture, language, gender, timetables)
 Attrition
20
POSSIBLE PROBLEMS IMPLEMENTING THE
PUBLIC INTERVENTION
No Overcome poverty-
associated educational
intervention problems (McLaughlin 1975)

Performance contracting-out
experiment, reading-teachers
Wrong/Partial
(Gramlich and Koshel 1975) intervention

Uncontrolled Project Head Start,


pre-school for underpiviliged
intervention (Cicirelli 1966)
21
RULES FOR SUCCESS?

 The implementation of the public intervention needs to:


 follow service-utilization and organizational plans
 be progressive and aim to reach universality
 appropriately communicated (services offered)
 start with the less favoured groups
 clearly define the profiles of access and intensity of services

22
METHODS AND KNOWLEDGE

23
MINIMUM KNOWLEDGE IN AN
IMPLEMENTATION EVALUATION

1) Material and economic resources: money, 5) Planning Inter- and intra-organization


buildings, information systems… relations (vertical and horizontal)
2) Staff (case-manager, high-skilled workers, 6) Services offered
low-skilled workers, administrative staff…
motivation, skills, leadership, work 7) Coverage
climate and engagement) 8) Services’ quality assessment
3) Operation processes (who does what, and 9) Context
how)
4) Organization structures (units, division of
labour and coordination)

24
HOW?
TECHNIQUES AND DIMENSIONS

Ivàlua guide
(p.34-40) Quantitative Qualitative

25

You might also like