Program Evaluation and Impact Measurement

September 14 to 16, 2022│ 8:00 AM – 12:00 PM

Via Zoom
SESSION NORMS
• Please join every session on time and from a stable and quiet location. If
you will be late, kindly inform us 24 hours in advance.
• Mute your microphone when you are not the one speaking.
• Be mindful of background noise, especially when you are speaking.
• Please “raise you hand” if you have questions. Once acknowledged by
the resource speaker of zoom master, kindly unmute yourself then
proceed with the question.
SESSION NORMS

• Please use a laptop in joining every session. It will not be possible to

complete some tasks using your phone.
• Maximize your learning by staying focused. Spacing out or with your
mind elsewhere, you might miss out on important information.
• Copies of the presentation will be shared/emailed after every day
session.
• Set aside 1-2 hours every day to review the lectures and/or complete
home works if there are.
Session Objectives:
1. Explain the monitoring and evaluation (M&E) process;
2. Discuss results-based framework as basis for effective
program evaluation;
3. Develop skills in measuring social impact;
4. Describe different types of assessment tools; and
5. Review current practices based on other corporate
examples.
CSR AND MONITORING AND
EVALUATION
Evolution of Business Concepts
“There is one and only one social
responsibility of business — to use
it resources and engage in activities
designed to increase its profits so
long as it stays within the rules of
the game, which is to say, engages
in open and free competition
without deception or fraud.” -
Milton Friedman,
Milton Friedman, New York Times
advocate of free
markets and 1976
Magazine, September 1970
Nobel Peace Prize
Winner for Economics
Evolution of Business Concepts
EVENT IMPACT

Business scandals Call for Transparency

Business Ethics

Awareness of Environment Triple Bottom Line

Impact Sustainable Development
Environmental
Accountability
Demand for Social Justice Development inclusion
Value chain approach
 Levels of Responsibilities
SOCIETY
Philanthropic
(being a good
citizen) Desires

Ethical (do what

is right and fair;
avoid doing Expects
harm)

Legal (obey law

and fulfill legal Requires
obligations)

Economic (be
profitable) Requires
Companies’ Sphere of Influence
Core Business - looks into company’s
internal operation to create maximum
business value to the community

Social Investment – initiatives where

companies use profits, employees’
competencies/expertise, products,
advertising and its influence to advance
and promote social agenda
Policy Advocacy – contributes to
societal change by:
•influencing key policy issues
•taking the leadership in catalyzing
discourses affecting business and society
Source : Nelson, J (1996) Business Partners in
Development. IBLF. London
Company Social Responsibility Framework

❑ Vision and Mission

❑ Core Values
❑ Business Principles
❑ SR Goals
Internal Drivers

❑ Internal resources of the organization

(strengths)
❑ Core business resources
❑ Reputation Management Plan
❑ Previous social investment/ responsibility
experience
External Drivers

❑ Community Needs
❑ Societal Pressures and Opportunities
❑ Business Challenges and Trends
❑ National Agenda/ Goals
❑ Conditions in the Environment
PRESSURE POINTS/BUSINESS ISSUES
• License to Operate
• Revenue Sharing
• Employee Satisfaction
• Stakeholder Expectation – more Social Development
projects (community/ area of operations)
• Corporate Reputation
• Regulatory institutions/ Compliance issues (e.g.
Environment, Health)
CSR BUSINESS IMPACT
❑ Performance enhancement
- revenues (sales)
- income
- cost savings
❑ Human Resource
- team building
- leadership skills
- job satisfaction
- recruitment, retention (employee loyalty)
CSR BUSINESS IMPACT
❑ Marketing/PR
- Corporate image
- Customer awareness
- PR opportunities
- sales possibilities
- expand business network
- testimonials
- Unsolicited praise
CSR BUSINESS IMPACT

❑ Operations
- Permits
- License to operate/Business continuity
- Cost Avoidance/Savings
- Legal fees
- Manpower
- Equipment/Capital
CSR BUSINESS IMPACT

❑ Community Relations
- Happy and contented community
- Trust
- Cooperation
- Good company image
- Harmonious relationship
- Improved communication
- Community safety
Why Monitoring and Evaluation
Greater transparency and accountability
(governments and development organizations)

Improved project performance and

development impact

Why is
Systematic and professional
M&E management of organizations
needed
Learning and data driven decision-
making

Demonstrates effective
resource allocation and use
of funds
Source: WHO, UNDP
Forces of Change: International and
External Initiatives
• Sustainable Development Goals
• Highly Indebted Poor Country (HIPC) Initiative
• International Development Association (IDA) funding
• World Trade Organization (WTO) membership
• European Union (EU) enlargement and European Union Structural
Funds
• Transparency International
• Global Reporting Initiative
• UN Global Compact Network
• Etc.
FOCUS: ACCOUNTABILITY

1. Accountability to Business Partners:

• This includes not only companies’ own performance but also

that of business partners and other actors throughout the
company’s value chain.
• The effective mechanism systems for increasing
accountability generally allow a company to be inclusive,
responsive, and engaged with its stakeholders.
FOCUS: ACCOUNTABILITY

2. Accountability at Senior Management level:

• Senior management needs related information

from the concerned stakeholders and is
answerable to them in many ways.
FOCUS: ACCOUNTABILITY
3. Accountability to Stakeholders:
• Stakeholders now expect companies to provide access to
information on impacts of their operations, to engage
stakeholders in meaningful dialogue, and to be responsive to
particular concerns unearthed in the dialogue process.
• To increase the credibility of what is disclosed, leadership
companies are also investigating carefully the value of
various types of assurances that might support their
reporting efforts.
IMPACT MEASUREMENT:
SOCIAL IMPACT
Definition and Model for Development
What does Social Impact mean?
Telling the story of the change we are
bringing to people’s lives and to the
organizations we work with

The net effect of an activity on a

community and the well-being of
individuals and families – Center for Social
Impact, 2017
What does Social Impact mean?

It means the consequences to human populations of

any public or private actions that alter the ways in
which people live, work, play, relate to one another,
and organize to meet their needs and generally cope as
members of society - Inter-organizational Committee
on Guidelines and Principles for Social Assessment
(1994)
What is Social Impact Assessment?

Efforts to assess or estimate, in advance, the

social consequences that are likely to follow
specific policy actions (including programs/
projects and the adoption of new policies and
specific government actions) (Centre for Good
Governance, 2006).
What is Social Impact Assessment?

It is a process that provides a framework for

prioritizing, gathering, analyzing, and
incorporating social information and
participation into the design and delivery of
developmental interventions.
 MONITORING AND
EVALUATION:
Definition, Concepts, Principles
REVIEW: THE PROJECT MANAGEMENT CYCLE

Project Replication Project Planning

Project Scaling-up and Development
Project Extension
Project Termination

Project Evaluation Project Implementation

Project Monitoring
Activity 1. Sample M & E Activities

• Describe 1 monitoring activity in your program

• Describe 1 evaluation activity in your program
• How would you differentiate monitoring from
evaluation?
 What is Monitoring?

• Monitoring is “an internal process undertaken by management

carried out to assess progress at regular intervals throughout the
life of a project or program.” (Laurence Taylor, Good Monitoring
and Evaluation Practice, Guidance notes, October, 2001)
• Activity which tracks actual performance against what is planned
• Provides regular feedback and indications of progress
(“indicators”)
• Identifies strengths and shortcomings and recommends
corrective measures
 Monitoring involves:
✓ Establishing indicators of efficiency, effectiveness
and impact;
✓ Setting up systems to collect information relating
to these indicators;
✓ Collecting and recording the information;
✓ Analyzing the information; and
✓ Using the information to inform day-to-day
management.
Why Do We Need It?

→Regular and systematic assessment of progress

→Continued review of partners’ capacity development needs
→Improve results-based reporting on achievements
→Strengthen teamwork and ownership of the program among
implementing partners
→Feeds into evaluation and real-time learning

Source: UNDP Handbook on Planning, Monitoring, and

Evaluating for Development Results, 2009
What is Evaluation?

• is a process and time bound exercise which seeks to

determine as systematically and as objectively as
possible the effectiveness, relevance, and impact of
activities in the light of the objectives of a certain
project or program; and
• the process of determining the value or worth of a
project, i.e., identifying and reflecting on its effects,
thus far, and judging the value of it.
Evaluation involves:
• Looking at what the project or organization intends to achieve – what difference did it
want to make? What impact did it want to make?
• Assessing its progress towards what it wanted to achieve, its impact targets.
• Looking at the strategy of the project or organization. Did it have a strategy? Was it
effective in following its strategy? Did the strategy work? If not, why not?
• Looking at how it worked. Was there an efficient use of resources?
• How sustainable is the way in which the project or organization works?
• What are the implications for the various stakeholders in the way the organization
works?
Why do we need to do it?
▪ Whether we are Doing the Right Things
• Relevance/rationale/justification
• Client satisfaction
▪ Whether we are Doing it Right
• Effectiveness/coherence
• Efficiency: optimizing resources
• Sustainability
• Impact
Source: UNDP Handbook on Planning, Monitoring, and
Evaluating for Development Results, 2009
Why do we need to do it?

▪ Whether there are Better Ways of Doing it

• Alternatives
• Good practices
• Lessons learned
• Improved positioning to influence next development planning
framework

Source: UNDP Handbook on Planning, Monitoring, and

Evaluating for Development Results, 2009
 Relationship Between M&E
• M & E are two different management tools that are closely related,
interactive and mutually supportive

• Through routine tracking of project progress, monitoring can provide

quantitative and qualitative data useful for designing and
implementing project evaluation exercises

• Through the results of periodic evaluations, monitoring tools and

strategies can be refined and further developed
Commonalities of M&E

• Both monitoring and evaluation must be

planned at the program/ project level
• Baseline data and appropriate indicators
of performance and results must be
established
 Monitoring vs. Evaluation
Adapted from UNICEF, A UNICEF Guide for Monitoring and Evaluation: Making a Difference?
New York, 1991, p.3
MONITORING EVALUATION
Resource deployment Effectiveness of strategy
Focus on project inputs, activities, outputs, work-plans Measures outcome, impact, sustainability

Purpose is to adjust work-plan Purpose is future programming

Structured top-down methods Participatory bottom-up methods
Regularly scheduled reporting processes Periodically schedules inquiring processes (episodic)

Emphasis on “decision making” Emphasis on “organizational learning”

Asks “are we doing what we said we would? Asks “are we doing the right
thing”
Project team accountable to project management Project management
accountable to stakeholders
Provides starting point for evaluation Provides endpoint for monitoring
Primary clients are “internal” Primary clients are “external”
M & E and RESULTS-BASED
PROGRAM MANAGEMENT
 RBM: Historical Context
PERIOD TOOL/FRAMEWORK FOCUS
1960s “Doing things right” rather than “Doing Financial Planning and Accountability, procedures
the right things”
1970s Management by Objective Objectives and good performance indicators
1980s Gantt Charts, Critical Path Method Program Management by activity methods
(CPM), Performance Evaluation and
Review Technique (PERT)
1980s later part ISO, TQM (Total Quality Management) Quality control assurance, service quality and standards
1990s “Results Revolution” More program effective use of development assistance
2000s Results Framework (Logical Framework, Consensus for achieving measurable results (MDGs, SDGs)
Theory of Change)
Results Management/also RBM thru Performance management, stakeholder participation,
the “Results Framework” devolution of management authority and responsibility,
stakeholder needs and preferences

Source: ADB, OECD

What is Results-based Management?

“Results based management provides a coherent framework for

strategic planning and management based on learning and
accountability in a decentralized environment. It is first a
management system and second, a performance reporting
system.” – World Bank, 1997
What is Results-based M&E?

• is a systematic approach to tracking results and

performance, based on a transparent and
reflective logical and results framework approach,
and to measure impact through evaluation.
- UN IOM Egypt
The Power of Measuring Results
• If you do not measure results, you cannot tell success from failure
• If you can not see success, you can not
reward it
• If you can not reward success, you are probably rewarding failure
• If you can not see success, you can not learn from it
• If you can not recognize failure, you can not correct it
• If you can demonstrate results, you can win
public support
Source: Adapted from Osborne & Gaebler 1992

45
Basis for Effective Monitoring and
Evaluation

• Results Framework
- Logic Framework
- “Theory of Change”
Results Framework
• An explicit articulation (graphic display, matrix, or summary) of the
different levels, or chains, of results expected from a particular
intervention—project, program, or development strategy – World Bank,
2012.

• Often depicted as a theory of change, logic model, or log frame—

identifies the intended results an intervention aims to achieve and the
logical cause-and-effect relationship between the intervention’s inputs,
activities, and their results. – Global Environment Facility, 2017

• Outcomes and impacts are the main focus of a results framework

Logical Framework Matrix

Core Elements of
A Logical Framework a Project’s Design
matrix displays key
Narrative
elements of a Project Summary
Indicators Data Sources Assumptions

Design (Columns 1 Goal

and 4) as well as
Information pertinent Purpose

for project Monitoring

and Evaluation Outputs

(Columns 2 and 3)
Inputs
Source: USAID

Roadmap for Project

Monitoring & Evaluation
Monitoring and Evaluation Elements in a Logframe

Indicators Means of
Verification

- Performance Indicators - Data Sources

- Baseline values - Data Collection/Analysis
- Targets: Quantity, Quality Methods
and/or Time - Frequency and schedule
for data collection

Source: USAID
Typical Logframe Matrix
 Results-Based: Result Chain
Impact is the overall and long-term effect of an
intervention; it is the ultimate improvement/
changes in people’s lives; IMPACT
intended/ unintended
Outcomes are institutional and
OUTCOME behavioral changes triggered by
the outputs;
Outputs are tangible, time-bound
product/services resulting from completion
of activities; are under the control of the OUTPUTS
project
Activities are the various
ACTIVITIES steps taken to carry out a
project
Inputs are means mobilized to
carry out activities
e.g. funds, staff time and other INPUTS
resources
Defining the Elements
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS

Resources What the programs The direct products of Specific results Benefits for
dedicated to or does with the inputs program activities (products & services) participants during
consumed by to fulfill its mission that emerge from and after program
the programs e.g. e.g. processing the inputs activities
e.g. •Provide food and •Number of classes e.g. e.g.
•Money shelter to homeless taught •Improved nutritonal •New knowledge
•Staff and staff families •Number of counseling status •Increased skills
time •Provide job trainings sessions conducted •Acquired skills •Changed attitudes
•Volunteers and •Educate the public •Number of • Prevention of drug or values
volunteer time about signs of child educational materials addiction •Modified behavior
•Facilities abuse distributed •Improved health •Improved
•Equiptment •Create mentoring •Number of hours conditions of pregnant conditions
and supplies relationships for service delivered women •Altered status
youth •Number of •Improved school
participants served performance
REVISITING YOUR PROGRAM/
PROJECT LOGFRAME
 SAMPLE COMPLETED LOGFRAME
Project Description Indicators Source of Verification Assumptions

GOAL Incidence of water-borne diseases Municipal hospital and clinic records

To contribute to improved health, reduced by 30% by 2012, specifically collected by mobile health teams.
particularly of under 5s and the general among low income families who live by
health of the river ecosystem. the river.

OBJECTIVES/OUTCOME Concentration of e. coli reduced by Monthly water quality surveys -The Clean River legislation is
Improved quality of river water. 20% (compared to levels in 2003) and conducted by the EPA and the River introduced by the EPA and enforced
meets national health and sanitation Authority. --Up river water quality remains
standards by 2012. unchanged

OUTPUT 1.1 60% of household fecal waste is 1.1 Annual sample survey conducted -Waste water treatment meets
1.1 Reduced volume of fecal waste disposed of via latrines or sewage by municipality between 2009 and national standards
discharged into river connections. 2012. -fishing cooperatives meet obligations
1.2 Reduced volume of household 1.2 … 1.2 to establish waste collection systems
refuse directly dumped into the river
system
ACTIVITIES 1.1.1 Baseline data (Knowledge 1.1.1 6 month progress report -Municipal budgets for improvements
1.1.1 Conduct baseline survey of Practice Coverage) for household waste 1.1.2 Extension team progress reports to sewage systems remain unchanged.
households management exists 1.1.3 Approved project charter from
1.1.2 Prepare and deliver public 1.1.2 Schedule of visits of mobile the Ministry of Public Works
awareness campaign teams completed Etc.
1.1.3 Procurement of materials for 1.1.3 …
latrines and sewage connections Etc.
1.1.4 Etc.
1.2.1 Etc.
 ALTERNATIVE LOGFRAME FORMAT
PROGRAM/PROJECT: _________________________________________
GOAL: To contribute to improved health, particularly of under 5s and the general health of the river ecosystem.
PURPOSE: Improved quality of river water
COMPONENT/ INPUT ACTIVITY OUTPUT (RESULTS) INTENDED DESIRED
OBJECTIVE OUTCOME IMPACT
Fecal Waste Conduct baseline survey of Baseline data on waste
households management established
Management
Prepare and deliver public X households received info on
awareness campaign environmental awareness % Reduced volume of fecal waste
discharged into river
Procurement of materials for X latrines constructed Incidence of water-borne
latrines and sewage X sewage connections diseases reduced by 30%
connections installed
specifically among low income
Conduct of waste X households segregating
families who live by the river.
Household waste
management seminars wastes
volume reduction Concentration of e. coli
Established bgy waste reduced by 20% from __year
collection system % Reduced volume of household values
refuse directly dumped into the
Provision of waste X households with waste bins river system
bins/containers
Construction of Materials MRF utilized by community
Recovery Facility (MRF)
END OF DAY 1