Strengthening Monitoring and Evaluation

System (SMES) Project

SMES MONITORING AND EVALUATION

TRAINING MANUAL

MODULE 1:
INTRODUCTION OF
RESULTS-BASED MONITORING
AND EVALUATION

(version-4)
November 2009
 SMES Monitoring and Evaluation Training Manual

Module 1: Introduction of Results-based Monitoring and Evaluation

Table of Contents

Abbreviations
Overview of Module 1 .......................................................................................I-1

Chapter 1: What is Monitoring and Evaluation? ..........................................I-2

1-1 Definition of Monitoring and Evaluation...............................I-2
1-2 Types of Evaluation ............................................................I-5
1-3 Methods and Tools of Monitoring and Evaluation ...............I-6

Chapter 2: Trend towards Results-based Monitoring and Evaluation ...........I-9

2-1 What is Managing for Development Results? .....................I-9
2-2 MfDR and Results-based Management ............................ I-11
2-3 Introducing Results-based Monitoring and Evaluation ......I-16
2-4 Ownership of Monitoring and Evaluation ..........................I-17

Chapter 3: Use of Monitoring and Evaluation Results ................................I-19

3-1 Different Uses of M&E Results..........................................I-19
3-2 To whom, When and How to Give Feedback ....................I-20

Chapter 4: M&E System and Stakeholders in Nepal ...................................I-22

Recommended Readings ...............................................................................I-25

Useful Website of International Development Partners ..................................I-26
 Abbreviations

AFD: Agence Française de Développement

AusAID: Australian Government’s Overseas Aid Program
CIDA: Canadian International Development Agency
DAC: Development Assistance Committee of the OECD
DFID: Department for International Development
GTZ: Gesellschaft für Technishe Zusammenarbeit (German Agency for
Technical Cooperation)
ISR: Implementation Status and Results Framework
JICA: Japan International Cooperation Agency
KfW: Kreditanstalt für Wiederaufbau (German Reconstruction Credit Institute)
Logframe: Logical Framework
M&E: Monitoring and Evaluation
MfDR: Managing for Development Results
OECD: Organization for Economic Co-operation and Development
PLA: Participatory Learning and Action
PRA: Participatory Rural Appraisal
PRISM: Performance Reporting Information System for Management
RBM: Results-based Management
RF: Results Framework
ROAR: Results Oriented Annual Report
SMES: Strengthening the Monitoring and Evaluation System (Project)
SRF: Strategic Results Framework
UNDP: United Nations Development Programme
USAID: United States Agency for International Development
Overview of Module 1

Module 1 consists of four Chapters: Chapter 1 - What is Monitoring and Evaluation;

Chapter 2 – Trend towards Results-based Monitoring and Evaluation; Chapter 3 –
Linkage and Use of Monitoring and Evaluation Results, and Chapter 4 - Monitoring and
Evaluation System in Nepal. Chapter 1 explains the basic definition, types, methods,
tools and stakeholders of monitoring and evaluation, Chapter 2 overviews the trend for
results-based monitoring and evaluation, Chapter 3

Objectives: - To understand the concepts, purposes, types, methods and

tools for monitoring and evaluation (M&E)
- To understand the concept of managing for development
results (MfDR) and results-based M&E
Process: - Presentation on the concept of M&E (30 minutes)
- Discussion about what the definitions and objectives of the
M&E are in their agencies and why M&E is important (20
minutes)
- Brief introduction on the concept of the MfDR and
results-based M&E (30 minutes)
- Questions and answers (10 minutes)
Key Words: - Monitoring and evaluation (M&E)
- Managing for development results (MfDR)
- Results-based management (RBM)
- Results-based monitoring and evaluation
Resources Required: - Module 1
- PowerPoint “Module 1”
Outputs Expected: - The concept of M&E and MfDR understood
- The importance of the use of M&E results understood

I-1
Chapter 1: What is Monitoring and Evaluation?

1-1 Definition of Monitoring and Evaluation

Monitoring and evaluation, often called by the acronym of M&E, is a
periodic review of interventions, which include policy, programs and projects. It is
conducted for the purpose of: (1) improving performance in terms of formulating,
designing and implementing policy, programs and projects, and (2) enhancing
transparency and accountability of the agencies which implement the interventions to
the tax payers, other stakeholders/investors, and intended target groups.

The definitions of monitoring and evaluation are shown below:

“Monitoring is a continuing function that uses systematic collection of data on

specified indicators to provide management and the main establishment of an on-going
intervention with indications of the extent of progress and achievement of objectives
and progress in the use of allocated funds.”

“Evaluation is a systematic and objective assessment of an on-going or completed

project, program or policy, in design, implementation and results. The aim is to
determine the relevance and fulfillment of objectives, efficiency, effectiveness, impacts
and sustainability. An evaluation should provide information that is credible and useful,
enabling the incorporation of lessons learned into the decision-making process of
implementing agencies. Evaluation also refers to the process of determining the worth
or significance of an activity, policy or program. An assessment, as systematic and
objective as possible, of a planned, on-going, or completed intervention.”
(Source: Glossary of Key Terms in Evaluation and Results Based management, OECD/DAC, 2002,
modified by the SMES Project Team)

Both of monitoring and evaluation are conducted to see the progress of interventions
such as policy, programs and projects. However, they have different characteristics, and
they are complementary to each other.

Monitoring is conducted, continuously and regularly as an internal management activity

during the implementation of interventions. The results of the monitoring are reported to
the higher levels of the implementation bodies of the interventions as well as to other
related stakeholders.

I-2
 On the other hand, evaluation is conducted at certain periods of the implementation of
interventions usually by external experts, who are equipped with technical knowledge
and analytical as well as communication skills, hired by the implementation bodies.
Evaluation reports, which cover the evaluation results, lessons and recommendations,
are prepared by the experts and shared with the stakeholders inside and outside of the
implementation bodies. The following table summarizes the differences between
monitoring and evaluation.

Table-1: Outline of Monitoring and Evaluation

Timing Objective Conducted by Tools

Monitoring - Conducted - To check if an - Designed, - Logframe
continuously and intervention is organized, and - Plan of
regularly during being implemented implemented by Operations
implementation as planned (status of the staff members - Monitoring
of an Inputs and Important of the System
intervention Assumptions, intervention sheets
progress of implementation
Activities, body
achievement of
Outputs and Project
Purpose)
- To improve the
quality of an
intervention by
revising its direction
based on the results
of monitoring
Evaluation - Conducted at - To check the - Designed and - Logframe
different times relevance, efficiency, organized by the - Five
of an effectiveness, impact staff members of evaluation
intervention and sustainability of the intervention criteria
along with its an intervention implementation
implementation - To improve the body
cycle: 1) before; quality of an - Implemented by
2) at the intervention external experts
mid-term; 3) at - To obtain lessons
the end, and 4) learnt for other
after the projects and future
intervention planning

I-3
The results of M&E may:
- help ministries and agencies manage activities at the sector, program and project
levels;
- enhance future planning of policies, programs and projects;
- help ministries in their policy analysis, and policy and program development;
- support ministries in evidence-based policy making, results-based budgeting
(budgeting is made based on predefined objectives and expected results that are
measurable), and performance-informed budgeting (the past and future performance
information are referred in the budget decision-making process).
- help ministries enhance transparency and accountability to the stakeholders including
related government agencies, donor agencies, tax payers and beneficiaries.

Box-1: Project Cycle Management (PCM)

The results of the project evaluation need to be reflected in future projects formulation.
Therefore, the project formulation-implementation-evaluation process is not a one-way
process but a cycle. The system to manage projects with this viewpoint is called “project
cycle management.” According to EC’s Project Cycle Management Guidelines, Project
Cycle Management is a term used to describe the management activities and
decision-making procedures used during the life-cycle of a project.

As shown below, three key components of the project cycle are “formulation (=Plan)”,
“implementation (=Do)”, and “monitoring and evaluation (=See).” The project cycle starts
with project formulation, and after implementation, ends with evaluation. During the
implementation, the progress of the intervention is being monitored.

Formulation of a
new project
PLAN
Formulation

Feedback

DO

Evaluation Implementation
Feedback

SEE Monitoring

I-4
1-2 Types of Evaluation
There are several types of evaluation categorized by stages of the project cycle and by
subject. Each agency has its own evaluation system, in which types and timing of
evaluation to be conducted are stipulated.

(1) Evaluation Types by Stages during the Project Cycle

In JICA, evaluation is classified into four types (ex-ante evaluation, mid-term

evaluation, terminal evaluation and ex-post evaluation) as Figure-1 and Table-2 show:

Figure-1: JICA Project Evaluation Types and Schedule

Start of the Completion of

Intervention the Intervention

Implementation of the Intervention

Ex-ante Mid-term Terminal Ex-post

Evaluation Evaluation Evaluation Evaluation

Feedback for future formulation and designing of interventions

Table-2: Outline of the Evaluation Types by Stages of Project Cycle

Evaluation
Timing Purpose
Type
Ex-ante Before To determine the necessity and conformity of an
Evaluation commencement intervention
of an intervention To clarify the details of an intervention and set
indicators
Mid-term At the mid- point To examine the progress of an intervention
Evaluation of implementation To revise the original plan and/or operation structure if
period necessary
Terminal Upon completion To review an intervention focusing on its efficiency,
Evaluation of an intervention effectiveness and sustainability
To determine if the follow-up is necessary or not
Ex-post After completion of To review an intervention focusing on impact and
Evaluation an intervention sustainability
To obtain lessons and recommendations for improving
formulation and implementation of future interventions

I-5
(2) Evaluation Types by Subject

Evaluation is also classified by subject such as project level, program level, sector level,
thematic level, country program level, and policy level. The definitions of the major
evaluation types by subject are summarized in below Table-3.

Table-3: Evaluation Type by Level

Type Definitions
Project-level evaluation Evaluation of an individual development intervention
designed to achieve specific objectives within specified
resources and implementation schedule
Program-level evaluation Evaluation of a set of interventions, which usually covers a
number of related projects or activities in one country,
seeking to attain specific development objectives,
Sector program evaluation Evaluation of a single sector or sub-sector such as health,
education, primary education, etc.
Thematic evaluation Evaluation of selected aspects, such as gender,
environment, etc. of various development interventions, or
evaluating a range of sector program in different countries
Policy-level evaluation Evaluation of a country development policy or sector
development policy

1-3 Methods and Tools of Monitoring and Evaluation

A sample of monitoring and evaluation tools, methods and approaches, including data
collection methods, analytical frameworks and types of evaluation and review is
introduced below. Some of the tools and approaches are complementary and can be
combined while some of them are substitutes. The choice of appropriate methods or
tools for any given context will depend on a range of considerations such as the
intended use of monitoring and evaluation results, the main audience for the findings,
how quickly the information is needed, and the cost available for monitoring and
evaluation.

(1) Logical Framework Approach

The Logical Framework clarifies objectives of any project, program or policy as well as
causal links among inputs, processes, outputs, outcomes and impacts. It also helps
identify indicators at each stage with which the performance and achievement are

I-6
measured, as well as risks which might impede the attainment of the objectives.
Because of this nature of Logical Framework, it serves as an effective tool to improve
quality and design of project/program and to prepare detailed operation plans. It is also
useful to review progress and take corrective action because it provides objective basis
for activity review, monitoring and evaluation.

(2) Rapid Appraisal Methods

Rapid appraisal methods are quick, low-cost ways to gather the views and feedback of
beneficiaries and other stakeholders to respond to decision-makers’ needs for
information. Opinions of beneficiaries can be obtained by key informant interview,
focus group discussion, community group interview, direct observation, and simple
survey. Rapid appraisal provides qualitative understanding of complex socio-economic
changes, highly interactive social situations, or people’s values, motivations, and
reactions. It also helps to provide context and interpretation for quantitative data
collected by more formal surveys.

(3) Participatory Evaluation

Participatory evaluation is an evaluation method in which representatives of project

implementing agencies and other stakeholders work together in designing, carrying out
and interpreting an evaluation. In order to ensure a fair representation of different needs,
participatory evaluation emphasizes the participation of key stakeholders, especially the
intended target groups and the involvement of special focus groups such as women,
children and other vulnerable groups through participatory techniques such as
Participatory Rural Appraisal (PRA), Participatory Learning and Action (PLA),
stakeholder analysis, and beneficiary assessment.

(4) Cost-benefit and Cost-effectiveness Analysis

Cost-benefit and cost-effectiveness analysis are tools for assessing whether or not the
costs of any activity can be justified by the outcomes and impacts. Cost-benefit analysis
measures both inputs and outputs in monetary terms. Cost-effectiveness analysis
estimates inputs in monetary terms and outcomes in non-monetary quantitative terms
(such as improvements in students’ scores). They are used for informing decisions about
the most efficient allocation of resources and for identifying projects that offer the
highest rate of return on investment.

I-7
(5) Impact Evaluation

Impact evaluation is a systematic identification of the effects on individual households,

institutions and the environment caused by a given development activity. Impact
evaluations can range from large scale sample surveys in which project populations and
control groups are compared before and after, and possibly at several points during
intervention, to small-scale rapid assessment and participatory appraisals where
estimates of impact are obtained by interviews, case studies and available secondary
data. Impact Evaluation is conducted to measure outcomes and impacts of an activity
and to distinguish these from influence from other factors, to clarify whether costs for
an activity are justified, and to strengthen accountability for results.

(6) Meta-evaluation

Meta-evaluation is an evaluation designed to aggregate findings from a series of

evaluations. Meta-evaluation can be used to denote the evaluation of an evaluation to
judge its quality and/or assess the performance of the evaluators.

I-8
Chapter 2: Trend towards Results-based Monitoring and
Evaluation

2-1 What is Managing for Development Results (MfDR)?

The international development community, since late 1990’s, has made efforts to
improve their evaluation activities. Key milestones include the Millennium Declaration
with the adoption of the Millennium Development Goals (MDGs) in 2000, the
Monterrey Consensus on Financing for Development in 2002, the Rome Declaration on
Harmonization in 2003, the Marrakech Memorandum on Managing for Development
Results in 2004, and the Paris Declaration on Aid Effectiveness in 2005.

“Managing for developing results (MfDR)” is one of the key issues emerged from such
efforts. With the definition agreed in Marrakech as shown in Box-2, MfDR calls for
developing countries to increase their commitment to policies and actions that promote
economic growth and reduce poverty, and developed countries to support them through
more effective aid and trade policies.

Box-2: Definition of MfDR:

Managing for Development Results (MfDR) is a management strategy focused on

development performance and sustainable improvement in country outcomes. It
provides a coherent framework for development effectiveness in which performance
information is used to improve decision making, and it includes practical tools for
strategic planning, risk management, progress monitoring and outcome evaluation.

MfDR Core Principles are:

(1) Focus the dialogue on results at all
phases;
(2) Align programming, monitoring
and evaluation with results;
(3) Keep measurement and reporting
simple;
(4) Manage for, not by results, and
(5) Use results information for learning and decision making.

(Source: Memorandum of the Marrakech Roundtable on Managing for Results, 2004)

I-9
MfDR is a new concept and tool to enhance the impact and effectiveness of
development and poverty reduction programs as well as to improve performance
management of development interventions.

The concepts that underlie MfDR are that global development assistance can be made
more effective by enhancing country ownership, aligning assistance with country
priorities, harmonizing development agencies’ policies and procedures, and focusing
more consistently on the achievement of development outcomes.

In the Paris Declaration of 2005, donors and partner countries expressed their
commitments to MfDR as follows (Source: OECD/DAC MfDR Source Book 2006):

Commitments to MfDR expressed by partner countries:

1) Strengthen the linkages between national development strategies and annual and
multi-annual budget processes.
2) Endeavour to establish results-oriented reporting and assessment frameworks
that monitor progress against key dimensions of the national and sector
development strategies; and that these frameworks should track a manageable
number of indicators for which data are cost-effectively available.

Commitments to MfDR expressed by donors:

1) Link country programming and resources to results and align them with
effective partner country performance assessment frameworks, refraining from
requesting the introduction of performance indicators that are not consistent with
partners’ national development strategies
2) Work with partner countries to rely, as far as possible, on partner countries’
results-based reporting and monitoring frameworks.
3) Harmonize their monitoring and reporting requirements, and, until they can rely
more extensively on partner countries’ statistical, monitoring and evaluation
systems, with partner countries to the maximum extent possible on joint formats
for periodic reporting.

Joint commitment to MfDR expressed by partner countries and donors:

1) Work together in a participatory approach to strengthen country capacities and
demand for results-based management (RBM).

In order to achieve MfDR, there are issues to be worked on at the national level as well
as at the sector program and project levels. The followings are some of the major topics
on which developing countries and development agencies are working at each level.

I-10
(1) At the National Level

Development countries need to better manage their development process toward desired
outcomes. MfDR begins with identifying national goals and developing the strategies to
achieve the goals.

Many countries are making progress in linking their poverty reduction strategy or
national development strategy to results-based expenditure management and
performance orientation in public administration. At the same time, central and line
ministries are developing more results-focused strategies accompanied by results
frameworks to monitor progress.

(2) In Sector Programs and Projects

Partner countries and development agencies work together in strengthening the focus on
results in their sector programs and projects through aligning national strategies and
sector development plans, harmonizing results reporting, and strengthening capacity to
manage for results in programs and projects.

Within the context of poverty reduction strategies and national development plans,
development agencies and partner countries are collaborating to identify priority sectors
in which targeted programs and projects can support achievement of country outcomes.

2-2 MfDR and Results-based Management (RBM)

What is RBM?

According to OECD/DAC, Results-based Management (RBM), which has been adopted

by many development agencies since 1990s in order to improve development
performance, is a management strategy focusing on performance and achievement of
outputs, outcomes and impacts.

Its main purposes are to improve organizational learning and to fulfill accountability
obligations through performance reporting. Although RBM is nearly synonyms with
MfDR, MfDR incorporates broader ideas such as collaboration, partnership, country
ownership, harmonization, and alignment, focusing on performance and improvements
in country outcomes whereas RBM can be used for shorter-term, smaller-scale
intervention.

I-11
Emergence of RBM and its development

In the 1990s, as governmental reform became especially active in North American

countries, RBM was adopted in the public administration field. RBM is used by the
governments of the US and Canada as a new management method for administrative
activities, and has been discussed at development-related international conferences.
RBM has now been adopted in a number of developing countries as well

Phases of RBM

The following table shows the seven phases of RBM. As shown below, in RBM, (1)
objectives are formulated, (2) indicators to show progress toward the objectives are
identified, and (3) targets are set.

These three phases are called Strategic Planning. The next two phases, which are (4)
monitoring results, and (5) reviewing and reporting the results are called Performance
Measurement. Finally, (6) evaluation is conducted and (7) the information obtained
through M&E are used for improvement of interventions (In RBM, evaluation is
considered to supplement monitoring information).

Thus, RBM entails not only setting objectives and implementing M&E activities but
also using M&E results to improve management.

Table-4: Seven Phases of RBM

1 Formulating objectives Strategic Performance Results-based

2 Identifying indicators Planning Measurement Management

3 Setting targets
4 Monitoring results
5 Reviewing and reporting results
6 Integrating evaluation
7 Using performance information
(Source: Annette Binnendijk, Results Based Management in the Development Co-operation Agencies: A
review of Experience Background Report, 2001)

I-12
Performance Measurement, Performance Indicators and Performance Monitoring

Performance Measurement is a system to assess performance of development

intervention against stated goals. Performance Indicators, which are variables that
allow verification of changes in development interventions or show results relative to
what was planned, provide means to measure achievement at all the levels (input,
process, output, outcome, and impact levels).

Performance monitoring is a continuous process of collecting and analyzing data to

compare how well a project, program, or policy is being implemented against expected
results.

Adoption of Performance Measurement by Development Agencies

Traditionally, in development interventions, reporting on the activities and achievements

had been conducted at the time of or after the termination of the interventions. In recent
years, however, many development agencies place more importance on measuring
performance during their interventions.

For example, some agencies such as CIDA, DFID and AusAID came to report the
performance of their major interventions annually in performance monitoring reports. In
addition to the performance monitoring, some agencies such as CIDA, DFID and World
Bank make it compulsory for the country offices to submit final reports with a focus on
the preventing and promoting factors of the project achievement and its process.
Evaluation also plays an important role in this context.

The following table shows how some of major development agencies, which employ
RBM, conduct Performance Measurement at the project/program level1.

1
As USAID and UNDP are not conducting performance measurement at the project level, the
description on the USAID and UNDP are about performance measurement at the program level.

I-13
Table-5: Performance Measurement System of Major Donors at the Project Level

Agency System of Performance Measurement

USAID USAID conducts performance measurement at the country program
level annually; all the Operating Units including country and regional
offices submit annual “Results Review and Resource Request” to the
Headquarters. Results Review is the review against the outcome
indicators set in the original Results Framework. All the reports are
reviewed at Headquarters for resource allocation and compiling annual
Performance and Accountability Report.
UNDP UNDP conducts performance measurement at the program level
employing Strategic Results Framework (SRF). Each country office
monitors its programs against SRF, and compiles Results Oriented
Annual Report (ROAR) annually. Headquarters produce organizational
annual report based on ROARs.
World Bank Implementation Status and Results Reports (ISR) are made annually by
reviewing progress against project Results Framework. Rating is also
conducted to measure performance of each project. Results of the
ratings are incorporated into Annual Report on Portfolio Performance,
which is the organization’s annual report.
DFID Each section reports performance through the data system, called
PRISM (Performance Reporting Information System for
Management), by measuring performance against indicators set on
Logframe. Rating is conducted as well. The results of the performance
measurement and ratings are compiled into an organizational report
annually (Autumn Performance Report) to be submitted to the
Parliament.
Source: USAID: http://evalweb.usaid.gov/index.cfm, UNDP: http://www.undp.org/eo/, World Bank:
http://www.worldbank.org/oed/, DFID: http://www.dfid.gov.uk/aboutdfid/evaluation.asp

Logical Framework and Results Framework

While the Logframe had been widely used among development agencies, the number of
agencies which adopt Results Framework is increasing as the following Table 6 shows.
Results Framework, usually laid out in diagrammatic form, is the program logic that
explains how the development objective is to be achieved, including causal
relationships and underlying assumptions. Although it has a lot in common with
Logframe, Results Framework helps us to comprehend link between objectives and

I-14
higher goals. Therefore, it can be used rather at the program or strategy level than at the
project level.

Table-6: Application Status of Logframe (LF) and Results Framework (RF) by

Major Development Assistance Agencies

Agency LF RF Application Status of LF/RF

DFID X LF is developed at the time of pre-appraisal of a project
and reviewed at the mid-term of project implementation.
The achievement of the output and outcome are assessed at
the terminal evaluation.
GTZ/KfW X LF, called Project Planning Matrix (PPM), is developed at
the time of pre-appraisal of a project.
JICA X LF, called Project Design Matrix (PDM), is developed at
the ex-ante evaluation and reviewed throughout the
implementation stage. JICA conducts mid-term and
terminal evaluation at the project level by using five
evaluation criteria of OECD-DAC.
AFD X Project evaluation is conducted according to the five
evaluation criteria of OECD-DAC based on the LF.
USAID X Strategic Objectives (SO), Intermediate Results and outputs
are set in RF. Evaluation is conducted at the SO level
(Project evaluation is not conducted).
UNDP X Strategic Development Objective, Intermediate
Results/Outcome and Outputs as well as those indicators
are set in the Strategic Results Framework (SRF). (Project
evaluation is conducted as well.)
World Bank X RF is developed at the project level. Performance
indicators are set for each outcome and used for M&E.
CIDA X The Framework of Result, which incorporates the concept
of RF of USAID, is used for project and program
evaluation.
Source: DFID: http://www.dfid.gov.uk/aboutdfid/evaluation.asp,
GTZ: http://www.gtz.de/en/leistungsangebote/6332.htm
JICA: http://www.jica.go.jp/english/operations/evaluation/
AFD: http://www.afd.fr/jahia/Jahia/lang/en/pid/18
USAID:http://evalweb.usaid.gov/index.cfm,
UNDP: http://www.undp.org/eo/,
World Bank: http://www.worldbank.org/oed/,
CIDA: http://www.acdi-cida.gc.ca/CIDAWEB/acdicida.nsf/En/JUD-111795644-KJX

I-15
As far as the above table tells us, as a general trend, Logframe is used by the agencies
which mainly conduct performance measurement and evaluation at the project level,
while Results Framework is adopted by the organizations which conduct performance
measurement and evaluation rather at the program level.

Box-3: RBM and Change of M&E Trend

It is often pointed out that introduction of RBM has changed the role of traditional
monitoring and evaluation. Conventionally, monitoring was conducted to check the
implementation process. As for evaluation, when to conduct an evaluation used to be
pre-determined according to the duration of the project period (ex-ante, mid-term,
terminal, ex-post, etc.), and performance analysis and reporting were conducted after a
project has terminated.
However, in RBM, monitoring is conducted continuously, not only to check the
process but to measure the performance. Also, many agencies started to conduct an
evaluation only when necessary in order to complement monitoring activities.
Regarding reporting, many development agencies now report their performance
annually instead of after the termination of an intervention. This way, RBM enabled
assessing performance in a timely manner and feeding the results into ongoing
strategies to better manage their interventions.

2-3 Introducing Results-based Monitoring and Evaluation

Traditional implementation-focused M&E systems are designed to address compliance
– the “did they do it” question. Conventional monitoring focuses on assessing how well
a project, program, or policy is being executed, and it was responsibility of a particular
unit of the implementing body. However, this approach does not provide policy makers,
managers, and stakeholders with an understanding about success or failure of the project,
program or policy.

Results-based M&E systems are designed to address the “so what” question. A
results-based M&E provides feedback on the actual outcomes and goals of government
actions. Results-based system helps answer the following questions:

What are the goals of the organization?

Are they being achieved?

How can achievement be proven?

I-16
 Box-4: Essential Actions to Build
Results-based M&E System

(1) Formulate outcomes and goals

(2) Select outcome indicators to monitor
(3) Gather baseline information on the current condition
(4) Set specific targets to reach and dates for reaching them
(5) Regularly collect data to assess whether the targets are being met
(6) Analyze and report the results
(Source: “Ten Steps to a Results-based M&E System” World Bank 2004)

Results-based M&E is a major component of the MfDR toolbox that helps both
countries and agencies systematically measure the progress of program and project
outcomes. Thus, effective results-based M&E is useful to better manage policies,
programs, and projects, and to demonstrate progress to civil society stakeholders. It also
shows the extent to which specific activities or programs contribute to achieving
national outcomes.

Box-5: The Power of Measuring Results

 If you do not measure results, you cannot tell the success from failure.
 If you cannot see success, you cannot reward it.
 If you cannot reward success, you are probably rewarding failure.
 If you cannot see success, you cannot learn from it.
 If you cannot recognize failure, you cannot correct it.
 If you can demonstrate results, you can win public support
(Source: Adapted from Osbourne and Gaeblier 1992)

2-4 Ownership of Monitoring and Evaluation

In light of Paris Declaration, it is important that developing countries strengthen their
ownership not only in implementation of projects but also in the area of monitoring and
evaluation. While the country-led evaluation enables developing countries to utilize
evaluation results effectively as described in Box 6, it requires those countries to exert
more active initiatives. Therefore, capacity development in the area of monitoring and
evaluation has become of greater significance.

I-17
 Box-6: Use of Aid Evaluation for Developing Countries

Evaluation enables countries to:

(1) Examine whether the project has achieved its intended effects;
(2) Assess the distribution of benefits among different groups;
(3) Analyze factors determining effectiveness in achieving project objectives;
(4) Institutionalize experiences and lessons from project implementation and
management, and
(5) Provide information for decision-makers on “replicability” of the project in the
future.
(Source: OECD/DAC MfDR Source Book 2006)

I-18
Chapter 3: Use of Monitoring and Evaluation Results

3-1 Different Uses of M&E Results

It is expected that the evaluation results are utilized for decision-making of policy
planning, budget allocation, personnel management and organizational learning. This
process is the so-called “evaluation feedback”.

As we have learned in the Chapter 1 in the Module 1 (See below Figure 1), project cycle
of “formulation (=Plan)”, “implementation (=Do)”, and “monitoring and evaluation
(=See)” not only applies to the project level, but also to the program and policy level.
In another word, the results of program/policy evaluation are reflected in the
formulation of program and policy plans such as PRSP.

Figure 2: Policy/Program/Project Management Cycle for Evaluation Feedback

PLAN
Formulation

Feedback
DO

Evaluation Implementation
Feedback

SEE Monitoring

Findings from monitoring and evaluation results can be used in a variety of ways, as
shown in Box7:

I-19
 Box 7: Ten Uses of Results Findings

1. Respond to elected officials’ and the public’s demands for accountability

2. Help formulate and justify budget requests
3. Help make operational resource allocation decisions
4. Trigger in-depth examinations of what performance problems exist and what
corrections are needed
5. Help motivate personnel to continue making program improvements
6. Formulate and monitor the performance of contractors and grantees
7. Provide data for special, in-depth program evaluations
8. Help provide services more efficiently
9. Support strategic and other long-term planning efforts (by providing baseline
information and later tracking progress)
10. Communicate better with the public to build public trust.
(Source:: Jody Zall Kusek, Ray C. Rist, World Bank, 10 Steps for Results-Based M&E, 2004,
Originally from Hatry, Harry, P, Performance Measurement, 1999)

3-2 To whom, When and How to Give Feedback

To whom to give feedback
Recipients of feedback can be categorized into the following three groups (Figure 3).

1) Stakeholders in the policy management cycle:

The evaluation results should be given to government agencies (such as agencies of
related sector, planning department, regional departments, etc.) where the findings,
lessons, recommendations derived from the evaluation are used for new and
on-going policies/programs/projects for improvement.

2) Stakeholders outside the policy management cycle:

Parliamentary committees, budgetary office ministers, boards, auditors, the general
public, NGOs, donors, the media, etc., who are not directly involved in the policy
management cycle but who should be informed of the evaluation results for
accountability reasons.

3) The budget office:

The budget office is involved in the feedback process when the evaluation results
are linked with a budget allocation. The office uses the evaluation results as a
decision-making tool for allocating budget.

I-20
Figure 3 Donors NGOs Beneficiaries
Budgetary Office
Auditors
Accountability
Media

General Public Boards Parliament

Learning Learning

Evaluation
Operational/Related
Operational Departments
Departments
Forming new policies/
Modifying ongoing
programs/projects
policies/programs/
projects

Implementation Planning

(Source: JICA, Textbook for the Area Focused Training Course in the Forum on Institutionalization of
Evaluation System, 2007)

When to give feedback

Providing evaluation feedback in a timely manner can be difficult often because of the
time required to finalize evaluation reports. However, for feedback to be effective, it is
critical that the evaluation results be shared at the right time with the right audience.

How to give feedback

Various media can be utilized to deliver the results of an evaluation depending on the
needs of the audience. Some of the common media used are listed below.
• Evaluation reports
• Evaluation report summaries (4-10 pages)
• Annual Evaluation Reports
• Websites
• Periodical Newsletters
• Press events
• Seminars/workshops
• Participation in evaluations

I-21
Chapter 4: M&E System and Stakeholders in Nepal

The government of Nepal has its own monitoring and evaluation system, the details of
which are explained in Module 6: Project Reporting for Monitoring. Many actors are
involved in the system, and each of them plays its role in the process. Key M&E
stakeholders and their functions in the context of development interventions in Nepal
are summarized as follows:

(1) Policy-makers/decision-makers such as the Cabinet, parliamentarian, Ministers and

Secretaries are to utilize M&E results in policy making;

(2) NPCS is to design and update monitoring reporting formats and guidelines to be
used by other ministries, line offices at the district level and project offices. NPCS is
also to compile M&E results collected from other ministries, and report the results
in NDAC meetings. NPCS provides other ministries with feedback on their M&E
results;

(3) All the officers of NPCS as well as Planning/M&E division or section officers of
other ministries are to design, conduct and report M&E activities to related offices.
At the same time, they are to utilize M&E results in future program and project
formulation and planning;

(4) Planning/M&E officers of departments, regional offices, district offices and project
offices are to plan, conduct and report M&E activities to related offices as well as to
utilize M&E results in future program and project formulation and planning;

(5) M&E experts and academicians of external institutions conduct evaluation surveys
for the government and prepare evaluation reports. They refer and utilize the
government M&E results/reports for their research as well;

(6) Target groups and other stakeholders join M&E activities of related interventions
which employ participatory M&E methods, and/or to be interviewed for M&E data
collection;

(7) Tax payers are to check how the government uses the financial resources and
achieves the intended goals based on the M&E results, and

(8) Development Partner agencies are to design, conduct and report M&E on their own

I-22
 assistance policy, programs and projects. Nepali counterpart agencies are to be
involved in the process.

Box-8: The System to Monitor the MDGs in Nepal - PMAS and DPMAS

The Government of Nepal is committed to the monitoring of the MDGs through

institutionalizing the Poverty Monitoring and Analysis System (PMAS), a framework
developed to monitor and evaluate progress in the implementation of the Poverty
Reduction Strategy Paper (PRSP). PMAS seeks to monitor the progress of PRSP and
communicate results effectively through:
 Implementation (or input/output) monitoring;
 Outcome or well-being monitoring;
 Impact assessment;
 Poverty Management Information System (PMIS), and
 Communication/Advocacy
Similarly, district-level monitoring system is called District Poverty Monitoring and
Analysis System (DPMAS), which aims to establish a district poverty monitoring and
analysis system for measuring district level poverty monitoring indicators, and monitor
resources, processes, outputs, outcomes and impacts of project and programs. The main
activities include: identify poverty monitoring indicators; collect data and information;
process and analyze data and information; analyze and present/disseminate quality
information, and institutionalize (and internalize) the organizational framework.

I-23
CONGRATURATIONS!! THIS IS THE END OF THE MODULE 1.
Please check whether or not you understand the contents of this module by
answering the following checklist.

CHECKLIST
 What is the difference between monitoring and evaluation?
 What types of evaluation are there in terms of stages of project cycle?
 What types of evaluation are there in terms of level?

 What is Managing for Development Results?

 What is Results-based Management?
 What is the significance of results-based monitoring and evaluation?
 For what purposes are evaluation results used?

 To whom the evaluation results are provided?

 What kind of media can be used to deliver the evaluation results to its audience?
 Who are the major M&E stakeholders in Nepal? What are they responsible for?

I-24
Recommended Readings:

 Annette Binnendijk, “Results Based Management in the Development

Co-operation Agencies: A review of Experience Background Report”, DAC
Working Party on Aid Evaluation, 2001

 DFID, “Essential Guide to Roles & Tools”, 2005

 GTZ, “Results-based Monitoring: Guidelines for Technical Cooperation Projects

and Programs”, 2004

 JICA, Textbook for the Area Focused Training Course in Forum on

Institutionalization of Evaluation System, 2007

 Jody Zall Kusek, Ray C. Rist, World Bank, “10 Steps for Results-Based M&E”,
2004

 Keith Mackay, Independent Evaluation Group, World Bank, “How to Build M&E
Systems to Support Better Government”, 2007

 OECD-DAC Joint Venture on Management for Development Results,

“Management for Development Results, Principle in Action: Source Book on
Emerging Good Practices”, 2006

 World Bank, “Monitoring and Evaluation: Some Tools, Methods and

Approaches”, 2004

I-25
Useful Website of International Development Partners
Agency Website
JICA Evaluation: http://www.jica.go.jp/english/operations/evaluation/
USAID 1) Evaluation: http://evalweb.usaid.gov/index.cfm
2) Guidelines (ADS) : http://evalweb.usaid.gov/adsreqs/adsrequirements.cfm
DFID Evaluation: http://www.dfid.gov.uk/aboutdfid/evaluation.asp
GTZ /KfW 1) GTZ (Evaluation): http://www.gtz.de/en/leistungsangebote/6332.htm
2) KfW (Evaluation):
http://www.kfw-entwicklungsbank.de/EN_Home/Ex-post_Evaluation_at_KfW/index.jsp
AFD Annual Report: http://www.afd.fr/jahia/Jahia/lang/en/pid/18
SIDA 1) Evaluation: http://www.sida.se/sida/jsp/sida.jsp?d=1509&language=en_US
CIDA 1) Evaluation: http://www.acdi-cida.gc.ca/CIDAWEB/acdicida.nsf/En/JUD-111795644-KJX
2) Annual Report: http://www.tbs-sct.gc.ca/dpr-rmr/2006-2007/index-eng.asp
NORAD 1) Evaluation: http://www.norad.no/default.asp?V_ITEM_ID=3507
2) Annual Report 2004-2005:
http://www.norad.no/items/3875/38/7079169984/ENGELSK_aarsrapport_2004-2005.pdf
DANIDA Evaluation: http://www.um.dk/en/menu/DevelopmentPolicy/Evaluations/
AusAID 1) Aid Effectiveness: http://www.ausaid.gov.au/ode/default.cfm
2) Annual Reports:
http://www.ausaid.gov.au/publications/pubout.cfm?ID=8691_5877_871_8496_1205
World Bank 1) Evaluation: http://www.worldbank.org/oed/
2) M&E Manual:
http://wbln0018.worldbank.org/Institutional/Manuals/OpManual.nsf/05TOCpages/Management
3) Project Cycle:
http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/0,,contentMDK:20120731~menu
PK:41390~pagePK:41367~piPK:51533~theSitePK:40941,00.html
4) Measuring Results:
http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/0,,contentMDK:20120723~menu
PK:41393~pagePK:41367~piPK:51533~theSitePK:40941,00.html
5) Annual Report on Operations Evaluation:
http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTANNREPOPEEVA/0,,content
MDK:21115852~menuPK:3073384~pagePK:64168427~piPK:64168435~theSitePK:3073310,0
0.html
UNDP 1) Evaluation: http://www.undp.org/eo/
2) Guidelines: http://www.undp.org/eo/methodologies.htm
Asian 1) Evaluation: http://www.adb.org/Evaluation/
Development 2) Evaluation Reports: http://www.adb.org/Evaluation/reports.asp
Bank

I-26