UNIT 8 PROJECT EVALUATION

Contents
8.0 Objectives
8.1 Introduction
8.2 Meaning and Objectives of Evaluation
8.3 Dimensions
8.4 Techniques and Criteria
8.5 Let Us Sum Up
8.6 Key words
8.7 Further Readings

8.0 OBJECTIVES
This unit aims at examining the concept and techniques of evaluating a
development project. After you have studied the unit, you should be able to:
• explain the concept of evaluation;
• state the objectives of evaluation;
• give its main dimensions; and
• indicate the techniques and criteria.

8.1 INTRODUCTION
Evaluation is an important function of modern management system. It tells us
about the effectiveness of the programme in achieving the stated objectives.
You may recall from the study of preceding units that evaluation would also
necessitate assessing the time and effort involved in reaching the final objective.
It needs to be clarified here that evaluation is different from appraisal with
respect to the time when it is carried out. As explained earlier, an appraisal is
an ex ante evaluation when the project details have been estimated for the
future. An evaluation is basically an ex post function when the project has
been implemented and events/activities concerned have occurred.

8.2 MEANING AND OBJECTIVES OF

EVALUATION
Since many developing nations seek to achieve growth with social justice
through planning, the evaluation of development policies and programmes
assumes importance in this context for the following reasons:
– The social and economic structures are so complex that the planning
process must take these into consideration and formulate strategies and
programmes on the basis of certain assumptions. Evaluation will enable
the policy makers to know to what extent the planning process was
responsive to development needs. It has been noticed, for instance, that 89
Rural Development some of the development programmes failed owing to wrong identification
Management
of the problems, incorrect assumptions, improper formulation or inefficient
administering of programmes.
– The resources at the command are limited. Evaluation will enable
allocation of funds to programmes. Selection of programmes will be guided
by efficiency and cost effective considerations.
– Evaluation aids decision makers to assess the various programmes in terms
of their relevance (to overall development goals), efficiency, effectiveness
and impact. This helps in better formulation and effective implementation
of programmes in the future.
The evaluation process is expected to make recommendations on the need for
modification of an existing programme, charting of a different course of action,
or formulation of new programmes. Thus, evaluation, in a way, helps to assess
the long-term implications of the present decisions and programme choices.
More specifically, the objectives of evaluation are:
– Measurement of the progress in both quantitative and qualitative aspects
in relation to the stated objectives, targets and course of action;
– Assessment of efficiency of the process of conversion of inputs into outputs
and the efficiency of the management apparatus;
– Identification of factors responsible for speeding up or slowing down of
the process; in other words, identification of reasons for a project’s success
or failure; and
– Providing feedback to decision makers, which will enable modifications,
where necessary, in the policy, strategy and objectives of the programme
and the operational aspects.
Check Your Progress I
Notes: a) Write your answers in the space provided.
b) Compare your answers with the text.
1) What is the main difference between project monitoring and project
evaluation?
.................................................................................................................
.................................................................................................................
.................................................................................................................
.................................................................................................................

2) Why is evaluation considered an essential tool of development

administration?
.................................................................................................................
.................................................................................................................
.................................................................................................................
.................................................................................................................
90 .................................................................................................................
 Project Evaluation
8.3 DIMENSIONS
Evaluation is now considered to be an integral part of the overall planning
system and a main ingredient of development administration. In simple terms,
evaluation is a performance or achievement audit, which assesses the whole
gamut of activities associated with programme formulation, implementation
and impact, and compares the observed results with those anticipated. In other
words, evaluation critically examines:
• Were the assumptions of the programme/project design valid?
• Were the programme objectives linked with specific schemes to achieve
them?
• Were the objectives quantified in the form of targets?
• Did the implementation adhere to the prescribed design?
• Were the critical inputs delivered in time and in right quantity and quality
by the agencies concerned?
• Was the manpower delivering the services trained?
• Was the scheduled path followed? If not, what were the reasons?
• Are the benefits commensurate with the costs?
• Has the intended impact occurred?
• Have the effects been beneficial or harmful?
It is obvious that answers to these questions indicate neither success nor failure
of the project, but certainly point out the problems, which affect its
implementation.
Concerns of Evaluation
Let us now discuss some of the concerns of evaluation.
Assessment of Outcome
Evaluation is concerned with assessment of outcome. This is after the
completion of the programme. Some of the issues considered in outcome
evaluation are:
• Whether the beneficiaries have realized the expected benefits.
• Whether beneficiaries are better off compared to non-beneficiaries on the
relevant indicators.
• To what extent the programme/project is responsible for the observed
benefits (both intended and unintended) and what have been the costs?
• What are the changes noticed in the behavioural dimensions of the target
population?
• Have there been any adverse, unintended effects of the project?
• Have any forces been set in motion, the dynamics of which may have far
reaching consequences?
The discussion of causation is one of the major aspects of an evaluation. The
91
isolation of the programme effects from those of other programmes is the real
Rural Development task involved in this exercise and demands a high level of expertise in the
Management
evaluation design.
Measurement of Project Efficiency
Sometimes, a major concern of evaluation is the measurement of project
efficiency.
Two yardsticks used to measure project efficiency are:
– Cost efficiency expressed in terms of outcome per unit of cost; and
– Project effectiveness measured in terms of outcome per unit of effort.
For example, if two projects A and B are equally effective, but A is cost efficient,
then the decision rule is to prefer A. But when A is more effective, but not cost
efficient, then the criterion for project selection is whether the additional benefit
is worth the additional cost. Another situation, which makes evaluation task
rather difficult is when projects have several characteristics. Each one brings
improvements in certain characteristics. The assignment of weight on an
objective basis to the progress made becomes the main focus. Arriving at a
composite measure for judging relative effectiveness and efficiency of the
individual is a major problem.
Purpose Oriented Evaluation
Sometimes, evaluation is purpose oriented. In this type of evaluation, the focus
is on the basic purpose for which the projects have been launched. For example,
the thrust of rural development programmes is to improve the quality of life in
rural areas. There may be several sub-purposes, such as provision of basic
amenities to rural population. Two important types of evaluation in this category
are:
i) Formative evaluation and
ii) Summative evaluation
Formative evaluation is basically concerned with the formulation of projects.
It is service oriented and aids planners through identification of potential
problem areas where project needs improvement, etc. In other words, the
purpose is to improve the planning design with a view to enhance the flow of
benefits from the project or to increase the efficiency of the delivery system.
Summative evaluation aims at selection of projects from a set of projects for
deciding on the continuance or termination of the project. In this sense, there
is a finality to summative evaluation, since the evaluation report may
recommend the withdrawal of the project. In brief, it provides a summary
statement about the effectiveness of the programme – incorporating the project’s
achievement of pronounced goals, unanticipated outcomes and comparison
with alternative projects.
Prerequisites for Evaluation
For a successful evaluation study, certain basic requirements have to be met.
A few are given below.
i) One of the foremost requirements is clear specification of the objectives of
92 the project and its likely impact. Vagueness with respect to objectives makes
the task of evaluation difficult, as evaluation seeks to assess the extent of Project Evaluation
realization of the objectives. There may exist a hierarchy of objectives, such
as immediate, intermediate, final and ultimate objectives. A project description
in terms of this hierarchy not only facilitates better formulation, but also helps
visualizing a programme/project as a sequence of interrelated activities. The
corresponding items in the project cycle are – inputs, outputs, effects and impact.
The immediate objective may be provision of inputs of right quality in adequate
quantity in time to the programme clients. The ‘inputs’ may as well include
services. For example, in the agricultural extension programme, training of
farmers by the extension worker is an input.
The intermediate objective could be generation of initial results (outputs) of
certain order as a sequel to the utilization of inputs. Adoption of improved
practices of agriculture by (contact) farmers is the immediate objective for the
agricultural extension programme.
The final objective is the same as the project objective and is usually referred
to as effects. The increase in yield (or incremental yield) is the effect measured
in quantitative terms.
The ultimate objective relates to sustenance of the effect of the project and,
thus, is concerned with the impact of the programme. For example,
modernization of agriculture could be an ultimate objective of agricultural
programmes.
ii) Another prerequisite for evaluation is the establishment of norms/targets
for comparison of results. In this context, it is useful to have bench-mark data
or base line information relating to prelaunching (programme) period. Also
relevant are a set of indicators for measuring the extent of progress and, thus,
determining the success or failure of the programme/project. Needless to say,
the selected indicators should relate to the various objectives of the project
and/or the various dimensions of these objective(s).
iii) Equally important is the selection of techniques for data collection and
analysis to make the findings of the study more meaningful. You will read
these in Course 5.
Check Your Progress II
Notes: a) Write your answer in the space provided.
b) Compare your answer with the text.
Why is clear specification of objectives of a project an important prerequisite
for evaluation?
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
.......................................................................................................................... 93
Rural Development
Management 8.4 TECHNIQUES AND CRITERIA
You will find a wide range of evaluation designs – from simple to very complex
and sophisticated ones. The application of various techniques depends on
several factors, such as time, cost, nature/type of questions the evaluation seeks
to answer, and the expertise available. Use of scientific method is desirable
since it involves careful observation and it also controls the subjective bias of
evaluators. The validity of inferences and findings of the study is greater if
techniques are scientific. These methods ensure:
– accurate and reliable data base, as the tools used for collection of data and
also the data collection procedures and sampling design are efficient, and
– the analytical techniques used help to disaggregate the whole complex
phenomenon, and, thus, assess the programme effects in an unbiased and
objective manner.
Criteria for Evaluation
The following five criteria are usually adopted in evaluation studies:
– efforts
– performance
– adequacy
– efficiency
– process
Efforts: In a summarized view, efforts refer to the total inputs to the programme.
For example, a criterion relating to the amount of effort put into the programme
would include quantity and quality of programme inputs (range of services),
the coverage of population, programme personnel, financial resources, etc.
Performance: This deals with the consequences of the programme inputs.
The composite index of performance should consider the magnitude and quality
of benefits and changes in behavioural dimensions. Thus, it refers to output
generated from the system as a result of programme inputs.
Adequacy: This is a relative measure indicating the relationship of the effort
and performance of the programme to the level of need for the programme. In
other words, it is the extent of successful (or effective) coverage of beneficiary
population in the total population. It, thus, refers to the relationship between
output and the total need.
Efficiency: This measure connects the above three criteria, viz. effort,
performance and adequacy and is based on the minimax principle, i.e.,
minimization of effort and maximization of performance. In other words, it is
simple input-output ratio. Naturally, the various inputs, such as money, time
and staff are being considered in evaluating the various alternatives.
Process: It is the study of the means of the programme in producing outputs
(results) with a view to establishing causality.

94
Indicators of Impact Project Evaluation

Indicators are simple measures reflecting a problem or changes in a situation.

Indicators are used as instruments for evaluating changes resulting from the
various activities/efforts of the programme. Impact indicators are designed to
show the degree of achievement of programme objectives. For example, the
percentage of beneficiary households crossing the poverty line once for all is
one of the (impact) indicators for the success of IRDP.
i) Characteristics of Indicators: In order to be effective, measures for assessing
impact/effect indicators should possess the following characteristics:
Validity: What is measured is the same as what is intended to be measured.
For instance, marks obtained by a student is a valid measure of scholastic
achievement. It is, however, not a valid measure of his behaviour in school.
Reliability: Same values should be obtained even if the evaluation/data
collection is made by others under similar conditions. In other words, repeated
collection of data should yield the same values.
Relevance: Indicators selected should be related to programme objectives.
Sensitivity: Indicators should be able to capture even small changes taking
place in important aspects.
Objectivity: Indicators should be free from subjective judgments of the
evaluator.
Simplicity: Measurement should not be a costly or time consuming affair.
From the above discussion, it is clear that indicators selected should relate to
the major dimensions of the programme. For instance, in case of service-
oriented programmes, the coverage (physical), range of service and quality of
service(s) rendered should be reflected by the selected indicators.
In case the goals are many and are heterogenous in nature, separate indicators
are to be considered. These indicators should be pivotal/key indicators and aid
the programme authorities to take remedial measures wherever the programme
path deviates from the planned trajectory. This also reduces the workload of
officials in data collection.
ii) Differential Impact: Indicators of impact, which are generally based on
time series data, reflect direction, speed and extent of change. The interregional
and intergroup comparisons help assess the differential impact of the
programme. These indicators may be direct or indirect in nature. The former
are amenable to quantification and are more easily measurable. The indirect
indicators are those which are neither quantifiable nor cost-effective. In such
cases, proxy indicators (partial measures) are usually employed. For example,
an assessment of the nutritional status of children utilizes the ‘age-weight’
indicator as a proxy indicator. Similarly, partial measures are considered to
capture behavioural changes.
iii) Impact Indicators of Programmes: The comparison of indicator values–
before and after programme of ‘with and without’ programme – will provide
an estimate of the magnitude of achievement. The indicators considered in
this context, for instance, are: impact of the programme on household income, 95
Rural Development use of incremental incomes (comsumption, savings and investment) and the
Management
living standards.
A few impact indicators are suggested for some programmes as an illustration.
Agricultural Programmes: Incremental yields, economic returns, employment
generated.
Health and Nutrition Programmes: Morbidity rate, death rates, infant mortality
rate, birth weight, weight for age, etc.
These indicators can be collected for different social groups to know the trends
in the incidence and prevalence among these groups.
One of the more common evaluation methodologies for measuring impact is
the ‘before and after’. In this, base line data (relevant to the change being
sought through the project) is collected before a project is launched. After the
project has been completed (or some phase of it), the data are again collected.
The difference is, then, interpreted in terms of project objectives, operations
and impact.
Another commonly used evaluation methodology is the experimental group–
control group design (i.e. with and without programme). Two population groups
are selected and matched. In one, the project is implemented (the experimental
group). In the other, no such project is under implementation (the control group).
The two are, then, compared on relevant indicators, and the difference is
interpreted in terms of project objectives, operations and impact.
iv) Miscellaneous: For any programme, the following four interrelated indices
of impact are suggested:
Goal Effectiveness Index (GEI): Indicates the extent of achievement of goals/
objectives, i.e., comparison of programme result with the objectives. It answers
the question whether the programme accomplished its objectives or not.

Programme Result
GEI =
Programme Task Objective
Potency Effectiveness Index (PEI): It points out the achievement as a
proportion of the total need. For instance, in the case of IRDP, it refers to the
number of beneficiaries who have crossed the poverty line to the number of
poor who need such programme assistance.

Programme Result
PEI =
Estimated size of the potential client needed for the r
Realised Efficiency Index (REI): It compares the result with the total effort
directed towards the programme and it is similar to the output-input ratio.

Programme Result
REI =
Total Programme Efforts
Capacity Efficiency Index (CEI): Compares programme outcome with
potential capacity of the programme in rendering its services as defined by its
physical, human and financial resources. How much can the programme
96 accomplish with the resources as its command is the issue under consideration.
 Programme Result Project Evaluation
CEI =
Established Potential Capacity of Programme
Concurrent and Terminal Evaluation
You may have come across the terms concurrent evaluation and terminal
evaluation. Concurrent evaluation is an ongoing activity and is carried out
when the programme is under implementation. It is usually undertaken at regular
intervals or at the end of a phase, to give an objective feedback on the progress
and direction of the programme. Concurrent evaluation does not provide a
feedback on impact, unless it is at the end of a phase when some impact was
visualized in the project design. The utility of concurrent evaluation is that it
can assist decision makers in modifying the objectives (if necessary), and change
the implementation strategy, so that overall objectives of the project can be
achieved. Concurrent evaluation also permits examination of the relevance or
validity of the assumptions sufficiently before the terminal stage.
Thus, the main purposes of concurrent evaluation are:
• to have a feedback on the need for modifying the project in relation to the
changes in the project environment or in the development goals; and
• to identify the problems affecting the programme success with a view to
provide corrective action.
Concurrent evaluation is an internal function of project managers who are also
the users of the information provided by the evaluation. Concurrent evaluation
can, thus, be viewed as a built-in ‘learning dimension’ of the programme/
project.
Terminal evaluation is done at the end of programme implementation and
focusses on the impact and other complex issues of programme performance.
Check Your Progress III
Notes: a) Write your answers in the space provided.
b) Compare your answers with the text.
1) What is the difference between validity and reliability?
...............................................................................................................
...............................................................................................................
...............................................................................................................
...............................................................................................................
...............................................................................................................

2) Is concurrent evaluation more important than terminal evaluation?

...............................................................................................................
...............................................................................................................
...............................................................................................................
...............................................................................................................
............................................................................................................... 97
Rural Development Activity I
Management
1) Prepare a list of ten indicators for evaluating the performance of IRDP.
Against each indicator, comment on the different characteristics, which
an indicator should ideally satisfy (i.e., validity, eligibility, etc.).
...................................................................................................................
...................................................................................................................
...................................................................................................................
...................................................................................................................
...................................................................................................................

2) Get hold of some evaluation studies and read about the evaluation design
and the concerns.
...................................................................................................................
...................................................................................................................
...................................................................................................................
...................................................................................................................
...................................................................................................................

8.5 LET US SUM UP

In this unit, we read about the role of evaluation and saw how it was an integral
part of the overall planning and implementation system. Evaluation provides a
feedback to the policy makers and enables them to judge the effectiveness of
strategies, thus, helping them to modify or work out alternative strategies and
programmes.
Evaluation enables not only measurement of progress in qualitative and
quantitative terms, but also assesses the impact and efficiency of the
implementation machinery and the identification of factors, which obstruct
development. We noted that evaluation has several dimensions viz., need,
process, outcome, efficiency and purpose. We briefly discussed each of these.
We stated that some basic requirements have to be met for a proper evaluation
of a programme. These were: clear specification of objectives, establishment
of norms and targets against which to measure progress, and selection of tools
on data collection. We saw that the criteria adopted in evaluation studies usually
covered efforts, performance, adequacy, efficiency and process. We observed
that the indicators to be used for measuring impact should meet the criteria of
validity, reliability, relevance, objectivity and simplicity.
While evaluation is often terminal, in recent years stress is being laid on
concurrent evaluation, which is an ongoing activity carried out during the
implementation of the project. The utility of concurrent evaluation is that it
enables decision makers to modify the project (if necessary), alter the
implementation strategy, examine the relevance or validity of the assumptions
relating to the programme, and identify the problems affecting the success of
the project with a view to provide corrective action.
98
 Project Evaluation
8.6 KEY WORDS
Concurrent : Ongoing, continuous.
Ex-ante : Before the event
Ex-post : After the event
Reliability : Repeat collection of data by the same person or another
person yields the same values.
Validity : Actually measures what is intended to be measured.

8.7 SUGGESTED READINGS

Baum, Warren C. and Stokes M. Tolbert (1985), Investing in Development:
Lessons of World Bank Experience, Oxford University Press, New York.
Casley, D.J. and D.A. Lury (1982), Monitoring and Evaluation of Agriculture
and Rural Development Projects, Johns Hopkins, Baltimore.
Clayton, E.S. (1983), Agriculture, Poverty and Freedom, Macmillan, London.

99