Selecting Performance Indicators: About Tips
Selecting Performance Indicators: About Tips
TIPS
SELECTING PERFORMANCE INDICATORS
ABOUT TIPS
These TIPS provide practical advice and suggestions to USAID managers on issues related to
performance monitoring and evaluation. This publication is a supplemental reference to the
Automated Directive System (ADS) Chapter 203.
1
WHY ARE FOR WHAT RESULTS (M&E) plan. The M&E plan
should be integrated in project
PERFORMANCE ARE PERFORMANCE management and reporting
INDICATORS INDICATORS systems (e.g., quarterly, semi-
annual, or annual reports).
IMPORTANT? REQUIRED?
Performance indicators provide THE PROGRAM LEVEL TYPES OF
objective evidence that an USAID’s ADS requires that at INDICATORS IN
intended change is occurring. least one indicator be chosen
Performance indicators lie at for each result in the Results
USAID SYSTEMS
the heart of developing an Framework in order to measure Several different types of
effective performance progress (see ADS 203.3.3.1)2. indicators are used in USAID
management system – they This includes the Assistance systems. It is important to
define the data to be collected Objective (the highest-level understand the different roles
and enable actual results objective in the Results and functions of these
achieved to be compared with Framework) as well as indicators so that managers can
planned results over time. supporting Intermediate Results construct a performance
Hence, they are an (IRs)3. These indicators should management system that
indispensable management tool be included in the Mission or effectively meets internal
for making evidence-based Office Performance management and Agency
decisions about program Management Plan (PMP) (see reporting needs.
strategies and activities. TIPS 8: Preparing a PMP).
Performance indicators can also CUSTOM INDICATORS
be used: PROJECT LEVEL Custom Indicators are
AO teams are required to performance indicators that
To assist managers in collect data regularly for reflect progress within each
focusing on the projects and activities, including unique country or program
achievement of inputs, outputs, and processes, context. While they are useful
development results. to ensure they are progressing for managers on the ground,
To provide objective as expected and are they often cannot be
evidence that results are contributing to relevant IRs and aggregated across a number of
being achieved. AOs. These indicators should programs like standard
To orient and motivate staff be included in a project-level indicators.
and partners toward monitoring and evaluation
achieving results. 2
Example: Progress on a
For further discussion of AOs and IRs
To communicate USAID (which are also termed impact and milestone scale reflecting
achievements to host outcomes respectively in other legal reform and
country counterparts, other systems) refer to TIPS 13: Building a implementation to ensure
Results Framework.
partners, and customers. 3
Note that some results frameworks credible elections, as follows:
To more effectively report incorporate IRs from other partners if
results achieved to USAID's those results are important for USAID Draft law is developed in
to achieve the AO. This is discussed in consultation with non-
stakeholders, including the further detail in TIPS 13: Building a
U.S. Congress, Office of Results Framework. If these IRs are governmental
Management and Budget, included, then it is recommended that organizations (NGOs) and
they be monitored, although less political parties.
and citizens. rigorous standards apply.
Public input is elicited.
2
PARTICIPATION IS ESSENTIAL designed to meet different INDICATORS AND DATA—SO
Experience suggests that needs. There is often a tension WHAT’S THE DIFFERENCE?
participatory approaches are an between measuring a standard Indicators define the particular
essential aspect of developing and across many programs and characteristic or dimension that will
maintaining effective performance be used to measure change. Height
selecting indicators that best
management systems. Collaboration is an example of an indicator.
with development partners reflect true program results and
The data are the actual
(including host country institutions, that can be used for internal measurements or factual information
civil society organizations (CSOs), management purposes. that result from the indicator. Five
and implementing partners) as well feet seven inches is an example of
as customers has important benefits. Example: Number of Laws or data.
It allows you to draw on the Amendments to Ensure
experience of others, obtains buy-in
Credible Elections Adopted They do not represent program
to achieving results and meeting
targets, and provides an opportunity with USG Technical performance, because the
to ensure that systems are as Assistance. indicator measures very high-
streamlined and practical as possible. level change.
In comparing the standard
Draft law is modified based indicator above with the Example: Score on the
on feedback. previous example of a custom Freedom House Index or
The secretariat presents indicator, it becomes clear that Gross Domestic Product
the draft to the Assembly. the custom indictor is more (GDP).
The law is passed by the likely to be useful as a
This sort of indicator may be
Assembly. management tool, because it
important to track to
The appropriate provides greater specificity and
understand the context for
government body is more sensitive to change.
USAID programming (e.g. a
completes internal policies
Standard indicators also tend to severe drop in GDP is likely to
or regulations to
measure change at the output affect economic growth
implement the law.
level, because they are precisely programming), but represents a
The example above would differ the types of measures that are, level of change that is outside
for each country depending on at face value, more easily the manageable interest of
its unique process for legal aggregated across many program managers. In most
reform. programs, as the following cases, it would be difficult to
example demonstrates. say that USAID programming
STANDARD INDICATORS
has affected the overall level of
Standard indicators are used Example: The number of freedom within a country or
primarily for Agency reporting people trained in policy and GDP (given the size of most
purposes. Standard indicators regulatory practices. USAID programs in comparison
produce data that can be
to the host country economy,
aggregated across many CONTEXTUAL INDICATORS
for example).
programs. Optimally, standard Contextual indicators are used
indicators meet both Agency to understand the broader
reporting and on-the-ground environment in which a
management needs. However, program operates, to track
in many cases, standard assumptions, or to examine
indicators do not substitute for externalities that may affect
performance (or custom success, failure, or progress.
indicators) because they are
3
WHAT ARE USAID’S Two overarching factors Indirect Indicator: Passage
determine the extent to which of the Freedom of
CRITERIA FOR performance indicators function Information Act (FOIA)
SELECTING as useful tools for managers
Direct Indicator: Progress
and decision-makers:
INDICATORS? on a milestone scale
The degree to which demonstrating enactment
USAID policies (ADS 203.3.4.2) performance indicators and enforcement of policies
identify seven key criteria to accurately reflect the that require open hearings
guide the selection of process or phenomenon
performance indicators: they are being used to The passage of FOIA, while an
measure. important step, does not
Direct actually measure whether a
Objective The level of comparability of target institution is more
Useful for Management performance indicators over transparent. The better
Attributable time: that is, can we example outlined above is a
Practical measure results in a more direct measure.
Adequate consistent and comparable
Disaggregated, as necessary manner over time? Level
Another dimension of whether
These criteria are designed to 1. DIRECT an indicator is direct relates to
assist managers in selecting An indicator is direct to the whether it measures the right
optimal indicators. The extent extent that it clearly measures level of the objective. A
to which performance the intended result. This common problem is that there
indicators meet each of the criterion is, in many ways, the is often a mismatch between
criteria must be consistent with most important. While this may the stated result and the
the requirements of good appear to be a simple concept, indicator. The indicator should
management. As managers it is one of the more common not measure a higher or lower
consider these criteria, they problems with indicators. level than the result.
should use a healthy measure Indicators should either be
of common sense and widely accepted for use by For example, if a program
reasonableness. While we specialists in a subject area, measures improved
always want the ―best‖ exhibit readily understandable management practices through
indicators, there are inevitably face validity (i.e., be intuitively the real value of agricultural
trade-offs among various understandable), or be production, the indicator is
criteria. For example, data for supported by research. measuring a higher-level effect
the most direct or objective Managers should place greater than is stated (see Figure 1).
indicators of a given result confidence in indicators that are Understanding levels is rooted
might be very expensive to direct. Consider the following in understanding the
collect or might be available example: development hypothesis
too infrequently. Table 1 inherent in the Results
includes a summary checklist Result: Increased Framework (see TIPS 13:
that can be used during the Transparency of Key Public Building a Results Framework).
selection process to assess Sector Institutions Tracking indicators at each level
these trade-offs. facilitates better understanding
and analysis of whether the
4
Figure 1. Levels relationship between the data are being collected. In
RESULT INDICATOR indicator and the result should other words, two people should
be well-understood and clearly be able to collect performance
Increased Real value of
Production agricultural articulated. The more information for the same
production. assumptions the indicator is indicator and come to the same
based upon, the weaker the conclusion. Objectivity is
Improved Number and indicator. Consider the critical to collecting comparable
Management percent of
Practices farmers using a following examples: data over time, yet it is one of
new technology. the most common problems
Result: Increased Household noted in audits. As a result,
Improved Number and Income pay particular attention to the
Knowledge percent of
definition of the indicator to
and farmers who can Proxy Indicator: Dollar
Awareness identify five out ensure that each term is clearly
of eight steps value of household defined, as the following
for expenditures examples demonstrate:
implementing a
new technology. The proxy indicator above Poor Indicator: Number of
makes the assumption that an successful firms
increase in income will result in
increased household Objective Indicator:
development hypothesis is
expenditures; this assumption is Number of firms with an
working. For example, if
well-grounded in research. annual increase in revenues
farmers are aware of how to
of at least 5%
implement a new technology, Result: Increased Access to
but the number or percent that Justice The better example outlines the
actually use the technology is exact criteria for how
not increasing, there may be Proxy Indicator: Number of ―successful‖ is defined and
other issues that need to be new courts opened ensures that changes in the
addressed. Perhaps the data are not attributable to
The indicator above is based on
technology is not readily differences in what is being
the assumption that physical
available in the community, or counted.
access to new courts is the
there is not enough access to
fundamental development Objectivity can be particularly
credit. This flags the issue for
problem—as opposed to challenging when constructing
managers and provides an
corruption, the costs associated qualitative indicators. Good
opportunity to make
with using the court system, or qualitative indicators permit
programmatic adjustments.
lack of knowledge of how to regular, systematic judgment
Proxy Indicators obtain legal assistance and/or about progress and reduce
Proxy indicators are linked to use court systems. Proxies can subjectivity (to the extent
the result by one or more be used when assumptions are possible). This means that
assumptions. They are often clear and when there is research there must be clear criteria or
used when the most direct to support that assumption. protocols for data collection.
indicator is not practical (e.g.,
2. OBJECTIVE 3. USEFUL FOR
data collection is too costly or
An indicator is objective if it is MANAGEMENT
the program is being
unambiguous about 1) what is An indicator is useful to the
implemented in a conflict zone).
being measured and 2) what extent that it provides a
When proxies are used, the
5
meaningful measure of change laws passed to promote invested in the process. In this
over time for management direct investment. case, the more useful example
decision-making. One aspect of better articulates the important
usefulness is to ensure that the More Useful for steps that must occur for a law
indicator is measuring the ―right Management: Progress to be passed and implemented
change‖ in order to achieve toward targeted legal reform and facilitates management
development results. For based on the following decision-making. If there is a
example, the number of stages: problem in meeting interim
meetings between Civil Society Stage 1. Interested groups milestones, then corrections
Organizations (CSOs) and propose that legislation is can be made along the way.
government is something that needed on issue.
can be counted but does not 4. ATTRIBUTABLE
Stage 2. Issue is introduced
necessarily reflect meaningful An indicator is attributable if it
in the relevant legislative
change. By selecting indicators, can be plausibly associated with
committee/executive
managers are defining program USAID interventions. The
ministry.
success in concrete ways. concept of ―plausible
Stage 3. Legislation is
Managers will focus on association‖ has been used in
drafted by relevant
achieving targets for those USAID for some time. It does
committee or executive
indicators, so it is important to not mean that X input equals Y
ministry.
consider the intended and output. Rather, it is based on
Stage 4. Legislation is
unintended incentives that the idea that a case can be
debated by the legislature.
performance indicators create. made to other development
Stage 5. Legislation is
As a result, the system may practitioners that the program
passed by full approval
need to be fine-tuned to ensure has materially affected
process needed in legislature.
that incentives are focused on identified change. It is
Stage 6. Legislation is
achieving true results. important to consider the logic
approved by the executive
behind what is proposed to
branch (where necessary).
A second dimension is whether ensure attribution. If a Mission
Stage 7. Implementing
the indictor measures a rate of is piloting a project in three
actions are taken.
change that is useful for schools, but claims national
Stage 8. No immediate need
management purposes. This level impact in school
identified for amendments to
means that the indicator is completion, this would not pass
the law.
constructed so that change can the common sense test.
be monitored at a rate that The less useful example may be Consider the following
facilitates management actions useful for reporting; however, it examples:
(such as corrections and is so general that it does not
improvements). Consider the Result: Improved Budgeting
provide a good way to track
following examples: Capacity
progress for performance
management. The process of Less Attributable: Budget
Result: Targeted legal
passing or implementing laws is allocation for the Ministry of
reform to promote
a long-term one, so that over Justice (MOJ)
investment
the course of a year or two the
Less Useful for AO team may only be able to More Attributable: The
Management: Number of report that one or two such extent to which the budget
laws have passed when, in produced by the MOJ meets
reality, a high degree of effort is
6
established criteria for good reliable data are available each level of resources available.
budgeting year, but the data are a year Too many indicators create
old, then it may be problematic. information overload and
If the program works with the become overly burdensome to
Ministry of Justice to improve Cost maintain. Too few indicators
budgeting capacity (by Performance indicators should are also problematic, because
providing technical assistance provide data to managers at a the data may only provide a
on budget analysis), the quality cost that is reasonable and partial or misleading picture of
of the budget submitted by the appropriate as compared with performance. The following
MOJ may improve. However, it the management utility of the demonstrates how one
is often difficult to attribute data. As a very general rule of indicator can be adequate to
changes in the overall budget thumb, it is suggested that measure the stated objective:
allocation to USAID between 5% and 10% of
interventions, because there are program or project resources Result: Increased Traditional
a number of externalities that be allocated for monitoring and Exports in Targeted Sectors
affect a country’s final budget – evaluation (M&E) purposes.
much like in the U.S. For Adequate Indicator: Value
However, it is also important to of traditional exports in
example, in tough economic consider priorities and program
times, the budget for all targeted sectors
context. A program would
government institutions may likely be willing to invest more In contrast, an objective
decrease. A crisis may emerge resources in measuring changes focusing on improved maternal
that requires the host country that are central to decision- health may require two or three
to reallocate resources. The making and less resources in indicators to be adequate. A
better example above is more measuring more tangential general rule of thumb is to
attributable (and directly linked) results. A more mature select between two and three
to USAID’s intervention. program may have to invest performance indicators per
5. PRACTICAL more in demonstrating higher- result. If many more indicators
A practical indicator is one for level changes or impacts as are needed to adequately cover
which data can be collected on a compared to a new program. the result, then it may signify
timely basis and at a reasonable that the objective is not
6. ADEQUATE properly focused.
cost. There are two dimensions Taken as a group, the indicator
that determine whether an (or set of indicators) should be 7. DISAGGREGATED, AS
indicator is practical. The first is sufficient to measure the stated NECESSARY
time and the second is cost. result. In other words, they The disaggregation of data by
should be the minimum gender, age, location, or some
Time
number necessary and cost- other dimension is often
Consider whether resulting data effective for performance important from both a
are available with enough management. The number of management and reporting
frequency for management indicators required to point of view. Development
purposes (i.e., timely enough to adequately measure a result programs often affect
correspond to USAID depends on 1) the complexity population cohorts or
performance management and of the result being measured, 2) institutions in different ways.
reporting purposes). Second, the amount of information For example, it might be
examine whether data are needed to make reasonably important to know to what
current when available. If confident decisions, and 3) the extent youth (up to age 25) or
7
adults (25 and older) are following describes a series of Build capacity for
participating in vocational steps to select optimal performance management
training, or in which districts performance indicators4. among partners, such as
schools have improved. Although presented as discrete NGOs and partner country
Disaggregated data help track steps, in practice some of these institutions.
whether or not specific groups can be effectively undertaken Ensure that systems are as
participate in and benefit from simultaneously or in a more practical and streamlined as
activities intended to include iterative manner. These steps possible. Often
them. may be applied as a part of a development partners can
larger process to develop a new provide excellent insight on
In particular, USAID policies PMP, or in part, when teams the practical issues
(ADS 203.3.4.3) require that have to modify individual associated with indicators
performance management indicators. and data collection.
systems and evaluations at the
AO and project or activity levels STEP 1. DEVELOP A A common way to begin the
include gender-sensitive PARTICIPATORY PROCESS process is to hold working
indicators and sex- FOR IDENTIFYING sessions. Start by reviewing the
disaggregated data if the PERFORMANCE INDICATORS Results Framework. Next,
activities or their anticipated The most effective way to identify indicators for the
results involve or affect women identify indicators is to set up a Assistance Objective, then
and men differently. If so, this process that elicits the move down to the Intermediate
difference would be an participation and feedback of a Results. In some cases, the AO
important factor in managing number of partners and team establishes the first round
for sustainable program impact. stakeholders. This allows of indicators and then provides
Consider the following example: managers to: them to other partners for
input. In other cases, key
Result: Increased Access to Draw on different areas of partners may be included in the
Credit expertise. working sessions.
Ensure that indicators
Gender-Sensitive Indicator: measure the right changes It is important to task the group
Value of loans disbursed, and represent part of a with identifying the set of
disaggregated by larger approach to achieve minimal indicators necessary
male/female. development impact. and sufficient to manage the
Build commitment and program effectively. That is, the
WHAT IS THE understanding of the group must go through a
PROCESS FOR linkage between indicators process of prioritization in order
and results. This will to narrow down the list. While
SELECTING increase the utility of the participatory processes may
PERFORMANCE performance management take more time at the front end,
system among key they almost always result in
INDICATORS? stakeholders. more coherent and effective
system.
Selecting appropriate and 4 This process focuses on presenting
useful performance indicators greater detail related specifically to
indicator selection. Refer to TIPS 7: STEP 2. CLARIFY THE RESULT
requires careful thought, Preparing a PMP for a broader set of Carefully define the result
iterative refining, collaboration, steps on how to develop a full PMP.
desired. Good performance
and consensus-building. The
8
indicators are based on clearly Be clear about what type of the full Assistance Objective
articulated and focused change is implied. What is Team. Ask, ―how will we
objectives. Review the precise expected to change—a know if the result is
wording and intention of the situation, a condition, the level achieved?‖
objective. Determine what of knowledge, an attitude, or a
exactly is meant by the result. behavior? For example, Consider other resources.
For example, if the result is changing a country's voting Many organizations have
―improved business law(s) is very different from databases or indicator lists
environment,‖ what does that changing citizens' awareness of for various sectors available
mean? What specific aspects of their right to vote (which is on the internet.
the business environment will different from voting). Each Consult with technical
be improved? Optimally, the type of change is measured by experts.
result should be stated with as different types of performance
much specificity as possible. If indicators. Review the PMPs and
the result is broad (and the indicators of previous
team doesn’t have the latitude Identify more precisely the programs or similar
to change the objective), then specific targets for change. Who programs in other Missions.
the team might further define or what are the specific targets
its meaning. for the change? For example, if STEP 4. ASSESS THE BEST
individuals, which individuals? CANDIDATE INDICATORS,
Example: One AO team For an economic growth USING THE INDICATOR
further defined their IR, program designed to increase CRITERIA
―Improved Business exports, does the program Next, from the initial list, select
Environment,‖ as follows: target all exporters or only the best candidates as
exporters of non-traditional indicators. The seven basic
Making it easier to do agricultural products? This is criteria that can be used to
business in terms of resolving known as identifying the ―unit judge an indicator’s
disputes, obtaining licenses of analysis‖ for the performance appropriateness and utility
from the government, and indicator. described in the previous
promoting investment. section are summarized in
STEP 3: IDENTIFY POSSIBLE Table 1. When assessing and
An identified set of key INDICATORS
policies are in place to comparing possible indicators,
Usually there are many possible it is helpful to use this type of
support investment. Key indicators for a particular result,
policies include laws, checklist to guide the
but some are more appropriate assessment process.
regulations, and policies and useful than others. In
related to the simplification of Remember that there will be
selecting indicators, don’t settle trade-offs between the criteria.
investment procedures, too quickly on the first ideas
bankruptcy, and starting a For example, the optimal
that come most conveniently or indicator may not be the most
business. obviously to mind. Create an cost-effective to select.
As the team gains greater initial list of possible indicators,
clarity and consensus on what using the following approaches: STEP 5. SELECT THE “BEST”
results are sought, ideas for PERFORMANCE INDICATORS
Conduct a brainstorming Select the best indicators to
potential indicators begin to session with colleagues to
emerge. incorporate in the performance
draw upon the expertise of management system. They
9
should be the optimum set of Ensure that the rationale for underlying problems. If a large
measures that are useful to indicator selection is recorded in number of indicators are
management and can be the PMP. There are rarely frequently changed, this may
obtained at reasonable cost. perfect indicators in the signify a problem with program
development environment—it management or focus. At the
Be Strategic and Streamline is more often a case of other end of the continuum, if
Where Possible. In recent years, weighing different criteria and no indicators were to change
there has been a substantial making the optimal choices for over a long period of time, it is
increase in the number of a particular program. It is possible that a program is not
indicators used to monitor and important to ensure that the adapting and evolving as
track programs. It is important rationale behind these choices necessary. In our experience,
to remember that there are is recorded in the PMP so that some refinements are inevitable
costs, in terms of time and new staff, implementers, or as data are collected and
money, to collect data for each auditors understand why each lessons learned. After some
indicator. AO teams should: indicator was selected. rounds of data collection are
Select indicators based on completed, it is often useful to
STEP 6. FINE TUNE WHEN discuss indicator issues and
strategic thinking about NECESSARY
what must truly be achieved refinements among AO team
Indicators are part of a larger members and/or with partners
for program success. system that is ultimately and implementers. In
Review indicators to designed to assist managers in particular, the period following
determine whether any final achieving development impact. portfolio reviews is a good time
narrowing can be done. Are On the one hand, indicators to refine PMPs if necessary.
some indicators not useful? must remain comparable over
If so, discard them. time but, on the other hand,
some refinements will invariably
Use participatory be needed to ensure the system
approaches in order to is as effective as possible. (Of
discuss and establish course, there is no value in
priorities that help continuing to collect bad data,
managers focus on key for example.) As a result, these
indicators that are necessary two issues need to be balanced.
and sufficient. Remember that indicator issues
are often flags for other
10
TABLE 1. INDICATOR SELECTION CRITERIA CHECKLIST
11
For more information:
TIPS publications are available online at [insert website].
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication, including Gerry Britan
and Subhi Mehdi of USAID’s Office of Management Policy, Budget and Performance (MPBP). This
publication was updated by Michelle Adams-Matson of Management Systems International.
12