MODULE 20

Monitoring and evaluation

PART 3: TRAINER’S GUIDE

The trainer’s guide is the third of four parts contained in this module. It is NOT a training course. This guide provides guidance
on how to design a training course by giving tips and examples of tools that the trainer can use and adapt to meet training
needs. The trainer’s guide should only be used by experienced trainers to help develop a training course that meets the needs
of a specific audience. The trainer’s guide is linked to the technical information found in Part 2 of the module.

Module 20 is about monitoring and evaluation (M&E) of nutrition interventions in emergencies. M&E is an important part of any
intervention, however, in emergencies this area is often neglected due to pressure on resources, time and personnel. In order
to learn from past interventions and to improve the impact for people affected by humanitarian disasters, simple but effective
M&E is essential. The trainer’s guide provides exercises to test the knowledge of trainees around the subject matter and poses
some likely situations that humanitarian workers may face. The examples given are taken or adapted from real situations. The
guide does not aim to provide an exhaustive training for M&E in emergencies but to raise pertinent issues around this complex
subject.

Navigating your way round the guide

The trainer’s guide is divided into seven sections:
1. Tips for trainers provide pointers on how to prepare for and organise a training course.
2. Training exercises from other key modules provide examples of related exercises that can be done in a classroom
context by participants individually or in groups.
3. Learning objectives et out examples of learning objectives for this module that can be adapted for a particular
participant group.
4. Testing knowledge contains an example of a questionnaire that can be used to test participants’ knowledge either
at the start or at the end of a training course.
5. Classroom exercises provide examples of practical exercises that can be done in a classroom context by participants
individually or in groups.
6. Case studies contain examples of case studies (one from Africa and one from another continent) that can be used to get
participants to think by using real-life scenarios.
7. Field-based exercises outline ideas for field visits that may be conducted during a longer training course.

HTP, Version 2, 2011 1

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

CONTENTS

1. Tips for trainers

2. Learning objectives

3. Training exercises from other key modules

4. Testing knowledge
Exercise 1: What do you know about monitoring and evaluation?
Handout 1a: What do you know about monitoring and evaluation?: Questionnaire
Handout 1b: What do you know about monitoring and evaluation?: Questionnaire answers

5. Classroom exercises
Exercise 2: Monitoring of CMAM programmes for the management of severe acute malnutrition
Handout 2a: Monitoring system for a CMAM programme: Part A
Handout 2b: Monitoring system for a CMAM programme: Part B
Handout 2c: Facilitator’s guide to Handouts 2a and 2b
Exercise 3: Evaluation of a supplementary feeding programme
Handout 3a: WFP targeted supplementary feeding in Ethiopia
Handout 3b: Facilitator’s guide to Handout 3a

6. Case studies
Exercise 4: Key issues for evaluations in emergencies
Handout 4a: Case study I: Real time evaluation of Pakistan flood response
Handout 4b: Case study I: Real time evaluation in Pakistan: Instructions for group work
Handout 4c: Facilitator’s guide to Handouts 4a and 4b
Exercise 5: Evaluation of WFP relief operations in Angola
Exercise 6: Evaluation of WFP response to 2004 tsunami
Handout 5a: Case study II: Evaluation of WFP relief operations in Angola: How to best use recommendations
Handout 5b: Facilitator’s guide to Handout 5a
Handout 6a: Case study III: WFP response to 2004 tsunami: Group work
Handout 6b: Facilitator’s guide to Handout 6a

7. Field-based exercises
Exercise 7: Assessment of the monitoring system of a nutrition intervention under field conditions
Handout 7a: Participants’ guide to field work exercise
Handout 7b: Facilitator’s guide to field work exercise

2 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

1. Tips for trainers

Step 1: Do the reading!

• Read Parts 1 and 2 of this module.
• Familiarise yourself with the technical terms from the glossary.
• Read through the following key documents and consult key websites (see full references and how to access them
in Part 4 of this module):
Websites
° ALNAP, http://www.alnap.org.
° Emergency Nutrition Network, Field Exchange, www.ennonline.net.
° Food and Nutrition Technical Assistance Project (FANTA), www.fantaproject.org.
° Humanitarian Reform, www.humanitarianreform.org.
° International Red Cross and the International Federation of the Red Cross and Crescent, www.ircr.org and www.ifrc.org.
° ODI, Humanitarian Practice Network, www.odi.org.uk.
° Relief Web, www.reliefweb.org.
° Sphere Project, www.sphereproject.org
° UNICEF, www.unicef.org;
° WHO, www.who.org;
° UNHCR, www.unhcr.org;
° FAO, www.fao.org.
° WFP, www.wfp.org
Key documents
° ALNAP (2006) Evaluating Humanitarian Action using the DAC criteria
° ECHO (2007) Evaluation of Humanitarian Aid by and for NGOs
° FANTA. (2006). Assessing Nutrition Situations in Emergencies. Washington: FANTA, www.fantaproject.og/bookmarks/
index.shtml.
° FAO (2005). Protecting and Promoting Good Nutrition in Crisis and Recovery Rome: FAO, www.fao.org;
email: [email protected].
° IFE Core Group. (2007). Infant and Young Child Feeding in Emergencies: Operational Guidance for Emergency Relief Staff
and Programme Managers Version 2.1. Oxford: IFE Core Group, www.ennonline.net/pool/files/ife/ops-guidance.
° Oxfam (2001). Oxfam’s Approach to Nutrition Surveys in Emergencies. Oxford: Oxfam, www.oxfam.org or SCUK. (2004).
Emergency Nutrition Assessment: Guidelines for Field Workers. ISBN 1 84187 090 0, www.savethechildren.org.uk, or WFP.
(2005). Food and Nutrition Handbook. Rome: WFP, www.wfp.org.
° The Sphere Project. (2011). Sphere Handbook: Humanitarian Charter and Minimum Standards in Disaster Response.
www.sphereproject.org.
° Valid International. (2006). Community-based Therapeutic Care: A Field Manual. Oxford: Valid International.
http://www.validinternational.org/demo/ctc/reports.php.
° WHO. (2000). Management of the child with a serious infection or malnutrition. Geneva: WHO.
www.who.int/nutrition/publications/malnutrition/en/.

HTP, Version 2, 2011 3

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Step 2: Know your audience!

• Find out about your participants in advance of the training:

° How many participants will there be?

° Do any of the participants already have experience of nutrition in emergencies?

° Could participants with experience be involved in the sessions by preparing a case study or contribute through
describing their practical experience?

Step 3: Design the training!

• Decide how long the training will be and what activities can be covered within the available time . In general,
the following can be guide can be used:

° A 90-minute classroom-based training can provide a basic introduction.

° A half-day classroom-based training can provide an overview and include a practical exercise.

° A one-day classroom-based training can provide a more in-depth understanding of nutrition in emergencies and
include a number of practical exercises and/or one case study.
• Identify appropriate learning objectives. This will depend on your participants, their level of understanding and
experience, and the aim and length of the training.
• Decide exactly which technical points to cover based on the learning objectives that you have identified.
• Divide the training into manageable sections. One session should generally not last longer than an hour.
• Ensure the training is a good combination of activities, e.g., mix PowerPoint presentations in plenary with more active
participation through classroom-based exercises, mix individual work with group work.

Step 4: Get prepared!

• Prepare PowerPoint presentations with notes (if they are going to be used) in advance and do a trial run. Time yourself!
• Prepare exercises and case studies. These can be based on the examples given in this trainer’s guide but should be
adapted to be suitable for the particular training context.
• Find the appropriate equipment for the session such as calculators.
• Prepare a ‘kit’ of materials for each participant. These should be given out at the start of the training and should include:

° Timetable showing break times (coffee and lunch) and individual sessions

° Parts 1 and 2 of this module

° Pens and paper

REMEMBER
People remember 20 per cent of what they are told, 40 per cent of what they are told and read, and 80 per cent of what
they find out for themselves.
People learn differently. They learn from what they read, what they hear, what they see, what they discuss with others
and what they explain to others. A good training is therefore one that offers a variety of learning methods which suit the
variety of individuals in any group. Such variety will also help reinforce messages and ideas so that they are more likely
to be learned.

4 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

2. Learning objectives

Below are examples of learning objectives for a session on M&E. Trainers may wish to develop alternative learning objectives
that are appropriate to their particular participant group. The number of learning objectives should be limited; up to five per
day of training is appropriate. Each exercise should be related to at least one of the learning objectives.

Examples of learning objectives

At the end of the training, participants will:
• Understand the basic concepts of M&E.
• Be able to describe the main characteristics of monitoring systems.
• Be able to describe key evaluation parameters and the importance of each of the parameters.
• Be aware of the importance of M&E for nutrition interventions in emergencies.
• Be able to design simple M&E systems for key nutrition interventions in emergencies.
• Have enough knowledge to use the findings of both monitoring systems and evaluation surveys.
• Be aware of the present gaps in practice in terms of M&E of nutrition interventions in emergencies.

HTP, Version 2, 2011 5

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

3. Training exercises from other key modules

The following section identifies relevant exercises for M&E found in other modules in this series.

When planning a training session it is important to be clear about the objectives and the level of practical knowledge trainees
will require in order to work more effectively. For example, if the training course is largely for practitioners who will be expected
to carry out anthropometric assessments in the field to study the impact of a supplementary feeding programme, concepts of
M&E can be drawn from Module 20, and used in conjunction with the practical training on supplementary feeding found in
Module 12.

If however the training course is largely for people who will be ùconsumersû of M&E information, the emphasis should be on
understanding basic concepts of M&E, the challenges of M&E in humanitarian disasters, and the practical use of the results of
M&E systems. In this case it is not necessary for the trainees to understand all of the potential M&E instruments for all of the
possible nutrition interventions, but rather understand the importance of the results produced.

Exercises for carrying out specific assessments and monitoring of nutrition interventions can be found in Part 3 of the following
modules. In addition, Part 4 of the all modules has extensive resource lists which can be referenced when designing specific
M&E systems for nutrition interventions in emergencies.

Module 6: Measuring malnutrition: individual assessment

Exercise 4: Calculating anthropometric indices and classifying anthropometric data is an important classroom exercise
for practitioners that are planning to carry out an anthropometric assessment.

Module 7: Measuring malnutrition: population assessment.

Exercise 1: What do you know about nutrition surveys? Case studies + advanced case studies for practitioners:
Questions for discussion and development of survey methodologies, including, survey planning, sampling frameworks,
logistics and implementation.

Module 9: Food security assessment and the link with nutrition

Classroom-based exercises
Exercise 4: Collecting 24-hour food frequency data
Exercise 5: Linking food security and nutrition data
Field-based exercises
Exercise 6: Testing different interview skills used in food security assessments

Module 11: General Food Distribution

Field-based exercise
Exercise 7: Evaluation of a general food distribution
Module 12: Management of moderate acute malnutrition
Classroom-based exercises
Exercise 3: Assessing SFP performance in relation to minimum standards
Case studies
Exercise 5: Evaluating a targeted SFP in Ethiopia
Field-based exercises
Exercise 7: Evaluation of SFP based on: analysis of monthly reporting forms, focus group discussions with beneficiaries and key
informant interviews with implementing staff

6 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Module 13: Management of severe acute malnutrition

This module provides tools for monitoring of interventions for the management of SAM that can be adapted to specific
circumstances (See Part 2 and Part 3 of the module.).
Classroom-based exercises
Exercise 4: Data collection and analysis: consolidating monthly reports
Field-based exercises
Exercise 7: Field activities in an outpatient care facility
Exercise 8: Field activities in an inpatient care facility

Module 18: HIV and AIDS nutrition

Field-based exercises
Exercise 8: Checklists for HIV and AIDS nutrition programming in emergencies

HTP, Version 2, 2011 7

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

4. Testing knowledge

When the facilitator has decided on the main purpose of the training session, PowerPoint presentations of the relevant sections
of Module 20 and the other modules referenced above should be prepared. The facilitator should make the presentations as
relevant as possible, bringing in examples from real situations and providing space for questions and answers.

This section contains a series of questions that can be used to ensure that all participants in the training session have gained
basic knowledge of M&E for nutrition interventions in emergencies.

The questions below should serve as a guide to a facilitator for the preparation of the classroom-based training course. Power-
Points/flip chart/handouts should be prepared summarising key points from Part 2 to enable participants to answer the questions
in Handout 1a.

Exercise 1: What do you know about monitoring and evaluation?

What is the aim?

• The aim is to ensure that basic knowledge of main issues around M&E are understood by practitioners
who will be working on nutrition in an emergency

When should this exercise be done?

• The exercise can be used to pre-test knowledge before a training session
if practitioners are experienced emergency workers
• The exercise can be used as a post training exercise

How long should the exercise take?

• 45 minutes to fill in 5-10 minutes to correct each test
(tests can be corrected in pairs by participants to save time if necessary)
• 20 minutes for discussion of results

What materials are needed?

• Handout 1a: Pre and/or post test sheets (answers in bold)

What does the trainer need to prepare?

• Adapt questions to training to be carried out, depending on the material taught during the course
• Photocopy the tests
• Photocopy the answers

Instructions
Step 1: Administer the test to practitioners, giving them 45 minutes.
Step 2: Ask the practitioners to exchange test sheets.
Step 3: Distribute answer sheets and ask practitioners to correct their neighbours tests
Step 4: Discuss problem areas (for pre-test this should guide the training – post training should be used
for additional session for clarification)

8 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 1a: What do you know about M&E and nutrition?: Questionnaire
Time for completion: 45 minutes

1. True or false?
a) A good monitoring system can provide information to improve the efficiency of an intervention.
b) Monitoring systems can help to identify aspects of the intervention that require programmatic attention.
c) The findings from a baseline study and final evaluation constitute an adequate monitoring system.
d) Monitoring systems are principally concerned with measuring the impact of interventions.
e) Regular collection of process indicators (to measure the functionality of an intervention),
are generally part of monitoring systems.
f ) Monitoring systems can be used to track resource use.

2. What is the value of improving participation of the affected community in M&E and how can participation be increased?

3. What are five commonly made mistakes in developing M&E systems?

4. The five basic evaluation parameters that are important when assessing nutrition interventions in emergencies are:
Effectiveness; Efficiency; Relevance (Appropriateness); Impact; Coverage. Provide a brief definition of the terms.
Effectiveness

Efficiency

Relevance

Impact

Coverage

5. What are the additional parameters for evaluation recommended by OECD- DAC when carrying out inter-agency
evaluations of large humanitarian disasters?

HTP, Version 2, 2011 9

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

6. What do you understand by ‘real time’ evaluation? And why is this concept important in emergencies?

7. Describe the principle elements of a programme that require monitoring during general food distribution (GFD).

8. What would an evaluation of GFD programmes need to assess in order to understand

whether the programmes had the expected impact?

9. If there were enough resources available for a complete evaluation, including the impact on micronutrient
deficiencies of a population dependant on distributed food rations (e.g., in an isolated refugee camp),
which micronutrients would be the most important to assess?

10. Against what indicators can programmes of the management of SAM be monitored and evaluated?
At what level does Sphere set the cut off for successful programmes?

11. In addition to the quantitative indicators identified in question 10, what additional qualitative information is useful
in monitoring the overall quality of programmes for the management of SAM?

12. What are the Sphere standards for supplementary feeding programmes?

13. To monitor the M&E of health interventions in emergencies, information is needed on which 3 areas:

14. Mention at least six diseases that are routinely monitored in emergency settings and have a potential impact
on nutrition status.

15. Which of the United Nations agencies and international relief agencies provide information about disease surveillance
in emergencies?

10 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

16. Explain why monitoring violations of the International Code of Marketing of Breastmilk Substitutes is important
in emergencies?

17. List 5 outcome indicators useful for monitoring the impact of interventions on IYCF practices:

18. When establishing M&E systems in areas of high HIV prevalence what additional factors should be taken into
consideration in the design of the systems in order to ensure adequate assessment of the intervention?

19. What are the most common proxy indicators used to assess the prevalence of HIV in an area? And why might they
be used instead of direct indicators?

20. Identify five major challenges for M&E of nutrition interventions in emergencies, and explain why these challenges
should be addressed.

HTP, Version 2, 2011 11

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 1b: What do you know about M&E and nutrition?: Questionnaire answers
1. True or false?
a) A good monitoring system can provide information to improve the efficiency of an intervention. True
b) Monitoring systems can help to identify aspects of the intervention that require programmatic attention. True
c) The findings from a baseline study and final evaluation constitute an adequate monitoring system. False
d) Monitoring systems are principally concerned with measuring the impact of interventions. False
e) Regular collection of process indicators (to measure the functionality of an intervention),
are generally part of monitoring systems. True
f ) Monitoring systems can be used to track resource use. True

2. What is the value of improving participation of the affected community in M&E and how can participation be increased?
Involving community members in M&E places the affected population at the heart of the response, providing
the opportunity for their views and perceptions to be incorporated into programme decisions and increases
accountability towards them. Moving to a more participatory approach to M&E requires greater involvement of
community members at all steps of the project cycle. Community members can become involved in the initial
design of the intervention, in collecting and analysing data, through adopting more qualitative approaches
to data collection and finally through ensuring findings are shared back and linked to action. Qualitative
approaches to M&E are of particular value, allowing voices to be captured and community members to tell
their story in a culturally appropriate and non-threatening way.

3. What are five commonly made mistakes in developing M&E systems?

• Trying to answer too many questions that sound interesting but do not contribute to ensuring programme
effectiveness
• Creating a framework that looks perfect in theory but too complicated in practice.
• Monitoring systems are often made too complex for community level workers to understand and implement
compromising the quality of the data collected.
• Over importance may be given to the reporting requirements of donors resulting in the neglect of
downward accountability to the beneficiaries.
• Over emphasis on the collection of input and process data (usually quantitative) providing information on
what is happening, with less focus on qualitative data to understand ‘why’ and ‘how’ it is happening.
• Limited attention is paid to the dissemination, feedback and utilisation of results.

4. The five basic evaluation parameters that are important when assessing nutrition interventions in emergencies are:
Effectiveness; Efficiency; Relevance (Appropriateness); Impact; Coverage. Provide a brief definition of the terms.
Effectiveness: Achieving objectives – doing the right thing at the right time, includes cost-effectiveness
Efficiency: Doing it right, making maximum use of resources: effort, time, money, people, material
Relevance/Appropriateness: Doing the right thing in the right way in relation to local context,
needs and priorities thereby increasing ownership and accountability
Impact: Contributing to changing the situation more profoundly and in the longer-term,
positively or negatively, intended or unintended
Coverage: Who has been reached by the intervention, where and why or why not? It is linked to effectiveness

12 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

5. What are the additional evaluation parameters recommended by OECD – DAC?

• Connectedness. Activities of a short term emergency nature are carried out in a context that takes longer
term and interconnected problems into account.
• Coherence. The extent to which all relevant policies (security, developmental, trade and military policies
as well as humanitarian) are consistent and take adequate account of humanitarian and human rights
considerations. Can also refer to the extent to which all humanitarian actors are working towards
the same goals.
• Timeliness. In emergencies there are different responses at different points in the emergency.
These are generally expressed as; (immediate) response and rescue; recovery; rehabilitation.
Interventions should be tailored to the phases.
• Coordination. All humanitarian actors and agencies need to coordinate activities to provide a complete
service to the affected community, avoid duplication of efforts, wasted resources or patchy coverage

6. What do you understand by ‘real time’ evaluation? And why is this concept important in emergencies?
A real time evaluation is one carried out whilst the programme is in full implementation, usually two to three
months after the onset of the emergency. They are meant to provide quick and practical feedback to the
programme for immediate use. Evaluators should be experienced and are either internal but not directly
involved with or responsible for the programme, or external with good familiarity with the agency’s work.
The approach is largely participatory with rapid dissemination of findings and recommendations, either in
verbal or written form and typically prior to leaving the field.

7. Describe the principle elements of a programme that require monitoring during GFD.
• Pipeline management (how much food is needed, how much is available and timing of
the arrival of food supplies)
• Food management (storage, warehousing, logistics, transport, etc.)
• Number and identification of beneficiaries (numbers of people in need, registration, ration criteria,
exit and entry criteria)
• Management of food distribution (modalities – wet or dry rations, frequency, location

8. What would an evaluation of GFD programmes need to assess in order to understand

whether the programmes had the expected impact?
• Changes in household food security
• Changes in nutrition status of beneficiaries (and non-beneficiaries, if possible)
• Mortality rates (over time)
• Micronutrient deficiency (particularly important for populations that are dependent on food rations)

9. If there were enough resources available for a complete evaluation, including the impact on micronutrient
deficiencies of a population dependant on distributed food rations (e.g., in an isolated refugee camp),
which micronutrients would be the most important to assess?
Vitamin A, iron (anaemia) and iodine (goitre), thiamine (beriberi), niacin (pellagra) and vitamin C (scurvy)

HTP, Version 2, 2011 13

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

10. Against what indicators can programmes of the management of SAM be monitored and evaluated? At what level does
Sphere set the cut off for successful programmes?
Distance from programme site: More than 90 per cent of the target population is within less than one day’s
return walk (including time for treatment) of the programme site.
Coverage: in rural areas is >50 per cent;
in urban areas: >70 per cent
in camp situations: >90 per cent
The proportion of discharges from therapeutic care who have:
i) died is <10 per cent,
ii) recovered is >75 per cent and
iii) defaulted is <15 per cent

11. In addition to the quantitative indicators identified in question 10, what additional qualitative information is useful
in monitoring the overall quality of programmes for the management of SAM?
Community awareness and understanding of the programme, community participation in the programme and
the programme’s acceptability

12. What are the Sphere standards for supplementary feeding programmes?
• 75 per cent of children who exit from an SFP should have ‘recovered’.
• Coverage of targeted supplementary feeding programs should be greater than 50 per cent in rural areas,
greater than 70 per cent in urban areas and greater than 90 per cent in camp situations.

13. To monitor the M&E of health interventions in emergencies, information is needed on which 3 areas:
• Health status and risks of the affected population
• Health resources availability (including services)
• Health system performance

14. Mention at least six diseases that are routinely monitored in emergency settings and have a potential impact
on nutrition status.
• Measles
• Diarrhoeal diseases, including cholera, dysentery
• Skin diseases, including scabies
• Upper respiratory tract infections (especially in children under five years old)
• Meningitis
• Typhoid
• Malaria

15. Which of the United Nations agencies and international relief agencies provide information about disease surveillance
in emergencies?
WHO, UNICEF, UNHCR, International Red Cross, International Federation of Red Cross and Crescent Societies

14 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

16. Explain why monitoring violations of the International Code of Marketing of Breast-milk Substitutes and subsequent
relevant WHA Resolutions, is important in emergencies?
Donations of milk powder and their inappropriate handling exposes both breastfed and non-breastfed infants
to risk. For artificial feeding to be ‘safe’ in emergencies it does not just require milk powder, it requires needs
assessment, planning, targeting, education, and monitoring by technical personnel with medical backup, as well
as provision of additional resources, e.g., clean water, fuel and cooking equipment. In past emergencies,
precious time, energy and resources were spent trying to sort out problems that have arisen with donations of
breast milk substitutes.

17. List 5 outcome indicators useful for monitoring the impact of interventions on IYCF practices:
• % infants aged less than 6 months who are fed exclusively with breast milk during the emergency response
(using standard methodology) and compared to pre-emergency rate.
• % infants put to the breast within 1 hour of birth during the emergency and compared to pre-emergency rate.
• % infants between 6-8 months with complementary foods introduced
• % young children still breastfeeding at 12 to 15 months and at 20 to 23 months
• % infants/young children aged 6-23m with minimum meal frequency1
• % infants/young children aged 6-23m with minimum dietary diversity2
• % of caregivers of infants/young children 0-23m who report receiving donations of BMS
• % of infants/young children aged 0-23m who are fed with a bottle (pre and post emergency)
• % infants/young children aged 6-23 months who receive an appropriate iron-rich food or iron-fortified food

18. When establishing M&E systems in areas of high HIV prevalence what additional factors should be taken into
consideration in the design of the systems in order to ensure adequate assessment of the intervention?
• Number of people living with HIV in the programme
• Number of people living with AIDS in the programme
• Feasibility of identifying people living with HIV and AIDS
• Possibility of using proxy indicators for HIV
• Measurement of additional resources required (people, time, funds, medicines, etc.)
due to the high levels of HIV)
• Adjustments in ‘success levels’ of the intervention

19. What are the most common proxy indicators used to assess the prevalence of HIV in an area and why might they used
instead of direct indicators?
• Percentage of households with chronically sick adult members
• Percentage of households with death of an adult household member after a chronic illness
Reasons why they might be used:
• Lack of reliable baseline information
• Testing facilities may not be available
• Testing and identification may lead to stigma and discrimination
• Testing and identification may lead to conflict

1
See Annex in Module 17 IYCF for definition
2
See Annex in Module 17 IYCF for definition

HTP, Version 2, 2011 15

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

20. Identify five major challenges for M&E of nutrition interventions in emergencies, and explain why these challenges
should be addressed.
Technical issues
• Lack of standardisation of nutrition outcomes for the variety of interventions undertaken. There are
questions linked to who should be measured to assess positive or negative impacts. Generally children
under age five have been used as a ‘population measure’ for malnutrition, but increasingly practitioners
are calling for adult anthropometric assessments to measure the variable impact of food insecurity and
malnutrition on the whole population.
• Insufficient information exists on the relative cost of interventions versus the efficacy of those interventions.
This information is sensitive as no agency wants to be accused of using resources inefficiently, however,
without critical oversight it will not be possible to correct errors and improve practice.
• Poor baseline information (from population studies or community/clinic based sources)
makes measurement of change and impact extremely difficult.
• Lack of M&E frameworks that can cope with rapidly evolving situations in emergency situations
• Minimal assessments, monitoring systems and evaluations are carried out due to the cost of these activities
in terms of funds, resources and time. In emergencies all three of these are in short supply. This feeds into
the vicious cycle of not having information for learning, therefore, repeating the same mistakes that
may be costly in terms of both resources and lives.
Institutional issues
• No one agency has a mandate to use the information produced on nutrition interventions (collate, draw
lessons learned, assess cost effectiveness, etc.). Information is produced under the health, food security,
livelihoods and needs assessment umbrellas. Taking the information forward to intervention guidance/
policy, or modification of intervention does not fall directly under the mandate of any one agency
(compounding the technical issue of lack of standardisation).
• A problem common to all M&E in emergencies is the lack of sharing of external and internal reviews
to add to the evidence base and improve nutrition interventions in the future. Some of the more innovative
nutrition interventions have instituted rigorous M&E systems as part of an advocacy strategy. Examples
include community-based management of acute malnutrition and the emergence of cash transfers in
developing countries. As these were new interventions, the advocates were careful to gather evidence and
have succeeded in providing evidence of impact. However, this does not appear to have stimulated the
practitioners espousing more traditional responses to develop improved M&E frameworks that counter
balance the claims of the newcomers.

16 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

5. Classroom exercises

This section contains three exercises to be carried out in the classroom to deepen understanding of the key issues for M&E of
nutrition interventions in emergencies. Students will need to refer to additional modules to fully answer the exercises.

These exercises are not sufficient to train people to carry out evaluations or design monitoring systems, but provide material for
discussion around this complex issue.

Exercise 2: Monitoring community-based management of acute malnutrition (CMAM)

for the management of SAM

What is the learning objective?

• To ensure participants can design a simple monitoring system (using CMAM as an example) and reflect on the
importance of incorporating key contextual information in the design of systems (here the example used is high
HIV prevalence)

When should this exercise be done?

• After the teaching sessions on the subject

How long should the exercise take?

• Part A: 90 minutes
• Part B: 60 minutes

What materials are needed?

• Handout 2a: Monitoring system for a CMAM programme: Part A
• Handout 2b: Monitoring system for a CMAM programme: Part B
• Handout 2c: Facilitator’s guide to Handouts 2a and 2b
• Module 13
• Module 18

What does the trainer need to prepare?

• Review the aspects that need to be taken into consideration for each group in order to facilitate the group work.
• During the role play take notes to provide feedback in the plenary.

HTP, Version 2, 2011 17

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Exercise 2: Monitoring community-based management of acute malnutrition (CMAM)

for the management of SAM (continued)

Instructions
This exercise is divided into two parts. In Part A, participants will be requested to design a simple monitoring system for
a therapeutic care programme for SAM. This will then be discussed in plenary and completed. In Part B, participants are
told that the area they are working in has high levels of HIV prevalence in under-fives. They are then requested to modify
the monitoring system to reflect the situation.
Step 1: Divide participants into groups of five or six people.
Step 2: Distribute resource material.
Step 3: Each group will present their system on flip charts.
Step 4: The trainer will facilitate the plenary to discuss the differences/similarities of the systems designed.
Step 5: In plenary, design an ùidealû system taking into account the systems designed in each group.
Step 6: Provide the group with the new scenario and allow a further 30 minutes to modify the system.
Step 6: In plenary, discuss modifications and brainstorm other contextual issues that could affect the way
in which a monitoring system was designed.

Discussion points for plenary

Part A
In the plenary the following aspects should be covered:
➡ Key aspects of a monitoring system
• A clear objective must be set for the intervention. If possible this should be measurable (use the Sphere standards
as a guide).
• The system designed must be able to monitor the appropriateness of the programme, the effectiveness, the
coverage of the intervention.
• Discuss the types of indicators.
• Discuss the practicalities of the systems designed (some will be elaborate and require too many resources, other
will be too simplistic and not capture essential information
• Reach agreement on a ùworkableû system.
Part B
➡ Is there a need to change objectives because of the additional information on HIV prevalence?
➡ Would there be changes in the expected results of the intervention?
➡ If changes are proposed to the M&E system what additional indicators would be needed; discuss the need
for additional resources and changes to the periodicity of collection/analysis of data?
➡ Would there be any change in the use of the information?
➡ Remind participants of the importance of understanding the context of a situation before designing M&E systems.

18 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 2a: Monitoring system for a CMAM programme: Part A

Design a simple monitoring system for a CMAM programme targeting 100 severely malnourished patients in a refugee camp in
northern Thailand.

The system should include the following:

• Objectives of the monitoring system
• Results (outcomes) expected from the intervention (with special attention to appropriateness of the intervention,
effectiveness and coverage)
• Indicators to be used in the system
• Frequency of collection of the information
• Who will collect the information
• How the information will be used

Resource material
• Module 13 in this series
• Sphere guidelines

HTP, Version 2, 2011 19

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 2b: Monitoring system for a CMAM programme: Part B

This exercise builds on the previous exercise. The NGO running the CMAM programme has received information that the area
they are operating in has high levels of HIV infection among children under five years of age (estimates of 20 per cent). It is
therefore necessary to review the monitoring system established and:
1. Indicate any changes in objectives.
2. Indicate any changes in the expected results.
3. Are there any additional indicators?
4. Indicate any changes in the frequency of the collection of information.
5. Indicate any changes in human resources for the monitoring.
6. Would there be any change in the use of the information?

Resource material
• Module 14 and Module 18
• Sphere guidelines

20 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 2c: Facilitator’s guide to Handouts 2a and 2b

Provide the trainees with reference material – Module 13 for Part A, add Module 18 for Part B.

Photocopy examples of monitoring tools from Module 14 and the module’s resource list, including, instruments for monitoring
individual treatment [a. Record cards (Medical and nutrition data including follow-up data); b. Ration cards, (key information
about the child and basic information on their progress (weight, height, ration received)].

Handout 2a: Monitoring system for a CMAM programme: Part A

The system developed should have the following objectives and include the following indicators:

Objectives of the monitoring system

• To reduce the mortality rate of severely malnourished individuals within the entire affected population,
including children, adolescents, elderly and adults

Key indicators for management of severe acute malnutrition: Recovery rates for severe malnutrition for under-fives
• Recovery rate is >75 per cent
• The proportion of discharges from therapeutic care who have died is <10 per cent,
recovered is and defaulted is <15 per cent

Monitoring the appropriateness of the programme

i. Quantitative indicators
Mortality, default and cure rates
ii. Qualitative information (specific to community-based programmes)
Perceptions of the programme at community level in particular information about coverage, access to the programme,
recovery rates, service delivery, cultural appropriateness

Monitoring the effectiveness of the programme

Quantitative indicators (minimum information)
• Total admissions, exits and number of children in the programme
• Number of admissions by category of patient
• Number of exits by category
• Information on exits, weight gain and lengths of stay

Monitoring programme coverage

Coverage surveys to ascertain the proportion of children in need of assistance who receive care in the programme (expressed
as a percentage)

The Sphere indicator for coverage is >50 per cent in rural areas, >70 per cent in urban areas and >90 per cent in camp situations

HTP, Version 2, 2011 21

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 2b: Monitoring system for a CMAM programme: Part B

Possible answers to the questions:
1. Indicate any changes in objectives.
There may be no changes in the overall objective, but one would expect to see some mention of additional care
for children living with HIV and AIDS who are severely malnourished. The group may also want to acknowledge the
burden of care of the child for a mother that may also be HIV-positive, and the possible implications for the programme.
2. Indicate any changes in the expected results.
Reaching the expected standard for recovery of 90 per cent would still be the goal, however, the discussions should
reflect the complex treatment of children living with HIV and AIDS (e.g., availability of paediatric ART, specialised care
for opportunistic infections, availability of drugs) and the potential problems of identification of children suspected
to be HIV-positive when there are no diagnostic facilities. The group may decide that the expected results will be
disaggregated to reflect the success rate with HIV-positive children and non-HIV-positive children.
3. Are there any additional indicators?
Disaggregated data for children identified as HIV-positive (may also include positive mothers if acceptable and feasible).
Addition of indicators linked to common opportunistic infections (if not already included). Indicators linked to
the amount of resources used (time, drugs, food ect.) for child (and mother or carer).
4. Indicate any changes in the frequency of the collection of information.
This probably will not change.
5. Indicate any changes in human resources for the monitoring.
This should not change after the initial adaptation of the system to provide disaggregated information
on HIV and AIDS status.
6. Would there be any change in the use of the information?
Possible additional uses for the data:
• Data for national HIV and AIDS programme
• Advocacy for the provision of testing/treatment facilities for people living with HIV and AIDS
• If the system indicates that additional resources or more specialised resources are necessary to run the programme,
the data could be used for advocacy purposes.
• At present there is little information about the effect of high HIV on CMAM programmes a good monitoring system
would add evidence for future policy.

22 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Exercise 3: Evaluation of a supplementary feeding programme

What is the aim?

• To ensure that the participants are aware of the issues around the design of an evaluation of a supplementary
feeding programme.

When should this exercise be done?

• During or after the formal training sessions

How long should the exercise take?

• 120 minutes (60 minutes group work and 60 minutes plenary)

What materials are needed?

• Handout 3a: WFP targeted supplementary feeding in Ethiopia
• Handout 3b: Facilitator’s guide to Handout 3a
• Modules 12 in this series
• Sphere guidelines
• Module 20

What does the trainer need to prepare?

• The trainer would need to have experience in M&E.
• Review all material mentioned above.

Instructions
Step 1: Divide the participants in groups of five or six people.
Step 2: Distribute resource material.
Step 3: Allow 60 minutes for group discussion.
Step 4: Each group will prepare a flip chart with the main considerations. Allow 60 minutes for the plenary to discuss
the group work and reach a consensus about main aspects to consider in an evaluation of a supplementary
feeding programme.

Key discussion points

Based on the restrictions given to the teams, the facilitator should assess in plenary how realistic the plans of each group
are. The following aspects are key to the discussion:
➡ The methodology should take into consideration the objectives and the key evaluation parameters.
➡ The questionnaire should relate closely to the objectives of the exercise. Discuss the differences between the key
topics/discussion questions. Arrive at a consensus as to a set of questions that would provide information
on each of the evaluation parameters.
➡ Discuss challenges that an assessment team would face: Emphasise the facilitator and participants’ real
experiences when discussing the challenges. After the discussion revisit the conclusions from the previous
two discussion points and adjust, if necessary, taking into consideration the challenges identified.

HTP, Version 2, 2011 23

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 3a: WFP targeted supplementary feeding in Ethiopia

Evaluation summary3
In October 2004, the Executive Board of the United Nations World Food Programme (WFP) approved Protracted Relief and
Recovery Operation (PRRO) 10362.0 for Ethiopia. This intervention aimed to address the food needs of 3.8 million beneficiaries
(relief 1.7 million, recovery 2.1 million) between 1 Jan 2005 to 31 December 2007.

PRRO 10362.0 comprises four main programme components including targeted supplementary feeding (TSF) for vulnerable
children and women working within the Government of Ethiopia’s framework of enhanced outreach strategy (EOS).

The EOS/TSF programme delivers a combination of key child and maternal health interventions including vitamin A supple-
mentation, measles vaccination, provision of insecticide-treated bed nets and de-worming on a six-month basis. Screening of
pregnant women, women with infants under six months of age and children under five years of age4 using mid-upper arm cir-
cumference (MUAC)5 also takes place in conjunction with delivery of health inputs. Those women and children who are found
to have a MUAC below the cut-off point of 21.0 cm and 12.0 cm, respectively, are given a ration card and referred to the TSF
programme. Those with a MUAC below 11.0 cm and/or with oedema are referred for treatment of severe malnutrition where
available. The TSF beneficiaries receive two three-month food supplements that comprises 25 kg of micronutrient-fortified
corn or wheat soy blend (CSB/WSB) and 3 litres of fortified vegetable oil. This provides 1690 kcals, 55 g of protein and 15 g of fat
per day. At the end of six months, beneficiaries automatically leave the programme.

The MUAC screening and TSF referral occurs every six months at designated EOS sites while the TSF distribution takes place
every three months at TSF designated sites. The Federal MoH/UNICEF are responsible for the EOS component while the Disaster
Preparedness and Prevention Bureau (DPPB)/WFP are responsible for the TSF component.

The overall aim of the combined components of the EOS/TSF is to çreduce morbidity and mortality in children under fiveé. The
TSF objectives are nutritional and are:
• To prevent the nutritional deterioration of children under age five and pregnant and lactating women
• To prevent those moderately malnourished becoming severely malnourished
• To rehabilitate moderately malnourished children and pregnant and lactating women through the provision of fortified
supplementary food
• To promote key nutrition messages

Group work
Based on the above description of the targeted supplementary feeding programme and the enhanced outreach strategy in
Ethiopia, the group should plan a theoretical field assessment that will take place over a period of 15 days at five sites. For this
assessment groups should:
1. Describe the methodology they would use to measure the impact of the programme (taking into consideration the
objectives and the key evaluation parameters).
2. Design a questionnaire, set of key informant topics and focus group discussion questions.
3. Describe potential challenges that an assessment team would face.

3
Summary Evaluation Report Ethiopia PRRP 0362.0. 10, October 2007. Available at http://documents.wfp.org/stellent/groups/public/documents/eb/wfp137560.pdf%20
4
The screening actually includes older children who are stunted, as the entry to the EOS programme is based on a height less than 110.0 cm.
5
Up until March 2006, MUAC screening was followed by weight-for-height measurements, but this was stopped after agreement among all stakeholders to simplify
the system and use only MUAC as a good predictor of mortality risks.

24 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 3b: Facilitator’s guide to Handout 3a

There are no fixed answers to this exercise and the facilitator needs to be aware of the principles of M&E and have had field
experience in order to guide this session.

Objective of the targeted supplementary feeding (TSF) and the enhanced outreach strategy (EOS): “The overall aim of
the combined components of the EOS/TSF is to” reduce morbidity and mortality in children under five. “The TSF objectives are
nutritional and are as follows:”
• To prevent the nutritional deterioration of children under five and pregnant and lactating women
• To prevent those moderately malnourished becoming severely malnourished
• To rehabilitate moderately malnourished children and pregnant and lactating women through the provision of fortified
supplementary food
• To promote key nutrition messages

Evaluation parameter
• Effectiveness: achieving objectives – doing the thing right, including cost-effectiveness
• Efficiency: doing it right, with as few resources as possible; effort, time, money, people, material
• Relevance/Appropriateness: doing the right thing in the right way at the right time)
• Impact: doing the right thing, changing the situation more profoundly and in the longer-term
• Coverage: who has been reached by the intervention, and where: linked to effectiveness
• Timeliness: Was the intervention carried out in a timely fashion that saved lives and prevented malnutrition?
• Connectedness (and coordination): Was there any replication or gaps left in programming due to a lack of coordination?
• Coherence: Did the intervention make sense in the context of the emergency and the mandate of
the implementing agency?

Timeframe
• 15 days for the evaluation

Key activities to be evaluated

• Vitamin A supplementation
• Measles vaccination
• Provision of insecticide-treated bed nets and de-worming on a six-month basis
• Screening of pregnant women, women with infants under six months of age and children under five years of age6 using
mid-upper arm circumference (MUAC)7
• Women and children who are found to have a MUAC below the cut-off point of 21.0 cm and 12.0 cm are referred to the
TSF programme.
• Those with a MUAC below 11.0 cm and/or with oedema are referred for treatment of severe malnutrition where available.

Implementing partners
• Targeted supplementary feeding component: Disaster Preparedness and Prevention Bureau (DPPB)/WFP
• Enhanced outreach strategy: Ministry of Health/UNICEF

6
The screening actually includes older children who are stunted as the entry to the EOS programme is based on a height less than 110.0 cm.
7
Up until March 2006, MUAC screening was followed by weight for height measurements but this was stopped after agreement among all stakeholders to simplify
the system and use only MUAC as a good predictor of mortality risk.

HTP, Version 2, 2011 25

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

In order to guide participants, the following should be taken into consideration when discussing the results of the group work:
1. Describe the methodology they would use to measure the impact of the programme (taking into consideration
the objectives and the key evaluation parameters).
Should include:
• Geographical location
• Timeframe
• Resources necessary to carry out the assessment
• Sampling (it is not necessary to indicate sample size but rather what sort of sampling they would contemplate –
cluster survey, household interviews, random sample, etc.)
• Secondary data to be collected (from monitoring systems)
• Primary data collection (anthropometric measurements, focus groups, individual interviews, quantitative or
qualitative survey)
• Analysis of information
• Dissemination plan
2. Design a questionnaire, set of key informant topics and focus group discussion questions.
The groups should identify for each set of informants the type of information they require in order to carry out the assessment.
Sets of informants
• Decision makers: (MoH/DPPB/ WFP/UNICEF)
• Practitioners (staff working in the EOS and TSP)
• Beneficiaries
• Non beneficiaries
Key information would need to be gathered on each component of the programme (See above for details.) –
• Number of beneficiaries versus number of potential beneficiaries (coverage)
• Success/failure rates (population level, individual case level)
• Process indicators for effectiveness and efficiency
• Any unintended benefits or negative impacts (appropriateness)
• Evidence for long-term impact: Does the package cover all aspects of the defined problem (coherence)?
• Coordination of the interventions (from decision makers) and the overall coherence within the disaster profile of
the country
3. Describe potential challenges that an assessment team would face.
The groups should consider:
• Short time frame
• Resources
• Availability of secondary data
• Ethical access to non-beneficiaries
• Political/institutional interests in the programme

26 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

6. Case studies

The case studies below illustrate some real problems faced by humanitarian workers in areas of M&E. There are no correct ans-
wers to the questions in these exercises. The idea of using case studies is to stimulate participants to discuss real and complex
situations.

Exercise 4: Key issues for evaluations in emergencies

What is the learning objective?

• To familiarise participants with evaluation parameters and to begin to discuss the real issues around planning
and executing an evaluation

When should this exercise be done?

• As part of the initial training sessions (when discussing evaluation parameters) or at the end of the training session

How long should the exercise take?

• 45 minutes for group work, 45 minutes for plenary discussion

What materials are needed?

• Handout 4a: Real time evaluation of Pakistan flood response
• Handout 4b: Real time evaluation in Pakistan: instructions for group work
• Handout 4c: Facilitator’s guide to Handouts 4a and 4b
• Module 20, Part 2
• Flip charts

What does the trainer need to prepare?

• Be familiar with M&E and with the case study material.

Instructions
Step 1: Divide the trainees into groups of five or six.
Step 2: Allow approximately 45 minutes for discussion of the key questions.
Step 3: Bring the group back for plenary debate for 45 minutes.

Key discussion points for plenary

➡ Measurement of change – how can this be achieved in this situation?
➡ Use of resources within a restricted timeframe. How can they be maximised without losing quality in terms of the
evaluation findings? How can the team ensure that the complex context is factored into the evaluation design?
➡ What are the skills needed to carry out this type of evaluation?
➡ Creating realistic causal net or causal chain – how have the team tackled this question?
Are there sufficient checks within the evaluation methodology to ensure that this will be possible?

HTP, Version 2, 2011 27

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 4a: Real time evaluation of Pakistan flood response

Evaluation summary8
The floods of 2007 devastated large areas of rural Sindh and Balochistan provinces in southern Pakistan, destroying homes,
crops and roads, and causing the temporary displacement of over 2.5 million people. The Government of Pakistan (GoP), through
its newly created the National Disaster Management Authority (NDMA) and with help from the Army, launched a major relief
operation. The United Nations, with other members of the international humanitarian community and local NGOs, mobilised
resources to help. The decision was taken by the Inter-Agency Standing Committee Country Team (IASC CT)9 to launch a full-
scale humanitarian response. An application was made to the Central Emergency Response Fund (CERF). Clusters were set up
and started work. A joint rapid assessment was carried out with NDMA and a flash appeal was announced and promoted.

For a variety of reasons, the GoP did not fully support the IASC CT’s decision and approach. In addition, the assessment was
delayed, the flash appeal was issued three weeks after the onset of the emergency and raised only 26 per cent10 of its target,
and the clusters failed to achieve their full potential as coordinating mechanisms. As a result, and despite substantial efforts, the
humanitarian community did not succeed to the extent it considered appropriate in delivering humanitarian relief to the
already-impoverished people of Sindh and Balochistan.

The results of the emergency response were varied in the two areas of the flooding. In the rural areas there was some indication
that nutrition status was less critical than in the towns hit by the flooding. There was no baseline data to compare the two areas
before the flooding occurred. (Added by author for the purposes of the exercise.)

A real time evaluation over a two-week period, staffed and operating independently of the United Nations, was set up to help
to understand the reasons for the results, and to suggest improvements for the future.

8
IASC, Inter-agency real time evaluation of the Pakistan floods/cyclone, October 2007.
9
Inter-Agency Standing Committee, see www.humanitarianreform.org.
10
Recorded as of 30 September 2007.

28 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 4b: Real time evaluation in Pakistan: Instructions for group work
Each group should discuss the following issues in relation to the situation set out above:
a) How can the evaluation team measure whether change occurred?
b) Discuss what resources are needed to carry out the assessment (people and material resources). The team have been
informed that they have two weeks in the field and a further two weeks of preparation/report writing to complete
the task.
c) Training needs of evaluators: What skill set would you ideally have to carry out the evaluation?
d) How would you ensure that the assessment captured the context of the complex situation?
e) How would the group attempt to ‘connect the dots’ and create a causal net or causal chain?

HTP, Version 2, 2011 29

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 4c: Facilitator’s guide to Handouts 4a and 4b

There are no fixed answers to the questions. Some possible considerations are below.
1. How can the evaluation team measure whether change occurred?
Take the evaluation parameters, including the DAC parameters, as a basis for the evaluation and adapt the parameters
to the situation. In addition as this is a ‘real time’ evaluation, the findings should be useful for the on-going recovery
and relief efforts.
2. Discuss the resources needed for the assessment. The team have been informed that they have two weeks of
field work and two additional weeks for preparation and report writing.
The size of the problem is overwhelming. The team will need to be focused and decide how best to get a cross-cutting
sample of agencies/government informants that are active in recovery and relief as well as incorporating beneficiary
opinions. These are real challenges that face all evaluators at the beginning of a task.
Some suggestions:
i. Conduct a thorough desk review of all documentation produced by the government coordinating bodies; the
clusters, the NGOs, United Nations agencies.
ii. Develop key topics lists (under the evaluation parameter headings) applicable to a broad range of agencies.
Methodologically, there are a number of ways in which this can be achieved:
• Half-day workshops with similar agencies: international NGOs, national NGOs, the United Nations, or
• Half-day workshops by ‘cluster’ – food, nutrition, protection etc., or
• Purposive selection of key actors (main national NGOs, small national NGOs, main international NGOs,
small international NGOs, United Nations agencies, donors, etc.)
iii. Identify key interviews with senior relief staff (government, United Nations and NGOs)
iv. Gather beneficiary opinions, including: compiling key topic lists standardised across the evaluators; identifying
areas where different emergency scenarios are taking place (e.g., accessible areas of high population density,
inaccessible areas with low population density, camps for displaced people or areas where households are hosting
large numbers of displaced). The field work should be based on solid information gathered from secondary
documents and preliminary key informant interviews.
v. Resources will include people, transport, logistics and funds. These are never sufficient. However, it is incumbent on
the evaluators to make the exercise as useful as possible for all stakeholders.
3. Training needs of evaluators: what skill set would you ideally have at your disposal to carry out the evaluation
Key competencies for evaluation team:
• experience with evaluation of humanitarian emergencies (preferably of the type of the disaster under review –
in this case a flood)
• practical knowledge of emergency relief and recovery (ERR) work
• knowledge of the coordination systems for ERR
• knowledge of the local context
• language skills (a bonus)
• independent from stakeholders
• ability to work in a team
• ability to work under pressure
If the evaluation team do not have a full compliment of competencies the team should factor in training sessions.

30 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

4. How would the group ensure that the assessment captured the context of the complex situation?
• Review extensive background literature.
• Conduct key informant interviews at an early stage of the evaluation (political leaders, key coordinators of the response).
• Have as much independent local involvement as feasible on the evaluation team.
5. How would the group attempt to ‘connect the dots’ and create a causal net or causal chain?
A causal net or causal chain is an attempt to explain why a certain outcome occurred. For example, why did people
in the remote mountainous area of the flood area have better nutrition status than the people in the more accessible
high density suburbs of the town affected by the floods?
In order to explain the findings the team would need to examine the risk factors, climate and natural factors,
access to food, livelihood options, health and sanitation situation, and cultural issues that could prevent or
promote access to assistance.
(This information is not available in the handout – the groups will need to use their imaginations to create credible
causal nets.)
In the example above the causal net may be as follows:
In the highlands, although houses were destroyed, the main sources of livelihoods are still intact – cross-border trading,
goat husbandry and cheese making. In addition, communities are based on close clan ties and sharing of scarce
resources took place, as well as the hosting of households that had lost their homes and assets. There was no major
outbreak of disease as population density is low and there were no rains. There were no significant after-shocks
in the area.
In the high density suburbs, food aid and other non-food assistance arrived rapidly. However, basic sanitation structures
were destroyed, there was a cholera outbreak, and there are few administrative or traditional systems in place that were
able to identify the most vulnerable households and/or provide a safety net for the most affected. Access to the rations
was problematic with no system developed to identify the most vulnerable for the distribution of food and non-food
items. In addition, livelihoods based on trading were disrupted with the influx of free goods.
In an evaluation that lacks baseline information where there is no possibility of carrying out a full-scale quantitative
survey, construction of these causal nets is the only option for the evaluation team.
There must be rigorous cross checking ‘triangulation’ of information, as well as the extensive use of secondary data
to ensure that the ‘causal nets’ are robust and grounded in reality. The team must be aware of maintaining a balance and
of spreading the interviews/meetings over the whole range of stakeholders.

HTP, Version 2, 2011 31

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Exercise 5: Evaluation of WFP relief operations in Angola

Exercise 6: Evaluation of WFP response to 2004 tsunami

What are the learning objectives?

Exercise 5
• To focus on the use of information from an evaluation and how this information can be used to improve practice
Exercise 6
• To focus on what can be learned about M&E (based on a real example) that can inform practice in the future

When should this exercise be done?

• At the end of the training session. These exercises are particularly useful for decision makers or people who will be
users of information from evaluations.

How long should the exercise take?

• 45 minutes for group work, 45 minutes for plenary discussion

What materials are needed?

• Handout 5a: Case study II: Evaluation of WFP relief operations in Angola: How to best use recommendations
• Handout 5b: Facilitator’s guide to Handout 5a
• Handout 6a: Case study III: WFP response to 2004 tsunami: Group work
• Handout 6b: Facilitator’s guide to Handout 6a
• Module 20, Part 2
• Flip charts

What does the trainer need to prepare?

• Be familiar with M&E and with the case study material.

Instructions
Step 1: Divide the trainees into groups of five or six.
Step 2: Allow approximately 45 minutes for discussion of the key questions.
Step 3: Bring the group back for plenary debate for 45 minutes.

32 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Exercise 5: Evaluation of WFP relief operations in Angola (continued)

Exercise 6: Evaluation of WFP response to 2004 tsunami (continued)

Key discussion points

Exercise 5
This exercise focuses on the usefulness of findings from evaluations. The following aspects should be covered in the
plenary discussion:
➡ Who is the audience for the findings and lesson learned? Is this clear? Could it have been made more explicit
by the authors?
➡ Ways of disseminating information. (depends in large part on the main audience)
➡ Clarify the difference between a lesson learnt and moving towards policy change. Use the examples in the text to
guide the discussion.
Exercise 6
This exercise uses the case to study to point out M&E issues. Some of the obvious ones are cited below, other may arise
in the discussion:
➡ Monitoring requires skilled staff and knowledge of the intervention/area
➡ Key monitoring tools are essential to measure the efficacy of an intervention
➡ Clear assessment can provide programme guidance
➡ Information is “political” and requires careful handling
➡ Lack of continuity is the death of monitoring
➡ Well-designed evaluations can identify unexpected impacts

HTP, Version 2, 2011 33

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 5a: Case study II: Evaluation of WFP relief operations in Angola: How to best
use recommendations
Summary of research11
The WFP assistance in Angola has alternated between relief and recovery since the mid-1970s. To date, nine emergency operations
(EMOPs) and six protracted relief and recovery operations (PRROs) have been implemented by the WFP in Angola. Following
the peace agreement in April 2002, WFP interventions continued to be of an emergency nature. In 2004, as the number of
food-insecure and highly vulnerable people declined significantly, WFP began to focus on recovery operations in the highly
food-insecure regions. The sharp decline in donations and the absence of government funds for recovery obliged WFP to re-
duce its coverage of most programmes. WFP began to concentrate on refining vulnerability analyses and implementing recovery
activities rather than implementing relief activities. Currently, WFP has reduced its humanitarian assistance in Angola, focusing
increasingly on consolidating its efforts and shifting responsibilities to national authorities.

WFP has evaluated the three most recent PRROs implemented from January 2002 to December 2004, as well as five special
operations (SOs) implemented to assist the PRROs in achieving their objectives.

Key findings
Emergency food distribution along with medical and social feeding programmes contributed to WFP’s objectives of saving
lives, improving nutritional status, and preventing malnutrition. WFP’s interventions contributed to nutrition objectives primarily
by reducing acute, rather than chronic, malnutrition. The operations were generally efficient reaching large numbers of bene-
ficiaries despite some delays and gaps. Concentrating efforts in the central highlands was appropriate for improved efficiency.

Many food-for-work and food-for-asset activities satisfied needs for infrastructure, but were mostly undertaken in exchange for
labour instead of constituting activities to improve livelihoods.

School feeding offered an opportunity for broader community participation in development, reconciliation, improving atten-
dance and addressing nutritional concerns. However, activities were hampered by weak government support, lack of funds
and technical capacity of implementing partners, and competing educational priorities.

WFP contributed considerably to enriching humanitarian coordination with the other United Nations agencies, governmental
agencies, NGOs and donors. However, it made little progress in linking short-term emergency relief measures with longer-term
recovery efforts. The marginal involvement of the government and its weak financial and technical engagement severely limited
the potential of WFP’s activities to contribute to sustainable recovery.

Overall, targeting methods improved and vulnerability analysis and mapping data (VAM) were used for geographical targeting,
but were not used systematically for beneficiary selection in particular areas. The M&E system improved in terms of efficient
data collection and storage, but operated separately from VAM and reported primarily on outputs rather than on outcomes.

WFP sought to ensure beneficiary protection, but the task became increasingly complex, involving prevention of discrimination
in aid programmes, ensuring access to basic services and protecting land tenure and property rights. Recovery activities involved
women, but failed to reflect their priorities for literacy, skills training and income generation. Moreover, gender imbalances in
decision-making bodies and the special needs of female-headed households were neglected.

11
Full and summary reports of the Evaluation of the Angola Relief and Recovery Operations are available at www.wfp.org/operations/evaluation.

34 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Key lessons included the following;

• In order to cultivate responsibility and ensure the ownership of results, partnership agreements must be based on joint
goals of WFP and its partners.
• M&E and reporting can be labour-intensive and costly. Clarifying staff responsibilities and allocating sufficient budget
at the start of operations will help ensure proper M&E.
• With better planning, and more consistent, timely and accurate monitoring of both food rations and the nutritional
status of the beneficiary populations, it is most likely that the pellagra outbreak in Kuito12 could have been prevented.
The experience could be useful in preventing similar situations in the future.
• Without adequate information on gender, and a thorough gender analysis of beneficiaries and relief recovery,
WFP’s task of supporting the enhanced commitment to women is rendered impossible.

Working in groups, discuss the findings and lessons learned from the real time evaluation in Angola.
1. Who could use the findings and lesson learned (what is the audience)?
2. What is the best way to disseminate the findings of this or similar evaluations?
3. What can be distilled from the lessons learned for general use in responding to humanitarian disasters – reformulate
the relevant lessons learned as policy points.

12
See Field Exchange, Issue 10, 2000.

HTP, Version 2, 2011 35

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 5b: Facilitator’s guide to Handout 5a

1. Who could use the findings and lesson learned (what is the audience)?
i. Internally for WFP in order to improve practice
ii. WFP implementing partners
iii. By the wider humanitarian community (United Nations and NGOs)
iv. Angolan government
v. Organisations linked to specific development initiatives, e.g., education, livelihood development, women and gender
Observations
The findings were not directed at any one particular group explicitly. This could result in the underutilisation of findings
as no one feels that they have a responsibility to take action on a specific point.
For example, the first lesson learned may be “In order to cultivate responsibility and ensure the ownership of results,
partnership agreements must be based on joint objectives of WFP and its partners.”
This lesson learned could be re-phrased as follows: “WFP and implementing partners have a responsibility to develop joint
(shared) objectives in order to ensure ownership of the results (impact) of the intervention.”
With the rephrasing there is clear attribution of responsibility for action. It is also interesting to note that the findings/
lessons learned do not reflect beneficiary voices and solely concentrate on the humanitarian actions taken on behalf of
the affected populations.
2. What is the best way to disseminate the findings of this or similar evaluations?
There are many ways of disseminating findings/lessons learned from this type of evaluation. Some suggestions are:
• Make the report widely available (distribution and on the Internet).
• Hold focused workshops with key decision makers (government and non-governmental).
• Conduct internal reflection (WFP, implementing partners).
• Encourage public debates in the media (national and international).
3. What can be distilled from the lessons learned for general use in responding to humanitarian disasters
– reformulate the relevant lessons learned as policy points.
Below are some suggestions as to how the lessons learned can be translated into policy points.
In order to cultivate responsibility and ensure the ownership of results, partnership agreements must be based
on joint objectives of WFP and its partners.
PP: WFP will establish jointly agreed objectives for programmes with all implementing partners
M&E and reporting can be labour-intensive and costly. Clarifying staff responsibilities and allocating sufficient
budget at the start of operations will help ensure proper M&E.
PP: Approved M&E systems must include an appropriate budget and indicate the implications for human resource
allocation
With better planning, and more consistent, timely and accurate monitoring of both food rations and the
nutritional status of the beneficiary populations, it is most likely that the pellagra outbreak in Kuito13
could have been prevented. The experience could be useful in preventing similar situations in the future.
PP: Consistent, timely, accurate (and appropriately resourced – see point above) M&E systems must be established
in order to prevent micronutrient deficiency developing in beneficiary populations
Without adequate information on gender, and a thorough gender analysis of beneficiaries and relief recovery,
WFP’s task of supporting the enhanced commitment to women is rendered impossible.
PP: In order to fulfil commitment to ‘enhanced commitment to women’ in humanitarian disasters resources must be
allocated for a thorough gender analysis of beneficiaries.

13
See Field Exchange, Issue 10, 2000.

36 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 6a: Case study III: WFP response to 2004 tsunami: group work
The World Food Programme (WFP) recently published the findings of a real time evaluationi (RTE) of their response to the 26
December tsunami that devastated coastal areas of India, the Maldives, Sri Lanka, Sumatra and Thailand. An estimated 260,000
people were killed or went missing and the lives and livelihoods of 2 million people were seriously affected by the disaster. The
evaluation team comprised three individuals, including a nutrition and health specialist. The focus was on the response at re-
gional level and in Indonesia and Sri Lanka.

The main findings were as follows:

The WFP responded rapidly to the emergency. By early January 2005, an emergency operation (Emergency 10405.0) had been
approved for an initial six months to the end of June. The two objectives of the emergency operation were:
• To save lives by preventing deterioration of the nutritional status of vulnerable children and mothers
• To promote the rehabilitation of housing, community infrastructure and livelihoods

WFP food aid made a major contribution to achieving the first objective, few deaths were reported from malnutrition or disease.
The second objective was more difficult to achieve in the timescale envisaged. Multi-sectoral rehabilitation has been slower
than expected, although the food aid did provide a safety net for people in need and an income transfer for displaced people.

Indonesia
The WFP had an established programme in Indonesia but it had limited logistical capacity of its own. For the first few weeks, a
private sector partner arranged trucks and got food supplies moving. Despite the WFPs strong initial response, it took some
months to develop a systematic food distribution programme because of programming and human resources constraints, as
well as the capacity limitations of some implementing partners. Hence, it took longer to reach the estimated 790,000 people
identified by the needs assessment. In addition, the WFP was still building up its caseload at the end of March 2005, when it had
been anticipated that the initial emergency phase would be over. While benefiting the population on the whole, general food
distributions (GFDs) made targeted programmes like food-for-work (FFW) more difficult to implement because there was less
incentive for people to work or for implementing partners (IPs) to divert resources to design and supervise FFW programmes.
Cash-for-work (CFW) interventions had started on a modest scale – at the time of the RTE they covered about 15 percent of be-
neficiaries assisted by the GFDs.

Monitoring was not as strong as it should have been, partly because only one of its non-governmental organisation (NGO)
partners had a prior presence in Aceh and because all the IPs, like WFP, faced problems such as difficulties recruiting and retain-
ing qualified staff. The Government of Indonesia was expected to take the lead in issuing ration cards to internally displaced
persons (IDPs), but no ration cards had yet been distributed in Aceh at the time of the mission’s visit, making it difficult for food
aid monitors to track assistance.

Supplementary feeding programmes were identified early on as necessary, but they took a long time to establish and were
only starting at the time of the mission’s visit.

Sri Lanka
It was easier to conduct a rapid response in Sri Lanka because there was less damage to infrastructure than in Indonesia. WFP
had a protracted relief and recovery operation (PRRO) in place in the north and east, areas badly affected by the tsunami. WFP
diverted 6000 MT of food from the PRRO to the emergency response in the first few days. WFP’s initial assessment found that
access to food was a problem for 650,000 people, but concluded that one-third would be able to recover their livelihoods
relatively quickly. It recommended that general distribution of full rations should cease after three months at the end of March
2005, with only targeted distributions after that date. The Government of Sri Lanka initially proposed a higher figure of 900,000
for GFDs, using its network of multi-purpose cooperative societies. A consensus was reached that this figure was too high be-
cause it included everyone living in or near the areas hit by the tsunami, including people who were only marginally affected.

HTP, Version 2, 2011 37

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Common findings
In both countries, security regulations impeded the emergency response. An excessive number of security rules meant that
staff could sometimes only work effectively by ignoring them. The emergency response roster (ERR) of WFP stand-by staff for
emergencies did not work because not enough suitable staff were available. As a result, the relief operation was sometimes run
at the field level by short-term international staff and inexperienced national staff, which had a negative impact on implementa-
tion and monitoring.

The tsunami had a greater impact on women than on men – many more women than men were killed. Problems related to
gender were emerging at the time of the evaluation. Women are frequently excluded from decision making in the restructuring
phase. For most people, returning home and economic self-sufficiency were the most important short-term objectives, but
often only men were considered for income-generating activities, free distributions of equipment, vocational training and
allocations of land and houses. Slow progress in rehabilitating housing, which forced many IDPs to remain in overcrowded,
isolated and inadequate shelters, was a major cause of vulnerability among women.

A positive aspect of the operation was the large amount of cash available from donors that enabled WFP to purchase food
locally or regionally, for example, rice in Indonesia and Sri Lanka.

Given the information summarised above from the RTE of WFP work in the 2004 tsunami, discuss in groups the key lessons
that can be learned specifically for future M&E. Draw out key lessons, present to the plenary and discuss.

38 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 6b: Facilitator’s guide to Handout 6a

Lessons for M&E

Measuring impact when there are unrealistic expectations in terms of the duration of interventions
The second objective was more difficult to achieve than the first in the time frame envisaged. Multi-sectoral rehabilitation has
been slower than expected, although the food aid did provide a safety net for people in need and an income transfer for
displaced people.

Lesson: Evaluations should set realistic parameters for measurement of change.

Monitoring requires skilled staff and knowledge of the intervention/area.

Monitoring was not as strong as it should have been, partly because only one of its non-governmental organisation (NGO)
partners had a prior presence in Aceh and because all the IPs, like WFP, faced problems such as difficulties recruiting and retain-
ing qualified staff.

Lesson: Training for monitoring is essential if systems are to be established.

Lesson: Prior knowledge of an area/intervention can improve the quality of monitoring systems.

Key monitoring tools are essential to measure the efficacy of an intervention.

The Government of Indonesia was expected to take the lead in issuing ration cards to internally displaced persons (IDPs), but
no ration cards had been distributed in Aceh at the time of the mission’s visit, making it difficult for food aid monitors to track
assistance.

Lesson: Agencies and government should identify key monitoring instruments before an intervention takes place. Increased
standardisation and clear guidelines would make for better prepared authorities.

Clear assessment can provide programme guidance

The WFP’s initial assessment found that access to food was a problem for 650,000 people, but concluded that one-third would
be able to recover their livelihoods relatively quickly. It recommended that general distribution of full rations should cease after
three months at the end of March 2005, with only targeted distributions after that date.

Lesson: Early assessments can lead to improved targeting and better use of resources.

Information is ‘political’ and requires careful handling

The Government of Sri Lanka initially proposed a higher figure of 900,000 for GFDs using its network of multi-purpose cooperative
societies. A consensus was reached that this figure was too high because it included everyone living in or near the areas hit by
the tsunami, including people who were only marginally affected

Lesson: Numbers produced by assessments are not objective. Humanitarian agencies must engage in dialogue in order to
fulfil the humanitarian mandate to save lives and do no harm.

Lack of continuity is the death of monitoring

As a result, the relief operation was sometimes run at the field level by short-term international staff and inexperienced national
staff, which had a negative impact on implementation and monitoring.

Lesson: Establish robust systems that are immune to staff changes or inexperience.

Well-designed evaluations can identify unexpected impacts

The tsunami had a greater impact on women than on men – many more women than men were killed. Problems related to
gender were emerging at the time of the evaluation.

Lesson: Evaluation must be able to distinguish between vulnerable groups (e.g., age, sex, health status, etc.) and have a robust
gender analysis component.

HTP, Version 2, 2011 39

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

7. Field-based exercises

Exercise 7: Assessment of the monitoring system of a nutrition intervention under field conditions

What are the learning objectives?

• To understand the monitoring system for the programme
• To assess the usefulness of the monitoring tools available in the field
• To identify gaps in monitoring
• To identify training needs

When should this exercise be done?

• In areas where there are any of the following programmes underway: General distribution programmes,
therapeutic care programmes or supplementary feeding programmes

How long should the exercise take?

• The exercise will take 2.5 days (not including extensive travel): 0.5 day of preparation;1 day in the field; 1 day
for analysis and discussion of the results.

What materials are needed?

• Handout 7a: Participants’ guide to field work exercise
• Facilitator’s guide to field work exercise

Methodology
i. Participants will prepare for the field trip by familiarising themselves with all the programmes currently
in operation (field reports). They will review material from M&E module and relevant modules in the series.
ii. Participants will be asked to interview staff members responsible for monitoring systems, collect examples of
monitoring tools and reports where possible.
iii. Participants will be asked to observe the data collection (if possible).

Back in the classroom participants will:

i. Prepare flipcharts that explain the design of the system – highlighting any gaps or problematic areas.
ii. The group will discuss the gaps and problematic areas and suggest improvements, resource requirements and
training needs (from observations).

40 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Exercise 7: Assessment of the monitoring system of a nutrition intervention under field conditions
(continued)

Key discussion points on Let’s get real

This is the point where the facilitator needs to marry the theoretical concepts that have been taught over the time of the
course with the reality of working in emergency settings. The discussions should focus on the following aspects of the
field visit
i. Differences between the reporting of the M&E systems and the reality
ii. If there are gaps/differences, why do they exist? Is it a question of time pressure, lack of training,
not seen as a priority by the health staff or practitioners or other reasons?
iii. Did they observe any feedback loops – were staff aware of how information collected was used,
did staff feel the M&E systems were useful for improving the outcomes of their interventions
(for example. improving survival rates)?
iv. What suggestions do the teams have to realistically improve the systems they have observed (must take
into account the resources on the ground, the context, timeframe for data management, added value of
any improvements against the opportunity cost of taking resources from other aspects of the intervention)?

HTP, Version 2, 2011 41

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Handout 7a: Participants’ guide to field work exercise

Participants must make sure they are aware of the types of programmes they will encounter in the field in order to prepare for
this exercise. The information should be made available by the facilitator or as part of the exercise participants will be asked to
extract the information from field reports.

Participants must ensure that:

• Key question lists are prepared before leaving for the field.
• Key observation points are prepared before leaving for the field.
• A list of who will be interviewed and places to visit is prepared (and if necessary responsibilities are divided to collect
information between participants).

Examples of a checklist of questions

Use the suggestions below as a guide, add additional questions/remove questions according to the local situation/nature of
the programme. Prepare checklists for each of the groups of people who will be interviewed.

Assessment of programme monitoring

• Understand the programme objectives.
• Identify the data that is routinely collected/periodicity of collection.
• Identify tool/instruments are used to collect data.
• Who collects the data (level of training, experience, etc.)?
• How is the data processed: tools, periodicity and location?
• Are the findings from the data used locally?
• Who uses the data and how (what level of training do the staff have, do they have any M&E training)?
• Taking into consideration the objectives of the programme are there any major gaps in data collected,
analysis and use (See example above of not been able to achieve one of the programme goals
because gender analysis was not available.)?

Examples of groups of people who could be interviewed during the field visit
a) Beneficiaries (and non-beneficiaries if relevant)
b) Staff working directly in the programme
c) Supervisory staff
d) M&E personnel

Information on the above points may be collected through interviews, direct observation or review of instruments/reports.

42 HTP, Version 2, 2011

Monitoring and evaluation MODULE 20
TRAINER’S GUIDE

Handout 7b: Facilitator’s guide to field work exercise

The facilitator must ensure that information on the existing programmes is made available to participants before going into the
field. If there is available time, the facilitator may make the information gathering part of the exercise. In this case it will be
necessary to have field reports available for participants to analyse. Participants should be asked to identify the nutrition inter-
ventions taking place, and make a brief summary of the following aspects of the programme:
• Type of intervention (brief description)
• Number of beneficiaries
• Number of sites the intervention is taking place
• Duration (for how long has the intervention been established)
• Number of staff members
• Any M&E information available in the reports
• Any other observations from the reports

The facilitator will ensure that:

• Key question lists are prepared before leaving for the field.
• Key observation points are prepared before leaving for the field.
• A list of potential interviewees and sites to visit has been drawn up and divided between the participants.

Examples of a checklist of questions

Use the suggestions below as a guide, adding additional questions/remove questions according to the local situation/nature
of the programme.

Assessment of programme monitoring

• Understand the programme objectives.
• Identify the data that is routinely collected/periodicity of collection.
• Identify tool/instruments are used to collect data.
• Who collects the data (level of training, experience, etc.)?
• How is the data processed: tools; periodicity; and location?
• Are the findings from the data used locally?
• Who uses the data and how (what level of training do the staff have, do they have any M&E training)?
• Taking into consideration the objectives of the programme, are there any major gaps in data collected,
analysis and use (See example above of not been able to achieve one of the programme goals
because gender analysis was not available.)?

Information on the above points may be collected through interviews, direct observation or review of instruments/reports.

HTP, Version 2, 2011 43

 MODULE 20 Monitoring and evaluation
TRAINER’S GUIDE

Back in the classroom, participants will analyse the information collected, cross-checking for accuracy and present
findings to the plenary for discussion.

Points to focus on in plenary:

• Differences between the reporting of the M&E systems and the reality
• If there are gaps/differences – why do they exist? Is it a question of time pressure, lack of training, not seen as a priority
by the health staff or practitioners or other reasons?
• Did they observe any feedback loops – were staff aware of how information collected was used, did staff feel the M&E
systems were useful for improving the outcomes of their interventions (for example. improving survival rates)?
• What suggestions do the team have to realistically improve the systems they have observed (must take into account
the resources on the ground, the context, timeframe for data management, added value of any improvements against
the opportunity cost of taking resources from other aspects of the intervention)?

44 HTP, Version 2, 2011