Monitoring & Evaluation

WHAT IS MONITORING?
Monitoring is about collecting information that will help you answer questions about your project. It is important that this information is collected in a planned, organized and routine way. You can use this information to report on your project and to help you evaluate. All organizations keep records and notes, and discuss what they are doing. This simple checking becomes monitoring when information is collected routinely and systematically against a plan. The information might be about activities or services, your users, or about outside factors affecting your organization or project. Monitoring information is collected at specific times: daily, monthly or quarterly. Here are some basic points for successful monitoring:

Build simple, user-friendly monitoring systems into everyday activities, collecting data at the most natural point Get commitment from those collecting the information, by explaining why they are doing it Make sure that everyone responsible for monitoring has clear and consistent guidelines Make sure that monitoring records are completed fully and accurately people may not regard it as a high-priority activity Give people collecting the information feedback on the results of their monitoring, and how it is being used to make the organization more effective Check that the project is not collecting the same piece of information more than once.

Get good advice on how your database can best serve your information needs. Talk to another project with a good management information system and to a database expert, if possible. Think about the links you want to create between different types of information. For example, you may want to be able to link user profile data with attendance at different activities. It is important to be familiar with the Data Protection Act. Make sure data is used for its intended purpose. If personal information is kept about individual service users, make sure that they know exactly what the evaluation is for, what data exists, that they can have access to it to check its accuracy, and that the project will preserve their confidentiality.

MONITORING OUTPUTS
When you set output indicators, focus these on the information that will be useful in planning activities and services. A pressure group might want to monitor its level of

Management Organizational Policies & Practices

Page 1 of 16

Monitoring & Evaluation

media coverage, whereas a service provider might want to monitor the scale and pattern of use. This could be:

The level of use at each session, for example, at a luncheon club The total level of service over a given period of time, for example, the number of advice sessions each month The total number of users, number of new and repeat users, and number of users who have left the service, over a given period of time Frequency or extent of use by individual people Use of premises, transport or catering.
Outputs Output Indicators Information Collection Methods When & By Whom How TO Report & Use

Objectives

Objective 3 To provide opportunities for high quality play, drama and dancing Play sessions Drama sessions Dance sessions Number of each type of session Length of sessions Number of children attending Profile of children attending Number of educational activities Log sheet On day by session worker Training planning meeting Quarterly reporting to management Annual evaluation Questionnaire Annual performance

Log sheet

Register

Log sheet

MONITORING THE PROFILE OF YOUR USERS

There are a number of good practice points to think about when you monitor the profile of your users. Information should be given voluntarily. Wherever possible, allow a relationship of trust to develop first and avoid guessing profile information. Using pre-set categories for ages, or for ethnicity, such as those used by the Equality and Human Rights Commission, will make it easier to analyze responses for a large group of people. In Scotland it will be helpful to use questions similar to the Scottish census questions. The Equality Commission for Northern Ireland recommends particular ethnic categories in its code of practice. People often wish to describe their characteristics themselves, so for a small group of users, or where you are working with a range of smaller minority groups, this may be preferable.

Management Organizational Policies & Practices

Page 2 of 16

Monitoring & Evaluation

MONITORING OUTCOMES
Outcomes are the changes, benefits, learning or other effects that happen as a result of your activities. Desired outcomes often describe positive change. They can also be:

About maintenance of the current situation. This involves ensuring things stay the same and dont get worse. About reduction in something, for example, criminal behavior or vandalism.

Outcomes can be:

Welcome or unwelcome Expected or unexpected.

You may be able to anticipate most of your outcomes. However, some things may happen that you did not plan for.

REPORTING MONITORING DATA

Monitoring data is usually reported to your trustees or to your funder or commissioner at regular points. This is usually quarterly, but for funders may be yearly. It will:

Keep trustees and others fully informed about how well the organization is doing against targets Prompt useful questions about how far activities are achieving expected benefits Prompt corrective action before it is too late.

Keep your life simple by collating information regularly, for example, weekly or monthly, to avoid too much work at one time. From the performance indicators you identified for your outputs, select key indicators to report on, for example, numbers of sessions, meetings, publications, profile of users or other target group. Set these out in a way that makes it easy to compare against the targets you set for your key indicators. Some of these may be performance levels or standards agreed with your funder. Make sure you distinguish between new and repeat users when you collect and report on take-up of services. Break down the data by age, gender, ethnic group or other relevant category to ensure that different levels and types of participation in the project and differences in benefits can be seen clearly. Keep monitoring data over a period of time, so that you can report on any trends that emerge. Outside factors, such as a changed policy or a new service in the area, may also affect trends. Using a consistent reporting format over time will help you to make comparisons.

Management Organizational Policies & Practices

Page 3 of 16

Monitoring & Evaluation

WHAT IS EVALUATION?
Evaluation is about using monitoring and other information you collect to make judgments about your project. It is also about using the information to make changes and improvements. Your monitoring information is likely to contain:

Profile information on your users Basic project record keeping, such as the minutes of meetings and case records Statistical information on take-up of services Feedback sheets from training courses and workshops Diaries and other records of events Complaints and compliments from users.

When you evaluate, you will use this information, but often you will need to carry out additional data gathering. Your monitoring information will probably suggest further questions that need an answer. You need to think clearly about where the focus of the evaluation will be and who and where you want to obtain information from. Make sure you set enough time aside for this additional information gathering. Questionnaires take time to develop, and should be tested with a small sample from your target group to see if they will capture the information you want. Interviews take time to organize and even longer to write up and analyze. Evaluation is a set of interlinked activities. Each of these is an important part of the overall process and needs adequate time built in to protect the quality of the evaluation.
Conducting an evaluation

OUTCOME EVALUATION
Outcomes are the changes that you can attribute to your organizational or project activities. These can be intended or not intended, positive or negative, and you may have set indicators for these.

Management Organizational Policies & Practices

Page 4 of 16

Monitoring & Evaluation

Once you have set indicators for these outcomes, that is, you have decided what will show you that outcomes are being achieved; you might have been able to monitor some of these outcomes regularly. During the evaluation stage you should be able to put together this information, place it in the context of your services and working methods, what you know about your users, and external factors, and make judgments about what is being achieved. Useful outcome information can also be obtained during the evaluation itself through one-off data collection. This is also a good opportunity to pick up on the unintended effects of activities. Projects often produce outcomes that were not expected, as well as planned changes, so it is important that you collect information in a way that will tell you about these. Be clear, particularly with funders, about what can be measured at any given stage. At an early point, for example a pilot phase, you may be able to measure only intermediate outcomes, for example increased knowledge or skills, changed levels of confidence, rather than longer-term outcomes, such as take-up of jobs, which might fulfill the project aims more completely. Longer-term outcomes may often be less easy to evaluate than intermediate ones. Some projects provide a service which is a link in a chain of services, or part of a multiagency intervention. For these projects, intermediate outcomes may be the final outcomes assessed as these are the only ones the project is involved with.

IMPACT EVALUATION
The term impact is used in a number of ways, but is usually taken to mean the effect of a project or program at a higher or broader level, cumulative effects or changes that affect a wider group than the original target. Impact relates to the overall aim or mission of your project. Impact is often longer term and therefore the timescale needed for impact evaluation may well be longer than that needed for outcome evaluation. Impact is generally achieved through the efforts or effects of a number of interventions. So you need to think carefully about the role of your project in relation to these longterm changes and place your projects activities within a broader picture. Even if you are able to follow up individual users later, to assess longer-term changes, it may be difficult to analyze clearly the role of the project in their successfully finding and keeping jobs or housing, reducing their alcohol dependency or maintaining family relationships.

WHY EVALUATE?
Monitoring and evaluation not only measure how well you are doing, but also help you to be more effective.

Management Organizational Policies & Practices

Page 5 of 16

Monitoring & Evaluation

Providing information to funders and other stakeholders

Monitoring and evaluation will provide useful information for funders about the level of activities and benefits for your users. You will be able to give examples of what has worked well, and what your users most value. You will also be in a better position to make a good case to funders for continuing or developing activities. You can also use your evaluation findings for publicity, for lobbying or campaigning, for advocacy and to highlight gaps in services.
Learning from evaluation

Sharing the information within your project will help you to become a learning organization. Management can improve its decision-making, and staff and volunteers will appreciate the value of the work that they do and understand how they can make further improvements. Once the immediate reporting back has taken place, make sure that dates are set for action so that impetus and enthusiasm are not lost.
Using evaluation for organizational planning and management

The evaluation can provide decision-makers with knowledge and information to make informed choices. Your evaluation should show which parts of the project are working, for what people and in what circumstances, and provide a warning if something is going wrong. These are key findings and you need to decide what action to take. Is extra funding needed? Are new activities required? Do staff need extra training or skills? The evaluation will also provide information for your next year plan. It will help you to review your objectives. Are your services or activities the right ones to achieve the intended change or benefits? If the project has brought about some unexpected results, how will you take those into account in future planning? You may need to gather more information about the outside world, for example local strategies and other service provision, before making decisions about changing services. The evaluation may give you clearer information about who is using your services, about your members, or who you are reaching with your information or publicity. This will help you to think more carefully about who you are not reaching. If the findings point out areas where need is greatest or least served, you may need to consider redefining your target group. You may need to carry out more publicity or establish new contacts and networks. It may be that you need to follow up your evaluation with a more in-depth needs analysis. Your evaluation will also allow you to review your targets for outputs and outcomes. If you have not met certain targets, or if you have exceeded them, then you should be able to set this against what you now know about the capacity of the project and the performance of other agencies. Your evidence should be strong enough to show if there were good reasons for a lower than expected performance, whether targets were set realistically and whether you should adjust them.
Management Organizational Policies & Practices Page 6 of 16

Monitoring & Evaluation

Use the lessons learnt about what you could do better, or differently, in your operational planning. Do you need to:

Change the way the project is managed? Reallocate resources? Expand or change direction?

Staff and volunteers are under pressure in their daily work routines and will need motivation to use evaluation findings and make changes. Work towards changing the culture of the organization, so that people are receptive to new ideas and challenging feedback.
Using evaluation to set service delivery standards

Evaluation should give you some important information about how you deliver your services to users, how this affects user satisfaction and how service delivery affects the outcomes for users. For example, user satisfaction and trainer observations should allow you to set standards for your training. These could be about such things as the maximum number of participants, the quality of training materials or accessibility for disabled people.
Reviewing key resources

Your evaluation should not just look at the results of your activities, but should relate these to the projects inputs. How have activities been affected by the projects management structure, its staffing or funding levels? Are the projects physical facilities unsuitable or is the project transport inadequate? Do you need improved information technology to support the project?
Using evaluation for policy change

Evaluation can play a key role in highlighting the effect that wider social structures and policies have on your own work and on the lives of the people you work with. Policy is influenced by a combination of factors, including the assumptions, personal ideology and interests of the policy-makers. Although the processes involved in policymaking are complex, evaluation can be designed not only to improve practice, but also to improve and change policies at many levels. When you report on your evaluation, think about who your audience is. If you intend to publish the evaluation, this clarity about your audience is essential. Communicate the lessons you have learnt directly and simply. Be direct about what types of policies might be effective, based on findings. Communicate what you know to community members and project users. This can gain you allies to help you influence policy-makers.

Management Organizational Policies & Practices

Page 7 of 16

Monitoring & Evaluation

Using evaluation for strategic planning

There is an important role for evaluation in strategic as well as operational planning. The strategic planning process starts when you have analyzed monitoring and evaluation data.
Steps in strategic planning

Monitoring and evaluation data can provide valuable information for analysis of organizational strengths, weaknesses, opportunities and threats. Effective monitoring and evaluation will help your organization to provide services of the highest possible quality, and embody the highest standards of integrity, credibility and accountability. It will make sure that you are working with the greatest possible effectiveness and efficiency that you provide value for money and, above all, that the work you do will make a real difference.

UNDERSTANDING AND CLARIFYING YOUR PROJECT

Stakeholders expect clear statements about the reason for the projects existence and what it is doing. This clarity is also needed to plan the projects work and to monitor and evaluate it. These statements are usually expressed in the following way:

Mission or overall aim your organizational or project purpose Values the organizational principles you work to Specific aims the changes or benefits you intend to achieve Objectives the activities you carry out.

Think about which of your stakeholders will be involved in thinking these through. If these statements were not clearly developed when the project was first set up, it will be helpful to do this as a first step in monitoring and evaluating the project.

DEVELOPING A MISSION STATEMENT

A mission statement expresses the overall purpose of an organization. It is sometimes called a strategic or overall aim. It defines and summarizes the long-term change or
Management Organizational Policies & Practices Page 8 of 16

Monitoring & Evaluation

difference you want to make. Because it expresses this in broad terms, it will need to be supported by more specific aims. The mission statement should be short and clear, and cover the essential aspects of your work. Mission statements usually include some core values that describe an organizations approach when carrying out its activities, for example, the way you work with service users. Values may also be expressed in a separate statement. A good mission statement tells people about an organization and its ideals in less than 30 seconds.

AIMS AND OBJECTIVES

Your mission or overall aim is likely to be too broad a statement to allow you to plan your work in detail, or to provide guidance on what you might monitor and evaluate. Specific aims statements about different aspects of your main purpose will allow you to think clearly about what you want to achieve. Aims should show an understanding of the needs of the target group. These more precise statements are also easier to evaluate. Specific aims are likely to remain relevant for several years, but should be reviewed regularly. Aims and objectives are linked in the following way:

Aims are the changes you are trying to achieve. Objectives are the methods or the activities by which you achieve your aims.

It is important to make this distinction in order to be clear about what you are doing, why you are doing it, and to be able to assess your achievements. Try not to have too many aims and objectives and dont make them vague or too ambitious. Aims will usually describe:

The people the service or activity is intended for The intended benefits, or changes you expect to make, for your target group.

Aims may also identify the area in which the project will work. Although they should be clear, aims do not specify exact details of what will be done to achieve them, nor do they spell out a timescale. Objectives identify what the organization will do and the services it will provide. Each aim will have one or more objective, and some objectives may relate to more than one aim. You can do the same thing with internal or organizational aims and objectives. These are mainly concerned with helping the project achieve its external objectives. They relate, for example, to management, finance and personnel.

Management Organizational Policies & Practices

Page 9 of 16

Monitoring & Evaluation

Once you have described your project in clear statements, with an overall aim and more specific aims and objectives, you will be able to describe your project activities and their effects in more detail, and in ways that can be evaluated. The terms output, outcome and impact are used. These relate directly to aims and objectives what you plan to do as shown in the following diagram.

Inputs are the resources you put into the project, such as staff and volunteer time, funding and technical resources. Inputs will directly affect the quality and level of your outputs. These in turn will affect your project outcomes.

PERFORMANCE INDICATORS
To check how your organization is performing, you will need to set performance indicators. These show what you do. They are the criteria or clues on which to base judgments about the progress and success of the project. Performance indicators:

Let stakeholders know what they can expect from the organization Provide a focus for managers Help you to focus on what you need to monitor and evaluate Help a project to judge its achievement and effectiveness Help comparison between projects.

Who defines the performance indicators for your project is important, as different stakeholders will have different views about what counts as success. If you involve them you will have a wider and richer perspective. It may be helpful to look at indicators that similar projects are using.
Management Organizational Policies & Practices Page 10 of 16

Monitoring & Evaluation

Funders may want to have performance indicators that are easily measurable or that link to their own strategies. This could mean measuring things that are not at the heart of the project. So discuss your key activities and intended benefits with funders early on.
Types of indicator

There are a number of different types of indicator. The most common are output and outcome indicators. These are often confused with impact indicators, but there is a difference.

Output indicators these demonstrate the work the organization does and show progress towards meeting objectives. Outcome indicators these demonstrate changes which take place as a result of the organizations work, and show progress towards meeting specific aims. Impact indicators these demonstrate longer-term change relating to the overall aim or mission of the organization.

SETTING TARGETS
Often it will be useful to compare monitoring data against targets that you have set in advance. Targets are expected levels of achievement in relation to inputs, outputs, outcomes and impacts against which you can measure performance. Targets specify the level and quality of activities and achievements that you expect or hope for. They may be needed for funding applications, but there are a number of reasons for setting targets. They can help with operational and individual work planning, help the project to improve performance, and help to demonstrate and improve value for money. They will provide a benchmark or standard to monitor and evaluate performance or progress against, and to document progress towards achieving aims and objectives.

PLANNING FOR MONITORING AND EVALUATION

Once you have a clear plan for your work, you can set out a monitoring and evaluation plan. In this, you may well be prompted by what a funder wants you to monitor. Think also about other information that you need to help you manage the project efficiently and effectively, and about who else might be interested in learning about what you are doing. In planning for your monitoring and evaluation activities, consider questions that will help you prioritize. For example:

What is most important to evaluate at this stage of the project? What evaluation questions do you want answered so you can plan next years work? What are the most significant outputs and outcomes?

Management Organizational Policies & Practices

Page 11 of 16

Monitoring & Evaluation

Think about how you will carry out monitoring and evaluation, who will do it, and what methods you will use. You also need to consider management, support and supervision issues. For example:

How can you best build self-evaluation activities into your routine work? How will the information be collated and analyzed? What skills are needed? How can you make sure that staff will collect the information and collect it consistently? When is the information needed?

You need to ask yourself:

When will different data be collected? Who will carry out and manage the activities? How will data be collected, analyzed and presented? How can routine monitoring activity support evaluation? How will evaluation findings be used in short-term planning and decision-making, quality reviews and other key processes?

Focusing your evaluation activities is vital. This focus, in turn, shapes the questions to which you want answers, and these should now be clearer. The following questions are quite different from each other:

How are resources being used? How appropriate is the management structure? How well are we meeting identified needs? How do we fit within a network of services? How well have we met our expected outcomes?

Try to build costs for evaluation into your budget right from the start and include them in funding applications. This will avoid having to divert scarce resources from other activities. If funding is uncertain, it may be difficult to plan your evaluation realistically. Your evaluation is more likely to succeed if:

Monitoring and evaluation plans are realistic Both evaluation and the project itself have good management support Staff and volunteers understand and value monitoring and evaluation tasks Enough time is set aside for monitoring and evaluation activities, including organizing and processing the data Evaluation is built into planning cycles, job descriptions and budgets Presentation of evaluation findings is timed so that they are available when decisions are being made.

DATA COLLECTION METHODS

Primary data is data collected, for example, through questionnaires or interviews designed specifically for the evaluation study.

Management Organizational Policies & Practices

Page 12 of 16

Monitoring & Evaluation

Secondary data is data collected by other organizations or researchers for their own purposes that can provide a useful source of information for the evaluation. This may be, for example, local population statistics, or a report on user experiences in other, similar projects. Quantitative data deals with numbers, asking questions such as: how much?, how many? and how often? Things are either measured or counted. It allows you to compare sets of numbers and look at their distribution. It is useful when you want accurate, precise data. You may be able to test whether there is a statistical relationship between an activity and some measured change. Qualitative data tells you about personal reactions, feelings, impressions and processes. It may be based on verbal or written records, and extracts from letters, questionnaires and minutes of meetings. In some cases data from open-ended questions can be transformed into quantitative data. This can be done by grouping statements or themes into larger broad categories and giving them a numerical value. For example: 26% of users made some negative comment about the waiting time, while 15% were unhappy about the adequacy of the waiting area, referring to the space and seating available, and inadequate provision of childrens toys. Be aware that such quantification of qualitative data may lose the essence of the personal views and experiences recorded.
Piloting

Before starting any data collection, it is a good investment of time to test the datacollecting instruments. For example, ask some project participants to fill in a questionnaire or carry out interviews with a small number of respondents. Then look at the results to see if the questions have been understood consistently by respondents and if you have captured the information you wanted.

DIVERSITY AND ACCESS

When designing the evaluation, be clear about differences between groups and recognize the potentially different viewpoints of minority groups, and how these differences might affect the way they experience the project. Be sensitive to cultural norms when choosing data collection methods. Think about how gender, age, disability, language and ethnicity may affect the ability of different groups to take part in the evaluation:

Which tools are most suitable for which groups? When would it be most convenient for people to meet? Would people prefer to meet with you alone or with someone else? Is an interpreter necessary?

Management Organizational Policies & Practices

Page 13 of 16

Monitoring & Evaluation

What type of evaluator or interviewer will put your respondent group most at ease? In some situations, for example, you may wish to consider a same-sex evaluator. What venue will be most suitable?

INVOLVING USERS IN EVALUATION

It is increasingly expected that users will be involved in evaluation, and this is sometimes requested by funders. Evaluation findings that users have contributed to could have a greater impact on policy-makers. Indeed, there are a number of good reasons to involve users in evaluation:

Users may have been involved in the design of services Involvement will make use of the expertise that users may have in relation to their own needs Users could gain a greater understanding of the project as a whole It could help to meet the projects aims of building capacity and confidence in the users and lead to greater user involvement in the project It will create a sense of ownership in the evaluation.

Be clear with users about what commitment is involved. Remember, if you are working with users you need to:

Plan in enough time it may take much more time than any other approach Provide proper training and support Encourage teamwork.

WRITING THE EVALUATION REPORT

You should have the basis of a useful evaluation if you planned and focused your evaluation carefully, if you have the right skills, if you chose an appropriate mix of methods to collect the data, and if you analyzed it carefully. However, there are five further important stages:

Interpreting your findings Drawing conclusions Writing your recommendations Presenting findings Using evaluation findings.

Management Organizational Policies & Practices

Page 14 of 16

Monitoring & Evaluation

INTERPRETING FINDINGS
Interpretation means looking beyond the data itself and asking what the results mean in relation to your evaluation questions. Be wary of assuming that there are links of cause and effect between your project activities and results. Involve other people in this level of interpretation and, where appropriate, acknowledge in your report the possibility of other interpretations. Remember to place data in context, bringing out the individual differences behind the responses. It is not always easy to get a sense of the significance of different findings when you first look at the data you may lose the most important point in studying and presenting the detail, or over emphasize a minor finding. So try to stand back from your findings and look at the broader picture before finalizing your report.

REPORTING
Drawing conclusions

Conclusions do not just repeat the data, but should link clearly to the evidence presented. Avoid generalizing from a particular finding and make clear in your report the difference between facts, respondents views and the evaluators interpretation. If your report is short, you might summarize your conclusions at the end of the findings section. In a longer report, summarize your conclusions on one section of your findings before reporting on the next topic. One approach to finalizing the report is to organize discussion groups on the draft report. Think of this as your final stage of data gathering, as you may be given new insights and different interpretations. It is more likely that recommendations developed in this way will be acted on. Make the distinction between data and interpretation clear. If differences of opinion arise, make sure that the evidence is reported clearly in the report, and offer alternative explanations.
Writing your recommendations

Recommendations are based on the conclusions. An evaluation need not always lead to recommendations, and be wary of making recommendations if you do not have enough evidence. It is helpful to be clear about where you are able to propose a course of action and where further information or consideration may be necessary. Be clear about the most important recommendations and those that are less significant. If possible, give a timescale and say how the recommendations should be implemented. Recommendations should be specific, realistic and achievable.

Management Organizational Policies & Practices

Page 15 of 16

Monitoring & Evaluation

Presenting findings

How you present the report will have an important bearing on your credibility, so consider what your evaluation report will look like before you start putting together your findings. Think first about the purpose of the evaluation and its audiences. Most reports will cover the following:

An introduction, which may include the projects aims and objectives The aims and objectives of the evaluation, including the main evaluation questions How the evaluation was carried out Findings Conclusions and recommendations.

It is also useful to produce a brief summary, placed at the front of the document, for people who will not read the whole report. When writing your report:

Remember your audience and the key questions covered by the evaluation Keep it short, including only what the reader really needs to know Use careful wording for negative points and avoid reflecting on individuals Use plain English, and avoid jargon Make sure the content follows a logical sequence Use a standard, readable sans serif font, of at least 12 points in size.

TABLES, DIAGRAMS AND CHARTS Tables, diagrams and charts get information across effectively and concisely. Look at other reports to see how charts and tables are used and set out.
Using evaluation findings

Once you have presented your report, it may be tempting to file the evaluation away. This will be a waste of all the time, energy and resources spent on the evaluation and risks losing the goodwill of everyone involved. Importantly, it will also be a lost opportunity to improve what you do. Using your monitoring and evaluation findings is what makes the time and energy you have invested worthwhile. Your efforts will be wasted if you file your report away in a drawer. Instead, you will have a powerful tool to demonstrate the value of what you are doing, plus practical lessons to develop the project.

Management Organizational Policies & Practices

Page 16 of 16