Writing better

questionnaires
getting better data

Writing better questionnaires: getting better

data
January 2014
Charities Evaluation Services
Acknowledgements

Written by Jean Ellis for Charities Evaluation Services.

With thanks to the Esme Fairbairn Foundation for its funding of this publication.

Design and production by Julia Essame

Published by Charities Evaluation Services
Charities Evaluation Services (CES) is a company limited by guarantee. Registered in
England and Wales number 2510318. Registered charity number 803602. Registered
charity in Scotland (number SCO40937).
Registered office address: 4 Coldbath Square, London, EC1R 5HL, UK.

Copyright
Unless otherwise indicated, no part of this publication may be stored in a retrievable
system or reproduced in any form without prior written permission from CES. CES will
give sympathetic consideration to requests from small organisations for permission to
reproduce this publication in whole or in part but terms upon which such reproduction
may be permitted will remain at CES discretion.
Charities Evaluation Services 2014
ISBN 978-0-9568090-1-8

Contents
Introduction 

Planning your questionnaire 

What is this guide about? 

How to use this guide 
Defining the terms 

First steps 
An overview of the process 

4
4
5

8
9

Drafting the questions 

11

Putting the questionnaire together 

18

Designing the questionnaire 

21

Administering the questionnaire 

25

Using the data from questionnaires 

32

Glossary 
Further information and reading 

35
37

Overview of the drafting process 

Main criteria when drafting questions 
Types of questions 
Advantages and disadvantages of types of questions 
Writing good questions 

The overall structure 

The opening 
Using helper questions 
The ending 
Your draft questionnaire a checklist 

Instructions 
Choosing your response options 
Design tips 
Designing an effective questionnaire 
Making the questionnaire accessible to different groups 

An overview 
Choosing your sample 
Using online surveys 
Ethical considerations 
Getting a good response rate 

Analysing the data 

Interpreting the data 
Writing the report 
Evaluate and learn from the process 

11
11
12
15
15

18
18
18
19
20

21
21
23
23
23

25
25
26
28
29

32
32
33
34

Introduction
What is this guide about?

More and more organisations are self-evaluating the work that they do, with a greater emphasis on
assessing their outcomes and impact. This means collecting data. One of the most commonly-used
data collection methods is the questionnaire, but organisations dont always put in enough time
and resources to get useful and valid evidence from them. It takes clear thinking and considerable
practice to get it right.
This guide is developed particularly for voluntary and community organisations. It is a practical
resource on ways to plan and design a questionnaire, and on how to write good questions and get a
response rate that will give you credible data. It specifically has in mind how to get information about
how users value your services, and about the outcomes and impact that your services achieve.
Charities Evaluation Services (CES) has a number of other publications that you may find helpful
when designing and carrying out your data collection. You can freely download the following from the
CES website.

Information Collection Methods: Choosing tools for assessing impact (2013)

The CES Resource Guide: Evaluating outcomes and impact, 2nd edition (2013)
Assessing Change: Developing and using outcomes monitoring tools (2010)

How to use this guide

You can choose to dip into the guide to find specific information or you can work through it
methodically. Following this introduction, the guide is divided into six main sections:

Planning your questionnaire

Drafting the questions
Putting the questionnaire together
Designing the questionnaire
Administering the questionnaire
Using the data from questionnaires.

Using questionnaires

Questionnaires are a relatively cheap and quick way to gather information from a large number of
people on a limited number of topics. The same questions are asked to each individual; responses
are summarised as statistics and can be easily compared. Online questionnaires provide statistical
analysis for you.
Your respondents may come from:

your target group those you would like to use your services
your user group those who actually use your services
potential users those in your target group who know about your services
non-users those in your target group who know about your services but choose not to use
them
other stakeholders
other professionals.

Voluntary sector organisations are increasingly using online questionnaires as a way of reaching
4

their target population efficiently to find out about user needs or stakeholder satisfaction (staff, users
or other stakeholders). More and more, online questionnaires are also being sent to users and other
stakeholders as part of an assessment of the difference organisations make their outcomes for
users and their broader impact.
For web-based surveys, participants access the questionnaire through clicking on a hyperlink sent in
an email message or by typing the web address into their browser. However, an online questionnaire
wont be suitable for all situations, and for small numbers of respondents, a questionnaire based in
an email, or attached to it, may be more appropriate. Paper-based questionnaires remain useful to
distribute in group situations, or to people who may not have access to the internet. However, care
must also be taken to consider how access to these questionnaires might also exclude certain groups
and evidence.
Whether questionnaires are online or paper-based, whether they are short and snappy or long and
detailed, drafting and designing your questionnaires well is key to getting good information. Even
when questionnaires are administered by phone or face to face, well-drafted questions are important.
There are some problems associated with questionnaires: not enough people returning information
(low response rate); an introduction of bias to the results caused by the choice of sample or low
response rates; the extent to which the validity of data is compromised if questions are not designed
well or people understand the questions differently. Good questions and good questionnaire design
are important to limit these potential difficulties.

Defining the terms

It will be useful to be clear about what we mean by some of the main terms used in this guide.
The terms survey and questionnaire are often used interchangeably, although one is more general
and the other more specific.

Survey

A survey is a process of systematically collecting information from different individuals. Surveys are
carried out for many different purposes and can be done in a number of different ways, including
through questionnaires and interviews. Whatever method is used, a survey will use standardised
procedures and a structured format, so that every participant is asked questions in the same way,
and responses can be statistically and thematically analysed. Participants being surveyed may
present their own views or represent those of an organisation to which they belong or are attached.

Questionnaire

A questionnaire is a research tool a specific type of information gathering which may be used in
surveys to collect information from different individuals. Questionnaires are paper-based or electronic
forms that have a series of simple or complex questions for individuals to complete on their own, or
with some assistance if necessary or appropriate.

Population

We refer to a population, meaning the whole group that the sample of individuals or organisations
you choose (ultimately your respondents) represent. For example, if you send a questionnaire to a
sample of the users of a helpline over a specific time period, the population would refer to all the
users of the helpline. If you distribute a questionnaire at the end of a series of training workshops,
the population would be all those participating in the training, not the 80 per cent that completed your
questionnaire.

Sample

You may be able to reach all of a specific audience, for example, your service users or other
professionals you are in touch with. But if large numbers are involved, you may decide to select
a sample as a more practical and efficient approach. A sample is the selection of individuals (or
organisations) from the target population to whom you will direct your survey, though not necessarily
the group that actually completes the research. There will be individuals who are unreachable as well
as non-respondents, but a good sample is one that is representative of the population from which it is
drawn.
How you decide to sample will be influenced by:

whether you are doing quantitative or qualitative research, or both

what resources you have available
what other methods of research you are using.

There are two main types of survey samples:

Probability samples. These allow you to use the findings from your sample to draw conclusions
about the total population. Organisations sampling from their total user population may use different
types of sampling:

Simple random sample each individual is chosen randomly and completely by chance. This
means that each individual is as likely to be selected to be part of the sample. This sampling
should provide unbiased sampling, but it requires the whole of your population (for example, all
your users calling a helpline, or all organisations downloading your resources) to serve as the
sampling frame from which you make your random selection. This option may not be available
or feasible.

Systematic random sample you start by choosing one from your list randomly and then
select at regular intervals, for example, the fourth name on your register, or every tenth
organisation in your database. This sampling technique isnt well suited when the population is
diverse in any way, as this diversity may not be well represented.

Stratified sample used when groups within a population vary. You divide members of the
population into subsets, with every individual assigned to one group only. You can then
randomly select within each group. This will help you make sure that all types of people or
organisations you work with are represented, although note that you need to have large
enough numbers in each group for the information to tell you something about that groups
particular views or experiences.

You could also have a probability sample by issuing pop-up invitations to your website users
randomly, or by surveying every nth visitor to the website.
Non-probability samples. If you dont have access to all the population that you want to represent,
you may not be able to use random selection procedures. Even with a random selection, you wont
be able to draw firm conclusions about the entire population based on the sample data, and it will be
important to describe the characteristics of your respondents (for example, geographical spread, age,
gender and other relevant characteristics) so that its clear where there may be possible sample bias
(see explanation of bias below).
Non-probability samples include:

6

Convenience sampling this allows people to self-select as to whether they respond, for

example, to a questionnaire posted on a website. People answering questionnaires are likely

to have a particular interest in a topic, whether positive or negative.

Volunteer opt-in panel where a sample is created by recruiting a panel of volunteers.This is

used for online surveys, for example, by using pop-up advertisements.

Purposive sampling in this, you select your participants with a purpose in mind. You may
want to focus on particular groups, based on demographic data, geographical area, or their
participation in activities. You may want to ensure that you reach groups and individuals who
have a different experience of your services or to give particular weight to specific groups or
perspectives.

Snowball sampling this is achieved when individuals or organisations are identified who can
refer your survey on to others.

Sampling frame

A sampling frame is a list of the individuals from whom you are going to obtain information. Sampling
frames can include email distribution lists, members of an organisation, or a database of service
users. A sampling frame may also divide this overall list according to a number of different criteria to
ensure that you get a representation of individuals or organisations with a diversity of characteristics
that you want to know about. Your sample will be drawn from this list.

Survey bias

Survey bias may occur when findings are unduly influenced by the way the data is collected,
analysed or interpreted. Bias can occur in a number of ways:

Sample bias this happens when the sample is selected in a way that makes it less likely for
some members of the target population to be included than others.

Coverage bias this happens when the survey is unable to reach portions of the given
population. For example, for online surveys this may happen if people are not able to access
the internet; this may particularly affect a specific group of people, for example, older people.

Response bias this happens because of the difference between those people included in
the sample who choose to respond to the survey and those who do not. As well as issues of
difference in access, those responding are likely to have strong positive or negative views.
Bias may also be introduced because people tend towards giving a response that is seen as
socially acceptable, and under-report behaviour that is regarded as negative. Numbers or
frequency related to sensitive topics may be inflated or underestimated.

Response rate

The response rate is the percentage of people invited to participate in the survey who respond.
Efforts should be made to encourage a high response rate so that you have accurate, useful results
which are representative of the target population.

Planning your questionnaire

First steps

When you start planning your questionnaire, you will have some idea of its broad purpose. Before
you go further to define more specific aims and objectives you should consider:

the budget and other resources you have available

your likely timeline from setting the research objectives and agreeing them through the
design phase, the administration of the questionnaire, its analysis, and writing your report and
disseminating the findings.

Thinking about this at the beginning will allow you to set realistic parameters for your survey, and
affect your decisions about what you want to cover in the questionnaire. With this in mind, define your
aims and objectives more clearly. It could be that you need to increase the resources that you were
going to allocate to the task.

Knowing your purpose

It is important that you are clear about the purpose of your questionnaire from the outset; you should
consider this before you think about how you are going to administer it. For example, a questionnaire
could be intended to assess the need for our services or to find out what difference we have made
to our user group. Once you have expressed your purpose broadly, it could be that you start to make
your aim more meaningful and more specific.
If you were putting out a questionnaire after a conference, you might originally consider your aim to
be: To get feedback on how well the conference went. However, you can clarify this in a way that will
lead you to be more specific. You may find that you have three main purposes:
To assess how satisfied participants were with conference arrangements, how far the conference
met with their aims and what benefits they gained.
At this stage, and before you start drafting the questions, its also important to clarify your objectives.
These will break down the issues you need to address in order to meet the purpose of the
questionnaire and should provide a framework for your questions. List these, as in the box below:

The post-conference questionnaire

Overall purpose: To assess how satisfied participants were with conference arrangements, how far
the conference met with their aims and what benefits they gained.
Questionnaire objectives: To collect information from participants about:







8

who the participants were

how they have used our resources in the past
what their aims were in attending the conference and if they were met
what elements of the conference programme worked well
what they learned from the conference
whether they made new contacts
what plans they have as a result of the conference programme.

A questionnaire can provide an opportunity to cover a number of things, but its important to prioritise.
Think carefully about whether its realistic to get a proper assessment or measurement of each
element of interest.

Thinking about how you will analyse in advance

It is important to think about whether you will report your data descriptively, or whether you will want
to draw conclusions for the general population based on your data. Inferential statistics allow you to
find out if your results are statistically significant or not (that the results have not just happened by
chance) and whether your findings can be generalised to the whole population. Your questions, your
measurement scales and your sample type and size will all be of consideration if you are going to
use statistical inference effectively. For example, your sample size should be about 10 per cent of the
total population, but not smaller than 30 and care will be needed if you are analysing your data by
subgroup, such as by age, ethnicity or size of organisation.
You may want to consult with someone who is familiar with statistical analysis in advance. However,
this sort of analysis may not be feasible or even appropriate for your information needs.

An overview of the process

Producing a questionnaire involves a process that starts with being clear about why you are doing it
and what you want to know, and ends with analysing the data, producing findings and getting these
out to people who need to know the information. This guide does not discuss managing, analysing
and interpreting the data, but its important to bear in mind issues concerning analysis and how you
will present the data when you are drafting and designing the questionnaire.
On the next page you will find a chart outlining the steps involved in producing a questionnaire.

Questionnaires the process

The process involves the following steps:
1.

Clarify your purpose

What do you want to know?
Why do you want to know it?
Who is the information for?
Who else will be interested?
What do they want to know?

2.

Define who you want information from

What is your target population?
Do you have a sampling frame, or can you create one?
How important is it to have a statistically representative sample?

3.

Plan your data collection

What is your budget?
What is your time frame?
What are your people resources?
How will you administer the questionnaire?

4.

Develop the questionnaire

Draft the questions.
Consult with others and redraft.
Design the questionnaire.
Pilot the questionnaire and amend.

5.

Collect data
Send an advance invitation wherever possible.
Administer the questionnaire.
Monitor responses.
Follow up non-responses when needed.

6.

Manage the data

Create a coding scheme.
Input the data or export from an online questionnaire.
Check the data for errors and inconsistencies.

7.

Analyse the data and report findings

Describe the distribution and frequency of responses.
Carry out any comparison of subsets.
Identify trends and relationships between variables.
Check for any potential bias.
Use a reporting method best suited for your target audience.
Make the findings available to respondents wherever possible.

Adapted from: Sue, VM and Ritter, LA (2007) Conducting Online Surveys, SAGE, page 2.

10

Drafting the questions

Overview of the drafting process

Before you consider the overall design and presentation of the questionnaire, you should create a
first draft of the questions you want to cover.

Follow these main stages when drafting your questions:

List what you want to cover, including any questions relating to outcome, output and process
indicators as appropriate.

Review the literature in relation to the topic or learn from the work of others.

Consider interviewing a few of your target group to identify issues and suggest the focus and
important areas to include. These preliminary interviews may also be useful to identify issues
about use of language and the way to present and address sensitive issues.

Draft questions together with a suggested response method.

Only include questions you need. Make sure that you dont already hold the data.

Consult with stakeholders, including your users, to review the questions and spot any gaps.

Refine your questions and choose the best response type.

Clarify who is eligible to answer the different parts of the questionnaire and use filter
questions to move respondents to questions which are relevant to them.

Check the logical flow of the questions.

Check that the questionnaire is an appropriate length for its purpose and target group.

Note: Questions related to outcomes refer to change brought about by your organisations
activities and services; questions about outputs concern the activities and services themselves;
questions about processes relate to how you do things, for example, how well you communicate
with users.

Main criteria when drafting questions

Throughout this process, you should be guided by three main criteria:

Questions should be easy to understand and answer, encouraging a response.

Questions should be relevant to the agreed survey aims and objectives.
Questions should be valid.

Ask questions that encourage a response

This means asking questions that are simple, self-explanatory and visually appealing. Avoid
questions that:

are lengthy
ask two or more questions at the same time
are ambiguous or can be misinterpreted
use jargon or unnecessary technical language.
11

Ask questions that are relevant

It may be tempting to use the questionnaire as an opportunity to pick up other information, but this
is likely to make it longer and less appealing to the participants. You will get a better response from
questionnaires that are short and to the point.
If you are assessing your project or organisations outcomes, your questions need to relate clearly to
the outcomes you wish to monitor, so take each of your outcome indicators in turn and think about
how best to express them as questions. For example, if the project is working with unemployed
young people, an outcome may be: Young people are more work ready. You may have the following
indicators:

level of motivation to find work

level of confidence in seeking work.

You could ask:

How keen are you to find a job at the moment? (motivation)

How confident do you feel about applying for work? (confidence).

However, you may get more objective evidence of both motivation and confidence with the following
question:

What steps are you taking to finding a job?

You may then ask follow up questions on having a CV ready, or job applications and interviews.
If you are designing your own questionnaire, you may not be able to test whether your questions
are valid scientifically, but it is worth reviewing questions to check whether they are really likely to
measure what you intend to measure, or to just give related information. For example, be aware that
questions about attitudes or feelings are unlikely to give you information about behaviour: asking
How many pieces of fruit have you eaten in the last week? will provide better information about
healthy eating habits, than How important is fruit in your diet?

Types of questions

There are two main types of questions:

Open questions used for descriptive answers.

Closed questions used to provide quantitative information.

It is often helpful to use a combination of both.

Open questions

Open questions allow people to respond in any way they wish. A question such as: What difference
has our support made to you? gives people the opportunity to provide greater detail about what they
value about your services, or the difference you have made to their lives, than a closed question
would. Open questions can be most helpful to find out about peoples experiences, perception, views
and feelings when used sparingly in self-report questionnaires, particularly when you want to explore
a topic more. Participants can be invited to supplement tick answers with further explanation in a
comment box, or you may have a final question, such as: Do you have any further comments?
However, people often use open questions because they have not fully identified more targeted
questions against which they could get quantifiable responses. Open questions give the respondent
12

a broad remit and the possibility of identifying what he or she considers to be most important. On the
other hand, respondents may also be influenced by what they think is expected as a response, and
may not identify something that is not uppermost in their mind, but significant.
You should recognise the amount of time required to analyse responses to open questions. If you
do use open questions, avoid the temptation to select a few quotes from the qualitative information
provided, but otherwise leave this information unanalysed; this would be to lose the full value of the
exercise.

Closed questions

Closed questions ask the respondent to choose between two or more possible responses, often by
ticking or circling a box.
There are different sorts of response options:
Two options only where there is a choice between two opposing options, such as Yes/No.
Multiple choice these should cover all potential options, which may be ordered or not. By adding
an other option with a text box to explain, this will allow for options that you dont know about, or
avoid a list that is too long. Items on the list should be mutually exclusive. Make it clear whether the
respondent should choose one or many options.
Scaled where a response is made against a continuous scale. The scale may mark frequency or
degree.
Ranked respondents rank options in order of importance. In online surveys you can use the
option to randomise the list, so that it is presented in a different order to each respondent. This will
counteract a tendency for people to choose options at the top of the list.
These are some points to note if you are using closed questions:

Avoid using too many rating scale type questions as people will get tired of them.
Make sure the scale you are using works for each item you are asking people to rate.
Where you are offering a choice of responses, try to avoid giving the impression that there is a
right answer.
Even where your questionnaire uses mainly closed questions, add a final question, such as, Is
there anything else you would like to add?

Scaled questions

Scaled questions are a type of closed question which allow you to gather information in a structured
way. There are several types of scales, which allow measurement against a predetermined rating.
They are often used to assess attitudes and satisfaction levels. Scales can be presented in different
ways:

number scales for example, rating from 1 to 5

symbols or pictures for example, smiley faces
physical objects for example, moving counters on a board
statements for users to agree/disagree with.

The most common scales require a response to a number of statements. There may be a simple
choice between agreeing or disagreeing with the statement. Or, as in the Likert-type scale (a
balanced five- or seven-point scale with a neutral middle point), one of the most popular methods
13

of measuring attitudes, there is a range of possible responses as shown below. Statements may be
worded in a positive or negative manner.
A scoring scheme is usually associated with the response. For example:
Strongly agree = 1
Agree = 2
Neither agree nor disagree = 3
Disagree = 4
Strongly disagree = 5.
A written scale may be replaced by a combination of written and numerical scores. For example:
Strongly agree 1 2 3 4 5 Strongly disagree.
You can use the same scale and rating to compare across time.
Bear the following in mind when using scales:

Avoid using too many different scale types in the same questionnaire.

Express scale rating consistently from left to right, for example, from 1 to 5 or from positive to
negative throughout the questionnaire.

In most cases, use a scale of between four and five points (maximum ten). A detailed
and long scale may be more sensitive than you really need and will require more time for
analysis. Often, when findings from longer scales are reported, detailed scale points are
collapsed anyway.

You may use a central neutral point, but use an even number of scale points if you want
to avoid respondents picking the middle or neutral option. However, note that forcing the
respondent to take any one side may provide invalid data if they really are neutral. You could
check for the effect of your choice when you pilot test the questionnaire.

Explain each point on a numerical scale as this will help users to understand exactly what
you mean and will give you better quality data. It will also be easier to analyse and present.

Allow a dont know or not applicable option to avoid a non-response where these apply. On
the other hand, this may give respondents an option to avoid answering the questions.

In most cases, opt for a balanced scale, with an equal number of positive and negative
statements. If you do choose an unbalanced scale (for example: poor; average; good; very
good; excellent), be aware that this may bias the responses.

Avoid using words such as frequently or rarely. Instead, explain what you mean by using
options such as at least once a week, or less than once a year.

14

Advantages and disadvantages of types of questions

The following table shows some of the main advantages and disadvantages of open and closed
questions.

Advantages of open questions

Disadvantages of open questions

Open questions:

Open questions::

allow respondents to answer freely

may produce detailed responses
dont lead respondents to a suggested
response
require respondents to think about their
answers
can provide insight into respondents true
feelings and views
can produce information on unexpected
outcomes.

may produce unclear responses

require more effort from respondents
resulting in unanswered questions in
questionnaires
produce a large amount of data for
analysis
make it hard to categorise and analyse the
responses.

Advantages of closed questions

Disadvantages of closed questions

Closed questions:

Closed questions:

are simple and quick to answer

tend to get a higher response rate
can help respondents to recall relevant
information
can convey areas of interest to the
respondent
can be easier to quantify and analyse
the responses.

may result in respondents ticking boxes

without thinking about their answers
may influence how the respondents
interpret the question through the answer
options
may not give respondents the choices that
reflect their real experiences, views and
feelings
do not show whether respondents have
understood the question or not
do not allow respondents to explain
responses or raise new issues.

Writing good questions

Good questions are ones that provide valid information against the important areas of enquiry and
more specific indicators, and that allow unexpected data to emerge. The foundation for this is drafting
questions that people can understand and are willing and able to answer.

Common mistakes

Questionnaires frequently have a list of questions that are too long. This is likely to discourage
respondents, so it will be better to split your list into two groups with different subheadings.
Avoid words that are not self-explanatory. For example, Did you find our procedures transparent?
may leave the respondent unsure as to what you really want to know. Sometimes respondents
may be uncertain how to respond to a question that is insufficiently clear. How long have you been
mentoring? could refer to a persons mentoring record with a specific service, or in a more general
sense. Option statements may be ambiguous, as in the following example:
15

In this case, both I feel happy and I feel confident are ambiguous. Does the question ask whether
the respondent feels happy or confident in general, or happy and confident with the relationship?
When you provide some explanation to illustrate what you mean by your statements, be careful that
you dont imply a value judgement. For example, the respondent may feel confused when being
asked to indicate agreement to the following statements:

I feel able to spend more time with positive people (eg, people who volunteer with the local
community).

I feel confident talking to people I dont know (eg, to share my problems and experiences).

In the first of these, this would be better expressed as I feel able to spend more time with people
who have a positive effect on my life. In the second case, the statement is likely to lead to confusion
between confidence in talking to new people and sharing problems and experiences; these are not
the same thing.

Tips for writing questions

It is important to make sure that all questions are relevant to the respondent and/or their organisation,
and can be answered by them without significant time or effort. However, writing good questions is
largely about how you phrase them:

Keep questions short a maximum of 20 words per question wherever possible.

Avoid using leading questions such as, Would you say you feel more a part of the local
community than before? It is better to ask neutral questions such as, How often per week
do you meet with friends locally? or What local activities are you involved in? Alternatively,
provide a statement with a range of agree/disagree options. Remember that you may have to
ask these same questions before and after engagement with a project.

Dont ask a question that contains two separate questions within it. For example, Has your
knowledge of the subject increased and your confidence improved as a result of the course?

Steer clear of any abbreviations, jargon, technical or other unfamiliar terms.

Avoid using double negatives, as in the statement: I dont feel confident when meeting new
people, followed by a response selection of: all the time/sometimes/never.

Relate questions to the present day or a specific, recent, short time period (for example, Over
the past week, how many times have you?)

Questions should be clear, unambiguous and straightforward. For example, Since, you started
with the project, how stressed are you feeling? (less stressed/about the same/more stressed)
rather than Have things improved for you?

Avoid questions requiring or suggesting a simple yes/no answer if you need more detailed

16

information. For example, Has the project helped you? could be replaced by: How have you
benefited from using our services? (Response options: no real benefit; get out of the house
more; made new friends; learned new skills; feel more confident).

When constructing your questions, avoid making assumptions. Ask, Did you gain new skills?
(yes/no) before asking, Please list the skills you have gained; or have a no new skills gained
in the response options.

Make sure responses to most questions are quantifiable. For example, questions should have
boxes to tick, or a choice of numbers on a scale; space for qualitative responses can add to
quantifiable information on a given topic. For example, a tick list against possible areas of
change could be followed by the question: Please tell us more about how these changes have
made a difference to you or Please explain how our services have helped you to make these
changes.

Avoid making sensitive questions appear too intrusive. For example, for questions about debt,
alcohol use or other personal issues, ask for responses in broad categories (such as a range
of alcohol units consumed) with tick boxes.

17

Putting the questionnaire together

The overall structure

As you start shaping your questionnaire, whether it is paper-based or online, start with direct, easy-toanswer questions, leaving more intrusive or complex questions later into the questionnaire. However,
note that if your questionnaire is too long, some people may leave the more complex questions
unanswered. As a general rule, group questions on the same topic together.
The following structure is a guide to ordering your questions.

The opening

In the introduction, make the following clear: the purpose of your survey; details of the organisation
(or possibly a funder) sponsoring it; who is carrying it out (possibly external consultants); and how the
information will be used. Emphasise that the information provided will be treated confidentially.
It is helpful to give an indication of how long it will take to complete the questionnaire. Give clear
information about the closing date and any incentives that you are offering, such as a prize draw,
even if this has been detailed in the covering email or letter. You may also wish to provide a contact if
participants have a query.

Using helper questions

Auxiliary or helper questions will make your questionnaire flow more easily and logically, and make it
more user-friendly.

Introductory questions

These warm up questions will help to capture interest in the survey, and may help move the
respondents into the more important questions, as well as giving you information that you need.

Screening questions

Screening questions find out whether the respondent meets the characteristics that make them
eligible to answer the question, or part of the question. People can receive the same survey but
answer different questions according to their function, or whether they had received different
services.
According to their answer, you may want to direct them to a different part of the questionnaire, as in
the following example:

18

Transition questions

Transition questions help move the respondent from one topic to another. In hard copy
questionnaires, these will follow a question such as: Have you read any of our guidance papers,
available on our website? with: If no, please go to question 6; if yes, please continue with
question 5.
With online questionnaires, you will be able to design the questionnaire to exclude respondents from
certain questions or pages which dont apply, and take them on to the appropriate question based on
their answer using skip logic.

The ending

Profile questions

You will usually need some profile information in order to compare your respondents against the
characteristics seen in the whole population of interest, if you know this information. This will tell you
how representative your respondents are in terms of demographics and other factors. You will also
need this information if you want to compare the responses for different subsets, for example, by age
group, or if your respondents represent organisations, by size or type of organisation.
Such data is particularly important to give responses about outcomes a context, and to help you
interpret and report findings. It will be useful to understand whether the outcomes achieved were
influenced by profile issues, by the frequency or length of their contact with the service or, for
example, whether they were receiving a complementary service from another agency. However, bear
in mind that too many questions may limit the quality and quantity of your responses, so focus on the
most useful information.
People may find personal questions sensitive (particularly if they are about ethnicity, financial
or employment status, for example) so it is helpful to place profile questions at the end of the
questionnaire, making it clear in your introduction to this section why you are asking the questions
and that the data will be confidential. You should also include a prefer not to say option.

Finishing the questionnaire

Thank respondents for completing the questionnaire. With an online survey, you should be able to
generate an automatic thank you email.

19

Your draft questionnaire a checklist

Will the respondent be clear about the survey deadline, how to complete and return it, and
how to answer each question? Are your instructions clear and simple?

Is your language easy to understand? Have you avoided abbreviations and eliminated
unnecessary jargon and technical terms?

Does the questionnaire present as competent and professional? Is it grammatically correct

and consistent in style throughout?

Are your questions short and simple in structure? Have you asked one question at a time?

Is there any danger that your questions could be misunderstood? Are questions, or different
response options, likely to be interpreted in the same way by different groups?

Have you avoided leading and loaded questions?

Have you constructed your response options as positive statements, avoiding negative and
double negative statements?

Have you consistently provided an other category, where none of the options apply?

Have you made a consistent use of rating scales? Will there be any confusion from different
types of response options used? Do lists work in the same way throughout the questionnaire
as far as possible?

Do your questions avoid making assumptions about the respondent and what they may be
familiar with? Have you used screening questions appropriately?

Are the questions and response options relevant to your target audience? Will they have the
information that you ask for?

20

Designing the questionnaire

Allowing enough time

When you are planning your questionnaire, allow enough time to design it once you have drafted
the questions. You will also need time to redraft after you have pilot tested the questionnaire (see
page 24). Not only is the design itself important, but you will find this stage gives you an opportunity
to see if your questions need revising or refining. If you are issuing a paper-based copy of an online
questionnaire for those who cant access the internet, you will need to check that the hard copy will
work well.

Instructions

Provide overall instructions at the beginning of the questionnaire, and specific instructions
for each question. For example, instructions may be simply: Please tick as many as apply
or more complex, as in: Please tell us about how youve been feeling over the last week by
ticking the statements that most accurately reflect how you are at the moment.

For online questionnaires, you may wish to explain if respondents can begin the survey and

return to it later. You can customise the next buttons to be most user-friendly to your target
group.

For online questionnaires, you will have the option to make any question compulsory, which
will prevent respondents moving on before they have answered and a compulsory question
may be necessary as part of filtering and guiding participants through the questionnaire. It may
be tempting to use compulsory questions more widely, but there is also an argument that this
goes against a principle of voluntary participation. It is not something you can do with a paperbased questionnaire, so this needs consideration.

Choosing your response options

The main design options for responses are the following:

Option buttons or boxes used for multiple choice or rating scales.

21

Tick boxes for choosing one or more options from a list.

Rank order matrices. These ask respondents to list items in order of importance or preference.

Open-ended text boxes used for open questions. In online questionnaires, choose the size of
box that will be most appropriate for your question and expected length of response, and keep
the box widths the same throughout.

For online questionnaires, a drop-down menu can be set up for a lengthy list of options.

For online questionnaires, it is helpful to become familiar with the different options for responses
before you start designing your questionnaire.

22

Design tips

There are some simple guidelines that you can follow to make your questionnaire look more inviting
and to get a better response, whether your questionnaire is online or a hard copy.

Designing an effective questionnaire

For online questionnaires, make sure that your welcome screen is welcoming.

Your first questions should be short, easy to answer and interesting enough to gain attention.

Use bold or italics to direct attention to important words.

Check the readability of the font, and any colours used in an online survey.

Use font and font size that will best serve your target respondents.

Limit the line length of your questions and response options.

Map questions out on the page to make sure it is attractive, while bearing in mind overall
length. For longer online surveys, think how to break the questionnaire into separate sections
and pages.

Make your question numbering clear either numbering sequentially throughout the
questionnaire or according to each page.

Use subheadings and clear explanation to guide people through the questionnaire.

For all questions, give simple, exact instructions on how to answer (for example: tick the box;
circle the number; add any comments in the space below).

For online questionnaires, choose the question type carefully.

For both online and hard copy questionnaires, make sure there is enough space for
qualitative answers.

Making the questionnaire accessible to different groups

It will be important to design your questionnaire so that it is accessible, making sure that the
experiences, views and perspectives of different groups are heard, and limiting coverage bias. It may
be useful to consult with organisations that have specific experience of working with particular target
groups to make your questionnaire more accessible, and there may be resources that you are able to
access for example, for people with learning disabilities or dyslexia.
Remember that if your target group is older people, a proportion may have sight difficulties. If you
are using an online survey, check whether you can create a questionnaire in an accessible format for
respondents with visual disabilities. Websurveyor can create surveys for respondents who use screen
readers. Use bright and distinct colours which provide contrast and are the easiest to distinguish.
Blind and partially-sighted people can access electronic documents by using software, and they will
be more accessible than paper-based questionnaires. The software can enlarge text on the screen,
read it out, or show it in braille. But it is important for the questionnaire to be designed appropriately,
and it will be helpful to get guidance on this.
Basic guidance on making printed documents more accessible includes the following:

Use a minimum font size of 12 point. Be prepared to print in 16 point or 20 point.

Use a sans-serif typeface like Arial or Helvetica, avoiding italics and ornate typefaces.
Make sure there is a good contrast between the background and the text.
23

Use a left alignment and allow for generous spacing between paragraphs.
Use bold rather than upper case for emphasis.

Pilot testing

Piloting (sometimes called pre-testing) is the live test of the survey and is really the last step in
finalising the questions and the design of the form; its a vital stage, so dont leave it out.
Ask your pilot test participants to complete the questionnaire and to provide feedback to specific
questions, such as:

How long did it take you to complete the questionnaire?

Did the questionnaire feel the right length?
Were there any questions that you didnt understand?
Were there important response options that were missing?
Did you feel comfortable answering all the questions?
Is there anything important missing?

Use your pilot responses to test out your data entry and analysis as well, to make sure that it will work
as expected before you roll it out. Analysing the responses will help you to see whether the wording
was clear, whether participants understood the questions in the same way and felt able to respond.
Most importantly, the pilot will show whether the questionnaire is producing information that is
relevant to your surveys aims and objectives. The pilot test should give you feedback on the design
as well as the content. Make sure that it doesnt take more than 20 minutes to complete, although
you will want to keep most questionnaires much shorter than this.

24

Administering the questionnaire

An overview

When we talk about administering the questionnaire, we mean how you get your questionnaire out
to your sample, and how you collect the responses and make sure that you get a good response rate.
The context in which you are working will often suggest how and when to administer the
questionnaire. For example, after a training course or a conference, you may be able to distribute a
questionnaire on the spot. On the other hand, you may want time to elapse before you ask questions,
in order to find out about further action that participants have taken as a result of your intervention. If
you are providing an advice service, you may obtain a sample of your clients over a period of time,
asking permission to send them out an immediate and later follow-up questionnaire, taking email
addresses if they have them, or postal addresses where not.
You should consider a number of factors that might allow people to participate or prevent them.

How likely are your potential respondents to have access to an email or internet?
What wider issues might prevent respondents from accessing or responding to the questions?
What particular needs might some groups have, and what can you do to help them to
participate?

There are a number of ways that you can make sure that you get good coverage, for example,
distributing the questionnaire through websites and email lists. If you decide to use an online survey,
you may want the option of printing a paper copy of questionnaires as an alternative for respondents;
there may be a number of reasons why people would prefer a hard copy.
Use intermediaries where helpful to distribute hard copies to named respondents. Think about how
you can work with partners and other agencies to distribute the questionnaire, for example, national
organisations, networks or intermediary agencies that may be able to advertise the questionnaire.
However you distribute the questionnaire, carefully compose a covering text explaining its purpose
and how you propose to use the data, making clear the deadline for response.

Choosing your sample

There are two main issues to consider before choosing your sample:

Do you want to be able to make inferences from your data to the whole population that is, to
present findings that will apply to a wider group than those in your sample?
What size do you want your sample to be?

Type of sample

If you want to present conclusions about your whole population based on findings from the sample,
then you will need a probability sample, drawing a random selection of participants from a defined
sampling frame (see page 6).
You may not be able to use probability sampling because you cant draw up a sampling frame
including the whole population; this will be the case with most online surveys that are not available to,
or accessed by, all your target audience. However, organisations frequently obtain information about
outcomes from a convenience sample, which is a type of non-probability sample, allowing people
to self-select to take part. You can still get valuable information and useful insights from this type of
sampling.
25

Whichever type of sample you use, it will be important to get as good coverage as possible, and
to make the responses as representative as you can. In some cases you may be able to use a
snowball approach to reach groups for whom you have no direct access by asking individuals or
organisations to refer your survey on. This can be useful to access hard-to-reach groups.
If you ask respondents to provide information about their characteristics (for example, where they
live, age, gender, other relevant characteristics), it may be possible to monitor the response rate from
particular groups; you will then be in a position to take specific action to encourage responses from a
group for which you notice there is a low response.

Size of sample

When choosing the size of your sample, assume anything between a 15 per cent and 40 per cent
response rate. There are no absolute guidelines for choosing a sample size for non-probability
samples, but the general rule would be to reach as many as you can within the time and resources
you have available, selecting the largest sample you can afford. However, if your sample size is too
small, it may call into question how reliable and useful the data is. Most guidance points to a sample
of 30 being the minimum that would be useful, while you may not need more than 500 while trying
to reach at least 10 per cent of your overall population.
Remember that if you want to analyse results for subsets within the sample (for example, those who
received different levels of service, or by age range, ethnicity or other demographic), you will need a
larger sample, so that there are enough responses in each subset.

Using online surveys

Bear in mind some of the advantages and disadvantages of using an online survey. These are set out
in the table on the next page.

26

Advantages of online surveys

Disadvantages of online surveys

They are cost efficient.

They are environmentally friendly.

It may not be efficient for a smaller group

of respondents.

There is a quick response time.

You should be able to import a respondent

list from another software programme.

Responses will be limited to those who

have internet access.

You should be able to import your email

address book.

You should have the option of sending

pre-notification emails and thank you
emails.

It may not be appropriate for feedback

that is close to the event or required
quickly.

Responses may be limited by your

sampling frame (for example, if you are
assessing need, using your contact
database will exclude other potential
users).

Care will be needed to maintain

anonymity.

You can save the questionnaire as a

template to use again.

The survey will be able to compute basic

statistics.

You will be able to export data to statistical

analysis software for more complex data

analysis.

It may not be appropriate for specific

groups or types of respondents.

They may not be appropriate for sensitive

information, where face-to-face interviews
based on a structured questionnaire might
be better.

They allow many design options, question

formats and response options.

People are more likely to answer

open questions in depth in an online
questionnaire.

Other methods (such as groupadministered questionnaires) may

encourage a greater response.

People may find it easier to answer

sensitive issues.

You will be able to track responses.

It may be less appropriate for longitudinal

studies as the response rate will be
difficult to control.

They are easier to use for skip logic.

You need to ensure that participants

respond only once.

Online questionnaires have many advantages and, if you are aware of the potential difficulties,
you can take steps to reduce their effects. For example, you can assign a unique identifier to each
respondent to prevent people from entering multiple responses.

27

Ethical considerations
Informed consent

Respondents should be given enough information to allow them to make an informed choice as to
whether to participate. Your questionnaire participants should receive the same ethical consideration
as if they were involved more deeply in a research study.

Informed consent in questionnaires

Participants should be clear:

what the survey is about

who is sponsoring the survey
how the data will be used and that their data will be confidential
how long it will take them to complete the survey
if they will be required to provide any further information
what benefits or discomfort/risks there may be (for example, if they are asked to give
sensitive information)
that participation is voluntary.

You can use your invitation letter or email to provide this information. It should also be contained in
the introduction to the survey itself.
The Data Protection Act 1998 requires you to make it clear in the introduction who is carrying out
the survey and how you will handle the data confidentially. For advice on data protection call the
Information Commissioners Advice Line on 01625 545745 or visit www.ico.org.uk

Confidentiality and anonymity

It is helpful to think about confidentiality and anonymity in advance, and it should really be part of your
early planning. Confidentiality (not disclosing the information in an unauthorised way) and anonymity
(making sure respondents are not identified) will require separate consideration.
In many cases, respondents will want to know that their responses are confidential, and you should
make proper arrangements to protect the data. When you report results, there are many situations
where you may risk jeopardising confidentiality, for example when using quotes, or case examples,
but also potentially with quantitative information. Survey results from small groups present difficulties,
and you should avoid presenting results from groups of less than ten.
In order to understand your responses better, its often important to gather some demographic
information from your respondents. You may want to know who has filled in your questionnaire, and
check if responses correspond to the profile of your user group, or other group(s) you are targeting.
Or you may wish to analyse the data by respondents age or other demographic to see if there are
any differences in the information provided. Very often you wont need to identify who has responded,
so you can reassure people that their responses will be anonymous.
However, you may want to follow up responses, for example by a short telephone interview. In this
case, you will need to have names and contact details, but you should always give people the option
not to provide this information. You may decide to code questionnaires with an identification number
so that you are able to connect information to individuals if you need to. You may be able to connect
answers to demographic or other information you already hold, for example, in a membership
database or case records. So, dont ask people to complete information that you are already able to
28

access elsewhere.

Getting a good response rate

There are no absolutes about what will be a good or an acceptable response rate, as this will depend
very much on the circumstances in which you are issuing the questionnaire: the better you know the
respondents, the more likely you are to have a higher response rate. Although you might want to aim
for a 50 per cent to 60 per cent response rate, research shows that response rates for email surveys
range between 24 per cent and 76 per cent, while for web-based surveys the response rates are
approximately 30 per cent lower or similar to the response rate for postal surveys.
A poor response rate means that you are introducing bias. Remember that those who return
questionnaires most readily are likely to be those that have either a really positive, or a really
negative, viewpoint. You need to proactively search out and encourage those who may have views
less fixed either way, or who do not normally have their voice heard, to return the questionnaire.
Response bias is a particular concern with online questionnaires as samples will be biased towards
those who have online access and are comfortable with the medium.
There are a number of ways that you can encourage a good response rate:

Whether you are distributing a postal questionnaire or an email questionnaire, write a brief
invitation to accompany the questionnaire. You may also contact your sample before to let
people know that the questionnaire will be on its way and invite them to complete it.

If you are posting a link to the questionnaire on different websites, or through electronic
newsletters and bulletins, include key points in a short paragraph.

Make it clear who you are.

Explain the purpose clearly.

Explain why the survey is important.

Tell people how long it will take to complete.

Offer assurances about confidentiality and anonymity.

Use plain English.

Keep the explanatory text short and simple.

Thank the reader in advance.

Provide contact details, for example, if respondents want an accessible version.

Once the questionnaire is live, check your responses regularly against your planned satisfactory
response rate. You are likely to increase the response rate by follow-up invitations, reminding people
of the deadline and briefly capturing the purpose of the questionnaire again. You may decide to
extend the deadline if you feel additional time is likely to have useful results. There are different
opinions as to whether reminders should be sent more than once, but a second reminder is definitely
a maximum.
If you are sending email invitations, you can send the reminder to everyone on a distribution list,
or you can remove the addresses of those who have completed the questionnaire, and send the
reminder only to those who havent yet responded. For online surveys, your software should allow
you to track the email address or name associated with each response.
If you add a unique number or identifier to each questionnaire, this may help you in tracking or
identifying your participants.
29

Using incentives

The main incentive to participate is that respondents feel that it will be in their own interest or that
of their organisation (although this may not motivate them so much). Illustrate how the survey is
important, for example to develop a better understanding of needs, to improve the service and
promote better outcomes, or to get new funding; tell respondents if important decisions will be
made on the basis of the findings. Reassuring them about how the data will be used and about
confidentiality will also help.
Research has shown that more obvious incentives are also effective for increasing response rates
for online surveys, so make it clear in your publicity for the questionnaire if respondents can enter a
prize draw, for example for gift vouchers. Choose your incentive carefully and make sure that it will be
appropriate and valued by the audience. You may be able to offer them vouchers to attend training,
or a free publication or other resource.

Thank-you notes

Remember that you may want to survey the same people again, so offering to send a summary of the
survey findings, and saying thank you is not only good practice, but may help future responses.
If you are using a web-based survey host you can write a thank you email and have it sent
automatically soon after the respondent completes the questionnaire. Another option is to wait until all
the responses are returned and then send the thank-you message to the entire list at once.
If you are using an online survey, let respondents know that once they click the submit button, their
responses have been submitted. You may also be able to redirect them to your own organisations
website.

Factors affecting response rates to questionnaires

Positive

Negative

The questionnaire is sent direct to the

person/department who will respond, with
a clear explanation of its relevance.

The questionnaire is distributed with

blanket coverage, with the hope that
enough appropriate people will see it.

The questionnaire looks attractive and

straightforward to complete.

The questionnaire looks messy, crowded

and complicated to complete.

The questionnaire is short and to the

point.

The questionnaire is lengthy and allembracing.

All questions are relevant to the

respondents and their organisation.

Many of the questions do not apply to the

person/organisation responding.
(cont. on next page)

30

Respondents can relate strongly to

the topic or the concerned project/
organisation.

Respondents dont feel invested in the

topic or project/organisation and are
reluctant to respond.

The organisation/individual can see a

benefit to them perhaps in terms of
service improvement or personally (like a
prize draw).

The organisation/individual can see no

benefit to them from responding and
providing the information.

The questions are lengthy, complicated

and ambiguous.

The information is not available without

a great deal of time and effort spent on
obtaining it.

The questions are clear and unambiguous.

The information to complete the form is

easily retrievable and accessible.

The questions are received as neutral and

objective.

The questionnaire includes clear

instructions for completion, and response
times are realistic.

The questionnaire includes the deadline

for responses and any return details.

The questions make assumptions, for

example about the organisation.

Instructions for completion and deadline

for responses are not clear.

There is inadequate time to get enough

responses.

31

Using the data from questionnaires

Analysing the data

Designing and administering your questionnaire, and getting a good response, is only the first half of
your survey process. When you were at the planning stage, you should also have considered how
you were going to analyse the data, write up the findings and distribute them.
Many organisations fall into a trap of attaching less importance than necessary to this stage. Even
if you choose an online survey and have the statistical analysis done for you automatically, you still
need to decide about any subset analysis, and have to carry out the analysis of the qualitative data.
Unless your questionnaire is quite short, it is likely to be useful to cross-tabulate relevant questions to
test a specific hypothesis, for example, whether those clients who had been attending the project for
longer than one year had better outcomes. You may also want to make a demographic comparison
between responses, testing whether there was any difference for different age groups, education
levels, or geographical areas, for example.
You need to consider issues to do with your sample, your overall response rate and that for individual
questions, and who responded. How well did the respondents reflect the overall population? It will
be useful to present this information alongside your survey findings. You should also consider how
questions were interpreted, and which ones gave you low response rates and how this affects your
ability to meet your questionnaire aims and objectives.

Interpreting the data

When you designed the questionnaire, you did so to make it as relevant as possible. But its also
important to check, when you are analysing and interpreting your data, to what extent you obtained
valid results, and to look at any likely bias that might have been introduced. Bear in mind how survey
error might have biased your data.
Your findings need to be clear and precise. You need to be careful not to misinterpret and make sure
that your findings are a true reflection of what the question asked and the response to that question.
For example, if you have asked whether respondents spend more money on social activities, dont
report responses as if they relate to the amount or quality of social activity.
There is a frequent tendency to smooth over results. A common error is to claim a broadly positive
result because a majority of responses were positive, without taking into account the size of the
majority, or without looking at what might be a significant minority of negative responses. For
example, Only 38 per cent of pupils reported very little change or no change in their behaviour could
be misleading because of the use of the term only. You need to ask how important this frequency is
and what it might mean.
It will also be important to identify in what conditions positive and negative responses were given.
If you are looking at changed knowledge, skills, attitudes and behaviour, for example, ask which
participants are showing this change, and clarify their original starting point.

What is a survey error?

A survey error is when the questionnaire results do not provide valid evidence about the population.
There will be some element of bias in any sample simply because of the natural level of variance
between any two samples. With a bigger sample, you can be more confident that this bias or error
will be less likely. This confidence level can be established for probability samples.
32

For either type of sample, you should consider both coverage and non-responses.

Coverage

Your original sampling frame may not have fully represented the population of interest. This may
happen when you dont have access information (such as postal addresses or email addresses) for
some of the relevant population, and they cant be included in the sample. Sometimes questionnaires
are accessed by people picking up a form as they come into an organisation over a period of
time; this will exclude those who did not visit, but who may be an important element in your target
group. For online surveys, people may not have access to the internet, and these people may have
important differences from those that can access it.

Non-responses

You may include potential respondents as part of your sample, but they choose not to participate:

They might choose not to respond at all.

They might skip particular questions.
They might close the questionnaire without completing it.

Those who dont participate or respond selectively may have different characteristics from those who
do participate, or participate fully a difference that you will not be able to capture.

Writing the report

Once your data has been analysed, the final important step is communicating your findings in a wellstructured report. Your report should contain all that people need to know about your survey and its
results. You will also possibly use other, briefer formats and a variety of media for communicating
headline results. Consider how you can report the results back to the survey respondents.
There are three main components of the report:

Background to the report, including the purpose of the survey, who carried it out and when,
sampling details and other information about collection methods.

A detailed breakdown of the results, highlighting data of particular significance. Use charts
and tables to make it easier to view the results, particularly important points, using descriptive
text to explain your charts. Look through all your charts to make certain that they offer a clear
illustration of the findings. Make clear if all or some of the respondents were asked particular
questions and what the response rate was for each question. Dont confuse the frequency of
responses with popularity or importance.

Present results against the precise question asked; be careful to present chart titles that
properly represent the question. At the foot or top of a table, show the total raw respondent
numbers and the total percentage.

Conclusions and recommendations suggested by the data. These should be expressed

clearly and concisely so that management decisions can be made and action plans built
around them.

It will be helpful to summarise your key findings in a short executive summary, which is usually
attached at the front of the report. You may also want to attach the full questionnaire as an appendix
to the report.

33

Evaluate and learn from the process

Make sure that you record some of the learning about what worked and what didnt as you went
through the process of drafting the questions, involving other people, designing the questionnaire and
administering it. Your pilot testing should have picked up on a number of learning points, but more will
become apparent when you do your analysis, so make sure that you take note of these.
Each time you carry out a questionnaire, it is likely to present its own challenges, but each time you
will become more skilled and are likely to get better results.

34

Glossary
Aims describe the changes you plan to achieve in your user or target group.
Closed question (or closed-ended question) is a question in which the respondent is given a choice
of replies.
Confidence level is an expression of how confident a researcher can be of the data obtained from
a sample. It is often expressed as a percentage; the most commonly used confidence level is 95 per
cent.
Leading question is a type of question that is phrased in a way that suggests what the answer
should be or indicates a certain point of view.
Evaluation involves using monitoring and other information to make judgements on how your project
is doing.
Inputs are the resources you put into an organisation to carry out an activity. Inputs may be human,
material, financial or expressed as time.
Indicators are well-defined information which can be observed or measured. They can be used to
assess whether outputs, outcomes and impacts are being achieved.
Impact is the long-term, broad and sustained change resulting from your activities and services, and
may relate to a wider population than your target group. Impact is likely to relate to your overall aim,
although there may be unexpected or negative impacts.
Monitoring is the routine, systematic collection of information for the purpose of checking your
projects progress against your project plans.
Non-respondents are individuals who have been invited to participate in a survey and do not
respond; it is not possible to draw conclusions about why there is non-response to the questionnaire
or any of its questions.
Open question (or open-ended question) is a question that does not include a list of possible
responses, and the respondent is free to answer in full.
Objectives describe the planned areas of activities by which you are going to achieve your aims.
Outcomes are all the changes, benefits, learning or other effects that happen as a result of your
activities. Outcomes can be end or intermediate (also called interim outcomes). Intermediate
outcomes are steps along the way to end outcomes. They are often smaller changes that happen
before the final, desired outcome can be reached.
Population is the total group of respondents of interest.
Pilot test (also pre-test) is a live test of the survey by a small group of respondents before it is
distributed more widely. Feedback on the content and design of the survey tool will allow problems to
come to light.
Reliability is the extent to which a measure provides consistent results across repeated testing.
35

Respondents are the individuals who participate in the survey.

Response option is a possible answer to a closed-ended questions.
Sample is a subset that is derived from a target population which is used to represent the whole
group.
Sampling frame is the source from which a sample is taken. It will list all the elements in the
population which can be sampled and may provide information which will help to provide a
representative sample.
Survey error is when survey results do not provide valid evidence, because bias has been
introduced through the sample and coverage, or through non-responses.
Target population is the entire group of possible respondents. Because it is improbable that you
will survey every individual in your target population, you must survey a smaller subgroup of your
population, known as a sample.
Validity refers to whether the measurement tool accurately and appropriately measures what it is
supposed to measure.
Web survey is a type of survey methodology in which a link is provided to a web host that takes the
respondent to an HTML form that is completed and submitted through the computer. Web surveys
can also be embedded on an organisations own website.

36

Further information and reading

CES services

CES is committed to improving the effectiveness of the voluntary sector by developing its use of
evaluation and quality systems. We provide a range of training courses, consultancy, support and
publications. For more information, contact CES on: 020 7713 5722 or visit our website:
www.ces-vol.org.uk

Other publications from Charities Evaluation Services

CES has produced a wide range of guides on monitoring and evaluation. Our publications list has full
details.
You might find the following particularly helpful:

Guide

Description

The CES Resource Guide: Evaluating outcomes

and impact (2013)

A guide to over 130 resources on monitoring

and evaluating outcomes and impact, including
approaches, methods and creative tools, subsector specific guidance, and links to useful
websites containing further resources.
Free to download:
www.ces-vol.org.uk/informationcollectionmethods

Assessing Change: Developing and Using

Outcomes Monitoring Tools (2010)

Practical handbook to help you identify and

develop ways to collect information on the
outcomes of your work.
Free to download:
www.ces-vol.org.uk/assessingchange

Information collection methods: Choosing tools

for assessing impact (2013)

This practical guide will help you think through

how to collect the right data to demonstrate the
outcomes and impact of your work.
Free to download
www.ces-vol.org.uk/informationcollectionmethods

First Steps in Monitoring and Evaluation (2002)

A basic guide for organisations who are looking

at monitoring and evaluation for the first time.
Free to download:
www.ces-vol.org.uk/firststepsme

37

Next Steps: Monitoring and Evaluation on a

Shoestring (2011)

A practical guide to help organisations develop

their monitoring and evaluation approach and
improve their effectiveness on a limited budget.
Free to download:
www.ces-vol.org.uk/shoestring

Practical Monitoring and Evaluation: A Guide for

Voluntary Organisations, 3rd edition (2009)

Comprehensive guide to monitoring and

evaluation in the voluntary sector, now in its third
edition.
Available from:
www.ces-vol.org.uk/practical

Your Project and its Outcomes (2007)

A guide focusing on how to identify outcomes

and their indicators and how to assess their
achievement. It uses case examples throughout
to demonstrate different stages of the process
and different types of organisation.
Free to download:
www.ces-vol.org.uk/freedownloads

Keeping on Track: A Guide to Setting and Using

Indicators (2008)

A guide to help third sector organisations

improve their skills in setting output, outcome
and process indicators.
Free to download:
www.ces-vol.org.uk/ontrack

Using ICT to Improve Your Monitoring and

Evaluation (2008)

This work book covers the steps and issues

you need to consider in developing appropriate
computer systems that will help you monitor and
evaluate your work.
Free to download:
www.ces-vol.org.uk/freedownloads

Making Connections: Using a theory of change

to develop planning and evaluation (2011)

This guide provides an introduction to the theory

of change approach to planning, monitoring and
evaluation, explaining why and how to use it.
Free to download:
www.ces-vol.org.uk/theoryofchange

38

Discussion papers

CES series of papers on evaluation covers

seven important topics, including outcomes
monitoring and assessing impact.
Available from:
www.ces-vol.org.uk/discussionpapers

Other publications
Barbour, R (2013) Introducing Qualitative Research: A Students Guide, Second edition, SAGE.
Blair, J, Czaja, FR and Blair, EA (2013) Designing Surveys: A guide to decisions and procedures,
Third edition, SAGE.
Kumar, Ranjit (2014) Research Methodology: A step-by-step guide for beginners, Fourth edition,
SAGE, London.
Sue, VM and Ritter, LA (2007) Conducting Online Surveys, SAGE.

39

Charities Evaluation Services

4 Coldbath Square
London EC1R 5HL
020 7713 5722
www.ces-vol.org.uk
[email protected]
Charity number in England and Wales 803602
Charity number in Scotland SCO40937
Company limited by guarantee
Registered in England and Wales number 2510318

40