SURVEY GUIDE

SURVEY FUNDAMENTALS
A GUIDE TO DESIGNING AND IMPLEMENTING SURVEYS

1
 SURVEY GUIDE

OFFICE OF QUALITY IMPROVEMENT

SURVEY FUNDAMENTALS

This guide describes in non-technical terms the underlying principles of good survey
design and implementation. Clear, simple explanations lead the reader through
methodology and logistics decisions, writing effective questions, and drawing
conclusions from the results. The guide also provides tips for maximizing the
response rate as well as an overview of human subject rules.

The material presented here is based on short courses presented by staff of the
University of Wisconsin Survey Center in 2008 in honor of the UWSC’s 25th
anniversary. The course materials were developed by Nora Cate Schaeffer,
Jennifer Dykema, Kelly Elver, and John Stevenson. Nancy Thayer-Hart compiled
this guide based on those courses, supplemented with additional material.

Version 2.0, December 2010

Copyright 2010 University of Wisconsin System Board of Regents

Contributors:
Nancy Thayer-Hart, Office of Quality Improvement
Jennifer Dykema, Kelly Elver, Nora Cate Schaeffer, John Stevenson, University of Wisconsin Survey Center

For additional information contact:

Office of Quality Improvement

Room 199 Bascom Hall, 500 Lincoln Drive
Madison, WI 53706-1380
608-262-6843 FAX: 608-262-9330
[email protected]
http://www.quality.wisc.edu

University of Wisconsin Survey Center

1800 University Avenue
Madison, Wisconsin 53726
608-262-9032 FAX: 608-262-8432
[email protected]
http://www.uwsc.wisc.edu
 SURVEY GUIDE

CONTENTS
Contents ............................................................................................................... 3
Survey Basics ....................................................................................................... 4
Design the Survey Process .................................................................................. 4
What are the goals? .......................................................................................... 4
What is the target population? .......................................................................... 5
What is the timing?............................................................................................ 5
What mode will be used? .................................................................................. 5
Develop Questions ............................................................................................... 6
Reliability ........................................................................................................... 6
Validity............................................................................................................... 6
Tips for Writing Good Questions ....................................................................... 7
Response Format ............................................................................................ 10
Design of Self-Administered Questionnaires .................................................. 11
Checklist for Effective Questionnaires ............................................................ 12
Test and Train .................................................................................................... 12
Collect Data ........................................................................................................ 14
Response Rate ............................................................................................... 14
Follow-up Procedures ..................................................................................... 14
Web Survey Challenges.................................................................................. 15
Analyze Data ...................................................................................................... 15
Coding and Analyzing Data ............................................................................ 15
Drawing Conclusions ...................................................................................... 16
Additional Considerations ................................................................................... 16
Human Subjects Protection............................................................................. 16
A Word About Climate Surveys....................................................................... 17
Getting Help with Your Survey ........................................................................... 17
Terminology ........................................................................................................ 18
References and Additional Resources ............................................................... 19
Works Cited..................................................................................................... 19
Additional Resources ...................................................................................... 19

3
 SURVEY GUIDE

SURVEY BASICS
A survey is often the best way to get information and feedback to use in
planning and program improvement. This guide is written to help you achieve
the results you need. The principles and practices described here have been
recommended by the University of Wisconsin Survey Center.

Designing and implementing a survey is a systematic process of gathering

information on a specific topic by asking questions of individuals and then
generalizing the results to the groups represented by the respondents. The
process involves five distinct steps. Figure 1 depicts the process of designing
and implementing a survey:

Design Survey Develop

Test & Train Collect Data Analyze Data
Process Questions

Figure 1. Survey Process

The type of survey you use for a particular purpose will be the result of decisions
you make about:
1. Contacting potential respondents (in-person, telephone, mail, email)
2. Presenting the questions (written, interviewer)
3. Recording responses (paper, electronic)

Your choices will be influenced by your research goals and timeline, how
sensitive or complex the study topic is, the characteristics, abilities and
resources of potential respondents (e.g., their access to and experience with
technology), and of course, your budget. The choices you make will affect the
quality, cost and timeliness of your results.

DESIGN THE SURVEY PROCESS

WHAT ARE THE GOALS?
The first step in planning for your survey is to determine the goal and how you
will use the information. The usual goal of a survey is to describe a population,
for example, the group of people using your services. Another common goal that
may require a very different study design is to make comparisons between
groups, such as comparisons between those using your services and those who
do not. Often surveys are used simply to get feedback from a specific group of
people and/or learn about their wants and needs.

4
 SURVEY GUIDE

WHAT IS THE TARGET POPULATION?

Who are the people you’re interested in hearing from? What criteria separate
those who should be included in the survey from those who should not?

When the target population is small and easy to contact, it is often feasible to
survey the entire population. However, when the target population is large or
widely dispersed and the survey is part of a rigorous research project, a
“sampling” process is typically used to select and survey only a subset of the
total population of interest. If a sampling process would enhance the results of
your survey, you will want to engage the services of a sampling statistician to
guide you.

One reason to survey a sample rather than an entire population is that with a smaller
sample size, you can include more follow-up contacts to encourage responses. As a
result of these efforts, participation and data quality may be improved.

In order to reach your target population, you will need current and accurate
contact information. Obtaining and verifying this information can be costly and
time-consuming, especially if your target population is highly mobile and/or
difficult to locate.

A service for mass emailing to campus audiences is available through Division

of Information Technology (DoIT). See http://www.doit.wisc.edu/lists/massemail/index.asp
for additional information. There is a fee for using this service.

WHAT IS THE TIMING?

Developing a timeline is a crucial step in planning your survey. When is the
survey data needed? When would be the best time to contact potential
participants? Do you want to survey during a break or at a time when
respondents are engaged in another activity with your unit? If your survey is
going to a campus population, you may want to avoid coinciding with one of the
periodic campus-wide surveys.

What other workload issues may affect those involved in sending the survey and
collecting and analyzing the results? How long will it take to design and obtain
approval for the survey? How much time is needed for collecting responses?
The answers to all these questions will help you develop a realistic timeline.

WHAT MODE WILL BE USED?

As noted above, there are a variety of options for how you conduct your survey.
The combination of choices you make about contacting sample members,
administering questions, and recording responses is called the “mode.” The
mode is influenced by the survey population, the study topic, and how you plan
to use the information you gather.

5
 SURVEY GUIDE

DEVELOP QUESTIONS
A survey question is a measuring device for things that are not directly
observable, in the same way that a bathroom scale is a measuring instrument
for weight. In order for your survey results to be useful and meaningful, the
questions you ask must have two characteristics: reliability and validity. These
terms are defined below.

Writing good survey questions requires keeping the goal of the survey firmly in
mind and then formulating each question from the perspective of the
respondent. It may be tempting to ask questions simply because it would be
interesting to know the answers, but if they are not essential to the goal of the
survey, such questions can actually detract from your survey results.
Unnecessary questions distract respondents or cause confusion.

A useful strategy for judging whether you have the right questions is to create
the tables for the final report before finalizing the survey. Thinking about how to
array the survey results will highlight extraneous questions and perhaps point
out areas where additional questions would strengthen your survey.

Respondents’ abilities to provide accurate and useful information are enhanced

when they can immediately understand what is being asked and the purpose of
the question. Respondents can then more easily recall the situation, event, or
information they are being asked about and decide how frank to be. In the “Tips
for Writing Good Questions” section below are general rules for framing
questions in a way that shortens the path between question and answer.

The placement of a question within the survey also has an impact on the results.
For example, respondents may have difficulty recalling specific dates or times.
If this information is absolutely necessary, reserve the question for the end of
the survey, when other questions have gotten the respondent recalling and
processing.

RELIABILITY
Just as you want to be able to rely on your bathroom scale to always give the
same reading if your weight is unchanged, you want your survey questions to be
reliable. Reliability is the extent to which repeatedly measuring the same
property produces the same result. Ideally, each survey question will mean the
same thing to everyone, including those administering the survey. This takes
careful design and refinement.

VALIDITY
Validity is the extent to which a survey question measures the property it is
supposed to measure. For example, a yardstick would not produce a valid

6
 SURVEY GUIDE

measure of the weight of an object. Your bathroom scale is more likely to

produce valid readings, but if it’s old and abused, the readings may be
systematically inaccurate.

TIPS FOR WRITING GOOD QUESTIONS

Whenever possible, build on what’s been done before. Somewhere, sometime,
someone has tried to get the same or similar information. Access previous
surveys conducted by your unit or elsewhere on campus or investigate online
repositories of questions so you don’t have to start from scratch.

An effective survey question provides the respondent with a context for the
question by announcing the topic and defining the timeframe for events or
behaviors that are to be included in the response. For example, “This question is
about commuting to work. When answering, please consider only events that
occurred during the fall semester.” This pre-question information or context is
known as the “preamble.”

The preamble should also explain what the respondent is being asked to do
(e.g., check one, list all) and define any concepts or terms that the respondent
needs to understand in order to answer the question. The preamble is an
important means of ensuring the reliability of the question.

The question can focus on an event or behavior that the respondent might have
observed or participated in, or it can inquire about their attitude, evaluation, or
judgment. The topic will determine the appropriate “response dimension.”

Common response dimensions for questions about events and behaviors include:
Opportunity to experience or know something
Occurrence of an event or behavior within a defined time period
Frequency (counts or rates)
Regularity (time interval)
Duration
Date or timing

Common response dimensions for questions about attitudes and judgment include:
Degree of attractiveness (like/dislike, favor/oppose)
Satisfaction
Intensity (a little, somewhat, or very much)
Certainty
Importance

Asking good questions is not as easy as it seems, and the unfortunate result of
ineffective questions is bad data. The resource list at the back of this guide

7
 SURVEY GUIDE

includes several basic texts that can be very helpful in writing good questions
(see, e.g., Schaeffer and Presser (2002), Fowler and Cosenza (2008), and Aday
and Cornelius (2006)). Some general rules for writing effective survey questions
are summarized below.

Question Wording
Avoid using complex words, technical terms, jargon, and phrases that are
difficult to understand. Instead, use language that is commonly used by the
respondents. For example:
Use . . . Instead of …
 Work  Employment
 Tired  Exhausted
 About  Regarding
 People who live here  Occupants of this household
 Your answers  Your responses to this questionnaire
 Job concerns  Work-related employment issues
 Providing health care  Health care provision

Other wording practices to avoid:

 Shorthand (contractions, abbreviations, symbols, slashes,
parentheses, brackets)
 Framing questions in the negative - how frequently do you not attend
classes?
 Using double negatives - do you agree or disagree that students
should never not go to class?
 Passive voice – how often were grade reductions made by your
teachers for absences?
 Words or phrases with a strong point of view

Question Structure
Just like the words and phrases used in your questions, the structure of the
question should be simple and easy for respondents to comprehend. Questions
should have only a single subject and verb, and should not combine two
questions into one.

Questions can become “double-barreled” when the word “or” is used and also
when two different types of response options are tucked into a single question.
For example, In what year was your husband born? Check here if not currently
married, asks for both the husband’s date of birth AND for whether the
respondent is married.

8
 SURVEY GUIDE

Refrain from asking questions with multiple response dimensions (e.g.,

regularity AND frequency, occurrence AND frequency). Separate these into two
separate questions.

Questions that start with a statement and ask the respondent to agree or
disagree with the statement (or answer true or false) may give the impression
that you are expecting a certain response. Instead, rephrase the question to
include the response dimension: How important is . . .? How sufficient is . . .?
How effective will . . .? Then supply appropriate response categories, such as
not at all, a little, somewhat, quite, extremely.

Maintaining a parallel structure for all questions immensely improves

respondents’ ability to comprehend and effectively respond to the survey. Use
the same words or phrases to refer to the same concepts. For example, don’t
mix “how old were you,” “at what age,” and “when.” Pick one phrase and stick
with it, and always position repeated words and clauses in the same place and
in the same order.

Where a definition would help to clarify what the question is about or to describe
what should be included in the response, provide the definition in the preamble,
before asking the question.

Question Type – Open or Closed?

Closed questions provide a list of acceptable responses. Checklists, multiple
choice questions, true/false questions, and attitude scales fit this category.
Respondents may find it easier to answer the question when response
alternatives are provided, and it is easier and less time-consuming to interpret
and analyze the responses to closed questions.

Open-ended questions such as “What is the most important problem facing our
department today?” allow respondents to answer in their own words. There are
several advantages to giving respondents the opportunity to use their own words
to express themselves:
 Respondents may give increased amounts of reflection to their
answers
 Unanticipated answers may emerge
 The respondent’s views may be described more accurately

However, some answers to open questions may be difficult to interpret and

answers may be spread over many categories, making it hard to analyze them.

Reference Period
The reference period is the time frame the respondent is being asked to
consider when answering a question. The reference period needs to be made

9
 SURVEY GUIDE

explicit, rather than leaving each respondent to make an assumption about how
far back they should try to remember.

It’s best to use specific dates rather than referring to “last week” or “last month,”
and the length of the reference period needs to be appropriate to the behavior or
event the respondent is being asked to recall. It might be difficult for many
people to recall how many times they’ve eaten in a restaurant during the past six
months, but most of us could be pretty accurate about how many times we ate in
a restaurant during a recent two-week period described with specific dates.

The place to announce the reference period is in the preamble at the beginning
of a section of related questions and whenever the reference period changes. It
is also helpful to repeat the reference period in abbreviated form as you ask
each question in the series.

RESPONSE FORMAT
Questions must match the desired format of the answer. For example, if you are
providing separate blanks for the city and the state of residence, ask In which
city and state do you live? rather than Where do you live?

If you provide response categories for a question, make sure they are
exhaustive – i.e., all respondents can identify one that fits (less than 20 years,
20-60 years, over 60 years). Categories should also be mutually exclusive so
that respondents can easily select only one response (0-9 and 10-19 rather than
0-10 and 10-20). Ensuring that response categories are both exhaustive and
mutually exclusive may require re-evaluating the intent of the question or
separating it into two or more questions, or both.

Response categories labeled with words (dissatisfied, satisfied) give more

reliable data than categories labeled with numbers (1, 5). The researcher can
always assign numeric values later to make it easier to analyze responses.
Categories should be fully labeled with exactly what each point on the scale
means (very dissatisfied, somewhat dissatisfied, neither satisfied nor
dissatisfied, somewhat satisfied, very satisfied).

For questions that only ask about one dimension (e.g., how satisfied are you?),
five fully labeled categories are usually sufficient. For bipolar questions (e.g.,
how satisfied or dissatisfied are you?), use seven fully labeled categories. The
midpoint should allow for mixed or in between feelings, whichever is most
appropriate.

When measuring the frequency of an event or behavior, ask for the actual
number of times the event or behavior occurred during the reference period,
rather than providing categories like often, seldom, never.

10
 SURVEY GUIDE

To measure occurrence, ask for number of times per day/week/month, rather

than asking respondents to select a category (several times a week, once a
week, once a month). However, if the event in question is difficult to count or
estimate, response categories (zero times, 1-3 times, 4 or more times) may help
respondents provide an answer.

DESIGN OF SELF-ADMINISTERED QUESTIONNAIRES

A self-administered questionnaire is a survey that collects data without the use
of a telephone or in-person interviewer. It can be either a paper or web-based
instrument. In both cases, the design of the questionnaire is crucial to reducing
non-responses and measurement error.

With self-administered questionnaires, it is especially valuable to provide a

context for the survey as a whole – how the data will be used, whether
responses are anonymous, etc. The earlier and more effectively this is done, the
less likely people will be to dismiss the survey before they even start
responding. The introductory email or letter or the front cover of a paper survey
are appropriate places to provide the context.

Important design considerations include:

 Physical format of a paper instrument (how pages are oriented, stapled,
folded, etc.) is simple for the respondent to work with and for data entry
staff to process
 Launching and moving through a web-based survey is intuitive and works
with multiple browsers
 Order of questions is polite and logical to the respondent
 Begin with questions that reflect the announced subject of the study,
catch the respondent’s attention, and are easy to answer
 Ask people to recall things in the order in which they happened

 Ask about topic details before asking for an overall assessment

 Group items that are similar in topic, then group items within the topic
that have similar response options
 Place personal and demographic questions at the end of the survey

 Visual layout is clean, simple, and consistent

 Distinguish question text, answer choices and instructions and be
consistent in how you do this
 Limit the number of variations in type or font, the use of white type on
black background, and the use of all capital letters
 Use indentation and white space to make it easy to navigate through
sections of the survey and to identify the next question
 On a paper survey, provide lines for answering open-ended questions
(the more lines, the more respondents will write)

11
 SURVEY GUIDE

 On an electronic survey, the response space should be expandable or

accommodate an ample number of words
 Desired first and last impressions are created by the front and back
covers on a paper survey and by the email and web page headers on an
electronic survey
 Include the organization’s name and logo (or an engaging but neutral
graphic) and the title of the survey on the cover or header
 Provide instructions for returning the completed questionnaire and a
number to call with questions on the back cover (or at the end of an
electronic survey), along with a note of thanks for responding.

CHECKLIST FOR EFFECTIVE QUESTIONNAIRES

The following table summarizes the key “dos and don’ts” for writing and
formatting your survey questionnaire.

Do: Do Not:
 Give clear instructions  Use jargon or complex phrases
 Keep question structure simple  Frame questions in the negative
 Ask one question at a time  Use abbreviations, contractions or
 Maintain a parallel structure for all symbols
questions  Mix different words for the same
 Define terms before asking the question concept
 Be explicit about the period of time being  Use “loaded” words or phrases
referenced by the question  Combine multiple response
 Provide a list of acceptable responses to dimensions in the same question
closed questions  Give the impression that you are
 Ensure that response categories are both expecting a certain response
exhaustive and mutually exclusive  Bounce around between topics or
 Label response categories with words time periods
rather than numbers  Insert unnecessary graphics or
 Ask for number of occurrences, rather mix many font styles and sizes
than providing response categories such  Forget to provide instructions for
as often, seldom, never returning the completed survey!
 Save personal and demographic
questions for the end of the survey

TEST AND TRAIN

Writing survey questions is an iterative process. Review, test, and revise the
questions and introductory scripts – maybe more than once!

12
 SURVEY GUIDE

Even if you’ve used the same questions before or copied them from a
government repository, you’ll want others to check your survey for spelling and
grammatical errors, readability and flow, and consistency with the current
survey’s goals. If the topic is sensitive or your results will be used to make high
stakes decisions, consulting an expert (e.g., the UW Survey Center) to review
your survey is a wise investment. You may also need to get final approval from
others in your unit or beyond.

Once your survey has passed muster internally, it’s time to “field test” it with a
sample of potential respondents to verify that your process is smooth and
completely understandable to your target population. Do people understand the
terms? Or are adjustments needed? Do people complete the survey as
intended? Or do they drop out before completing it? Are certain questions
regularly skipped? If the survey is electronic, does it launch properly and work
as expected with different browsers? The purpose of the field test is to get
estimates of the survey’s reliability and validity and to identify any final changes
that might be needed to ensure success.

Those involved in contacting potential respondents, conducting interviews, and

analyzing survey responses need to be briefed on the purpose of the study and
provided with training tailored to their role in the project. The goal is to have
everyone following a consistent process so that as little variation as possible is
introduced into the process.

If interviewers are administering the survey in person or by phone, develop a

script for them to follow so that the survey is presented in exactly the same way
to every respondent. Interviewers will also need to be trained to behave in
neutral ways to control their influence on the answer, and they need actual
practice reading the questions and script. For mail surveys, the mail and data
entry staff needs instruction in how the study will be mailed out and how the data
will be coded and entered.

A common question is whether those who participate in the field test can later be
respondents. The answer depends on how the pretest respondents were drawn
and whether the instrument has changed, and how much time has passed
between the field test and the main study. A good way to identify pretest
respondents is to draw a miniature sample like that to be used in the main study.
This approach allows field procedures to be tested as well as the instrument.
When this method is used, pretest respondents can sometimes be combined
with respondents from the main study.

13
 SURVEY GUIDE

COLLECT DATA
RESPONSE RATE

The whole point of conducting a survey is to obtain useful, reliable, and valid
data in a format that makes it possible to analyze and draw conclusions about
the total target population. Although there is no agreed-upon minimum response
rate (Fowler, 2002), the more responses you receive, the more likely it is that
you will be able to draw statistically significant conclusions about the target
population.

Every component of your survey process – everything you do or don’t do – will

affect the response rate, so seemingly simple decisions like whether to use a
mailed survey or a web survey should not be taken lightly. The UW Survey
Center normally receives a 60-70% rate of response to mailed surveys. For web
surveys, a 30-40% response rate is common, even with populations that are
young and have easy access to the web.

From your very first contact with potential respondents to obtain cooperation,
you have the opportunity to affect the response rate. How can you interest
potential respondents so that they are more likely to respond? People like to
know the purpose of the survey and what you will do with the results. Studies
show it helps if the initial communication is personalized and presents an
altruistic purpose for participation. Clear identification of the source and authority
of the organization conducting the survey and assurances about confidentiality
are also extremely important (Odom, 1979).

Incentives are often used to maximize the response rate. Mailed surveys or
advance letters are often accompanied by a crisp bill. When it is provided before
the survey is taken, even a small denomination encourages the respondent to
complete and return the survey. Prize drawings for a gift certificate are popular
incentives for completing a web survey, however the evidence about their
potential effectiveness in gaining participation is mixed at best.

Calculating response rates can be quite complicated. The American Association

for Public Opinion Research (AAPOR) has developed a set of “Standard
Definitions” that are used by most social research organizations. See
http://www.aapor.org/Standard_Definitions/1818.htm .

FOLLOW-UP PROCEDURES
Advising potential respondents of a deadline for completing the survey helps
make it a priority. CustomInsight (2010) suggests giving a window of 7 to 10
days, with a follow-up reminder sent a few days before the end date.

14
 SURVEY GUIDE

Depending upon how the survey is being administered, popular means of

prompting potential respondents to complete the survey include email,
telephone, or a mailed postcard. A reminder that includes a duplicate copy of the
survey may yield the best response.

Investigators often want to survey the same population on a regular basis. A

simple but timely thank you note is helpful in retaining people’s interest in
participating in your future surveys. If you can share some of the survey findings,
it will engage a feeling of responsibility for the results, which in turn will increase
their commitment to future surveys. Whatever you have promised respondents
in the way of incentives and/or post-survey follow-up, be sure you deliver!

WEB SURVEY CHALLENGES

Web surveys are an increasingly popular mechanism for collecting data, but
come with their own set of issues and aren’t appropriate for every use. Web
surveys eliminate the need for data entry, but not the need to verify accuracy
and account for missing data.

An obvious drawback is that the samples in web surveys are skewed to Internet
users, and the target population may or may not be skilled in or have access to
the necessary technology. Reliability is at risk because the survey may appear
differently to different respondents, depending upon browser and computer
platform. People also differ in how and when they read email, and spam filters
can wreak havoc with your delivery schedule.

Unless the survey has only a couple questions and the number of potential
respondents is very limited, simply putting the survey in the body of an email will
not suffice (Couper, 2008). Commercial web survey tools are available
(Zoomerang and SurveyMonkey are two examples) as well as a UW-Madison
tool called Qualtrics (see http://survey.wisc.edu/). The UW Survey Center also
conducts Internet web surveys. These tools provide assurance about anonymity,
templates for laying out the survey, and assistance with compiling responses
that go far beyond the capabilities of an email.

ANALYZE DATA
CODING AND ANALYZING DATA
Thoughtful decisions made during the planning phase about what data are
needed and the format in which it will be collected will pay dividends when it
comes time to analyze the data. A consistent process for organizing and
analyzing survey data should be established and clearly documented well ahead
of receiving the first responses, with everyone involved receiving ample training.
How will incomplete surveys and missing data be handled?

15
 SURVEY GUIDE

What checks will be conducted to find errors in coding or data entry?

Will some questions be weighted more heavily than others?

Scaled responses (e.g., dissatisfied, somewhat dissatisfied, neutral, somewhat

satisfied, satisfied) are usually converted to numerical values (1, 2, 3, 4, 5) for
easier analysis. Use of a template or mask can help coders convert the data
more quickly and accurately.

Careful thought needs to be given to coding responses to open-ended items.

The process usually involves examining some preliminary data to identify
potential categories, and then testing to ascertain how consistently the
categories are assigned by various coders.

To analyze responses to open-ended questions, you can copy the comments

onto individual cards and then group similar comments together. This will give
you a sense of the most frequent ideas. Alternatively, there are software
packages that help in analyzing responses to open-ended questions.
ThemeSeekr (http://themeseekr.com/) was developed to aid in processing the
thousands of comments received during the UW-Madison 2009 reaccreditation
self-study, and uses “word clouds” as a visual analysis tool to show the relative
frequency of the themes into which responses have been categorized.

DRAWING CONCLUSIONS
Visual displays can be helpful in understanding the data. For responses to
multiple choice questions, you might:
Create a frequency histogram of the responses for each question to
demonstrate the variation in the responses
Use a bar chart to display the percent of respondents selecting
particular responses

ADDITIONAL CONSIDERATIONS
HUMAN SUBJECTS PROTECTION
If the results of your survey will be written for publication, you may need to
receive advance approval from an Institutional Review Board (IRB). Contact the
Graduate School for assistance with identifying the appropriate board (see
http://www.grad.wisc.edu/research/hrpp/irblinks.html ).

The UW Survey Center notes that survey work frequently raises issues of
interest to the IRB, including:
What is the consent process and need it be written?
How will confidentiality be protected?
How will people be sampled or recruited?

16
 SURVEY GUIDE

Will incentives be offered?

Will HIPAA (Health Insurance Portability and Accountability Act)
privacy rules be applicable?

Since IRB approval cannot be granted after the fact, it is extremely important to
follow the appropriate protocol prior to conducting your survey, if there is even a
remote chance that you will someday want to publish the results of your
research.

A WORD ABOUT CLIMATE SURVEYS

Frequently initiated by a school or college’s Equity and Diversity Committee,
climate surveys can be a useful way to gather data on job satisfaction, the quality
of interactions within the workplace, and potential issues requiring intervention.

It is always a good idea to find out what other climate assessment may have
been conducted with your target population. For example, on the UW-Madison
campus, the Women in Science and Engineering Leadership Institute (WISELI)
has been studying the quality of work life for faculty since 2003, and makes the
survey data available at the school/college level.

WISELI has also developed extensive materials for enhancing department

climate (see http://wiseli.engr.wisc.edu/climate.php), including a template climate
survey for individual departments.

So You Want to Run a Climate Survey?! (Frehill, 2006) offers numerous tips and
design considerations specific to climate surveys.

GETTING HELP WITH YOUR SURVEY

The University of Wisconsin Survey Center (see www.uwsc.wisc.edu) offers a
complete range of survey research capabilities and serves a wide variety of
clients including faculty, staff, and administration at the University of Wisconsin;
faculty and staff at other universities; federal, state, and local governmental
agencies; and not-for-profit organizations.

The Survey Center can consult with you at one or more points in the survey
process or Survey Center staff can take the lead role in designing and
implementing your survey. Specific tasks on which it can be helpful to get
assistance include:
Developing protocols that yield high response rates
Reviewing the instrument and refining questions
Sampling
Developing cost proposals
Foreign language and international work

17
 SURVEY GUIDE

TERMINOLOGY
Instrument: The survey questionnaire or booklet.

Institutional Review Board (IRB): Campus governance bodies responsible for

ensuring the protection of human subjects participating in research studies,
including surveys.

Mode: The combination of how sample members/participants are contacted,

questions are administered, and responses are recorded.

Preamble: An introductory statement that provides definitions and instructions

for completing a section or question in the survey.

Reference period: The time frame the respondent is being asked to consider
when answering a question (e.g., August 1 through August 15, 2009).

Reliability: The extent to which repeatedly measuring the same property

produces the same result.

Respondent: An individual providing answers to survey questions.

Response dimension: The scale or descriptor that a survey question asks the
respondent to use to describe their observations, actions, attitude, evaluation or
judgment about an event or behavior.

Response rate: The number of completed surveys divided by the number of

eligible potential respondents contacted.

Sample: A list of people drawn from the group from which information is
needed.

Self-administered questionnaire: A paper or web-based survey that collects

data without the use of a telephone or in-person interviewer.

Target population: The group of people whose activities, beliefs or attitudes

are being studied.

Validity: The extent to which a survey question accurately measures the

property it is supposed to measure.

18
 SURVEY GUIDE

REFERENCES AND ADDITIONAL RESOURCES

WORKS CITED
Couper, Mick P. 2008. Designing Effective Web Surveys. Cambridge:
Cambridge University Press.

"Customer Satisfaction Surveys: Maximizing Survey Responses."

CustomInsight. Web. <custominsight.com>; 24 Nov. 2010.

Fowler FJ. Survey Research Methods. 3rd ed. Thousand Oaks, CA: Sage
Publications; 2002.

Frehill, Lisa; Elena Batista; Sheila Edwards-Lange; Cecily Jeser-Cannavale; Jan

Malley; Jennifer Sheridan; Kim Sullivan; and Helena Sviglin. “So You Want to
Run a Climate Survey?!” (pp. 29-35) Using Program Evaluation to Ensure the
Success of Your ADVANCE Program. May 2006.
http://wiseli.engr.wisc.edu/docs/Report_Toolkit2_2006.pdf

Odom, John G. "Validation of Techniques Utilized to Maximize Survey

Response Rates." Tech. ERIC document reproduction service no. ED169966.
Education Resource Information Center. Web. <http://eric.ed.gov>. (09 April 1979).

ADDITIONAL RESOURCES
Department Climate Surveys: http://wiseli.engr.wisc.edu/climate.php

IRB Contacts: University of Wisconsin-Madison Graduate School,

http://grad.wisc.edu/research/hrpp/irblinks.html

Online Survey Tools:

 UW-Madison Survey Hosting Service, http://survey.wisc.edu/
 ThemeSeekr, http://themeseekr.com

Question Comprehensibility Testing:

http://mnemosyne.csl.psyc.memphis.edu/QUAID/quaidindex.html

Reference Materials List: Compiled by Nora Cate Schaeffer, UW Survey Center,

http://www.ssc.wisc.edu/cde/faculty/schaeffe.htm

Response Rates: Standard Definitions, American Association for Public Opinion

Research, http://www.aapor.org/Standard_Definitions/1818.htm

19
 SURVEY GUIDE

Survey Methodology:
 Dillman, Don A., Jolene D. Smyth, and Leah M. Christian. 2009. Internet,
Mail, and Mixed-Mode Surveys: The Tailored Design Method, Third
Edition. New York: John Wiley & Sons.

 Groves, Robert M., Floyd J. Fowler, Mick P. Couper, James M.

Lepkowski, Eleanor Singer, and Roger Tourangeau. 2004. Survey
Methodology. Hoboken, New Jersey: Wiley.

 Journal of Survey Research Methodology (surveymethods.org),

http://w4.ub.uni-konstanz.de/srm/

Training Interviewers: http://www.cdc.gov/brfss/training/interviewer/index.htm

Usability Guidelines for Web Design:

http://www.usability.gov/guidelines/guidelines_book.pdf

Writing Good Questions

 Aday, Lu A. and Llewellyn J. Cornelius. 2006. Designing and Conducting
Health Surveys: A Comprehensive Guide, 3rd Edition. New York: Wiley.

 Fowler Jr., Floyd J. and Carol Cosenza. 2008. "Writing Effective

Questions." Pp. 136-60 in International Handbook of Survey
Methodology, edited by Edith D. de Leeuw, Joop J. Hox, and Don A.
Dillman. Lawrence Erlbaum.

 Schaeffer, Nora Cate and Stanley Presser. 2003. “The Science of Asking
Questions.” Annual Review of Sociology, Vol. 29: 65-88
http://www.annualreviews.org/eprint/rU4UOoizjrXROhijkRIS/full/10.1146/annure
v.soc.29.110702.110112

20