Lecture 5

Ch. 4. Questionnaires
February 22nd, 2022
Agenda for lecture
1. Basic considerations for designing a questionnaire
• The variable plan
• How variables are converted into a data matrix
2. Best practices for creating your own questions, items, and measures
• question structure
• objectivity
• “look and feel”
Last (2-day) lecture we introduced measurement.
Questionnaires are one (extremely important and prevalent) way of recording
measurements.
• Especially when the variables are psychological (e.g., attitudes), or
personal/private (e.g., income, race, etc.).
• In practice, we use the terms questionnaire and survey interchangeably.
• Technically, “survey” is a kind of research design that uses questionnaires.
• The benefit of questionnaires is that we can standardize our measurements
across all participants.
• Standardized = exact same measures and scales for all participants, which allows all
participants to be compared (apples-to-apples).
• In contrast, focus groups, interviews, case studies (all of which we’ll cover later) are unstandardized
(different variables, or varied measurements), limiting comparisons (apples-to-oranges).
• Other designs (e.g., observational research) are also standardized, but these designs
cannot measure psychological or private variables.
Okay, how do I do a questionnaire?
1. Determine your research objectives.
• Is the research descriptive? Diagnostic? Predictive? (review Lecture 1 if necessary)
• If the research is descriptive, then the questionnaire may be more open-ended and exploratory.
You can include many measures of many things.
• If it is diagnostic or predictive, then hypotheses will drive the questionnaire. You will limit your
questionnaire to the specific variables under scrutiny.
2. Create a “variable plan.” (see next slide)
• List the constructs you intend to measure or analyze.
• Identify whether there are validated measures for these variables.
• Sometimes, there are more than one measure for constructs.
3. Translate the “variable plan” into an actual questionnaire.
• Here, you make decisions about measurement.
• Consider question structure, objectivity, and “look and feel.”
• E.g., transitions, a welcome statement, customization features, a debriefing statement, etc.
The variable plan
• An outline of the constructs you are measuring.
• Helps you to organize and plan:
• How your constructs are operationally defined.
• How many variables will be measured in your study.
• Some constructs may be measured more than once.
• The number of total items that participants will respond to.
• Some variables will be measured with multi-item scales. Some variables may be measured in an open-
ended way.
• Approximately how long the study will be (which helps you determine compensation).
• The goal of the plan is to plan how your questionnaire addresses your research
questions or hypotheses, before you begin creating the questionnaire itself.
• You’ll have many low-level decisions to make once you begin creating the questionnaire;
the variable plan focuses you on the high-level problems.
 Sample variable plan
I have a student collaborating with researchers from San Francisco State University on a
study of well-being, culture, and obligation motives for giving.
Construct Definition Variable # of items/scale Scale
Individualistic-
Individualism- 1-7 scale (disagree –
Culture Collectivistic 20 items
Collectivism score agree)
Orientation
Sense of Duty to Care Obligation motives 1-5 scale (not important –
Obligation 5 items
for Others score extremely important)
Positive Appraisal of Satisfaction with Life 1-7 scale (disagree –
Well-Being 5 items
Life score agree)
Good Feelings minus Positive and Negative 1-5 scale (not at all –
24 items
Bad Feelings Affect score extremely)
Eudaemonic 1-7 scale (disagree –
Sense of Purpose 17 items
Fulfillment score agree)

The final questionnaire will consist of five measures, totaling 85 items. This will take people
roughly five minutes to complete.
Here is an example of a questionnaire, with 7 variables.
What do the data look like, when collected?

Every survey respondent is assigned a row. Their answers to the survey items
are organized by columns. There are numbered values for all answer options,
even for categorical variables. The column ‘id’ provides a numbered code, to
identify individual respondents (subject 1, subject 2, subject 3, etc…)
Building your own questions,
items, and measures
Consider:
- question structure
- objectivity
- “look and feel.”
Question Structure #1
Limit your use of open-ended response questions.
• Advantages of open-ended responses:
• Particularly beneficial in exploratory research, especially when the range of responses
is not known.
• May suggest alternatives not listed in a closed-end response format.
• Can assist participants with explaining their responses to scaled items.
• Ex: On a 1-9 scale, how satisfied were you with your banking experience?
• Please explain _______________________________________________.
• Disadvantages of open-ended responses:
• Difficult/expensive to analyze.
• Requires coding the responses, which often requires human labor.
• Exceptions: web scraping, word frequency analysis, or machine learning methods
Question Structure #2
Be sensitive when asking personal questions. But be
aware of how measurement constrains reporting.

What is your age?

1: 18 – 24
Example Question: 2: 25 – 34 What is your age? ____
3: 35 – 44
4: 45 - 54

15
10
5 4
What the report Mean Age: 34
might look like: Median Age: 28
18-24 25-34 35-44 45-54
Question Structure #3
Don’t ask unnecessary/redundant questions
• Date of birth and age  redundant; unnecessary to have both!
• Zip code and city/state  redundant; unnecessary to have both!
• Exception: attention checks
• Attention checks are questions that are thrown in to ensure your
respondents’ data can be trusted.
Question Structure #4
Multiple choice questions need to be mutually
exclusive and exhaustive
NOT mutually
Mutually exclusive/exhaustive
exclusive/exhaustive
What did you eat for breakfast? Was your breakfast?

 Pancakes  Vegetarian
 Waffles  Not vegetarian
 Omelet  N/A

What is your age? What is your age?

 0 – 20  0 - 20
 20 - 30  21 - 30
 30 - 40  31 – 40
 …
Objectivity #1
Avoid Subjective Language
In a typical month, how often do you shop in department stores?
_____ Never
_____ Occasionally
_____ Sometimes
_____ Often
_____ Regularly (Incorrect)

In a typical month, how often do you shop in department stores?

_____ Less than once a month
_____ 1 or 2 times a month
_____ 3 or 4 times a month
_____ More than 4 times a month (Correct)
Objectivity #2
Avoid Leading Questions
Would you agree that our customer service is excellent?

How would you rate our customer service?

Objectivity #3
Avoid Double-Barreled Questions
“Do you think Coca-Cola is a tasty and refreshing soft drink?”

Do you think Coca-Cola is a tasty soft drink?

Do you think Coca-Cola is a refreshing soft drink?
Objectivity #4
Randomize Multiple Choices
What is most important to you in a laptop
computer?
 Price
 Weight
 Speed Randomize these choices
 Brand
 Service
 Other: ___________
Look and Feel #1
Have a survey introduction
A good introduction does the following:

• Explains who is conducting the survey

• Explains how the data will be used
• Reveals whether it is anonymous, confidential, or neither
• Reveals whether there is an incentive or not
• Sets expectations for the time requirement
• Expresses gratitude for the respondents’ time and insights

Related. Introductions may also require signatures documenting consent to

participate. (E.g., “subject understands what is being asked of him/her and
agrees to participate.”)
Look and Feel #2
Include transitions and instructions
Guide the respondent and manage their expectations throughout.

• “First, we are going to ask you to think about ____. Please think
about ____ as you answer the questions below.”
• “Great! You are finished with section 1. Please click the button below
to proceed to section 2.”
• “You are now finished answering questions about your household.
We will now ask you questions about your love life.”

It is often a good idea to include a progress bar, when your survey is online.
Sorry this is so small. But there’s the
progress bar, indicating how much longer
the subject can expect the research to
last.