>appendix13a

Crafting Effective Measurement Questions

Numerous issues inluence whether the questions we ask also points to two other problems of scope and coverage:
on questionnaires generate the decision-making data that the double-barreled question and the imprecise question.
managers sorely need. Each of the issues summarized in
Exhibit 13-5 is developed more fully here. Double-Barreled Questions Does the question
request so much content that it should be broken into two
Question Content or more questions? While reducing the overall number of
questions in a study is highly desirable, don’t try to ask
Should This Question Be Asked? double-barreled questions. The Albany Clinic question
about lu (“Have you ever had or been treated for a re-
Purposeful versus Interesting Questions that cent cold or lu?”) ires more than two barrels. It asks four
merely produce “interesting information” cannot be justi- questions in all (Ever had cold? Ever had lu? Been treated
ied on either economic or research grounds. Challenge each for cold? Been treated for lu?).
question’s function. Does it contribute signiicant informa- Here’s another common example posed to menswear
tion toward answering the research question? Will its omis- retailers: “Are this year’s shoe sales and gross proits
sion limit or prevent the thorough analysis of other data? higher than last year’s?” Couldn’t sales be higher with
Can we infer the answer from another question? A good stagnant proits, or proits higher with level or lower
question designer knows the value of learning more from sales? This second example is more typical of the problem
fewer questions. of double-barreled questions.
A less obvious double-barreled question is the question
Is the Question of Proper Scope
we ask to identify a family’s or a group’s TV station pref-
and Coverage? erence. Since a single station is unlikely, a better question
would ask the station preference of each family member
Incomplete or Unfocused We can test this con- separately or, alternatively, screen for the group member
tent issue by asking, Will this question reveal all we who most often controls channel selection on Monday
need to know? We sometimes ask participants to reveal evenings during prime time. Also, it’s highly probable that
their motivations for particular behaviors or attitudes by no one station would serve as an individual’s preferred sta-
asking them, Why? This simple question is inadequate tion when we cover a wide range of time (8 to 11 p.m.).
to probe the range of most causal relationships. When This reveals another problem, the imprecise question.
studying product use behavior, for example, we learn
more by directing two or three questions on product use Precision To test a question for precision, ask, Does
to the heavy-use consumer and only one question to the the question ask precisely what we want and need to know?
light user. We sometimes ask for a participant’s income when we re-
Questions are also inadequate if they do not provide ally want to know the family’s total annual income before
the information you need to interpret responses fully. If taxes in the past calendar year. We ask what a participant
you ask about the Albany Clinic’s image for quality pa- purchased “last week” when we really want to know what
tient care, do different groups of patients or those there he or she purchased in a “typical 7-day period during the
for the irst versus the third time have different attitudes? past 90 days.” The Albany Clinic’s patients were asked for
To evaluate relative attitudes, do you need to ask the same cold and lu history during the time frame “ever.” It is hard
question about other companies? In the original Albany to imagine an adult who has never experienced a cold or
Clinic survey, participants were asked, “Have you ever lu and equally hard to assume an adult hasn’t been treated
had or been treated for a recent cold or lu?” If participants for one or both at some time in his or her life.
answer yes, what exactly have they told the researcher that A second precision issue deals with common vocabu-
would be of use to the eye surgeon? Wouldn’t it be likely lary between researcher and participant. To test your ques-
that the surgeon is interested in medication taken to treat tion for this problem, ask, Do I need to offer operational
colds or lu within, say, the prior 10 days? This question deinitions of concepts and constructs used in the question?

328
 >chapter 13 Questionnaires and Instruments 329

Can the Participant Answer Adequately? determining the content and appropriateness of a question.
In some studies, the degree of participant expertise can be
Time for Thought Although the question may ad- substantial, and simpliied explanations are inappropriate
dress the topic, is it asked in such a way that the participant and discourage participation. In asking the public about
will be able to frame an answer, or is it reasonable to as- gross margins in menswear stores, we would want to be
sume that the participant can determine the answer? This sure the “general-public” participant understands the na-
is also a question that drives sample design, but once the ture of “gross margin.” If our sample unit were a merchant,
ideal sample unit is determined, researchers often assume explanations might not be needed. A high level of knowl-
that participants who it the sample proile have all the an- edge among our sample units, however, may not eliminate
swers, preferably on the tips of their tongues. To frame a the need for operational deinitions. Among merchants,
response to some questions takes time and thought; such gross margin per unit in dollars is commonly accepted as
questions are best left to self-administered questionnaires. the difference between cost and selling price; but when
offered as a percentage rather than a dollar igure, it can
Participation at the Expense of Accuracy Par- be calculated as a percentage of unit selling price or as a
ticipants typically want to cooperate in interviews; thus percentage of unit cost. A participant answering from the
they assume giving any answer is more helpful than de- “cost” frame of reference would calculate gross margin at
nying knowledge of a topic. Their desire to impress the 100 percent; another participant, using the same dollars
interviewer may encourage them to give answers based and the “selling price” frame of reference, would calculate
on no information. A classic illustration of this problem gross margin at 50 percent. If a construct is involved and
occurred with the following question:1 “Which of the fol- differing interpretations of a concept are feasible, opera-
lowing statements most closely coincides with your opin- tional deinitions may still be needed.
ion of the Metallic Metals Act?” The response pattern
shows that 70 percent of those interviewed had a fairly Recall and Memory Decay The adequacy prob-
clear opinion of the Metallic Metals Act; however, there lem also occurs when you ask questions that overtax par-
is no such act. The participants apparently assumed that ticipants’ recall ability. People cannot recall much that
if a question was asked, they should provide an answer. has happened in their past, unless it was dramatic. Your
Given reasonable-sounding choices, they selected one mother may remember everything about your arrival if
even though they knew nothing about the topic. you were her irst child: the weather, time of day, even
To counteract this tendency to respond at any cost, what she ate prior to your birth. If you have several sib-
ilter or screen questions are used to qualify a participant’s lings, her memory of subsequent births may be less com-
knowledge. If the MindWriter service questionnaire is dis- plete. If the events surveyed are of incidental interest to
tributed via mail to all recent purchasers of MindWriter participants, they will probably be unable to recall them
products, we might ask, “Have you required service for correctly even a short time later. An unaided recall ques-
your laptop since its purchase?” Only those for whom ser- tion, “What radio programs did you listen to last night?”
vice was provided could supply the detail and scope of might identify as few as 10 percent of those individuals
the responses indicated in the investigative question list. who actually listened to a program.3
If such a question is asked in a phone interview, we would
call the question a screen, because it is being used to de- Balance (General versus Specific) Answering
termine whether the person on the other end of the phone adequacy also depends on the proper balance between
line is a qualiied sample unit. This same question asked generality and speciicity. We often ask questions in terms
on a computer-administered questionnaire would likely too general and detached from participants’ experiences.
branch or skip the participant to a series of classiication Asking for average annual consumption of a product may
questions. make an unrealistic demand for generalization on people
Assuming that participants have prior knowledge or who do not think in such terms. Why not ask how often
understanding may be risky. The risk is getting many an- the product was used last week or last month? Too often
swers that have little basis in fact. The Metallic Metals Act participants are asked to recall individual use experiences
illustration may be challenged as unusual, but in another over an extended time and to average them for us. This
case a Gallup report revealed that 45 percent of the persons is asking participants to do the researcher’s work and en-
surveyed did not know what a “lobbyist in Washington” courages substantial response errors. It may also contrib-
was and 88 percent could not give a correct description of ute to a higher refusal rate and higher discontinuation rate.
“jurisdictional strike.”2 This points to the need for opera- There is a danger in being too narrow in the time frame
tional deinitions as part of question wording. applied to behavior questions. We may ask about movie at-
tendance for the last seven days, although this is too short
Presumed Knowledge The question designer a time span on which to base attendance estimates. It may
should consider the participants’ information level when be better to ask about attendance, say, for the last 30 days.
330 >part III The Sources and Collection of Data

>Exhibit 13a-1 A Test of Alternative Response Strategies

A. What is your favorite brand of ice cream? _________________________

B. Some people have a favorite brand of ice cream, while others do not have a favorite brand. In which group are you? (please check)
I do not have a favorite brand of ice cream.
I have a favorite brand of ice cream.
What is your favorite (if you have a favorite)? _________________________

>Exhibit 13a-2 Results of Alternative Response Strategies Test

Response Version A Version B

Named a favorite brand 77%* 39%*
Named a favorite flavor rather than a brand 19 18
Had no favorite brand 4 43
Total 100% 100%
n 5 57 n 5 56

*Significant difference at the 0.001 level.

There are no irm rules about this generality-speciicity topics are considered too sensitive to discuss with strang-
problem. Developing the right level of generality depends ers. These vary from person to person, but one study sug-
on the subject, industry, setting, and experience of the gests the most sensitive topics concern money matters and
question designer. family life.4 More than one-fourth of those interviewed
mentioned these as the topics about which they would be
Objectivity The ability of participants to answer ad- “least willing to answer questions.” Participants of lower
equately is also often distorted by questions whose content socioeconomic status also included political matters in
is biased by what is included or omitted. The question may this “least willing” list.
explicitly mention only the positive or negative aspects of Participants also may be unwilling to give correct an-
the topic or make unwarranted assumptions about the par- swers for ego reasons. Many exaggerate their incomes, the
ticipant’s position. Consider Exhibit 13a-1, an experiment number of cars they own, their social status, and the amount
in which two forms of a question were asked. Fifty-seven of high-prestige literature they read. They also minimize
randomly chosen graduate business students answered their age and the amount of low-prestige literature they
version A, and 56 answered version B. Their responses read. Many participants are reluctant to try to give an ad-
are shown in Exhibit 13a-2. The probable cause of the dif- equate response. Often this will occur when they see the
ference in level of brand preference expressed is that A is topic as irrelevant to their own interests or to their percep-
an unsupported assumption. It assumes and suggests that tion of the survey’s purpose. They participate halfheartedly,
everyone has a favorite brand of ice cream and will re- often answer with “don’t know,” give negative replies, give
port it. Version B indicates the participant need not have stereotypical responses, or refuse to be interviewed.
a favorite. You can learn more about crafting questions dealing
A deiciency in both versions is that about one partici- with sensitive information by reading “Measuring Atti-
pant in ive misinterpreted the meaning of the term brand. tudes on Sensitive Subjects” on the text website.
This misinterpretation cannot be attributed to low educa-
tion, low intelligence, lack of exposure to the topic, or Question Wording
quick or lazy reading of the question. The subjects were
students who had taken at least one course in marketing in Shared Vocabulary Because surveying is an ex-
which branding was prominently treated.* change of ideas between interviewer and participant, each
must understand what the other says, and this is possible
Will the Participants Answer Willingly? only if the vocabulary used is common to both parties.5
Two problems arise. First, the words must be simple
Sensitive Information Even if participants have enough to allow adequate communication with persons of
the information, they may be unwilling to give it. Some limited education. This is dealt with by reducing the level
of word dificulty to simple English words and phrases
*Word confusion difficulties are discussed later in this appendix. (more is said about this in the section on word clarity).
 >chapter 13 Questionnaires and Instruments 331

Technical language is the second issue. Even highly Unsupported Assumptions Unwarranted assump-
educated participants cannot answer questions stated in tions contribute to many problems of question wording.
unfamiliar technical terms. Technical language also poses A metropolitan newspaper, Midwest Daily, conducted a
dificulties for interviewers. In one study of how corpo- study in an attempt to discover what readers would like
ration executives handled various inancial problems, in- in its redesigned lifestyle section. One notable question
terviewers had to be conversant with technical inancial asked readers: “Who selects your clothes? You or the
terms. This necessity presented the researcher with two man in your life?” In this age of educated, working, in-
alternatives—hiring people knowledgeable in inance and dependent women, the question managed to offend a
teaching them interviewing skills or teaching inancial signiicant portion of the female readership. In addition,
concepts to experienced interviewers.6 This vocabulary Midwest Daily discovered that many of its female readers
problem also exists in situations where similar or identical were younger than researchers originally assumed and the
studies are conducted in different countries and multiple only man in their lives was their father, not the spousal
languages. or romantic relationship alluded to by the questions that
A great obstacle to effective question wording is the followed. Once men reached this question, they assumed
choice of words. Questions to be asked of the public that the paper was interested in serving only the needs of
should be restricted to the 2,000 most common words female readers. The unwarranted assumptions built into
in the English language.7 Even the use of simple words the questionnaire caused a signiicantly smaller response
is not enough. Many words have vague references or rate than expected and caused several of the answers to be
meanings that must be gleaned from their context. In a uninterpretable.
repair study, technicians were asked, “How many radio
sets did you repair last month?” This question may Frame of Reference Inherent in word meaning
seem unambiguous, but participants interpreted it in two problems is also the matter of a frame of reference.
ways. Some viewed it as a question of them alone; oth- Each of us understands concepts, words, and expres-
ers interpreted “you” more inclusively, as referring to sions in light of our own experience. The U.S. Bureau
the total output of the shop. There is also the possibility of the Census wanted to know how many people were
of misinterpreting “last month,” depending on the tim- in the labor market. To learn whether a person was em-
ing of the questioning. Using “during the last 30 days” ployed, it asked, “Did you do any work for pay or proit
would be much more precise and unambiguous. Typi- last week?” The researchers erroneously assumed there
cal of the many problem words are these: any, could, would be a common frame of reference between the in-
would, should, fair, near, often, average, and regular. terviewer and participants on the meaning of work. Un-
One author recommends that after stating a question as fortunately, many persons viewed themselves primarily
precisely as possible, we should test each word against or foremost as homemakers or students. They failed to
this checklist: report that they also worked at a job during the week.
This difference in frame of reference resulted in a consis-
• Does the word chosen mean what we intend? tent underestimation of the number of people working in
• Does the word have multiple meanings? If so, does the United States.
the context make the intended meaning clear? In a subsequent version of the study, this question
• Does the word chosen have more than one pronunci- was replaced by two questions, the irst of which sought
ation? Is there any word with similar pronunciation a statement on the participant’s major activity during the
with which the chosen word might be confused? week. If the participant gave a nonwork classiication, a
second question was asked to determine if he or she had
• Is a simpler word or phrase suggested or possible?8
done any work for pay besides this major activity. This re-
We cause other problems when we use abstract con- vision increased the estimate of total employment by more
cepts that have many overtones or emotional qualiica- than 1 million people, half of them working 35 hours or
tions.9 Without concrete referents, meanings are too vague more per week.10
for the researcher’s needs. Examples of such words are The frame of reference can be controlled in two ways.
business, government, and society. First, the interviewer may seek to learn the frame of refer-
Shared vocabulary issues are addressed by using the ence used by the participant. When asking participants to
evaluate their reasons for judging a retail store as unat-
following:
tractive, the interviewer must learn the frames of refer-
• Simple rather than complex words. ence they use. Is the store being evaluated in terms of its
particular features and layout, the failure of management
• Commonly known, unambiguous words.
to respond to a complaint made by the participant, the
• Precise words. preference of the participant for another store, or the par-
• Interviewers with content knowledge. ticipant’s recent dificulty in returning an unwanted item?
332 >part III The Sources and Collection of Data

>Exhibit 13a-3 Split Test of Alternative Question Wording

Should the United States do any of the following at this time?

A. Increase our armed forces further, even if it means more taxes.

Should the United States do any of the following at this time?

B. Increase our armed forces further, even if you have to pay a special tax.

Eighty-eight percent of those answering question A thought the armed forces should be increased, while only 79 percent of those
answering question B favored increasing the armed forces.

Source: Hadley Cantril, ed., Gauging Public Opinion (Princeton, NJ: Princeton University Press, 1944), p. 48.

Second, it is useful to specify the frame of reference name in a survey question asked of teen boys interested in
for the participant. In asking for an opinion about the new basketball. The power of aspirational reference groups to
store design, the interviewer might specify that the ques- sway opinion and attitude is well established in advertis-
tion should be answered based on the participant’s opinion ing; it shouldn’t be underestimated in survey design.
of the layout, the clarity and placement of signage, the ease We also can bias response through the use of superla-
of inding merchandise, or another frame of reference. tives, slang expressions, and fad words. These are best
excluded unless they are critical to the objective of the ques-
Biased Wording Bias is the distortion of responses tion. Ethnic references should also be stated with care.
in one direction. It can result from many of the problems
already discussed, but word choice is often the major Personalization How personalized should a ques-
source. Obviously such words or phrases as politically tion be? Should we ask, “What would you do about . . . ?”
correct or fundamentalist must be used with great care. Or should we ask, “What would people with whom you
Strong adjectives can be particularly distorting. One al- work do about . . . ?” The effect of personalization is shown
leged opinion survey concerned with the subject of prepa- in a classic example reported by Cantril.12 A split test—in
ration for death included the following question: “Do you which a portion of the sample received one question, with
think that decent, low-cost funerals are sensible?” Who another portion receiving a second question—was made
could be against anything that is decent or sensible? There of a question concerning attitudes about the expansion of
is a question about whether this was a legitimate survey or U.S. armed forces in 1940, as noted in Exhibit 13a-3.
a burial service sales campaign, but it shows how sugges- These and other examples show that personalizing
tive an adjective can be. questions changes responses, but the direction of the inlu-
Congressional representatives have been known to use ence is not clear. We cannot tell whether personalization
surveys as a means of communicating with their constitu- or no personalization is superior. Perhaps the best that can
encies. Questions are worded, however, to imply the issue be said is that when either form is acceptable, we should
stance that the representative favors. Can you tell the rep- choose that which appears to present the issues more real-
resentative’s stance in the following question? istically. If there are doubts, then split survey versions
should be used (one segment of the sample should get one
Example: Would you have me vote for a balanced question version, while a second segment should receive
budget if it means higher costs for the sup- the alternative question version).
plemental Social Security benefits that you
have already earned? Adequate Alternatives Have we adequately ex-
We can also strongly bias the participant by using pressed the alternatives with respect to the purpose of the
prestigious names in a question. In a historic survey on question? It is usually wise to express each alternative ex-
whether the war and navy departments should be com- plicitly to avoid bias. This is illustrated well with a pair
bined into a single defense department, one survey said, of questions that were asked of matched samples of par-
“General Eisenhower says the army and navy should be ticipants.13 The question forms that were used are noted in
combined,” while the other version omitted his name. Exhibit 13a-4.
Given the irst version (name included), 49 percent of the Often the above issues are simultaneously present in
participants approved of having one department; given the a single question. Exhibit 13a-5 reveals several questions
second version, only 29 percent favored one department.11 drawn from actual mail surveys. We’ve identiied the prob-
Just imagine using Kobe Bryant’s or Dirk Nowitzki’s lem issues and suggest one solution for improvement.
 >chapter 13 Questionnaires and Instruments 333

>Exhibit 13a-4 Expressing Alternatives

The way a question is asked can influence the results. Consider these two alternative questions judging companies’ images in the
community in the face of layoffs:
A. Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs
and give steady work right through the year?
B. Do you think most manufacturing companies that lay off workers in slack periods could avoid layoffs and provide steady work
right through the year, or do you think layoffs are unavoidable?
The Results:

When Asked . . . A B
Company could avoid layoffs 63% 35%
Could not avoid layoffs 22 41
No opinion 15 24

Source: Hadley Cantril, ed., Gauging Public Opinion (Princeton, NJ: Princeton University Press, 1944), p. 48.

While the suggested improvement might not be the only Closed questions are better when there is a clear frame
possible solution, it does correct the issues identiied. of reference, the participant’s level of information is
What other solutions could be applied to correct the prob- predictable, and the researcher believes the participant
lems identiied? understands the topic.
Open-ended questions also help to uncover certainty
Response Strategy of feelings and expressions of intensity, although well-
designed closed questions can do the same.
The objectives of the study; characteristics of participants,
especially their level of information, level of motivation Thoroughness of Prior Thought If a participant
to participate, and ease of communication; the nature of has developed a clear opinion on the topic, a closed ques-
the topic(s) being studied; the type of scale needed; and tion does well. If an answer has not been thought out, an
your analysis plan dictate the response strategy. Examples open-ended question may give the participant a chance to
of the strategies described in Chapter 13 and discussed in ponder a reply, and then elaborate on and revise it.
detail in Chapters 11 and 12 are found in Exhibit 13-6.
Communication Skill Open-ended questions re-
Objective of the Study If the objective of the quire a stronger grasp of vocabulary and a greater ability
question is only to classify the participant on some stated to frame responses than do closed questions.
point of view, then the closed question will serve well.
Assume you are interested only in whether a participant Participant Motivation Experience has shown that
approves or disapproves of a certain corporate policy. A closed questions typically require less motivation and an-
closed question will provide this answer. This response swering them is less threatening to participants. But the
strategy ignores the full scope of the participant’s opinion response alternatives sometimes suggest which answer is
and the events that helped shape the attitude at its foun- appropriate; for this reason, closed questions may be biased.
dation. If the objective is to explore this wider territory, While the open-ended question offers many advan-
then an open-ended question (free-response strategy) is tages, closed questions are generally preferable in large
preferable. surveys. They reduce the variability of response, make
Open-ended questions are appropriate when the ob- fewer demands on interviewer skills, are less costly to ad-
jective is to discover opinions and degrees of knowledge. minister, and are much easier to code and analyze. After
They are also appropriate when the interviewer seeks adequate exploration and testing, we can often develop
sources of information, dates of events, and suggestions closed questions that will perform as effectively as open-
or when probes are used to secure more information. ended questions in many situations. Experimental stud-
When the topic of a question is outside the partici- ies suggest that closed questions are equal or superior to
pant’s experience, the open-ended question may offer open-ended questions in many more applications than is
the better way to learn his or her level of information. commonly believed.14
334 >part III The Sources and Collection of Data

Exhibit 13a-5 Reconstructing Questions

Problem/Solution Poor Measurement Question Improved Measurement Question

Problems: Checklist appears to offer If your purpose for THIS hotel stay Which reason BEST explains your purpose for
options that are neither exhaustive included personal pleasure, for what THIS personal pleasure hotel stay?
nor mutually exclusive. Also, it doesn’t ONE purpose specifically? Dining
fully address the content needs of Visit friend/relative Shopping
understanding why people choose Weekend escape Entertainment
a hotel when they travel for personal Sporting event . . . was this for a . . . Sport-related event?
reasons versus business reasons. Sightseeing Theater, musical, or
Family event other performance?
Solution: Organize the alternatives. Vacation Museum or exhibit?
Create subsets within choices; use
Other: __________________ Visit friend/relative
color or shading to highlight subsets.
. . . was this for a special event? YES NO
For coding ease, expand the alterna-
Vacation
tives so the participant does not fre-
. . . was this primarily for . . . Sightseeing?
quently choose “Other.”
Weekend
escape?
Other:_______________

Problems: Double-barreled ques- When you eat out, do you frequently Considering your personal eating experiences
tion; no time frame for the behavior; order appetizers and dessert? away from home in the last 30 days, did you
“frequently” is an undefined construct YES NO purchase an appetizer or dessert more than half
for eating behavior; depending on the time?
the study’s purpose, “order” is not as More Than Less Than
powerful a concept for measurement Half the Time Half the Time
as others (e.g., purchase, consume, Purchased
or eat) an appetizer
a dessert
Solution: Split the questions; expand Purchased neither appetizers nor desserts.
the response alternatives; clearly
define the construct you want to
measure.

Problem: Nonspecific time frame; Have you ever attended a college In the last six months, have you been a spectator
likely to experience memory decay; basketball game? at a basketball game played by college teams on
nonspecific screen (not asking what YES NO a college campus?
you really need to know to qualify a YES NO
participant).
Solution: Replace “ever” with a more
appropriate time frame; screen for the
desired behavior.

Problem: Question faces serious How much did you pay for the last Did you purchase a dress coat for your personal
memory decay as a coat may not be coat you purchased? use in the last 60 days?
purchased each year; isn’t asking if YES NO
the coat was a personal purchase or Thinking of this dress coat, how much did
for someone else; nor do you know you pay? (to the nearest dollar) $ _______.00
the type of coat purchased; nor do Was this coat purchase made at a
you know whether the coat was pur- discounted price?
chased for full price or at a discount. YES NO

Solution: Limit the time frame; spec-

ify the coat type.