Quantitative Data

Collection Methods
Prof. Dr. Jegak Uli
Learning Outcomes
1. Able to determine what needs to be measured to address a research
question or hypothesis
2. Able to distinguish levels of scale measurement
3. Be able to explain the three criteria for good measurement
4. Able to assess scale reliability and validity
5. Able to explain the different ways of doing survey research.
6. Able to explain the three steps of developing research instrument.
7. Able to produce survey questionnaire.
8. Able to explain how to carry pre-test of questionnaire as well as pilot
study.
What Do I Measure?
Measurement
• The process of describing some property of a phenomenon, usually by
assigning numbers in a reliable and valid way.
Concept
• A generalized idea about a class of objects, attributes, occurrences, or
processes that you study.

13–3
Operational Definitions
Operationalization
• The process of identifying scales that correspond to variance in a
concept involved in a research process.
Scales
• A device providing a range of values that correspond to different
characteristics or amounts of a characteristic exhibited in observing a
concept.
Correspondence rules
• Indicate the way that a certain value on a scale corresponds to some
true value of a concept.
Constructs
• Concepts measured with multiple variables (constructs).
The Different Level of Scale Measurement
Four levels of scale measurement include
1. the nominal scale,
2. the ordinal scale,
3. the interval scale, and
4. the ratio scale.
The Nominal Scale
Nominal measurement is the process of classifying different objects into categories
(groups) based upon some defined characteristics.

The properties of nominal data are as follows:

1.Data categories are mutually exclusive (an object can belong to only one category/no
overlapping in category).
2.Data categories have no logical order.

Examples are sex, color of hair/eyes, ethnic background, makes of car and so on.

Summary:
• a nominal scale simply classifies without order.
• Assigns a value to an object for identification or classification purposes (e.g. : 1 = Male, 2 =
Female).
• Most elementary level of measurement.
The Ordinal Scale
• With the ordinal scale one additional property is present over those of the nominal scale - a
logical ordering of the categories/groups .
• The variable measured is such that differences in the amount of the characteristic possessed
are discernible (obvious).

The properties of ordinal data are as follows:

1. Data categories are mutually exclusive.
2. Data categories have some logical order.
3. Data categories are scaled according to the amount (e.g. : Grade A = 80-90, A+ = 91-100) of the
particular characteristic they possess.

Examples are course grade: A+, A, B+, B, B-, C+, C, C-, D, D- and F; level of satisfaction: low,
medium and high and so forth.

Summary:
• Ranking scales allowing things to be arranged based on how much of some concept they possible.
• Have nominal properties
Category of Satisfaction level for different Likert Scale
Category Satisfaction Scale (Likert Scale)
level
10-point 7-point 5- point

Low 1.00 – 3.99 1.00 – 3.00 1 – 2.33

1 (Dissatisfied)

Moderate 4.00 – 6.99 3.01 – 4.99 2.34 – 3.67

2 (Satisfied)

High 7.00 –10.00 5.00 – 7.00 3.68 – 5.00

3 (Very Satisfied)
 The Interval Scale
• Interval data possess all the properties of ordinal scale with one additional property. This property is
that the differences between the various levels of the categories on any part of the scale reflect
equal differences in the characteristic measured. That is, equal unit is established in the scale
(sometimes referred to as equal unit scale).
The properties of interval data are as follows:
1.Data categories are mutually exclusive.
2.Data categories have some logical order.
3.Data categories are scaled according to the amount of the particular characteristic they possess.
4.Equal differences in the characteristic are represented by equal differences in the numbers
assigned to the categories.
5.The point zero is just another point on the scale.

An Example of interval scale is temperature (0o Celsius, 32o F).

Summary:
• Capture information about differences in quantities of a concept.
• Have both nominal and ordinal properties.
The Ratio Scale
• The highest level in the hierarchy of measurement scales.
• The scale has one property in addition to the properties listed above for the interval scale. This
property is a known or true zero point that reflects an absence of the characteristic measured.

The properties of ratio data are as follows:

1. Data categories are mutually exclusive.
2. Data categories have some logical order.
3. Data categories are scaled according to the amount of the particular characteristic they possess.
4. Equal differences in the characteristic are represented by equal differences in the numbers
assigned to the categories.
5. The point zero reflects an absence of the characteristic.

Examples of ratio scales are income, age, weight, height, score (including composite/mean
composite score), etc.

Summary:
• Highest form of measurement.
• Have all the properties of interval scales with the additional attribute of representing absolute
quantities.
Nominal, Ordinal, Interval, and Ratio Scales Provide Different Information
Mathematical and Statistical Analysis of Scales
1. Discrete Measures
• Measures that can take on only one of a finite number of values.
• As opposed to a continuous variable, a discrete variable can take on only designated
values.
For example, number of cars a household possessed.
• This variable can take on only integer values; that is 1, 2, 3 and so on.
• There can be no fractional value at any one time.

2. Continuous Measures
• Measures that reflect the intensity of a concept by assigning values that can take on any
value along some scale range.
• A continuous variable is one that can take on any values on the measurement scale
under consideration. Examples of continuous variables are weight, height, income, age,
etc.
• We sometimes refer to a continuous variable as one that has underlying continuity.
That is, there are no breaks in the possible values it can assume.
Index Measures
Attributes
• Single characteristics or fundamental features that pertain to an object,
person, or issue (e.g.: gender/sex).
Index Measures
• Assign a value based on how much of the concept being measured is
associated with an observation.
• Indexes often are formed by putting several variables together. (e.g. :
cost of living index/consumer price index)
Composite Measures
• Assign a value to an observation based on a mathematical derivation of
multiple variables.
Formulas for Computing a Composite Score
Example you have item a1 to a19 to measure a concept/construct, you
add these item (a1 to a19) to compute the composite score using the
following formula:
1. Custom formula:
a1+a2+a3+a4+a5+a6+a7+a8+a9+a10+a11+a12+a13+a14+a15+a16+a1
7+a18+a19
2. SUM function – the default formula:
SUM(a1, a2, a3, a4,a 5, a6, a7, a8, a9, a10, a11, a12,
a13, a14, a15, a16, a17,a17, a18, a19)
3. SUM function – the shortcut formula:
SUM (a1 to a19)
Computing a Composite Score Composite
score

Mean
Composite
score
Computing Scale Values
Summated Scale
• A scale created by simply summing (adding together) the response
to each item making up the composite measure.
Reverse Coding
• Means that the value assigned for a response is treated oppositely
from the other items.
 Recoding Made Easy
1. Click on transform.
2. Click on recode.
3. Choose to recode into the
different /same variable.
4. Select the variable(s) to be
recoded.
5. Click on old and new values.
6. Use the menu that appears
to enter the old values and
the matching new values.
Click add after entering each
pair.
7. Click continue.
 A scale of the physical activities of daily living in the elderly
Table 1Physical activity items
NO. ITEM NO. ITEM LABEL +VE/- Scale
VE
1 2 3

1 ITEM1 Can eat +VE

2 ITEM2 Can dress and undress +VE

3 ITEM3 Can take care of own appearance +VE

4 ITEM4 Can walk +VE

5 ITEM5 Can get in and out of bed +VE

6 ITEM6 Can take a bath or shower +VE

7 ITEM7 Can get to bathroom on time +VE

8 ITEM8 Has been able to do tasks for 6 months +VE

Has not been able to do tasks for 6 months -VE

Note: 1 = If the patient was unable to perform the activity, 2 = if the patient was able to
perform the activity with assistance, and 3 = if the patient re­quired no assistance to
perform the activity
The goal of the scale is to assess an elderly person's competence in the
physical activities of daily living. Item8 can be written +ve or –ve statement
Three Criteria for Good Measurement

1.
1.Reliability
Reliability 2.
2.Validity
Validity

Good
Good
Measurement
Measurement

3.
3.Sensitivity
Sensitivity
 Test-retest reliability

Stability
Parallel-form reliability
Reliability
(accuracy in
measurement) Inter-item consistency reliability
[Carry out Reliability Analysis using Cronbach’s Alpha (α)]
Goodness of Consistency
Measures Split-half reliability
(data)
Validity (are we
measuring the
right things?)

Criterion-related Congruent validity

Logical validity
validity (construct)
(content)

Face validity Predictive Concurrent Convergent

(construct) Discriminant
(Carry out during pre-test)

Goodness of measures: Forms of reliability and

validity These are done by
statistics program for you
1. Reliability
Reliability
• The degree to which measures are free from random
error and therefore yield consistent results.
• An indicator of a measure’s internal consistency.
• A test of how consistently a measuring instrument
measures whatever concept it is measuring.
Internal Consistency
• Represents a measure’s homogeneity or the extent to
which each indicator of a concept converges on some
common meaning.
• Measured by correlating scores on subsets of items
making up a scale.
Internal Consistency: 2 ways to measure/assess
1. Split-half Method/Split half reliability
• Assessing internal consistency by checking the results of one-half
of a set of scaled items against the results from the other half.
2. Coefficient alpha (α)/Inter-item consistency reliability [Cronbach’s
alpha value (α)]
• The most commonly applied estimate of a multiple item scale’s
reliability.
• Represents the average of all possible split-half reliabilities for a
construct.
• Cronbach’s alpha value should be 0.7 and above to be reliable.
A Rule of Thumb to Interpret Cronbach’s
Alpha (α)
A Rule of Thumb to Interpret Cronbach’s Alpha (α)

α > 0.9 – excellent

α > 0.8 – good
α > 0.7 – acceptable
α > 0.6 – questionable
α > 0.5 – poor
α < 0.4 – unacceptable
Source: George, Darren & Mallery, Paul (2001). SPSS for Windows Step by Step: A Simple Guide and Reference,
10.0 Update. Third Edition. Allyn & Bacon, USA. (Chapter 18/Relaibility Analysis, P: 217).
How to run reliability analysis using
Cronbach’s Alpha Method?
How to run reliability analysis using Cronbach’s Alpha Method?

Reliability Statistics Reliability analysis result:

Cronbach’s Alpha = 0.922
Cronbach's Alpha N of Items indicating excellent reliability
.922 19
Test-Retest Reliability
Test-retest Method
• Administering the same scale or measure to the same respondents
at two separate points in time to test for stability.
• Represents a measure’s repeatability.
Problems:
• The pre-measure, or first measure, may sensitize the respondents
and subsequently influence the results of the second measure.
• Time effects that produce changes in attitude or other maturation
of the subjects.
2. Validity
Validity
• The accuracy of a measure or the extent to which a score truthfully
represents a concept.
• Does a scale measure what was intended to be measured?
• A test of how well an instrument that is developed measures the particular
concept it is intended to measure
Establishing Validity:
• Is there a consensus that the scale measures what it is supposed to
measure?
• Does the measure correlate with other measures of the same concept?
• Does the behavior expected from the measure predict actual observed
behavior?
Content Validity
Content Validity
• The degree that a measure covers the breadth (all the scope) of
the domain/construct of interest.
• Does the measure adequately measure the concept?
Face Validity
• A scale’s content logically appears to reflect what was intended to
be measured.
• Do “expert” validate that the instrument measures what its name
suggests to measure
Criterion-related Validity
Criterion Related Validity
• The ability of a measure to correlate with other standard
measures of similar constructs or established criteria.
• Does the measure differentiate in a manner that helps to predict
a criterion variable?
Methods:
Concurrent Validity
• Does the measure differentiate in a manner that helps to predict
a criterion variable currently?
Predictive Validity
• Does the measure differentiate individuals in a manner that
helps predicts a future criterion?
Congruent Validity (construct)
Construct Validity
• Exists when a measure reliably measures and truthfully
represents a unique concept.
• How well the results obtained from the use of the measure
fit the theories around which the test is designed.
Methods of assessment:
Convergent Validity
• Another way of expressing internal consistency; highly reliable scales contain
convergent validity.
• Do two instruments measuring the concept correlate highly?
Discriminant Validity
• Represents how unique or distinct is a measure; a scale should not correlate too
highly with a measure of a different construct.
• Does the measure have a low correlation with a variable that is supposed to
unrelated to this variable?
Reliability and Validity on Target
3. Sensitivity
Sensitivity
• A measurement instrument’s ability to accurately measure
variability in stimuli or responses.
• Generally increased by adding more response points or
adding scale items (e.g. from 5-point Likert to 10-point Likert
scale).
• 5-point Likert (1-5: 1, 2, 3, 4, 5)
• 10-point Likert (1-10: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
Ways of Doing Survey Research
1. Personal Interview
2. Telephone Interview
3. Self-Administered Questionnaire
1. Types of Personal Interviews
a. Door-to-Door Interview
b. Shopping Mall Intercept Interview
1. Personal Interviews

• A personal interview is a form of direct communication in

which an interviewer asks respondents questions face-to-
face.
• Versatile and flexible
• Truly interactive (feedback are possible)
Advantages of Personal Interviews
Opportunity
for Feedback
Probing Complex
High Participation Answers

Personal
Personal
Props and Visual Interviews
Interviews Length of
Aids Interview

Completeness of
Questionnaire
Disadvantages of Personal Interviews
Lack
Lackof
of
Interviewer
Interviewer Anonymity
Anonymityofof
Influence
Influence Respondent
Respondent

Personal
Personal
Interviews
Interviews

Cost
Cost
a. Door-to-Door Interviews
• Personal interviews conducted at respondents’ doorsteps in
an effort to increase the participation rate in the survey.
• Callbacks (follow-up)
• Attempts to re-contact individuals selected for a sample
who were not available initially.
Door-to-Door Interview Characteristics
• Speed of data collection: • Degree of interviewer influence of
moderate to fast answer: high
• Geographical flexibility: limited • Supervision of interviewers: moderate
to moderate • Anonymity of respondent: low
• Respondent cooperation: • Ease of call back or follow-up: difficult
excellent
• Cost: highest due to travel costs
• Versatility of questioning: quite
versatile • Special features: visual materials may
be shown or demonstrated; extended
• Questionnaire length: long probing possible
• Item nonresponse: low
• Possibility of respondent
misunderstanding: lowest
b. Mall Intercept Personal Interview
• Personal interviews conducted in a shopping mall.
• Interviewers typically intercept shoppers at a central point
within the shopping center or at the main entrance.
Mall Intercept Interview Characteristics
• Speed of data collection: fast • Degree of interviewer influence of
• Geographical flexibility: confined, answers: highest
urban bias • Supervision of interviewers:
• Respondent cooperation: moderate to high
moderate to low • Anonymity of respondent: low
• Versatility of questioning: • Ease of call back or follow-up:
extremely versatile difficult
• Questionnaire length: moderate to • Cost: Lower than door-to-door
long
• Special features: taste (rasa) tests,
• Item nonresponse: medium viewing of TV commercials possible
• Possibility of respondent
misunderstanding: lowest
Personal Interviews
• Global Considerations
• Variations in willingness to participate
• Sensitivity to interview subject matter (few would
consent to be interviewed on certain topics)
• Beliefs about appropriate business conduct (difficult
to conduct personal interview during business hours
in Japan)
2. Telephone Interviews
Telephone Interviews
• Personal interviews conducted by telephone.
• The mainstay 9 (cara yg kerap digunakan) of commercial
survey research.
• “No-call” legislation has limited this capacity.
Telephone Interviews
Mobile Phone Interviews
• In U.S., no telemarketing can be directed toward
mobile phone numbers by law.
• Recipient of call is even more likely to be distracted
(driving, etc).
• Area codes not necessarily tied to geography – this
could pose a problem for sampling.
• Phones have varying abilities for automated responses
and differing keypads.
Phone Interview Characteristics
• Speed – many can be conducted
• Cost – high but becoming inexpensive
• Absence of face-to-face contact - impersonal
• Cooperation – response rate have fallen
• Incentives to respond – financial incentive
• Representative samples – a problem, list not complete and do not
reflect geographical location.
• Callbacks – improve response rate by making multiple attempts
• Limited duration – people are impatience with long interview & can
hang up – make it short.
• Lack of visual medium - cannot use visual aids
Types of Telephone Interviews
a. Central Location Interviewing
• Conducting interviews from a central location allowing firms to
hire a staff of professional interviewers and to supervise and
control the quality of interviewing more effectively.
b. Computer-Assisted Telephone Interviewing (CATI)
• Allows answers to telephone interviews to be entered directly into
a computer for processing.
c. Computerized Voice-Activated Telephone Interview
• Combining computerized telephone dialing and voice-activated
computer messages to allow researchers to conduct telephone
interviews without human interviewers.
Automated Phone Surveys of
Teens
• Computer-assisted telephone interviewing (CATI) was
used to ask teens about smoking.
• By pressing keys on the keypad in the self-administered
survey, teens were more likely to say they smoked.
Telephone Interview Recap
• Speed of data collection: very fast • Supervision of interviewers: high,
• Geographical flexibility: high especially with central location
interviewing
• Respondent cooperation: good
• Anonymity of respondent:
• Versatility of questioning: moderate moderate
• Questionnaire length: moderate • Ease of call back or follow-up: easy
• Item nonresponse: medium
• Cost: low to moderate
• Possibility of respondent
• Special features: fieldwork and
misunderstanding: average
supervision of data collection are
• Degree of interviewer influence of simplified; quite adaptable to
answer: moderate computer technology
3. Self-Administered Questionnaires Can Be Either Printed or
Electronic
Mail Questionnaires
Characteristics of Mail Questionnaires
• Geographical flexibility – good for reaching geographically
dispersed sample simultaneously
• Cost - inexpensive
• Respondent convenience – can response at when free
• Anonymity of respondent – confidentiality is high
• Absence of interviewer – interviewer bias is low
• Standardized questions – very structured questions
• Time is money – specific the time to complete the
questionnaire – follow-up mailing when the necessary
• Length of mail questionnaire – not too long
Self-Administered
Questionnaires
Response Rate
• The number of questionnaires returned or completed
divided by the number of eligible people (sample size)
who were asked to participate in the survey.
Factors that Bias the Response Rate
• Persons who will complete questionnaires versus those
persons who will not.
• Person filling out survey is not the intended subject.
Increasing Response Rates for Mail Surveys
• Cover letter
• Money helps
• Interesting questions
• Follow-ups
• Advance notification- use letter or
telephone
• Survey sponsorship like well known
organization increase response rate
• Other techniques – envelope with
stamps, color of questionnaire, etc
• Keying mail questionnaires with codes
A Cover Letter Requesting Participation in a Survey
Fax Surveys
• A survey that uses fax machines as a way for respondents to
receive and return questionnaires.
Advantages
• Reduce sender’s printing and postage costs
• Is quicker than traditional mail surveys
Disadvantage
• Only respondents with fax machines who are willing to exert
the extra effort will return questionnaires.
E-Mail Surveys
• Surveys distributed through electronic mail.
• Ways to contact respondents:
• Include a questionnaire in the body of an e-mail.
• Distribute questionnaire as an attachment.
• Include a hyperlink within the body of an e-mail.
Advantages
• Speed of distribution
• Lower distribution and processing costs
• Faster turnaround time
• More flexibility
• Less handling of paper questionnaires
Disadvantage
• Not all e-mail systems have the same capacity
Internet Surveys
• A self-administered questionnaire posted on a
website.
• Respondents provide answers to questions
displayed online by highlighting a phrase, clicking
an icon, or keying in an answer.
Internet Survey Characteristics
• Speed and cost effectiveness • Callbacks – automatically resend e-
• Visual appeal and interactivity mail remainder
• Respondent participation and • Personalized and flexible
cooperation questioning
• Representative samples • Respondent anonymity
• Accurate real-time data capture – • Response rates
each respondent’s answers are • Security concerns – worry about
entered directly into researcher’s PC hackers others accessing the
as soon as the questionnaire is websites.
submitted
Other Approaches
Kiosk Interactive Surveys
• Placed in high-traffic locations (e.g., airports).
Mixed-Mode Survey Research
• Employs any combination of survey methods.
Text-Message Surveys
• May use SMS (short-message service) or MMS (Multi-
Media Service).
Mixed Mode Data Collection: The Case of Web and Telephone Surveys

• A major criticism of web surveys

is the low response rate.
• A “sequential mixed-mode data
collection” approach showed this
to be true compared to
telephone surveys.
• But the information was the
same and was obtained faster
and more cost-effectively.
INSTRUMENTATI
ON:
QUESTIONNAIRE
S
Why Use Instrument/questionnaire
• The questionnaire translates the research objective into specific
questions.
• The answers to those questions provide the data for testing the
research hypothesis.
• Questions must also interest the respondents enough that they
will provide the information.
Instrumentation - THREE Steps

STEP 1
STEP 2
Create the STEP 3
Conceptual Produce the
Model Questionnaire Pre-test and Pilot
test
STEP 1
Create Research Problem:
the • Some senior managers are not making
Conce good decisions on the job
ptual
Research Purpose:
Model
• Determine which factors lead to better
decision-making on the job
Research Design:
• Cross-sectional survey of senior managers
Population:
• Currently employed senior managers
STEP 1 (Cont’d)
Data Analysis:
Create
• Indicate the level of measurement needed for each question
the
Hypotheses:
Conce
• The more training in quantitative methods, the better the decision-
ptual making
Model • The more recent the training in quantitative methods, the better the
decision-making
• The quality of decision making will vary across departments
Variables:
• Decision-making quality (ratio level)
• Amount of training in quantitative methods (ratio level)
• Recency of training in quantitative methods (ratio level)
• Department of employment (nominal level)
STEP 1 (Cont’d)
Variable Name Operational Definition: Measurement Indicator

Decision-making quality The extent to which supervisors' personnel Q1. How many personnel decisions did you make between
decisions are questioned by superiors or January 1 and June 30? ___
subordinates (percentage) Q2. How many of the personnel decisions you made between
January 1 and June 30 were questioned by either superiors or
subordinates? ___

Amount of quantitative methods The amount of training in quantitative Q3. How many quantitative methods
training methods courses did you complete in college? ___

Recency of training in The recency of training in quantitative Q4. In what month and year did you complete your last
quantitative methods methods university quantitative methods
course?
month___ year___

Employment Department in the organization in which the Q5. In which department are you currently employed?
supervisor is employed _____Accounting
_____Sales
_____Engineering
STEP 2 (Cont’d)
Prod A. Write the introduction
uce The introduction:
the • explains the nature and purpose of the survey
Que
stion • names the person to contact in case of questions
nair • says whether the questionnaire is anonymous
e • says whether the information will be kept
confidential
• enumerates the respondent's rights
• stresses the importance of the study
• thanks the respondent for participating
STEP 2 (Cont’d)
Prod B. Write the questions
uce • 1. Bring together key actors in the question and response
the writing session, to make sure that the type of data that is
Ques needed will be produced by the questionnaire.
tionn • 2. There must be at least one questions for every
aire variable of importance in the study; and there may be
more than one question for each variable.
For example, to measure a worker's job satisfaction, there
may be three questions:
(a) satisfaction with working conditions,
(b) satisfaction with pay, and
(c) satisfaction with promotional opportunities.
 STEP 2 (Cont’d)
Prod B. Write the questions
uce 3. Questions must be designed to elicit data measured at
the the desired level--nominal, ordinal, interval or ratio--so that
Ques the appropriate statistics may be applied and the questions
tion posed by the research may be adequately answered.
naire
4. Questions can elicit also different types of information:
facts, values, and knowledge.
Factual questions: age, sex, marital status, annual income,
job title, education;
Value questions: attitudes, opinions, expectations;
Knowledge questions: tests, etc.
STEP 2 (Cont’d)
Prod B. Write the questions
uce 5. Questions can have two general types of response options:
the 1. open-ended or 2. closed-ended.
Que a. Closed-ended response options provide respondents with
stio a limited set of response choices, such as Yes or No; low,
nnai medium or high; high school, bachelor, masters; etc.
re
Example 1: "Why did you leave your last job?" (Check the one
most important reason)
1. _____Better pay
2. _____Better opportunities
3. _____Fired
4. _____Personal reasons.
Advantages & Disadvantages of Closed-Ended
Questions
Advantages:
• Require less interviewer skill
• Take less time to answer
• Are easier for the respondent to answer
• Provides comparability of answers
• Easier to analyze
Disadvantages:
• Lack of range in the response alternatives
• Tendency of respondents to choose convenient
alternative
 STEP 2 (Cont’d)
Prod B. Write the questions
uce 5. Questions can have two general types of response options: 1.
the open-ended or 2. closed-ended.
Que a. Closed-ended response options provide respondents with a
stio limited set of response choices, such as Yes or No; low, medium or
high; high school, bachelor, masters; etc.
nnai
re • Example 2: "What is your marital status?" (Check the box that
applies)
1. Single, never married
2. Married
3. Divorced
4. Separated
5. Widowed
6. Other:_____
STEP 2 (Cont’d)
Pro B. Write the questions
duc 5. Questions can have two general types of response
e options: open-ended or closed-ended.
the a. Closed-ended response options provide respondents
Qu with a limited set of response choices, such as Yes or No;
esti low, medium or high; high school, bachelor, masters; etc.
onn
aire
Example 3: "How satisfied are you with your job?" (Circle
the number that represents your response)
Very dissatisfied Dis-satisfied A little Satisfied Satisfied Very satisfied
1 2 3 4 5
STEP 2 (Cont’d)
Pro B. Write the questions
duc 5. Questions can have two general types of response options: open-ended or
e closed-ended.
the a. Closed-ended response options
Qu Matrix questions are merely a series of questions that share the same set
esti of closed-ended response options. For example:
onn Question Very
Dissatisfied
Dissatisfied Satisfied Very
Satisfied
aire
1. How satisfied are you with your working 1 2 3 4
conditions?

2. How satisfied are you with your pay? 1 2 3 4

3. How satisfied are you with your supervisor? 1 2 3 4

STEP 2 (Cont’d)
Pro B. Write the questions
duc 5. Questions can have two general types of response
e options: open-ended or closed-ended.
the
a. Closed-ended response options provide respondents
Qu
with a limited set of response choices, such as Yes or No;
esti
low, medium or high; high school, bachelor, masters; etc.
onn
aire
Example 3: "How satisfied are you with your job?" (Circle
the number that represents your response)
Very dissatisfied Dis-satisfied A little Satisfied Satisfied Very satisfied
1 2 3 4 5
STEP 2 (Cont’d)
B. Write the questions
Pro 5. Questions can have two general types of response options: open-ended or closed-ended.
duc b. Open-ended response options provide respondents with the opportunity to respond to the
e question in whatever way they see fit.
the For example, "Why did you leave your last job?"
(followed by blank space for the respondent to write in the response)
Qu
esti Advantages of open-ended response options:
onn • allow the respondents to suggest a range of possibilities unknown to the researchers;
aire • avoid the bias of a list of response possibilities;
• generate rich details, key words and phrases;
• are sometimes easier (e.g., write in your state rather than pick from a list of all states);
• are often used in exploratory research or in early stages of a project;
• can generate ideas for future closed-ended response options;
• can gather data to explain responses to closed-ended response options (if you said yes,
why?);
• allows the respondent to have input to the researcher or the sponsor of the research;
• can promote a positive feeling for the respondent, that the input is valuable;
STEP 2 (Cont’d)
Pro B. Write the questions
duc 5. Questions can have two general types of response options: open-ended
or closed-ended.
e
b. Open-ended response options provide respondents with the
the opportunity to respond to the question in whatever way they see fit.
Qu
esti
Disadvantages of open-ended response categories:
onn
• need careful pre-testing;
aire
• data may be more difficult to code and analyze;
• may be left unanswered more often than closed-ended response
categories;
• requires motivation on the part of the respondent;
• requires well-trained interviewers for in-person interviews;
• takes more time for analysis for large samples.
STEP 2 (Cont’d)
Pro C. Question Wording and Response Wording
duc 1. Questions need to be clear, specific, and short
e 2. Responses must fit the question
the 3. Questions can only ask one thing; divide compound questions into
two
Qu 4. Responses need to elicit data measured on the appropriate scale
esti 5. Endeavor (try) to lessen discomfort, harm, or embarrassment to
onn the respondent
aire 6. Respondents tend to choose socially accepted answers
7. Include equal numbers of positive and negative options on rating
scales
8. Avoided "loaded" or "hot button" words
9. Avoid acronyms, abbreviations, jargon, etc.
10. Avoid questions with conjunctions (and, but, or)
STEP 2 (Cont’d)
Pro D. Order the questions
duc 1. Include clear directions on how to answer questions;
e whenever the type of response category changes, be sure to
the include relevant directions.
Qu
2. Questions that establish eligibility
esti
onn 3. Put relevant and easy to answer questions first, to build
aire interest and rapport with the respondent
4. Group major questions into sets by subject or issue (by
your IV’s and DV). Provide transition statements to orient
the respondent when the issue or subject changes.
5. "Response Set": avoid putting many questions together
which the respondent is likely to answer all the same way.
STEP 2 (Cont’d)
Pro D. Order the questions
duc 6. Put demographic or personal information questions at
e the end. This is called a funnel sequence, with broader and
the more general questions first, narrowing to more sensitive or
Qu open-ended questions at the end.
esti
7. Put a brief "thanks" at the end of the questions
onn
aire 8. Include instructions on when and where to return the
questionnaire if applicable.
 STEP 3
Pre- A. Pre-test the questionnaire
test 1. Show it to colleagues and key decision-makers. Are the questions
and and responses valid and reliable? Are they appropriate? Are they
Pilot both necessary and sufficient?
test 2. Administer the questionnaire to a small number of people who
resemble or are drawn from the population of interest, but it does
not have to be a random sample.
• Measure how much time it takes to complete each questionnaire.
• Debrief the respondents (and the interviewers if applicable) after
they complete the questionnaire.
• Analyze the information provided to clarify directions, question
wording, or response categories where necessary.
• Revise as needed.
 STEP 3 (Cont’d)
Pre- B. Pilot Study
test • A pilot study is a dress referral of the full project,
and including the questionnaire, the interviewers, and all
Pilot other aspects.
test • Often a sample of 10-30 responses are obtained, coded,
and analyzed.
• Then, run reliability analysis (RA) using Cronbach’s
Alpha (α)
• Questions that are not providing useful data are
discarded, and the final revisions of the questionnaire
are made.
 Thank
you
For Listening