Designing Research Instrument

Quantitative Research Analysis

STAT 201/ RESED 321

Fernando T. Herrera, PhD

Department of Mathematics
College of Mathematics & Natural Sciences
OBJECTIVES

Identify the
appropriate strategy
for questionnaire
design

Develop an instrument
🞂 INSTRUMENT 🞂 INSTRUMENTATIO
🞂 is a tool used to N
collect, measure, 🞂 is the process of
and analyze data constructing
related to your research instruments
research interests. that could be used
appropriately
in gathering data.
Some considerations/steps
Define all
constructs

Consider the
SOP or
Pilot testing Objectives

Identify
variables that
represent the
construct
Consult expert
for review

Revise to Develop
correct issues specific items
(structure,
grammar, etc)
Consider this…
Validating questionnaire
establish face •content expert
•psychometrician
validity
•on a subset of your intended population
pilot test the survey •20 participant per question, or 60 for 8-15
items

code & examine • enter the responses into a spreadsheet and clean the
data
data • remediate missing value / outliers

check the internal

•standard test of internal consistency is Cronbach’s Alpha
(CA)
•at least 0.70 or higher although a value from 0.60 to 0.70
consistency is acceptable

revise the survey • based on information gleaned from the PCA and CA
• repeat process for major changes
Face validity is an informal review of a
questionnaire by non-experts, who assess its
clarity, comprehensibility, and appropriateness
for the target-group, whilst content validity
involves a formal assessment by subject
experts, to determine appropriateness of
content and identify any misunderstandings or
omissions.
Considerations in establishing face validity

Avoid questions that prompt or motivate the respondent to say what

you would like to hear

Consider the social desirability of responses to questions, and write

questions that elicit honest responses

Use specific questions instead of general ones, Keep it simple

if possible. and clear

Ask only one Pay Decide

concept per attention to whether to
Avoid double question use open or
question… avoid
negatives. double-barreled order closed
questions. effects questions.
Most common ways to
construct scale
🞂 Likert scale
🞂 Thurstone scale
🞂 Guttman scale
🞂 Semantic Differential
LIKERT SCALE
🞂 A Likert scale is a 🞂 Items generation
psychometric scale
commonly involved in
🞂 Rating items to
research that employs generate scale 1-
questionnaires. It is the most least favorable to 5-
widely used approach to
scaling responses in survey most favorable)
research, such that the term 🞂 Use of mean &
is often used standard deviation
interchangeably with rating
scale, although there are
for selection of
other types of rating scales. items
Description How to generate?
Likert Scale
… is typically a five, seven, or nine-point
agreement scale used to measure respondents'
agreement with various statements.
THURSTONE SCALE
🞂 Thurstone scale is 🞂 Items generation
defined as a 🞂 Ranking items to
unidimensional scale
generate scale 1-
that is used to track
respondent's behavior, least favorable to
attitude or feeling 11-most favorable)
towards a subject. 🞂 Use of mean & IQR
for selection of
items
 GENERATING
THURSTONE SCALE
Step 1 - Develop statements

Develop a large
number of
agree/disagree
statement

Click this link for the google form.

https://forms.gle/nYMCbqnqG7qBZxo
n8
Step 2 – Rank each statement
with a panel of judges

Judges rate each

item on a scale of
1 to 11 where 1 is
the least
favorable attitude.

Link for the google sheet

https://docs.google.com/spreadsheets/d/
1thgD_TB8ECUmQ2HSj2CHXzgWxIX3pOpx
JQWC4KcZHqI/edit?usp=sharing
Step 2 – Rank each statement
with a panel of judges
It is important to note that
the…
 Judges are required to rate each
option and not agree or disagree
with them.
 Judges will use all ratings from 1 to
11 allowing ties with option ratings.

Link for the google sheet

https://docs.google.com/spreadsheets/d/
1thgD_TB8ECUmQ2HSj2CHXzgWxIX3pOpx
JQWC4KcZHqI/edit?usp=sharing
Step 3 – Calculate median and
Interquartile range (IQR)

The median is the middle

number in a sorted, ascending or
descending, list of numbers

IQR is the difference between

smallest value and the largest
value of the middle 50% of a set of
data.
Step 3 – Calculate median
mean and Interquartile range
(IQR)

Link to excel file

Step 3_Thurstone scale generation.xlsx
Step 4 – Sorting the table

The statements (or options) have

to be sorted as follow
 smallest to largest median, and
 for each median, IQRs have to be in a
descending order.

Link to excel file

Step 3_Thurstone scale generat
ion.xlsx
Step 5 – Selecting final
variables or options
 Select one item from
each mean/median
value
 Statements with the most
agreement between judges
(meaning lowest IQR)

Link to excel file

Step 3_Thurstone scale generat
ion.xlsx
Step 5 – Selecting final
variables or options
You don’t have to
choose this item based
on median & IQR. If you
decide it’s poorly
worded or ambiguous,
choose the item above it
(with the next lowest
IQR).

Link to excel file

Step 3_Thurstone scale generat
ion.xlsx
Semantic Differential
Scale
Semantic differential is a type of a rating scale
designed to measure the connotative meaning of
objects, events, and concepts - to derive the attitude
towards the given object, event or concept

A semantic differential scale

is a survey or questionnaire
rating scale that asks people
to rate a product, company,
brand, or any 'entity' within
the frames of a multi-point
rating option.
Important Principles Relating Reliability and Validity

🞂 First, a test can be considered reliable, but not

valid.
🞂 Second, validity is more important than reliability.
🞂 Finally, the most useful instrument is both valid
and reliable.
RELIABILITY TESTS
🞂 Inter-Rater/Observer Reliability
The degree to which different
raters/observers give consistent answers or
estimates.
🞂 Test-Retest Reliability
The consistency of a measure evaluated
over time.
🞂 Parallel-Forms Reliability
The reliability of two tests constructed the
same way, from the same content.
🞂 Internal Consistency Reliability
The consistency of results across items, often
measured with Cronbach’s Alpha.
Internal Consistency Measures
🞂 Average Inter-Item Correlation
🞂 Uses all items on the instrument that are
🞂 designed to measure the same construct.
🞂 Here, we first compute the correlation
🞂 between each pair of items.

🞂 Average Itemtotal Correlation

🞂 This approach also uses the inter-item
🞂 correlations. In addition, we compute a total
🞂 score for the six items and use that as a
🞂 seventh variable in the analysis.
🞂 Split-Half Reliability
🞂 In split-half reliability we randomly divide all items

🞂 that intend to measure the same construct into

🞂 two sets. We administer the entire instrument to
a
🞂 sample of people and calculate the total score for
🞂 each randomly divided half. the split-half
reliability
🞂 estimate
🞂 Cronbach’s Alpha (a)

Cronbach's alpha is a way of assessing

reliability by comparing the amount of shared
variance, or covariance, among the items
making up an instrument to the amount of
overall variance. The idea is that if the
instrument is reliable, there should be a great
deal of covariance among the items relative to
the variance.
🞂 Cronbach’s alpha quantifies the level of agreement on
a standardized 0 to 1 scale. Higher values indicate
higher agreement between items. High Cronbach’s
alpha values indicate that response values for each
participant across a set of questions are consistent.
LET US COMPUTE
CRONBACH ALPHA…