Instrumentasi:

Pembentukan Instrumen,
Kesahan dan
Kebolehpercayaan
Instrumen

Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

Main Steps in Quantitative

Research:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Theory
Hypothesis
Research design
Plan/Develop/Create measures of concepts
Select research site(s)
Select research subjects/respondents
Administer research instruments/ collect data
Process data
Analyse data
Write up findings and conclusions
Powerpoint Templates

Examples of Quantitative Research

Methods:
Experiments
Social surveys
Cross-sectional
Comparative (cross-national)
Longitudinal

Content Analysis
Statistical Analysis
Official Statistics
Demography
Epidemiology

Field stimulations
Structured Interviews and Observation
Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

Survey/Tinjauan
Terms survey and questionnaire are
often used interchangeably
Survey refers to a descriptive
research technique or methodology
(also includes interviews, and focus
groups)
Questionnaire refers to a paper and
pencil instrument through which
information is obtained

Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

 Useful for surveying groups of any size

Typically designed to determine
opinions, attitudes or present practices
Limitation: rely on self report
information v. observed behavior
Challenges to validity become a
concern

Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

Validity
Validity: the extent to which the test measures
what it significances/meaning/senses to
measure
Internal Validity: Relates to instrument
(questionnaire) quality. Are the appropriate
questions asked clearly and logically?
External validity: can the results be
generalized?
Measurement error: difference between the
characteristics of your sample and the
characteristics of the population
Non-response bias: answers of respondents
do not mirror the characteristics of those
who refused participation.
Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

Create a questionnaire that is clear

and logical
Content Relevance And Content
Representativeness
Content Relevance: are the questions relevant to
the purpose of the questionnaire? Unimportant
content should be eliminated.
Content Representativeness: Are the questions an
adequate representation of the universe of
possible questions?
Solicit the input of experts in scale development
and the content of the questionnaire for an
objective evaluation
Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

QUESTIONNAIRE PLANNING
Determine the Objective
Is the development of a new questionnaire
Necessary?
Feasible ? (practicable/viable/achievable/possible..etc)

What do you want to ask and of whom?

Consider how the data will be analyzed. What information
is needed?

Select mode of administration

Web Based?
Paper and pencil?
Combination?

Powerpoint Templates

QUESTIONNAIRE PLANNING
Determine the Sampling Methodology
To whom will you send the questionnaire? Who can supply
the information?
Will everyone be surveyed?
ALL current students? ALL alumni?
If yes, this is a census of the finite population
The selection of a representative sample of the population
may be more effective based on time and cost of
administration
Decision depends on the size of the finite population

Powerpoint Templates

CONSTRUCTING the QUESTIONNAIRE

Consider what objective each question measures
Three to five questions per objective is recommended

How many questions?

Find a balance: long enough to obtain the necessary
information but short enough so respondents wont lose
interest.
Questionnaires that are too long wont be completed
Goal: 25-30 minutes MAX

Powerpoint Templates

DESIGNING THE
QUESTIONNAIRE

Question Formats: Open Ended

Questions
Allow respondents to express feelings and expand on ideas.
Question should be phrased to avoid a one word response.
What aspects of the Sociology program best prepared
you for your current position?
Open ended questions at the beginning of the questionnaire
may be discouraging to respondent include one or two at
the end
Analysis of responses can be challenging

Powerpoint Templates

QUESTIONNAIRE
DEVELOPMENT
Question Formats: Closed Questions
Ranking
Forces the respondent to rank order responses based on a
value judgment.
Rank the following items with regard to how you spend time at
Bucknell from 1 (spend the most time) to 5 (spend the least
time).
_____
Studying
_____
Socializing
_____
Volunteering
_____
Working at a part time job
_____
Exercising
Powerpoint Templates

QUESTIONNAIRE
DEVELOPMENT
Question Formats: Closed Questions
Checklist
A number of possible answers are provided and the
respondent is asked to choose one or choose all that apply.
How did you finance your Bucknell University education?
Choose all that apply.
_____
Parental contribution
_____
Student Loan
_____
Work Study
_____
Off campus employment
_____
Other (please specify): _____________
Powerpoint Templates

QUESTIONNAIRE
DEVELOPMENT
Question Formats: Closed Questions
Scaled items
Very common
Indicate strength of agreement or disagreement with a
statement.
Numbers are assigned to each response on the continuum
The English curriculum at Bucknell University adequately
prepared me for my current position.

2 e 3 / No 4
y
l
g
ee
ral on
gre
t
r
n
a
u
g
o
r
s
i
A
Ne pin
St gree Di
O
a
Dis
Powerpoint Templates

gly
n
o
Str ree
Ag

QUESTIONNAIRE
DEVELOPMENT
Question Formats: Closed Questions
Suggestions for writing scaled items
Use positively worded or negatively worded items
consistently
Use an odd number of responses along the continuum to
allow for a neutral response
Anchors may be different based on focus of the item
(Strongly Agree Strongly Disagree; Frequently Never;
Very Good Very Poor)
Keep items short
Powerpoint Templates

QUESTIONNAIRE
DEVELOPMENT

Question Formats: Categorical

Typical demographic questions:

Yes/No/?, Male/Female, F/S/J/S/G
Answers fall into certain categories respondents pick the one
that applies to them
Analyzed using frequency, proportion, percentages
2 considerations:
Questions on a continuum should stay on a continuum if they
will be directly analyzed
Example: Age allow respondent to write in their age
rather than check off a box for their age range
Categories must be collectively exhaustive & mutually
exclusive (next slide)
Powerpoint Templates

QUESTIONNAIRE
DEVELOPMENT
Problematic:
Number of hours spent
studying per week:
18 21
21 24
24 27
Categories are not
mutually exclusive

Corrected:
Number of hours spent
studying per week:
18 21
22 25
26 29

Powerpoint Templates

IMPORTANT
CONSIDERATIONS
APPEARANCE AND DESIGN
Have clear directions that are age appropriate & easy to
follow
Format, size, reproduction should facilitate completion
No less than 12 pt
Times, Arial, Courier
PILOT STUDY
Give the questionnaire to a few people to try it out
Have them evaluate readability and clarity of questions
Estimate time required to complete the questionnaire
Suggestions for revisions
Powerpoint Templates

SENDING IT OUT
INITIAL CONTACT / COVER LETTER
Short & to the point
Identify person by name
Convince respondent that participation is
important
Ensure confidentiality
Deadline for return of questionnaire
SASE for return (mailed questionnaires only)
Powerpoint Templates

FINAL THOUGHTS
FOLLOW UP PROCEDURES
Send to all participants to preserve confidentiality (If you
havent already returned the questionnaire..)
May include a second copy if available
Usually done in 2 week intervals

RESPONSE RATE FOR MAILED QUESTIONNAIRES

Typically low - 30% is considered good
Higher response rate with web-based questionnaires
Good follow-up increases response rate

Powerpoint Templates

SUMMARY OF STEPS
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Determine the Objective (consider analyses)

Determine the Mode of Administration
Determine the Sampling Methodology
Construct the Questionnaire
Institutional Approval
Conduct the Pilot Study
Write the Initial Communication
Send the Questionnaire
Follow up
Analyze the Results

Powerpoint Templates

Research design and methods

Develop your own

research design
and methods of
your study

Powerpoint Templates

DEVELOPING QUALITATIVE
INSTRUMENT

Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

Main Steps in Qualitative

Research:
1.
2.
3.
4.
5.
6.
7.
8.
9.

General research question

Select relevant site(s) and subjects
Collection of relevant data
Interpretation of data
Conceptual and theoretical work
Tighter specification of the research question
Collection of further data
Conceptual and theoretical work
Write up findings
Powerpoint Templates

Examples of Qualitative Research

In-depth Interviews
Focus Groups
Ethnography/Field Research
Historical-Comparative Research
Discourse Analysis
Narrative Analysis
Media Analysis

Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

RELIABILITY & VALIDITY OF

INSTRUMENT - QUANTITATIVE
INSTRUMENT

Powerpoint Templates

Mengilham harapan, mencipta masa depan

Inspiring Futrures, Inspiring Possibilities

Criteria of Measurement Quality

How do we judge the relative success (or
failure) in measuring various concepts?
Reliability consistency of measurement
Validity confidence in measures and design

Powerpoint Templates

Reliability and Validity

Reliability focuses on measurement
Validity also extends to:
Precision in the design of the study ability to
isolate causal agents while controlling other
factors
(Internal Validity)

Ability to generalized from the unique and

idiosyncratic settings, procedures and participants
to other populations and conditions
(External Validity)
Powerpoint Templates

Reliability
Consistency of Measurement
Reproducibility over time
Consistency between different
coders/observers
Consistency among multiple indicators

Estimates of Reliability
Statistical coefficients that tell use how
consistently we measured something
Powerpoint Templates

Measurement Validity
Are we really measuring concept we defined?
Is it a valid way to measure the concept?

Many different approaches to validation

Judgmental as well as empirical aspects

Powerpoint Templates

Key to Reliability and Validity

Concept explication
Thorough meaning analysis

Conceptual definition:
Defining what a concept means

Operational definition:
Spelling out how we are going to measure
concept

Powerpoint Templates

Four Aspects of Reliability:

1. Stability
2. Reproducibility
3. Homogeneity
4. Accuracy

Powerpoint Templates

1. Stability
Consistency across time
repeating a measure at a later time to
examine the consistency
Compare time 1 and time 2

Powerpoint Templates

2. Reproducibility
Consistency between observers
Equivalent application of measuring
device
Do observers reach the same conclusion?
If we dont get the same results, what are we
measuring?
Lack of reliability can compromise validity

Powerpoint Templates

3. Homogeneity
Consistency between different measures
of the same concept
Different items used to tap a given concept
show similar results ex. open-ended and
closed-ended questions

Powerpoint Templates

4. Accuracy
Lack of mistakes in measurement
Increased by clear, defined procedures
Reduce complications that lead to errors

Observers must have sufficient:

Training
Motivation
Concentration

Powerpoint Templates

Increasing Reliability
General:

Training coders/interviewers/lab personnel

More careful concept explication (definitions)
Specification of procedures/rules
Reduce subjectivity (room for interpretation)

Survey measurement:

Increase the number of items in scale

Weeding out bad items from item pool

Content analysis coding:

Improve definition of content categories

Eliminate bad coders
Powerpoint Templates

Indicators of Reliability
Test-retest
Make measurements more than once and see if they
yield the same result

Split-half
If you have multiple measures of a concept, split
items into two scales, which should then be correlated

Cronbachs Alpha or Mean Item-total Correlation

Powerpoint Templates

Reliability and Validity

Reliability is a necessary condition for
validity
If it is not reliable it cannot be valid

Reliability is NOT a sufficient condition for

validity
If it is reliable it may not necessarily be valid

Example:
Bathroom scale, old springs
Powerpoint Templates

Not Reliable or Valid

Powerpoint Templates

Reliable but not Valid

Powerpoint Templates

Reliable and Valid

Powerpoint Templates

Types of Validity

1. Face validity
2. Content validity
3. Pragmatic (criterion) validity
A. Concurrent validity
B. Predictive validity

4. Construct validity

A. Testing of hypotheses
B. Convergent validity
C. Discriminant validity

Powerpoint Templates

Face Validity
Subjective judgment of experts about:
whats there
Do the measures make sense?

Compare each item to conceptual definition

Do it represent the concept in question?
If not, it should be dropped
Is the measure valid on its face
Powerpoint Templates

Content Validity
Subjective judgment of experts about:
what is not there
Start with conceptual definition of each
dimension:
Is it represented by indicators at the
operational level?
Are some over or underrepresented?
If current indicators are insufficient:
develop and add more indicators
Example--Civic Participation questions:
Did you vote in the last election?
Do you belong to any civic groups?
Have you ever attended a city council
meeting?
What about protest participation or online
organizing?
Powerpoint Templates

Pragmatic Validity
Empirical evidence used to test validity

Compare measure to other indicators

1. Concurrent validity

Does a measure predict simultaneous

criterion?
Validating new measure by comparing to existing measure
E.g., Does new intelligence test correlate with established test

2. Predictive validity

Does a measure predict future criterion?

E.g., SAT scores: Do they predict college GPA?
Powerpoint Templates

Construct Validity
Encompasses other elements of validity
Do measurements:

A. Represent all dimensions of the concept

B. Distinguish concept from other similar concepts

Tied to meaning analysis of the concept

Specifies the dimensions and indicators to be

tested

Assessing construct validity

A. Testing hypotheses
B. Convergent validity
C. Discriminant validity

Powerpoint Templates

A. Testing Hypotheses
When measurements are put into practice:

Are hypotheses that are theoretically derived,

supported by observations?
If not, there is a problem with:

A. Theory
B. Research design (internal validity)
C. Measurement (construct validity?)

In seeking to examine construct validity:

Examine theoretical linkages of the concept to

others
Must identify antecedent and consequences
What leads to the concept?
What are the effects of the concept?

Powerpoint Templates

B. Convergent Validity
Measuring a concept with different methods
If different methods yield the same results:
than convergent validity is supported
E.g., Survey items measuring Participation:
Voting
Donating to money to candidates
Signing petitions
Writing letters to the editor
Civic group memberships
Volunteer activities
Powerpoint Templates

C. Discriminant (Divergent) Validity

Measuring a concept to discriminate that

concept from other closely related concepts
E.g., Measuring Maternalism and Paternalism
as distinct concepts

Powerpoint Templates

Dimensions of Validity for

Research Design
Internal
Validity of research design
Validity of sampling, measurement,
procedures

External
Given the research design, how valid are
Inferences made from the conclusions
Implications for real world
Powerpoint Templates

Internal and External Validity in

Experimental Design
Internal validity:

Did the experimental treatment make a

difference?
Or is there an internal design flaw that invalidates the results?

External validity:

Are the results generalizable?

Generalizable to:
What populations?
What situations?

Without internal validity, there is no external validity

Powerpoint Templates

RELIABILITY & VALIDITY

OF
INSTRUMENT QUALITATIVE
INSTRUMENT

Powerpoint Templates

Core Qualitative Methods

Semi- or Un-structured, Open-Ended:
In-depth Interviews

(in the field, face-to-face)

Participant Observation
Archival Research

(field/site visits)
(document review and analysis)

Powerpoint Templates

Guiding Principles
Qualitative research designs consider ways to foster:
Reflexivity (an ongoing process of reflecting on the researchers subjective

experience, ways to broaden and enhance this source of knowing, & examining how it
informs research)

Iteration (a spiraling process: sequential and repetitive steps in examining

preliminary findings for the purposes of guiding additional data collection and analysis)

Intersubjectivity (a process of reaching a shared/ objective agreement about how

to assign meaning to a social experience - with insiders and outsiders)

Powerpoint Templates

The Iterative Process of

Qualitative Research: A Model

Analysis
Data Collection

Powerpoint Templates
Reflection

Qualitative Research Techniques

Instrumentation:
Key Informants (question development and piloting of instrument)
Unstructured to Semi-structured
Probing
Data Processing and Analytic Tools

Powerpoint Templates

Qualitative Research Techniques/

Considerations
Sampling
Single v. Multiple Cases (not an individual)
Expert and Key Informants (identification and recruitment of sample)
Roles of the Researcher (identification and recruitment of sample)

Powerpoint Templates

Qualitative Research Techniques

Data Collection
Participants as Data Collectors
Field Notes (personal reflections, observations, emerging concepts/theories)
Debriefing (a participant, a participating researcher, a non-participating
researcher)

Powerpoint Templates

Qualitative Research Techniques

Analysis
Key Informant Feedback
Codebooks (specifies definitions and relationships of concepts and terms)
Memos (emerging patterns, concepts; documentation of analytic pathways)
Case Analysis Meeting (a meeting of a research team for the purposes of reflecting on analytic
process, tools, and findings)
Matrices or Diagrams (to identify and examine time sequencing, the structure of relationships,
conditions of cross case events)

Powerpoint Templates

Qualitative Research Techniques

Analysis
Key Informant Feedback
Codebooks (specifies definitions and relationships of concepts and terms)
Memos (emerging patterns, concepts; documentation of analytic pathways)
Case Analysis Meeting (a meeting of a research team for the purposes of reflecting on analytic
process, tools, and findings)
Matrices or Diagrams (to identify and examine time sequencing, the structure of relationships,
conditions of cross case events)

Powerpoint Templates

Three Qualitative Methods:

Ethnography
Focus: study human behaviour in the
cultural context in which it is embedded
Ethnography is the work of describing a
culture the way of life of a cultural group
Associated with Cultural Anthropology

Example: David Counts and Dorothy Ayer

Counts: An Ethnography of RVing
Seniors
Powerpoint Templates

Three Qualitative Methods:

Grounded Theory
Focus: develop a theory to explain underlying
social processes of a cultural group
Useful in areas where little is known or when a
new perspective is needed
Used for exploratory, descriptive studies
Because the theory emerges from the data, it is
said to be grounded in the data
Foundation in Symbolic Interactionism
Example: Kerry Daly and the social
construction of fatherhood
Powerpoint Templates

Three Qualitative Methods:

Phenomenology
Focus: reveal the meaning of the lived experience from
the perspective of participants
Describe the essences of lived experience
Essences: elements related to the true meaning of
something that gives common understanding to the
phenomenon under study
Conveyed with descriptive language
Drawn from Philosophy; used across disciplines
Example: J.E. Solchany: A phenomenological study of
womens preadoptive experiences
Powerpoint Templates

Data Collection Methods

in Qualitative Studies
Three data collection strategies
introduced:
1. Participant observation
2. In-depth interviews
3. Focus group interviews

Qualitative researchers may combine

more than one method
Powerpoint Templates

Data Collection Methods:

1. Participant Observation
Intensive, usually long term, examination of a
social group, an organization, etc.
Researcher becomes a participant in the
lives of group members
Observes their behaviour and learns meaning
systems (which are tied to language)

Most closely associated with Ethnography, as

developed in Classical Anthropology
Now done in a variety of disciplines
Powerpoint Templates

1. Participant Observation (contd)

Many classic participant observations studies
E.g., Asylums (Erving Goffman), Tallys
Corner (Elliot Liebow), Street Corner
Society (William F. Whyte), etc.
Today most ethnographers take an overt role
I.e., their identity as a researcher is known
to the people being studied
Covert participation (i.e., identity concealed
from participants) is fraught with ethical issues
e.g., Humphreys Tearoom Trade
Powerpoint Templates

Steps Involved in
Participant Observation Research
A. Gaining entry into the group
B. Developing and maintaining rapport
C. Developing a method for taking field
notes
D. Integrating data collection and data
analysis

Powerpoint Templates

Steps in Participant Observation:

A. Gaining Entry into the Group
Take into consideration the type of group
formal organizations require formal entry;
involves letter writing, permission requests,
etc.
Informal groups different strategy needed
Access may be gained through a gatekeeper
(an individual with special status)
Want to involve key informants (those who are
most knowledgeable about the group)
Powerpoint Templates

Steps in Participant Observation:

B. Developing/Maintaining Rapport
Researcher must work hard to develop
and maintain good relationships in the
field
E.g., be sure not to become associated
with one faction in a group or organization

Researcher could be blamed for

problems that arise in the setting

Powerpoint Templates

Steps in Participant Observation:

C. Strategies for Taking Field Notes
Field notes integral to participant observation
Include descriptions and interpretations of individuals,
interactions, and events
Distinguish descriptions from interpretations
Record time and location of observations, as well as
key information (weather, events happening and their
significance)
Keep theoretical memos which are the tentative
interpretations emerging and being assessed through
further data collection

Powerpoint Templates

Field Notes (contd)

May not be possible or advisable to take
notes while in the field
Important that they be done as soon after
field observation as possible

Note-taking is time-consuming because

it is integral to guiding the data
collection and continuing the analysis
E.g., field notes for When Prophecy Failed
were well over 1,000 typed pages
Powerpoint Templates

Steps in Participant Observation:

D. Integrating Data Collection and

Analysis
Organizing field notes
into different types of files

facilitates data analysis

Master field file complete journal of field notes;
number pages and include entry dates
Background, history file subfile organizing
background material
Key character files subfiles on key players in
the group or organization
Analytic files subfiles for different types of
observations or relationships
Powerpoint Templates

Data Collection Methods:

2. In-depth Interviews
Some studies cannot employ the participant
observation method
E.g., Desrochess study of bank robbers
In-depth interviews allow participants to describe
their experiences and the meaning of events
taking place in their lives
Verbatim quotes capture the language and
meaning expressed by participants
Interviews are flexible and allow for probing
Interview method is quite diverse, adaptive
Powerpoint Templates

2. In-depth Interviews (contd)

Spradley three key elements for the interview

method to be successful
1. Explicit purpose researcher and informant are
aware that the discussion has a purpose
2. Ethnographic explanations researcher tries out
explanations on the participants to see if they
make sense
Encourage the informants to use colloquial
language, and teach the researcher its
meaning
Powerpoint Templates

2. In-depth Interviews (contd)

3.

Ethnographic questions include:

i. Descriptive questions ask participants to
describe their experiences (e.g., their ideas,
circumstances, viewpoints, dilemmas, etc)
ii. Structural questions ask participants how they
organize their world (e.g., activities)
iii. Contrast questions ask participants what is
meant by specific terminology
Prus & Grills stress the value of the interview method
in a multi-method approach
Powerpoint Templates

Data Collection Methods:

3. Focus Group Interviews
Interview format, but in a group setting
6-12 participants with common experience
Dates back to the 1940s used to assess
effectiveness of morale-boosting radio shows
1970s onward used by market researchers
1980s onward used by academics
Transcript of discussion is the data
Plus accompanying notes
Use content analysis or grounded theory
approach to analyze the data
Powerpoint Templates

3. Focus Group Interviews (contd)

Strengths:
Open-ended question
Spontaneously deal with issues as they arise
Cost-effective method of collecting data
Less time-consuming
Weaknesses:
One or two participants may dominate
Not done in a natural setting, so little observation to
help understand the experience of the participants

Powerpoint Templates

Implications for Handling

Threats to Validity and
Reliability
In quantitative research, threats to
validity are addressed by prior
design features (such as
randomization and controls)

Powerpoint Templates

Implications for Handling

Threats to Validity and
In qualitative research, such prior elimination of
Reliability
threats to validity is less possible because:
qualitative research is more inductive, and
it focuses primarily on understanding particulars rather
than generalizing to universals.

Qualitative researchers view threats as an

opportunity for learning
- e.g. researcher effects and bias are part of the story that is
told; they are not controlled for

Powerpoint Templates

Validity
In general, validity concerns the degree
to which an account is accurate or
truthful
In qualitative research, validity
concerns the degree to which a finding
is judged to have been interpreted in a
correct way

Powerpoint Templates

Assessing the Validity of

Qualitative Research
Can another research read your field (and other types
of) notes (i.e., the explication of your logic) and come
to the same understandings of a given phenomenon?
Concern about validity (as well as reliability) is the
primary reason thick description is an essential
component of the qualitative research enterprise
Resources:
Handout: Different Types of Notes
Example: ACY Site Visit Toolkit

Powerpoint Templates

Major Types of Validity in

Qualitative Research
Descriptive Validity
Interpretive Validity
Theoretical Validity
External Validity (i.e.,
generalizability)
Powerpoint Templates

Descriptive Validity
Concerned with the factual accuracy of an
account (that is, making sure one is not making
up or distorting the things one hears and sees)
All subsequent types of validity are dependent
on the existence of this fundamental aspect of
validity

Powerpoint Templates

Descriptive Validity
Behavior must be attended to, and
with some exactness, because it is
through the flow of behavior or,
more precisely, social action that
cultural forms find articulation.
Geertz 1973:17

Powerpoint Templates

Interpretive Validity
Interpretive accounts are grounded in
the language of the people studied
and rely, as much as possible, on
their own words and concepts
At issue, then, is the accuracy of the
concepts as applied to the
perspective of the individuals
included in the account

Powerpoint Templates

Interpretive Validity: Design

Consideration
While the relevant consensus about the terms
used in description rests in the research
community, the relevant consensus for the
terms used in interpretation rests, to a
substantial extent, in the community studied
An important design element, for increasing
interpretive validity, therefore, is to employee,
at some level/to some degree, a participatory
research approach (e.g., through member
checks, peer to peer research model, etc.)
Powerpoint Templates

Theoretical Validity
Theoretical understanding goes beyond
concrete description and interpretation; its
value is derived based on its ability to explain
succinctly the most amount of data
A theory articulates/formulates a model of
relationships as they are postulated to exist
between salient variables or concepts
Theoretical validity is thus concerned, not only
with the validity of the concepts, but also their
postulated relationships to one another, and
thus its goodness of fit as an explanation

Powerpoint Templates

Major Threats to Validity

Type I error: believing a principle
to be true when it is not (i.e.,
mistakenly rejecting the null
hypothesis)
Type II error: rejecting a principle
when, in fact, it is true
Type III error: asking the wrong
question
Powerpoint Templates

Triangulation: An Important
Theoretical Validity Check

Case example: Parable of the blind

men and the elephant

Powerpoint Templates

Triangulation: An Important
Theoretical Validity Check
The most fertile search for validity comes from a
combined series of different measures, each with
its own idiosyncratic weaknesses, each pointed to a
single hypothesis. When a hypothesis can survive
the confrontation of a series of complementary
methods of testing, it contains a degree of validity
unattainable by one tested within the more
constricted framework of a single method.

Powerpoint Templates

Webb et al. 1966:174

External Validity in Qualitative

Research
There is broad agreement that generalizability
(in the sense of producing laws that apply
universally) is not a useful standard or goal for
qualitative research
This is not to say, however, that studies
conducted to examine a particular
phenomenon in a unique setting cannot
contribute to the development of a body of
knowledge accumulating about that particular
phenomenon of interest
Consensus appears to be emerging that for
qualitative researchers generalizability is best
thought of as a matter of the fit between the
Powerpoint
situation studied
and Templates
others to which one
might be interested in applying the concepts

Enhancing External
Validity
Thick descriptions are crucial.
Such descriptions of both the site in which the
studies are conducted and of the site to which
one wishes to generalize (or apply ones
findings) are critical in allowing one to search
for the similarities and differences between the
situations.
Analysis of these similarities and differences
makes it possible to make a reasoned
judgment about the extent to which we can
use the findings from one study as a working
hypothesis about what might occur in another
situation.
Powerpoint Templates

Multi-site Studies: Another Way

to Enhance Generalizability
A finding emerging repeatedly in the study of
numerous sites would appear to be more likely
to be a good working hypothesis about some
as yet unstudied site than a finding emerging
from just one or two sites.
A finding emerging from the study of several
very heterogeneous sites would be more
robust and, thus, more likely to be useful in
understanding various other sites than one
emerging from the study of several very
similar sites.
Heterogeneity may be obtained by creating a
sampling frame
that maximizes
the variation
Powerpoint
Templates
inherent in the sample, specifically in terms of

Reliability
Reliability concerns the ability of
different researchers to make the
same observations of a given
phenomenon if and when the
observation is conducted using the
same method(s) and procedure(s)

Powerpoint Templates

Enhancing the Reliability of

Qualitative Research
Researchers can enhance the reliability of
their qualitative research by:
Standardizing data collection techniques and
protocols
Again, documenting, documenting, documenting
(e.g., time day and place observations made)
Inter-rater reliability (a consideration during the
analysis phase of the research process)
Powerpoint Templates