Essential Research Methods for Social

Work
Fourth Edition, Allen Rubin. Earl Babbie

Chapter 12
Experiments and
Quasi-Experiments

© 2016 Cengage Learning

 Overview

• Criteria for Inferring Causality

• Internal Validity
• Pre-Experimental Pilot Studies
• Experimental Designs
• Quasi-Experimental Designs
• Additional Threats to the Validity of Experimental
Findings and Quasi-Experimental Findings
• External Validity
• Cross-Sectional Studies
• Case-Control Design

© 2016 Cengage Learning

 Criteria for Inferring Causality

1) Cause (independent variable) must precede the effect

(dependent variable) in time
2) The two variables are empirically correlated with one
another
3) The observed empirical correlation between the two
variables can not be due to the influence of a third
variable that causes the two under consideration

© 2016 Cengage Learning

 Internal Validity (1 of 2)

• Depends on the extent to which the 3 criteria for

causality are met
• Refers to whether or not causal inferences can be
drawn from the results
• Prominent threats to internal validity:
– History
– Maturation or the passage of time
– Testing
– Instrumentation changes
– Statistical regression

© 2016 Cengage Learning

 Internal Validity (2 of 2)
– Selection bias
– Ambiguity regarding the direction of causal inference

© 2016 Cengage Learning

 Pre-experimental Pilot Studies
• Very limited degree of internal validity
• Exploratory or descriptive purposes, such as:
– To generate tentative exploratory or descriptive
information regarding a new intervention about which little
is known
– To learn whether it is feasible to provide the new
intervention as intended
– To identify obstacles in carrying out methodological
aspects of a more internally valid design that is planned
for the future
– To see if the hypothesis for a more rigorous study
remains plausible based on the pilot study results

© 2016 Cengage Learning

Three common pre-experimental designs

• One-shot Case Study

• One-Group Pretest-Posttest Design
• Posttest-Only Design with Nonequivalent Groups

© 2016 Cengage Learning

One-Shot Case Study

© 2016 Cengage Learning

One Group Pretest-posttest Design

© 2016 Cengage Learning

Posttest Only Design with Nonequivalent
Groups

© 2016 Cengage Learning

 Introduction to Experimental Designs

• Best causal evidence comes from designs with strong

internal validity
• Experimental designs provide maximum control for
threats to internal validity

© 2016 Cengage Learning

Diagram of a Basic Experimental Design

© 2016 Cengage Learning

 Pretest-Posttest Control Group Design

R O1 X O2
R O1 O2

• Does not control for possible impact of testing and

retesting

© 2016 Cengage Learning

 Posttest-only Control Group Design

R X O
R O

• Avoids testing/retesting impact

• Pre-testing not possible in some experiments
• Assumes that the process of random assignment
removes any significant initial differences between
experimental and control groups

© 2016 Cengage Learning

 Solomon Four-Group Design

R O1 X O2
R O1 O2
R X O2
R O2

• Assesses amount of pretest-posttest change while

checking for testing effects

© 2016 Cengage Learning

 Alternative Treatment Design with Pretest
(1 of 2)
R O1 XA O2
R O1 XB O2
R O1 O2

• If the first two rows both show approximately the same

amounts of improvement, and both amounts are more
than in the third row, that would indicate that both
treatments are approximately equally effective. But if the
third row shows the same degree of improvement as in
the first two rows, then neither treatment would appear to
be effective. Instead, we would attribute the improvement
in all three rows to an alternative explanation such as
history or the passage of time.
© 2016 Cengage Learning
Alternative Treatment Design with Pretest
(2 of 2)
• Some experiments use the first two rows of this design
but not the third row. They compare the two treatments
to each other but not to a control group. They can have
conclusive, valid findings if one group improves
significantly more than the other. But suppose they
both have roughly the same amount of improvement.
The temptation would be to call them equally effective.
However, with no control group, we cannot rule out
threats to internal validity, such as history or the
passage of time, as alternative explanations of the
improvement in both groups

© 2016 Cengage Learning

 Dismantling Studies

R O1 XAB O2
R O1 XA O2
R XB O2
R O2

• Shows not only whether an intervention is effective, but

also which components of the intervention may or may
not be necessary to achieve its effects.

© 2016 Cengage Learning

 Randomization

• Controls for selection bias in experimental designs

• Participants are divided into groups using procedures
based on probability theory
• Improves the likelihood that the control group
represents what the experimental group would look like
had it not been exposed to the experimental stimulus

© 2016 Cengage Learning

 Providing Services to Control Groups

• Ethical considerations require that services be provided

to many control groups
• Often, control groups receive usual care in place of the
experimental intervention
– Compares experimental intervention to treatment as
usual
• Control group participants may also be given waitlist
priority

© 2016 Cengage Learning

 Quasi-experimental Designs (1 of 2)

• Designs that attempt to control for threats to internal

validity and thus permit causal inferences but are
distinguished from true experiments primarily by the
lack of random assignment of subjects
• Useful when it is not feasible to obtain a control group
• When conducted properly can obtain a reasonable
degree of internal validity

© 2016 Cengage Learning

 Quasi-experimental Designs

• Three common quasi-experimental designs will be

examined:
– Nonequivalent comparison group designs
– Simple time-series designs
– Multiple time-series designs

© 2016 Cengage Learning

 Nonequivalent Comparison Groups
Designs
• Two existing groups that appear to be similar are
identified or created
• The dependent variable is assessed before and after an
intervention is introduced to one of the groups
• Comparison group does not receive the intervention

O1 X O2
O1 O2

© 2016 Cengage Learning

 Time-Series Designs

• A simple interrupted time-series design attempts to

develop causal inferences based on a comparison of
trends over multiple measurements before and after an
intervention is introduced and requires no comparison
group

O1 O2 O3 O4 O5 X O6 O7 O8 O9 O10

• The more measurements, the stronger the design

© 2016 Cengage Learning

Additional Threats to the Validity of Experimental
& Quasi-experimental Findings
• Measurement Bias
• Research Reactivity
• Diffusion or Imitation of Treatments
• Compensatory Equalization, Compensatory Rivalry, or
Resentful Demoralization
• Attrition (or Experimental Mortality)

© 2016 Cengage Learning

 Measurement Bias

• Credibility of study conclusion can be compromised if

measures are biased
• Bias can be avoided if blind raters who are unaware of
the hypotheses and whether or not a participant has
received the experimental intervention

© 2016 Cengage Learning

 Research Reactivity

• Refers to changes in outcome data that are caused by

researchers or research procedures rather than the
independent variable
– Measurement Bias
– Experimental Demand Characteristics
– Experimenter Expectancies
– Obtrusive Observation
– Novelty and Disruption Effects
– Placebo Effect

© 2016 Cengage Learning

Placebo Control Group Design

R O1 X O2
R O1 O2
R O1 P O2

© 2016 Cengage Learning

 Diffusion and Imitation of Treatments

• A threat to the validity of an evaluation of an

intervention’s effectiveness that occurs when
practitioners who are supposed to provide routine
services to a comparison group implement aspects of
the experimental group’s intervention in ways that tend
to diminish the planned differences in the interventions
received by the groups being compared

© 2016 Cengage Learning

 Compensatory Equalization, Compensatory
Rivalry, or Resentful Demoralization (1 of 2)
• In Compensatory Equalization comparison condition
practitioners compensate for differences in treatment
groups by providing enhanced services that go beyond
the routine-treatment regimen for their clients
• In Compensatory Rivalry comparison condition
practitioners compete with therapists in experimental
group; extra efforts might improve the effectiveness of
routine-treatment

© 2016 Cengage Learning

 Compensatory Equalization, Compensatory
Rivalry, or Resentful Demoralization (2 of 2)
• In Resentful Demoralization practitioners or clients in
the comparison condition become resentful and
demoralized because they did not receive special
training or treatment; decline in confidence or
motivation may explain their inferior outcomes

© 2016 Cengage Learning

 Attrition

• A threat to the validity of an experiment that occurs

when participants drop out of an experiment before it is
completed
• Strategies to minimize attrition:
– Reimbursement
– Avoid intervention or research procedures that
disappoint participants
– Utilize tracking methods

© 2016 Cengage Learning

 Intent-To-Treat Analysis

• Data analysis technique used when attrition is high

• Assesses outcomes of all who originally enrolled in the
study, including dropouts

© 2016 Cengage Learning

 External Validity

• The extent to which we can generalize the findings of a

study to settings and populations beyond the study
conditions

© 2016 Cengage Learning

 Cross-sectional Studies

• Study based on observations that represents a single

point in time
• May have exploratory, explanatory or descriptive
purposes
• May use multivariate statistical procedures to rule out
the plausibility of alternative hypotheses

© 2016 Cengage Learning

 Case-Control Design

• Compares groups of cases that have had contrasting

outcomes
• Collects retrospective data about past differences that
might explain the difference in outcomes
• Possible limitations
– Representativeness of sample
– Faulty memories
– Ambiguity in the direction of causal influence
– Recall bias

© 2016 Cengage Learning