Quasi-Experimental Designs

Fearon J D 1991 Counterfactuals and hypothesis testing in behavioral sciences: how can researchers make a valid
political science. World Politics 43: 169–95 causal inference when units cannot be randomly
Goldberger A 1983 Abnormal selection bias. In: Karlin S, assigned to conditions? Like all experiments, quasi-
Amemyia T, Goodman L A (eds.) Studies in Econometrics, experiments deliberately manipulate presumed causes
Time Series, and Multi
ariate Statistics. Academic Press, New
York
to discover their effects. However, quasi-experiments
Goldstein H 1987 Multile
el Models in Educational and Social are also defined by what they do not do: the researcher
Research. Oxford University Press, New York does not assign units to conditions randomly. Instead,
Heckman J J 1988 The microeconomic evaluation of social quasi-experiments use a combination of design
programs and economic institutions. In: Chung-Hua Series of features, practical logic, and statistical analysis to
Lectures by In
ited Eminent Economists no. 14. The Institute show that the presumed cause is likely to be responsible
of Economics, Academia Sinica, Teipei, China for the observed effect, and other causes are not. The
Hsiao C 1986 Analysis of Panel Data. Cambridge University term nonrandomized experiment is synonymous with
Press, Cambridge, UK quasi-experiment; and the terms observational study
Isaac L W, Griffin L J 1989 A historicism in time-series analyses and nonexperimental design often include quasi-
of historical process. American Sociological Re
iew 54:
experiments as a subset. This article discusses the need
873–90
Jackman R 1986 The politics of economic growth in industrial for quasi-experimentation, describes the kinds of
democracies, 1974–1980: Leftist strength or North Sea oil? designs that fall into this class of methods, reviews the
Journal of Politics 49: 242–56 intellectual and practical history of these designs, and
Janoski T, Hicks A (eds.) 1994 The Comparati
e Political notes important current developments.
Economy of the Welfare State. Cambridge University Press,
Cambridge, UK
Lange P, Jarrett G 1985 The politics of growth. Journal of
Politics 47: 792–827
Maddala G S 1983 Limited-dependent and Qualitati
e Variables 1. The Need for Quasi-experimentation
in Econometrics. Cambridge University Press, Cambridge, Given the desirable properties of randomized experi-
UK
Nickell S 1997 Unemployment and labor market rigidities:
ments, one might question why quasi-experiments are
Europe versus North America. Journal of Economic Per- needed. When properly implemented, randomized
specti
es 11(3): 55–74 experiments yield unbiased estimates of treatment
Nieuwbeerta P, Ultee W 1999 Class voting in Western industrial- effects, accompanied by known probabilities of error
ized countries, 1945–1990: systematizing and testing explan- in identifying effect size. Quasi-experimental designs
ations. European Journal of Political Research 35: 123–60 do not have these properties. Yet quasi-experiments
Przeworski A, Alvarez M R, Cheibub J A, Limongi F in are necessary in the arsenal of science because it is not
pressDemocracy and De
elopment: Political Regimes and always possible to randomize. Ethical constraints may
Material Welfare in the World, 1950–1990. Cambridge Uni- preclude withholding treatment from needy people
versity Press, New York based on chance, those who administer treatment may
Ragin C 1987 The Comparati
e Method. University of California
Press, Berkeley, CA
refuse to honor randomization, or questions about
Ragin C 1994 A qualitative comparative analysis of pension program effects may arise after a treatment was
systems. In: Janoski T, Hicks A M (eds.) The Comparati
e already implemented so that randomization is im-
Political Economy of the Welfare State. Cambridge University possible. Consequently, the use of quasi-experimental
Press, Cambridge, UK, pp. 320–45 designs is frequent and inevitable in practice.
Stimson J A 1985 Regression in time and space: A statistical
essay. American Journal of Political Science 29: 914–47
Western B 1998a Causal heterogeneity in comparative research:
A Bayesian hierarchical modeling approach. American Journal
of Political Science 42: 1233–59 2. Kinds of Quasi-experimental Designs
Western B 1998b Between Class and Market: Postwar Unioniza-
The range of quasi-experimental designs is large,
tion in the Capitalist Democracies. Princeton University Press,
Princeton, NJ
including but not limited to: (a) Nonequivalent control
group designs in which the outcomes of two or more
G. Esping-Andersen and A. Przeworski treatment or comparison conditions are studied but
the experimenter does not control assignment to
conditions; (b) Interrupted time series designs in which
many consecutive observations over time (proto-
typically 100) are available on an outcome, and
treatment is introduced in the midst of those observa-
tions to demonstrate its impact on the outcome
Quasi-Experimental Designs through a discontinuity in the time series after treat-
ment; (c) Regression discontinuity designs in which
The theory of quasi-experimentation addresses one of the experimenter uses a cutoff score on a measured
the great unanswered questions in the social and variable to determine eligibility for treatment, and

12655
Quasi-Experimental Designs

an effect is observed if the regression line (of the why researchers may be wrong about the causal
assignment variable on outcome) for the treatment inferences they draw. Statistical conclusion validity
group is discontinuous from that of the comparison concerns inferences about whether and how much
group at the cutoff score; (d) Single-case designs in presumed cause and effect co-vary; examples of threats
which one participant is repeatedly observed over time to statistical conclusion validity include low statistical
(usually on fewer occasions than in time series) while power, violated assumptions of statistical tests, and
the scheduling and dose of treatment are manipulated inaccurate effect size estimates. Internal validity con-
to demonstrate that treatment controls outcome. cerns inferences that observed co-variation is due to
In the preceding designs, treatment is manipulated, the presumed treatment causing the presumed out-
and outcome is then observed. Two other classes of come; examples include history (extraneous events
designs are sometimes included as quasi-experiments, that could also cause the effect), maturation (natural
even though the presumed cause is not manipulated growth processes that could cause an observed
(and often not even manipulable) prior to observing change), and selection (differences between groups
the outcome. In (e) case–control designs, a group with before treatment that may cause differences after
an outcome of interest is compared to a group without treatment). Construct validity concerns inferences
that outcome to see if they differ retrospectively in about higher-order constructs that research operations
exposure to possible causes in the past; and in (f ) represent; threats include experimenter expectancy
correlational designs, observations on possible treat- effects whereby participants react to what they believe
ments and outcomes are observed simultaneously, the experimenter wants to observe rather than to the
often with a survey, to see if they are related. Because intended treatment, and mono-operation bias in which
these designs do not ensure that cause precedes effect, researchers use only one measure that reflects a
as it must logically do, they usually yield more construct imperfectly or incorrectly. External validity
equivocal causal inferences. concerns inferences about generalizing a causal
relationship over variations in units, treatments,
observations, settings, and times; threats include
interactions of the treatment with other features of
3. The History of Quasi-experimental Designs the design that produce unique effects that would not
otherwise be observed.
Quasi-experimental designs have an even longer his- Third, Campbell’s theory emphasized addressing
tory than randomized experiments. For example, threats to validity using design features—things that a
around 1,850 epidemiologists used case–control meth- researcher can manipulate to prevent a threat from
ods to identify contaminated water supplies as the occurring or to diagnose its presence and potential
cause of cholera in London (Schlesselman 1982), and impact on study results (see Table 1). For example,
in 1898, Triplett used a nonequivalent control group suppose maturation (normal development) is an an-
design to show that the presence of audience and ticipated threat to validity because it could cause a
competitors improved the performance of bicyclists. pretest–post-test change like that attributed to the
In fact, nearly all experiments conducted prior to treatment. The inclusion of several consecutive pre-
Fisher’s work were quasi-experiments. tests before treatment can indicate whether the rate of
However, it was not until 1963 that the term quasi- maturation before treatment is similar to the rate of
experiment was coined by Campbell and Stanley change from during and after treatment. If it is similar,
(1963) to describe this class of designs. Campbell and maturation is a threat. All quasi-experiments are
his colleagues (Cook and Campbell 1979, Shadish combinations of these design features, thoughtfully
et al. in press) extended the theory and practice of chosen to diagnose or rule out threats to validity in a
these designs in three ways. First, they described a particular context. Conversely, Campbell was skep-
large number of these designs, including variations of tical about the more difficult task of trying to adjust
the designs described above. For example, some quasi- threats statistically after they have already occurred.
experimental designs are inherently longitudinal (e.g., The reason is that statistical adjustments require
time series, single case designs), observing participants making assumptions, the validity of which are usually
over time, but other designs can be made longitudinal impossible to test, and some of which are dubious
by adding more observations before or after treatment. (e.g., that the selection model is known fully, or that
Similarly, more than one treatment or control group the functional form of errors is known).
can be used, and the designs can be combined, as when Other scholars during this time were also interested
adding a nonequivalent control group to a time series. in causal inferences in quasi-experiments, such as
Second, Campbell developed a method to evaluate Cochran (1965) in statistics, Heckman (1979) in
the quality of causal inferences resulting from quasi- economics, and Hill (1953) in epidemiology. However,
experimental designs—a validity typology that was Campbell’s work was unique for its extensive emphasis
elaborated in Cook and Campbell (1979). The ty- on design rather than statistical analysis, for its theory
pology includes four validity types and threats to of how to evaluate causal inferences, and for its
validity for each type. Threats are common reasons sustained development of quasi-experimental theory

12656
 Quasi-Experimental Designs

Table 1
Design elements used in constructing quasi-experiments
Assignment (Control of assignment strategies to increase group comparability)
Cutoff-based assignment. Controlled assignment to conditions based solely on one or more fully measured
covariates. This can yield an unbiased effect estimate
Other nonrandom assignment. Various forms of ‘haphazard’ assignment that sometimes approximate
randomization (e.g., alternating assignment in a two condition quasi-experiment whereby every other unit is
assigned to one condition, etc.)
Matching and stratifying. Efforts to create groups equivalent on observed covariates in ways that are stable, do
not lead to regression artifacts, and are correlated with the outcome. Preference is for pretreatment measures of
the outcome itself
Measurement (Use of measures to learn whether threats to causal inference actually operate)
Post-test observations
Nonequi
alent dependent
ariables. Measures that are not sensitive to the causal forces of the treatment, but are
sensitive to all or most of the confounding causal forces that might lead to false conclusions about treatment
effects (if such measures show no effect, but the outcome measures do show an effect, the causal inference is
bolstered because it is less likely due to the confounds)
Multiple substanti
e post-tests. Used to assess whether the treatment affects a complex pattern of theoretically
predicted outcomes
Pretest observations
Single pretest. A pretreatment measure on the outcome variable, useful to help diagnose selection bias
Retrospecti
e pretest. Reconstructed pretests when actual pretests are not feasible—by itself, a very weak design
feature, but sometimes better than nothing
Proxy pretest. When a true pretest is not feasible, a pretest on a variable correlated with the outcome—also
often weak by itself
Multiple pretest time points on the outcome. Helps reveal pretreatment trends or regression artifacts that might
complicate causal inference
Pretests on independent samples. When a pretest is not feasible on the treated sample, one is obtained from a
randomly equivalent sample
Complex predictions such as predicted interactions. Successfully predicted interactions lend support to causal
inference because alternative explanations become less plausible
Measurement of threats to internal
alidity. Help diagnose the presence of specific threats to the inference that A
caused B such as whether units actively sought out additional treatments outside the experiment
Comparison groups (Selecting comparisons that are ‘less nonequivalent’ or that bracket the treatment group at the
pretest(s))
Single nonequi
alent groups. Compared to studies without control groups, using a nonequivalent control group
helps identify many plausible threats to validity
Multiple nonequi
alent groups. Serve several functions. For instance, groups are selected that are as similar as
possible to the treated group but at least one outperforms it initially and at least one underperforms it, thus
bracketing the treated group
Cohorts. Comparison groups chosen from the same institution in a different cycle (e.g., sibling controls in families
or last year’s students in schools)
Internal (
s. external) controls. Plausibly chosen from within the same population (e.g., within the same school
rather than from a different school)
Treatment (Manipulations of the treatment to demonstrate that treatment variability affects outcome variability)
Remo
ed treatments. Showing an effect diminishes if treatment is removed
Repeated treatments. Reintroduces treatments after they have been removed from some group—common in
laboratory sciences or where treatments have short-term effects
Switching replications. Reverses treatment and control group roles so that one group is the control while the other
receives treatment, but the controls receive treatment later while the original treatment group receives no further
treatment or has treatment removed
Re
ersed treatments. Provides a conceptually similar treatment that reverses an effect—e.g., reducing access to a
computer for some students but increasing access for others
Dosage
ariation. Demonstrates that outcome responds systematically to different levels of treatment

and method over four decades. Both the theory and and his terms like internal and external validity became
the methods he outlined were widely adopted in so much a part of the scientific lexicon that today they
practice during the last half of the twentieth century, are often used without reference to Campbell.

12657
Quasi-Experimental Designs

4. Contemporary Research about capacity to generate unbiased effect estimates is

Quasi-experimental Design hamstrung by the same lack of knowledge of selection
that thwarts selection bias models.
4.1 Statistics and Quasi-experimental Design
Although work on the statistical analysis of quasi-
experimental designs deserves separate treatment, 4.2 The Empirical Program of Quasi-experimental
several contemporary developments deserve mention Design
here. One is the work of statisticians such as Paul Many features of quasi-experimentation pertain to
Holland, Paul Rosenbaum, and Donald Rubin on matters of empirical fact that cannot be resolved by
statistical models for quasi-experimental designs (e.g., statistical theory or logical analysis. For these features,
Rosenbaum 1995). They emphasize the need to a theory of quasi-experimental design benefits from
measure what would have happened to treatment empirical research about these facts. Shadish (2000)
participants without treatment (the counterfactual), has presented an extended discussion of what such an
and focus on statistics that can improve estimates of empirical program of quasi-experimentation might
the counterfactual without randomization. A central look like, including studies of questions like the
method uses propensity scores, a predicted probability following:
of group membership obtained from logistic regres- (a) Can quasi-experiments yield accurate effect
sion of actual group membership on predictors of estimates, and if so, under what conditions?
outcome or of how participants got into treatment. (b) Which threats to validity actually occur in
Matching, stratifying, or co-varying on the propensity practice (e.g., pretest sensitization, experimenter ex-
score can balance nonequivalent groups on those pectancy effects), and if so, under what conditions?
predictors, but they cannot balance groups for un- (c) Do the design features in Table 1 improve causal
observed variables, so hidden bias may remain. Hence inference when applied to quasi-experimental design,
these statisticians have developed sensitivity analyses and if so, under what conditions?
to measure how much hidden bias would be necessary Some of this empirical research has already been
to change an effect in important ways. Both propensity conducted (see Shadish 2000, for examples). The
scores and sensitivity analysis are promising develop- methodologies used to investigate such questions are
ments warranting wider exploration in quasi-experi- eclectic, including case studies, surveys, literature
mental designs. reviews (quantitative and qualitative), and experi-
A second statistical development has been pursued ments themselves. Until very recently, however, these
mostly by economists, especially James Heckman and studies have generally not been systematically used to
his colleagues, called selection bias modeling (Winship critique and improve quasi-experimental theory.
and Mare 1992). The aim is to remove hidden bias in
effect estimates from quasi-experiments by modeling
the selection process. In principle the statistical models
are exciting, but in practice they have been less 5. Conclusion
successful. A series of studies in the 1980s and 1990s
found that effect estimates from selection bias models Three important factors have converged at the end of
did not match results from randomized experiments. the twentieth century to create the conditions under
Economists responded with various adjustments to which the development of better quasi-experimen-
these models, and proposed tests for their appropriate tation may be possible. First, over 30 years of practical
application, but so far results remain discouraging. experience with quasi-experimental designs have pro-
Most recently, some economists have improved results vided a database from which we can conduct empirical
by combining selection bias models with propensity studies of the theory. Second, after decades of focus on
scores. Although selection bias models cannot yet be randomized designs, statisticians and economists have
recommended for widespread adoption, this topic turned their attention to improving quasi-experimen-
continues to develop rapidly and serious scholars must tal designs. Third, the computer revolution provided
attend to it. For example, other economists have both theorists and practitioners with increased ca-
developed useful econometrically based sensitivity pacity to invent and use more sophisticated and
analyses. Along with the incorporation of propensity computationally intense methods for improving quasi-
scores, this may promise a future convergence in experiments. Each in their own way, these three factors
statistical and econometric literatures. have taken us several steps closer to answering that
A third development is the use of structural equation great unanswered question with which this article
modeling (SEM) to study causal relationships in quasi- began.
experiments, but this effort has also been only partly
successful (Bollen 1989). The capacity of SEM to See also: Campbell, Donald Thomas (1916–96);
model latent variables can sometimes reduce problems Comparative Studies: Method and Design; Exper-
of bias caused by unreliability of measurement, but its imental Design: Overview; Experimental Design:

12658
 Queer Theory

Randomization and Social Experiments; Experi- explicable solely in terms of identity categories, such as
mentation in Psychology, History of; Panel Surveys: gender, race, class, or sexual orientation. It is radically
Uses and Applications anti-essentialist, in that it challenges a notion of
homosexuality as intrinsic, fixed, innate, and univer-
sally present across time and space.
Bibliography Queer theorists reject any mode of thought that
relies on a conception of identity as unified and self-
Bollen K A 1989 Structural Equations with Latent Variables. evident (e.g., I have sex with people of the opposite
Wiley, New York sex, therefore I must be heterosexual), and instead
Campbell D T, Stanley J C C 1963\1966 Experimental and
demonstrate that desires, sexual practices, and gen-
Quasi-experimental Designs for Research. Rand-McNally,
Chicago dered identities are performances and enactments,
Cochran W G 1965 The planning of observational studies in rather than expressions of ‘true’ subjectivity. Hetero-
human populations. Journal of the Royal Statistical Society, sexuality is therefore challenged by queer theory
Series A General 128: 234–66 not simply as a ‘hegemonic’ mode of identity, but as a
Cook T D, Campbell D T 1979 Quasi-experimentation: Design false claim to unity and coherence that is constantly
and Analysis Issues for Field Settings. Rand-McNally, Chicago undermined by the incoherencies of sex and gender,
Heckman J J 1979 Sample selection bias as a specification error. incoherencies that the queer analytic hopes to expose
Econometrica 47: 153–61 and celebrate.
Hill A B 1953 Observation and experiment. The New England
Journal of Medicine 248: 995–1001
Rosenbaum P R 1995 Obser
ational Studies. Springer-Verlag, 1. Intellectual Origins
New York
Schlesselman J J 1982 Case–Control Studies: Design, Conduct, In the broadest sense, queer theory emerged in what
Analysis. Oxford University Press, New York might be called the postmodern moment, when in-
Shadish W R 2000 The empirical program of quasi- tellectual unease with unitary and cohesive frame-
experimentation. In: Bickman L (ed.) Validity and Social works of knowing reached a fever pitch. While
Experimentation: Donald Campbell’s Legacy. Sage Publica- impossible to summarize here, queer theory’s alle-
tions, Thousand Oaks, CA giance to postmodern and\or poststructural modes of
Shadish W R, Cook T D, Campbell D T in press Experi-
thought can be traced in its challenge to the notion of
mental and Quasi-experimental Designs for Generalized Causal
Inference. Houghton-Mifflin, Boston unitary identity (as in ‘gay’ or ‘straight’), its refusal to
Triplett N 1898 The dynamogenic factors in pacemaking and understand sexuality through a singular and unified
competition. American Journal of Psychology 9: 507–33 lens (homosexual desire, feminist theory, gender), a
Winship C, Mare R D 1992 Models for sample selection bias. rejection of binary models (gay\straight, man\
Annual Re
iew of Sociology 18: 327–50 woman, biological\social, real\constructed), and a
more generic critique of identity-based theories and
W. R. Shadish politics that, according to poststructuralist accounts,
invariably reproduce the very conditions of repression
Copyright # 2001 Elsevier Science Ltd. they desire to challenge. For example, the term ‘gay’ or
All rights reserved. ‘homosexual’ might be critiqued as a (fictional) cat-
egory that shores up the binary opposition between
Queer Theory ‘gay’ and ‘straight’ that is itself part of the repressive
logic of identity. To claim ‘gayness’ is therefore not
‘Queer theory’ is a notoriously unstable phrase, and simply or solely an act of self-revelation but is also a
one much in contention. As a new theoretical move- way of corralling sexuality within the framework of a
ment with equally new political counterparts, it is category that only appears coherent but that, when
in constant flux and development, and is charac- opened up, reveals its instability (e.g., Am I still gay if
terized more by what it challenges and contests than I sleep with a person of the opposite sex? Or if I sleep
by what it offers in the shape of a unified social theory. with those of the same sex but only in certain
Drawing on the work of theorists such as Eve conditions and in certain ways? Or if my self-under-
Sedgwick and Judith Butler, queer theory ‘describes standing is of myself as ‘straight?’). Queer theory, in
those gestures or analytical models which dramatise that sense, has developed within and through the
incoherencies in the allegedly stable relations between deconstructive impulse of poststructuralism, chal-
chromosomal sex, gender and sexual desire (Jagose lenging assertions of unitary identity and necessary
1996). In this sense, queer theory is a challenge to the linkages (between, say, sexual desire and gender
‘obvious categories (man, woman, latina, jew, butch, orientation) and arguing instead for a more pro-
femme), oppositions (man vs. woman, heterosexual vs. visional, contingent, and fluid conception of the ‘queer’
homosexual), or equations (gender l sex) upon which in contemporary culture.
conventional notions of sexuality and identity rely’ While queer theory emerges as coterminous with
(Hennessy 1993). Queer theory argues instead that postmodern impulses, it also traces its intellectual
sexual desire and sexual practices are not reducible or origins in lesbian\gay studies and feminist theory even

12659

International Encyclopedia of the Social & Behavioral Sciences ISBN: 0-08-043076-7