Invited presentation, plenary session on Mixed Methodologies and the Politics of Evidence,

Third International Congress of Qualitative Inquiry, Urbana, IL, May 2007.

How Can “Mixed” Methods Constitute a Coherent Research Strategy?

Joseph A. Maxwell
George Mason University
[email protected]

In this presentation, I want to focus on what it means for a mixed

methods research strategy, or for any research strategy, to be coherent. I
will argue that coherence is not a matter of similarity (of methods, data,
paradigmatic assumptions, or anything else), but of the interaction and
complementarity of different methods and assumptions. I will then draw
some inferences from this position for how we conceptualize “evidence.” In
this view of coherence, I argue, there are no "rules" for what constitutes
evidence; evidence is anything that we draw from our data that contributes
to our understanding of the phenomena we study. However, I also argue
that this does not conflict with the rigorous assessment of such evidence and
of the conclusions we draw from it.

I want to begin with a definition of mixed methods. For most

researchers, “mixed methods” means combining quantitative and qualitative
methods, although the term could easily be extended to include the
combining of disparate qualitative methods—for example, Caroline Mears’s
award-winning dissertation (2005) on the experience of parents of students
at Columbine High School, which combined oral history methods, Eisner's
educational connoisseurship and educational criticism, and poetic
representation of her interview data. The term “mixed methods” is almost
always used to include what Tashakkori & Teddlie (1998, p. 16) called
“mixed model” research, in which not only quantitative & qualitative
methods, but also paradigms, research questions, and inference procedures,
are combined. The basic idea is that “mixed” methods combine what are
viewed as contradictory or dissimilar premises and methods.

The coherence and legitimacy of mixed methods research has been

problematic for many qualitative researchers. The main reason for this is the
assumption of paradigm incompatibility—that the paradigms of quantitative
and qualitative research (positivism/postpositivism and constructivism,
respectively) can’t be reconciled or integrated, and that attempts to combine
the two will create contradictions and incompatibilities in the research (e.g.,
Smith & Heshusius, 1986).

The main rejoinder to this view has been that paradigms are not
essential to the use of particular methods, and that methods can be
combined on the basis of their practical value without worrying about their
How can “mixed” methods constitute a coherent research strategy? 2

paradigmatic presuppositions (e.g., Patton, 1990). However, this view has

been criticized by other scholars, and a middle-ground view of this issue
seems to be dominant at present. This view holds that while paradigmatic
assumptions have important implications for the use of different methods,
they do not determine this use, nor do they preclude the joint use of
methods associated with different paradigms (e.g., Howe, 2003).

However, more recently, a second concern has taken center stage in

the qualitative community—that combining qualitative and quantitative
approaches, in the current climate of “science-based research,” will lead to
the subordination and misrepresentation of qualitative research, forcing it
into a positivistic framework. A recent paper asserted the likelihood of the
latter outcome by referring to mixed methods research as “positivism in
drag” (Giddings, 2006)

There are two points that I want to make about this argument. The
first is that different research methods are not typically tied to a single
paradigmatic position. This is particularly true of qualitative research, which
can be practiced from a variety of ontological, epistemological, and
axiological stances, including constructivism, realism, postmodernism, and
pragmatism, as well as more specific positions such as feminism and critical
theory.

The second point is that there are now a large number of examples of
mixed methods studies that combine qualitative and quantitative methods
and premises without subordinating one to the other, and provide valuable
insights into the phenomena they studied (e.g., Tolman and Szalacha,
2004). It seems clear that mixed methods research can be coherent without
imposing a single paradigmatic viewpoint.

However, neither of these arguments settles the issue of how mixed

method research achieves a coherent integration of methods and
assumptions that are connected to different paradigms. In the rest of this
paper, I will focus on the ways in which mixed methods studies do this,
rather than whether they can do so. To do this, I will try to unpack the term
“coherence,” and argue for a view of coherence that is at odds with some
widely-held assumptions about what “coherence” requires.

The term “mixed” in mixed methods is problematic in this respect,

because it has connotations that I think are misleading. To “mix” is normally
thought of as combining different things to make a single homogeneous
substance or mass, as in mixing cake batter. It can also have the meaning of
integrating disparate and distinct parts, but this is a less obvious sense, and
may still have the implication of creating a uniform product, as in “mixed
How can “mixed” methods constitute a coherent research strategy? 3

nuts.” The term “mixed methods” is by now too well established to change,
but I prefer to think of this strategy as one of integrating different research
approaches and methods, because it has less of a connotation of creating
homogeneity or similarity.

There is a widespread assumption in Western societies, in both

everyday thinking and social thought, that integration, coherence, and
solidarity are based on similarity—shared properties, characteristics, or
beliefs (Maxwell, 1996, 1999). One place where the assumption of creating
homogeneity or similarity has appeared in the mixed methods literature is in
the idea that integrating qualitative and quantitative data involves
transforming the data to make these more similar, and thus more
compatible (e.g., Sandelowski et al., 2006).

I argue, in contrast, that integration and coherence, while they can be

facilitated by similarity, are primarily the result of connection rather than
similarity, and that connection is not incompatible with, and may be
dependent on, difference. More broadly, coherence is not a matter of
similarity or uniformity of the components of a research design, but of their
connection and complementarity (Maxwell, 2003, 2005). Differences do not
need to be reduced, eliminated, or reconciled in order to create a coherent
design.

An example of this view of coherence is the structure of DNA. Once the

basic fact that DNA was composed of interlocking helical strands, each
having a sugar-phosphate backbone and containing four different bases
(adenine, guanine, cytosine, and thymine), the question became how the
strands were connected to one another. Essentially everyone investigating
the problem, including Pauling and Watson and Crick, initially assumed that
the connection must involve the pairing of similar bases (adenine with
adenine, etc.). Watson and Crick’s breakthrough discovery was realizing that
the links holding together the two chains were between dissimilar bases
(adenine with cytosine, and guanine with thymine). This structure was the
key to understanding how DNA is able to replicate, and thus how cells are
able to divide and reproduce, transmitting their genetic information to the
daughter cells (Watson, 1969).

The view of coherence that I am advocating here is best exemplified

by the type of mixed methods research that Caracelli and Greene (1997;
Greene & Caracelli, 2003) termed “dialectical.” Dialectical research involves
the juxtaposition of opposed or contradictory ideas; differences between
paradigms are viewed as valuable because they require the researcher to
engage with the tensions they create, rather than to ignore or attempt to
reconcile these. This perspective does not privilege any particular set of
How can “mixed” methods constitute a coherent research strategy? 4

assumptions or way of knowing, but sees different stances as providing

partial but valuable insights into the phenomena studied. It also argues for a
dialectical approach to reaching understanding of these phenomena. I would
argue that this approach is quite different from following a pre-specified
sequence or combination of methods and analyses, and has important
advantages for mixed methods research.

This understanding of coherence has been applied much more broadly

than just to research. For example, the philosopher Richard Bernstein
(1992), in an analysis of the work of Habermas and Derrida, tried to
demonstrate that, while these two thinkers cannot be reconciled, they do
supplement each other, and constitute "two intertwined strands" of the
modern/postmodern condition. Using Theodor Adorno's concepts of "force-
field” and “constellation," he argued that:

I do not think there is a theoretical position from which we can

reconcile their differences, their otherness to each other--nor do I
think we should smooth out their "aversions and attractions." The
nasty questions that they raise about each other's "project" need to be
relentlessly pursued. One of the primary lessons of
"modernity/postmodernity" is a radical skepticism about the possibility
of a reconciliation--an aufhebung, without gaps, fissures, and
ruptures. However, together, Habermas/Derrida provide us with a
force-field that constitutes the "dynamic, transmutational structure of
a complex phenomenon"—the phenomenon I have labeled
"modernity/postmodernity.” (1992, p. 225)

Jay, commenting on Adorno's work, defines "constellation" as

a juxtaposed rather than integrated cluster of changing elements that

resist reduction to a common denominator, essential core, or
generative first principle. (Jay, 1984, p. 15)

However, I think Jay’s denial that a constellation is “integrated” applies

a too-narrow meaning of this term. I see Adorno’s concept of “constellation”
as a postmodern form of coherence or integration that does not depend on
similarity or require reconciliation of difference, but is dependent on dialogue
and interaction among complementary differences. This view of coherence
requires that paradigmatic differences be integrated in a “constellation” that
provides useful insights and understandings, while preserving their
distinctive characteristics.

This sort of integration is at odds with the strict definitions of science

and scientific method that are now influencing educational and other social
science research policy. Such views promote a single paradigmatic
How can “mixed” methods constitute a coherent research strategy? 5

perspective on what research methods are appropriate for answering

different types of research questions, and are inimical to the kinds of
integration of methods that I am advocating. In the process, they
systematically dismiss or distort the value of qualitative research for
anything besides “exploratory” or descriptive research (Maxwell, 2004a, b).

Finally, I want to say something about coherence and evidence. The

dictionary definition of evidence is “facts or information indicating whether a
belief or proposition is true or valid.” I would expand this to include a range
of implications for our understanding of some phenomena, not only whether
a belief or proposition is true. Thus, “evidence” is any data (in a broad
sense) that has implications for your understanding or conclusions. Unlike in
a court of law, there is no “inadmissible evidence”, no general rule for
deciding what “counts” as evidence. What constitutes evidence in a
particular study depends on the specific goals and circumstances of that
study.

This point has been made repeatedly with respect to the concept of
validity. For example, Brinberg and McGrath argue that "Validity is not a
commodity that can be purchased with techniques. . . Rather, validity is like
integrity, character, and quality, to be assessed relative to purposes and
circumstances" (1985, p. 13). Similarly, a classic work on survey research,
after critiquing one study, asks "May one, therefore, conclude that the
Gluecks' measurements of these variables are invalid? In order to answer
this question, it is necessary to ask what the Gluecks wish to learn from
their data" (Hirschi and Selvin, 1967, p. 195).

My critique here is in part a critique of what I see as “scientism”: an

attempt to impose a narrowly logical definition of science and scientific
method (usually that promoted by positivist philosophers) and to create
rules determining what counts as scientific methods and scientific evidence,
rules that illegitimately exclude some methods, types of data, and
paradigmatic positions (Maxwell, 2008; cf. Cartwright, 1999; Haack, 2003).

My views have some similarity to what the philosopher Paul

Feyerabend, in his book Against Method, called an “anarchist theory of
knowledge.” Feyerabend argued that science does not proceed by any fixed,
universal set of rules, criteria, or methods; that any method could in
principle be valid in some circumstances; and that methodological rules only
exist situationally. My main disagreement with Feyerabend is that he focuses
on a critique of scientism, and provides little guidance in how we should
actually proceed in collecting, assessing, and using evidence in specific
situations. I believe (and I think many excellent mixed methods studies
exemplify this) that this is a matter of critically and insightfully connecting
How can “mixed” methods constitute a coherent research strategy? 6

your data to the conclusions and interpretations you draw, carefully

considering their implications for your understanding of what you are
studying, and systematically assessing possible alternative interpretations
and conclusions in light of these data.

I am not opposed to “evidence-based research” in the literal meaning

of the phrase; I am opposed to a priori judgments and rigid rules about what
constitutes valid evidence and what methods can provide this. To use the
classic qualitative mantra, “It depends.” As the Hirschi & Selvin and Brinberg
& McGrath quotes I used earlier argue, what counts as evidence depends on
the purposes for which it’s being used and the particular circumstances of
the research. Evidence, like meaning, is local; it is not an intrinsic property
of data, but of the relationship between data and the conclusions for which it
is seen as evidence.

Michael Scriven, in his article on causality in the Encyclopedia of

Evaluation (2004), argues that no method can provide certainty about causal
conclusions, but many methods can establish causal conclusions beyond a
reasonable doubt. An additional point that Scriven doesn’t make explicitly,
but that is implied by his argument, is that when particular methods can do
this depends on the specific circumstances of the research—what validity
threats are most relevant, and what would help us decide between
competing interpretations or conclusions.

I see the position that I present here as an application of some key

ideas of postmodernism to the issues of evidence and of mixed methods
research. It is an affirmative, rather than skeptical, postmodernism
(Rosenau, 1992) that accepts irreducible difference, but that also assumes
that it is possible to gain more valid and useful understandings of the world
we live in, rather than denying the possibility of such understandings and
embracing a radical relativism. However, my position applies postmodern
insights not just to our understandings, but to the methods we use to gain
these understandings, arguing that we need to jointly employ radically
dissimilar approaches without trying to assimilate one to the other or to
reconcile them in some more fundamental uniformity.

Lee Shulman, in his classic article on “Paradigms and programs,”

argued that

In asserting that no single research program can capture the full set of
educational events, I imply that the insufficiencies of particular
programs can be overcome through proper blending with the
insufficiencies of other programs. This image of a yoking of
inadequacies to produce a hybrid more vigorous than either of its
How can “mixed” methods constitute a coherent research strategy? 7

parents is certainly not alien to the practice of agriculture, but it has

not been widely touted in the social sciences. (Shulman, 1990, p. 10)

Shulman quoted Schwab (1978, p. 221):

Yet as long as resources for research are limited, there is an itch to

believe that one of the several strategies available to science is the
best one. . . . I have tried to show that this hope must betray us.
Consensus over a single pattern of choices will merely enable us to
overlook what we have not done in our inquiry. (Shulman, 1990, p.
81)

References:

Bernstein, Richard (1992). The new constellation: The ethical-political

horizons of modernity-postmodernity. Cambridge, MA: MIT Press.

Brinberg, D., & McGrath, J. E. (1985). Validity and the research process.
Thousand Oaks, CA: Sage Publications.

Caracelli, V. J., & Greene, J. C. (1997). Crafting mixed-method evaluation

designs. In Advances in Mixed-Method Evaluation: The Challenges and
Benefits of Integrating Diverse Paradigms, pp. 19-32. New Directions
for Evaluation, no. 74 (Summer 1997). San Francisco: Jossey-Bass.

Cartwright, N. (1999). The dappled world: A study of the boundaries of

science. Cambridge: Cambridge University Press.

Feyerabend, P. (1993). Against method, third edition. London: Verso.

Giddings, L. S. (2006). Mixed-methods research: Positivism dressed in drag?

Journal of Research in Nursing 11(3), pp. 195-203.

Greene, J. C., & Caracelli, V. J. (2003). Making paradigmatic sense of mixed

methods practice. In A. Tashakkori & C. Teddlie (Eds.), Handbook of
mixed methods in social and behavioral research, pp. 91-110.
Thousand Oaks, CA: Sage.

Haack, S. (2003). Defending science—within reason: Between scientism and

cynicism. Amherst, NY: Prometheus Books.

Hirschi, T., and Selvin, H. C. (1967). Principles of survey analysis. New

York: The Free Press.
How can “mixed” methods constitute a coherent research strategy? 8

Howe, K. R. (2003). Closing methodological divides: Toward democratic

educational research. Dordrecht, The Netherlands: Kluwer.

Jay, M. (1984). Adorno. Cambridge MA: Harvard University Press.

Maxwell, J. (1996). Diversity and methodology in a changing world.

Pedagogía 30, pp. 32-40 (1995-96).

Maxwell, J. (1999). A realist/postmodern concept of culture. In E. L.

Cerroni-Long (Ed.), Anthropological theory in North America, pp. 143-
173. Westport, CT: Bergin & Garvey.

Maxwell, J. (2004a). Causal explanation, qualitative research, and scientific

inquiry in education. Educational Researcher 33 (2), 3-11.

Maxwell, J. (2004b). Using qualitative methods for causal explanation. Field

Methods 16(3), 243-264 (August 2004).

Maxwell, J. A. (2005). Qualitative research design: An interactive approach,

second edition. Thousand Oaks, CA: Sage Publications.

Maxwell, J. A. (2008). Scientism. In William A. Darity, Jr. (Ed.), International

Encyclopedia of the Social Sciences, 2nd ed., Vol. 7, pp. 364-365.
Detroit: Macmillan Reference USA, 2008.

Maxwell, J. & Loomis, D. (2003). Mixed methods design: An alternative

approach. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed
methods in social and behavioral research, pp. 241-271. Thousand
Oaks, CA: Sage.

Mears, C. L. (2005). Experiences of Columbine parents: Finding a way to

tomorrow. Unpublished doctoral dissertation, University of Denver.

Patton, M.Q. (1990). Qualitative research and evaluation methods, second

edition. Thousand Oaks, CA: Sage Publications.

Rosenau, P. M. (1992). Post-modernism and the social sciences. Princeton:

Princeton University Press.

Sandelowski, M., Voils, C. I., & Barroso, J. (2006). Defining and designing
mixed research synthesis studies. Research in the Schools, 13(1), 29-
40.

Schwab, J. J. (1978). Science, Curriculum, and Liberal Education (selected

essays). Chicago: University of Chicago Press.
How can “mixed” methods constitute a coherent research strategy? 9

Scriven, M. (2004). Causation. In S. Mathison (Ed.), Encyclopedia of

evaluation (pp. 43-47). Thousand Oaks, CA: Sage

Shulman, L. S. (1990). Paradigms and programs. New York: Macmillan.

Originally published in the Handbook of research on teaching, ed. M.
Wittrock. New York: Macmillan, 1986.

Smith, J. K., & Heshusius, L. (1986). Closing down the conversation: The
end of the qualitative-quantitative debate among educational
researchers. Educational Researcher. 12(3), 6-13.

Tashakkori, A., and Teddlie, C. (1998). Mixed Methodology: Combining

Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage
Publications.

Tolman, D. L., & Szalacha, L. (2004). Dimensions of desire: Bridging

qualitative and quantitative methods in a study of female sexuality. In
S. Hesse-Biber & P. Leavy (Eds), Approaches to Qualitative Research,
pp. 97-129. Oxford: Oxford University Press.

Watson, J. D. (1969). The Double Helix: A Personal Account of the Discovery

of the Structure of DNA. Signet.