[Week 3]Small,Mario_How-Many-Cases-Do-I-Need
[Week 3]Small,Mario_How-Many-Cases-Do-I-Need
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms
Sage Publications, Ltd. is collaborating with JSTOR to digitize, preserve and extend access to
Ethnography
Etlino Igraphy
Copyright © The Author(s), 2009. Reprints and permissions: http://www.sagepub.co.uk/journalsPermissions.nav
http://eth.sagepub.com Vol 10(1): 5-38[DOI: 10.1177/1466138108099586]
In the South Seas there is a Cargo Cult of people. During the war they
airplanes land with lots of good materials, and they want the same thi
happen now. So they've arranged to make things like runways, to put f
along the sides of the runways, to make a wooden hut for a man to sit
[the controller] - and they wait for the airplanes to land. They're doing
thing right. The form is perfect. It looks exactly the way it looked be
But it doesn't work. No airplanes land. So I call these things Cargo
Science, because they follow all the apparent precepts and forms of
tific investigation, but they're missing something essential, because the
don't land. (Feynman, 1999: 208-9)
Feynman's lecture was devoted largely to practices such as ESP, but the h
about social science were difficult to miss. (In fact, he criticized a ps
ogist's advice to his student not to replicate the studies of others.) P
scientists were expert imitators but terrible practitioners, adopting the f
of science but nothing else. In this respect, his analogy might have
stronger had he noted that some New Guinean cargo cultists had fash
their own airplanes out of logs, sticks, and leaves, remarkably accu
replicas that, lacking engines and a foundation in aerodynamics, wo
never fly (Harris, 1974; Worsley, 1968).
While Feynman probably underestimated the successes of social sc
his observations are worth noting by at least one major segment of co
porary ethnographers, for whom the temptations of imitation have n
been stronger. The problem of imitation is not new to social scientists, w
from the start have argued heatedly (and repeatedly) over the mer
emulating the natural sciences in pursuit of social scientific me
(Dilthey, 1988; Lieberson and Lynn, 2002; Saiedi, 1993). But toda
important subset of ethnographic researchers - and of qualita
researchers more generally - faces its own version of that dilemma: whet
to emulate basic principles in quantitative social sciences in establish
standards of evidence for qualitative work. Some background is neces
We may set aside for the moment whether any of these critiques is valid in
order to highlight the larger predicament: ethnographers in these fields can
expect to be assessed on their methods by quantitative researchers.
Second, ethnographic studies in urban poverty, social inequality, and
immigration operate in an environment at once thirsty for in-depth case
studies that describe conditions in poor, minority, immigrant, or otherwise
non-mainstream groups, neighborhoods and communities, and either skep
tical or uncertain about the relationship between these 'small-«' studies and
the larger population of groups, neighborhoods, and communities that the
case studies are expected to represent (Lieberson, 1991). Consider a
contrast. When Geertz (1973) wrote on the cockfights in a small Balinese
village, many expected his theoretical model (of how games can embody
societal power relations) to be applicable to other sites, but few expected
the empirical findings to be so applicable - that is, for cockfights to look
similar or to follow the same rules in other villages throughout or outside
of Indonesia. The latter would be wholly beside the point.4 But ethnogra
phers in the aforementioned fields work in a professional, intellectual, and
policy environment that demands empirical findings applicable to other
cases. When ethnographers today describe conditions in, for example, one
poor black neighborhood in St Louis, MO, many of their readers in the
urban poverty literature expect to be learning about the conditions in poor
black neighborhoods in general - in Boston, Los Angeles, New York, and
perhaps even London and Rio de Janeiro - not merely that neighborhood
in St Louis. In fact, ethnographies of single neighborhoods are often cited
to argue that conditions in 'the ghetto', 'the underclass', 'the immigrant
enclave', or other similar categories exhibit some set of empirical charac
teristics, such as desolation, graffiti, street disorder, gang presence, street
entrepreneurship, etc. (on the perils of this practice, see Small, 2007, forth
coming b). Similar expectations surround the interpretation of in-depth
large and complex problem that I cannot hope to cover entirely in one
article. Instead, I will examine one issue, that of 'generalizability', one of
the most central concerns that quantitative researchers raise about ethno
graphic research in the aforementioned fields. To make the problem clear,
I will discuss two common scenarios, one from in-depth interview work
and one from participant observation, under which ethnographers
concerned about science attempt to make their work 'generalizable'. After
critiquing these forms of thinking about generalizability, I discuss two major
alternatives.6
people (for large populations, the exact size does not matter very much
In this case, she would need 385 respondents.10 If Jane narrowed her focus,
and only wanted to be confident about the 1000 working-class blacks
one city, she would need 278.
In fact, nothing Jane could do with her three dozèn people would ens
that they matched, with high certainty, the characteristics of the U
working-class black population. Some interview-based researchers prefer
ignore these two problems and refer to studies such as Jane's as 'repre
tative' because they were based on a 'random' sample. In doing so, they
simply adopting words without adopting their meaning, referring
wooden tubes as airplanes. If Jane sought a representative study, she should
have conducted a survey (and assumed all of its limitations, such as hav
to construct a formal questionnaire and reducing its length to maintain hig
response rates).
For example, we could first select our community very carefully in order to
make sure that it is especially representative of the rest of the country ...
We might ask a few residents or look at newspaper reports to see whether
it was an average community or whether some nonsystematic factor had
caused the observation to be atypical ... This would be the most difficult
part of the case-study estimator, and we would need to be very careful that
bias does not creep in. Once we are reasonably confident that
mized, we could focus on increasing efficiency. To do this, w
many weeks in the community conducting numerous sepa
(King et al., 1994: 67-8)"
One could question, again, whether Bill's strategy is not superior to any
alternative. An alternative is to search for unique cases (Small, 2004; van
Velsen, 1978 [1967]). Suppose that Bill had chosen a neighborhood with a
40 percent poverty rate but little garbage or graffiti and a unique architec
tural design due to the influence of a mayor interested in promoting
architecture in the city. It is tempting to assume that inferences based on
Alternatives
Jane and Bill's designs, and related approaches to render field studies more
generalizable, simply fail in their objectives. I see only three solutions to
this predicament: 1) to ignore the problem; 2) to qualify the work by
explaining that it is provisional or 'hypothesis generating'; and 3) to
conceive and design the work from a different perspective and language of
inquiry.
While ignoring the problem may seem to be inappropriate, it remains
a suitable option under some circumstances, given the different goals that
underlie different theoretical perspectives. As in the case of Geertz's (1973)
cockfight study, the theoretical perspective informing a case study may
render irrelevant whether the study provides empirical information on
conditions in other cases. For example, ethnomethodologists aim to study
practical life as it is experienced, and they assume that social reality is
both inter-subjective and radically situational, such that the meaning of
actions and events can only be explained within the particular set of
circumstances in which they occurred (Garfinkel, 1967). This assumption
(that meanings are indexed to particular situations) implies that whether
other sites - for example, other neighborhoods - would exhibit the same
empirical characteristics is not merely beside the point, but it would
erroneously presume that the analyst can simply extract the characteristics
of a neighborhood from the context in which actors experienced them
and expect them to have the same consequences elsewhere. Similarly,
narrative theories fundamentally assume that identity derives from
personal experiences and the narratives actors tell themselves about them.
And while conditions shared with others, such as the experience of race
Manchester revisited
Conclusions
For ethnographers in many fields of study, the issues discussed in this articl
do not arise. These ethnographers have no intellectual engagement wit
quantitative researchers, no expectation that the latter will ever be reviewer
no need to assess the work against a larger body of quantitative studie
asking similar or related questions. For many others, however, the issue
are critical. In urban sociology, the study of immigration, and soci
inequality - as well as, to some extent, the sociology of organizations an
of education - ethnographers must contend, explicitly or implicitly, wi
scholars trained in radically different traditions who claim expertise on the
same questions and may well assume a unity of method.
In this context, I have argued that no matter what they do within th
parameters of their original projects, Jane and Bill will never build airplanes
with the capacity to fly. The 'representative' single neighborhood does n
exist.28 Jane and Bill should not be building airplanes; they should
building, say, boats, vessels that are equally important for transportatio
and, during some circumstances, much more effective. And rather tha
build boats that try to fly, they should build boats that sail effectively. I ha
defended Mitchell's elaboration of the extended case method and propose
sequential interviewing - along with the (common) recommendations t
consider sampling for range, snowballing, and identifying unique cases -
one set of tools, by no means the only ones, to build better vessels. Gen
ally, the approaches call for logical rather than statistical inference, for case
rather than sample-based logic, for saturation rather than representation as
the stated aims of research. The approaches produce more logically sensib
hypotheses and more transparent types of empirical statements. Regardle
of the method, ethnographers facing today's cross-methods discourse an
critiques should pursue alternative epistemological assumptions bett
suited to their unique questions, rather than retreat toward models designed
for statistical descriptive research. After all, to push the metaphor to it
limits, even the best built airplane will never do well at sea, and I do no
see any pilots losing sleep over it.
Notes
Social Problems and City and Community, among the mainstream journals
that publish papers in these fields, have exhibited stronger inclinations to
publish ethnographic work.
In fact, the National Science Foundation recently felt compelled to organize
two conferences, with accompanying reports, aimed at discussing what
constitutes 'science' in the context of ethnographic or interview-based
qualitative work (see Lamont and White, forthcoming; Ragin et al., 2004).
The purpose was clear, to help qualitative researchers negotiate a pre
dominantly quantitative discourse:
The more standard practice to assess the work against other cases would
resemble Sallaz's (2008) recent study, which examines not cockfights but
casino card games in South Africa to assess the extent to which they also
embody social power relations and reflect larger structural transformations.
To be clear, these and other manuals certainly address questions of research
design, at length. However, they tend to say little explicitly about what
might be the most important questions for quantitative researchers. In fact,
the question of representativeness and statistical generalizability, probably
the most sensitive and ambiguous in the discourse between quantitative
and qualitative researchers, receives scant attention in the widely used
Hammersley and Atkinson (1995; see pages 42-5). It receives even less
coverage in Lofland and Lofland (1995).
Readers should note that they will not find in these pages an attack on
quantitative methods (which I often employ in my work). At issue is the
difference between an approach to science in which different questions are
asked within different epistemological frameworks and one in which all
social scientists attempt to answer all questions from one framework. It
should become obvious why this article defends epistemological pluralism.
I think Lamont's study of 160 upper-middle-class men in France and the
United States is a methodologically sophisticated interview-based study.
However, I do not believe, as others have commented, that it is sophisti
cated because 'she had a representative sample'. The study's response rate
was very low, between 42 percent and 58 percent, by liberal estimates
(Lamont, 1992). (As Lamont [1992: 285] writes, the figures 'do
potential respondents who did not provide the information n
determine whether they qualified or not' for the study. Thu
could overstate the response rate.) In addition, the samples ar
80 individuals in each country (40 in each site). The methodol
tication of the book comes from the sensitivity of the interv
Lamont's ability to interpret the meaning of respondents' stat
their cultural contexts; her use of a comparative model to
concepts; her judicious use of both semi-structured interviews,
findings to emerge inductively, and a structured survey, wh
comparative data across the cases; and her thoughtful selectio
sites (Lamont, 1992, Appendix II).
8 For a discussion of these issues from researchers aimed at
qualitative/quantitative divide, see King et al. (1994, pp. 63ff)
9 A more accurate statement of her question is this: Jane
know that, were she to sample the same population repeatedly
100 times the true number of working-class blacks w
immigration reform would fall within +/- five percentage po
estimated value.
10 The formula is n = (Z2 X p X (1 - p)) / C2, where Z is the Z value (1.96
for a 95% confidence level); C is the confidence interval, expressed as a
decimal (in this case, .05); and p is the percentage of people who are
expected to be, in this case, pro-reform. We assume .5, the most conserv
ative assumption. (If 51% are pro- and 49% are anti-reform, the room for
error is high, so a large sample is needed; if 90% were pro-reform one
could get by on a much smaller sample of 139.) There are dozens of sample
size calculators on the Internet, where one can manipulate the assumptions.
For example, [www.raosoft.com/samplesize.html].
11 In this passage, the authors were discussing much broader issues, so this
selection does not do justice to their book. The purpose here is not to
produce a full-fledged critique of the authors' book. Rather, it is to show
the pitfalls of this particular way of thinking about case study selection,
which the authors share with many others. One could argue that the
authors' flaw in this instance stems from their attempt to use a case study
to find an 'estimator', in the quantitative sense. From that perspective, we
may quote Campbell and Stanley, who said of 'the one-shot case study'
that 'such studies have such a total absence of control as to be of almost
no scientific value' (Campbell and Stanley, 1963: 6). I disagree strongly with
Campbell and Stanley's implied assessment of the purpose of a case study;
however, I agree that, if the purpose were to make a statistically general
izable statement of the effect of one variable on another, the authors would
be right, and their critique would apply to King et al.'s 'case study
estimator' (1994).
References
Evens, T.M. S. and D. Handelman (eds) (2006 [1983]) The Manchester School
New York: Berghahn Books.
Feynman, R.P. (1999) The Pleasure of Finding Things Out. Cambridge, MA
Perseus.
Fordham, S. and J. Ogbu (1986) 'Black Students' School Success: Coping with
the 'Burden of "Acting White"', Urban Review 18(3): 176-206.
Frankfort-Nachmias, C. and D. Nachmias (2000) Research Methods in the
Social Sciences, 6th edn. New York: Worth.
Garfinkel, H. (1967) Studies in Ethnomethodology. Cambridge: Polity Press.
Geertz, C. (1973) The Interpretation of Cultures. New York: Basic Books.
Glaser, B.G. and A.L. Strauss (1967) The Discovery of Grounded Theory.
Hawthorne, NY: Aldine de Gruyter.
Gluckman, M. (1961) 'Ethnographic Data in British Social Anthropology',
Sociological Review 9: 5-17.
Goering, J. and J.D. Feins (eds) (2003) Choosing a Better Life? Washington,
DC: Urban Institute Press.
Hammersley, M. and P. Atkinson (1995) Ethnography, 2nd edn. London:
Routledge.
Harris, M. (1974) Cows, Pigs, Wars, & Witches. New York: Random House.
Hermanowicz, J.C. (1998) 'The Presentation of Occupational Self in Science',
Qualitative Sociology 21(2): 129-48.
Katz, J. (1997) 'Ethnography's Warrants', Sociological Methods and Research
25(4): 391-423.
Katz, J. (2001a) 'From How to Why: Luminous Description and Causal
Inference in Ethnography, Part 1', Ethnography 2(4): 443-73.
Katz, J. (2001b) 'Analytic Induction', in N.J. Smelser and P.B. Baltes (eds) Inter
national Encyclopedia of the Social and Behavioral Sciences. London:
Elsevier.
Katz, J. (2002) 'From How to Why: Luminous Description and Causal
Inference in Ethnography, Part 2', Ethnography 3(1): 63-90.
King, G., R.O. Keohane and S. Verba (1994) Designing Social Inquiry.
Princeton, NJ: Princeton University Press.
Klinenberg, E. (2002) Heat Wave. Chicago, IL: University of Chicago Press.
Lamont, M. (1992) Money, Morals, & Manners. Chicago, IL: University of
Chicago Press.
Lamont, M. and P. White (eds) (forthcoming) Report from Workshop on Inter
disciplinary Standards for Systematic Qualitative Research. Washington,
DC: National Science Foundation.
Levitt, P. (2001) The Transnational Villagers. Berkeley: University of California
Press.
Lieberson, S. (1991) 'Small N's and Big Conclusions: An Examination of the
Reasoning in Comparative Studies Based on a Small Number of Cases',
Social Forces 70(2): 307-20.
O'Connor, A., C. Tilly and L. Bobo (eds) (2001) Urban Inequality. New
Russell Sage Foundation.
Pattillo, M. (1999) Black Picket Fences. Chicago, IL: University of C
Press.
Portes, A. and R. Rumbaut (2001) Legacies. Berkeley: University of Calif
Press.
Portes, A., L.E. Guarnizo and W.J. Haller (2002) 'Transnational Entrepreneurs:
An Alternative Form of Immigrant Economic Adaptation', American
Sociological Review 67(2): 278-98.
Ragin, C.C. and H.S. Becker (eds) (1992) What Is a Case? Cambridge:
Cambridge University Press.
Ragin, C.C., J. Nagel, and P. White (eds) (2004) Workshop on Scientific
Foundations of Qualitative Research. Washington, DC: National Science
Foundation.
Rumbaut, R. and A. Portes (2001) Ethnicities. Berkeley: University of
California Press.
Saiedi, N. (1993) The Birth of Social Theory. Landham, MD: University Press
of America.
Sallaz, J.J. (2008) 'Deep Plays: A Comparative Ethnography of Gambling
Contests in Two Post-Colonies', Ethnography 9(1): 5-33.
Schudson, M. (1992) Watergate in American Memory. New York: Basic
Books.
Shaw, C.R. (1930) The Jack-Roller. Chicago, IL: University of Chicago Press.
Shaw, C.R. (1931) The Natural Flistory of a Delinquent Career. Chicago, IL:
University of Chicago Press.
Shea, C. (2000) 'Don't Talk to the Humans', Lingua Franca 10(6): 26-
Small, M.L. (2004) Villa Victoria. Chicago, IL: University of Chicago P
Small, M.L. (2007) 'Is There Such a Thing as "The Ghetto"? The Per
Assuming that the South Side of Chicago Represents Poor Black Neigh
hoods', City 11(3): 413-21.
Small, M.L. (2008a) 'Lost in Translation: How Not to Make Qual
Research More Scientific', in M. Lamont and P. White (eds) Report
Workshop on Interdisciplinary Standards for Systematic Qualita
Research. Washington, DC: National Science Foundation.
Small, M.L. (2008b) 'Four Reasons to Abandon the Idea of "The Ghe
City and Community.
Small, M.L. (forthcoming) Unanticipated Gains. New York: Oxford Unive
Press.