Qualitative Literacy: A Guide to Evaluating Ethnographic and Interview Research

Qualitative Literacy

PRAISE FOR QUALITATIVE LITERACY

"Instead of judging qualitative research by the standards of other methods, Mario Luis Small and Jessica McCrory Calarco consider the purpose and aspirations of in-depth interviewing and ethnography and then offer criteria with which to evaluate if a piece of research hits the mark (or not). At once practical and sophisticated, Qualitative Literacy reflects the wisdom of two of the most talented qualitative researchers in the field today. It is an invaluable resource for methods teachers, funders, policy makers, and students."

MARY PATTILLO, author of Black Picket Fences: Privilege and Peril among the Black Middle Class

This excellent, accessible book is written by authors with an impeccable reputation in the field. It brings focus to what we know and agree on as practitioners of qualitative research and how we should be thinking about the craft versus how it is typically taught.

D’LANE R. COMPTON, Full Professor of Sociology, University of New Orleans

This is a stellar book. The authors have crafted a clearly written manuscript that will be useful both for teaching and as a handbook for practitioners. I have not seen anything quite like it, and this book helps me think reflexively about my own work and that of my students.

TANYA GOLASH-BOZA, Professor of Sociology, University of California, Merced

The publisher and the University of California Press Foundation gratefully acknowledge the generous support of the Atkinson Family Foundation Imprint in Higher Education.

Qualitative Literacy

A GUIDE TO EVALUATING ETHNOGRAPHIC AND INTERVIEW RESEARCH

Mario Luis Small

Jessica McCrory Calarco

UNIVERSITY OF CALIFORNIA PRESS

University of California Press

Oakland, California

© 2022 by Mario Small and Jessica Calarco

Library of Congress Cataloging-in-Publication Data

Names: Small, Mario Luis, 1974– author. | Calarco, Jessica McCrory, 1983– author.

Title: Qualitative literacy : a guide to evaluating ethnographic and interview research / Mario Luis Small, Jessica McCrory Calarco.

Description: Oakland, California : University of California Press, 2022. | Includes bibliographical references and index.

Identifiers: LCCN 2022005233 (print) | LCCN 2022005234 (ebook) | ISBN 9780520390652 (cloth) | ISBN 9780520390669 (paperback) | ISBN 9780520390676 (epub)

Subjects: LCSH: Ethnology—Qualitative research—Methodology. | Interviewing—Methodology.

Classification: LCC GN346 .S58 2022 (print) | LCC GN346 (ebook) | DDC 305.80072/1—dc23/eng/20220304

LC record available at https://lccn.loc.gov/2022005233

LC ebook record available at https://lccn.loc.gov/2022005234

Manufactured in the United States of America

31  30  29  28  27  26  25  24  23  22

10  9  8  7  6  5  4  3  2  1

For ARABELLA, LAYLA, LEO, and ODIN

Contents

Preface

Introduction

1 Cognitive Empathy

2 Heterogeneity

3 Palpability

4 Follow-Up

5 Self-Awareness

Conclusion

Acknowledgments

Appendix: A Note on Proposals

Notes

References

Index

Preface

This book is animated by a simple question. Suppose you were given two books, each based entirely on one year of ethnographic observation, and were told that one of them is a sound piece of empirical social science and the other, though interesting and beautifully written, is empirically unsound. What criteria would you use to tell the difference? One can ask the same question of a different kind of qualitative research. Suppose the two books were instead based on in-depth interviews with the same set of respondents, and you were informed that one is empirically sound and the other is not. What criteria would you use?

This question should matter to readers of the many qualitative studies regularly published on socially important topics: inequality, education, poverty and wealth, immigration, the family, crime and punishment, management and formal organization, public health, neighborhoods, labor, discrimination, housing and homeownership, aging, and the relationship between environment and society. Over the past two decades, qualitative research on each of these topics has come to be consumed by a large and diverse readership, a group that includes not only ethnographers and interviewers but also quantitative sociologists, demographers, economists, psychologists, applied statisticians, and others who themselves work on the topics, as well as the policy makers and legislators, think tank authors and foundation officers, practitioners and activists, journalists and lay readers who may not conduct formal research but whose work or beliefs are nonetheless deeply affected by what the field researchers report. All of these readers have a stake, in varying degrees, in the empirical soundness of the studies.

We have posed our criteria question to many people who would especially want to have an answer: quantitative social scientists who review qualitative studies in their topic of expertise for panels or journals; ethnographers of all stripes who produce such work themselves; private and national foundations’ program officers who determine whether to fund such work; university deans, provosts, and presidents who evaluate such work for hiring and promotion; journalists who report on such work; and practitioners and policy makers who use such work in hopes of improving others’ lives. When asked, some of these people proposed that experience and intuition should enable them to distinguish the empirically weak from the strong book—while also often admitting that they could not quite articulate what they would look for. Others pointed to criteria common in quantitative social science, such as reliability or representativeness, without offering a clear idea of how these would be applied to single-case ethnographies. Indeed, just about everyone who answered our question expressed some uncertainty, and the single most common answer was some version of the phrase, I’m not sure.

That uncertainty partly reflects what one of us has called an absence of qualitative literacy, the ability to read, interpret, assess, and evaluate qualitative evidence competently.¹ For several decades now, scientists, educators, and others have recognized the importance of quantitative literacy across society and have pushed for its dissemination with at least some success. School districts and colleges have buffered instruction to improve numeracy. Graduate training programs across the social sciences have deemphasized high theory in favor of empirical training in quantitative methods. In fact, the public discourse on the important topics we listed above is far more quantitatively literate than it was twenty or thirty years ago; newspapers and magazines routinely produce more and more accurately reported quantitative data and at times even make available the full dataset informing a story.²

Nevertheless, there has been no parallel increase in the public discourse’s qualitative literacy. And this lapse is equally prominent in the social sciences themselves. For example, while most strong sociology PhD programs require training in quantitative research, the majority do not require courses in qualitative methods, even though their graduates produce many of the major ethnographies and interview studies on contemporary social problems.³ Furthermore, even surface-level exposure to any qualitative coursework remains all but nonexistent in many of the other social sciences, such as psychology, political science, and economics—which would seem like the natural state of affairs but for the fact that today many of their graduates will have to evaluate qualitative research, within their topics of expertise, for foundations, tenure committees, and other venues. An economist teaching in an education graduate program, having never studied the evaluation of qualitative research, can nonetheless expect over the course of their career to vote on whether a school ethnographer deserves tenure. Indeed, in spite of the welcome intellectual openness of today’s research on social problems, many quantitative researchers, and others in gatekeeping roles, are admittedly at a loss to explain how to evaluate qualitative empirical research, not on whether it is interesting or informative or well-written or a source of good ideas or of telling anecdotes, but on whether it is good social science.

That state of affairs represents a problem. The distinct kinds of knowledge produced by interview-based and participant observation research are indispensable to how much we know and how effectively we address inequality, housing, public health, discrimination, immigration, education, and the many other issues noted earlier. It is important to know whether a given set of qualitative findings is scientifically believable. And there is an enormous gap, we believe, between the knowledge needed to make that determination and the knowledge collectively possessed by those poised to assess, fund, support, report on, learn from, or make decisions on the basis of qualitative research. Our short book aims to fill that gap.

Introduction

In the not-too-distant past, qualitative and quantitative researchers in sociology and other disciplines were embroiled in a protracted conflict sometimes referred to as the paradigm wars.¹ Field-workers accused their presumed opponents of conducting positivistic, unreflective, or simple-minded research; quantitative analysts chided theirs for doing soft or unrigorous work, or of writing just-so stories with little scientific backing. To the extent it was a war, quantitative research clearly held the upper hand, as economics, demography, statistical analyses, and public opinion surveys deeply shaped national policy decisions.² But quantitative researchers who ignored fieldwork did so at their peril, as a vast body of ethnographic and interview-based research had documented and offered important insight into the experiences of many of the populations that those national discussions were concerned with, such as school students, the unemployed, married couples, low-income families, employers, and immigrants.³ In all, the wars were a highly counterproductive conflict that made evident just how young, in historical terms, most social sciences are.⁴

But much has changed. While quantitative research arguably remains dominant in social science debates on important social problems, over the past two decades qualitative scholarship has dramatically shaped how scientists, policy makers, and the public think about inequality, poverty, race and ethnicity, gender, education, health, organizations, immigration, neighborhoods, and families.⁵ It has helped us understand

why neighborhoods matter;⁶

how schools shape children’s opportunities, expectations, and understanding of themselves;⁷

why people risk life and limb to cross the southern U.S. border;⁸

how employers evaluate potential hires;⁹

how people make decisions about marriage, employment, and child-rearing;¹⁰

how people manage legal institutions;¹¹

how social and economic conditions affect people’s everyday lives; and much more.¹²

In contrast to the past, economists, demographers, political scientists, and quantitative sociologists of all stripes today often cite qualitative studies in their own work and use those studies to generate hypotheses, illustrate discoveries, or interpret findings.¹³ In turn, qualitative researchers have testified before Congress, helped governments set policy, advised local practitioners, contributed to the public discourse, and shaped how corporate and nonprofit boards invest and spend their resources.¹⁴ At this juncture, the importance of interview and ethnographic methods to social science, and to society, is not in question.

But in spite of this progress, social scientists have not come to agree on what constitutes good qualitative social science—in fact, they do not even agree on whether qualitative research should be thought of as scientific, as opposed to merely informative, work.¹⁵

How to Assess Quality

Consider the recent history. In 1994, Gary King, Robert Keohane, and Sydney Verba’s Designing Social Inquiry was supposed to lay to rest questions about the scientific foundations of qualitative studies by presenting clear guidelines for conducting and evaluating such work based on basic principles that guide quantitative research, such as reliability, unbiasedness, and efficiency.¹⁶ The book was unusually detailed, comprehensive, and full of examples, promising to unite qualitative and quantitative researchers under a common view of rigorous empirical science. Instead, it sparked even more controversy.¹⁷ To this day, the work strongly divides researchers, and qualitative researchers have repeatedly complained that its guidelines are inappropriate.¹⁸

In the early 2000s, the National Science Foundation (NSF), which funds social science scholarship, was facing an increasing number of [submissions for] qualitative research projects in sociology and recognized both that many reviewers did not know how to evaluate the work and that those who did seemed to disagree on the appropriate criteria.¹⁹ The NSF convened a team of sociologists and later another of social scientists across several disciplines to clarify the standards by which qualitative research should be deemed rigorous and the criteria against which reviewers should evaluate proposals.²⁰ Distinguished social scientists participated in both teams, and each working group produced a volume of papers with summary guidelines.

But the researchers had so little in common in approach and perspective that many of their summary recommendations merely restated basic principles one would expect any proposal would have, rather than offering many new criteria specific to qualitative work. For example, the set of standards released by the first team recommended that researchers should write clearly and engagingly for a broad audience, locate the research in the relevant literature, provide evidence of the project’s feasibility, and so forth.²¹ The second team had similar ideas, such as situate the research in appropriate literature, pay attention to alternative explanations, and specify the limitations of the research and anticipate potential reviewer objections.²² All of these recommendations have value, but they are suggestions that most experienced researchers in any discipline would already know to follow, rather than guidelines either specifically relevant or new to field-workers.

Not all of the NSF recommendations had this character; some were in fact distinctively appropriate to qualitative research. For example, the volumes recommended that researchers assess the possible impact of the researcher’s presence [and] biography; discuss the researcher’s cultural fluency, language skill, . . . knowledge of particular research context; and describe and explain case selection.²³ But the guidelines nonetheless sparked controversy: at least one team member, who is one of the most distinguished and influential ethnographers in American history, openly challenged the NSFs conceptions of good research and produced an extraordinary minority report.²⁴ In the end, the NSF efforts, important though they were, did not bring closure to the questions the experts had been tasked to address.

In fact, over the ensuing years this ongoing ambiguity has contributed to several high-profile disputes involving fieldwork. Famous ethnographers with widely discussed books have been accused of handling data poorly, of making implausible claims, and of rendering their findings unverifiable.²⁵ A recent book has interrogated ethnography, proposing that claims in several major ethnographic studies do not stand up to legal standards of verification.²⁶ Indeed, some of the most contentious reviews of qualitative research today are produced by other qualitative researchers. Among the long list of controversies are those over grounded theory, cowboy ethnography, replicability in recent ethnographies, exaggeration in anthropology, sampling in case studies, the value of interviews in studying behavior, and the ethics of masking identities in qualitative research.²⁷

These controversies have left budding field-workers uncertain about how to conduct their own work; reviewers unclear about what signs of quality to look for; and scholars, journalists, and other consumers unsure about how to judge the work that qualitative researchers are generating.

Improving Social Science

That general uncertainty exists in a societal context where social science as a whole is looking more closely at its methods. In recent years, quantitative social science has experienced a reckoning, as common research practices in psychology, economics, political science, and quantitative sociology have come under extensive scrutiny. Critics have pointed to flaws large and small. Some problems are serious but relatively benign, faults of omission rather than commission; for example, studies in many subfields have been shown to be all too dependent on WEIRD samples, from white, educated, industrialized, rich, democratic respondents, a dependency that calls into question the work’s empirical generalizability.²⁸ Other problems lie at the heart of scientific practice itself. Authors of high-profile quantitative studies have been found guilty of p-hacking (mishandling data to produce significant findings), HARKing (writing up hypotheses after the results are known), and even fabricating results.²⁹ In fact, the findings of numerous major experimental studies have been uncovered to be nonreplicable.³⁰ These practices undermine not only the quality of the science but also the public’s trust, a problem that, in an era in which some politicians have sought to question the reliability of science, can be especially pernicious.³¹

Faced with this reckoning, however, quantitative researchers have instituted solutions. Journals have formalized many practices to improve how transparent and accountable quantitative studies are, including publishing more papers that aim to replicate prior findings, encouraging researchers to preregister their hypotheses in public repositories, requiring authors to publicly post the data and code that produced their analyses, and more.³² At least some of these efforts are working. Malfeasance, mistakes, and questionable analytical decisions are often discovered quickly, discussed openly, and retracted as needed.³³ Bad research habits that were commonplace a decade ago are increasingly rare in the disciplines’ best journals, as editors impose greater transparency, expect multiple robustness checks, and require making code and data easily available to reviewers.

These developments might have been great news for qualitative researchers but for the fact that many of the recommended practices are inappropriate for their work. For example, calling for researchers to replicate more ethnographies will often make no sense: an observational case study of a single event—for example, the Arab Spring or the George Floyd protests—cannot be replicated, since a future researcher cannot return to the past, recreate the event, and experience what the real-time observer did.³⁴ Similarly, requiring all field-workers to preregister their hypotheses would undermine major traditions in which the research is inductive; an ethnographer who is entering a study site without intending to test a prior hypothesis would have nothing to preregister.

In recent years, some qualitative researchers have proposed alternative ways of increasing the transparency of qualitative research.³⁵ Some scholars have discouraged ethnographers from anonymizing a field site