Grassroots Action and Learning for Social Change:

Evaluating Community Organizing

Catherine Crystal Foster Justin Louie Blueprint Research & Design, Inc. March 2010

2010 Center for Evaluation Innovation. All rights reserved. May not be reproduced whole or in part without written permission from Center for Evaluation Innovation.

Grassroots Action and Learning for Social Change:

Evaluating Community Organizing

Catherine Crystal Foster and Justin Louie Blueprint Research & Design, Inc. March 2010

n recent years, the practice of community organizing has matured and evolved, gaining visibility as a vibrant and potent force for social change. This has coincided with a growing focus on systemic reforms throughout the social sector, reforms that effective community organizing can help advance. As this work has gained more prominence, and its potential has been more broadly understood, interest in assessing the contributions and impact of organizing activities has surged. This interest is part of a larger trend seeking metrics to drive program improvement and innovation in the social sector. Growing numbers of funders, evaluators, and nonprofits are striving to develop meaningful program assessments, particularly for systemic reforms such as organizing, advocacy, and other policy change work. While it was once common to hear questions about whether advocacy efforts could be meaningfully evaluated, those have long since given way to discussions of how to take advocacy evaluation to the next level. Development of frameworks and an array of tools to measure progress along the multiple dimensions of policy change and advocacy have emerged. And now, after implementing and vetting these approaches, practitioners have been adapting them to address the particular needs and circumstances of community organizing, as distinct from other forms of social change or advocacy work. Community organizing for social change shares many characteristics with policy advocacy, but it differs in significant ways, and the approaches to evaluating the two also differ. As evaluators, we have partnered with organizers, advocates, and their funders over the last five years, and we have seen these differences first-hand. Since developing our initial framework for assessing advocacy and policy change activities in 2005,1 we have worked closely with seasoned and emerging community organizers on a range of evaluation projects, covering multiple issue areas in a variety of locations. We have provided technical assistance and tools to organizers, helped systematize their existing assessment processes, and led external evaluation efforts in partnership with them. Throughout our work, we have continued to learn from organizers and refine our own thinking and processes.

Community organizing shares characteristics with policy advocacy, but it differs in important ways. Approaches for evaluating the two also differ.

Guthrie, K., Louie, J., David, T., & Crystal Foster, C. (2005). The challenge of assessing policy and advocacy activities: Strategies for a prospective evaluation approach. San Francisco, CA: Blueprint Research & Design. 1

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

This brief describes our vision of community organizing evaluation. It is grounded in a set of principles that have emerged from our experiences with organizers, as well as the collective wisdom of others working in this field. We have learned that evaluation of organizing should be:

Participatory, rather than purely third-party Prospective (contemporaneous and forward-looking), rather than retrospective Learning-based, rather than part of a pass-or-fail mentality Real-time, rather than delayed and academic Respectful of the culture of organizing Attentive to leadership development as well as policy wins Focused more on evidence than proof.

We begin the brief with an examination of the similarities and differences between advocacy and organizingdifferences that shape how our approach to evaluation of organizing differs from evaluation of advocacy. We then present a framework for evaluating organizing, and discuss some of the lessons we have learned through the ongoing process of implementation.

What is Community Organizing?

ommunity organizing is a democratically-governed, values-driven process that catalyzes the power of individuals to work collectively to make the changes they want to see in their communities. Community organizers honor and develop the leadership potential in everyday people by helping them identify problems and solutions, and then by supporting them as they take action to make those solutions a reality. In so doing, organizing challenges the existing power structure. Relationships lie at the heart of organizing, and the one-to-one relational conversation between an organizer and a community member is the building block of organizing. As those community members participate in social change work, build skills, and take on responsibilities, they become leaders within the organizing group. Developing these leaders and building the base of leaders and other community members is an ongoing focus of community organizing. A given community organizing group may emerge on its own and remain independent, or be associated with one of several national organizing networks, such as the PICO National Network, Association of Community Organizations for Reform Now (ACORN), Industrial Areas Foundation (IAF), Direct Action and Resource Training Center (DART), Gamaliel Foundation, and others. Some are faith-based, and others are institutionally- or place-based. Organizing may also coalesce around an issue, like immigration, health, or education; or around populations, such as youth, or a specific ethnic group.

Community organizing catalyzes the power of individuals to work collectively for the changes they want in their communities. It places a strong emphasis on developing community leaders.

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

Organizing Shares Qualities with Policy Advocacy

ike policy advocacy, community organizing efforts frequently require long timeframes to come to fruition. The process of growing and mobilizing a base, building power, shifting attitudes, and changing policies or practices does not happen overnight. Holding those in power accountable following change also takes time. The work of organizing, like advocacy, is complex and iterative. Strategies shift, often rapidly and multiple times, based on the external environment and other factors. Ultimate outcomes may not be readily quantifiable. The work is frequently collaborative, which brings with it benefits and challenges. And organizing campaigns, like advocacy campaigns, often spring from a sense of crisis and cause the participants to approach the work with a high level of emotional intensity.

But Organizing is Distinct from Advocacy

rganizing and advocacy differ, however, at a core level. Community organizing is emphatically bottom-up. It is the community members who select the issues, proffer the solutions, and drive strategy and execution. Most advocacy is fundamentally top-down, even if the work is authentically undertaken on behalf of community members. Advocates speak for others, while organizers inspire community leaderseveryday peopleto speak for themselves. Tellingly, the so-called Iron Rule of organizing is, Never do for people what they can do for themselves. Organizers and leaders also believe that community members can be experts, and that expertise is not the sole domain of policy professionals. A low-income mother with little formal education can be an expert on local educational needs just like a senior think tank fellow, through her own experience or by conducting communityled action research in her neighborhood school. The leader-focused lens also points to another difference from advocacy. In organizing, leadership development is a central concern and a key outcome in addition to policy change objectives. This has major implications for priorities and goals. It makes capacity development look different in organizing than in advocacy, since the capacities to attract and develop leaders are a top priority in organizing. Finally, certain logistical aspects of organizing differ from advocacy in a significant way. Organizers operate in a predominantly oral culture, in contrast to the more archived, written culture of advocacy. Organizing often places a premium on process and ritual, particularly as it concerns base-building and direct actions. In addition, organizing takes place in a more diffuse setting: in homes, churches, schools, or community venues, rather than in a central office or the corridors of the state house.

Evaluation of Organizing Presents Unique Challenges

n many respects, the similarities between advocacy and organizing present similar evaluation challenges. But for organizing, evaluation requires additional considerations that reflect the particular qualities of the work. Most important, the bottom-up nature of organizingdriven by the community, not by organizational managers or external professionalscreates a whole new set of complexities. This

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

orientation collides with the inherently top-down nature of traditional third-party evaluation, in which outside experts ask the questions, set the terms, and make judgments. As we have noted, organizers have a fundamentally different view from advocates not only of how decisions are made and priorities are set but also in where expertise resides. That affects how organizers view evaluation generally, and what role they see for themselves and their leaders in that process. If the community-defined, bottom-up goals for the work do not align completely with a funders goals, an evaluator measuring against those goals faces the difficult task of navigating between the two. When the work unfolds as part of a multi-site initiative in which multiple communities have been funded to work on an issue, those complications are compounded. Since the goals, strategies, and tactics of organizing bubble up in ways that are highly context-specific, multi-site evaluation of an organizing effort is particularly hard. It is quite difficult to standardize methodologies and roll up results when the work and processes are driven by the needs and approaches of each community. As well, the intense focus on leadership development in organizing, and the emphasis on process within some schools of organizing lead to identification of interim benchmarks and goals that often differ from those in an advocacy campaign targeting similar policy change objectives. Organizing requires additional benchmarks and goals related to the processes of growing leadership and power, and organizers may prioritize them differently from advocates. Finally, certain practical considerations implicit in organizing work can impact evaluation. Many organizers value reflection quite highly, and incorporate it in their work more explicitly than some advocates. This is particularly true for faithbased organizers. As a result, evaluation may be more about systematizing informal reflection and helping to focus it more on impact than process, not about teaching the value of it. Yet, while they do reflect regularly, organizers have very little time for formal evaluation and the rigorous, uniform, and documented processes of data collection and analysis that formal evaluation can imply. They pride themselves on never being in the office, instead spending their time in the community. Leaders who carry out the work are community members who may have entirely separate day jobs, making systematic evaluation far more challenging than when partnering with advocates working in a more traditional office environment.

Organizing requires benchmarks to capture the process of growing leadership and power.

How an Evaluator Can Address the Unique Challenges of Evaluating Organizing

hen confronted with these complicating factors, how can an evaluator respond? In our work, we have started by setting an overall approach to the evaluation that reflects key attributes of community organizing. First and foremost, a community organizing evaluation is most effective when it involves the participants in an ongoing and meaningful way, and frames the evaluation as a learning opportunity for everyone. Embracing the bottom-up approach of organizing and the organizers highly democratic notion of expertise leads to an evaluation that leverages the strengths of organizing rather than fighting its values. This is a stark contrast to accountability-focused third-party analysis. We have found that the participatory approach is the most effective way to capture useful and reliable
4

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

information, interpret that information, and ensure that data collected are actually used to improve the work. Intentional relationship building with organizers and leaders is crucial to success in such an evaluation. The dynamic nature of most organizing work and the intensity of organizing campaigns make real-time data collection and feedback particularly important. Organizers need feedback today to move their campaigns forward tomorrow. They are not as interested in academic examinations of long-past work or detailed compilations of proof. Experienced organizers know what power looks like and they know where they want to go. What they often welcome is a process for more systematically examining the multitude of interrelated forces and activities that will take them there. Evaluation feedback for organizers becomes most useful when it is oriented toward learning rather than judgment or a pass-or-fail accountability. A focus on information for learning and growth has particular resonance for organizers who devote themselves to developing individual leaders and the process of building power. Faith-based organizers and those adopting an action-reflection method of organizing naturally gravitate toward a more formative, developmental approach to evaluation. 2 Given this receptivity, and our interest in promoting evaluation for organizational learning, we have found it productive to help organizers build their own internal evaluation capacities. This enables them to work in greater partnership with us and to carry on authentic assessment activities in the long-term, both with and without external evaluation assistance. We have also found that in contrast to program or advocacy evaluation, organizing evaluation must measure and credit achievement of leadership development and capacity development goals as much as policy goals, since that mirrors the organizers work. The evaluation also works best when it accommodates organizing style and culture. This requires attention to the language and philosophical underpinnings of a given organizing groups approach, recognition of the volunteer-based staffing structure, and the more oral culture implicit in organizing work.

Evaluation for organizers is most useful when it is oriented toward learning rather than judgment or a pass-or-fail accountability.

Basic Scaffolding for Evaluation of Community Organizing

U
2

nderstanding the differences between community organizing evaluation and advocacy evaluation at the conceptual level we have described is crucial, but how does an evaluator use that understanding to implement a real evaluation on the ground?

Formative evaluation, in contrast to summative evaluation, assesses a program or activities while they are forming or unfolding. It looks at the context, inputs, and processes of implementation to help improve and refine the program or activities. Developmental evaluation, a term coined by Michael Quinn Patton, is a team effort that supports program, project, personnel and/or organizational development through a process of continuous improvement and adaptation. The evaluator in a developmental evaluation facilitates data-based decision-making by the team.

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

We have found in our work that to structure an evaluation of community organizing that yields timely information that organizers and their funders can use, we need to start by erecting some basic scaffolding with the organizers. There is little new about this part of the process, particularly to those familiar with advocacy evaluation, but it is important to establish what the supporting beams look like. As we describe, it is the next stages of construction that add the particular approaches and perspective that effective organizing evaluation requires. To begin, in partnership with organizers:

Articulate a theory of change,3 by whatever name and in whatever form is most accessible to the organizers and their leaders, making it clear that the theory does not need to be linear or static. Articulate evaluation questions, thinking collaboratively about what organizers, leaders, funders, and other stakeholders want to learn. Define benchmarks and indicators, paying particular attention to mid-term benchmarks and to the range of outcomes that are important to reaching organizing goals, recognizing that they may shift over time. Establish multiple data collection methods and collect data by involving evaluators, organizers, and leaders in the process. Provide real-time feedback, to the extent feasible. Document progress toward benchmarks and goals, and re-evaluate and refine the goals and benchmarks as strategies shift. Engage with funders on findings and processes.

Next, based on what has been learned:

Organizers refine their strategy. Organizers and evaluators refine evaluation practices and maintain on-going evaluation.

Building on the ScaffoldingWhat We Have Learned

he basic elements of community organizing evaluation are relatively straightforward, but how does an evaluator build on them in practice to create a learning partnership that feels authentic to organizers and yields useful results? Categorize Organizing Goals Systematically We have learned that when determining evaluation questions, setting benchmarks, and selecting data collection methods, it helps to categorize the work in a way that incorporates the values and orientation of organizing. One useful framework has

An organizers theory of change describes, in broad terms, the ultimate goals of the organizing effort and the logical chain of activities and interim outcomes that the organizers believe will get them to those goals. 6

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

been developed by the Alliance for Justice (AFJ).4 We adapted the AFJ frame, using the following core components of organizing:

Participation and membership (the base): This includes, but is not limited to, the numbers of members of an organizing group, their attendance at organizing events, and the extent to which they represent the community being organized. It encompasses their engagement in issue and strategy selection and their identification with the goals of the organizing group. Constituent leadership and power: Flowing from participation, real leadership is the distinguishing hallmark of community organizing, as opposed to advocacy or other forms of social change work. As members gain skills, take ownership of change processes, and begin to lead others in their community, they become leaders. They then use that leadership capacity to gain power and make change. Organizational power: The collective power of a community organizing group is what gives it the ability to name problems, demand specific solutions, and hold those in power accountable. It gives the organization visibility and leverage. Organizing wins: Wins are the most valued currency in community organizing. When organizers and leaders succeed in a campaign, show power in a demonstrable way, or get what they seek in an ask, they chalk up a win. A win may not be as obvious as passage of a desired policy. It may also be a key step along the way, the prevention of a less-desired outcome, or a shift in relationships. Meaningful impact following wins: Implementation and accountability after a win are as important as the win itself. Having the power to realize the promise of a win is critical in organizing. Organizational capacity: As with any successful social change entity, a community organizing group requires structures, resources, knowledge, and vision to achieve its goals. Given the way community organizing operates, the ability to forge and strengthen relationships, create strategic alliances, and manage a base are vital capacities in community organizing. Reflection and innovation: An organizing groups process for self-assessment, refinement of strategy, and adaptability in a rapidly shifting environment are key to separating skilled organizing from mere activism (direct action that may not be grounded in strategy or oriented toward sustained and effective follow-up or realignment of power relationships). Clearly, this aspect of organizing has strong resonance in the evaluation context.

When developing the evaluation, the core components of organizing can be a guide.

This Core Components of Community Organizing framework and additional useful information on evaluating organizing can be found at AFJs Resources for Evaluating Community Organizing website, http://www.afj.org/for-nonprofits-foundations/reco/. 7

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

Craft Evaluation Questions Meaningful to Organizers

e have engaged both funders and grantees in the development of evaluation questions at the very start. This keeps the evaluation relevant and useful, and promotes greater buy-in and participation by the organizers. As we continuously look to organizers for cooperation in data collection and for engagement on interpretations and findings, it has been quite helpful that many of the questions we are asking are the ones they want answered. Evaluation questions are highly individualized to the needs of the evaluation audience and sponsors. We found, however, that in our organizing evaluations, they generally fall within the core organizing component categories listed above. By referencing the components when developing questions, it may be easier to see what is missing and to surface any unintended biases in proposed areas of inquiry. Training attention on organizational capacity and constituent leadership and power, for example, can balance what might otherwise be a more exclusive focus on policy wins.

Set Clear Interim Benchmarks, and Be Willing to Refine Them

ata collection and analysis may seem like the hardest work of an evaluation. Yet setting meaningful, sufficient, and achievable interim benchmarks that are expected to occur between activities and longer-term social change goals can often be a major struggle in organizing work. With varying degrees of success, we have encouraged hard thinking about interim benchmarks focused more on substantive outcomes. We have found it can be helpful to look explicitly at the full range of organizing components and the ways in which they interrelate when setting meaningful interim benchmarks. Similarly, we have relied on regular, in-depth dialogue with organizers about the way progress within each component develops and what it looks like, since organizing benchmarks rarely remain static over time. Referring back to the relatively high-level components of organizing we set forth earlier adds a consistency to the work when the shifting environment in which organizers operate causes strategies and tactics to evolve or quickly change. The table on page 15 illustrates examples of some of the benchmarks we have used and the corresponding data collection methods used to track progress toward those benchmarks.

Use Data Collection Methods that Fit with Organizing Culture

e have used a wide array of data collection methods in our organizing evaluations. We found that we obtain the most useful information, with the most productive organizer participation, by using methods that fit the style and existing practices of the organizing group and are tailored to capture the different components of organizing. When possible, we support organizers in creating their own data collection tools. For example, we worked with a multi-issue, faith-based community organizing group in California on evaluating their efforts to expand access to health care in their county. They wanted to look deeply at the strength and breadth of the local organizing committees of their member congregations (aspects of Participation

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

and Membership and Constituent Leadership and Power). These committees were the core teams working to move the organizing groups health access agenda. The organizers led an inclusive, thorough process to select the key participation measures they wished to track. They then worked with a consultant to customize free software (Zoho) that would allow each organizer to track their leaders by committee (even when out in the field, using their iPhones), and to provide clear charts and graphs that highlighted progress and areas for growth. Looking at a very different aspect of growth, we helped education organizers in Pennsylvania think about how to document their progress in building external champions. These champions are key indicators of development of organizational power in order to leverage wins. The organizers created an online survey tool that they update regularly, which tracks relationships and interactions with specific targets over time. We can refer to the results as we assess progress, while the organizers can use the data to refine their own strategies and activities. We have shared the templates for this and other organizer data collection tools with other organizing groups in the multi-site collaborative in which the Pennsylvania organizers participate. We plan to post them on a private website so that organizers across sites can adapt each others methods (taking care not to publicize matters that reveal too much about strategy in ways that could jeopardize the organizing work). One other methodology that we have found particularly productive across almost all aspects of organizing work is what we call the critical incident debrief.5 Immediately after a win, loss, or inflection point in an organizing drive, we conduct a series of 360-degree interviews with individuals and groups both inside and outside of the campaign, including opponents, allies, observers, and decision makers. We analyze what took place, what went well, what could be improved, what growth occurred, and what can be learned as the work moves forward. We write up our findings in a narrative for the organizers to use internally and share with their funders. Our findings are generally revealing, sometimes raw, sometimes validating, and close enough in time to be actionable. In one example, our debrief following a coalitions painful legislative loss in Colorado helped provide insight on leadership and strategy that guided the coalition in a renewed effort to move the bill during the next legislative cycle.

Evaluations tend to work best when internal evaluation champions are engaged.

Develop Buy-In and Trust to Ensure Successful Implementation

uccessful implementation of an organizing evaluation hinges on the relationship between the organizers and the evaluators, and on the extent to which the evaluation meets the organizers where they are. Our evaluation efforts have sparked the most learning and have worked best when we have had internal evaluation champions within the organizing group. The champions help ensure engaged participation, which we rely upon for data collection, interpretation, and use. For example, we have worked with executive directors of organizing groups whose enthusiasm for evaluation has created a culture of

Our methodology was inspired by the Intense Period Debrief focus group protocol used by Innovation Network. http://www.innonet.org/?section_id=101&content_id=581. See also Bagnell Stuart, J. (2007). Necessity leads to innovative evaluation approach and practice. The Evaluation Exchange, 13(1), 10-11. 9

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

learning and inquiry within their organizations. This has led them to infuse data and documentation into the organizations reflection practices and to institutionalize evaluation. An organizing group in Denver, for example, used one of our debriefing narratives to call a meeting with their funders to start a dialogue on some of the provocative questions we raised. This led to a productive exchange and noteworthy shift in power relationships and credibility for the organizing grantees. Our work with a California group gave the executive director and a key staff person the opportunity to develop and implement a new suite of planning, tracking, and reflection protocols that the organizers now use routinely. But we have also seen that it is important that the evaluation is widely held within an organization, not merely by the champion. Absent broader buy-in, evaluators may not receive consistent, complete, and sustainable flows of information, particularly given how busy organizers can be and the usual rate of turnover in organizing groups. Our evaluation champions all were naturally attuned to learning, but our ability to cultivate them as evaluation champions depended on our efforts to develop relationships with them. This relationship-building takes time, and requires evaluators to work to earn the organizers trust. Honesty, preservation of the confidentiality of strategic information, using the unique language of organizing, and showing understanding of the groups organizing philosophy all make a real difference. Similarly, demonstrated interest in digging deep into the policy and political issues the organizers are confronting, and understanding their political environment and the players in it, yields major dividends. For some organizers, that means having the kind of one-to-one relational meetings that organizers have with their leaders. For others, it means poring over blog posts and news accounts to get a detailed grounding in the political realities of a campaign so that we can speak more as partners and use organizers time respectfully. Ultimately, the relationship and the evaluation need to have enough value to the organizers to justify the expense of time and effort. And the relationship needs to be strong enough that evaluators can maintain the ongoing data collection and feedback activities that prospective, real-time evaluation requires, even when the feedback delivered may not be what the organizer wants to hear. Beyond the basics of interpersonal relationship-building, in our work with organizers we have secured additional stipends for the organizers to assist with local data collection. While the stipends do not fully compensate the organizers for the time they spend on evaluation, the fact that we sought the stipends is a gesture of respect and helps solidify our partnership. We have also found it important to respect the culture and existing organizational systems and approaches of our organizing partners. In some cases, that means using more rudimentary data collection approaches than desired, or relying on existing systems. For others, it means waiting until the organization can more organically arrive at systems that have meaning to them and are sustainable. For example, we created an online media tracking tool for one organization, which they modified and piloted. They determined it was not a process they could maintain, nor was it yielding the information they really needed. We gave them the space to create what they needed and they now are focusing on tracking the growth of their relationships with journalists in targeted markets rather than logging coverage. They have found this useful and so have we.
Grassroots Action and Learning for Social Change: Evaluating Community Organizing 10

When developing data collection systems, it is important to respect the culture of organizing groups and their existing evaluation capacity or systems.

Set Achievable Expectations to Maximize Learning and Sustainability

or organizations that were building internal evaluation mechanisms rather than relying on external evaluation, we have encouraged them to go slow and start small. That may require focusing on a single aspect of the work. The organization that examined its local organizing committees using Zoho software, for example, was initially going to evaluate its health policy work more directly. Instead, the leadership development work provided a comfortable and helpful entry point. Starting with activities that offer useful, timely insights into ongoing work has been most effective in our organizing evaluations to date. In fact, despite our recommendations, some organizers with whom we worked have been so excited about implementing new tracking and evaluation systems that they have forged ahead, only to become overwhelmed part-way through. Our eagerness to see these organizations embrace evaluation probably inhibited us from forcing the issue and requiring them to scale down their expectations and not bite off more than their organization could handle. As approaches have been tested, they have learned and retooled the approaches to work for them in the longer term.

Use Evaluation Results to Create Impact, But Recognize the Challenges of Getting Results in Real-Time

e firmly believe that evaluation has value only if the results are used. We have been gratified by the extent to which organizers and their funders have learned from our evaluation efforts and acted on what they have learned. We have seen that ownership of the data and willingness to learn (even from hard truths) can lead to powerful impact. The Colorado organizers who have used evaluation to refine campaign strategy and explore issues with funders in a new way are just some examples. One coalition of organizing partners in Chicago is using evaluation data on a win to make certain that implementation realizes its full potential. Another Chicago organizer coalition is learning from our debriefing on a win to ensure that efforts to repeat the success are executed most effectively. Several organizers elsewhere are refining their base-building efforts after examining areas for growth revealed through data. Using results well, however, is not always easy. Timeliness matters, and providing real-time or close-in-time data is often challenging. Collecting enough data from a sufficient number and range of sources is not quick, particularly when the information sources are external to the organizing effort. Organizers assisting with data collection may be too busy to collect data regularly enough for it to be useful. While we attempt to provide feedback as rapidly as possible, we have realized over time that we needed to be realistic with organizers about how long it can take to get sufficient and accurate information back to them. Nevertheless, we continue to try to make our feedback happen as quickly as possible. It can also be difficult to propagate the learning in ongoing, fast-paced, and sensitive campaigns. Similarly, the politics and complex communications paths in coalitions and networks can slow down or complicate information sharing, particularly when there are questions of attribution or responsibility. We have found it helpful to be as patient, focused, and transparent as possible and to provide our analysis in an accessible way. Lifting up examples from peer organizing groups has also effectively

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

11

motivated organizers to focus on and use evaluation themselves.

Clarify What Success Means

efining and measuring success plays a critical role in evaluation, whether that means a policy win, the achievement of learning goals, or achieving growth in organizational capacity. Through our work in a variety of social change contexts and with a range of funders, it has become clear how important it is to surface assumptions about what success looks like from the perspective of an organizer, versus an advocate, versus a funder (and even among different types of funders). As we have noted, success for organizers encompasses leadership development and positive shifts in power dynamics as well as desired policy change. Success for others may be more limited. In addition, our evaluation work with organizers and advocates has reinforced that for many organizers who work in a values-based context, how you win is often as important as winning itself. Passing a bill may be an important goal, but it may be equally important for the individual leaders pushing for that passage to have their role and identity framed in a particular way as the bill is considered, or for the compromises to evince a particular power alignment. This may not be as true for some allied advocates or funders. Evaluation of an organizing campaign needs to account for such nuances when measuring success. Moreover, funders of organizing may differ about the role that organizing can and should play in a policy success, depending on whether they come at the grant as issue-oriented funders or as community or civic engagement funders. Issue-based funders (health care, education, environment, etc.) may see organizing as simply one tool to be used in reaching a policy change or social change goal, while community or civic engagement funders may see organizing as the central component of the work and an end in and of itself. When framing the evaluation questions and analyzing the results, evaluators must ensure that there is clarity on the funders orientation and how funders articulate success.

Success for organizers encompasses leadership development and positive shifts in power dynamics as well as desired policy change.

Conclusion

e have learned from our work with organizers that expertise comes in many forms and from a variety of perspectives. This brief presents our perspective on evaluating community organizing based on the evaluations we have conducted to date. We are still learning, and there is much we would like to develop further. How can we provide more useful, usable tools that fit with organizing systems and culture? How can we find ways to get valid feedback to organizers more quickly? How can we balance the needs and expectations of funders, organizers, and community leaders? We see this brief as part of a continuing conversation on how to evaluate community organizing in a way that benefits all involved. We look forward to more discussions that validate, challenge, or refine our thinking, and we welcome new voices to the conversation. As we continue our evaluation efforts, we will work to uncover and communicate new lessons for the field, and bring the voices of organizers into the conversation. As part of our recent evaluation capacity- building efforts with community organizing grantees of The California Endowment, we developed a series of videos in which
12

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

organizers discuss their experiences developing and conducting evaluations of their efforts. These can be found at http://www.blueprintrd.com/do-it-yourself-communityorganizing-evaluation. We invite you to take a look and to join our conversation.

About the Authors

atherine Crystal Foster is an independent strategist, advocate, and evaluator with over 20 years experience in the nonprofit community. She serves a strategic partner of the San Francisco-based philanthropy strategy firm Blueprint Research & Design, where she co-leads evaluations of policy, advocacy, and community organizing initiatives. Catherine also partners with nonprofit organizations and funders to advance their policy and advocacy activities. These partnerships are grounded in Catherines experience directing child advocacy and welfare policy initiatives before opening her consulting practice. She has also served on numerous nonprofit boards of directors and is an active participant in civic affairs in her home city of Palo Alto, California. Catherine can be reached at [email protected]. Justin Louie is a senior analyst at Blueprint Research & Design, and has spent the last 13 years developing and conducting evaluations for nonprofits, for-profit corporations, foundations and government agencies. He leads Blueprints practice area in organizational learning and evaluation. Early on in his career he was a community and youth organizer focused on HIV prevention, and he has served on the boards of a number of grassroots, state, and national policy and political organizations. He has degrees in sociology (BA) and public policy (MPP) from the University of California, Berkeley. Justin can be reached at [email protected].

Acknowledgements
This brief was developed with generous support from The California Endowment, with special thanks to Astrid Hendricks and Gigi Barsoum.

About Blueprint Research & Design, Inc.

lueprint Research & Design helps grantmaking foundations, individual and family donors, and philanthropic networks achieve their missions. Blueprint believes that philanthropy is a critical part of our global society and that research and systems thinking help individual organizations best achieve their goals. Services include program strategy and design; organizational learning and evaluation; and philanthropic industry analysis. Blueprint has been a leader in the advocacy evaluation field, and with support from The California Endowment, produced a seminal report on the field in 2005 titled The Challenge of Assessing Policy and Advocacy Activities: Strategies for a Prospective Evaluation Approach. That report can be found at www.blueprintrd.com.

About the Center for Evaluation Innovation

he Center for Evaluation Innovation is pushing evaluation practice in new directions and into new arenas. The Center specializes in areas that are hard to measure and where fresh thinking and new approaches are required. This includes advocacy and policy change, communications, and systems change efforts. The Center works with other organizations to develop and then share new ideas and solutions to evaluation challenges through research, communications, training development, and convening. www.evaluationinnovation.org.

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

13

About Advocacy Evaluation Advances

n 2009, 120 advocates, evaluators and funders gathered at The California Endowments Center for Healthy Communities for two days of thought-provoking presentations and discussions on recent advocacy evaluation advances. The convening, sponsored by The California Endowment with support from The Atlantic Philanthropies and Annie E. Casey Foundation, focused on real-life experiences with advocacy evaluation and what has been learned from testing different tools and approaches in this emerging field over the last several years. It also focused on challenges that still must be addressed, and identified priorities for the field moving forward. Some of the content and findings featured in this brief were presented and discussed during the convening. To access other convening and presenter resources, visit the Advocacy Evaluation Advances web page at www.calendow.org/article. aspx?id=3774.

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

14

Sample Benchmarks and Data Collection Methods for the Core Components of Organizing
Organizing Component
PartiCiPatiOn anD MeMBershiP

Benchmarks
Changes in numbers, demographics or location of members Changes in attendance (numbers, types of events, who attends) Changes in attitudes, skills, and knowledge Changes in self-esteem and self-efficacy Changes in stature within community or among decision makers Development of relationships with decision makers, media, and influential figures Changes in stature within community or among decision makers Changes in membership Changes in turnout to events Policy wins Policy wins Shifts in norms or content of debate Holding the line against negative actions Implementation of policies Changes in practices Public accountability for action or inaction Sustained shifts in norms or content of debate Impact on community Changes staffing Changes in infrastructure Changes in skills Changes in resources Building on and systematizing internal processes Infusing data and documentation into reflection Use of data in refinement of strategy or tactics

Data Collecting Methods

Membership tracking (including demographic and geographic info) Attendance tracking Tracking elements of leadership growth Organizer check-ins and debriefs Documenting 1-to-1s Journaling/portfolios Focus groups Power analysis Relationship/champion tracking Base-building/mobilization tracking Media tracking Policy developments tracking Interviews Critical incident debriefs or case studies Policy tracking Collection of archival documents Media tracking Critical incident debriefs or case studies Policy implementation tracking Community indicators tracking Action research (accountability surveys, interviews, focus groups) Interviews Critical incident debriefs or case studies Organizational capacity assessments Most Significant Change1 Interviews and check-in calls or meetings Interviews and check-in calls or meetings Collection of assessment documents or examination of systems

COnstituent leaDershiP anD POwer

OrganizatiOnal POwer

Organizing wins

Meaningful iMPaCt fOllOwing wins

OrganizatiOnal CaPaCity

refleCtiOn anD innOvatiOn

This is a form of participatory monitoring and evaluation developed by Rick Davies and Jess Dart that involves the participants collection and discussion of stories about the most significant changes resulting from a program or action. See http://www.mande.co.uk/docs/ MSCGuide.pdf.

Grassroots Action and Learning for Social Change: Evaluating Community Organizing

15