Theory, Measurement, and Analysis
Evaluating Community Initiatives: A Progress Report
Anne C. Kubisch, Karen Fulbright-Anderson, and James P. Connell
The Roundtable on Comprehensive Community Initiatives for Children and Families is a forum where representatives of many groups engaged in comprehensive community initiatives (CCIs) can meet to discuss lessons emerging from this relatively new approach to community revitalization. Established in 1992, the Roundtable moved to the Aspen Institute in 1994 and now includes approximately 30 foundation sponsors, program directors, technical assistance providers, evaluators, and public sector officials. Also in 1994, the Roundtable formed the Steering Committee on Evaluation, which attempts to address problems encountered by stakeholders in their efforts to learn from and judge the effectiveness of CCIs. (Members of the Roundtable and the Steering Committee on Evaluation are listed on pages 253-56 of this volume.)
Soon after its formation, the Steering Committee on Evaluation commissioned several papers on some of the more pressing issues in CCI evaluation. The papers were published in 1995 in New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, edited by James Connell, Anne Kubisch, Lisbeth Schorr, and Carol Weiss. The current volume follows up on concerns raised in that first collection, hereinafter referred to as Volume 1.
The Emergence of Comprehensive Community Initiatives
In the United States, the history of locally based initiatives to improve poor urban neighborhoods and the lives of their residents dates back to the late nineteenth century. Comprehensive community initiatives (CCIs) represent the most recent generation in a line of neighborhood-level efforts that includes settlement houses, Grey Areas, Model Cities, Community Action Programs, and community development corporations.1 CCIs build on and incorporate lessons from each of these models.
Individual CCIs take different forms and structures, yet all are guided by principles of comprehensiveness and community building. As comprehensive initiatives, CCIs seek to strengthen all sectors of neighborhood well-being, including social, educational, economic, physical, and cultural aspects, and in so doing seek to achieve a level of synergy among them. Although CCIs may not implement programmatic strategies for all sectors at the outset, they pay attention to the interrelationships among sectors while pursuing opportunities to improve community conditions. By claiming comprehensiveness, CCIs also indicate a commitment to change at many levels, including individual, family, institutional, and community-wide, through processes that involve collaboration and coordination within the community and between the community and the broader society.
The community-building dimension of CCIs emphasizes how this change should come about. It places priority on a resident-driven approach that values local knowledge and participation in all stages of the revitalization effort, from initial goal setting through evaluation and policy change. It aims to build capacity at the neighborhood level for a continuous process of local improvement. It also aims to build or rebuild "social capital"—that is, the social fabric among residents—in the community. Finally, community building operates according to values of racial equity, social and economic justice, and respect for community assets, including local culture and history.
Most CCIs have been created by, and receive support from, private foundations, although there is a growing number of federal, state, and locally sponsored initiatives. In addition, many institutions have undertaken activities designed to provide technical assistance to CCIs, to evaluate and carry out research in order to increase understanding of their work, and to share knowledge among CCIs and disseminate information about them to a broader audience. Perhaps more important, however, is the way in which the principles that underlie CCIs are increasingly guiding a wide range of publicly and privately sponsored social welfare and economic development efforts. A growing number of sector-specific or outcome-
specific initiatives in education, health, youth development, crime prevention, substance abuse, child development, child welfare, and other areas are being guided by principles of comprehensiveness and community building. Thus, while CCIs may represent the "purest" application of comprehensiveness and community building, they are by no means the exclusive expressions of those principles.
Why is Good Evaluation of CCIs So Important?
CCIs bear an enormous responsibility to ensure that we learn as much as possible from their collective experience for three sets of reasons, one political, one practical, and one more technical.
Politically, CCIs are public efforts that seek to be both democratic in process while also committed to results. As such, they benefit from the support of a wide range of constituents, all of whom need to be kept informed of progress and outcomes. Funders look for evidence that their investments are "paying off," quite often within the time frame of their funding cycles. But in CCIs, neighborhood residents and "influentials" outside the community are equally important audiences for information about the initiative. They too need evidence that their investments—whether in time, energy, or social or political capital—are having an effect. Other partners in the public, nonprofit, and corporate sectors also need to be kept informed of progress.
The second press for good evaluation is the need to generate useful feedback to guide implementation. Surely any good program or agency manager wants formative feedback to inform planning, management and administration, and mid-course correction. In CCIs, this need is intensified by the defining mandate to create new and different ways of doing business at the neighborhood level. No stock blueprint shows how this should happen: structures and activities must be tailored to individual neighborhood circumstances. Thus, contemporaneous learning from each CCI’s unique, progressive experimentation is necessary to guide practitioners’ decisions and actions and to help ensure that CCIs have the best chance of success.
The third reason for good evaluation has to do with what Alice O’Connor (1995) calls the need for "social learning" over the longer run. Essentially laboratories for applying the best lessons from previous inner city revitalization efforts, CCIs offer opportunities for practitioners and researchers to test their knowledge and experience and to examine some of the most fundamental and cutting-edge questions in the antipoverty field today—questions concerning the configuration of model programs, for example, or the role that "social capital" and "neighborhood capacity" play in promoting healthy communities. This information, ultimately, cycles back to inform and shape both the policy and research agendas.
The Challenge of Evaluating CCIs
Broad interest in CCIs means that CCI evaluations are being asked to serve multiple purposes for a host of audiences. Fortunately, many CCI funders have recognized the value of investing in learning as much as possible from the current cohort of interventions, and a number of evaluations are underway.2
At the same time, the very characteristics of CCIs that give hope to both stakeholders and observers are also the ones that challenge our ability to evaluate and learn from them. In their introduction to the first Aspen Roundtable volume on evaluation, Kubisch and colleagues (1995) describe these CCI features and why they raise evaluation-related difficulties:
This enumeration of evaluation-related difficulties helped identify the scope of the problem facing the field, yet a different framework was required for structuring work to address those difficulties. One of the first priorities of the Roundtable’s Steering Committee on Evaluation was to reshape this list of complex and inter-related challenges into a more manageable and accessible framework within which to organize new thinking and new work. That process led to the identification of three fundamental questions that have guided the committee’s work over the last three years; those questions, in turn, form the structure of this volume.
- Horizontal complexity. They work across multiple sectors (social, economic, physical, political, and others) simultaneously and aim for synergy among them.
- Vertical complexity. They aim for change at the individual, family, community, organizational, and systems levels.
- Community building. They aim for strengthened community capacity, enhanced social capital, an empowered neighborhood, and similar outcomes.
- Contextual issues. They aim to incorporate external political, economic and other conditions into their framework, even though they may have little power to affect them.
- Community responsiveness and flexibility over time. They are designed to be community-specific and to evolve in response to the dynamics of the neighborhood and the lessons being learned by the initiative.
- Community saturation. They aim to reach all members of a community, and therefore individual residents cannot be randomly assigned to treatment and control groups for the purposes of assessing the CCI’s impact; finding equivalent comparison communities is also not feasible.3
Volume 1: Concepts, Methods, and Contexts
- Given the breadth, complexity, and evolving nature of CCIs, what can be done to clarify the short, interim, and long-term outcomes of an initiative, as well as the projected pathways for achieving them, in a way that guides initiative evaluation?
- Given the value that CCIs place on community building, on the process of change, and on community-level change, what are compelling indicators of all CCI outcomes—early, interim and long-term—and how should they be measured?
- Given the community-specific and community-wide goals of CCIs, how should data be collected and analyzed to ensure that the causal links between initiative activities and outcomes—and the role of contextual variables in those links—are as fully understood as possible?
Perhaps the most important contribution of Volume 1 was to introduce the potential applicability of theory-based evaluation to the CCI field. In "Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives," Carol Weiss hypothesized that a key reason that CCIs and other complex programs are so difficult to evaluate is that the "theories of change" that underlie the structures, strategies, and goals of CCIs are poorly articulated. Weiss challenged CCI designers to be specific about theories of change and suggested that doing so would improve their overall evaluation plans while also providing guidelines for data collection and analysis.4
In the three years since the publication of Weiss’s paper, the importance of achieving clarity about the theory of change underlying a CCI has gained considerable popularity among CCI funders, evaluators, and practitioners. This turn of events was perhaps predictable, given that many CCIs did indeed begin with only very general theories of change: as funders and communities entered into agreements to work together toward neighborhood transformation, they deliberately left aspects of their initiatives open so that collaborative processes of planning and implementation could unfold. But why were concerns about the clarity of the theory of change triggered by questions regarding CCI evaluation, as opposed to, say, planning or technical assistance? There are two answers to that question.
One answer, offered in this volume by James Connell and Anne Kubisch, is that the process of designing an evaluation requires participants to be explicit about aspects of an initiative that might otherwise be left vague for technical, political, administrative, or conceptual reasons. The authors state that "it is in the process of designing an evaluation that specific decisions must be taken regarding what is meant by key terms (such as ‘collaboration’), the type and degree of change being sought, and the measures that would indicate whether change is occurring."
The second answer is that many CCI funders decided early in the process that good evaluation should accompany these new and high-risk efforts, not only as a means for tracking their considerable investments closely and gathering formative feedback but also to ensure that these important social experiments would generate useful knowledge along the way. As a result, evaluation became an important locus for creative thinking about the range of planning, implementation, and research challenges facing CCIs.
These circumstances meant that, at each stage, CCI evaluators have been confronted with challenges for which traditional evaluation tools have seemed inadequate. Problems at the outset included vaguely defined interventions; at the measurement stage, the absence of indicators for many of the community-building concepts; and at the analysis stage, the difficulty of causal attribution. For CCI evaluators, the "theory of change" notion offers a framework for breaking down this complex set of problems into discrete and more manageable questions.
Volume 2: Theory, Measurement, and Analysis
This second volume offers a progress report on work accomplished over the last few years. The first paper, by Connell and Kubisch, attempts to take Carol Weiss’s formulation of a theory of change evaluation to the next level: that is, to apply it specifically to the circumstances of a CCI. They define a theory of change approach to CCI evaluation and describe steps stakeholders can follow to carry one out.
In the five contributions that follow, several practitioners share their experiences with the theory of change approach to evaluation, together providing an up-to-date portrait of the field. As a group, the papers provide an overview for understanding how to apply the approach and examples of its benefits and challenges when used in different settings and at different stages of an initiative.
The paper by Sharon Milligan, Claudia Coulton, Peter York, and Ronald Register was written as part of an effort by the Aspen Roundtable’s Steering Committee on Evaluation to "field test" the concept on the ground, in the context of a comprehensive community initiative. A fortunate convergence of interests and timing enabled the Roundtable to form a partnership with the Cleveland Community-Building Initiative in the early phases of its evaluation design process. The Cleveland site benefited from technical and financial resources provided by the Roundtable, while the Roundtable gained from the opportunity to test the approach. In addition, evaluators whose recent work incorporates a theory of change approach were invited to write short, analytical papers describing the strengths and weaknesses of the approach. The contributions by Prudence Brown, Lynn Kagan, Susan Philliber, and Scott Hebert and Andrea Anderson are intended to give readers a broad set of perspectives on how the approach actually works, in practice, for evaluators.
The volume then turns to issues regarding measurement of change and the availability of data. The problem of measuring the range of types and levels of change that CCIs expect to influence has been at the forefront of the agenda of the Steering Committee on Evaluation since its inception. Because CCIs aim for change at many levels (individual, family, personal network, institutional, and community), across many domains (economic, social, physical, and community building), and over different time periods (near-term, interim, and long-term or ultimate), the measurement task is substantial. Surveying even well-understood measures in so many areas would be a daunting process for any evaluator—but for CCI evaluators, good measures simply do not exist for many of the most important desired outcomes, such as community building.
Since the publication of Volume 1, the Roundtable has embarked on several activities to identify measurement issues and organize measures of community-level change. One step was to convene a meeting of researchers and evaluators in December 1996. Michelle Gambone took that wide-ranging discussion and distilled from it a framework, presented in this volume, for thinking through the various CCI dimensions that need measuring, covering context, progress, and early, interim, and long-term outcomes. The Steering Committee on Evaluation also prevailed upon two of its members, Claudia Coulton and Robinson Hollister, to share their extensive knowledge of administrative, census, and survey data in a paper that reviews the use of existing data to measure change at the neighborhood level.
Turning to the thorny issue of attributing causality—the problem that dominates evaluators’ discussions about CCIs—Robert Granger takes up a challenge issued by Hollister and Hill in Volume 1 and suggests several strategies evaluators can use to strengthen the case that neighborhood outcomes can be attributed convincingly to a CCI.
A final theme woven throughout the discussion is the changing role of the evaluator in the context of CCIs and other complex community-based initiatives. In Volume 1, Prudence Brown argued for a more engaged role than the evaluator has traditionally adopted, and the authors in this volume for the most part concur. The five "practitioner reflections" presented here all describe significant participation in the theory articulation process, the research design, collection of data, and other steps. The papers are rich in their discussion of the advantages of this approach, but they are also honest about the difficulties that arise.
This, in turn, raises the question of who should be responsible for the theory articulation process. A theory of change that is accepted by all stakeholders is a product of a CCI’s work.5 It comes out of an in-depth planning process, but as Connell and Klem (1996) and Schorr (1997) suggest, theories can be informed by knowledge and experience from outside the community, implying a different, and perhaps less intensive, process of producing a theory of change that is nonetheless community owned. Experts in fields relevant to community revitalization, therefore, may be more appropriate facilitators of the process of creating the theory of change than are evaluators.
How do we define a CCI’s theory of change? How should we measure the change? How can we be sure that the CCI caused the change? What is the role of the evaluator in this new approach? This volume describes the progress that has been made in the field as a whole on each of those questions over the last three years. Greater advances have been made in some areas than in others, and in some cases the progress has been more conceptual than applied. Nonetheless, the members of the Roundtable’s evaluation committee believe that the papers included here offer new ways of thinking, new insights, and new information, which together can help CCI stakeholders approach the evaluation enterprise with greater confidence that the result will meet their various needs.
- For further discussion of the evolution CCIs and the state of the field today, see Kubisch et al., 1995; Kingsley et al., 1997; Stone, 1996; Walsh, 1996; Lieterman and Stillman, 1993; Kubisch et al., 1997; Pitcoff, 1997, 1998; Jackson and Marris, 1996; Halpern, 1994; O’Connor, 1995; O’Connor, forthcoming; Ferguson and Dickens, forthcoming; and Wright, 1998.
- See, for example, Chapin Hall Center for Children, 1997; Chaskin, 1992; Chaskin and Joseph, 1995; Chaskin, Dansokho, and Joseph, 1997; Hirota, Brown, and Butler, 1998; Sviridoff and Ryan, 1996; Nathan, 1997; OMG, Inc., 1995, 1998; Hebert and Anderson, this volume; Milligan et al., this volume; and Stillman et al., 1996.
- The lack of suitable comparison groups in CCI evaluations is discussed at greater length in Hollister and Hill, 1995.
- Chen (1990) and Weiss (1995, 1997) offer further thoughts on the challenges of theory-based evaluation.
- For more on the role of the evaluator, see Connell, 1997.
Chapin Hall Center for Children. 1997. The Partnership for Neighborhood Initiatives: Report of the Chapin Hall Center for Children at the University of Chicago. Chicago: Chapin Hall Center for Children.
Chaskin, Robert. 1992. The Ford Foundation’s Neighborhood and Family Initiative: Toward a Model of Comprehensive Neighborhood-Based Development. Chicago: Chapin Hall Center for Children.
Chaskin, Robert, Selma Chipenda Dansokho, and Mark Joseph. 1997. The Ford Foundation’s Neighborhood and Family Initiative: The Challenge of Sustainability. Chicago: Chapin Hall Center for Children.
Chaskin, Robert, and Mark Joseph. 1995. The Neighborhood and Family Initiative: Moving toward Implementation. Chicago: Chapin Hall Center for Children.
Connell, James P. 1997. "Render unto Evaluators…: Some Cautions from Early Experience with a Theory of Change Approach." Paper presented at the Annie E. Casey Foundation Evaluation Conference, Baltimore.
Connell, James P., and Adina Klem. 1996. "Using a Theory of Change Approach to Evaluate Investments in Public Education." Paper presented at a meeting convened by Independent Sector, Washington, DC.
Chen, Huey-tsyh. 1990. Theory Driven Evaluations. Thousand Oaks, CA: Sage Publications.
Ferguson, Ronald, and William T. Dickens, eds. Forthcoming. Urban Problems and Community Development. Washington, DC: Brookings Institution.
Halpern, Robert. 1994. Rebuilding the Inner City: A History of Neighborhood Initiatives to Address Poverty in the United States. New York: Columbia University Press.
Hirota, Janice M., Prudence Brown, and Benjamin Butler. 1998. Neighborhood Strategies Project: Report on Initial Implementation, July 1996-March 1998. Chicago: Chapin Hall Center for Children.
Hollister, Robinson G., and Jennifer Hill. 1995. "Problems in the Evaluation of Community-Wide Initiatives." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Jackson, Maria-Rosario, and Peter Marris. 1996. Collaborative Comprehensive Community Initiatives: Overview of an Emerging Community Improvement Orientation. Washington, DC: Urban Institute.
Kingsley, Thomas G., Joseph McNeely, and James O. Gibson. 1997. Community Building: Coming of Age. Washington, DC: Urban Institute.
Kubisch, Anne C., Prudence Brown, Robert Chaskin, Janice Hirota, Mark Joseph, Harold Richman, and Michelle Roberts. 1997. Voices from the Field: Learning from Comprehensive Community Initiatives. New York: Roundtable on Comprehensive Community Initiatives for Children and Families, Aspen Institute.
Kubisch, Anne, Carol H. Weiss, Lisbeth B. Schorr, and James P. Connell. 1995. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Leiterman, M., and J. Stillman. 1993. Building Community. New York: Local Initiatives Support Corporation.
Nathan, Richard, P. 1997. Empowerment Zone Initiative: Building a Community Plan for Strategic Change: Findings from the First Round of Assessment. Albany: Nelson A. Rockefeller Institute of Government, State University of New York.
O’Connor, Alice. Forthcoming. "Swimming Against the Tide: A Brief History of Federal Policy in Poor Communities." In Urban Problems and Community Development. Washington, DC: Brookings Institution.
O’Connor, Alice. 1995. "Evaluating Comprehensive Community Initiatives: A View from History." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
OMG, Inc. 1998. Final Assessment Report: Comprehensive Community Revitalization Program in the South Bronx. New York: Comprehensive Community Revitalization Program.
OMG, Inc. 1995. Final Assessment Report: The Planning Phase of the Rebuilding Communities Initiative. Baltimore: Annie E. Casey Foundation.
Pitcoff, Winton. 1998. "Redefining Community Development, Part II: Collaborating for Change." Shelterforce 20 (1):2-17.
Pitcoff, Winton. 1997. "Redefining Community Development." Shelterforce 19 (6):2-14.
Schorr, Lisbeth. 1997. Common Purpose. New York: Doubleday.
Stillman, Joseph, Benjamin Butler, Prudence Brown, and Lenneal J. Henderson. 1996. Sandtown-Winchester Community Building in Partnership: 1990-1994: Interim Evaluation Report. New York: Conservation Company.
Stone, Rebecca. 1996. Core Issues in Comprehensive Community-Building Initiatives. Chicago: Chapin Hall Center for Children.
Sviridoff, Mitchell, and Willam Ryan. 1996. Investing in Community: Lessons and Implications of the Comprehensive Community Revitalization Program. New York: Comprehensive Community Revitalization Program.
Walsh, Joan. 1996. Stories of Renewal: Community Building and the Future of Urban America. New York: Rockefeller Foundation.
Weiss, Carol Hirschon. 1997. "How Can Theory-Based Evaluation Make Greater Headway?" Evaluation Review 21, no. 4.
Weiss, Carol Hirschon. 1995. "Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for Comprehensive Community Initiatives for Children and Families." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Wright, David J. 1998. "Comprehensive Strategies for Community Renewal." Rockefeller Institute Bulletin, pp. 48-66.
Back to New Approaches to Evaluating Community Initiatives index.
Copyright © 1999 by The Aspen Institute
Comments, questions or suggestions? E-mail email@example.com.
This page designed, hosted, and maintained by Change Communications.