Theory, Measurement, and Analysis
Challenges of Measurement in Community Change Initiatives
Michelle Alberti Gambone
Community change initiatives (CCIs), as is clear to everyone associated with them, are very complex endeavors. Whatever the particular focus of individual initiatives, all CCIs have in common the ambitious goal of catalyzing and sustaining significant change in fundamental aspects of social, economic, and political structures and their functioning in communities. They all begin with the premise, in some form, that activities can be undertaken that will alter basic patterns of social interaction, values, customs, and institutions in ways that will significantly improve the quality of life of a community’s residents.
This characteristic distinguishes CCIs from more traditional social or economic interventions, which typically attempt to meet social policy goals by using a relatively defined and discrete mechanism (such as a new service or program) to produce desired changes in the lives of targeted individuals. CCIs attempt to change the everyday environment in communities in ways that will result in better outcomes for everyone living within a designated geographic area. This crucial difference in strategy poses a new and complex set of challenges for those involved in implementing CCIs, and at the same time multiplies and complicates the issues that need to be addressed by those evaluating the initiatives.
This chapter builds on a meeting attended by evaluators and researchers convened by the Aspen Institute Roundtable on Comprehensive Community Initiatives for Children and Families focusing on the specific measurement-related challenges facing those who attempt to assess the progress and impact of these initiatives. Over the course of the meeting, the participants articulated the range of theoretical and methodological evaluation challenges that are surfaced by CCIs, raising questions that ran the gamut from the philosophical ("How do we attribute causality?") to the practical ("What instruments exist for the range of variables to be measured?"). At the same time, it was also apparent that the state of the field is such that evaluation strategies, generally, and measurement strategies, specifically, are unfolding as these community efforts proceed.
The purpose of this chapter is to identify and briefly discuss some of the most pressing issues that stem from the particular problem of measuring change in CCIs and, in so doing, to improve understanding of one of the key challenges in the overall process of researching community endeavors.
The Role of Theory in Measurement
The most basic measurement-related questions facing evaluators of CCIs are the same as those facing any evaluator: What should be measured, how, and when? As other chapters in this volume make clear, such seemingly straightforward technical issues are complicated in CCIs both because of the comprehensive and evolving design of the interventions and because of the multiple purposes of evaluation, ranging from formative feedback to assessment of impact to social learning. Once again, theory helps orient the researcher, in this case on measurement issues.
First, no research design with finite time, money, and human resources can test all the possible relationships among activities, outcomes, and contexts in a community. As posited by Weiss (1995), a theory of change requires, first and foremost, that all stakeholders be clear about an initiative’s intended goals, which in turn should provide general guidance about what to measure at baseline and over the long term. Also, a theory of change demands some level of clarity about the pathways intended for getting to the longer-term goals. Those pathways are generally made up of a series of activity-outcome sequences which are, in principle, measurable. Thus, a theory of change should help guide decisions about what aspects of an initiative should be measured and in what order the measurement should take place.
Second, given the experimental nature of CCIs, the "social learning" dimension of evaluation is critical and affects decisions about how to structure and carry out measurement. The progress of "normal science" is rooted in the principle that knowledge development occurs only when high-quality, reliable measurement strategies are combined with well-specified theories so that hypotheses can be tested. As the resulting information is used to confirm or discard hypotheses, new "knowledge" is gained, and disciplines make progress in their ability to explain and predict phenomena. Constructing research designs that yield valid and reliable information, therefore, is more than a matter of methodological mechanics. From this perspective, data collected without a theory has the status of "information" and is limited to describing phenomena, while data collection guided by theory produces what can be called "knowledge."
Thus, specifying the underlying theory of change of an initiative not only provides practical guidance about what to measure, how, and when, but is necessary if the field of community change is to progress. Having said this, it is also true that the field is, in many respects, so young that theory specification is as much an art as a science. This fact is at the center of the difficulty of designing the high-quality research that can best serve all CCI constituencies. The theories underlying these initiatives are often developed and elaborated while the initiatives are underway, and research teams are often integral participants, and sometimes even catalysts, in the process of specifying those theories.
Measuring Effects, Progress, and Context in CCIs
Evaluations of CCIs are intended to (1) assess the effectiveness of the initiatives in achieving the outcomes as specified; (2) provide information for monitoring and reporting on the progress made by initiatives in implementing the activities that are expected to catalyze and sustain change in a community; and (3) develop a knowledge base about the circumstances that facilitate or hamper positive or negative outcomes and the linkages specified in the initiative’s theory of change.
In order to meet the first purpose, the research design needs to address the question of whether the CCI activities ultimately have the intended effects that are sought, over the long term as well as during interim periods. To meet the second purpose, a research design needs to address the question of whether progress is being made in catalyzing and sustaining change in a community, and if so, how. This entails designing a strategy for measuring what activities a CCI generates in a community, how it attempts to do so, and the outcomes of those activities. The knowledge development goal requires both of the above, as well as an examination of the context in which a CCI is undertaken. This requires that evaluators develop strategies for measuring the conditions under which change is attempted and the influence of those conditions on the implementation and effects of the CCI. The accompanying figure provides a graphic representation of a general measurement model that can serve these multiple purposes.
This section treats the three issues of measuring effects, progress, and context individually, describing the key dilemmas associated with each. The intention is not to resolve these dilemmas, since specific design decisions need to be made within the context of each initiative, but to elaborate on the issues that must be considered to produce a multi-purpose research design. The discussion notes how a theory of change can be one mechanism to guide evaluators through choices about the focus and timing of measurement.
Figure. A Measurement Model for CCI Evaluations
Historical conditions and ongoing dynamics
Question: Under what conditions is the initiative operating?
(For example, history, geography, economics, politics, population, or relationships)
Activities and interim outcomes
Question: How are activities catalyzed and implemented?
(For example, involvement of individuals or groups, decision making, roles, facilitators, or obstacles)
Question: What CCI activities are undertaken?
(For example, mobilizing people or organizations, new or restructured activities, or new or restructured resources)
Question: What early and intermediate outcomes result?
(For example, meeting predetermined thresholds for participation, civic activities, or events)
Question: What long-term outcomes result from CCI activities?
(For example, change at the level of individuals, family, organizations, institutions, or community)
Measuring the Long-Term Effects of CCIs
Specifying the expected long-term outcomes of a CCI effort in terms of measurable variables is in many ways the most traditional and straightforward aspect of developing a CCI evaluation plan. Whether the long-term goals are set by the designers of an initiative, local implementers, or community residents, they are usually specified earlier, with more clarity and consensus, and in more concrete terms than other elements of the theory. Nevertheless, the scope of a typical CCI—in both number and nature of effects sought—constitutes a major challenge for researchers. As with discrete social programs, the ultimate success or failure of these efforts will be judged on the basis of long-term, measurable outcomes. But developing a fair and interpretable strategy for assessing change in CCIs brings to the forefront several issues that differ from those of more traditional program evaluation.
What and When to Measure
A hallmark of CCIs is their intention to improve the overall quality of life in communities by effecting change across multiple "strands," such as housing quality and affordability, economic opportunities, civic pride and social cohesion, service quality, family health, and youth development. The first challenge for a CCI evaluation is simply to define and assemble a catalog of reliable measures for the long-term outcomes that stakeholders have defined as important.
A complicating issue in defining and measuring long-term outcomes is the expectation that CCIs will spark "synergy" across these multiple strands. Thus, change in one strand can have spillover effects or may even be a necessary condition for change in another. Therefore, relationships among long-term outcome areas need to be specified in advance as part of the theory of change in order to define precisely what should be measured, when it is most sensible to assess effects in a particular area, and a framework for interpreting findings. For example, an initiative may target housing quality and affordability, stability of neighborhood residence, and neighborhood cohesion as long-term outcomes for improvement. At the same time, the underlying theory of change may be that improved housing quality and affordability are expected to change migration rates and patterns within the target neighborhood, which in turn will increase the level of neighborhood cohesion. Therefore, under a highly specified and sequenced theory of change, the housing outcomes should be assessed first, with migration and cohesion outcomes evaluated only after the housing effects are evident.
A further issue in defining the measurement of long-term outcomes is the common CCI tactic of implementing activities in stages throughout a geographic area. An accurate assessment should recognize that some activities are focused initially on an area smaller than the "target community" as a whole. Continuing the previous example, a CCI might select a specific area of ten square blocks to redevelop first, and then move on to another area. This would have clear implications for how the numerator and denominator should be specified for assessing changes in housing stock, migration rates, and social cohesion at any particular point in time. The denominator for assessing changes in housing stock would be the number of dwellings in the smaller area, not the full target community. At baseline and at the earliest follow-up point, the numerator would consist of the number of substandard (or high priced) dwellings in the ten-block area. The need to calculate such effects at particular points in time has clear implications for how statistical, survey, and other types of data are collected from the start.
The complexity of CCIs also raises issues about what is considered significant change. Although the challenge of establishing causality is examined elsewhere in this volume, it bears noting here that assessing and interpreting the effectiveness of CCIs can be problematic because the thresholds that would define meaningful improvement are unknown or unspecified. There may be "tipping points" that must be reached before significant change can be expected in certain areas. Without a body of empirically based, practical knowledge about, for example, the degree of improvement in housing stock needed before change in migration rates becomes evident, it will be difficult to interpret the effectiveness of CCIs.
How to Measure
The sheer number and variety of long-term effects sought by CCIs present significant measurement challenges. Assembling the needed measurement "tools" and collecting the breadth of data can be an overwhelming task, especially in communities that have not been the subject of previous community-level research. The methodological issues surrounding the construction of small area data sets from administrative records, conducting neighborhood-level surveys, creating organizational capacity for record keeping, and other techniques for assessing neighborhood change are, fortunately, receiving increasing attention from researchers and evaluators (Sommer et al., 1996; Urban Institute, 1996). In addition, there is an effort underway to catalogue measures that are currently being applied to or developed and tested in CCIs and other community-based programs.1 As progress is made in both these areas, the design and implementation of CCI evaluations will become increasingly efficient.
A significant challenge arises from the fact that most evaluations measure effects, even community effects, at the individual level. Aggregation of individual-level information to describe communities is more easily accomplished in some domains than in others. For example, Coulton, Korbin, and Su (1996) found that aggregating perceptions of individual residents yielded promising levels of reliability in some key areas of interest to CCIs, such as facility availability, usage, and quality and block club activity. Other community-level outcomes, such as neighborhood social interactions, economic opportunity structures, and institutional coordination, require different measurement strategies that take significant time and resources to develop.
Finally, because most CCI neighborhoods have not been the subject of previous research, many measures that are used have been adapted from other research settings. One particular problem is that the populations of CCI neighborhoods are largely minority, and few existing measures have been developed with particular attention to their validity when applied to individuals of minority racial, ethnic, and cultural backgrounds. While there have been some efforts to test the reliability of common measures for minority groups (Jones, 1996), it is difficult to know in advance the validity of such measures in CCI settings.
Tracking Progress and Early Results
What and When to Measure
As with measuring the long-term effects of a CCI, measuring more immediate progress raises a set of issues and challenges that are somewhat more difficult than those encountered in traditional evaluation research. As long-term investments, CCIs are not expected to achieve their ultimate goals quickly, sometimes charting a course toward change over five, ten, or more years. Yet even early implementation steps have observable effects. The need for initiatives themselves to have access to information for formative feedback and the need for research in this field to contribute to knowledge development are both best served by specifying and measuring not only long-term outcomes, but also the early and interim effects that lead to them. This makes it critical that evaluation designs include mechanisms for assessing the progress these initiatives are making years before long-term effectiveness can be assessed.
On one hand, having a theory of change helps to move the initiative and its evaluators beyond what are traditionally called implementation data—that is, descriptive information about the intervention and how is it implemented—by tying such information directly to outcomes of interest. On the other hand, this careful specification and assessment of early, interim, and long-term outcomes, the linkages among them, and the strategies to be used closes off some traditional evaluation shortcuts to CCI evaluators. In more circumscribed program evaluations, aspects of the research design may be minimized or even excluded to conserve resources without causing significant loss to the utility of the study: this might be possible, for example, if the program model is predetermined and unchanging, if options for how it is to be implemented are relatively constrained, or if ultimate outcomes—the effects on individual participants—are available to be assessed in a relatively short time frame.
By contrast, measuring progress in CCIs requires that linkages be specified among the strategies and approaches a CCI undertakes, the activities it implements, and the early, interim, and longer-term outcomes it is expected to achieve, and that all these linkages be measured with equal intensity. No short-cuts are immediately apparent. For example, because CCIs are meant to be community-driven efforts to produce change, the questions surrounding how governing entities are formed and decisions are made need to be monitored throughout the life of the initiative. In turn, the degree and quality of implementation of activities should be specified and measured, and their early outcomes must be monitored. Thus, understanding and assessing progress toward ultimate effects require that three components—how a CCI is undertaken, what activities it implements, and what and when early and intermediate outcomes are achieved—be measured. A predictable feature of CCI evaluation is that this stage of the research design process is sure to be complex.
Perhaps the greatest obstacle to measuring activities and their intermediate outcomes is the dearth of well-developed theory or systematic data about the crucial elements for effective community mobilization, planning, and governance and how those aspects of a community initiative, in turn, lead to effective strategies for producing short-term and long-term outcomes. As a result, researchers must devote a great deal of effort to working with CCI operators to specify what activities will be undertaken in a community, why they have been chosen, and what indicators will mark interim progress or success. As with long-term outcomes, determining the threshold of activity required to meet goals is problematic, given the absence of existing systematic research to inform this process. Nevertheless, this time-consuming activity is a necessary step if benchmarks included in the research design are to carry weight with all the consumers of the evaluation, from program managers to funders to community residents. Progress is being made in the CCI context in conducting this type of research, and new knowledge will help the design and implementation of future evaluations to become more efficient.
Determining when progress should be measured can also be difficult. Depending on the outcome being studied, some data must be collected on a continuous basis, while others can be collected periodically. As new strategies are chosen and new activities planned over the life of the initiative, documenting baseline conditions in the target neighborhood can be an ongoing requirement of the research design.
How to Measure
To operationalize a plan for measuring progress in early and intermediate outcomes, CCI evaluators must first contend with the volume and variety of outcomes requiring documentation. Because there are so few existing, reliable measures of early and intermediate activities and outcomes, CCI researchers are usually left with the task of developing those measures as an initiative proceeds. Further, much of the data must be collected on a continuous basis, which means that CCI participants, on-site researchers, or both may be needed as partners in carrying out the research design.
For example, to collect information on meeting attendance, organizational participation rates, civic activities, and informal neighborhood events, evaluators might depend on strategies that can be incorporated into the CCI implementation or observations made as events occur. Working with community members to develop and incorporate data collection activities and recruiting and managing local data collectors are challenges that require ongoing attention and substantial resources.
Measuring the context—social, economic, political, cultural, institutional, and so on—in which a CCI takes place raises issues that differ in degree but not kind from those previously discussed. Here again, theory and research to support the work are scarce, and therefore the researcher needs to apply the same degree of creativity in measurement design and breadth of measurement strategies that were required in measuring effects and progress. Once again, if the evaluation findings are to be interpreted and fully understood, linkages must be specified, in this case, among contextual factors, implementation strategies and activities, and their early, intermediate, and long-term outcomes.
What and When to Measure
Identifying what components of context are important to a CCI and deciding how and when to measure them is complicated, but necessary, because of the dynamic nature of communities and the interventions. Each community has a unique history, and ongoing dynamics are influenced by internal and external factors. These dynamics, in turn, affect the shape and progress of a CCI as it seeks to achieve its desired outcomes. Thus, even as a CCI is attempting to influence factors within its purview, the initiative is subject to the influence of factors over which it has little or no control. Understanding and comparing the circumstances under which CCIs can be effective requires that any theory of change take into account at least three types of contextual factors to be identified and measured as part of a research design: historical conditions, ongoing dynamics, and critical events.
In setting a CCI underway, a community starts from a particular point in its history, which forms the context into which change is introduced. The physical geography, local economy, political history, population, and other factors in both the target community and its broader surroundings (such as the city) need to be documented historically and at the time the initiative begins. It is also important to understand the community’s trajectory: for example, although two communities may begin with similar poverty rates, one may be in the midst of a ten-year economic decline while the other has experienced economic growth over the last three years. Understanding how, when, and why a CCI works necessitates linking these conditions to the rate of progress seen in establishing activities and producing interim and long-term outcomes.
The ongoing dynamics of a CCI community are also important to measure and link with the other parts of the research model. These conditions include surrounding social, political, and economic factors—such as population dynamics, racial relations, or political activism—that are not directly targeted for change but can influence the extent to which the CCI activities are implemented and the strength of their results. For example, an initiative may not be targeting migration into and out of the community, but a growing foreign immigrant population or an out-migration of families with school-age children can have a significant impact on the CCI.
Clearly, these factors need to measured and linked with other research findings about progress and effectiveness. Even so, it is important to ensure that dynamics directly targeted by the CCI are treated as outcomes, and that only those not directly targeted are treated as contextual factors. These factors should be measured on a continuous basis over the course of a CCI.
A third group of contextual factors to be measured can be thought of as critical events; that is, discrete, powerful events that can directly or indirectly affect the progress of a CCI by changing the ongoing dynamics in a community. These factors include a broad range of possible events, such as the award of significant new grants (for example, through designation as an Empowerment Zone), natural disasters (such as floods or earthquakes), political change (such as an election, new legislation, or a school closing), or social relations (such as civil unrest or police conflict). It is important to document the occurrence of such events and trace those effects that can have an impact on CCI implementation and outcomes.
How to Measure
Measuring community context requires the full range of techniques at researchers’ disposal. Social factors such as population, economic history, and political participation require the creative use of statistical data for the target area. Other contextual factors, such as outbreaks of civil unrest, shifts in political climate or racial relations, and existing attitudes toward key organizations or institutions, require careful qualitative data collection and analysis. These might be documented by site-based researchers or through key informants interviewed on a regular basis by a research team. In either case, it is important to ensure that sufficient financial and human resources are dedicated to this type of data collection or the understanding of a CCI’s progress and effectiveness will be incomplete. Indeed, as noted by O’Connor (1995), understanding the contextual conditions under which CCIs are relatively more or less effective is one of the most important research and policy questions to answer.
Discussion and Recommendations
Conducting high-quality research on CCIs is critical if knowledge development is to occur, but doing it well is difficult and expensive. Whether one is inclined to devote resources to theory building and measurement development or to developing other techniques (such as random assignment within a community context or constructed comparison groups for attributing causality), intensive human and dollar investments are required. The complex nature of these initiatives demands that significant resources be devoted to the research design effort, both before and during implementation of the initiative. A few strategies and practices have the potential for moving the field forward and ultimately making evaluation research on CCIs more efficient.
Develop new measures. The demands of CCI evaluation research are extensive and diverse, largely by virtue of the sheer volume and newness of the needed measures. Although some measure development can and should be done within the context of specific initiatives, efforts specifically aimed at constructing and validating new tools or developing models of threshold points could yield significant benefits. Better tools and models could reduce CCIs’ reliance on expensive community surveys, which to date have been used to collect much of the data during all phases of measurement.
Increase the availability of existing measures. CCI initiatives and evaluators should attempt to publish their evaluation designs and measures as early as possible. Although this is not standard practice in program evaluations (or is done only in conjunction with a final report), the CCI field needs to take a more active approach to sharing information if the field is to grow in the next five years, while initiatives are on their way toward measuring long-term outcomes and producing final reports.
Support the specification of underlying theories. Given that theory specification is a critical element in evaluation, funders could help by building in incentives for CCI stakeholders to specify their theories of change early and reassess those theories during the course of an initiative. Related incentives could encourage evaluators to incorporate a theory of change approach into their evaluation designs.
Undertake basic research on community change. One of the reasons that evaluators and others who are undertaking research on CCIs must spend so much time developing new methods and measures is because the "science" of community research is underdeveloped. The process of "natural" community change is poorly understood, and little investment has been made in the tools for undertaking such research. A longitudinal panel study of communities would provide a rich source of basic information about the processes and contextual factors associated with natural and planned change. It would also provide the context for development of new research methods.
Encourage the development of small area data in existing data collection efforts. Standard censuses and surveys still focus on individuals, giving little attention to the neighborhoods and community contexts in which they reside. The federal government is considering adding questions about small areas in the census to be conducted in the year 2000. Such information would greatly enhance both design and implementation of community-based initiatives.
- The Roundtable is developing an annotated catalogue of outcome measures, drawn from current and past evaluations of CCIs, other complex initiatives, and basic research on communities.
Coulton, Claudia, Jill Korbin, and Marilyn Su. 1996. "Measuring Neighborhood Context for Young Children in an Urban Area." American Journal of Community Psychology 24:5-32.
Jones, Reginald, ed. 1996. Handbook of Tests and Measurements for Black Populations. Hampton, VA: Cobb and Henry.
O’Connor, Alice. 1995. "Evaluating Comprehensive Community Initiatives: A View from History." In New Approaches to Evaluating Community Initiatives Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Sommer, T. Erich, et al. 1996. The Creation of a Community Information Infrastructure. Chicago: Chapin Hall Center for Children.
Urban Institute. 1996. Democratizing Information: The First Year Report of the National Neighborhood Indicators Project. Washington, DC.
Weiss, Carol Hirschon. 1995. "Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for Comprehensive Community Initiatives for Children and Families." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Back to New Approaches to Evaluating Community Initiatives index.
Copyright © 1999 by The Aspen Institute
Comments, questions or suggestions? E-mail firstname.lastname@example.org.
This page designed, hosted, and maintained by Change Communications.