Theory, Measurement, and Analysis
Using a Theory of Change Approach in a National Evaluation of Family Support Programs:
Sharon L. Kagan
Family support programs have emerged as a comparatively recent phenomenon, gaining operational currency in the last two decades (Weissbourd, 1987). Designed originally as community-based entities, these programs defined themselves as antidotes to, and mediators of, more formal and highly bureaucratized social services. Rather than rigidly adhering to structured rules and regulations, family support programs sought to be flexible, contouring their activities to respond to parent and family needs. Rather than meeting the needs of one family member, family support programs sought to meet the needs of entire families and communities. Rather than focusing on interventions once problems were identified, family support programs sought not only to prevent the onset of problems, but to optimize human development and capacity. Rather than relying primarily on the expertise of professionals, family support programs sought to recognize and maximize the talents and competencies of involved families. In short, family support programs were established to be "of," "by," and "for" the families and communities in which they were enmeshed.
Not surprisingly, these family support programs raised considerable challenges for evaluators. Small, highly idiosyncratic, and designed to modify their activities continually, family support programs did not offer large populations or uniform and stable treatments to evaluate. As programs that serve all members of the family, interventions are multifocused and diffuse, often changing with family needs. As programs that are, by design, voluntary and often serve highly transient populations, they do not afford optimum settings for randomly assigning participants or for sustaining durable participant involvement. Indeed, random assignment to non-treatment groups violates a fundamental principle of family support—notably that all who need and want services are eligible for them. Moreover, identifying valid comparison groups—so necessary for quasi-experimental design—is difficult, as the environments in which family support programs exist are often contaminated by myriad other interventions. Indeed, researchers contend for good reason that using conventional approaches to evaluate family support programs is extremely problematic (Powell, 1989, 1994; Weiss and Jacobs, 1988).
Despite these caveats, the growth of family support programs and the press for public support to fund them has created a demand for a more systematic understanding of their efficacy. Do these efforts really work? For whom? Under what conditions? How do and can they mesh with conventional services? What are the factors that help them to effect community engagement and community change? How do we know that these programs are worth the investment? Indeed, policymakers and analysts alike have asked whether the family support "emperor" really has new clothes.
A National Evaluation of Family Support Programs
To seek answers to such questions, and in conjunction with the passage of the Family Preservation and Support Services Program (PL 103-66), a national evaluation of family support was authorized for the five-year period from 1996 to 2001. Conducted by Abt Associates and Yale University, the National Evaluation of Family Support Programs was intended to "document, describe and assess the relative impact and effectiveness of family support programs funded under Title IV-B of the Social Security Act Subpart 2, and any other Federal or non-Federal programs that are designed to achieve the same purposes." Inherent in such a charge are three allied goals:
After reviewing the evaluations of many family support programs (Barnes, Goodson, and Layzer, 1995) and creating a typology to categorize them (Kagan, Cohen, Hailey, Pritchard, and Colen, 1995), it became clear that conducting a national evaluation of so diffuse a treatment demanded innovation. Standard random assignment would not work, nor would the use of conventional evaluation strategies. In searching for an approach that would lend fidelity to the process while also respecting the variation inherent in family support, a "theory of change" strategy was sought.
- to provide a state-of-the art knowledge base on family support by gathering and synthesizing existing descriptive information on current family support programmatic and research efforts
- to enhance that knowledge base by designing a research/evaluation strategy to fill important gaps in understanding of the effects of family support
- to synthesize and analyze data in order to advance understanding of the effectiveness of different approaches to family support
On first review, the theory of change approach seemed ideally suited to an evaluation of family support because it sought to discern "how and why an initiative works" (Weiss, 1995) by making pathways of change explicit. Moreover, and consonant with family support, it includes key participants in the design process and gives voice to those participants’ views in ways that conventional evaluation approaches do not. In so doing, it also realigns the balance of power between researchers and practitioners, according weight to each.
The theory of change process begins with program staff identifying the outcomes they hope their programs will achieve. Participants—with evaluators—then discern the nature of activities offered by the initiative, linking them through interim accomplishments to ultimate goals. Through a series of iterative discussions, the lifeline of the program is plotted so that the paths from activities to outcomes are graphically portrayed to form a map of the program’s theory of change. Inherent in the process of creating the map, participants’ ideas about how change occurs are manifest. Clarity is often achieved regarding the precise nature of the goals to be achieved, as well as regarding which frameworks will guide the evaluation.
The rationale for using the theory of change approach in the National Evaluation of Family Support Programs had several facets. First, the approach is aligned with family support in that it captures the non-linearity and iterative nature of the programs. Rather than expecting defined and static interventions, the theory of change approach welcomes the diversity and changeable nature of interventions. It regards critical contextual factors not as theoretical nuisances, but as important variables to be understood and examined. In respecting the role of context, the approach appreciates, rather than tolerates, variation within and across sites—a critical element in family support programs. The approach is responsive to interventions that are multi-, not uni-, dimensional—precisely the kinds of interventions that are found in most family support programs.
But perhaps the most appealing aspect of the theory of change approach is that it is as helpful to program personnel as it is to evaluators. By demanding precise thinking about the purpose and outcomes of various activities, the process of building a theory of change forces program staff to examine their own assumptions and beliefs about what works, for whom, and under what conditions. With the evaluators and participants present, activities that are and are not likely to yield the desired outcomes can be identified and the program refined accordingly. In this respect, the process is valued by providers.
From the perspective of the evaluators, the approach evokes both a new and valuable role and a new set of responsibilities. Evaluators become collaborators with program personnel; the hierarchical relationship that often elevates the evaluators’ status is minimized. Indeed, the theory of change process engages evaluators as co-explicators in reaching consensus regarding the language attached to the program’s intentions. For the most part, theory of change evaluators are not the shapers of intentions, but thoughtful allies in explicating them. Their role is less to pass judgment on the validity of the theory than to identify it.
It should be noted, however, that the collaborative relationship developed between program personnel and evaluators—and the way theory of change information is used—may vary significantly, framed by the interests and experiences of the evaluators and program participants, the stated purpose of the exchange, and the maturity and nature of the programs involved. If, for example, the evaluators are using the process to help distinguish and select among programs for further evaluation, then the potential for program improvement may be limited, as might the duration of the involvement of the evaluator. If, however, the purpose of conducting the theory of change process is to engage the evaluator in a more sustained or consultative relationship, then the programmatic implications of the process might be more intense.
Allied with these issues are considerations regarding whether the evaluation is more formative or summative. It may be that in more formative evaluations the role of the evaluator has greater programmatic impact than in more summative evaluations. Moreover, more mature programs may be less willing or able to alter their efforts, while those at earlier stages of development may have more flexibility, and hence may be better able to use the results for programmatic purposes. The point is that the dynamic theory of change process can be adapted for a variety of evaluative and programmatic purposes.
The Theory of Change Approach: The Process
Once it was determined that the theory of change process made sense for the evaluation, we clarified our intention to use it to improve our understanding of the 16 programs that were candidates for the final evaluation. Clearly, the theory of change process could not be the only criterion used to determine a program’s appropriateness as a subject for evaluation, but we decided that it would be helpful in discerning which programs would best qualify for first-round inclusion in the study.
Then we embarked on the process: site visits were made to the candidate programs by teams of workers, typically a researcher and a program specialist. Prior to visiting each site, written materials were extensively reviewed by the site visitors. On site, the visitors held entry interviews with the directors and key staff, followed typically by program observations. Initial meetings were held to discuss the theory of change process with key staff; and then at least two and sometimes as many as five follow-up sessions were held to come up with a theory of change map for each program.
To assure the fidelity and accuracy of the map, the evaluators finalized the map and sent it back to the program for approval or refinement, if necessary. For each program visited, an individual map portraying the program’s theory of change was created. Similarities and differences among programs were discerned, as were similarities and differences in the outcomes desired. Ultimately, a "generic" family support theory of change was created, and from it, "generic" instruments were identified. In addition, individual instruments were also selected, tailored to site-specific needs.
The theory of change process, then, served to illuminate what was to be evaluated; it did not provide specific answers regarding how to conduct the evaluation. Having said that, however, using a theory of change approach—because it uncovers such richness of program detail—cannot help but underscore the necessity of getting beneath the numbers to reveal the nuances of a program’s impact. As such, the theories of change process points toward increased use of qualitative information, typically in combination with more quantitative data.
In discerning the "what" of the evaluation, the process was also extremely useful in guiding the thinking and collaborative process between evaluators and program personnel that was to ensue as the precise methodology of the evaluation unfolded. The process was highly regarded by both program staff and evaluators. Program personnel—including many who were not selected to participate in the first round of the evaluation—indicated that the work had been beneficial to their long-term planning. Although many expressed appreciation for the maps as blueprints for action, they were even more grateful for the opportunity to learn a valuable process for strategic planning. From the perspective of the evaluators, the theory of change maps provided an informative operational overview of the family support field.
Although it is still too early to comment on the actual impact of the theory of change process on the outcome of the evaluation, it may be helpful to speculate on several points. First, it appears that the process heightens the awareness and intentionality of program efforts. Because efforts are more closely aligned with the ends that are being sought, program results may increase. Second, for programs that are designed to have flexible interventions—including family support programs—the theory of change approach may need constant updating. As such, evaluators will need to consider how much adaptiveness they can incorporate into their evaluation methods and tools, as programs change and become far more explicit about those changes. Third, the process beckons evaluators to examine how inventive evaluation methodologies can be developed and/or used. The process strongly reaffirms the challenge of using conventional methods for unconventional programs and services.
As productive as the theory of change approach has proven to be, its innovative nature raises many important, but unresolved, theoretical and practical issues. Dialogue on these knotty issues may help evaluators refine the process based on experience and thoughtful consideration. Several theoretical issues deserve focused attention:
At the same time, several key practical questions demand further discussion:
- Is there only one theory of change per program or do many theories coexist? What are the benefits and liabilities of each?
- Do theory of change models change over the lifespan of a particular program?
- Does the approach work better with some kinds of initiatives than with others (for example, those that are more circumscribed and less global)?
- What constitutes the unit of analysis for the theory of change approach? Is it the whole comprehensive program or its sub-parts? Who decides, and by what process?
- What is the appropriate level of detail for the outcomes? How specific should the theory be?
Using the theories of change approach has been rewarding because it fits so well with the intentions of family support. It has served program and evaluation personnel by establishing the basis for egalitarian relationships, providing conceptual cogency to an array of efforts, and, perhaps most important, creating a viable template for mounting an evaluation of extremely complex phenomena.
- How do we capture the synergy among the efforts described in the map?
- How much should the evaluator know about the program before embarking on the theory of change process?
- What kind of training, experience, and knowledge do evaluators need to do their work successfully?
- What is the optimal nature of the relationship between evaluators and program staff? Does too close a relationship limit objectivity? Does too little impair understanding?
- When interviewing groups about their theories of change, is it preferable to have cross-sections of program participants or to have role-alike groups (for example, parents, staff, or board members)? If role-alike groups, is there a preferred order?
Barnes, Helen V., Barbara D. Goodson, and Jean I. Layzer. 1995. Review of the Research on Supportive Interventions for Children and Families. Volume 1. Cambridge, MA: Abt Associates.
Kagan, Sharon L., Nancy Cohen, Linda Hailey, Eliza Pritchard, and Hope Colen. 1995. Toward a New Understanding of Family Support: A Review of Programs and a Suggested Typology. Cambridge, MA: Abt Associates.
Powell, Douglas R. 1994. "Evaluating Family Support Programs: Are We Making Progress?" In Putting Families First, eds. Sharon L. Kagan and Bernice Weissbourd. San Francisco: Jossey-Bass.
Powell, Douglas R. 1989. Families and Early Childhood Programs. Washington, DC: National Association for the Education of Young Children.
Weiss, Carol Hirschon. 1995. "Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for Comprehensive Community Initiatives for Children and Families." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Weiss, Heather B., and Francine Jacobs. 1988. Evaluating Family Programs. Hawthorne, NY: Aldine.
Weissbourd, Bernice. 1987. "A Brief History of Family Support Programs." In America’s Family Support Programs, ed. Sharon L. Kagan, Douglas Powell, Bernice Weissbourd, and Edward Zigler. New Haven: Yale University Press.
Back to New Approaches to Evaluating Community Initiatives index.
Copyright © 1999 by The Aspen Institute
Comments, questions or suggestions? E-mail firstname.lastname@example.org.
This page designed, hosted, and maintained by Change Communications.