Concepts, Methods, and Contexts
Evaluating Comprehensive Community Initiatives: A View From History
Although both community-based initiatives1 and evaluation research have been prominent in the fight against poverty since the 1960s, they have taken distinctly divergent paths over the past three decades. Evaluation research has grown steadily in prestige since the 1960s, its results featured prominently in debates over the Family Support Act of 1988 (Wiseman et al. 1991; Manski and Garfinkel 1992). The field built its reputation for scientific objectivity and policy relevance on experimental design and has been preoccupied with its requisites: finite, measurable program goals; discernible program components; the ability to control for internal and contextual contingencies; and generalizability across locality. Community-based initiatives, in contrast, have been largely abandoned by the federal government, left to rely on foundations and other private sources for their chief means of support. If anything, these initiatives have increased their emphasis on the "intangibles" of community building such as strengthened social bonds, their conviction that the whole of the intervention is more than the sum of its parts, and their determination to become immersed in the needs and strengths unique to their communities. While groups such as the Roundtable continue to search for appropriate evaluation methodologies, a deep-seated skepticism persists in policy circles about the efficacy of community-based initiatives.
This paper explores the roots of this impasse--and, perhaps, some ways out of it--in an examination of past experience in evaluating a variety of community-based initiatives designed to combat poverty. It starts out by discussing the historical relationship between community-based initiatives, social scientific knowledge, and policy, and then reviews past evaluation strategies to examine what has been learned, both about the capacity of community-based initiatives to combat poverty and about the institutional and political capacity to learn from their experience. This review indicates that past evaluation efforts have yielded important findings and insights about community-based initiatives. At the same time, it serves as a reminder that evaluation is conducted within a political context, in which decisions are driven more by values, political interests, and ideology than they are by scientific evidence (Aaron 1978). Moreover, the practice of evaluation is itself a profoundly political and value-laden process, involving judgments about the validity of program objectives and choices about how progress can be measured (Weiss 1993). Recognizing the political nature of evaluation does not mean that evaluators cannot come up with valid assessments of program effectiveness. However, it does suggest that evaluators need to be aware of the context within which they are conducting their work, the function that evaluation can and should play in programs and policies, and the values and assumptions that they themselves bring to the evaluative enterprise. Finally, an analysis of the historical record points to the questions that have not been asked as well as those that have been asked in past evaluation efforts, and points to issues that will need to be addressed in efforts to create more effective evaluative paradigms for community-based initiatives.
Community Initiatives, Knowledge and Policy
Community-based initiatives and evaluation research can be seen as two different strands of an approach to social change that relies on the purposeful application of knowledge--whether knowledge takes the form of experience, scientific investigation, or both--as a strategy for social improvement. Long before evaluation research existed as a field within the social sciences, community reformers regarded their efforts as social experiments, part of the process of accumulating knowledge about the nature of social problems and their solution (Bremner 1956; Carson 1990; Davis 1984). Similarly, in its earliest years American social science was motivated by the cause of social reform (Haskell 1977). Community-based interventions and evaluation research have also been shaped by similar historical forces and processes, three of which have been especially important in their evolution as separate fields: professionalization, which has contributed to increased specialization in the production of knowledge on the one hand and in the design and implementation of social welfare programs on the other; the growth of the welfare state and the new demands for knowledge it created; and changes in the social, economic, and political context within which both community-based interventions and evaluation have been developed. The shaping influence of these historical forces can be seen in a brief overview of community-based social change efforts, and their changing relationship to knowledge and policy, since the late nineteenth and early twentieth centuries.
Comprehensive community intervention has been used as a strategy for responding to a variety of social problems since the Progressive Era, and its history encompasses such diverse efforts as the settlement house movement, neighborhood-based projects to combat juvenile delinquency in the 1930s and again in the 1950s, the Community Action and Model Cities programs during the War on Poverty, and the wide range of community-based economic development projects, including community development corporations (CDCs), that have emerged in recent decades (Halpern 1994). What links these diverse initiatives together is a set of assumptions and operating principles that have proved remarkably stable in the face of changing circumstances. These include:
Applied in various ways, primarily in low-income urban settings, these principles and assumptions have translated into an approach to social change that seeks to be comprehensive, firmly rooted in the community, participatory, and informed by accumulated knowledge, either in the form of ongoing research or experience (Davis 1984; Trolander 1987; Carson 1990; Bremner 1956; Halpern 1994). And yet, this characterization perhaps suggests more coherence in action principles than actually was there. In fact, community-based initiatives have historically faced persistent dilemmas--balancing individual with social change objectives, revitalizing poor communities while increasing outward mobility for residents, combining the requisites of social experimentation with immediate community priorities--that have caused some ambiguity in defining objectives and designing programs.
- An analysis of social problems that emphasizes their environmental origins, their complexity, and their interrelatedness: Over the course of the twentieth century, environmentalist explanations have pointed to a number of external factors to explain social inequality and behavioral "pathology," some more difficult to measure than others. Thus, economic and social conditions, neighborhood environment, family structure and processes, and group culture have all been identified as explanatory factors. In the face of the periodic resurgence of eugenic and other biologically based explanations for inequality, those who adhere to environmentalist explanations have insisted that individual and group differences stem from outside forces and hence are susceptible to intervention.
- A recognition of the importance of the geographically bounded area, whether rural small town or urban neighborhood, as the basis of communal social bonds, as a manageable landscape for achieving social reform, as a laboratory for social experimentation, and as a legitimate unit for social scientific analysis: At times these functions have overlapped or co-existed in tension with one another within the ideological framework of community-based social reform.
- An emphasis on institutions as key leverage points for stimulating change, both in individuals and in society at large: The institutional approach has taken two main forms in community-based reform movements. One, exemplified by the settlements and by community action agencies in the 1960s, embraces institutional innovation as a goal, and often establishes new or "parallel" institutions in order to stimulate innovation. The other works within existing institutions, often the school, to reform or coordinate institutional practices. The emphasis on institutions stems in part from the need to steer a path between individual and social change goals, from the conviction that existing institutions are not set up to address local needs, and from a desire to make lasting changes in the community.
- A faith in knowledge as the basis of planning, public education, and learning for the sake of social betterment: This, too, has been expressed in many different ways in community-based initiatives, with some quite consciously basing their programs on social science theory and evaluation, and others relying on experience.
- A belief that the isolation of the poor is a key factor in causing and perpetuating poverty and social inequality: Concepts of social isolation have expressed themselves in many different ways in the history of community-based intervention--middle-class settlement workers or "neighbors" saw their presence in poor communities as a form of cultural uplift for the poor; initiatives based on organizing and empowerment strategies have sought to combat the political isolation of poor communities. In those and other forms, the concept of social isolation can be linked to a strategy that embraces the principle of local participation as a means of individual transformation, of transcending social differences, and of building essential resources from within the community.
Community-based initiatives have undergone important changes over the course of the twentieth century. In earlier decades, they were tied to local reform movements and often worked through nongovernmental institutions (NGOs). Beginning in the 1950s, community-based initiatives were linked more directly to efforts to reform urban governing structures, which were seen as too narrow and fragmented to deal with the complex of changes brought about by "metropolitanization." Reacting to the "bricks and mortar" and downtown business orientation of federal urban renewal policy, advocates of the approach known as "community action" called attention to the "human face" of redevelopment and insisted on a reorientation of the bureaucracy to respond to the needs of the inner-city poor. That led to a series of experiments designed to stimulate local government reform--often from a position outside of local governing structures--which were eventually absorbed into the War on Poverty (O'Connor, forthcoming). The 1970s marked a retreat from looking to reformed governing institutions as "change agents" and devolution to the states and localities for dealing with community needs. This period also featured a turn to market forces as instruments of reform, as government looked for ways to "privatize" service delivery and CDCs formed public/private alliances to create markets in ghetto neighborhoods. Those trends have continued into the 1990s.
The past three decades have also brought important changes in the role and production of knowledge related to community-based initiatives. One of the most important changes has been the emergence of national sponsorship for community-based reform, and with it the emergence of national audiences for the results of evaluation. Thus, from the Progressive Era through the 1920s the application of community-based social research was, for the most part, immediate and localized (although widely recognized reform-movement leaders such as Jane Addams did reach a national audience) (Carson 1990). Gradually, under the auspices of the emerging social work profession and New Deal initiatives, the use of social scientific theory to plan and evaluate local community-based programs for a national audience began to take hold (Trolander 1987). The Chicago Area Project, for example, combined social work practice with sociological theories of neighborhood ecology to train neighborhood workers and community organizers for juvenile delinquency prevention programs. Launched by Clifford Shaw of the Illinois Institute for Juvenile Justice and sociologist Ernest Burgess of the University of Chicago, the Project grew out of a decade's worth of "Chicago School" research on poor neighborhoods, which convinced Shaw that delinquency could be tied to a breakdown or weakness in neighborhood socializing institutions, especially the family, brought about by pressures from the industrial economy and immigration. Informed by this theoretical perspective, the Project targeted three neighborhoods, and focused its efforts on developing capacity in existing and newly established neighborhood institutions to respond to delinquency and provide alternative activities for gang-prone youth. In a practice that would later be replicated in anti-delinquency programs of the 1950s, the Project recruited "curbstone counsellors"--young male graduate students or former delinquents "gone straight"--to act as role models and mentors for local youth. Though not formally evaluated in terms of the outcomes it produced, the Project provided models for replication and was used as evidence to bolster the ecological theories that would inform the next generation of community-based initiatives (Halpern 1994). In this sense, the Project could be seen as an experiment in the application of social scientific theory to human problems, as well as in delinquency prevention practice. Some of the New Deal agencies, such as the Farm Security Administration, also sponsored community-based experiments that were grounded in social scientific theory (Gilbert 1994; Kirkendall 1966; Baldwin 1968).
With a growing emphasis on using community-based initiatives as demonstration experiments for national policies, the 1950s ushered in a transformative period in the use of research, placing greater importance on learning for the sake of widespread replication, and introducing new, less community-based standards for evaluating program success. Local demonstrations in juvenile delinquency prevention, housing, and mental health services were funded by the National Institute of Mental Health, the Children's Bureau, and other federal agencies to inform national policy initiatives. Major foundations also funded demonstration projects in areas such as education reform with an eye to learning for the sake of national policy (Lageman 1989). Reflecting the growing faith in scientific knowledge that characterized the immediate post-World War II years, social science became a more important component of these centrally orchestrated local demonstration projects: beyond informing the initiatives, social scientists would be part of the national planning team as program monitors, evaluators, and liaisons with the outside funders. At the same time, social science would be called upon to judge programs according to standards generated by the national funders: Did the initiative address a problem of national significance? Could it be replicated? Did it achieve the outcomes funders were reaching for? As evaluation came to be associated with "outside" priorities, local administrators became more skeptical about its value for their indigenous concerns.
Evaluation: Policy, Science, and Politics
In addition to changes in the nature, purposes, and sources of support for community-based initiatives, developments in policymaking, in the social sciences, and in the politics of evaluating social action programs were also important to the changing relationship between research and community-based programs. Beginning in the 1960s, the introduction of a new federal planning and budget system, the shaping of evaluation as a science, and the political use of evaluation in legislative battles over social welfare together created an environment that favored the development of quantitative experimental design techniques and discouraged sustained attention to theories and methods--including ethnographic, combined quantitative/qualitative, comparative case studies, and theory-based techniques--that might have contributed to better understanding of comprehensive community-based initiatives.
During the New Deal and continuing throughout the post-WWII years, federal government expanded into the unfamiliar terrain of social welfare, steadily escalating the demand for social scientific expertise while also making new demands on knowledge. Evaluation became part of program planning in several New Deal agencies, taking the form of program monitoring and, in a smaller number of cases, impact assessment using control groups (Deutscher n.d.). It was not until the 1960s, however, when the Johnson Administration mandated the adoption of the Planning-Programming-Budgeting System (PPBS) in all executive branch agencies, that the policy environment became ripe for more widespread use of the experimental, outcomes-oriented research that is most often associated with scientific evaluation. This bureaucracy-wide introduction of PPBS was an attempt to introduce rationality and long-range planning into the otherwise chaotic annual budget process. Following detailed instructions and strict timetables set by the Bureau of the Budget, each agency was to establish an internal process for "setting goals, defining objectives, and developing planned programs for achieving those objectives" as "integral parts of preparing and justifying a budget submission" (Bureau of the Budget 1965, 1). This system placed a premium on cost/benefit analysis of existing programs as well as on the range of alternative programs, on "objectives and planned accomplishments" expressed in "quantitative non-financial terms," and, above all, on an "output-oriented . . . program structure . . . which presents data on all of the operations and activities of the agency in categories which reflect the agency's end purposes or objectives" (Bureau of the Budget 1965, 2, 6; emphasis in original). Increasingly, this process would be handled at the agency level by separate divisions for research, planning, and program evaluation, staffed by social scientists or career bureaucrats who were one step removed from actual program administration; indeed, PPBS assumed "[t]he existence in each agency of an Analytic capability which carries out continuing in-depth analyses by permanent specialized staffs of the agency's objectives and its various programs to meet these objectives" (Bureau of the Budget 1965, 2; emphasis in original). Evaluation, once informal and often carried out by program staff or locally hired consultants if at all, would now be recognized as a separate function, to be undertaken by professionally trained social scientists using scientific methods. Moreover, PPBS demanded a particular kind of evaluation: focused on program outcomes rather than processes, stated in quantitative terms, analyzed in categories that corresponded with policy goals that were generated from the top.
Partly fueled by the expansion of government demand for evaluation, it was also in the 1960s that evaluation became more widely recognized as a distinct research field, informed by theoretical frameworks and, eventually, undergirded by a set of institutions devoted to evaluation training and practice (Rossi 1994). Reflecting a more generalized social scientific tendency to emulate the "hard" sciences, the social science research community responded to the growing policy demand for quantitative outcomes research by adapting the methods of controlled experimentation that had been developed over several decades and applied in engineering and psychological research after World War II, and that had been used in educational research beginning in the 1920s (Haveman 1987; Campbell and Stanley 1966). The idea behind the experimental approach was to approximate a laboratory setting as closely as possible, so that analysts could establish the cause and effect of particular interventions with some precision and with a measure of statistical validity. In the absence of ideal laboratory conditions, wherein researchers can manipulate most of the variables, evaluations of human interventions became an exercise in control: establishing comparable "control" groups to determine what would happen in the absence of intervention; "controlling" for contextual factors or natural processes not directly tied to the intervention; keeping careful control over sampling, measurement, testing, and other evaluative techniques to avoid the introduction of bias. More important than understanding the management and substantive program content of the interventions was the ability to identify a measurable set of inputs and a corresponding set of outputs. Such "black box" evaluations would have little to say about how and under what conditions programs were implemented, nor much about the configuration of components within the intervention. They were designed to answer, that is to say, exactly the questions that were assuming priority in the policy world of PPBS.
Of course, quantitative experimental design was not without critics among evaluation practitioners, and was by no means the only method used in the profession. Mirroring similar disputes in the social sciences, advocates of more qualitative approaches criticized quantitative researchers for their positivist assumptions and de-contextualized world view (Rossi 1994; Quane 1994). Even the most steadfast practitioners of experimental design became more realistic about its limitations, and began to develop more flexible, "quasi-experimental" techniques while also calling for theory-based and combined qualitative/quantitative approaches (Weiss 1972a; Cook and Campbell 1979; Quane 1994). Evaluators' experience in assessing the social action programs of the 1960s could only reinforce this tendency toward flexibility, by underscoring the mismatch between experimental theory and program reality: program goals were complicated and hard to measure; local administrators were often uncooperative; true experimental design was politically if not technically impossible; and the results were used out of context and for political purposes (Weiss 1972b; Rossi and Williams 1972). The findings from early social action program evaluation were no more encouraging: program impacts, if positive at all, were small when compared to program costs. These findings caused particular dilemmas for evaluators who, committed to program objectives, were unable to substantiate their convictions with the hard numbers policymakers were asking for (Weiss 1993).
If such doubts might have created an opportunity for developing more integrated methods for evaluating community-based social action programs, the political realities of evaluating Great Society programs quickly squelched it. In efforts to keep control over the new social programs, Congressional monitors, too, discovered the merits of cost/benefit accounting and quantitative outcomes-oriented evaluation. Soon after the War on Poverty was launched and with increasing regularity thereafter, legislators began to mandate regular evaluation--even specifying the use of control groups and experimental design as a condition of continued funding. Social scientists invested their intellectual capital in the kind of research that would have a pay-off in policy circles, developing quantitative and experimental techniques for use in negative income-tax experiments, employment and training research and, beginning in the early 1970s, welfare reform schemes designed to combine work and welfare. Both within and outside the profession, the political constituency for evaluation research was built around the large-scale, experimental approach, and power was ceded to the "quants" (Rossi 1994). With the help of foundations and federal contracts, a substantial research industry was created to carry out this type of research. As funding agencies retreated from community action, evaluators had little reason to develop and gain scientific legitimacy for qualitative, process-oriented, and theory-based methods that would be more appropriate for evaluating community-based initiatives.
Past Evaluation Efforts
Despite the divergent paths taken by research and community action, much of contemporary value can be learned from past experiences with evaluating community-based social programs. In addition to some important findings about the potential of local initiatives in the fight against poverty, a selective review of evaluations over the past three decades says a great deal about the function and politics of evaluation, underscoring the point that the job of the evaluator is complex and multi-layered. (See Prudence Brown's chapter in this volume.) And although evaluators have not exactly made great methodological or theoretical inroads in their approach to comprehensive, community-based programs over the past thirty years, the variety of approaches that have been used is indicative of the methods and critical issues that must be pursued in future efforts. In addition, a review of past experience highlights some crucial questions that have been raised but not pursued by evaluation--and must be if the current generation of reformers is going to learn about whether and how community initiatives can achieve what they set out to do.
Evaluation as "Social Learning": Gray Areas and Juvenile Delinquency
In the mid-1950s the Ford Foundation initiated a series of experiments in urban areas, the capstone of which was the "Gray Areas" program started in 1961. These experiments grew out of the Foundation's effort to respond to the "urban crisis" brought about by deindustrialization, white middle-class suburbanization, and the massive in-migration of poor minorities. They were variously designed to promote metropolitan governance and planning structures, create new community-based mechanisms for dealing with juvenile delinquency, increase the capacity of inner-city schools to educate "culturally deprived" newcomer populations, and improve employment and human services to the inner-city poor (O'Connor, forthcoming). Combining research, local planning, and community action, many of the experimental programs attempted to reform local government agencies to coordinate and make services more responsive to the needs of the inner-city poor. Alternatively, the initiatives set up new institutional structures designed to overcome the fragmentation and other problems associated with government bureaucracies. As the culmination of this series of experiments, the Gray Areas program was the product of an extended period of social learning in which continual program evaluation played a central role.
The chief architect of Ford's urban programs was a Harvard-trained public administrator named Paul Ylvisaker, who had most recently been an aide to Philadelphia reform mayor Joseph Clark. Aware that the staff was working from limited knowledge and experience with what would work in hard-pressed inner-city areas, Ylvisaker was deliberately experimental when he first set out to define the Foundation's urban program, funding a variety of locally planned initiatives aimed at different points of intervention. "We are confronted with the task of dealing with problems which have no criteria upon which to base decisions," he noted in a speech to a group of urban grantees in 1963. "We have to deal with the unknown. We have to have an experimental period, searching for the unknown, and then justify this activity" (Ylvisaker 1963). The purpose of such experimentation was as much to test the underlying problem analysis as it was to explore new methods for responding; lessons from earlier, more tentative urban interventions would be used to help refine and bring focus to the overall urban program. Thus, having started out with a somewhat vague notion that the "urban crisis" required government reorganization along metropolitan lines, Ylvisaker shifted his focus over time to what he called the "people problems" that were being ignored in government reform movements and in "bricks and mortar" urban-renewal efforts. These "people problems" were embodied in the "newcomer" populations: black, Puerto Rican, and Appalachian white migrants from rural backgrounds who were seen to suffer from low skills, low educational achievement, and overall "cultural deprivation." They were geographically concentrated in the urban "gray areas," neighborhoods in between the central city and the suburbs that in bygone days had served as staging grounds for immigrant upward mobility. The key to restoring this historic function still lay in governance, only now it was not for the purposes of metropolitan planning but to force social service agencies to be more comprehensive, coordinated, and responsive to the needs of the newcomers. Having thus narrowed the focal point for the urban program, the purpose of local experimentation then became a search for the institution that the Foundation could use as an "entry point" to stimulate comprehensive reform in the gray areas. Following short-lived attempts to use school systems and housing as the point of intervention, Ylvisaker's search eventually led him to the conclusion that such institutions did not exist and would have to be invented, leading to the creation of nonprofit organizations to provide the needed coordinating functions (O'Connor, forthcoming).
Although it was given rhetorical emphasis, evaluation remained a vaguely defined component of the Foundation's urban experiments. Individual programs were required to establish "self-evaluation" mechanisms, usually with the help of local universities, and they were periodically visited by staff members and outside consultants. Little effort was made to introduce any uniformity or validation process to these self-evaluations, however. Much more important to the social learning process was a different, less formal type of evaluation carried out by staff members and a changing cast of "expert" consultants appointed by Ylvisaker to "accompany" the process of program development, serving as what he called a "gadfly committee" to observe and generate new ideas. This eclectic group of experts included recognized scholars such as Peter Marris and Lloyd Ohlin, urban "practitioners" such as Chicago city planner Clifford Campbell, former mayors who shared Ylvisaker's reform ideas, and journalists. With few marching orders, they conducted site visits, participated in staff meetings, and helped to plan special training programs and conferences that brought grantees together, and eventually served as advisors to the grantees as well as to the foundation. This dual function did not raise a problem because, as Ylvisaker saw it, evaluation should involve the local project directors, the expert consultants, and the funders in a constant dialogue that would result in steady program improvements; the evaluation team members were to act as "teachers and learners" for "people on both sides of the table." In this sense, evaluation was a collective process (Ylvisaker 1963).
In addition to fostering the intangible social learning that accompanied the programs, this unstructured evaluation process produced a wealth of narrative reports, correspondence, internal memos, and the now-classic study, The Dilemmas of Social Reform , by Peter Marris and Martin Rein (1967). Together, these documents track the continuing learning process that informed the Gray Areas program, offering a kind of running commentary not only on how various local efforts were faring organizationally, but also on the conceptual revisions that accompanied program development. Thus, it is in these exchanges that one can detect the emergence of poverty, and the notion of a "vicious cycle" of cultural and material deprivation, as the underlying problem definition. Similarly, these exchanges articulate the theory of institutional reform that informed the Gray Areas approach (Marris and Rein 1967; O'Connor, forthcoming). In this sense, the Foundation's social learning process offers glimmers of that kind of theory-based evaluation strategies that methodologists have been calling for since the 1970s (Weiss 1972a). They also contain crucial insights into the process of community-based social action. A number of themes in particular stand out, perhaps because they are virtual advance warnings of the kinds of problems that would later be experienced by the local efforts sponsored by the Community Action Program during the War on Poverty. One is the sheer difficulty of achieving the kind of administrative cohesion and program coordination the Foundation was looking for in its attempt to change the way local governing institutions dealt with the poor. Here, more often than not, bureaucratic rivalries and historically rooted local power struggles got in the way of comprehensive, integrated approaches, as was the case in Boston, where the housing establishment struggled with social welfare leaders for control over the program. A second important theme is that racial inequality--particularly if left unacknowledged--could serve to undermine the effectiveness of anti-poverty efforts. Third, several of the Foundation's urban programs revealed an ambivalence about the nature and importance of citizen participation. Finally, some of the programs made mistakes with the media that later got them into trouble with public expectations and perceptions (Ford Foundation 1964).
As helpful as it was in the process of program development, the Foundation's social learning process did not systematically address the question of what the programs were actually accomplishing and did little to assess the connections between programs and their stated objectives. However, given the rudimentary documentation and short time-frame they were working with, it is questionable whether evaluators could have made reasonable statements about outcomes. By the time such evidence was becoming available, the Foundation was proudly handing its experimental programs off to the government's anti-poverty initiative, for use as grantees and model programs in the community action strategy. The Foundation's short attention span was only one aspect of its broader institutional incapacity to conduct and absorb the lessons of evaluation. At about the time when Ylvisaker's staff was assessing its urban experiments, Foundation president Henry Heald established a central office of Policy and Planning, the purpose of which was to rationalize planning and to be more systematic about feeding program lessons into the federal pipeline. Given the mandate to establish Foundation-wide program-evaluation procedures, this office was widely interpreted as Heald's effort to centralize control over his independent-minded staff, and few program officers wanted any part of it (Sheldon 1974). The Gray Areas programs were still new, had taken the Foundation into unfamiliar territory, and had introduced a much more activist style of philanthropy than had been followed in the past; though willing to conduct an internal dialogue with grantees, program officers were understandably wary of what the results of outcomes-oriented evaluation would say to the trustees, let alone to the outside world. This sensitivity was not confined to the Gray Areas program, and evaluation remained a problematic issue within the Foundation for the next several years.
The reluctance to subject programs to outside evaluation and to create accountability beyond the confines of the funder–grantee relationship had its costs. In the first place, the absence of more systematic, widely agreed-upon criteria for gauging program progress meant that choosing among alternative strategies was largely subjective and politically opportunistic; local program directors did take part in the evaluative dialogue, but only the most entrepreneurial among them had much influence on decision-making. Nor did the Foundation establish a way of following up on the "early warnings" coming from the consultants' reports. Second, the Foundation staff had difficulty accepting the critical perspectives on the conceptual underpinnings of their program--perspectives that challenged its essentially behavioral notions of poverty and its failure to grapple with the structural underpinnings of gray-area problems. Most important, because evaluation remained a dialogue among insiders, many valuable lessons were simply overlooked as the urban experiments became caught up in the federal anti-poverty efforts and the Foundation's priorities shifted from achieving local reform to providing replicable models for the War on Poverty. In the rush to promote community action as an anti-poverty strategy, those closest to the urban experiments proved willing to overlook problems that had given them reason for caution. In large degree, success was being measured in the volume of Administration visitors to the Gray Area sites; the passage of Title II, the Community Action clause of the Economic Opportunity Act of 1964, was deemed the program's "proudest achievement" (Ford Foundation 1964).
A slightly different version of evaluation as social learning can be seen in evaluations of juvenile delinquency demonstration programs in the 1950s and early 1960s. Delinquency was virtually a national obsession in the 1950s, the subject of numerous reports, special commissions, Congressional investigations, and popular fiction (Gilbert 1986). Public concern was matched by a flood of new research, much of it either challenging or building on the University of Chicago ecological theories from earlier decades. Youth organizations, universities, social welfare programs, and law-enforcement agencies--now armed with a broad spectrum of theoretical perspectives that implicated individual psychology, female-headed families, adolescent transitions, lower-class culture, and the absence of opportunity in explanations for delinquent behavior--became involved in a proliferation of community-based demonstration programs during the 1950s and '60s. Funded by the Children's Bureau, the National Institute of Mental Health, the Ford Foundation, and, later, the President's Committee on Juvenile Delinquency, these demonstrations were more firmly grounded in theory than the Gray Areas and related urban experiments were, and the accompanying evaluation plans were meant to reflect on the validity of one or more of these theories. However, as part of relatively short-lived and sometimes elaborate demonstration projects, these evaluations could not possibly achieve the requirements of time necessary to subject theories--most of which were tied to some form of life-course framework--to empirical assessment.
If they did not exactly offer definitive backing for any single theory, the findings from these theoretically informed evaluations of delinquency programs did play a prominent role in social scientific debates over the nature of delinquency and, later, poverty. (See, for example, Miller 1968.) Ultimately, however, pragmatism and ideology--and not evaluation research--were responsible for the growing popularity of the opportunity theory of delinquency among liberal policymakers in the early 1960s, and its incorporation into the Juvenile Delinquency Control Act of 1961. This theory was propounded by sociologists Richard Cloward and Lloyd Ohlin in their book Delinquency and Opportunity (1960), and embodied in Mobilization for Youth, a community action program on New York's Lower East Side. It shifted attention away from deep-seated psychological and cultural factors and toward more malleable environmental conditions, arguing that the absence of a viable "opportunity structure" for social and economic advance was what pushed disadvantaged youth into delinquent behavior. With its emphasis on creating new pathways for young people to share in the American Dream, opportunity theory captured the imagination of Kennedy staff members and fit in with the ideological predilections of the new Administration. Unlike the more psychological and culturally oriented explanations, opportunity theory also pointed to concrete targets for intervention, in the form of educational, job training, and other opportunity-enhancing programs (Hackett 1970; Marris and Rein 1967).
The Gray Areas and juvenile delinquency evaluation strategies shared a common, if not widely articulated, view of evaluation as an essential part of the social learning that would guide advocates of social change as they sought out new ways of responding to urban problems. Emphasizing the educational function of evaluation, they focused on underlying concepts, on administrative processes, and on site-specific details that could only be picked up in close observation. If not entirely successful in the sense of having produced definitive evidence regarding the effectiveness of the experimental program strategies or the validity of the assumptions that informed them, these evaluations did produce a rich body of information that was certainly relevant to advocates of community-based change. In the rush to implement community action on a national scale, however, the mostly cautionary lessons from this brief period of social learning were often overlooked, and the potential lessons for evaluation methodology were never developed.
Evaluation as Impact Assessment: PPBS and Community Action Programs
As part of its broader strategy against poverty, the Johnson Administration established the Community Action Program (CAP) in Title II of the Economic Opportunity Act of 1964, providing funds for rural and urban Community Action Agencies (CAAs) to mobilize local resources on behalf of the poor. Modeled on the Gray Areas and Juvenile Delinquency demonstration programs, CAP was conceptualized as a mechanism for stimulating program coordination at the local level and for ensuring "maximum feasible participation" by the poor. The program was centrally administered by the Office of Economic Opportunity (OEO), which required CAAs to submit planning and action proposals, and to report regularly on their progress. Local CAAs were also encouraged to develop local capacity for evaluation. As the agency responsible for research and evaluation of the poverty program as well as for administering CAP, OEO also had the more difficult job of evaluating the impact of CAP with respect to the goals of national anti-poverty policy.
OEO's attempts to evaluate CAP should be seen as part of the overall governmental effort to evaluate the new "social action" programs in light of Great Society objectives. This posed a substantial challenge to social science and to the federal bureaucracy, which was faced with the task of generating both the knowledge and the staff necessary to implement and act on the results of evaluation. Unlike other agencies responsible for administering social programs, OEO had a sizeable research budget and had established a separate division for Research, Plans, Programs, and Evaluation (RPP&E) from its outset. This division was staffed primarily by economists and sociologists, who through a combination of primary and contracted research were responsible for meeting the diversified knowledge needs of the entire War on Poverty. In addition to basic research on poverty, RPP&E had to generate a substantial amount of information in response to the requirements of the PPBS system, and to the persistent political pressure to convince Congress and the public that the government was winning the War on Poverty. In addition to knowledge of labor markets, community processes, statistical techniques, and evaluation methodology, the RPP&E division needed versatility, political savvy, and the ability to produce under pressure from its staff. Director Joseph Kershaw and assistant director Robert Levine were both economists with previous government experience and had also worked at the Rand Corporation, where they had become familiar with the PPBS system and cost/benefit analysis; over the years, they recruited colleagues from the Department of Defense, where PPBS had first been instituted. As a result, the RPP&E staff was closely tuned in to the importance of systematic, quantitatively supported, goal-oriented data for the sake of making the case for poverty programs to the Administration and to Congress.
As one of its first steps, this office came up with what it hoped would be the road map against which its annual appropriation requests would be judged: a five-year plan that would take the country well on the way toward the ultimate goal of eliminating poverty within a generation (Levine 1970). The five-year plan retained the comprehensive vision of the War on Poverty, and included provisions for expanded community action to deal with the local political and institutional barriers to opportunity, expanded job training to reach a broader segment of the population, and a significant public job creation component to deal with structural slack in demand for low-skill workers. However, the most prominent feature of this and subsequent OEO five-year plans was fundamental reform of income maintenance policy. Whether they took the form of family allowances or a negative income tax, such ideas were quite popular among economists across the political spectrum throughout the 1960s and early 1970s, when it appeared the country had the means, and could the develop the technological know-how, to eliminate "income poverty" while simultaneously working to expand opportunities (Levine 1970). Motivated by the possibility of reform along these lines, RPP&E staff contracted with the University of Wisconsin's Institute for Research on Poverty and Mathematica Policy Research, Inc., to design and run the New Jersey Income-Maintenance Experiment, a social experiment that was unprecedented in scope and ambition (Kershaw and Fair 1976; Watts and Rees 1977). This experiment, versions of which would be launched in other urban and rural locations, was seen at the time to represent the cutting edge of evaluative policy research, using sophisticated sampling techniques and random assignment, and testing several different versions of the income package. Launched in 1967, it also coincided with officialdom's preference--bordering on a creed, to hear some tell it--for large-scale controlled experiments as testing grounds for proposed policy innovations. This preference, which helped to generate support for additional income maintenance, health care, work/welfare, and other social experiments in the 1970s, became especially pronounced and spread to a wider circle of policymakers in the 1980s when experimental design became the mantra of Congressional welfare reform advocates on both sides of the aisle (Wiseman et al. 1991; Fishman and Weinberg 1992; Manski and Garfinkel 1992).
While the demands of long-range planning and policy innovation created an institutional impetus for supporting large-scale quantitative experimental research on income maintenance, evaluating CAP and related social programs became increasingly tied to the political need to produce measurable results--and to produce them quickly. Such pressures were foremost in the minds of RPP&E staff members as they worked with CAP program staff to develop a comprehensive evaluation strategy beginning in late 1965. In efforts to stave off simplistic demands for mere numerical reporting, Kershaw and Levine took it upon themselves to educate program administrators and the Congress about the purpose of evaluation and the particular difficulty posed in evaluating CAP. Systematic reporting on program content, participation levels, and other gauges of program input, though essential, had to be distinguished from assessments of CAP's outputs, if not yet in terms of reducing poverty rates then in terms of reasonable interim measures. In practice, however, identifying and measuring the appropriate effectiveness indicators was not a straightforward process. CAP as a program not only had multiple objectives, it also was responsible for administering several individual programs that themselves had multiple objectives. This posed the immediate problem of whether impact should be assessed by focusing on the comprehensive, integrating functions unique to local CAAs, or by accumulating evaluations of the individual programs administered under their auspices. The former approach, which was methodologically the more challenging task, was clearly the one that captured the essence of the community action idea. Recognizing the complexity of the task, Kershaw attempted early on in his tenure to engage the talents of top-ranked researchers and social theorists in evaluating CAP. He first approached Yale University, which, because of its academic prestige and proximity to New Haven's "model" Gray Areas/CAA, seemed ideally situated to conduct a comprehensive evaluation of a local project while also developing methodology for evaluating community-based interventions more generally. To the chagrin of OEO director Sargent Shriver, himself under the gun to testify on CAP in Congress, Kershaw envisioned a process of scholarly conferences and research that would take a minimum of two years to have any concrete pay-off. The differences between Shriver and Kershaw themselves turned out to be academic: despite expressions of enthusiasm from Yale president Kingman Brewster, the University failed to take up Kershaw's offer of funding for such an undertaking, claiming that it was impossible to find the senior scholar with the ability and the commitment to take responsibility for the project (OEO 1965). Nor could Kershaw seem to persuade any other major academic institution to take it on, despite numerous overtures. As a result of a more general request for proposals, OEO eventually made individual contracts with several universities and nonprofit research organizations to conduct evaluations of local CAAs, jettisoning the more ambitious effort to develop new evaluation methodologies. In the meantime, the pressure for evidence of concrete program achievements was unyielding, whether coming from an increasingly skeptical Congress or from the need to justify budget requests to an increasingly stringent Administration. OEO thus had to make due with what it considered a partial, short-range evaluation plan combining site visits with "single component" evaluations of specific programs such as Head Start and Job Corps. Thus, for a combination of political and academic reasons, Kershaw's hope of making a serious methodological contribution to evaluating comprehensive social action programs-- and ultimately of gathering evidence to show how, in CAAs, the whole could be far more than a sum of its parts--was never seriously pursued.
The political pressures on OEO's evaluation function only intensified in 1967 when, as part of an effort to rein in the War on Poverty, Congress passed a series of amendments explicitly requiring that all OEO programs be subject to "careful, systematic and continuing" evaluation using "control groups, cost-benefit analysis, data on the opinions of participants about the strengths and weaknesses of programs," and the like, requirements that would "outstrip the capabilities" of OEO's evaluation division, according to Levine (1968). CAP and the work-training program were singled out as programs in need of published standards of effectiveness that would be used in deciding whether they would be renewed, tying evaluation of these programs even more closely to the budget allocation process than they already were under PPBS. Under these mandates, evaluation for the sake of learning, of developing methodology, and of program improvement would take a back seat to evaluation for the sake of political accountability. In response to the 1967 amendments, RPP&E revamped its evaluation practices, distinguishing between overall program effectiveness and program-specific evaluation (Levine 1968).
Increasingly, then, under the influence of Congressional critics, evaluation of politically vulnerable programs such as CAP were supposed to take on a "bottom line" quality. Though this legislative intent was well understood by OEO in the waning months of the Johnson Administration, it was not until the Nixon years that this somewhat single-minded approach to program evaluation was fully accepted as an objective. As part of a Congressionally mandated review of OEO evaluation procedures, a contractor for the General Accounting Office approvingly cited an internal OEO memo by a staff member appointed under the new regime, articulating this philosophy of evaluation and in the process drawing a sharp distinction between large-scale impact studies "based on national samples and using sophisticated research designs with control groups and longitudinal measures of change" and studies aimed at improving program operations. Leaving no doubt about which approach was preferable for policy purposes, the memo went on to say that " . . . social programs should be evaluated on the basis of a generalized quantitative technique very much like the framework within which business investment decisions are made. The ideal methodology would remove judgment from the process and, in essence, permit the public authorities to settle society's problems by computation" (Resource Management Corporation 1969, 23, 30). Achieving this ideal type of evaluation would require more far-reaching changes at OEO than those initiated by Levine, the report indicated. And those changes had direct implications for the future of CAP: one of the chief obstacles to more "effective" evaluation in the past, it found, was the OEO's dual mandate as an operating and an "R&D" organization. The implication was that one of those functions would have to be relinquished.
Prior to the release of the Government Accounting Office (GAO) report, Levine and his staff had managed to get a start on the overall impact assessment of CAP called for in the 1967 legislation, and the fate of those efforts as the Nixon Administration took over are revealing with regard to the uses of evaluation under changing political conditions. This overall impact assessment was based on eight case studies conducted by separate outside contractors, and a national impact assessment jointly conducted by The University of Chicago's National Opinion Research Corporation (NORC) and Barss Reitzel and Associates, Inc., of Cambridge, Massachusetts. As noted by Joseph Wholey in his study of federal evaluation policy, little effort had been made to coordinate the design of the case studies and they were generally regarded to be of limited value in the larger assessment task (Wholey 1970). The NORC/Barss Reitzel efforts, however, held more potential interest, for they were based on a national sample of 100 CAAs, and included multiple-survey, personal interview, and observational techniques. The study's objective was to determine CAP's effectiveness in stimulating local institutional change, as measured by indicators such as the changes in the size and demographic make-up of the population served by local agencies, changes in the demographic composition of various community boards, and the number of agency referrals generated by CAAs. Its major finding was that CAAs did indeed bring about local institutional change, and that those dedicated to community mobilization and empowerment had a greater impact on local institutions than did CAAs serving primarily as local service coordinators. Reporting on these findings, the authors were careful to point out that change was well within the law and the boundaries of political acceptability; CAAs were making the system work better, without unduly shaking up the status quo. However, the question of whether institutional change translated into better services and employment outcomes for the poor was left open to interpretation in the early reports from the study (Vanecko 1970; Vanecko and Jacobs 1970). In a separate analysis based on face-to-face interviews with a range of community respondents drawn from the same sample, a Barss Reitzel study expressed more skepticism about the anti-poverty effect of the CAAs, claiming that on the "gut issues" of jobs, income, and political power, CAAs had little actual impact (Barss-Reitzel and Associates, Inc. 1969).
By the time the results of these various studies were available, the Nixon OEO was in place, and the future of both CAP and OEO were in doubt. Convinced that the future for OEO was in acting principally as a social "R&D" laboratory rather than in the actual administration of politically vulnerable programs, OEO director Donald Rumsfeld and his successors acceded to the phase-out of CAP and the transfer of administrative responsibility for Head Start and other programs to the appropriate outside agencies. OEO's research agenda reflected the Administration's priorities: reforming income maintenance was one and assuring the continuation of the Negative Income Tax (NIT) experiments was another; community action was not, and OEO would not follow up on the effort to learn from the CAP experience. However, the OEO research staff under the Nixon Administration was responsible for "drafting . . . a replacement for the current Title II of the Economic Opportunity Act. . . . " (Redenius 1970, 1). In the process of gathering a range of evidence to support the recommendations they would make, the staff contracted with the Institute for Research on Poverty to conduct a comprehensive and synthetic review of the eight case studies that had been done, and to review the findings that were then coming out from the NORC/Barss Reitzel study. The legislative timetable would not wait for an eight-case CAP synthesis, as it turns out, but sociologist Myron J. Lefcowitz did conduct a review of the NORC study. In a thorough critique, Lefcowitz called the findings into question, noting that the investigators had not adequately specified their indicators of institutional change, and had relied on simple correlational findings rather than more rigorous causal analysis to draw their conclusions. Nor did the study address the issue of the magnitude of change that resulted from various practices. These and other methodological flaws made the study virtually useless for policy purposes, he concluded. However, he added, "there can be little doubt that the material available . . . is one of the social scientific mother lodes of the decade" (Lefcowitz 1969, 1).
While Lefcowitz's critique was heavily influenced by the standards of cost/benefit analysis and experimental evaluation research, his documentation of methodological and analytic flaws was convincing. Equally plausible were his conclusions regarding the political issues raised by this and other assessments of CAP, which, he suggested, could account for problems in the study. One was the "tremendous time pressure to weave a story from a mass of data," for which OEO "could not wait for the year or two necessary to do the proper analytic job. The Community Action Program is continuously under the political gun and any evidence that would justify its past performance and legitimate its future continuance was and is essential" (1). But political considerations had also influenced the substance of the evaluations findings, Lefcowitz suggested, noting a "strain" in the analysis to show that CAP "is accomplishing something and not upsetting the system" (1). What Lefcowitz may not have anticipated was the political use to which his own report on the evaluation might be put.
Soon after submitting the Lefcowitz report, it became clear to Harold Watts, the director of the Institute for Research on Poverty (IRP), that OEO's motivation in commissioning the review was political. "We have now moved to the point where your office has a lack of interest in (or even impatience with) this task and seems to me to want us to shoulder at least some `line responsibility' for formulation of new policy direction," he wrote in a pointed memo to Assistant OEO director John Wilson. "Moreover, there seem already to be presumptions about that direction which we are expected to validate (endorse, sanctify?)," he continued. "Namely, the direction seems to be how do we cool out the CAP operation and make a nice, well-behaved (emasculated?) service-distribution mechanism" (Watts 1970, 1). For technical but more importantly for political reasons, then, there was no serious and sustained effort to learn from the evaluations of the CAP program.
That the War on Poverty yielded neither a thoroughgoing assessment of the impact of CAPS nor any significant methodological inroads into evaluating comprehensive community interventions represents a significant lost opportunity, a loss due to political pressures, to a lack of academic engagement in the issues, and to the fact that there was insufficient organized political interest in establishing whether and how a sustained investment in CAP could contribute to anti-poverty goals. Even the evaluations that were conducted--partial or flawed though they may have been--had considerable value in clarifying the difficult methodological issues that would have to be addressed: how to operationalize the many and conflicting goals of CAP; how to measure the existence and extent of such complex, long-range processes as institutional change; identifying appropriate interim indicators for measuring progress against poverty; and developing the appropriate framework for comparing findings across sites. Equally significant, an unfortunate and artificial distinction between policy-relevant outcome evaluation and program-relevant operations monitoring began to take hold and became institutionalized as Congress increased the pressure for the "bottom line" on OEO programs. An RPP&E staff that was initially open to exploring new ways of assessing CAP's impact--and even to questioning some of the precepts of PPBS--never got the chance.
Evaluation as Policy Learning: The Case of Model Cities
As CAP was getting under way, the Administration was also trying to breathe new life into its national urban policy by establishing priorities for the newly established Department of Housing and Urban Development (HUD). In 1965 LBJ appointed a special task force for this purpose, chaired by Massachusetts Institute of Technology (M.I.T.) urbanist Robert C. Woods, and instructed its members to be bold and innovative, leaving the legislative politics to the White House. With the sense of urban crisis looming, exacerbated by recurring incidents of racial unrest in impoverished inner cities, the task force came up with a set of recommendations designed to make up for the failings of urban renewal and to avoid the political mistakes of the anti-poverty program. Included in its report was a proposal for what would become the Model Cities program: grants to a designated number of demonstration cities to concentrate and coordinate existing federal resources for physical and social development in inner-city neighborhoods. The task force recommended that a total of 66 cities be chosen as experimental sites, which would be eligible for the full range of existing categorical grants on a "priority" basis, a new block grant administered by HUD to provide for planning and implementation of comprehensive redevelopment plans in poor neighborhoods, and technical assistance from a federal coordinator working as a liaison with local agencies. While the bulk of the funding would come from existing social welfare, housing and employment programs administered by the Departments of Health, Education, and Welfare (HEW), HUD, and Labor, HUD would be designated as the "lead agency" responsible for getting cooperation from the others in making grants available. To avoid some of the tensions that had been created by making Community Action Agencies independent of mayors, the Model Cities Demonstration Agencies would be run by the mayor's office, and the mechanisms for assuring citizen participation would be left up to the localities. The task force recommendations also included proposals requiring racial integration in new federally funded housing units. The estimated cost of the program was $2.3 billion in new appropriations for the planning grants, with an anticipation that as much as $12 billion would be available through the existing categorical grant programs. Through these measures, the task force hoped to achieve a concentration of resources in the neediest areas, coordination of fragmented social welfare and housing programs, local flexibility in the use of categorical grants, and a federal commitment to careful experimentation: the program envisioned one year for planning and five for implementation (Haar 1975).
The proposed legislation arrived in Congress with several handicaps that made it difficult to build a solid constituency behind the program. Critics saw it as a proposal from "on high," cooked up by an overly secretive task force set on adding yet another array of programs to an already overwhelming domestic agenda. Mayors, though generally supportive, insisted that the projected funding levels were much too low. Others worried that it would drain money from cities not in the program. The Administration, on the other hand, was trying to restrict new requests for domestic programs, aware of escalating costs in Vietnam and wary that Congressional willingness to approve additional spending had already been stretched to its limits. By the time it got through Congress, the Model Cities legislation had been expanded to twice the number of cities, its appropriation cut to $900 million, and its initial life span reduced from six years to two years. The bill had also been gutted of racial integration requirements, and designated no central federal coordinating authority, not even offering clarifying language for how the program was to be implemented. Equally significant, the legislation was now being pitched as the Administration's response to Watts, to the Kerner Commission Report, to the threat of future "long hot summers"; Model Cities would appease the inner city. All of this left a sense of deep ambivalence about the ultimate purposes of the program: was it a national experiment, to guide future urban policy, or was it simply another way of providing aid to "distressed"--and dangerous--areas? One thing seemed certain to the original architects of the Model City idea: what had started out as an ambitious effort to demonstrate what could be achieved by concentrating substantial resources and know-how in disadvantaged neighborhoods was in danger of becoming simply another bureaucratic mechanism for helping cities to use available resources--however inadequate--more efficiently (Haar 1975; Frieden and Kaplan 1975).
The same ambivalence and vagueness of purpose that accompanied the watered-down Model Cities program was reflected in efforts to institute evaluation procedures once the legislation was passed. In contrast to the trend initiated in other Great Society legislation, neither the task force nor Congress had paid much attention to evaluation, and no money had been set aside for it. Thus, it was up to administrators in HUD, working within a new bureaucracy, with virtually no trained staff and little idea of the appropriate methods, to come up with an evaluation plan for a program whose ultimate purposes were uncertain. More immediately damaging to the evaluation cause were the deep divisions within HUD over the function and control of evaluation. The Model Cities Administration (MCA) was established under HUD's jurisdiction as the agency responsible for implementing the new legislation. Its evaluation staff came up with an elaborate plan that reflected the legislation's experimental intent, seeking to learn from the experience of the initial demonstration sites for the purpose of refining the program in future rounds while also trying out a variety of evaluation methodologies. This plan was premised on the notion that evaluation was an integral part of the program design and implementation; optimally, local programs would be designed with the needs of policy evaluation in mind. Included in the MCA's proposed evaluation plan were documentations of the local planning processes, assessments of existing and prospective local information systems, and measures of Model City's local impact as indicated by changes in neighborhood attitudes, levels of citizen participation, institutional change in the targeted neighborhoods, and improved economic and social conditions for local residents. Reflecting HUD's limited internal capacity to conduct such an ambitious, multi-pronged evaluation plan, the MCA staff proposed that it be conducted by outside contractors.
In what turned out to be a protracted and ultimately fruitless internal bureaucratic struggle, the HUD deputy undersecretary's office objected to the plan, arguing that the MCA should have control over implementation but not over evaluation, which was more properly a function of a separate monitoring and oversight office. Indicating a very different orientation to evaluation, the undersecretary's office criticized the MCA plan for jumping the gun; evaluation would not even be relevant until the after the cities had begun implementing their plans (Haar 1975).
Eventually, the MCA did receive funding for Model City evaluation, only then to be thwarted when it attempted to issue requests for proposals (RFPs). The lingering bureaucratic tensions were ultimately settled by the changeover to the Nixon Administration, which incorporated Model Cities into its broader "new federalism" policy. Strengthening local capacity--and not learning for the sake of a future, expanded federal policy--became the rationale for Model Cities. Within that framework, the immediate needs of program implementation became the standard for determining the kinds of evaluation studies HUD would undertake, placing emphasis on troubleshooting, and internal management issues. As a result, the national, broader impact evaluation plan was never implemented.
The Model Cities experience nevertheless did suggest important lessons for efforts to implement a national strategy of community-based intervention. Two of them stand out as especially pertinent for this discussion. First, Model Cities illustrated on a federal scale what the Gray Areas experience did on the local level: agency coordination is very difficult and should not to be taken for granted. Never clearly designated and given no leverage to act as the "lead agency" for the effort to mobilize the categorical grants available through various federal agencies, HUD ran into tremendous difficulties in getting the other agencies to cooperate for the sake of concentrating resources. Second, Model Cities offers a stark example of the problems built in to the demonstration model of social programming, problems also evident in the Gray Areas, Juvenile Delinquency, and in subsequent demonstration programs. Inevitably, these demonstrations have run into tensions between the goals of experimentation for the sake of replication, policy revision, and future program improvement, and the goals of effective local management--tensions that arise not from an inherent incompatibility between these goals, but from the existence of differing priorities among different players in coordinating multi-level programs. Thus, national funders want to know how local demonstration sites are furthering national policy objectives, and try to be sure that local programs are designed to test the relationship between intervention strategy and outcome. Local program directors, while often sharing the need and desire to learn about the relationship between intervention and outcome, often find that the needs of the experiment might conflict with the local responsibilities of the program. These tensions have been played out in the evaluation process, raising basic questions about who should have responsibility for evaluation, how criteria for success will be determined, and to what end evaluation results will be used. There is no clear answer to these dilemmas, but the Model City and subsequent experiences suggest the need for critical reexamination of arrangements for funding and learning from local programs.
Finally, Model Cities is, in the words of one recent analysis, "a lesson in the dissipation of limited resources" (Edelman and Radin 1991, 53). As such, it points to another potential pitfall in the use of local demonstrations for the purpose of policy learning. Time and again, proposed multi-site demonstrations have been stretched thin in the legislative process, expanding to cover a large number of Congressional districts even as appropriation levels are pared down. Such inadequately funded experiments only reinforce skeptics' expectations when they fail to achieve the ambitious goals of federal policymakers.
Evaluation and Context: Community Development Corporations
Almost from its inception, the War on Poverty was subject to a critique of its assumption that poverty could be eliminated without significant job creation, a critique coming from within the ranks of the Administration as often as from outside commentators. This criticism took on added weight as the extent and impact of inner-city joblessness were highlighted in such "events" as the debate over the Moynihan Report, civil unrest, and extended hearings on inner-city conditions headed by Senators Javits and Kennedy in 1966. One product of those hearings was an amendment to the Equal Opportunity Act providing federal funds for the Special Impact Program (SIP) for economic development in neighborhoods and communities where "dependency, chronic unemployment and community deterioration" (Abt 1973, 6–7) converged, and under this program to support a project in New York's Bedford Stuyvesant neighborhood that promised to add job creation and economic development to the mix of services and housing the federal government was trying to achieve in Model Cities. Like CAP and Model Cities, the projects funded under SIP were designed to target the neediest neighborhoods, concentrate available resources, and coordinate interventions in a total, comprehensive approach. Unlike the other efforts, however, the strategy in Bedford Stuyvesant and other neighborhoods aimed directly at regenerating the local economy through community economic development and at creating its own infrastructure of community-based services rather than seeking to reform government bureaucracies. At the center of several of the community-based development efforts was the community development corporation (CDC), a private entity, often combining nonprofit and for-profit branches, set up with multiple purposes, such as building housing and other neighborhood infrastructure, providing job training and technical assistance to local small businesses, and leveraging public and private funds for community-building. To a greater degree than past community action efforts, the CDC also represented an effort to build on several genuinely "home grown" movements--many of them headed by minorities--to promote comprehensive redevelopment and to control the future of the community. Over the next few years, federal funds were extended to approximately forty urban and rural CDCs under SIP and its successor programs, although overall funding levels remained relatively low. The Ford Foundation and others also established a major philanthropic presence in community-based economic development, lobbying for Congressional funding and providing direct support for major CDCs in the late 1960s, and then moving to create intermediaries to provide technical and financial support in the 1970s. In the 1980s, Ford once again began to provide direct subsidies for "emerging" CDCs, partly in response to severe federal funding cutbacks (Halpern 1994; Peirce and Steinbach 1987; Vidal 1992).
From the start, federally sponsored community development in impoverished areas was burdened by inadequate funding levels and a cumbersome administrative set-up, which presented particular challenges for evaluation. Centrally administered by OEO, the SIP program actually consisted of a series of development projects that were proposed by several different agencies, including the Departments of Labor, Agriculture, Commerce, and OEO itself. Working with relatively small sums of money, a specially designated OEO committee would choose a number of projects from among these proposals for funding, using as criteria the legislative requirements that the programs be targeted on areas of concentrated poverty, that they show promise of stimulating local economic development and employment, and that they be large enough to have a significant impact on the community. Given the limitations of funding, the OEO also "made an early decision to use this money to test and evaluate rigorously the various approaches with a view to finding out the best means of utilizing the operating impact programs," according to then-OEO director Bertrand Harding (Harding 1968). If evaluators were thus to play a prominent role in the SIP strategy, they also had to respond to a number of challenges, as noted in the evaluation RFP issued by OEO. One was the multiplicity of agency sponsors and the varying emphases they brought to the program. A second was the "complexity of the poverty impact indicators" (OEO 1968), which would require measures not only of job creation and other aspects of economic opportunity, but also of changes in neighborhood environment, attitudes, and other less tangible effects. A third difficulty, according to the instructions for potential evaluators, was the "shortness of time" (OEO 1968)
available for evaluation. In order to meet Congressional reporting requirements, OEO needed "hard data" within one year of issuing the evaluation contract, and suggested that evaluators might have to settle for reports on "before" rather than "after" data (Harding 1968). As it turned out, time would also be a frustrating factor in the OEO effort to conduct rigorous evaluation of the anti-poverty impact of community economic development strategies: by the time longer-range measures would have been available, OEO was being dismantled and responsibility for SIP re-assigned.
In light of these administrative and political realities, the early CDC movement did not put much emphasis on basic research and knowledge-generating activities. Evaluations commissioned by the funding agencies, the Ford Foundation, and other sponsors generally relied on individual case studies, ranging from the highly technical to the highly anecdotal. Little was done to develop comparative frameworks that would help to inform or draw lessons from these ongoing case studies. In part, the relatively low priority given to research and evaluation can be attributed to the activist roots and pragmatic ethos that have pervaded the CDC movement, along with an understandable impatience with social scientific concepts not grounded in local reality. But it is also the case that government and other funding sources did not invest significant resources in CDC evaluation, for lack of persistent political pressure to do so, and in the absence, following the demise of OEO, of consistent centralized administrative accountability for their contribution to anti-poverty goals. A third barrier has to do with the kind of evaluation required to understand the impact of CDCs. As the efforts most explicitly designed to change the economic and physical environment in poor communities, CDCs called for the kind of contextual analysis that was not being pursued in evaluation research at the time.
This conclusion can be drawn from three studies, two conducted in the early 1970s and one in the mid-1980s, that did attempt to develop broader frameworks for assessing CDC performance and effectiveness as an approach to reducing neighborhood poverty. One, conducted by Abt Associates in the early 1970s, was an evaluation of SIP commissioned by OEO. Using statistical measures of CDC growth rates, job-creating capacity, average wage rates in the jobs created, and resident attitudes and perceptions in urban and rural CDCs, the study attempted to measure the potential for "appreciable and continuing impact" on the local area, and to determine how particular characteristics of CDCs influenced their effectiveness. While most of the SIP program grantees showed only limited success in the various performance indicators, the Abt study concluded that CDCs did have the potential to achieve appreciable impact and that their commitment to the comprehensive approach to community needs and citizen participation made them more effective than other types of minority business ventures as local economic development agencies (Abt Associates 1973). In another study, commissioned by the Ford Foundation in 1973, the Urban Institute attempted to develop a framework for assessing individual CDC performance standards based on quantifiable "milestones" of profitability, production levels, and efficiency set for each of their major program areas. Using case studies of Bedford Stuyvesant, The Woodlawn Organization (TWO) in Chicago, and The Zion Non-Profit Charitable Trust in Philadelphia, the evaluation found that the CDCs varied considerably in their areas of strength, and although they showed some progress, they generally fell short of the milestones set by the evaluation team. Despite or perhaps because of these somewhat discouraging findings, the report concluded that CDCs were viable community institutions that could not be judged solely in terms of profitability or efficiency indicators. Calling for alternatives means of assessing the effectiveness of CDCs, the report pointed out that they were uniquely vulnerable to external political and economic conditions; evaluation would need some way of taking the effects of such externalities into account (Urban Institute 1976).
By the time the Urban Institute completed its report, the policy climate for CDCs had changed considerably: the economy was in decline, OEO had been dismantled and the Special Impact Program transferred to the Community Services Administration, housing programs were at a virtual standstill, funding for CDCs from all sources was in decline, and the emphasis among funders was increasingly on weaning them away from direct subsidy to promote self-sufficiency. "The combination of directions in federal policy and national economic trends paints a very serious and bleak picture for both CDCs and the communities they serve," the Urban Institute authors concluded (Urban Institute 1976, 130). Nor was any attempt made to follow up on either of the early attempts to develop frameworks for assessing and monitoring CDCs on anything more than a strictly localized basis. In this sense, the failure to develop an overall evaluation strategy can be seen to reflect more than the culture of the movement or the difficulty of comparison. It was an acknowledgment of the federal retreat from direct involvement in poor communities, the end of the short-lived experiments of the 1960s. It was also an expression of skepticism about social science research, a retreat from the Great Society notion that applied research could be used to make and improve policies. While thousands of community-based anti-poverty initiatives continued their work, there was no longer a clearly identifiable audience, no visible bureaucratic or political demand, for broadly based evaluations that would assess local interventions as contributors to national anti-poverty policy.
Thus, in the mid-1980s a third effort to assess the overall anti-poverty potential of CDCs had no empirical or theoretical base on which to build. Undertaken by the Community Development Research Center of the New School for Social Research, this third effort involved a study based on a national sample of 130 successful CDCs. Characterizing its findings as "exploratory" in nature, a report issued in 1992 described a record of moderate success according to quantitative indicators of housing production, commercial real estate, and, to a lesser extent, small business development, linking success to a number of CDC characteristics, such as size, quality of leadership, and record of past success. Also important to CDC success, the report emphasized, were several contextual factors, including the overall conditions of the neighborhoods they served, the existence of a local network of CDCs, and local political and institutional support (Vidal 1992). Relying as it did on numerical indicators of success in each area of CDC activity, this report did not address the issue of how the CDC acted as a comprehensive, multi-faceted neighborhood presence rather than simply a sum of its programs, nor did it address the question of how effectively CDCs were meeting neighborhood needs. Housing represented the least risky of the economic development activities pursued by CDCs, but did housing development respond to community needs and priorities? These and related issues continue to be explored in subsequent, related research, suggesting that this study will continue to be an important empirical base for future evaluative inquiries.
Conclusions and Lessons
This historical review suggests three major conclusions that should help to guide contemporary efforts to evaluate comprehensive community-based initiatives. Drawing attention to the political barriers that have hampered effective evaluation, to the persistent dilemmas facing community-based initiatives, and to the relatively limited role that scientific evaluation has played in determining the fate of past community-based initiatives, these conclusions suggest concrete ways to develop new strategies for conducting, learning from, and building a constituency for the results of evaluation.
First, the barriers to developing effective evaluation strategies for community-based initiatives have been political and institutional as much as they have been methodological and substantive. This paper focuses on the former; others in this volume address the latter in more detail. Not all of these political and institutional barriers are easily susceptible to change. However, there are steps that researchers, foundations, program administrators, and others interested in the future of community-based initiatives can take to improve the situation, even as they work on developing better methods for measuring program impact and effectiveness. One is to create legitimacy--among social scientists as well as policymakers--for the theory-based, qualitative, contextual, and integrated quantitative/qualitative techniques that are necessary for assessing community-based initiatives. Related to that is the need to create incentives and opportunities for evaluators to explore these and other appropriate methodologies, in collaboration with local program directors and community residents. A particularly useful step in this direction would be to engage current theorists of community and neighborhood effects in the project of evaluation and developing evaluation methodologies. In addition, funding agencies should examine their own internal procedures for evaluating community-based initiatives, asking, among other things, whether those procedures are based on a shared and realistic set of expectations for program achievement, whether local programs and funding agencies have the flexibility to absorb the lessons from evaluation, and whether evaluation strategies adequately respond to the knowledge needs of various collaborators and "stakeholders" in community-based initiatives. Only by responding to these varied needs can a broad-based constituency for continuing evaluation efforts be built and sustained.
Second, evaluators should explore in greater depth the persistent dilemmas that community-based initiatives have encountered over the years, in the hope of promoting historical as well as cross-initiative learning, and for the pragmatic purpose of helping funders and practitioners to think through these recurring dilemmas. One example of a recurring dilemma experienced by leaders of community-based initiatives is the tension between the need to maintain a comprehensive vision of problems and objectives and the practical demand to focus, whether on a discernible set of problems and goals or on a clearly defined geographic area. A second persistent frustration for community-based initiatives has been the difficulty of achieving agency coordination and institutional reform, as illustrated particularly in continuing conflict between the "bricks and mortar" and the human service bureaucracies at the federal and local levels. A third persistent difficulty has been in achieving genuine integration among the individual components that make up community-based initiatives--making the whole more than a sum of its parts. Appropriate responses to these and other built-in dilemmas could be illuminated by historical and comparative analysis. By the same token, the recurrence of common problems across time and space may provide the basis of a comparative framework for conducting cross-site evaluations.
Third, several useful models for evaluating comprehensive community-based initiatives are suggested by past experience, each of them offering lessons in the role evaluation can usefully play in program and policy, as well as insights into the programs themselves. But experience also raises a cautionary note for evaluators: no matter how rigorous the scientific method, evaluative evidence will play only a limited--and sometimes unpredictable--role in determining the political fate of social programs. In the past, decisions about community-based initiatives--or about welfare reform, for that matter--have been driven not, primarily, by science but by the values, ideologies, and political interests of the major constituencies involved. Historically, comprehensiveness and citizen participation have been pursued in community-based initiatives because they make sense, they are strategies that grow out of a long tradition of local democratic activism, because experience shows that the "piecemeal" categorical alternatives are not effective, and because they embody values that are worthwhile ends in themselves. Analysts have yet to produce scientific evidence about whether these strategies "work" in terms of measurable outcomes. Developing the appropriate methods and indicators to assess the impact of community-based initiatives remains an important goal (Schorr 1994). Nevertheless, it is also important to keep in mind that evaluation results are only part of what governs program and policy design.
I will conclude, then, by suggesting that a more "policy-relevant" type of evaluation research should be asking questions that have hitherto not been adequately pursued in evaluation design: questions that focus a little less on how community-based initiatives can achieve modest goals while working against incredible odds and much more on changing those odds by identifying the socioeconomic conditions and policy environments under which local initiatives can be more effective vehicles for change. Such evaluation would be more contextual in nature, making social, economic, political, and geographic factors integral parts of the evaluation equation, rather than trying to control for them. It would actively explore how contextual factors influence the nature and effectiveness of community-based initiatives on several levels: Are the programs informed by theories about the relationship between individuals, institutions, and the larger environment? Do such factors as local labor markets, degree of political support for the initiative, strength of the local philanthropic and nonprofit sector, and degree of ethnic and class heterogeneity in the targeted neighborhood create opportunities and/or constraints in program effectiveness? What contextual factors does the initiative attempt to change? In devising ways of answering those and related questions, such evaluations would also take advantage of the recent increase in social scientific interest in macro and micro-level contextual analysis, as seen in the resurgence of literature exploring "neighborhood effects" on poverty and in efforts to measure such intangible concepts as group or neighborhood-based social networks and social capital. And such contextual analysis would of necessity be historical, examining how the legacy of past community activism has shaped the current generation, using lessons from experience to build comparative, longitudinal frameworks for evaluation, and understanding the initiatives as products of a particular configuration of historical circumstances that may help as well as hinder them.
In addition to exploring such issues as how community-based initiatives fare in varying macro-economic circumstances, contextual analysis would focus on local governance issues, exploring, for example, the links between metropolitan and region-wide development plans and the economic fate of poor neighborhoods. Conversely, such analyses would provide empirical means of assessing prospective policies for the impact they would have on poor neighborhoods and on the possibility for generating job creation and resource development in those neighborhoods. In these and other ways, a more contextual approach would shift the lens of evaluation away from a sole focus on individual initiatives and onto the economy and the policy environment, away from the behavior that takes place within poor communities and onto the choices being made--or not being made--by policymakers and funders. In effect, a more contextual approach to evaluation would take stock of one of the most important lessons from historical experience: that even the most effective community-based initiatives cannot, by themselves, reverse the interacting economic, social, and political trends that have generated growing inequality. This is an important starting point for a discussion of future directions in evaluating, and learning from, community-based initiatives.
An earlier version of this paper was presented at the Workshop on Comprehensive Community Initiatives held at the Aspen Institute in August 1994. I am grateful to the workshop participants for their helpful comments. I am especially indebted to Anne Kubisch, Donald Schon, and Lisbeth Schorr for their detailed and thought-provoking comments.
I would also like to thank the Ford Foundation for permission to quote from sources in its archival holdings.
- Throughout this essay, I use the term "community-based initiatives" to refer to a diverse range of programs, including the Progressive Era settlement house movement, a variety of community action programs sponsored by foundations and the federal government in the 1950s and '60s, model cities, community development corporations, and the comprehensive cross-systems reform efforts that have emerged in several communities over the past few years. Without diminishing the differences among these very diverse initiatives, my analysis suggests that they share several common assumptions and pose similar evaluation challenges.
Aaron, Henry J. 1978. Politics and the Professors: The Great Society in Perspective. Washington, DC: The Brookings Institution.
Abt Associates. 1973. "An Evaluation of the Special Impact Program: Final Report." Cambridge, Mass.: Abt Associates.
Baldwin, Sidney. 1968. The Rise and Decline of the Farm Security Administration. Chapel Hill: University of North Carolina Press.
Barss Reitzel and Associates, Inc. 1969. "Community Action and Institutional Change." An Evaluation Prepared for the Office of Economic Opportunity, July.
Bremner, Robert. 1956. From the Depths: The Discovery of Poverty in the United States. New York: New York University Press.
Bureau of the Budget. Bulletin # 66-3 (October 12, 1965). Austin, Tex.: Lyndon Baines Johnson Library Archives.
Campbell, Donald T. and Stanley, Julian C. 1966. Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally.
Carson, Mina. 1990. Settlement Folk: Social Thought and the American Settlement Movement, 1885-1930. Chicago: University of Chicago Press.
Cloward, Richard A. and Lloyd E. Ohlin. 1960. Delinquency and Opportunity: A Theory of Delinquent Gangs. New York: Free Press.
Cook, Thomas D. and Donald T. Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Skokie, Ill.: Rand McNally.
Davis, Allen F. 1984. Spearheads for Reform: The Social Settlements and the Progressive Movement 1890–1914. New Brunswick: Rutgers University Press.
Deutscher, Irwin. n.d. "The Evaluation of Social Programs: Some Consequences of Historical Insensitivity, Naiveté and Contempt." Unpublished paper.
Edelman, Peter B. and Beryl A. Radin. 1991. "Effective Services for Children and Families: Lessons from the Past and Strategies for the Future." In Effective Services for Children and Families: Report of a Workshop. Washington, DC: National Academy Press.
Fishman, Michael E. and Daniel H. Weinberg. 1992. "The Role of Evaluation in State Welfare Reform Waiver Demonstrations." In Evaluating Welfare and Training Programs, ed. Charles F. Manski and Irwin Garfinkel. Cambridge: Harvard University Press.
Ford Foundation. 1964. "Public Affairs: Gray Areas Program." Unpublished Report #2845, Ford Foundation Archives, September.
Frieden, Bernard J. and Marshall Kaplan. 1975. The Politics of Neglect: Urban Aid from Model Cities to Revenue Sharing. Cambridge: M.I.T. Press.
Gilbert, James. 1986. Cycle of Outrage: America's Reaction to the Juvenile Delinquent in the 1950s. New York: Oxford University Press.
Gilbert, Jess. 1994. "Democratic Planning in the New Deal: The Federal-County Agricultural Planning Program." Paper presented at the Newberry Library Rural History Seminar, Chicago, February 1994.
Haar, Charles M. 1975. Between the Idea and the Reality: A Study in the Origin, Fate and Legacy of the Model Cities Program. Boston: Little, Brown.
Halpern, Robert. 1994. Rebuilding the Inner City: A History of Neighborhood Initiatives to Address Poverty in the United States. New York: Columbia University Press.
Hackett, David. 1970. Oral History Interview, Boston, Massachusetts, July 22, 1970. John F. Kennedy Memorial Library.
Harding, Bertrand. Letter to Senator Joseph S. Clark, May 1968. Harding Papers. Austin, Tex.: Lyndon Baines Johnson Library.
Haskell, Thomas. 1977. The Emergence of Professional Social Science: The American Social Science Association and the Nineteenth Century Crisis of Authority. Urbana: University of Illinois Press.
Haveman, Robert. 1987. Poverty Policy and Poverty Research: The Great Society and the Social Sciences. Madison: University of Wisconsin Press.
Kershaw, David and Jerilyn Fair. 1976. The New Jersey Income-Maintenance Experiment. Vol. 1. New York: Academic Press.
Kirkendall, Richard. 1966. Social Science and Farm Politics in Age of Roosevelt. Columbia: University of Missouri Press.
Lageman, Ellen C. 1989. The Politics of Knowledge: The Carnegie Corporation, Philanthropy and Public Policy. Chicago: University of Chicago Press.
Lefcowitz, Myron. 1969. "CAP Evaluation: Some Comments." Unpublished paper. Madison, Wisc.: Institute for Research on Poverty Archives.
Levine, Robert A. 1968. "Memorandum to RPP&E Staff," January 1968. Madison, Wisc.: Institute for Research on Poverty Archives.
-----. 1970. The Poor Ye Need Not Have with You. Cambridge: M.I.T. Press.
Manski, Charles F. and Irwin Garfinkel, eds. 1992. Evaluating Welfare and Training Programs. Cambridge: Harvard University Press.
Marris, Peter and Martin Rein. 1967. Dilemmas of Social Reform: Poverty and Community Action in the United States. New York: Atherton Press.
Miller, Walter. 1968. "The Elimination of the American Lower Class as National Policy: A Critique of the Ideology of the Poverty Movement of the 1960s." In On Understanding Poverty: Perspectives from the Social Sciences, ed. Daniel P. Moynihan. New York: Basic Books.
O'Connor, Alice. Forthcoming. "Community Action, Urban Reform, and the Fight Against Poverty: The Ford Foundation's Gray Areas Program." Journal of Urban History.
Office of Economic Opportunity (OEO). 1965. "Yale University." OEO Agency Files. Austin, Tex.: Lyndon Baines Johnson Library.
-----. 1968. "Request for Proposals for Evaluation of the Special Impact Program." Harding Papers. Austin, Tex.: Lyndon Baines Johnson Library.
Peirce, Neal R. and Carol F. Steinbach. 1987. Corrective Capitalism: The Rise of America's Community Development Corporations. New York: Ford Foundation.
Quane, James. 1994. "Back From the Future: Can Evaluation Survive Dissention in the Ranks?" Paper presented at the Annual Conference of the American Evaluation Association, Boston, Massachusetts, November 2–5, 1994.
Redenius, Richard. 1970. "Memorandum to John Wilson," March 20, 1970. Madison, Wisc.: Institute for Research on Poverty Archives.
Resource Management Corporation. 1969. "Evaluation of the War on Poverty: Status and Prospects at the OEO." Santa Monica, Calif.: Resource Management Corporation.
Rossi, Peter H. 1994. "The War Between the Quals and the Quants: Is a Lasting Peace Possible?" In The Qualitative -Quantitative Debate: New Perspectives, ed. C. C. Reichardt and S. F. Rallis. New Directions for Program Evaluation, No. 61. San Francisco: Jossey-Bass.
Rossi, Peter H. and Walter Williams, eds. 1972. Evaluating Social Programs: Theory, Practice and Politics. New York: Seminar Press.
Schorr, Lisbeth B. 1994. "The Case for Shifting to Results-Based Accountability." Improved Outcomes for Children Project. Washington, DC: Center for the Study of Social Policy.
Sheldon, Richard. 1974. Oral History Interview, September 30, 1974. Ford Foundation Archives.
Trolander, Judith A. 1987. Professionalism and Social Change From the Settlement House Movement to Neighborhood Centers, 1886 to the Present. New York: Columbia University Press.
Urban Institute. 1976. Evaluating CDCs: A Summary Report. Washington, DC: Urban Institute.
Vanecko, James J. 1970. "Community Organization in the War on Poverty: An Evaluation of a Strategy for Change in the Community Action Program." Final Report of a National Evaluation of Urban Community Action Programs Prepared for the Office of Economic Opportunity. Chicago: National Opinion Research Center.
Vanecko, James J. and Bruce Jacobs. 1970. "The Impact of the Community Action Program on Institutional Change: Assistance to Community Organization as a Successful Strategy." Joint Report from NORC and Barss, Reitzel and Associates, Inc. Chicago: National Opinion Research Center.
Vidal, Avis C. 1992. Rebuilding Communities: A National Study of Urban Community Development Corporations. New York: Community Development Research Center.
Watts, Harold. 1970. "Memorandum to John Wilson," April 1970. Madison, Wisc.: Institute for Research on Poverty Archives.
Watts, Harold W. and Albert Rees, eds. 1977. The New Jersey Income-Maintenance Experiment. Vols. 2 and 3. New York: Academic Press.
Weiss, Carol H. 1972a. Evaluation Research: Methods for Assessing Program Effectiveness. Englewood Cliffs, N.J.: Prentice-Hall.
-----, ed. 1972b. Evaluating Action Programs. Boston: Allyn and Bacon.
-----. 1993. "Where Politics and Evaluation Research Meet." Evaluation Practice 14, no. 1: 93–106.
Wholey, Joseph S., John W. Scanlon, Hugh G. Duffy, James S. Fukumoto, and Leona M. Vogt. 1970. Federal Evaluation Policy: Analyzing the Effects of Public Programs. Washington, DC: Urban Institute Press.
Wiseman, Michael, P. L. Szanton, E. B. Baum, R. Haskins, D. H. Greenberg, and M. B. Mandell. 1991. Research and Policy: A Symposium on the Family Support Act of 1988. Madison, Wisc.: Institute for Research on Poverty.
Ylvisaker, Paul. 1963. "Remarks to the Conference on Urban Extension," October 1963. Staff Speeches Collection Ford Foundation Archives. New York: The Ford Foundation.
Back to New Approaches to Evaluating Community Initiatives index.
Copyright © 1999 by The Aspen Institute
Comments, questions or suggestions? E-mail firstname.lastname@example.org.
This page designed, hosted, and maintained by Change Communications.