New Approaches
to Evaluating
Community
Initiatives

Volume 1
Concepts, Methods, and Contexts


Using Community-Level Indicators of Children's Well-Being in Comprehensive Community Initiatives
Claudia J. Coulton

Children and their families live within local communities, which are often explicit targets of change in comprehensive community initiatives. It is indeed at the level of the local community that many of the processes that affect children transpire. Children interact with neighbors; participate in local institutions; receive social, health, and educational services; develop a sense of safety and belonging; form a vision of their opportunities; know what is expected of them and what they can expect from others. Parents' implicit understanding of the importance of local community is reflected in the serious thought that many of them give to their residential choices. Yet the locational options of a significant number of families are constrained by racism, low income, insufficient information, or public policy. Describing the variation in child well-being across local communities is, thus, a crucial task for comprehensive community initiatives.

Developing relevant and sensitive indicators of child well-being at the local community level, however, poses numerous conceptual, methodological, and practical challenges. Those challenges, as well as the important benefits of undertaking this level of measurement, are the focus of this paper.

Local Communities as Units of Analysis

Although the term "community" is a social rather than a geographic unit, this paper focuses on local communities that are bounded spatially. Such communities can serve as units for the measurement of child and family well-being. The local community of interest is typically the neighborhood, although for some purposes it is a political jurisdiction such as a ward or town, or a service delivery zone such as a school or health district.

Neighborhood boundaries are often difficult to draw because there is little consensus about what constitutes a neighborhood. Social scientists hold varying perspectives on the degree to which the term implies homogeneity, social interaction, and place identity on the part of the residents (White 1987). Most definitions of neighborhood imply a degree of social cohesion that results from shared institutions and space, but it is also widely accepted that neighborhoods differ in their levels of community social organization and integration (Lyon 1987). Further, it seems that neighborhoods that are least cohesive and organized may be the poorest community environments for rearing children (Coulton et al., forthcoming; Garbarino and Sherman 1980; Sampson 1992).

Despite the definitional ambiguities of neighborhood or other meaningful localities, local community indicators typically require geographic boundaries. These boundaries may be phenomenological, interactional, statistical, or political.

At the phenomenological level, each resident has a sense of the boundaries that are personally meaningful. These vary even for the same individual depending upon the context (Galster 1986). However, under some circumstances it is possible to use the consensus of residents as the basis for drawing geographic boundaries for neighborhoods. In our research on Cleveland's neighborhoods we have found some areas where there is considerable agreement among neighbors on the boundaries of their neighborhood while in other locales neighborhood boundaries seem virtually idiosyncratic. Consensus seems to be greater in areas with higher levels of community identity and attachment (Korbin and Coulton 1994). Where adequate consensus exists, the residents' perceived boundaries can be used to form units for the development of indicators. However, this consensus may change over time, making consensual boundaries problematic for tracking changes in communities over time.

A second approach to generating community-area boundaries is to use the patterns of social interaction of residents. This involves a process of "mapping locally-based social interaction onto a spatial grid" (Entwisle 1991). Friendship patterns and daily activities have both been used as methods of tying interaction patterns to spatial locations.

Statistical definitions of local community areas are a third approach. Census tracts have been most widely used to date in local indicator development even though concerns have been raised about the degree to which these units resemble the space that is meaningful to residents (Tienda 1991). Nevertheless, census tracts have proven quite useful for local planning and research on neighborhoods (White 1987; Kasarda 1993; Galster and Mincey 1993; Pandey and Coulton 1994). Block groups have also served as proxies for neighborhoods in some studies (Taylor, Gottfredson, and Brower 1984).

In cities such as Chicago, Cleveland, Philadelphia, and elsewhere, designated neighborhoods traditionally are used for planning purposes, and the Census Bureau has supported such local designations as well. Unlike census tracts, these designated neighborhoods can vary considerably in size but take into account local sentiments on natural boundaries. Although designated neighborhoods do not always match resident perceptions, they have been used successfully in research (for example, Galster and Hesser 1982).

Political units such as wards, districts, and towns are a fourth approach to defining community areas but they can seldom be equated with neighborhoods and they lack a social meaning. Nevertheless, when community indicators are used for planning or evaluation, political units may be appropriate for analysis.

Choosing a set of geographic boundaries for community indicators depends upon several considerations. First, it is important that the unit be constant over time so that trends can be tracked. Second, it must be possible to allocate available data to the unit of analysis that is chosen. Third, the choice of units should be appropriate given the assumptions and purposes underlying the set of indicators. In this regard, varying conceptual perspectives on community indicators are discussed in the next section.

In the remainder of this paper I use the term "community area" to refer to the unit of geography that has been chosen, be it a census tract, neighborhood, or town. I use the term "community indicator" to refer to measures that are made on these units of geography.

Perspectives on Community-Level Management

Two quite different perspectives can be taken on community indicators for families and children. One I will label outcome orientation, the other contextual orientation.

The outcome orientation views community areas as valuable units for measuring the status of children on various social, health, and developmental outcomes. Comparing the status of children across community areas can reveal inequalities that suggest levels of need and indicate where resources should be targeted. Such comparisons also can point to differences in program effectiveness or practices across community areas. Those are the purposes for which community indicators are often obtained, and outcomes do indeed vary considerably across community areas within counties or metropolitan areas (Coulton and Pandey 1992).

This outcome orientation makes few assumptions about the relationships between community areas and their families and children. The outcome measures represent the status of a population of children who live in specified local areas. The ways in which the local communities themselves affect these outcomes remain unspecified.

An alternative view of communities, though, is to measure them as environments for families and children. This contextual orientation is based on the assumption, for which there is some empirical support, that community areas (for example, neighborhoods) can affect children and their families positively or negatively. Some of those effects are reflected in higher or lower rates of health, social, and developmental outcomes. But from this perspective it is the community structure and processes that are the relevant focus of measurement.

The contextual orientation makes some strong assumptions about how communities affect families and children. I have reviewed the several extant lines of research on the effects of neighborhoods on families and children in another paper (Coulton 1994), but I will summarize them here. For convenience, this summary discusses the research under four broad headings that are not mutually exclusive: compositional effects, community context of effective parenting, effects of stressful neighborhood environments, and community social organization.

First, recent interest in extreme-poverty neighborhoods has spawned a series of studies designed to determine whether neighborhood socioeconomic composition affects life chances of children over and above family background factors. Although not adopting any uniform theoretical perspective, these studies can be loosely classified as subscribing to a model of socialization processes within neighborhoods through adult, peer, or institutional influences (Jencks and Mayer 1990). Research on the socioeconomic composition of communities reveals that having sufficient affluent families in a neighborhood promotes school achievement, cognitive development, and the avoidance of teen childbearing (Brooks-Gunn et al. 1993; Clark 1992; Crane 1991; Duncan 1993). These studies have found that the positive effects of middle-class and affluent neighbors are more important as a context for children than the negative effects of having poor neighbors.

Second, the community context for parenting has been explored in several important studies. These studies shed light on how social networks, resources, local institutions, and environmental stressors shape parenting style. Parents adapt to dangerous and depleted environments by restricting their children's activities and isolating themselves from the surrounding area (Furstenberg 1993). These adaptations, while understandable and necessary for safety, may not promote academic achievement and future economic success. Also, individual parents who adopt effective child-rearing styles will not be as successful when they are surrounded by less effective parents. The distribution of effective parents differs across types of communities (Steinberg et al. 1992; Steinberg and Darling 1994).

Third, the negative effects on families and children of stressful conditions in poor, urban environments have been explored in many studies. Getting most attention in recent years has been the negative impact of chronic exposure to violence in the community (Martinez and Richters 1993; Garbarino et al. 1992; Zapata et al. 1992). Daily hassles, though less dramatic, have been found to be a significant cause of parental distress in poor neighborhoods, too (Caspi, Bolger, and Eckenrode 1987; Garbarino and Sherman 1980). Considerable work on how resourceful parents adapt to these stressful conditions is contained in this line of research as well. Kinship networks and neighbors are used quite effectively by some parents (Lee, Campbell, and Miller 1991; McAdoo 1986). For others, network relationships can actually be a further source of strain (Riley and Eckenrode 1986).

Fourth, community social organization is proving to be a useful framework for understanding the relationship between macro-structural change and the experience of families and children within neighborhoods (Sampson 1992). As neighborhoods decline economically, experience population turnover, and begin to contain large numbers of children in female-headed families, the community's internal control is diminished. Studies of crime and delinquency, in particular, support the contention that this diminishing control occurs through the effects of the macro-structure on processes within the community such as friendship networks, institutional participation, normative consensus, and monitoring of the environment (Bursik and Grasmick 1993; Sampson and Groves 1989; Sampson 1991). Dimensions of community structure such as economic resources, residential mobility, family structure, and age distribution of the population have been linked to varied childhood outcomes including child maltreatment, delinquency, teen childbearing, and low-birthweight rates (Coulton et al., forthcoming).

Regardless of whether an outcome orientation or contextual orientation is chosen, caution must be exercised in interpreting differences among communities. Community selection processes are complex and difficult to isolate but can be important explanations for variation among local areas (Tienda 1991). On the one hand, self-selection of families into particular communities and forced selection of communities by families due to discrimination, affordability, or public policy may be responsible for variation in outcomes or community context. On the other hand, pre-existing differences in social, economic, and institutional structures and processes can affect children within communities regardless of the selection processes that lead to their presence in a particular community in the first place.

Even with these caveats in mind, community-area boundaries need to be consistent with the orientation that is chosen. The outcome and contextual orientation also call for somewhat different types of indicators to represent the well-being of children in communities. Examples of such indicators are provided in the next section.

Community Indicators of Children's Well-Being

Community indicators are measures of child and family well-being tied to local community areas. Indicators that reflect an outcome orientation include social, health, and developmental outcomes for the population living in local areas. Indicators that reflect a contextual orientation include measures of the community structure and process that are believed to affect children and family life.

Neighborhoods and other community areas can change rapidly. Thus, community indicators from either orientation should be calculated and available annually or bi-annually.

Outcome Indicators

Outcome indicators that are useful at the community level include many of those that have been proposed for use at the national and state levels. However, data sources and availability differ at the community level, placing limitations on what is practical. In particular, large-scale surveys seldom have sufficient sample sizes to make estimates for small areas. Furthermore, the base rates of some outcomes useful at the state or national level are too low to allow valid measurement at the local community area level.

Table 1 (170k) offers a list of indicators that we have used in Cleveland's neighborhoods and the surrounding metropolitan area as an illustration of what is currently possible. This is not an exhaustive list. The indicators are organized according to the general categories suggested by Zill (1991). Our system in Cleveland allows calculation of the indicators for block groups, census tracts, neighborhoods officially designated as planning areas, or any other sub-areas of the county that can be aggregated from block groups such as areas defined by residents or neighborhood leaders. This type of flexibility is highly desirable in local indicator work.

Measures of health and safety of children are the types of outcomes most readily available at the local level. Local health departments, hospitals, police jurisdictions, child welfare agencies, and coroners are all potential collaborators for developing indicators in this area. Infant death rates and low-birthweight rates and other measures of infant and child health can be calculated from birth and death certificates that are readily available and can be geo-coded for aggregation into small areas. Rates from year to year are quite labile in small areas and three-year averages are preferred.

Child maltreatment rates can be calculated from official reports and are reasonably comparable in terms of definitions and criteria within one agency jurisdiction. However, over- and under-reporting biases may differ across community areas and must be assessed carefully. (See Reporting Bias and Error, below.)

Trauma rates can be calculated for children if trauma registries exist in the emergency departments of most major hospitals serving the communities of interest. We are currently in the planning stages for such a registry system for Cleveland. The trauma events will be geo-coded using the home address of the patient and aggregated by age to yield rates for children. A seriousness threshold for inclusion in this indicator will need to be developed.

Child homicide rates and suicide rates as well as gun-related deaths for community areas can be calculated from the coroner's data. Because these are rare events, multiple years need to be averaged and rather large community areas must be used for analysis.

Measures of what Zill (1991) labels moral and social development and emotional development are more difficult to obtain at the local level without resorting to impractical neighborhood surveys. One useful measure, though, is the teen childbearing rate, which is based on births to teens per 1,000 females ages 12 to 17.

The delinquency rate is another possible indicator of moral and social development and is derived from court records that are classified as to level of offense and age of offender and then geo-coded. Delinquency filings are counted per 1,000 males and females ages 10 to 17. Further refinements of this indicator can include separate rates for males and females, separate rates for violent offenses, and direct age standardization.

Teen drug violation arrest rates are also available and can be calculated from arrest records of the police departments. Caution must be exercised when comparing these rates across police precincts or municipalities because police practices may differ. While these rates do suggest the communities in which youth are interacting with the criminal justice system as a result of drug-related activities, they cannot be used as valid measures of drug use or involvement as a whole.

Measures of cognitive development and academic achievement can be developed for communities in collaboration with local school systems. High school graduation rates require student-level data from the schools that are geo-coded so students can be assigned to neighborhoods. If multiple years of data files are available, counts of students entering the ninth grade in each community area can be divided into counts of students graduating four years later. Unfortunately, this only measures students' graduation rates and does not include persons who obtain a general educational development (GED) credential or who complete high school later.

Student performance is measured as the mean grade level achieved by students at selected grade levels on standardized achievement tests. These scores can be usefully compared across neighborhoods even though the role of standardized testing in education is undergoing rapid change. Nevertheless, longitudinal comparisons should be made cautiously.

These school-based measures are more practical in neighborhoods where most of the children attend public schools. We have found some neighborhoods in Cleveland, though, in which fewer than 50 percent of the children are enrolled in public schools. A valid school graduation indicator for those areas would require obtaining student data with home addresses from more than 30 private and parochial schools, all of whom have differing methods of data collection and storage. Such a task has been impractical thus far.

Youth employment can also be considered an indicator of achievement. It uses the decennial census to calculate the labor force status of young men and women, ages 16 to 25, who are not in school. Unfortunately, we have not yet found a measure of youth employment at the community level that is available more frequently than each decade. However, the use of the state reporting system related to unemployment compensation is being explored.

The economic status of families, a final category suggested by Zill (1991), is available at the community level from the decennial census. Family poverty rate, child poverty rate, and family median income can all be calculated easily. However, we know that the actual economic status of families in a neighborhood can change rapidly during a decade. Therefore, we are developing a model for estimating these rates in each subsequent year using variables derived from Aid to Families with Dependent Children) (AFDC) and food-stamp recipients in each community area. Our previous experience in estimating overall poverty rates for census tracts between 1980 and 1990 showed fairly high accuracy. Public assistance counts for various programs were benchmarked to census counts of poor persons and the model was used as an estimator in intercensal years.

The child public assistance rate is an additional indicator of the economic status of families that is available yearly. Public assistance families typically have income that is well below the poverty threshold. This indicator reflects, therefore, the children with the most extreme economic deprivation. It can be calculated using the monthly average caseload of children receiving public assistance in each community area divided by the number of children living in the area. The monthly public assistance case files are geo-coded and counts are produced for the desired units of geography.

Contextual Indicators

The search for practical measures of community environments has an extended history (Rossi 1970). However, the identification of indicators of community context that may be important factors in the well-being of children requires either scientific research or a set of assumptions that link aspects of community structure and process to effects on families and children. Unfortunately, research that pinpoints those aspects of community that affect children and families has yet to yield definitive connections. (See Connell, Aber, and Walker in this volume.) Nevertheless, the research described in the section on perspectives can be used to suggest a set of indicators of community context that are worthy of experimentation.

Table 2 (190k) presents a set of indicators of community context for children that we have been exploring in Cleveland's neighborhoods. At the aggregate level, they have been linked to rates of child maltreatment, teen childbearing, low birthweight and delinquency (Coulton et al., forthcoming). Ethnographic studies conducted in selected neighborhoods representing varying levels of risk for children generally confirm that these factors coincide with residents' perceptions of the neighborhood as "good or bad places for raising children" (Korbin and Coulton 1994). The data sources for these contextual indicators are much more limited at the community level than they are for the outcome types of indicators. Since many come from the decennial census they only are available at ten-year intervals.

Economic status of neighbors is suggested as an important contextual indicator for the well-being of children both in the compositional effects and the community social organization research. The compositional research, though, emphasizes that measurement needs to reflect not only overall economic status, such as median income or poverty rate, but must also include an indicator of the presence of middle-class or affluent neighbors (Brooks-Gunn et al. 1993; Clark 1992; Crane 1991; Duncan 1993).

The importance of the age and family structure of a community is also implicated as an important factor in the well-being of children. Specifically, community areas with a higher percentage of elderly persons, a more balanced ratio of men to women, a greater percentage of two-parent families, and a more favorable adult–child ratio are found to correlate with lower rates of poor outcomes and to be perceived by residents as promoting a better environment for children (Coulton et al., forthcoming; Korbin and Coulton 1994).

Several indicators of residential mobility are deemed important because population turnover has been repeatedly connected to aspects of community process (Freudenburg 1986). Most important as a context for children is the negative effect of residential mobility on parent-to-parent networks and support for institutions serving children.

Indicators of environmental stress are potentially useful because they may directly affect parents' ability to protect and nurture their children and because of the negative effects of these factors on community social organization. Substandard and abandoned housing is associated with growing disorder and fear of crime (Skogan 1990). High levels of personal crime and drug selling are seen by residents as a source of anxiety and distraction that affects their parenting (Furstenberg 1993; Korbin and Coulton 1994).

On the positive side, some contextual supports for effective parenting are also suggested. Parental involvement with social institutions, neighbor-to-neighbor relations, and community resources for families are but a few of the features of community that seem important (Zill and Nord 1994; Garbarino and Sherman 1980). Unfortunately, few data sources are readily available for measuring resources at the local community level. Community resources for children have been studied in national and local surveys, but surveys are seldom practical for local indicators. New sources of data need to be developed to measure these aspects of community context.

Relationship of Contextual and Outcome Indicators

The distinction between contextual and outcome indicators is not as clear as may be implied by the discussion thus far. A given indicator could be viewed as contextual or outcome depending upon the circumstances. For example, decreasing the number of children in poverty is often an outcome objective of community initiatives, especially those that include job creation and workforce development components. Yet, a similar indicator, the economic status of the families in a neighborhood, is thought to represent an important aspect of the context for child-rearing. Improvement in this aspect of context would presumably benefit children even if their own families remained poor.

Contextual and outcome indicators may also have complex and difficult-to-detect relationships with one another. While it might generally be assumed that context affects outcome, the direction of the influence may be the other way around. For example, the presence of large numbers of vacant and boarded houses may be viewed as an indicator of declining community social organization, and drug trafficking by teens a consequence that can flourish in such a context. However, heavy drug trafficking may actually be a reason that owners abandon their property and police often close and board homes of known drug dealers.

Because of these complications, monitoring both types of indicators is desirable. Panel studies can help to unravel the reciprocal relationships among processes and outcomes (for example, Pandey and Coulton 1994) but undoubtedly there are unique stories that need to be told within specific communities. Ethnographic and observational studies can be useful in interpreting changing indicators in specific locales (Korbin and Coulton 1994).

Methodological Considerations for Community-Level Indicators

There are numerous methodological problems pertinent to making measurements of child and family well-being for small geographic areas such as communities. While they are formidable, many can be managed through careful definition and interpretation.

Assignment of Geographical Location

Because definitions of community areas typically have some geographic boundaries, data used for community indicators must be suitable for assignment to geographic units. Administrative agency data, which are often the preferred source for local community indicators, must be obtained with the street addresses intact. The addresses can be geo-coded using the TIGER files (census files containing street addresses) and aggregated to the desired geographic boundaries--for example, block groups, census tracts, resident-defined neighborhoods, wards, catchment areas, school zones, and so on.

In our experience, agencies differ considerably in the accuracy and validity of their addresses for this purpose. Problems include the timeliness of the address, whether it is verified or not, and administrative conventions that can be misleading. For example, some agencies overwrite addresses when there is a move so the address is the most current one. The most current address, however, may differ from the address at which an event of interest occurred, such as an arrest or a child abuse incident. Also, the address given may not be a home address, which is usually the one desired for community analysis, but an office or agency where a service was delivered. Finally, when the address is obtained by the agency for informational rather than service delivery purposes, accuracy may be low and a substantial number of addresses may not be codable without considerable effort expended to correct errors.

Finally, for some indicators, there may be ambiguity as to which geographic area to assign a case. For example, infant deaths are ordinarily assigned to the community area in which the death occurred. However, since infant death is highly related to conditions in the prenatal period, it may be more useful to assign the death to the community area in which the birth occurred.

Small Area Limitations

The geographic units for community indicators are typically fairly small. Block groups vary considerably in their population size but may have anywhere from just a few to hundreds of housing units. While census tracts have an average population of 4,000, many are quite a bit smaller, especially in central cities that have been losing population. Designated neighborhoods can be of any size depending on the methodology used for drawing the boundaries. This small geography poses several limitations.

Unavailability of Survey Data. No national surveys are available with sample sizes that are adequate to provide valid estimates for small areas such as neighborhoods and census tracts. Even statewide or metropolitan-wide surveys are seldom adequate for these purposes. Only the decennial census, in which 15 percent of the households complete the long form, provides some estimates of family structure and economic status that can be used for small geographic areas. The Public Use Microdata (PUMS) 5 percent sample from the census can be used to make estimates at the sub-city level, but these areas of 100,000 minimum population seldom correspond to any meaningful definition of community area.

Surveys are periodically undertaken locally that are capable of generating measures of child and family well-being for small, geographically defined communities. However, it is seldom feasible to draw adequately sized samples for all neighborhoods in a region so a multistage sampling method can be used. Thus, these surveys do not provide measures for all community areas but only for a sample, selected randomly or otherwise. Furthermore, these expensive surveys are seldom repeated and so they do not yield measures over time, which is desirable for indicators.

Low Base Rates. Outcomes of significant interest such as childhood deaths from trauma can show extreme variation in rates because they are rare. Since aggregating geography to achieve sufficient population size would often negate the purpose of community-area analysis, multi-year averages must be used to obtain a stable trend. The disadvantage, of course, is that important changes in conditions may be obscured in the short run.

Unequal Population Sizes. A third problem is the fact that meaningful geographic units often have widely differing population sizes. The stability or reliability of an indicator will be better in larger areas than in smaller areas. An extreme rate in a smaller area must be viewed with considerable caution. For some purposes, such as statistical modeling, the geographic units can be weighted for their population size, but such weighting does not typically make sense when the indicators are being used for local planning or evaluation purposes.

Reporting Bias and Error

Although error and bias must be considered in all work on indicators, two problems are particularly troubling at the local community level. First, because local community indicators rely so heavily on administrative agency data, they are beset by the reporting bias and error in those data sources. The nature of these problems is likely to vary from one indicator to another. Birth and death certificates, for example, are known to be quite complete. However, cause of death on death certificates and information about the mother's health contained in birth certificates are prone to error. These errors differ depending upon the hospitals and physicians involved in their completion. Thus, the degree of error will differ in an unknown way across community areas.

Reports of criminal or deviant acts are subject to the most severe and troubling sorts of bias. Police reports and court records are known to underestimate the true levels of criminal and violent events (O'Brien 1985). More importantly, they are also biased by differences that may exist across jurisdictions in victims' or observers' tendencies to report (Sampson 1985) and government officials' tendency to file reports and take action (Sherman 1989; O'Toole, Turbett, and Nalpeka 1983). Unfortunately, the direction of the bias in each of the community areas cannot be known but could account for some part of the observed differences.

The problem of errors and bias in administrative records requires careful investigation in each instance. Few generalizations can be made across regions. Generally, though, errors will be fewer when the data element used serves a mandated function or vital purpose of the agency. Information gathered by the agency for descriptive purposes only can often be misleading due to large amounts of missing data or coding errors. Reporting bias is particularly troubling when the direction of the bias is not the same in all community areas.

It is desirable, therefore, that efforts be made to validate widely used indicators based on administrative records against other data sources. Specially designed community surveys can be useful for establishing the validity of indicators derived from administrative agencies or other sources. For example, Jill Korbin and I have a study in the planning stage that will use a survey instrument to measure aspects of child abuse and neglect. The survey will be conducted in a random sample of neighborhoods whose rates of child abuse and neglect have been calculated based on official reports to the county authorities. These two sources of data can be compared to illuminate the issue of reporting bias and error in both the survey and administrative agencies.

Another example of validating administrative data with another source are the infant mortality reviews that are being carried out as a part of Cleveland's Healthy Start Program and are being performed in other Healthy Start cities. In Cleveland, the infant mortality review has revealed considerable variation across hospitals and physicians in classifying causes of death and in deciding what is a live birth. When corrected, the quality of this source of administrative data will be improved.

An additional issue that is pertinent to local community indicators is the amount of undercounting and missing data. This problem has received considerable attention with respect to the decennial census. Most troubling for community indicators is that the amount of undercounting and missing data are not uniform across community areas. Census counts are more likely to be undercounts when they pertain to young men and minorities in central city neighborhoods, for example. Furthermore, the amount of missing data that the census bureau imputes is greater in low-income, minority neighborhoods leading to differential reliability of census indicators (White 1987). Adjusting for the estimated undercount in some neighborhoods may be necessary but also introduces another source of error since the true undercount cannot be known in each location.

Small Area Population Estimates and Change

Community indicators are often reported as rates in which the area's population is used as the denominator. Rates may be reported per 100, 1,000, or 100,000 persons. Unfortunately, population estimates are not universally available at the block group or census-tract level between censuses, so rates in non-census years will be less valid. While established methods of population estimation are used at state and county levels, their application to areas as small as block groups, census tracts, or neighborhoods has not been widespread (Heeringa 1982).

Nevertheless, the sources of data needed to perform these population estimates can generally be obtained for community areas through geo-coding. The housing unit method of population estimates, for instance, can use building and demolition permits, utility hookups and disconnections or county assessor tax records to update housing unit counts post-censally. The component-cohort method relies on birth and death certificates. While estimating small-sized areas results in greater error, there is evidence that useful estimates are possible (Smith and Cody 1994).

Equally important as changes in the size of the population are shifts in the composition of the population. For example, the poverty rate of residents of a neighborhood may rise due to declining wages of existing residents, in-migration of disadvantaged residents, out-migration of more affluent residents, or departure of previously poor residents who recently became more affluent, perhaps as a result of an initiative. Tracking changes in population size or poverty rates will not reveal what combination of processes are operative, yet the processes themselves are important to understanding comprehensive community initiatives. For example, using a method developed by Jargowsky and Bane (1990), we estimated that approximately one-third of Cleveland's growth in extreme-poverty neighborhoods in the 1980s was due to the flight of the middle class as opposed to actual decline in earnings of area residents or in-migration of poor from elsewhere (Coulton, Pandey, and Chow 1990). More sensitive and precise methods of decomposing population changes are needed to properly assess the impact of interventions on these types of indicators.

Standardization

Community-level indicators are applied to areas of widely differing demographic makeup. In some instances, therefore, it may be useful to apply age or gender standardization. Small geographic areas often display marked variability in age and gender distribution. Some childhood indicators are sensitive to the particular age distribution that is present in the community. For example, teen births are concentrated in older teens and occur with less frequency in younger teens. Therefore, if two communities have a similar number of teens but the teens in one community tend to be older, the community with older teens would be expected to have a higher teen birthrate. Age standardization can compensate for differing age distributions and is probably worth the extra computational steps when children's ages put them at greater or lesser risk for particular outcomes.

Certain indicators may also be sensitive to gender distribution in the community. For example, delinquent behavior is known to be more frequent among boys. Gender-specific rates should be calculated for these types of indicators.

Other forms of statistical adjustment have been suggested for indicators, such as adjusting outcome indicators for the economic status or ethnicity of the population (Zill 1991). Such adjustment must be done cautiously, since it can obscure important ways in which more affluent communities differ from lower-income communities or ways in which some communities may be favored over others in resource distribution.

However, where community indicators are being used for evaluating specific programs, known risk factors for a particular outcome that are not amenable to the effects of the intervention would be suitable factors to use in adjustment. For example, mothers' age, ethnicity, and educational attainment are known risk factors for infant mortality. Community area rates of infant mortality could be adjusted for these factors in an evaluation of an infant mortality prevention program.

Corruption of Indicators

Many of the community indicators mentioned in this paper are collected by agencies of government for their own purposes. Indicators can become corrupted when the employees within the agency believe they are being judged by an indicator and act to change the indicator itself but not the underlying phenomenon it is supposed to measure. Indicators can also be affected when an initiative leads to less underreporting, such as increased reports of child abuse as community residents become more aware of the problem.

A careful examination of each community indicator is needed to determine the extent to which corruption could become a source of invalidity. To the degree that the indicator data come from agencies not directly involved in the comprehensive community initiative, direct corruption by employees is unlikely. However, more subtle influences within each agency may result in corruption or changes in reporting rates, and it is important that these be taken into account when interpreting trends in the indicators at the community level.

Role of Community Indicators in Evaluating Comprehensive Community Initiatives

Community indicators can contribute to understanding comprehensive community initiatives, but they must be used very carefully. It is most important that they be interpreted within a framework of metropolitan and regional dynamics. A neighborhood that is the target of an initiative cannot be viewed in isolation because it will be affected by the trends in surrounding areas. Furthermore, changes in a target neighborhood may affect contiguous areas and this fact, also, cannot be ignored.

The application of community indicators to comprehensive community initiatives should also be embedded within a long-term perspective. Specifically, target neighborhoods have experienced a history of change. The patterns and rates of past changes must be taken into account when prospectively tracking changes into the future. Expectations regarding improvement in key indicators must be based on the reality of past experience and trends.

Analysis and Interpretation of Community-Level Indicators

The analysis of community indicators can take several forms. First, each target neighborhood and surrounding areas can be examined over time to determine the amount and direction of change in each indicator. When areas are small, multi-year averages must be used and a fairly long-time trend is needed to detect significant variation. Nevertheless, for practical purposes neighborhood residents and organizations are often interested in monitoring these types of trends to get a sense of whether they are making desired progress.

Second, target neighborhoods and surrounding areas can be compared with one another cross-sectionally on each indicator. Areas can be ranked on selected indicators and maps can be used to determine the location of communities that are relatively high and relatively low on the various indicators. Cross-sectional analyses of this type are often useful for planning purposes such as choosing locations for programs or deciding which issues should receive priority. Also, they allow areas that are performing poorly to identify areas that are performing better and seek their advice and assistance for improvement. These cross-sectional comparisons must be made cautiously, though, due to differences in reporting biases and an expected amount of random variation at any given period in time.

Third, community areas can be grouped according to their similarities on a set of indicators. Such clusters can aid community leaders in recognizing the interrelationships among several aspects of children's well-being. The recognition that several troubling outcomes or conditions are concentrated in a few areas can lead to greater collaboration and service integration. Maps that allow overlaying of several indicators can be powerful visual aids in this process and promote community approaches to problem solving.

Fourth, panel studies of change in multiple indicators across multiple community areas are possible (Pandey and Coulton 1994). Such analyses can suggest the degree to which change in an indicator leads or lags behind change in other indicators. This knowledge can allow communities to anticipate improvement or deterioration and to react accordingly.

Finally, indices that capture the metropolitan-wide distribution of community indicators are quite useful for understanding the regional context for a comprehensive community initiative. For example, an important concern today is that poor children and their families are often isolated from the rest of the population in inner-city enclaves and this concentration of families at risk is particularly detrimental (Wilson 1987). A commonly used method of determining this level of concentration is to establish a threshold that is considered detrimental. With respect to poverty, census tracts with poverty rates of 40 percent or more are considered extreme-poverty areas (Kasarda 1993). A resulting index of concentration pertinent to child well-being is the percentage of all children in a county or metropolitan area who live in these extreme-poverty neighborhoods.

Another method of determining the local distribution of child well-being is to calculate a D index of dissimilarity (Lieberson 1980) for selected child outcomes. For example, the degree to which low-birthweight babies are segregated from babies whose weights are in the normal range could be calculated for a metropolitan area. The D index varies from 0 to 1 and represents the proportion of those babies who would have to be moved to achieve an even distribution of low-birthweight babies throughout the metropolitan area. It reveals the amount of segregation of childhood outcomes within the metropolitan area.

Reducing concentration and segregation could be an explicit objective of comprehensive community initiatives. Even if not an official target of change efforts, such concentration and segregation may be impediments to community improvement that are important to understand.

Community Involvement and Dissemination of Indicators

The development of community indicators requires involvement of local residents and leaders. They need to be involved in designating the appropriate geographic units to be studied as well as setting priorities regarding the types of indicators to be sought. Because of the demands of data cleaning and geo-coding, the generation of community indicators can be quite expensive and the community needs to influence the choices that are made.

Community residents and leaders also play an important role in interpreting trends and patterns that are observed in the indicators. They are aware of changes that are occurring in their communities that may account for the findings. They are also the vehicle for converting the information that indicators provide into action.

Since local and state administrative agencies are the source of much community indicator data, their involvement is essential, too. Collaborative relationships need to be established so that the agency as well as the community can benefit from the information that is generated. Data preparation may be burdensome for the agency and there are often serious concerns about the protection of confidentiality, especially since addresses are needed for geo-coding. These barriers can be overcome when all parties see the benefit of producing the information.

Getting community-level indicators into the hands of the public in a useful format is not a trivial problem, though. Because so many units of geography are of potential interest, the indicators need to reside in a system that can be quickly and easily manipulated, preferably by users as well as analysts. Taking Cleveland and its suburbs as an example, there are 1,535 block groups, 495 census tracts, 35 city neighborhoods, and 58 suburban municipalities. Indicators for each of these units need to be readily available if comprehensive community initiatives in target neighborhoods are to be properly examined within their metropolitan and historical contexts.

To accommodate these hierarchically structured units of geography and over 80 indicators for a thirteen-year period, we have created an interactive information system that is available to community-based organizations and initiatives (Chow, forthcoming). Using the system that we have named CANDO (Cleveland Area Network for Data and Organizing), leaders and ordinary citizens can generate their own geography and trend analyses for indicators of their choice. We also produce a hard-copy report of selected indicators each year for one unit of geography, Cleveland's thirty-five city neighborhoods.

Conclusion

Community indicators have a place in planning and evaluating comprehensive community initiatives. They are important adjuncts to other methods for monitoring progress and determining whether desired objectives are being achieved. To be interpretable, however, they must be available not only for target neighborhoods but also for surrounding areas. Historical trends must be linked to projections into the future as well.

The use of community indicators in isolation presents a danger because they are sensitive to a variety of economic and demographic processes that occur along with deliberate actions of any initiative. Also, it may appear that an initiative has not met its stated objectives when key indicators do not improve, but it is difficult to determine what the trends would have been in the absence of the initiative. Furthermore, because general knowledge about community change is sparse, expectations regarding the speed and amount of change in indicators may be unrealistic. Nevertheless, the availability of a comprehensive set of community indicators that can describe metropolitan trends will aid in the interpretation of findings from within target neighborhoods and contribute to the overall process of evaluation.


Note

Portions of this paper were prepared for a Conference on Indicators of Children's Well-Being, November 17–18, 1994, sponsored by The Office of the Assistant Secretary for Planning and Evaluation in the U.S. Department of Health and Human Services; Child Trends, Inc.; and the Institute for Research on Poverty, The University of Wisconsin-Madison.


References

Brooks-Gunn, J., G. J. Duncan, P. K. Klebanov, P. K., and N. Sealand. 1993. "Do Neighborhoods Influence Child and Adolescent development? American Journal of Sociology 99: 353–95.

Bursik, R. J., and H. G. Grasmick. 1993. Neighborhoods and Crime. New York: Lexington Books.

Caspi, A., N. Bolger, and J. E. Eckenrode. 1987. "Linking Person and Context in the Daily Stress Process." Journal of Personality and Social Psychology 52: 184–95.

Chow, J. Forthcoming. CANDO (Cleveland Area Network for Data and Organizing) Users Guide. Cleveland: Case Western Reserve University, Center for Urban Poverty and Social Change.

Clark, R. 1992. Neighborhood Effects on Dropping out of School among Teenage Boys. Washington, DC: The Urban Institute.

Coulton, C. 1994. "Effects of Neighborhoods in Large Cities on Families and Children: Implications for Services." In Children and Their Families in Big Cities. Ed. A. Kahn and S. Kammerman. Seminar held at the Columbia University School of Social Work, New York.

Coulton, C., S. Pandey, and J. Chow. 1990. "Concentration of Poverty and the Changing Ecology of Low-income, Urban Neighborhoods: An Analysis of the Cleveland Area." Social Work Research and Abstracts 26: 5–16.

Coulton, C., J. Korbin, J. Chow, and M. Su. Forthcoming. "Community Level Factors and Child Maltreatment Rates." Child Development.

Coulton, C. J. and S. Pandey. 1992. "Geographic Concentration of Poverty and Risk to Children in Urban Neighborhoods." American Behavioral Scientist 35: 238–57.

Crane, J. 1991. "Effects of Neighborhoods on Dropping out of School and Teenage Childbearing." In The Urban Underclass, ed. C. Jencks and P. E. Peterson, pp. 299–320. Washington, DC: The Brookings Institution.

Duncan, G. J. 1993. Families and Neighbors as Sources of Disadvantage in the Schooling Decisions of White and Black Adolescents. Ann Arbor: University of Michigan.

Entwisle, B. 1991. "Micro-Macro Theoretical Linkages in Social Demography: A Commentary." In Macro-Micro Linkages in Sociology, pp. 280–86. Newbury Park, Calif.: Sage Publications.

Freudenburg, W. R. 1986. "The Density of Acquaintanceship: An Overlooked Variable in Community Research." American Journal of Sociology 92: 27–63.

Furstenberg, F. F. 1993. "How Families Manage Risk and Opportunity in Dangerous Neighborhoods." In Sociology and the Public Agenda, ed. W. J. Wilson, 231–58. Newbury Park, Calif.: Sage Publications.

Galster, G. C. 1986. "What Is Neighborhood? An Externality-Space Approach." International Journal of Urban and Regional Research 10: 243–61.

Galster, G. C., and G. W. Hesser. 1982. "the Social Neighborhood: An Unspecified Factor in Homeowner Maintenance?" Urban Affairs Quarterly 18: 235–54.

Galster, G. C., and R. B. Mincey. 1993. "Understanding the Changing Fortunes of Metropolitan Neighborhoods: 1980 to 1990." Housing Policy Debate 4 303–48.

Garbarino, J., and D. Sherman. 1980. "High-risk Neighborhoods and High-risk Families: The Human Ecology of Child Maltreatment." Child Development 51: 188-98.

Garbarino, J., N. Dubrow, K. Kostelny, and C. Pardo. 1992. Children in Danger. San Francisco: Jossey Bass.

Heeringa, S. G. 1982. "Statistical Models for Small Area Estimation." Readings in Population Research Methodology, 5: 126–32.

Jargowsky, P. A., and M. J. Bane. 1990. "Ghetto Poverty: Basic Questions." In Inner-city Poverty in the United States, ed. L. E. Lynn and M. G. H. McGeary, 16–67. Washington, DC: National Academy Press.

Jencks, C., and S. E. Mayer. 1990. "the Social Consequences of Growing up in a Poor Neighborhood." In Inner-city Poverty in the United States, ed. L. E. Lynn and M. G. H. McGeary, 111-86. Washington, DC: National Academy Press.

Kasarda, J. 1993. "Inner-city Concentrated Poverty and Neighborhood Distress: 1970 to 1990." Housing Policy Debate 4: 253–302.

Korbin, J. E., and C. J. Coulton. 1994. Final Report: Neighborhood Impact on Child Abuse and Neglect. Washington, DC: National Center for Child Abuse and Neglect.

Lee, B. A., K. E. Campbell, and O. Miller. 1991. "Racial Differences in Urban Neighboring." Sociological Forum 6: 525–50.

Lieberson, S. 1980. A Piece of the Pie: Blacks and White Immigrants since 1880. Berkeley, Calif.: University of California Press.

Lyon, L. 1987. The Community in Urban Society. Chicago. Dorsey Press.

Martinez, J. E., and P. Richters. 1993. "The Nimh Community Violence Project: Children's Distress Symptoms Associated with Violence Exposure." Psychiatry 56: 22–35.

McAdoo, H. P. 1986. "Strategies Used by Black Single Mothers Against Stress." Review of Black Political Economy 14: 153–66.

O'Brien, R. M. 1985. Crime and Victimization Data. Beverly Hills, Calif.: Sage Publications.

O'Toole, R., P. Turbett, and C. Nalpeka. 1983. "Theories, Professional Knowledge, and Diagnosis of Child Abuse." In The Dark Side of Families: Current Family Violence Research, ed. D. Finkelhor, R. Gelles, G. Hotaling, and M. Strauss. Newbury Park, Calif.: Sage Publications.

Pandey, S., and C. J. Coulton. 1994. "Unraveling Neighborhood Change Using Two-wave Panel Analysis: A Case Study of Cleveland in the '80s. Social Work Research 18: 83–96.

Riley, D., and J. Eckenrode. 1986. "Social Ties: Subgroup Differences in Costs and Benefits." Journal of Personality and Social Psychology 51: 770–78.

Rossi, P. 1970. Community Social Indicators. Baltimore: Johns Hopkins University.

Sampson, R. J. 1985. "Neighborhood and Crime: The Structural Determinants of Personal Victimization." Journal of Research in Crime and Delinquency 22: 7–40.

-----. 1991. "Linking the Micro- and Macro-level Dimensions of Community Social Organization." Social Forces 70: 43-64.

-----. 1992. "Family Management and Child Development: Insights from Social Disorganization Theory." In Advances in Criminological Theory, vol. 3: 63–93. New Brunswick: Transition Press.

Sampson, R. J., and W. B. Groves. 1989. "Community Structure and Crime: Testing Social-Disorganization Theory." American Journal of Sociology 94: 775-802.

Sherman, L. 1989. "Hot Spots of Predatory Crime: Routine Activities and the Criminology of Place." Criminology 27: 27–56.

Skogan, W. G. 1990. Disorder and Decline. Berkeley, Calif.: University of California Press.

Smith, S. K., and S. Cody. 1994. "Evaluating the Housing Unit Method." APA Journal 60: 209–21.

Steinberg, L., and N. Darling. 1994. "the Broader Context of Social Influence in Adolescence." In Adolescence in Context, R. Silbereisen and E. Todt. New York: Springer.

Steinberg, L., S. D. Lamborn, S. M. Dornbush, and N. Darling. 1992. "Impact of Parenting Practices on Adolescent Achievement: Authoritative Parenting, School Involvement and Encouragement to Succeed." Child Development 63: 1266–81.

Taylor, R. B., S. D. Gottfredson, and S. Brower. 1984. "Block Crime and Fear: Defensible Space, Local Social Ties, and Territorial Functioning." Journal of Research in Crime and Delinquency 21: 303–31.

Tienda, M. 1991. "Poor People and Poor Places: Deciphering Neighborhood Effects on Poverty Outcomes." In Macro-Micro Linkages in Sociology, ed. J. Huber, 244-62. Newbury Park, Calif.: Sage Publications.

White, M. 1987. American Neighborhoods and Residential Differentiation. New York: Russell Sage Foundation.

Wilson, W. J. 1987. The Truly Disadvantaged: The Inner City, the Underclass, and Public Policy. Chicago: The University of Chicago Press.

Zapata, B. C., A. Rebolledo, E. Atalah, B. Newman, and M. C. King. 1992. "The Influence of Social and Political Violence on the Risk of Pregnancy Complications." American Journal of Public Health 82: 685–90.

Zill, N. 1991. Improving Kids Count: Review of an Annual Data Book on the Condition of Children in the 50 States of the U.S. Washington, DC: Child Trends.

Zill, N., and C. W. Nord. 1994. Running in Place: How American Families Are Faring in a Changing Economy and Individualistic Society. Washington, DC: Child Trends.
 


Back to New Approaches to Evaluating Community Initiatives index.


Copyright © 1999 by The Aspen Institute
Comments, questions or suggestions? E-mail webmaster@aspenroundtable.org.
This page designed, hosted, and maintained by Change Communications.