U. S. Charter Schools and Ethnic Segregation. Cobb & Glass

U. S. Charter Schools and Ethnic Segregation: Inspecting the Evidence Casey D. Cobb University of Connecticut Gene V Glass Arizona State University Among the major concerns surrounding school choice programs is their potential to stratify students along the dimensions of race, ethnicity, or socioeconomic class (Corwin & Flaherty, 1995; Elmore, 1988; Henig, 1995; Moore & Davenport, 1990; O'Neil, 1996; Thrupp, 1999; Wells, 1993a; Wells & Crain, 1992; Willms, 1986). Related concerns are that they will "cream" academically talented students off of the public schools (see e.g., Buechler, 1996; Fitzgerald, Harris, Huidekoper & Mani, 1998; Lee & Croninger, 1994; Moore & Davenport, 1990; and Wells, 1993b. Charter schools, as schools of choice, have been targets of these same allegations. Reports appear mixed as to whether charter schools disproportionately serve white students or whether they have contributed to increased segregation among publicly funded schools. Studies conducted by charter advocacy groups have found no evidence of ethnic or racial separation. Other, more prominent national evaluations have concluded that charter schools do not stratify students nor predominantly serve white children (e.g., Buechler, 1996; U.S. Department of Education, 1997). Finally, a number of investigations report evidence that contradict these national evaluations (e.g., Cobb & Glass, 1999; Crockett, 1999; Horn & Miron, 1999; Wells, 1999; Wells, Holme, Lopez, & Cooper, 2000). We turn next to these national evaluations. Their findings tend to be cited often, so they deserve close inspection. National Charter School Evaluations The first national evaluation of charter schools reported that in most states charter schools had "a racial composition similar to statewide averages or [had] a higher proportion of students of color" than traditional public schools (U.S. Department of Education, 1997, p. 24). As we shall argue, such statements may serve to misrepresent charter schools and their potential to ethnically and racially stratify students. In the first place, there is an overreliance on aggregate data to answer the question of whether ethnic separation occurs among schools; such aggregate data are incapable of achieving that purpose. Second, such statements overgeneralize the circumstances of charter schools, which operate under varying conditions, often as a result of differing state laws and regulations. The Fourth-Year Report: A Closer Look The latest among four U.S. Department of Education national evaluations of charter schools again reports no evidence that charter schools are predominantly white or that they segregate students (U.S. Department of Education, 2000). Such conclusions remain in question, however, for several reasons. First, since not all charter schools are "schools of choice" to the same degree, generalizations can be misleading if not inappropriate. Indeed, the degree of choice offered by charter schools depends largely on the laws under which they operate. Given the variation in charter legislation among the 27 states with operating charters, generalizations should be restricted to at most the state in which the charter schools reside. For instance, some state charter laws do not require that schools maintain particular ethnic/racial balances (e.g., Arizona), while others require ethnic/racial compositions to reflect that of the sponsoring district (e.g., California), while still others must reflect the ethnic/racial diversity of the surrounding area (e.g., Minnesota) (U.S. Department of Education & RPP, 1999). Even within-state assessments can be problematic when one considers the various types of charter schools. There are urban charter schools, at-risk charter schools, grassroots charter schools, and public and private conversion charter schools. Other characterizations include teacher-led, parent-led, and entrepreneur- initiated charter schools (Wells, 1999). Moreover, methodological inadequacies have made the detection of stratification impossible in cases where it might very well exist. The Fourth-Year Report analyses rely too heavily on aggregate state and national data, which are incapable of showing between-school ethnic/racial separation. These reports have not found evidence of stratification because they fail to consider the circumstances under which it is most likely to occur, namely, among schools within a district, town, or community. The U.S. Department of Education (2000) investigated enrollment compositions at the national, state, and local levels. Percentages of white/nonwhite students were aggregated and comparisons were made between charter and traditional public schools. From these data, the report makes this case: Critics and advocates alike have feared that charter schools would primarily serve white students. This has not turned out to be the case. Overall, charter schools enrolled a larger percentage of students of color than all public schools in the states with open charter schools. (p. 30) And further, charter schools in approximately three-fifths of the charter states enrolled a higher percentage of nonwhite students than all public schools in those states. (p. 32) (These statements are not removed from a broader context--they are among the main conclusions from the report.) Taken in the literal sense, these statements are not incorrect. However, such comparisons between charters and "all public schools" are inappropriate if the intent of these findings is to provide evidence that charter schools do not stratify students. Including all public schools in the comparison group compares what might be going on in a particular neighborhood with what might be going on in an entire state. For instance, why would one include in this comparison group average hundreds of public schools located several hundreds of miles away from any charter school? Such aggregated data could not speak to between-school segregation, if it existed. If one is interested in seriously investigating the possibility of ethnic/racial separation, a more appropriate comparison group would include those public schools that are in proximity to charter schools. To its credit, the report makes an attempt to examine the ethnic/racial variability between schools (the "local level" analysis), but the manner in which this was done again places the conclusions in question. After comparing the percentage of nonwhite students among charter schools to surrounding districts' percentage nonwhite students, the report concluded: Sixty-nine percent of charter schools were within 20 percent of their surrounding district's percentage of nonwhite students, while almost 18 percent had a distinctly higher percentage of students of color than their surrounding district. Approximately 14 percent of schools had a lower percentage of students of color than their surrounding districts. (p. 30) It is problematic that these figures are tallied across states, without regard to size of school, size of district, ethnic/racial heterogeneity, or presence of charter schools in any one state. Moreover, one might question the generous leeway given charters when a 20% ethnic/racial imbalance in 69% of the charter schools is dismissed as not evidencing segregation. (One wonders if the Washington DC administration's benevolent attitude toward charter schools--borne of a wish to stave off the even more radical reform of vouchers, we believe--was intruding at this point in its analysis. We have seen in the past at the federal level how the same research on class size, for example, can be interpreted in radically different ways by different political parties.) "Surrounding district" represents a better comparison group than all public schools, but still falls short of the mark. Some states, such as Arizona, permit-- even encourage--nondistrict sponsors. In fact, only a handful of Arizona charter schools are sponsored by public school districts, making within-district comparisons less meaningful in that state. Further, most of these district-sponsored schools were located well outside the boundary of the sponsoring district. Yet the Department of Education's "local level" analysis relied on district comparisons for all charter schools in their national sample. Even in those instances where charters do belong to districts, comparisons to district averages may not be the most sensitive technique for detecting ethnic/racial segregation. Segregation can easily be hidden in district level analyses. District schools, after all, can exhibit extreme variability in their ethnic/racial compositions, for they are often highly segregated. Averaging the percentage of white students among several district schools masks this variability. Furthermore, in urban, secondary districts, which can span wide geographic areas, stratification could be occurring in one corner of that district (e.g., between two high schools and one charter school), but the averaged figures obscure any evidence. Intra-district comparisons may make more sense for smaller, rural districts that tend to have only one or two high schools. Lastly, the Department of Education's local level analysis relied predominantly on charter schools reporting data about the districts in which they reside (see footnote 2 on page 31 in full report). Sound research requires that the quality of such data be ensured by use of independent auditing of reports. Evidence from Three States Next, we present evidence of ethnic/racial stratification among charter schools in Arizona, California, and Michigan. These three states currently enroll over half (52%) of all charter school students in the United States and contain nearly half of the nation's charter schools (U.S. Department of Education, 2000). The conclusions drawn here rest primarily upon findings from three statewide studies. Arizona Arizona has arguably the most lenient charter legislation in the nation, which is borne out by the sheer numbers of charters in that state. Arizona contains nearly one quarter of the nation's charter schools. Charters are sponsored by one of three boards. Two of these boards may approve up to 25 charter schools per year; the third may grant an unlimited number. Virtually any individual or organization inside or outside the state is eligible to receive a charter, and very few applicants are turned down. Successful charter applicants include entrepreneurs, former public school educators, school districts, for-profit companies, nonprofit organizations, and private citizens. Teachers in charter schools are not required to be certified. Despite recent legislative attempts to amend the law to prevent abuses (the Senate passed a bill that introduced familiar state regulations such as increased financial accountability, more auditing of books, bringing charters under state procurement laws, and the like), 11th-hour amendments were slapped on by the Arizona House Majority Leader that stripped out nearly all of the proposed new regulations. The governor and superintendent of schools (the latter an ardent supporter of charter schools) expressed shock and dismay at this almost inexplicable political maneuvering. Data reported at the state level suggest that Arizona charter schools serve an ethnically and racially diverse group of students, though they underrepresent Hispanic students. For instance, in 1996, traditional public schools in Arizona collectively served 56.8% white students while all charter schools enrolled 55.2% white students (Cobb & Glass, 1999). Put this way, there appears to be little difference in the ethnic compositions between charter and traditional public schools. But, as we have argued, averaged figures do not speak to the possibility of between-school ethnic/racial separation. Cobb and Glass (1999) compared the 1996 ethnic/racial compositions of over 100 Arizona charter schools with those of nearby traditional public schools. Geographic maps were used to analyze the ethnic/racial makeup of each urban charter school (n = 55) in relation to nearby traditional public schools of the same grade level. The maps provided rich, contextual information. Various geographical characteristics such as major streets and highways, reservation lands, mountainous regions, canals, military bases, census tracts, and district boundaries comprised these digital maps. Descriptive data relevant to the census tracts, district boundaries, and--most important-- nearby schools, were also available. Results indicated that the charter schools were typically more white (on the order of 15 to 20% higher in the percentage of white students enrolled) than the nearest traditional public schools. Moreover, the charter high schools appeared to fall naturally into either college preparatory schools that were largely white, or at-risk, vocational schools that were predominantly minority. Intra-district analyses of 57 rural charter schools (which often entailed comparing one charter school to one or two traditional public schools due the smallness of rural school districts) showed similar levels of ethnic/racial separation. These results confirmed, at least in the case of Arizona charter schools, the often-mentioned claim that schools of choice have the propensity to sort students along ethnic and racial lines (e.g., Whitty, 1997; Willms, 1986, 1996). A more recent study analyzed 1998 enrollment data in much the same manner as the original investigation to determine if the degree of ethnic separation had lessened, remained the same, or worsened two years later (Cobb, 2000). Numerous charter schools opened while others closed in the two years following the previous analysis, resulting in significant changes in enrollment patterns. For example, there has been a steady decline in the African American population of charter school students in Arizona as predominantly African American charter schools have encountered various problems with nearly nonexistent state regulations, as remarkable as that might seem. As in the previous analysis, this study (Cobb, 2000) also benefited from the urban analysis. We present one of those here. Figure 1 depicts a scenario that provides evidence of ethnic/racial separation. The charter school at the center of the map is an elementary-middle level Montessori school. Of the 336 students it enrolled, 86% were white. This stands in contrast to the percentages of white students served by surrounding traditional public schools of the same grade level (43, 28, 27, 18, and 34%). The traditional public school located in the northwest corner of the map is largely white (74%), but this school resides far away from the cluster of other schools and is separated by a major interstate. There is little reason to believe the charter school is drawing from that area. No other schools are located to the immediate north due to a large mountainous region. The ethnic composition of the charter school located in the southwest corner of the map reflects that of nearby traditional public schools, and thus does not contribute to ethnic separation. To remain consistent with the previous study, each of the 98 urban charter schools was directly compared to the nearest traditional public school of comparable grade level. Admittedly, this method lacks the capacity for detecting ethnic/racial separation that the more inclusive mapping technique offers; however, it summarily portrays charter/traditional public school differences in ethnic/racial composition in a simple, straightforward manner. Figure 2 displays the differences in the proportion of white students between each charter and the closest traditional public school of the same grade level. Overall, two thirds of the charter schools were more white than their traditional public school neighbor. Of those that contributed to ethnic/racial separation--that is, those that demonstrated at least a 15% difference in percentage of white students--the majority (about a 3 to 1 ratio) did so in the direction of serving more white students than their nearest traditional public school (see the right side of the figure). This is perhaps suggestive of "white flight." The overall results of this latest study indicated that nearly a third of Arizona's charter schools contributed to ethnic/racial separation during 1998–99. The encouraging news is that this percentage is considerably down from two years prior when 46% of the charter schools were found to contribute to this sort of stratification. However, when the number of charter schools that are suspect of contributing are added, the difference across years narrows significantly from 53% in 1996 to 47% in 1998 (see Cobb & Glass, 1999, or Cobb, 1999, for a complete explanation of what constitutes the "suspect of contributing" classification). Furthermore, although the proportion of charter schools that appear to have contributed to ethnic/racial separation has lessened over the past two years, the numbers of students and schools that have been affected has clearly increased. More Arizona students attended ethnically and racially stratified charter schools in 1998 than they did two years prior (76 schools in 1998 versus 45 schools in 1996). This level of segregation is disturbing and deserves the attention of policymakers in that state. Charter schools offer more than just choice of a school for students and parents, they offer schools (or those that sponsor new schools) opportunities to select students and parents. Indeed, charter schools can be selective primarily due to their start-up nature. Consider that those that start up can (1) limit size and thus enrollment, (2) narrow their curricular scope to attract or target certain types of students (e.g., Ben Franklin schools in Arizona), and (3) choose geographic location. This notion of selectivity is not limited to charter startups. Even conversion schools--especially private conversion schools--already have missions, students, and enrollment numbers in place. One charter school in Arizona, founded by the wife of a Libertarian economics professor at the University of Arizona, advertised for an academically elite clientele and told parents who inquired about admittance for their learning disabled child that the charter school would "not be a good fit" (personal communication). Another charter school in Arizona that was predominantly white in an ethnically diverse area prominently advertised its Mormon mission. It promised young Mormons a school tailored to them with its "10 Reasons LDS Parents Should Choose Life School" (Arizona Republic, 1998). To be sure, these are two extreme examples; however, they are testimony to the notion that charter schools, at least those operating under few regulations, may well result in worse levels of stratification than other "pure choice" models, such as vouchers, because of their start-up nature. We encourage the use of improved methodologies to study the potential stratifying effects of charter schools. For instance, student address data would strengthen the mapping techniques employed by Cobb and Glass (1999). But gaining access to these data can be difficult. We also urge researchers and policymakers to make reasonable and appropriate comparisons when looking at the enrollment compositions of charter and traditional public schools. As we have demonstrated, it makes little sense to look at highly ethnically and racially homogeneous areas to find evidence of segregation. Charter schools do locate in predominantly white districts, and probably should not be included in overall averaged figures. Lastly, we suggest that investigations be done by research teams with representatives from pro-charter and anti-charter positions. Michigan In the aggregate, Michigan's charter schools--called "public school academies"-- serve proportionally more students of color than regular public schools (Horn & Miron, 1999). But again, such averaged figures can mask underlying disparities at regional and local levels. It would be inappropriate to conclude that Michigan charter schools do not ethnically/racially segregate. A more in-depth analysis would be required to answer that question. In its 1999 study, the Evaluation Center at Western Michigan University reported that, on average, charter schools in Michigan (at least the 62 schools in their study) enrolled relatively more students of color than noncharter public schools (51 to 33% respectively). The evaluators were quick to point out, however, that these numbers do not necessarily support the conclusion that charter schools are attracting more students of color than traditional public schools, or that they have not contributed to ethnic/racial segregation in their vicinity. Most charter schools in Michigan are located in urban areas, which are predominantly minority. In fact, the data indicate that the charter schools are actually serving disproportionately fewer minorities in diverse areas. Sponsoring districts were 41% white on average while charter schools in those districts were nearly 60% white on average. Horn and Miron, while recognizing the limitations of charter-to-district comparisons (in this instance, some students attend charters from outside districts in which they are located), suggested that this provides evidence of ethnic segregation. They reported: "in fact, in relation to the host districts, the [charter schools] as a whole have fewer minorities. Thus, there is support for those who argue that the charter schools are skimming and increasing segregation" (Horn & Miron, 1999, p. v). They also state that, "while some schools . . . strive[d] to increase racial and social diversity of the students, others [had] few, if any minorities or students with special needs" (p. iv). In the appendices of their report, Horn and Miron (1999) present comparisons of the ethnic compositions among 61 charter schools and their host districts. We calculated that 26 of 61 charter schools, or 43%, demonstrated at least a 15% difference in percentage of white students. Of these 26, 14 were in the direction of enrolling more minority students while 12 were in the direction of serving more white students. However, after removing charter schools located in host districts that were ethnically and racially homogeneous (that is, over 95% white, on average), the proportion of charter schools that presented with at least a 15% difference increased to well over half (24 of 45). Further, the percentage of charter schools that were significantly more white than their host district increased to 27%. Perhaps more troubling is the declining trend in enrollment among minority students over a four-year period (see Figure 3). The evaluators attributed this decline to newly formed charter schools, which were enrolling greater numbers of white students. To the extent that new charters are locating in ethnically/racially heterogeneous areas, this could be indicative of white flight. Given that districts sponsoring charter schools are 41% white (compared to the overall state average of 80% white), it does appear that charters are locating in more ethnically and racially diverse communities. We learned from the Horn and Miron study that state-averaged data may not accurately portray a complete picture. Once again, we return to the comparisons drawn in the U.S. Department of Education's Fourth-Year Report: In order to examine the racial/ethnic variability across schools, we also calculated the average of the schools' racial/ethnic percentages. On average, charter schools enrolled a significantly lower percentage of white students (50 percent versus 63 percent) and a much larger percentage of black students (27 percent versus 17 percent) than all public schools in the 27 charter states. (2000, p. 30) We want to point out that this "50%" figure is an average across schools, and that there are many data points on either side of this mean. It could be the case that those schools that comprise the upper part of this range (say, 70% and higher white) were mostly located in non-white areas, as Horn and Miron had found in Michigan. Cast in this light, the data tell a different story. California A number of studies have reported that charter schools in California overrepresent white students. For instance, the national First-Year Report revealed that 37% of California charter schools, compared to 17% of traditional public schools, had enrollments of 80% or more white students (U.S. Department of Education, 1997). A separate study, one commissioned by the California Legislative Analyst's Office, concluded that, On the whole, charter schools served a population that was demographically similar to the student population statewide. Within-district comparisons, however, showed that in about 40% of charter schools students were more likely to be White, and in about 60% of charter schools students were less likely to be low income than other students in their sponsoring districts. (SRI, 1997, p. II-1) The UCLA Charter School Study found that in 10 of the 17 charter schools it investigated, at least one ethnic or racial group was over- or underrepresented by 15% or more in comparison to the districts' composition (Wells, 1999). Crockett (1999) found similar evidence of ethnic/racial segregation in her comprehensive study of well over 100 California charter schools. We explore this study in more detail below. Crockett (1999) conducted a statewide analysis of all 123 California charter schools that operated during the 1997–1998 school year. The analysis was an attempt to discern, to the extent it existed, the ethnic/racial distinctness between charter schools and their sponsoring districts. California charter law requires charter schools to reflect the ethnic/racial balance of the district in which they reside. (Wells [1999] previously reported that this rule was not being enforced, at least among the 10 California districts in her study.) If discrepancies in ethnic/racial student compositions were evident, Crockett sought to explain those differences by way of school and district characteristics (e.g., urbanness, grade level, geographic location and size, charter mission, and the like). Racial distinctness was defined by a 25% charter-district difference in ethnicity/ race for at least one of the seven ethnic/racial categories. (California maintains seven classifications of ethnicity/race: American Indian, Asian, Pacific Island, Filipino, Hispanic, Black, and White.) Results indicated that nearly one third (n = 38) of the charter schools were ethnically/racially distinct from their sponsoring district (see Table 1). Further inspection demonstrated that urban charter schools were far more likely to exhibit distinctness than suburban and rural schools. A closer examination of the 38 charter schools that were 25% and greater distinct in ethnic/racial composition from their sponsoring districts revealed that 20 (or 52%) of these exhibited a white–Hispanic inverse relationship; that is, charter schools typically served more white students than the district, on average, and conversely, the district schools served more Hispanic students, on average. These percentages virtually offset one another. Of the 20 charter schools, 19 were in the direction of more white. This is strongly suggestive of ethnic/racial separation. Crockett was particularly interested in those charter schools that were whiter than their district. Overall, 78 of the 123 charter schools (63%) were whiter than their sponsoring districts. One in five charter schools (n = 26) exhibited at least a 20-point difference in the percentage of white students enrolled (all in the direction of the charter being more white). The average difference in percentage white among these charter schools was 32%. These schools tended to be located in urban areas, span the elementary and middle school grades, espouse an academic (versus vocational) mission, and be start-up (versus conversion) schools. In most instances, the difference in percentage white (i.e., charter minus district average) was matched by a corresponding deficit in the percentage of Hispanic students (see Table 2). Crockett noted the methodological weakness of using district averages (of percentage ethnic/racial enrollments), as it ignores within-district ethnic/racial heterogeneity. She further cautioned that "some charter schools may reflect their locations in a way that puts them out of balance with their sponsoring districts . . . particularly if the district is large." That said, Crockett affirmed that "the findings of ethnic separation . . . are not limited to one or two districts, but are in effect statewide" (p. 74). A map analysis of California charter schools is currently under way, which we believe presents a more powerful manner to investigate the possibility of ethnic/racial separation. Nevertheless, the findings reported by Crockett are strongly indicative of ethnic/racial stratification. Finally, we think it is important to note that many charters in California have experimented with mandatory parental involvement contracts, which can serve to be exclusionary (Becker, Nakagawa, & Corwin, 1996). In a study of 10 California school districts Wells (1999) also reported that charter schools exercise considerable control over the types of students they serve. Conclusions The evidence presented here runs counter to some of the claims intimated by highly regarded national evaluations of charter schools; namely, that charter schools have not resulted in the ethnic/racial separation of students. More careful inspection demonstrates that ethnic and racial stratification can and does exist on the part of some charter schools. Comparisons among proximal charter and traditional public schools in Arizona, Michigan, and California suggest that a significant number of charter schools are disproportionately more white by about 15 to 20% on average. These three states account for over half of the nation's charter schools. Although we do not generalize our findings to all charter schools, we do believe that substantial evidence exists that charter schools in ethnically and racially diverse neighborhoods are contributing to ethnic and racial segregation in the public schools of our nation. Moreover, certain among state charter policies appear to permit such sorting. These incidences of ethnic/racial separation are not isolated nor insignificant. Note. An earlier version of this paper was presented at the annual meeting of the American Educational Research Association (New Orleans, April 2000). References Arizona Republic. (1998, May 14). Charter school ad targets Mormons. p. A1. Becker, H. J., Nakagawa, K., & Corwin, R. G. (1996). Parent involvement contracts in California's charter schools: Strategy for educational improvement or method of exclusion. Teachers College Record, 98(3), 511–536. Buechler, M. (1996). Charter schools: Legislation and results after four years (Indian Education Policy Center, Policy Rep. PR-B13). Bloomington, IN: School of Education Office. Cobb, C. D. (1999). Charter schools as schools of choice: Ethnic and racial separation in Arizona. Cobb, C. D., & Glass, G. V (1999). Ethnic segregation in Arizona charter schools. Education Policy Analysis Archives, 7(1). Retrieved from http://olam.ed.asu.edu/v7n1. Corwin, R. G., & Flaherty, J. F. (Eds.) (1995, November). Freedom and innovation in California's charter schools. Los Alamitos, CA: Southwest Regional Laboratory. Crockett, C. (1999). California charter schools: The issue of racial/ethnic segregation. Unpublished doctoral dissertation, Arizona State University, Tempe. Elmore, R. F. (1988). Choice in public education. In W. L. Boid & C. T. Kerchner, Politics of excellence and choice in education: 1987 yearbook of the politics of education association (pp. 79–98) New York: Falmer Press. Fitzgerald, J., Harris, P., Huidekoper, P., & Mani, M. (1998, January). 1997 Colorado charter schools evaluation study: The characteristics, status and student achievement data of Colorado charter schools. Denver: Clayton Foundation. Henig, J. R. (1995). Race and choice in Montgomery county, Maryland, magnet schools. Teachers College Record, 96(4), 729–734. Horn, J., & Miron, G. (1999, January). Evaluation of the Michigan Public School Academy initiative: Executive summary. Evaluation Center of Western Michigan University. Retrieved from http://www.wmich.edu/evalctr/. Lee, V., & Croninger, R. G. (1994). Parental choice of schools and social stratification in education: The paradox of Detroit. Educational Evaluation and Policy Analysis, 16(4), 434–457. Moore, D., & Davenport, S. (1990). School choice: The new, improved sorting machine. In W. Boyd & H. Walberg (Eds.), Choice in education (pp. 187–223). Berkeley, CA: McCutchan. O'Neil, J. (1996). New options, old concerns. Educational Leadership, 54(2), 6–8. SRI International. (1997). Evaluation of charter school effectiveness. Menlo Park, CA. Report prepared for the State of California Office of Legislative Analyst. Written by J. Powell, J. Blackorby, J. Marsh, K. Finnegan, & L. Anderson. Thrupp, M. (1999). Schools making a difference: Let's be realistic! School mix, school effectiveness and the social limits of reform. Buckingham, PA: Open University Press. U.S. Department of Education, Office of Educational Research and Improvement. (1997). A study of charter schools: First-year report 1997. RPP International and the University of Minnesota. U.S. Department of Education & RPP. (1999). A comparison of charter legislation: Thirty-three states and the District of Columbia incorporating legislative changes through October, 1998. Retrieved from http://www.uscharterschools.org. U.S. Department of Education. (2000). The state of charter schools 2000--Fourth-year report, January 2000. Retrieved from http://www.uscharterschools.org. Wells, A. S. (1993a). The sociology of school choice: Why some win and others lose in the educational marketplace. In E. Rassel & R. Rothstein (Eds.), School choice: Examining the evidence (pp. 29–48). Washington, DC: Economic Policy Institute. Wells, A. S. (1993b). Time to choose: America at the crossroads of school choice policy. New York: Hill and Wang. Wells, A. S. (1999). Beyond the rhetoric of charter school reform: A study of ten California school districts. UCLA Charter School Study. Wells, A. S., & Crain, R. L. (1992). Do parents choose school quality or school status? A sociological theory of free market education. In P. W. Cookson Jr. (Ed.), The choice controversy, (pp. 65–82). Newbury Park, CA: Corwin Press. Wells, A. S., Holme, J. J., Lopez, A., & Cooper, C. W. (2000). Charter schools and racial and social class segregation: Yet another sorting machine? In Richard D. Kahlenberg (Ed.), A notion at risk: Preserving public education as an engine for social mobility (pp. 169–221). New York: Century Foundation Press. Whitty, G. (1997). Creating quasi-markets in education: A review of recent research on parental choice and school autonomy in three countries. Review of Research in Education, 22, 3–47. Willms, D. J. (1986). Social class segregation and its relationship to pupils' examination results in Scotland. American Sociological Review, 51, 224–241. Willms, D. J. (1996). School choice and community segregation. In A. C. Kerckhoff (Ed.), Generating social stratification: Toward a new research agenda. Boulder, CO: Westview.

Me & Saul

Me & Saul Saul – Kripke, that is – has been labeled the most influential philosopher of the second half of the 20th Century. Wikipedia says, “[f]rom the 1960s until his death, he was a central figure in a number of fields related to mathematical and modal logic, philosophy of language and mathematics, metaphysics, epistemology, and recursion theory.” Me? I play tennis and feed hummingbirds. I only saw Saul once. It was at the 1957-58 Nebraska High School Science Fair. The fair was held on the campus of the University of Nebraska – which is now known as UNL. But I’m getting ahead of the story. Saul Aaron Kripke was born in Long Island, five months after my birth in Lincoln, Nebraska; but he was raised from age 6 in Omaha. I was born in the house once owned and later dedicated as a hospital by William Jennings Bryan. (That’s about the end of my celebrity.) Saul was the eldest of three children – one boy, two girls – born to Rabbi Myer and Dorothy Kripke. Myer was the rabbi of Beth El, the only conservative synagouge in Omaha. Saul taught himself ancient Hebrew by the age of six, read the Shakespearean plays by age nine, and mastered the writings of Descartes and complex mathematics before the end of sixth grade. I took up tennis at age 40. Our friend Marlyne Freedman was BFF with Saul’s younger sister Netta. Occasionally, the two would have an overnight at the Kripkes. It was clear to Marlyne that something was not quite right about her friend’s brother Saul. Saul would frantically walk up and down stairs from the first to the upper level for hours after everyone had turned in. Nobody talked about Aspergers in those days. But back to the science fair. I hated high school, or rather, I hated the things they taught in high school until second semester of grade 12 when it became obvious that the coaches were no longer interested in me. At that point I picked out chemistry as something to like. As the science fair approached – it was held in February – a couple of us cast about for a project to put on display. I put two things together and came up with an atom weighing machine. If you put a copper plate and a carbon rod in a beaker of copper-sulfate solution and connect them to a battery, you’d have something. Actually, if you could figure out how many electrons passed through that system at a certain amperage for a certain period of time, and you knew that two electrons would plate out one copper atom, and you knew that Coulomb and Avogadro – oh, forget it. It actually made sense. And the first time I tried it, it produced the weight of a copper atom with only 5% error. The second time – on a Saturday with my friends Dave and Rod on hand to exercise additional control – the estimated weight was off by 30%. My first experience with reliability. All the same, even if I couldn’t nail down the exact weight of one copper atom, the system was beautiful: a big beaker of crystal-clear blue solution, wires, amp meters, a big battery. In fact, my display was a huge hit at the science fair. Attendees huddled around my display as I proudly described the first run-through of the atom weighing machine. No point boring them with the story of the second attempt. The display right next to mine – stage right – was a white posterboard with some letters and funny looking scribbles on it. It was entitled “The System LE.” Its creator stood in front of the poster behind a table with his knuckles flat on the table, rocking back and forth. Davening; I had never seen such a thing before; neither the rocking nor the stuff on the posterboard. Everybody was fascinated by my bubbling solution and wires. No one was paying attention to Saul. He was a Finalist for the Westinghouse Science Talent Award that year. The 1958 winner eventually in 1990 invented a device for performing jaw exercises. Saul once described the fair as “one of the memorable experiences of my life.” It certainly was such for me. It was a Friday in February. The fair wrapped up around 4 PM. As Dave and I walked out of the auditorium we noticed Saul, his classmate (see Note below), and an adult, who was probably their teacher, in animated conversation. My buddy Dave said that they were worried about whether they could make it back to Omaha before sundown. I never saw Saul again. But my friend Dave did. Six months later, Dave went off to Harvard, while I went off to the tiny college two miles away, the only college I could afford or be admitted to. Saul also went off to Harvard. Dave and Saul were enrolled in the same Freshman calculus course. Six-weeks exam. The bluebooks are passed out, the testing period was ON. No Saul. Twenty minutes later, Saul rushes in, plops down in a seat in the front row, and starts working … and davening. In fact, the davening was so extreme and distracting that a couple of students went up to the proctor and told him to get that guy outta here! Dave says that was the last time he saw Saul. Rumor had it that in the very next year, Saul was teaching a graduate-level seminar in logic at MIT. My tennis buddy Jerry was finishing his PhD in Psychology at Princeton in the mid-1970s when Saul accepted an endowed professorship there in philosophy, of course. His most famous book, Naming and Necessity (1980), is one of the major philosophical works of the 20th century, I am told. I bought it a few years ago; I gave up on page 4. Saul was apparently the object of a good deal of curiosity around Princeton. His wife, Margaret Gilbert – also a philosopher and British, whose family name was once Goldberg – drove Saul wherever he had to go, bought his clothes, cooked his meals, and generally did all the mundane tasks of ordinary life, like buying tennis balls and cleaning hummingbird feeders. They divorced. Saul died on September 15, 2022, of pancreatic cancer. Note Saul's classmate was Richard Speier , no small celebrity himself.

Politics of Teacher Evaluation

1993

Glass, G. V & Martinez, B. A. (1993, June 3). Politics of teacher evaluation. Proceedings of the CREATE Cross-Cutting Evaluation Theory Planning Seminar (ED364581, pp. 121–134). ERIC. https://files.eric.ed.gov/fulltext/ED364581.pdf

Review of "The handbook of research synthesis"

1995

Cooper, H. and Hedges, L. V. (Eds.) The handbook of research synthesis. New York: Russell Sage Foundation, 1994.
573 pp.
ISBN 0-87154-226-9. $49.95

Reviewed by Gene V Glass
Arizona State University
June 19, 1995

The Handbook of research synthesis is the third volume of a coordinated publication program on meta-analysis sponsored by the Russell Sage Foundation. Starting in 1987 under the direction of a Research Synthesis Committee (Harris Cooper, Thomas Cook, David Cordray, Heidi Hartmann, Larry Hedges, Richard Light, Thomas Louis and Frederick Mosteller), the project has previously produced The future of meta-analysis (Wachter and Straf, 1990) and Meta-analysis for explanation (Cook et al., 1992). The Handbook is by far the largest and most comprehensive publication of this project. It means to be the "definitive vade mecum for behavioral and medical scientists intent on applying the synthesis craft."(p. 7) At nearly 600 hundred pages and three pounds, researchers will have to leave their laptops behind.

Although the editors and many of the chapter authors eschew the term "meta-analysis" in favor of the broader "research synthesis," potential readers should understand that the former (statistical analysis of summary statistics from published reports) is the subject of the Handbook and not the more general concerns of theory commensurability or the planning of coordinated investigations suggested by the latter.

The organization of the Handbook follows the common logic of producing a meta-analysis: formulate the question, search the literature, code the information, analyze it, write a report. Some of the chapters are unremarkable, since much of the craft of doing research is routine; this only speaks to the completeness of the work. Chapter 6, "Research Registers" by Kay Dickersin, points to new possibilities. Medicine has databases of prospective, on-going and completed studies; Dickersin identifies 26 of them. Expand them slightly to include the actual data from clinical trials and other forms of study and many of the more vexing problems of meta- analysis (which arise from the telescoping of primary data into summary statistics--and the discarding of the former) will be solved. It is past time when behavioral research, both on-going and completed, is catalogued and archived. Telecommunications has driven the costs of information storage and retrieval to near zero. Who will create the Internet Behavioral Research Archives?

Two themes imparted by the editors and the committee, one presumes, give the Handbook of research synthesis its distinctive character. Chapter 1 by the editors, Harris Cooper and Larry Hedges, is entitled "Research Synthesis as a Scientific Enterprise." Research synthesis is likened to doing science itself: both are seen as involving problem formulation, data collection, data evaluation, analysis and publication. These stages in both the pursuit of science and the conduct of research synthesis give the Handbook its section titles, and perhaps its entire bent. Although these stages might reasonably describe the stages in carrying out a meta-analysis, they do not capture what is distinctive about science. The stages describe as well how one may conduct the evaluation of a device, a drug and program or what-have-you. In effect, the Handbook draws no clear or convincing line between the pursuit of scientific theory and the evaluation of technology. This line is quite important and must be drawn.

To cast meta-analysis as dedicated to the construction of science disposes the discussion of it in the direction of classical statistical methods that evolved alongside quantitative science in the 20th century. In particular, the methods of statistical hypothesis testing have come to be associated with the scientific enterprise. The unwholesome effects of this association are the subject of a brilliant article by Paul Meehl (1990) on the progress of "soft psychology"; see particularly the Appendix where Meehl briefly addresses meta-analysis. Just as scientists bring forth hypotheses to be accepted or rejected by data, so do statisticians devise the strategies by which data are judged to be in accord with or at odds with the hypotheses. This view of statistics gives the Handbook its other defining theme: meta-analyses involve the testing of statistical hypotheses about parameters in populations of research studies.

The appropriate role for inferential statistics in meta- analysis is not merely unclear, it is seen quite differently by different methodologists. These differences are not reflected in the Handbook. In 1981, in the first extended discussion of the topic, McGaw, Smith and I raised doubts about the applicability of inferential statistics in meta-analysis. Inference at the level of persons within studies (of the type addressed by Becker in Chapter 15, "Combining Significance Levels") seemed quite unnecessary to us, since even a modest size synthesis will involve a few hundred persons (nested within studies) and lead to nearly automatic rejection of null hypotheses. Moreover the chances are remote that these persons or subjects within studies were drawn from defined populations with anything approaching probabilistic techniques; hence, probabilistic calculations advanced as if subjects had been randomly selected are dubious. At the level of "studies," the question of the appropriateness of inferential statistics can be asked again, and the answer again seems to be negative. There are two instances in which common inferential methods are clearly appropriate: when a defined population has been randomly sampled and when subjects have been randomly assigned to conditions in a controlled experiment. In the latter case, Fisher showed how the permutation test can be used to make inferences to the universe of all possible permutations. But this case in of little interest to meta-analysts who never assign units to treatments. The typical meta-analysis virtually never meets the condition of probabilistic sampling of a population (though in one instance (Smith, Glass & Miller, 1980), the available population of drug treatment experiments was so large that it was in fact randomly sampled for the meta-analysis). Inferential statistics has little role to play in meta-analysis: "The probability conclusions of inferential statistics depend on something like probabilistic sampling, or else they make no sense." (p. 199)

It is common to acknowledge that many data sets fail to meet probabilistic sampling conditions, but to argue that one might well treat the data in hand "as if" it were a random sample of some hypothetical population. Under this supposition, inferential techniques are applied and the results inspected. The direction taken by the Handbook editors and authors mirrors the earliest published opinion on this problem, expressed by Mosteller and his colleagues in 1977: "One might expect that if our MEDLARS approach were perfect and produced all the papers we would have a census rather than a sample of the papers. To adopt this model would be to misunderstand our purpose. We think of a process producing these research studies through time, and we think of our sample--even if it were a census--as a sample in time from the process. Thus, our inference would still be to the general process, even if we did have all appropriate papers from a time period." (Gilbert, McPeek and Mosteller, 1977, p. 127; quoted in Cook et al., 1992, p. 291) This position is repeated in slightly different language by Hedges in Chapter 3, "Statistical Considerations": "The universe is the hypothetical collection of studies that could be conducted in principle and about which we wish to generalize. The study sample is the ensemble of studies that are used in the review and that provide the effect size data used in the research synthesis." (p. 30)

These notions appear to be circular. If the sample is fixed and the population is allowed to be hypothetical, then surely the data analyst will imagine a population that resembles the sample of data. Or as Gilbert, McPeek and Mosteller viewed it, the future will resemble the past if the past is all one has to go on. Hence all of these "hypothetical populations" will be merely reflections of the samples in hand and there will be no need for inferential statistics. Or put another way, if the population of inference is not defined by considerations separate from the characterization of the sample, then the population is merely a large version of the sample. With what confidence is one able to generalize the character of this sample to a population that looks like the sample writ large? Well, with a great deal of confidence, obviously. But then, the population is nothing but the sample.

Hedges and Olkin have developed inferential techniques that ignore the pro forma testing (because of large N) of null hypotheses and focus on the estimation of regression functions that estimate effects at different levels of study characteristics; nearly all of them appear in the Handbook. They worry about both sources of statistical instability: that arising from persons within studies and that which arises from variation between studies. As they properly point out, the study based on 5 persons deserves greater weight than the study based on 500 persons in determining the response of the treatment condition to changes in study conditions. The techniques they present are based on traditional assumptions of random sampling and independence. It is, of course, unclear precisely how the validity of their methods are compromised by failure to achieve probabilistic sampling of persons and studies.

The irony of traditional hypothesis testing approaches applied to meta-analysis is that whereas consideration of sampling error at the level of persons always leads to a pro forma rejection of "null hypotheses" (of zero correlation or zero average effect size), consideration of sampling error at the level of study characteristics (the study, not the person as the unit of analysis) leads to too few rejections (too many Type II errors, one might say). Hedges's homogeneity test of the hypothesis that all studies in a group estimate the same population parameter is the focus of much attention in the Handbook. Once a hypothesis of homogeneity is accepted by Hedges's test, one is advised to treat all studies within the ensemble as the same. Experienced data analysts know, however, that there is typically a good deal of meaningful covariation between study characteristics and study findings even within ensembles where Hedges's test can not reject the homogeneity hypothesis. The situation is nearly exactly parallel to the experience of psychometricians discovering that they could easily interpret several more factors than inferential solutions (maximum- likelihood; LISREL) could confirm. The best data exploration and discovery is more complex and credible than the most exact inferential test. In short, classical statistics seems not able to reproduce the complex cognitive processes that are commonly applied by data analysts.

Rubin (1990) addressed most of these issues squarely and staked out a radical position that appeals to the author of this review : "...consider the idea that sampling and representativeness of the studies in a meta-analysis are important. I will claim that this is nonsense--we don't have to worry about representing a population but rather about other far more important things." (p. 155) These more important things to Rubin are the estimation of treatment effects under a set of standard or ideal study conditions. This process, as he outlined it, involves the fitting of response surfaces (a form of quantitative model building) between study effects (Y) and study conditions (X, W, Z etc.). Of the 32 chapters in the Handbook, only the contribution of Light, Singer and Willett, Chapter 28, "the visual presentation and interpretation of meta-analyses," comes close to illustrating what Rubin has in mind. By far most meta-analyses are undertaken in pursuit not of scientific theory but technological evaluation. The evaluation question is never whether some hypothesis or model is accepted or rejected but rather how "outputs" or "benefits" or "effect sizes" vary from one set of circumstances to another; and the meta-analysis rarely works on a collection of data that can sensibly be described as a probability sample from anything.

Rubin's view of the meta-analysis enterprise would have produced a volume substantially different from that which Cooper and Hedges edited. So we can expect the Handbook of research synthesis to be not the last word on the subject, but one important word on meta-analysis.

References

Glass, G.V; McGaw, B. & Smith, M.L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: SAGE.

Rosenthal, R. (1984). Meta-Analytic Procedures for Social Research. Beverly Hills, CA: SAGE.

Rubin, D.R. (1990). A new perspective. Chp. 14 (pp. 155-165) in Wachter, K.W. and Straf, M.L. (Eds.), The Future of Meta-Analysis. N.Y., N.Y.: Russell Sage Foundation.

Smith, M.L.; Glass, G.V & Miller, T.I. (1980). Benefits of Psychotherapy. Baltimore, MD: Johns Hopkins University Press.

Gilbert, J.P.; McPeek, B. & Mosteller, F. (1977). Progress in surgery and anesthesia: benefits and risks of innovative surgery. In J. P. Bunker, B.A. Barnes & F. Mosteller (eds.) (1977). Costs, Risks and Benefits of Surgery. NY: Oxford University Press.

Cook, T.D.; Cooper, H; Cordray, D.S.; Hartmann, H; Hedges, L.V.; Light, R.J.; Louis, T.A.; & Mosteller, F. (1992). Meta-analysis for explanation: A casebook. New York: Russell Sage Foundation.

Meehl, P.E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195- 244. (Monograph Supplement 1-V66)

Deciding for One or Deciding for Many

2018

The U.S. Charter School Movement and Ethnic Segregation

2000

Cobb, C. D., Glass, G. V & Crockett, C. (2000). The U.S. Charter School Movement and Ethnic Segregation. Paper presented at the Annual Meeting of the American Educational Research Association. New Orleans, LA. April 2000.