A Slice of Advice

1992

Gene V Glass. (1992). A slice of advice. Educational researcher, 21(3), 23.

How May Salience of a Membership Group Be Increased?

1964

Gene V Glass. How May Salience of a Membership Group Be Increased? Journal of Educational Measurement , 1(2), (Dec., 1964), 125-129.

Critiquing a Follow Through Evaluation

1978

Ernest R. House, Gene V. Glass, Leslie D. McLean and Decker F. Walker. (1978). Critiquing a Follow Through Evaluation. The Phi Delta Kappan , 59(7), 473-474.

When Statistical Significance Hides More Than It Reveals

Powers, J.M., & Glass, G.V (2014). When statistical significance hides more than it reveals. Teachers College Record. When Statistical Significance Hides More Than It Reveals Jeanne M. Powers Gene V Glass Arizona State University ABSTRACT Background & Purpose: The What Works Clearinghouse (WWC) recently released a summary of five CREDO studies of charter school outcomes produced between 2009 and 2013. We compared the WWC’s summary, which highlighted the statistical significance of the findings, to the effect sizes reported in the individual reports. We also addressed the findings reported in the original studies that were not summarized in the WWC reports. Research Design: Analytic essay highlighting the gaps between the findings reported in the summary WWC report, the individual WWC reports for each study, and the original CREDO studies. Findings: We argue that focusing on statistical significance is potentially misleading. The WCC summary invites the reader to conclude that charter schools had a greater effect on students’ achievement gains than traditional public schools. Comparing across the studies’ effect sizes suggests that the average effect of charter schools on students’ achievement gains is negligible. The WWC reports also do not address the considerable variation in achievement gains within and across subgroups of students and schools. Conclusion: Summaries generated from research studies should provide an accounting of findings that allows practitioners to assess their practical importance. When these and similar reports are hard to understand and misleading, they run the risk of eroding practitioners’ trust in research and increasing rather than bridging the gulf between research and practice. INTRODUCTION Table 1 is a simplified version of a table produced for the What Works Clearinghouse’s (WWC) Review of the Center for Research on Education Outcomes (CREDO) Charter School Studies (WWC, 2014a). The WWC review summarized the individual reviews of five studies examining charter school student outcomes. The original studies were produced by CREDO between 2009 and 2013. The WWC reviews (three quick reviews and two single study reviews) were released about six months after the release of each study. The last of these reviews and the summary were released in January 2014. i Retrieved February 21, 2014 from http://ies.ed.gov/ncee/wwc/SingleStudyReview.aspx?sid=220. A “+” indicates that charter school students had greater achievement gains after one year than their traditional public school counterparts. A “-“ indicates that traditional public school students made greater achievement gains. The text above the table notes that all of the findings were statistically significant except for the negative math gains observed in the national study. The table is surrounded by technical details about the studies summarized. But the power of the table lies not in its fidelity to technical details; it is technically correct. The power of the table is in its simple visual appeal. In this research note we argue that this table can also provide important lessons about statistical and practical significance. BACKGROUND The goal of the WWC is to allow educators to make “evidence-based decisions” by identifying studies that “provide credible and reliable evidence of the effectiveness of a given practice, program, or policy” (WWC, n.d.). These are studies that have been vetted and summarized using protocols for assessing the research designs and findings of studies (WWC, 2014b). These summaries of “what works” are explicitly intended to help people working in schools figure out how to work more effectively with students and their families — these short summaries are presumably easier to access and understand than the full research reports. Quick reviews provide preliminary reviews of analyses of program or practice effectiveness; single study reviews are more detailed reviews of studies that underwent quick review. Single study reviews “are designed to provide education practitioners and policymakers with timely and objective assessments of the quality of the research evidence from recently released research papers and reports” (WWC, 2012). (Note 1) The CREDO studies were reviewed by the WWC because they had received significant media attention (WWC, 2014a). They are also important because they are the studies of school choice outcomes in the WWC that are the largest in scope. The 14 other studies of school choice in the WWC tend to focus on specific settings (Milwaukee, the District of Columbia, New York City) or smaller samples of specific types of charter school (KIPP schools, charter schools run by charter management organizations (CMOs)). Our discussion is focused on the WWC reviews, although we consulted the CREDO studies to clarify our understanding of the WWC reviews as needed (WWC 2010a, 2010b, 2011, 2013, 2104a, 2014c). Briefly, the CREDO studies all used the same technique: charter school students were matched to similar students attending “feeder” traditional public schools (Note 2) by grade level, baseline achievement, and demographic characteristics (eligibility for free or reduced price lunch, special education status, gender, and race). The analyses compared the two groups’ achievement gains on standardized reading and math tests between the baseline year and the following year. The main difference between the studies was the scope of the analytic samples. Two were multi-state (a national and a 16 state study), two focused on specific states (Indiana and New Jersey), and the final study was of New York City charter schools. The studies also varied by time frame and the grades included in the analyses. ANALYSIS Table 2 provides a summary of the effect sizes reported in the individual WWC reviews. For the purposes of accuracy, in Table 2, we reproduced much of the text from the individual reports and organized it into a chart similar to Table 1. Leave aside for the moment the possibility that all of these comparisons are confounded by a differential regression effect (Campbell & Stanley, 1966); one of the common reasons a student enrolls in a charter school is because they earned poor grades at their traditional public school. (Note 3) We paraphrased when necessary to present the findings in table form and also facilitate comparison across the reports. Table 2 lacks the impact on the reader of Table 1. Table 1 invites the reader to count the number of cells with a “+” and come to the conclusion that on balance, charter schools had a much greater effect on students’ achievement gains than traditional public schools. Yet for more than 30 years it has been clear that counts of statistically significant and non- significant results violate some of the most fundamental properties of statistical hypothesis testing (Hedges & Olkin, 1980). Table 2 suggests that on average, the effect of charter schools on students’ achievement gains, positive or negative, was negligible. The strongest positive results for charter schools were reported for New York City students in mathematics. While positive, these effect sizes are among the lowest of those reported across the 15 studies of school choice in the WWC database that met WWC evidence standards. (Note 4) They are also well under the definition of a substantially important finding provided in the glossary of all WWC single study reviews: a substantively important finding “has an effect size of .25 or greater, regardless of statistical significance” (WWC, 2014a). (Note 5) By comparison, the typical effect size of a year’s growth in reading achievement at the elementary school level is about 1.0; in math, it is slightly larger (Levin, Glass & Meister, 1986, 1987). The latter figure allows us to interpret more correctly the practical significance of the studies’ findings because we can compare the effect sizes yielded across the studies to an appropriate benchmark, typical yearly achievement growth. Finally, while the discussion above focuses on the findings that the WWC reported and assessed, these are a relatively small part of the analyses in the CREDO studies. The WWC reports focus largely, although not exclusively, on the average achievement gains of charter school students compared to traditional public school students across the full sample of students in each study. However, the CREDO studies also compare achievement gains within subgroups of students by race, special education, and English language learner status, deciles of prior achievement, and the number of years enrolled in charter schools. Other analyses include assessments of charter school and traditional public school students’ achievement gains in different locales (the “local markets” charter schools serve, metropolitan areas, and states depending on the scope of the report). These findings are substantially more complex and nuanced than the summary table might suggest and are only minimally and inconsistently addressed in three of the five WWC reports and not addressed in two. (Note 6) Moreover, marked differences in achievement gains within groups of students or schools indicate that simple overall averages are not appropriate for understanding the “effectiveness of a given practice, program, or policy” (WWC, n.d.). In other words, the WWC summary and individual reports are largely silent on a key finding of the CREDO studies—charter school achievement effects vary considerably across settings and subgroups of students—findings that have important implications that practitioners and researchers need to consider. CONCLUSION While the table that is arguably the center of the WWC Review of the CREDO Charter School Studies is a technically correct summary of statistical significance, it provides potentially misleading visual cues that may overemphasize and exaggerate the success of charter schools, adding to the myth of their superiority (Thaler & Sunstein, 2009; see also Fischman & Tefera, 2014 and Berliner, Glass & Associates, 2014). This review is unique in the WWC database because it summarizes across multiple single study and quick reviews of studies using the same research design. It is also the starkest representation of an inconsistency in how WWC reports are presented to the public. Half of the 14 other studies of school choice summarized as either quick or single study reviews address the practical significance of the studies’ findings on each review’s web page – the easiest to access source of information about a given study. More detailed reports can be easily downloaded in PDF form. In the latter, the statistical significance of findings and effect sizes are reported in narrative form, much like the information presented in Table 2 above. In the WWC Review of the CREDO Charter School Studies, the effort to summarize across multiple studies resulted in an oversimplification that obscures more than it illuminates. Likewise, the summary report and the individual reviews largely do not address a significant aspect of all of the CREDO analyses—the variations in achievement gains across subgroups of students and different types of charter schools. If summaries generated from research studies are intended to be useful guides to practitioners, they must provide a consistent and careful accounting of findings that allows them to assess their practical importance. Summaries of research that are hard to understand and misleading run the risk of eroding practitioners’ trust in research and increasing rather than bridging the gulf between research and practice. REFERENCES Berliner, D. C., Glass, G. V & Associates. (2014). 50 myths and lies that threaten America’s public schools: The real crisis in education. NY: Teachers College Press. Campbell, D. T. & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand-McNally. Center for Research on Education Outcomes. (2012, December 12). Charter school performance in Indiana. Retrieved from http://credo.stanford.edu/pdfs/IN_2012_FINAL_20130117nw.pdf Center for Research on Education Outcomes. (2013, February 20). Charter school performance in New York City. Retrieved from http://credo.stanford.edu/documents/NYC_report_2013_FINAL_20130219_000.pdf Hedges, L. V. & Olkin, I. (1980). Vote-counting methods in research synthesis. Psychological Bulletin, 88(2), 359-369. Fischman G. E. & Tefera A. A. (2014). Qualitative inquiry in an age of educationalese. Education Policy Analysis Archives, 22 (7). Konstantopolis, S. & Hedges, L. (2008). How large an effect can we expect from school reforms? Teachers College Record 110(8), 1611-1638. Levin, H.M., Glass, G.V & Meister, G.R. (1986). The political arithmetic of cost-effectiveness analysis. Kappan, 68, No. 1, 69-72. Levin, H. M.; Glass, G. V & Meister, G.R. (1987). Different approaches to improving performance at school. Zeitschrift fur Internationale Erziehungs und Sozial Wissenschaftliche Forschung, 3, 156-176. Lipsey, M.W., Puzio, K., Yun, C., Hebert, M.A., Steinka-Fry, K., Cole, M.W., Roberts, M., Anthony, K.S., Busick, M.D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. (NCSER 2013-3000). Washington, DC: Institute of Education Sciences, U.S. Department of Education. Thaler, R. H., & Sunstein, C. R. Nudge: Improving decisions about health, wealth, and happiness. New York: Penguin Books. What Works Clearinghouse. (n. d.) Topics in education. http://ies.ed.gov/ncee/wwc. Retrieved February 23, 2014 from http://ies.ed.gov/ncee/wwc/topics.aspx. What Works Clearinghouse. (2010a, February). WWC Quick Review of the Report “Multiple Choice: Charter School Performance in 16 States.” Retrieved from http://ies.ed.gov/ncee/wwc/pdf/quick_reviews/charterschools_021710.pdf What Works Clearinghouse. (2010b, July). WWC review of the report “Charter School Performance in New York.” Retrieved from http://ies.ed.gov/ncee/wwc/pdf/quick_reviews/nyccharter_070710.pdf. What Works Clearinghouse. (2011, September). WWC review of the report “Charter School Performance in Indiana.” Retrieved from http://ies.ed.gov/ncee/wwc/pdf/quick_reviews/incharter_093011.pdf. 9 What Works Clearinghouse. (2013, October). WWC review of the report “Charter School Performance in New Jersey.” Retrieved from es.ed.gov/ncee/wwc/pdf/single_study_reviews/wwc_njcharter_100113.pdf. What Works Clearinghouse. (2014a, January). WWC review of the CREDO charter school studies. Retrieved from http://ies.ed.gov/ncee/wwc/SingleStudyReview.aspx?sid=220. What Works Clearinghouse. (2014b, January). WWC review of the report “National Charter School Study: 2013.” Retrieved from http://ies.ed.gov/ncee/wwc/pdf/single_study_reviews/wwc_ncss_012814.pdf. What Works Clearinghouse. (2014c). Procedures and standards handbook, Version 3.0. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_stand ards_handbook.pdf. NOTES The WWC provides four types of review: a) intervention guides that review analyses of specific interventions; b) practice guides that are produced by an expert panel and are aimed at providing clear, research-based guidance for practitioners; c) quick reviews; and d) single study reviews. Feeder schools are traditional public schools that had students transfer to one of the charter schools in the study sample. All of the reviews provide a similar note of caution about the research design that alludes to the possibility of regression effects. These vary somewhat across the five reports. The most elaborate version states the following: “[U]nobserved differences between [charter school and traditional public school students] may have existed. For example, charter school students may have been more motivated to do well in school or may have had other unobserved characteristics that influenced student achievement. This means the study’s results do not necessarily isolate the effect of charter schools.” (WWC, 2013) According to the WWC protocol, the highest rating that quasi-experimental studies such as the CREDO charter school studies can receive is “meets WWC group design standards with reservations” because even if groups are equivalent on observed characteristics, there may be important differences between a treatment group and the comparison group on unobserved characteristics that may “introduce bias into an estimate of the effect of the intervention” (WWC, 2014b, pp. 10-11). All five of the CREDO studies met the WWC’s standards with reservations. As the WWC Review of the CREDO Charter School Studies noted, the effect sizes for the CREDO studies are not directly comparable to the effect sizes reported for other studies because the CREDO studies compare achievement gains between charter school students and traditional public school students whereas other studies compare the two groups on different outcomes (e.g., reading and math achievement, high school graduation, and college attendance). While outside the scope of the discussion here, this standard is also problematic because it provides a normative standard for interpreting findings without a clear rationale (Konstantopolis & Hedges, 2008; see also Lipsey et al., 2012). It is significant here because this is ostensibly the information a practitioner, the target audience for WWC reviews would have at hand to assess the findings reported in single study reviews. The WWC report of the 16-state study discusses the school-level comparisons (WWC, 2010), the Indiana report summarizes the decile comparison (WWC, 2011), and the New Jersey report highlights the findings for Newark (WWC, 2013).html>

High Button Shoes and Education Reform

2014

High Button Shoes and Education Reform Gene V Glass As a very young child, I was fascinated by my grandmother’s collection of button hooks. It was the mid-1940s and high button shoes had been out of style for decades. She collected the hooks that were used to pull the tiny buttons through the holes that ran up the sides of the ladies’ shoes back in the early 20th century. Often made of silver with decorative handles of porcelain or glass, the hooks made for an attractive display. Warren Buffett and Charlie Munger, his long-time partner at Berkshire-Hathaway, also had a connection to shoe buttons. Munger’s grandfather had managed to corner the market on shoe buttons back around 1900. The grandfather exercised a virtual monopoly over their production and sale. Emboldened by his business acumen, the old man grew to believe that he not only knew more than anyone about shoe buttons but that he knew more than anyone about anything—and he preached and proclaimed at length on such. Munger and Buffett named the syndrome the Shoe Button Complex, and they encountered it frequently in their dealings with successful business practitioners. Now Buffett struggled assiduously to avoid developing the Shoe Button Complex. As one of the richest persons in the world, the temptation to succumb would surely have been great. He was careful to restrict his actions and speaking to what he called his Circle of Competence. He recognized that there were a limited number of things he could know well, and he did not presume to act as though he was expert of those things lying outside the Circle. Buffett’s recognition of his Circle of Competence led to some surprising decisions as he approached and passed his 80th birthday. He did not understand the emerging digital technology, and while the masses were losing their shirts trading high tech companies, Buffett was buying stocks in railroads and underwear manufacturers. Poseurs with enormous Circles of Competence scorned the old man when he reminded them that in the past 120 years in the U.S. more than 1,000 automobile manufacturers came and went, leaving just four. Buffett’s rise to the top of the list of the world’s billionaires left him with a problem as he saw the end of his career approaching. How does one with a small Circle of Competence give away tens of billions of dollars to some good end? Bill Gates and the Shoe Button Complex Buffett startled the world when he announced a few years ago that he was leaving his great wealth to the Bill & Melinda Gates Foundation. Buffett spoke thus in addressing the question of why he turned over his philanthropy to the Gateses: Bill Gates is the most rational guy around in terms of his foundation. He and Melinda are saving more lives in terms of dollars spent than anyone else. They’ve worked enormously hard on it. He thinks extremely well. He reads thousands of pages a year on philanthropy and health care. You couldn’t have two better people running things. They have done incredible work, they’ve thought it through, their values are right, their logic is right. (p. 768 in The Snowball: Warren Buffett and the Business of Life) Well, Buffett’s decision presents us with a conundrum. If he is not competent to determine the worthwhile recipients of his beneficence, then how is it that he knows that the Gates Foundation is dispensing beneficences in a worthwhile way? Education is an arena particularly prone to attracting Shoe Button Complexes. Everyone has been to school; everybody thinks they know what is wrong with schools. And so we come to the question, How is the Gates Foundation doing these days? This question is of more than passing interest on account of the fact that the Foundation is dispensing roughly $400 Million a year to education related causes. Moreover, some utterances by the benefactors have raised eyebrows among groups that have long made the search for understanding education their preoccupation. For example, Melinda Gates startled her interviewer on an NPR program in 2007 when she seemed to suggest that 100% of high school students should continue their education into college. (Interviewer) I just want to ask you about a statement on the Bill and Melinda Gates Foundation's Web site that I read, which is that, "All students in the United States can and must graduate from high school, and they must leave with the skills necessary for college, work, and citizenship." I think everyone would agree that they better leave with the skills for citizenship because everyone can vote at age 18, and we urge them to. College. Can we reasonably expect 100 percent of high school students to become college students? (Melinda Gates) Yes, I think we can. And, in fact, I'm here today in the Chicago school district visiting with students – huge number of Latinos and African-American populations, and guess what? I'm in schools where 95 to 98 percent of these kids are going on to college, and it's because they started freshman year with teachers who believe in them and said, 'These kids can do it.' … (Interviewer) That would be a dramatic increase of the share of high school students, if 100 percent went on to college. I mean, you would be effecting an enormous social change if you could reach – (Melinda Gates) Correct, and that is the idea. (Interviewer) How many years do you think it would take to achieve that particular – (Melinda Gates) I think it is going to take us quite a while. I think that this is a long-term effort and I think it's one that the foundation is going to be at for a very long time. ... (See the entire transcript here.) We may be looking at a Circle of Competence problem here. And the situation may not be much better with her husband, who teeters dangerously close to the edge of a Shoe Button Complex. Bill Gates has referred to Diane Ravitch as “public enemy #1” of effective education. Whether either Diane Ravitch or the nation’s schools fall within his Circle of Competence is questionable. A few years ago when Gates testified to Congress on the current state and future of American education, he spent most of his time complaining about difficulties in obtaining visas and green cards for young tech employees of Microsoft. That his interest might be stronger in promoting the health of this business than in promoting the development of the nation’s children may be understandable. After all, what does the richest man in America really know about the needs of the nation’s children—the vast majority of whom will hold a half dozen low-level jobs during their lifetimes in industries like recreation, food services, child care, health care, and the like? Now my own Circle of Competence does not extend much past some understanding of what is happening to K-12 public education in America. Thanks to Ken Libby of the National Education Policy Center and his analysis of the Gates Foundation grants to U.S. education, we have at hand new information about what Bill & Melinda Gates consider to be efforts to improve schooling that are worthy of their own and Warren Buffett’s support. The following table shows Libby’s breakdown of where Gates Foundation money for education went in the three years from 2008 through 2010. Category Total $ 2008-2010 % Charters $73.1M 7 Alternative Public Schools $97.8M 9 Private Schools $47.4M 4 Small Schools $30.1M 3 School Reforms $61.4M 6 Government Agencies $9.0M 1 Advocacy $116.8M 11 Think Tanks $10.0M 1 Research $78.6M 7 Development $112.6M 10 College/Career Ready $52.4M 5 College Completion $145.4M 13 Common Core $18.5M 2 Early Learning $46.8M 4 Conferences $10.0M 1 STEM $26.0M 2 Human Capital $104.0M 9 Media $17.0M 2 RTTT & i3 $4.7M 0 Other $40.2M 4 Total $1,101.8M 100% I leave it to the readers to make their own interpretations of the mind-set that lies behind these kinds of allocations. As for me, that mind-set shows little faith in the development of better education for the vast majority of America’s children, particularly children in poverty. It is a mind-set that comes about from drinking the Kool-Aid that the “market” will lead America’s schools to the promise land. Consider the following example of how money from the Gates Foundation is being spent. A proposed law in Florida named Parent Empowerment was pushed this legislative season by a California-based group called Parent Revolution. Parent Revolution is funded by Gates, the Eli Broad Foundation, and the Walton Foundation. If Parent Empowerment became law in Florida, then 51 percent of the parents in a public school could sign a petition that would give them control of a school and give them the power to decide whether to close it or turn it over to a charter management organization. It is not difficult to see who the true beneficiaries of this bill would be. In fact, the parent Empowerment bill could have been written by lobbyists for the education management industry. (See Diane Ravitch's report on the true parents uprising that defeated this bill that was nonetheless backed by former governor Jeb Bush and current governor Rick Scott.) (Also, see the NEPC report on the EMOs for a revealing look at the breadth of this industry.) The Gates Foundation support of dubious enterprises didn't start with the Parent Empowerment bill. As Diane Ravitch recently remarked, "The Bill & Melinda Gates Foundation puts up the money to ensure that ["Waiting for Superman"] this morality tale of good reformers and bad teachers is shown to state legislatures, to civic groups, to people living in housing projects. The movie itself is financed in part by an evangelical billionaire (Philip Anschutz) who contributes heavily to libertarian and ultra-conservative causes." The Shoe Button Complex in Arizona Jan Brewer, Republican governor of Arizona and famous for issuing a tongue wagging to President Obama, appointed Intel ex-CEO Craig Barrett to chair a council—Ready Arizona--to study and recommend public education reform for the state. It is unclear what Barrett knows about education. One suspects that we are encountering another case of the Shoe Button Complex. Barrett is urging businesses to push school reform. His public utterances strike familiar chords: the future of the entire state rests on the test scores of little kids; more science and math majors will attract businesses to the state; it’s a global economy. After all, the public schools are “suppliers” of labor for businesses. And at Intel, “if a supplier didn’t meet our specifications, we would call the supplier and say, ‘Meet our specifications or we will fire you.’” Apparently, Barrett shares his fellow Republican Mitt Romney’s pleasure in firing people. Of course, what Barrett is actually and unknowingly talking about is crony capitalism: Linking government and business in relationships that favor the economy. Whether the intellectual, moral, physical, and aesthetic well-being of young people is benefited by their education probably never occasions to Barrett and his ilk. Or perhaps "well-being" to Barrett means having acquired a taste for consumerism and a job to support it. In fact, most industry leaders would like to see specialized training pushed down as early in the curriculum as possible so that high school graduates appear in their HR departments job-ready, trained at public expense. And if training kids for Intel just happens to involve piping a bunch of online courses into Arizona public schools, well so much the better since Barrett also serves on the board of K-12 Inc., the nation’s #1 supplier of cyber-courses. Whether the former CEO of Intel knows everything there is to know about selling microprocessors AND education, or whether this is merely another manifestation of the Shoe Button Complex remains to be seen.

The Effect of Experimental Mortality on Internal and External Validity

1971

Stephen G. Jurs & Gene V Glass. (1971). The Effect of Experimental Mortality on the Internal and External Validity of the Randomized Comparative Experiment. The Journal of Experimental Education, 40(1), 62-66.

Lawyers and Courts as Architects of Educational Policy

1979

Jon E. Getz and Gene V Glass. (1979) Lawyers and Courts as Architects of Educational Policy: The Case of Minimal Competence Testing The High School Journal, 62(4), 181-186.

The Financial Impact of Ohio’s Charter Schools

2006

Gene V Glass. (2006). “The Financial Impact of Ohio’s Charter Schools”

Review of As the Twig Is Bent...: Lasting Effects of Preschool Programs

1984

Glass, G.V. & Ellwein, M.C. (1984). Review of As the Twig Is Bent...: Lasting Effects of Preschool Programs. Science, 223, 273-4.