Review of "The handbook of research synthesis"

1995

Cooper, H. and Hedges, L. V. (Eds.) The handbook of research synthesis. New York: Russell Sage Foundation, 1994.
573 pp.
ISBN 0-87154-226-9. $49.95

Reviewed by Gene V Glass
Arizona State University
June 19, 1995

The Handbook of research synthesis is the third volume of a coordinated publication program on meta-analysis sponsored by the Russell Sage Foundation. Starting in 1987 under the direction of a Research Synthesis Committee (Harris Cooper, Thomas Cook, David Cordray, Heidi Hartmann, Larry Hedges, Richard Light, Thomas Louis and Frederick Mosteller), the project has previously produced The future of meta-analysis (Wachter and Straf, 1990) and Meta-analysis for explanation (Cook et al., 1992). The Handbook is by far the largest and most comprehensive publication of this project. It means to be the "definitive vade mecum for behavioral and medical scientists intent on applying the synthesis craft."(p. 7) At nearly 600 hundred pages and three pounds, researchers will have to leave their laptops behind.

Although the editors and many of the chapter authors eschew the term "meta-analysis" in favor of the broader "research synthesis," potential readers should understand that the former (statistical analysis of summary statistics from published reports) is the subject of the Handbook and not the more general concerns of theory commensurability or the planning of coordinated investigations suggested by the latter.

The organization of the Handbook follows the common logic of producing a meta-analysis: formulate the question, search the literature, code the information, analyze it, write a report. Some of the chapters are unremarkable, since much of the craft of doing research is routine; this only speaks to the completeness of the work. Chapter 6, "Research Registers" by Kay Dickersin, points to new possibilities. Medicine has databases of prospective, on-going and completed studies; Dickersin identifies 26 of them. Expand them slightly to include the actual data from clinical trials and other forms of study and many of the more vexing problems of meta- analysis (which arise from the telescoping of primary data into summary statistics--and the discarding of the former) will be solved. It is past time when behavioral research, both on-going and completed, is catalogued and archived. Telecommunications has driven the costs of information storage and retrieval to near zero. Who will create the Internet Behavioral Research Archives?

Two themes imparted by the editors and the committee, one presumes, give the Handbook of research synthesis its distinctive character. Chapter 1 by the editors, Harris Cooper and Larry Hedges, is entitled "Research Synthesis as a Scientific Enterprise." Research synthesis is likened to doing science itself: both are seen as involving problem formulation, data collection, data evaluation, analysis and publication. These stages in both the pursuit of science and the conduct of research synthesis give the Handbook its section titles, and perhaps its entire bent. Although these stages might reasonably describe the stages in carrying out a meta-analysis, they do not capture what is distinctive about science. The stages describe as well how one may conduct the evaluation of a device, a drug and program or what-have-you. In effect, the Handbook draws no clear or convincing line between the pursuit of scientific theory and the evaluation of technology. This line is quite important and must be drawn.

To cast meta-analysis as dedicated to the construction of science disposes the discussion of it in the direction of classical statistical methods that evolved alongside quantitative science in the 20th century. In particular, the methods of statistical hypothesis testing have come to be associated with the scientific enterprise. The unwholesome effects of this association are the subject of a brilliant article by Paul Meehl (1990) on the progress of "soft psychology"; see particularly the Appendix where Meehl briefly addresses meta-analysis. Just as scientists bring forth hypotheses to be accepted or rejected by data, so do statisticians devise the strategies by which data are judged to be in accord with or at odds with the hypotheses. This view of statistics gives the Handbook its other defining theme: meta-analyses involve the testing of statistical hypotheses about parameters in populations of research studies.

The appropriate role for inferential statistics in meta- analysis is not merely unclear, it is seen quite differently by different methodologists. These differences are not reflected in the Handbook. In 1981, in the first extended discussion of the topic, McGaw, Smith and I raised doubts about the applicability of inferential statistics in meta-analysis. Inference at the level of persons within studies (of the type addressed by Becker in Chapter 15, "Combining Significance Levels") seemed quite unnecessary to us, since even a modest size synthesis will involve a few hundred persons (nested within studies) and lead to nearly automatic rejection of null hypotheses. Moreover the chances are remote that these persons or subjects within studies were drawn from defined populations with anything approaching probabilistic techniques; hence, probabilistic calculations advanced as if subjects had been randomly selected are dubious. At the level of "studies," the question of the appropriateness of inferential statistics can be asked again, and the answer again seems to be negative. There are two instances in which common inferential methods are clearly appropriate: when a defined population has been randomly sampled and when subjects have been randomly assigned to conditions in a controlled experiment. In the latter case, Fisher showed how the permutation test can be used to make inferences to the universe of all possible permutations. But this case in of little interest to meta-analysts who never assign units to treatments. The typical meta-analysis virtually never meets the condition of probabilistic sampling of a population (though in one instance (Smith, Glass & Miller, 1980), the available population of drug treatment experiments was so large that it was in fact randomly sampled for the meta-analysis). Inferential statistics has little role to play in meta-analysis: "The probability conclusions of inferential statistics depend on something like probabilistic sampling, or else they make no sense." (p. 199)

It is common to acknowledge that many data sets fail to meet probabilistic sampling conditions, but to argue that one might well treat the data in hand "as if" it were a random sample of some hypothetical population. Under this supposition, inferential techniques are applied and the results inspected. The direction taken by the Handbook editors and authors mirrors the earliest published opinion on this problem, expressed by Mosteller and his colleagues in 1977: "One might expect that if our MEDLARS approach were perfect and produced all the papers we would have a census rather than a sample of the papers. To adopt this model would be to misunderstand our purpose. We think of a process producing these research studies through time, and we think of our sample--even if it were a census--as a sample in time from the process. Thus, our inference would still be to the general process, even if we did have all appropriate papers from a time period." (Gilbert, McPeek and Mosteller, 1977, p. 127; quoted in Cook et al., 1992, p. 291) This position is repeated in slightly different language by Hedges in Chapter 3, "Statistical Considerations": "The universe is the hypothetical collection of studies that could be conducted in principle and about which we wish to generalize. The study sample is the ensemble of studies that are used in the review and that provide the effect size data used in the research synthesis." (p. 30)

These notions appear to be circular. If the sample is fixed and the population is allowed to be hypothetical, then surely the data analyst will imagine a population that resembles the sample of data. Or as Gilbert, McPeek and Mosteller viewed it, the future will resemble the past if the past is all one has to go on. Hence all of these "hypothetical populations" will be merely reflections of the samples in hand and there will be no need for inferential statistics. Or put another way, if the population of inference is not defined by considerations separate from the characterization of the sample, then the population is merely a large version of the sample. With what confidence is one able to generalize the character of this sample to a population that looks like the sample writ large? Well, with a great deal of confidence, obviously. But then, the population is nothing but the sample.

Hedges and Olkin have developed inferential techniques that ignore the pro forma testing (because of large N) of null hypotheses and focus on the estimation of regression functions that estimate effects at different levels of study characteristics; nearly all of them appear in the Handbook. They worry about both sources of statistical instability: that arising from persons within studies and that which arises from variation between studies. As they properly point out, the study based on 5 persons deserves greater weight than the study based on 500 persons in determining the response of the treatment condition to changes in study conditions. The techniques they present are based on traditional assumptions of random sampling and independence. It is, of course, unclear precisely how the validity of their methods are compromised by failure to achieve probabilistic sampling of persons and studies.

The irony of traditional hypothesis testing approaches applied to meta-analysis is that whereas consideration of sampling error at the level of persons always leads to a pro forma rejection of "null hypotheses" (of zero correlation or zero average effect size), consideration of sampling error at the level of study characteristics (the study, not the person as the unit of analysis) leads to too few rejections (too many Type II errors, one might say). Hedges's homogeneity test of the hypothesis that all studies in a group estimate the same population parameter is the focus of much attention in the Handbook. Once a hypothesis of homogeneity is accepted by Hedges's test, one is advised to treat all studies within the ensemble as the same. Experienced data analysts know, however, that there is typically a good deal of meaningful covariation between study characteristics and study findings even within ensembles where Hedges's test can not reject the homogeneity hypothesis. The situation is nearly exactly parallel to the experience of psychometricians discovering that they could easily interpret several more factors than inferential solutions (maximum- likelihood; LISREL) could confirm. The best data exploration and discovery is more complex and credible than the most exact inferential test. In short, classical statistics seems not able to reproduce the complex cognitive processes that are commonly applied by data analysts.

Rubin (1990) addressed most of these issues squarely and staked out a radical position that appeals to the author of this review : "...consider the idea that sampling and representativeness of the studies in a meta-analysis are important. I will claim that this is nonsense--we don't have to worry about representing a population but rather about other far more important things." (p. 155) These more important things to Rubin are the estimation of treatment effects under a set of standard or ideal study conditions. This process, as he outlined it, involves the fitting of response surfaces (a form of quantitative model building) between study effects (Y) and study conditions (X, W, Z etc.). Of the 32 chapters in the Handbook, only the contribution of Light, Singer and Willett, Chapter 28, "the visual presentation and interpretation of meta-analyses," comes close to illustrating what Rubin has in mind. By far most meta-analyses are undertaken in pursuit not of scientific theory but technological evaluation. The evaluation question is never whether some hypothesis or model is accepted or rejected but rather how "outputs" or "benefits" or "effect sizes" vary from one set of circumstances to another; and the meta-analysis rarely works on a collection of data that can sensibly be described as a probability sample from anything.

Rubin's view of the meta-analysis enterprise would have produced a volume substantially different from that which Cooper and Hedges edited. So we can expect the Handbook of research synthesis to be not the last word on the subject, but one important word on meta-analysis.

References

Glass, G.V; McGaw, B. & Smith, M.L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: SAGE.

Rosenthal, R. (1984). Meta-Analytic Procedures for Social Research. Beverly Hills, CA: SAGE.

Rubin, D.R. (1990). A new perspective. Chp. 14 (pp. 155-165) in Wachter, K.W. and Straf, M.L. (Eds.), The Future of Meta-Analysis. N.Y., N.Y.: Russell Sage Foundation.

Smith, M.L.; Glass, G.V & Miller, T.I. (1980). Benefits of Psychotherapy. Baltimore, MD: Johns Hopkins University Press.

Gilbert, J.P.; McPeek, B. & Mosteller, F. (1977). Progress in surgery and anesthesia: benefits and risks of innovative surgery. In J. P. Bunker, B.A. Barnes & F. Mosteller (eds.) (1977). Costs, Risks and Benefits of Surgery. NY: Oxford University Press.

Cook, T.D.; Cooper, H; Cordray, D.S.; Hartmann, H; Hedges, L.V.; Light, R.J.; Louis, T.A.; & Mosteller, F. (1992). Meta-analysis for explanation: A casebook. New York: Russell Sage Foundation.

Meehl, P.E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195- 244. (Monograph Supplement 1-V66)

Deciding for One or Deciding for Many

2018

The U.S. Charter School Movement and Ethnic Segregation

2000

Cobb, C. D., Glass, G. V & Crockett, C. (2000). The U.S. Charter School Movement and Ethnic Segregation. Paper presented at the Annual Meeting of the American Educational Research Association. New Orleans, LA. April 2000.

Scholarly Electronic Journals: Economic and Technical Issues

2000

Ganesh, T. G., Glass, G. V, Andrews, S., Middleton, J. A., Jennings, T. A. & Leavy, A. (2000). Scholarly electronic journals: Economic and technical issues. In K. E., Sparks & M. Simonson (Eds.), 22nd Annual Proceedings: Selected Research and Development Papers Presented at the 2000 National Convention of the Association for Educational Communications and Technology (pp. 129-136). Columbus, OH: RTS & Associates, Inc.

NEPC Review: Student Assessment During COVID-19

2020 Glass, G.V, Mathis, W.J., & Berliner, D.C. (Sept. 29, 2020). NEPC Review: Student Assessment During COVID-19. Boulder, CO: National Education Policy Center. NEPC Review: "Student Assessment During COVID-19." Gene V Glass William J. Mathis David C. Berliner National Education Policy Center University of Colorado, Boulder Report Being Reviewed: Student Assessment during COVID-19 Laura Jimenez Center for American Progress September 10, 2020 SUMMARY School closings and the ever-increasing number of deaths provide the backdrop for a proposal by the Center for American Progress (CAP) to deny waivers of the federally mandated administration of standardized tests in Spring 2021. Further, the federal government proposes to add to those assessments in ways that CAP argues would make the test results more useful. In its recent report, CAP sides with the Department of Education's policy of denying such requests for waivers, and it calls for additional assessments that "capture multiple aspects of student well-being, including social-emotional needs, engagement, and conditions for learning" as well as supplementary gathering of student information. The report contends this will assure greater equity in the time of the pandemic, supposedly through the addition of the new measures to annual assessments. Although there have been attempts in the past at multi-variable, test-based accountability schemes, the report endorses this less-than-successful approach, citing studies that do not address the complexity of the undertaking or the effects of its implementation. Considering the massive disruption now occurring in schools and the limited utility of standardized tests even in ordinary times, state agencies and local districts are too hard-pressed by fiscal and time demands and the ramping up of health costs to consider even more costly programs of dubious value. For these reasons, the CAP proposal is ill-timed, unrealistic, and inappropriate for dealing with the exigencies arising from the pandemic. I. Introduction With the inexorable ticking of the pandemic death count, historic political unrest in the nation, floods and fires of biblical proportions, and racial unrest, the citizenry searches for respite and a return to better times. It is not hyperbole to say that the nation's democratic institutions are in a state of crisis. The electorate divides, and traditional political stances harden. When the nation's public schools closed early in the Spring of 2020 due to the coronavirus pandemic, education was inescapably drawn into the debate. Public education, the historic reflector as well as the shaper of democracy, saw one wing claiming the nation's educational ills could be healed by continuing to employ high stakes testing and punitive accountability schemes. On the other end of the spectrum, progressives asked schools to embrace practices that support heightened learning and address socio-emotional needs. Into this debate, the Center for American Progress (CAP) released their report, Student Assessment during COVID-19. (Note 1) Directed by a Board of liberal politicians - Tom Daschle, Stacey Abrams, John Podesta, and others - the Center surprisingly finds itself closely allied with the opinion of the conservative Betsy DeVos-led Department of Education. She and the Department ordered states to administer standardized achievement tests in spring 2021 to all public school children, despite the dangers of COVID-19 and the uneven attendance (and enrollment) of many of these students. On September 3, 2020, DeVos sent a letter to the Council of Chief State School Officers informing them that no testing waivers would be granted. (Note 2) The annual every-student testing program has become a common feature of the public education landscape since the passage of NCLB and its successor ESSA. Billions of dollars from the coffers of state agencies and local school districts are spent administering tests required by the federal government if they are to maintain eligibility for federal support. But the costs of mandated state assessments are not solely those levied by the education assessment industry. Annual assessments cost teachers and students weeks focused on "teaching to the tests" and then administering them. State assessments for accountability have the power to turn the activities of the classroom away from a curriculum valued by educators and toward the content of the commercial paper-and-pencil tests. II. Findings and Conclusions of the CAP Report The tenor of the report reviewed here proves to be quite different from what one might assume from its statement supporting testing in 2021. After contrasting the circumstances of spring 2020 when waivers from annual assessment were granted and with what may likely be the circumstances under which the public schools operate in 2021, the report goes on to state: States have sufficient time to plan how to administer not just the state academic assessment next year [2021], but also to establish protocols through which schools can gather additionally critical information about students. A wider spectrum of data can better guide principals, teachers, and families in fulfilling students' needs this school year, which continued disruptions will almost certainly exacerbate until students can return to the classroom (p. 1). It takes an act of blind optimism to expect that schools will be able to pull off such state assessment in 2021, plus have the time and resources to collect other critical information. It is safe to say that neither CAP, the USDOE, nor any of the 50 states knows how many schools will be operating in one form or another at the end of the year. However, the assessment that CAP calls for has little in common with the multiple-choice achievement tests that typify state mandated testing in the post-NCLB era: Assessing refers to the process of collecting data about students in all forms, including academic and nonacademic information .... the author is referring to the entire process of administering assessments and collecting data about student performance through an array of methodologies [including qualitative methods] (p. 2). At this point in the report, it becomes obvious that the thing it calls "assessment" differs markedly in purposes and form from the contemporary practice of state mandated testing. To the burden of delivering instruction in new and unusual ways due to the pandemic, CAP wishes to add the burden of redesigning state assessment systems to include nonacademic data and a new "array of methodologies." The theme of expanding assessment to include new methods and purposes permeates the report - e.g., evaluate impacts of the pandemic and alternative methods of delivering instruction, "assess social-emotional needs, student engagement and attendance, and family engagement." It is addressed less to the question of whether 2021 assessments would be worthwhile, and instead advocates a thorough redesign of education assessment and its purposes. Schools and districts caught in this unanswered dilemma will find little to illuminate their decisions here. III. The Report's Rationale for its Findings and Conclusions The report's rationale is vague and reads: "Parents, educators, administrators, and policymakers need more information about how students are doing and being served, not less." The purpose is to "better understand and address . . . the gaps that have been made worse by the coronavirus pandemic" (p. 2). Notably absent is any rationale for either the practicality or the purpose of this expanded assessment. Also absent is any reference to the inequitable and unequal resourcing of our urban schools. Further ignored is any evidence that assessment, in and of itself, improves educational outcomes. The report is remarkably free of specifics on what new data will be collected and used, offering only statements like "[b]efore students can learn, their well-being, engagement, and conditions for learning must be addressed, and in order to do so, schools must collect these data to inform how they should respond to the challenges raised by the COVID-19 pandemic" (p. 6). The reader is left to speculate on whether the document is designed to improve instruction and education or to serve some other purpose such as perpetuating the test-based accountability systems despite their meager record of success. IV. Use of the Literature There exists in the scholarly literature a plethora of research on the disruption to teaching and learning occasioned by outside mandated testing (Note 3), the redirection of the curriculum caused by mandated assessment (Note 4), the negative effect of such assessment on teachers' instructional approaches (Note 5), as well as the irrelevance of such activities to the practice of classroom teachers (Note 6). Scholars have raised strong objections to the mandated assessments connected to federal accountability programs for the purposes of diagnosing the needs of individual students or the efficacy of individual teachers (Note 7). The CAP report addresses none of this scholarship in shaping its recommendations. On one point, scholars will agree wholeheartedly with a CAP recommendation: "test results should not be used to formally rate teachers or schools" (p. 6). Test-based teacher evaluation - often through Value Added Measurement - is an abysmal failure (Note 8). If state assessment data cannot be used to "formally rate" teachers and schools, they cannot be used to analyze the multiple influences on pretest-posttest score gains. If they are so flawed, then they can hardly be expected to clarify questions about how the pandemic affected test scores, what modes of coping with the pandemic were more or less effective, and similar challenges of causal analysis that CAP puts forward. V. Usefulness of the Study for Policy Making In final form, the general recommendation offered by CAP is as follows: Administering the annual state academic assessments in their current form is likely not practical in the circumstances of next school year. That is why, starting now, states must work with their test vendors and technical advisory committees to identify what is feasible regarding the statewide annual assessment (p. 5). CAP notes six critical questions: How can the assessment be condensed in content and length and still provide useful results? How can assessment results be provided in a timely manner - in 30 days or fewer - so that their results can affect the current school year? How can the assessment be cognizant of digital and connectivity equity concerns and be administered under different scenarios, including at home, at school, or virtually at an off-site location? How must the assessments be adjusted to accommodate the needs of students with disabilities and English-language learners? How can schools be supported in using high-quality curricula that are aligned to statewide standards and assessments? What state assessment policies must be revised to allow for these changes? (pp. 5-6). These threshold issues are not resolved. What CAP calls for in 2021 bears little resemblance to the mandated achievement testing of the last two decades. Assessment, as CAP sees it, must be directed toward new purposes. Among those new purposes is impact analysis - basically causal analysis using non-experimental data. According to the CAP report, the new assessment instruments and strategies being advocated are supposed to evaluate the impacts of the pandemic, alternative methods of delivering instruction, the disadvantages of inadequate internet connections for poor children, "the efficacy of recovery funding for schools," and a raft of other influences. Even attempts at mounting controlled experimental studies to answer questions of this type have produced equivocal findings; but this fact is ignored in the report. It is as though by enumerating grand goals for a mandated assessment the case is made for denying waivers from the assessment. That these goals are unattainable undercuts the argument for insisting that the assessments go forward in 2021. CAP argues that assessments are necessary to do what assessments have never done. The report does not take account of the long development and validation timelines required by the test industry. No attention is given to the increase in home schooling and various choice mechanisms that would invalidate any data used for long-term baselines or measures of progress. VI. Validity of the Findings The CAP report is of little relevance in the current debate regarding continuation of federally mandated assessments in 2021. However, when read as a critique of the current form of education assessment, there is value in its position. The report clearly sees current methods of assessment falling short of having utility for teachers and administrators. Test data, uninformed by the circumstances under which they have been collected, are of little value. If assessment is to contribute to the development of informed policy, test results must be contextualized by the addition of a great deal of data on students' schools, homes, and communities. The assessments imagined by the report do, in fact, "... need to capture multiple aspects of student well-being, including social-emotional needs, engagement, and conditions for learning ...." (p. 1). Of course, this is true but is never done. The sheer cost of such an ambitious plan relegates it to the dead letter file. Before the virus, our urban areas suffered great financial deficits. With Covid-19, schools will find hiring nurses, counselors, and teachers to be more imperative than expending funds on a testing system of little proven value. The report seeks to reinforce its recommendations for a different and better form of assessment by appealing to the need for greater equity in public education noting that "[t]he annual statewide assessment provides critical data to help measure equity in education" (p. 2). It is an arguable point whether state testing has exacerbated inequities among racial, ethnic, and socio-economic groups in the delivery of schooling. Test-based school improvement strategies have been common since the basic skills movement in the 1970s. Likewise, any number of researchers have examined multiple measures to tease out relevant variables. Yet, the achievement gap remains. Instead of a boon, the model assessment envisioned by CAP may be a bane. Test results are used by realtors and home-buyers in crypto-redlining that leads to greater segregation of public schools (Note 9). School choice has contributed to resegregation of public education by offering charter schools to white families seeking to flee diverse schools, is perversely sold as an "equity issue." Indeed, one can argue that mandated assessment in 2021 would be even more unfair than in the past. Poor children are likely to suffer the greatest loss in opportunities to learn and hence show the greatest deficits in test performance. It is an article of faith unsupported by history that these deficits would prompt greater efforts at remediation. One can imagine instead increased use of "retention in grade" as a result of lowered test performance, a practice shown repeatedly not to be in the interest of the future success of those retained (Note 10). To the extent that the nation needs to know how badly the COVID-19 pandemic affected school learning, that question may be addressed by the 2021 National Assessment of Educational Progress (NAEP) - at least in so far as NAEP's methods permit such causal analysis. The NAEP's Governing Board voted 12 to 10 in favor of administering NAEP in 2021. That ten NAGB Board members demurred is in itself noteworthy. And there is some sense to moving forward with NAEP. NAEP tests fewer than 1 in 1,000 students in grades 4, 8, and 12. Its disruption to the curriculum is minimal. The results might help clarify the extent of the devastation of the coronavirus, though a clear and convincing answer is anything but guaranteed. VI. Usefulness of the Report Paradoxically, the shortcomings of the report highlight the case for suspending the federally mandated state assessments for the 2020-21 school year. It is already apparent only weeks into the academic year that the current school year will operate under unusual circumstances: start times will differ not only among states but among districts within states; the suspension of face-to-face instruction that occurred in spring 2020 could well take place again in spring 2021; methods adopted by districts to cope with the pandemic will differ substantially. A promised benefit of spring 2021 assessments is the parsing of the effects of these multiple influences on test data. Unfortunately, this has been tried countless times under far more favorable circumstances and has not proven successful. An augmented assessment simply interferes with the schools' orderly recovery and strengthening from the multiple traumas of the past year. While it is hoped that serendipitous advantages will come to light, schools need vast latitudes - not expanded mandates from the federal government. Such mandates are not only tone deaf, they are disruptive and ill advised. For reasons contained in this CAP report, in fact, states should not have to request waivers of annual testing from the USDOE. The Department should announce suspension of the mandate without delay. Notes Jimenez. L. (September 10, 2020). Student Assessment During COVID-19. Retrieved September 18, 2010 from https://www.americanprogress.org/issues/education-k-12/reports/2020/09/10/490209/student-assessment-covid-19/ Chambers, J. (2020, September 3). DeVos rejects waiver requests for state assessments. Detroit News. Retrieved September 23, 2020 from https://www.detroitnews.com/story/news/education/2020/09/03/michigan-schools-still-need-give-students-state-exams/5708833002/ Smith, M.L., Heinecke, W., & Noble, A J. (1999). Assessment policy and political spectacle. Teachers College Record, 101(2), 157-191. Cole, H., Hulley, K., & Quarles, P. (2020). Does Assessment Have to Drive the Curriculum? Forum on Public Policy. Retrieved September 23, 2020 from https://files.eric.ed.gov/fulltext/EJ864817.pdf Prince Charles made headlines regarding the National Curriculum and examinations in Great Britain: “I want to encourage teachers to enrich their teaching despite the straitjacket of assessment.” And he continued, “More frequent exams mean that the time for learning has shrunk and that leads to defensive teaching.” Quoted in Peterson, B. (2002). Testing reigns in Britain but resistance is growing. Rethinking Schools Online. Retrieved October 12, 2020 from https://rethinkingschools.org/articles/e-s-e-a-watch-2/ McNeil, L.M. (1988) Contradictions of school reform: Educational costs of standardized testing. London: Routledge. Dawson, P., Bearman, M., Boud, D.J., Hall, M., Molloy, E.K., Bennet, S. & Joughin, G. (2013). Assessment Might Dictate the Curriculum, But What Dictates Assessment? Teaching & Learning Inquiry: The ISSOTL Journal, 1(1), 107-111. See generally the blog VAMboozled: A Blog About Teacher Evaluation, Accountability, and Value-Added Models (VAMs). Retrieved October 12, 2020, from http://vamboozled.com/ DeRoche, T. (2020). A fine line: How most American kids are kept out of the best public schools. Los Angeles: Redtail Press. Shepard, L. A. & Smith, M. L. (Eds.) (1989). Flunking grades: Research and policies on retention. London: The Falmer Press.

An Extendor Role for Nurses

1965

Morgan, J.R., Glass, G.V, Stevens, H.A., & Sindberg, R.M. (1965). A study of an extender role for nursing service personnel. Nursing Research, 14, 330-334.

Editorial, Review of Educational Research

1970

Gene V. Glass (1970). Editorial, Review of Educational Research, 40(3).

The first volume of the Review of Educational Research which was published in 1931 contained a complete listing of all 329 members of the American Educational Research Association! AERA has grown to nearly 10,000 members, and educational research has grown with it. As the organization and the discipline have changed, AERA's publication program has changed. In 1964, the American Educational Research Journal was created as an outlet for original contributions to educational research. In April 1969, the Editorial Board of the Review of Educational Research proposed to the Association Council that the nearly forty-year old Review be reorganized so it might better serve the purposes of the Association's publications program. Since 1931, the Review has published solicited review manuscripts organized around a topic for each issue. Generally, a slate of fifteen topics was chosen by the Editorial Board and each topic was reviewed once each three years in one of the five issues per volume. An issue chairman was selected by the Editor and given the authority to choose chapters and authors for the issue. The Review has served well for many years both the discipline of educational research and the profession of education. As an organization and as a profession, we are grateful for the generosity and efforts of the more than 1,000 scholars who have contributed to the Review.

The purpose of the Review has always been the publication of critical, integrative reviews of published educational research. In the opinion of the Editorial Board, this goal can now best be achieved by pursuing a policy of publishing unsolicited reviews of research on topics of the contributor's choosing. In reorganizing the Review, AERA is not turning away from the task of periodically reviewing published research on a set of broad topics. The role played by the Review in the past will be assumed by an Annual Review of Educational Research, which AERA is currently planning. The reorganization of the Review of Educational Research is an acknowledgment of a need for an outlet for reviews of research that are initiated by individual researchers and shaped by the rapidly evolving interests of these scholars.

Knowledge about education is not increasing nearly as fast as is alleged, but the proliferation of the educational research literature is obvious. A body of literature can grow faster than a body of knowledge when it swells with false knowledge, inconclusive or contradictory findings,, repetitive writing or simple dross. If knowledge is not subjected to scrutiny, it cannot be held confidently to be true. Moreover, if knowledge is to be "known" it must be "packed down" into assimilable portions either in reviews of literature or in textbooks. The integration of isolated research reports and the criticism of published works serve an essential purpose in the growth of a discipline. The organization and maintenance of old knowledge is no less important than the discovery of new knowledge. It is hoped that the new editorial policy of the Review, with its implicit invitation to all scholars, will contribute to the improvement and growth of disciplined inquiry on education.

The third number of the fortieth volume of the Review is the first issue to be published under the new editorial policy.

Gene V Glass
Editor

Testing for competence: Translating reform policy into practice

1991

Ellwein, M. C. and Glass, G. V (1991). Testing for competence: Translating reform policy into practice. Educational Policy, 5(1), 64-78.