NEPC Review: Student Assessment During COVID-19

2020 Glass, G.V, Mathis, W.J., & Berliner, D.C. (Sept. 29, 2020). NEPC Review: Student Assessment During COVID-19. Boulder, CO: National Education Policy Center. NEPC Review: "Student Assessment During COVID-19." Gene V Glass William J. Mathis David C. Berliner National Education Policy Center University of Colorado, Boulder Report Being Reviewed: Student Assessment during COVID-19 Laura Jimenez Center for American Progress September 10, 2020 SUMMARY School closings and the ever-increasing number of deaths provide the backdrop for a proposal by the Center for American Progress (CAP) to deny waivers of the federally mandated administration of standardized tests in Spring 2021. Further, the federal government proposes to add to those assessments in ways that CAP argues would make the test results more useful. In its recent report, CAP sides with the Department of Education's policy of denying such requests for waivers, and it calls for additional assessments that "capture multiple aspects of student well-being, including social-emotional needs, engagement, and conditions for learning" as well as supplementary gathering of student information. The report contends this will assure greater equity in the time of the pandemic, supposedly through the addition of the new measures to annual assessments. Although there have been attempts in the past at multi-variable, test-based accountability schemes, the report endorses this less-than-successful approach, citing studies that do not address the complexity of the undertaking or the effects of its implementation. Considering the massive disruption now occurring in schools and the limited utility of standardized tests even in ordinary times, state agencies and local districts are too hard-pressed by fiscal and time demands and the ramping up of health costs to consider even more costly programs of dubious value. For these reasons, the CAP proposal is ill-timed, unrealistic, and inappropriate for dealing with the exigencies arising from the pandemic. I. Introduction With the inexorable ticking of the pandemic death count, historic political unrest in the nation, floods and fires of biblical proportions, and racial unrest, the citizenry searches for respite and a return to better times. It is not hyperbole to say that the nation's democratic institutions are in a state of crisis. The electorate divides, and traditional political stances harden. When the nation's public schools closed early in the Spring of 2020 due to the coronavirus pandemic, education was inescapably drawn into the debate. Public education, the historic reflector as well as the shaper of democracy, saw one wing claiming the nation's educational ills could be healed by continuing to employ high stakes testing and punitive accountability schemes. On the other end of the spectrum, progressives asked schools to embrace practices that support heightened learning and address socio-emotional needs. Into this debate, the Center for American Progress (CAP) released their report, Student Assessment during COVID-19. (Note 1) Directed by a Board of liberal politicians - Tom Daschle, Stacey Abrams, John Podesta, and others - the Center surprisingly finds itself closely allied with the opinion of the conservative Betsy DeVos-led Department of Education. She and the Department ordered states to administer standardized achievement tests in spring 2021 to all public school children, despite the dangers of COVID-19 and the uneven attendance (and enrollment) of many of these students. On September 3, 2020, DeVos sent a letter to the Council of Chief State School Officers informing them that no testing waivers would be granted. (Note 2) The annual every-student testing program has become a common feature of the public education landscape since the passage of NCLB and its successor ESSA. Billions of dollars from the coffers of state agencies and local school districts are spent administering tests required by the federal government if they are to maintain eligibility for federal support. But the costs of mandated state assessments are not solely those levied by the education assessment industry. Annual assessments cost teachers and students weeks focused on "teaching to the tests" and then administering them. State assessments for accountability have the power to turn the activities of the classroom away from a curriculum valued by educators and toward the content of the commercial paper-and-pencil tests. II. Findings and Conclusions of the CAP Report The tenor of the report reviewed here proves to be quite different from what one might assume from its statement supporting testing in 2021. After contrasting the circumstances of spring 2020 when waivers from annual assessment were granted and with what may likely be the circumstances under which the public schools operate in 2021, the report goes on to state: States have sufficient time to plan how to administer not just the state academic assessment next year [2021], but also to establish protocols through which schools can gather additionally critical information about students. A wider spectrum of data can better guide principals, teachers, and families in fulfilling students' needs this school year, which continued disruptions will almost certainly exacerbate until students can return to the classroom (p. 1). It takes an act of blind optimism to expect that schools will be able to pull off such state assessment in 2021, plus have the time and resources to collect other critical information. It is safe to say that neither CAP, the USDOE, nor any of the 50 states knows how many schools will be operating in one form or another at the end of the year. However, the assessment that CAP calls for has little in common with the multiple-choice achievement tests that typify state mandated testing in the post-NCLB era: Assessing refers to the process of collecting data about students in all forms, including academic and nonacademic information .... the author is referring to the entire process of administering assessments and collecting data about student performance through an array of methodologies [including qualitative methods] (p. 2). At this point in the report, it becomes obvious that the thing it calls "assessment" differs markedly in purposes and form from the contemporary practice of state mandated testing. To the burden of delivering instruction in new and unusual ways due to the pandemic, CAP wishes to add the burden of redesigning state assessment systems to include nonacademic data and a new "array of methodologies." The theme of expanding assessment to include new methods and purposes permeates the report - e.g., evaluate impacts of the pandemic and alternative methods of delivering instruction, "assess social-emotional needs, student engagement and attendance, and family engagement." It is addressed less to the question of whether 2021 assessments would be worthwhile, and instead advocates a thorough redesign of education assessment and its purposes. Schools and districts caught in this unanswered dilemma will find little to illuminate their decisions here. III. The Report's Rationale for its Findings and Conclusions The report's rationale is vague and reads: "Parents, educators, administrators, and policymakers need more information about how students are doing and being served, not less." The purpose is to "better understand and address . . . the gaps that have been made worse by the coronavirus pandemic" (p. 2). Notably absent is any rationale for either the practicality or the purpose of this expanded assessment. Also absent is any reference to the inequitable and unequal resourcing of our urban schools. Further ignored is any evidence that assessment, in and of itself, improves educational outcomes. The report is remarkably free of specifics on what new data will be collected and used, offering only statements like "[b]efore students can learn, their well-being, engagement, and conditions for learning must be addressed, and in order to do so, schools must collect these data to inform how they should respond to the challenges raised by the COVID-19 pandemic" (p. 6). The reader is left to speculate on whether the document is designed to improve instruction and education or to serve some other purpose such as perpetuating the test-based accountability systems despite their meager record of success. IV. Use of the Literature There exists in the scholarly literature a plethora of research on the disruption to teaching and learning occasioned by outside mandated testing (Note 3), the redirection of the curriculum caused by mandated assessment (Note 4), the negative effect of such assessment on teachers' instructional approaches (Note 5), as well as the irrelevance of such activities to the practice of classroom teachers (Note 6). Scholars have raised strong objections to the mandated assessments connected to federal accountability programs for the purposes of diagnosing the needs of individual students or the efficacy of individual teachers (Note 7). The CAP report addresses none of this scholarship in shaping its recommendations. On one point, scholars will agree wholeheartedly with a CAP recommendation: "test results should not be used to formally rate teachers or schools" (p. 6). Test-based teacher evaluation - often through Value Added Measurement - is an abysmal failure (Note 8). If state assessment data cannot be used to "formally rate" teachers and schools, they cannot be used to analyze the multiple influences on pretest-posttest score gains. If they are so flawed, then they can hardly be expected to clarify questions about how the pandemic affected test scores, what modes of coping with the pandemic were more or less effective, and similar challenges of causal analysis that CAP puts forward. V. Usefulness of the Study for Policy Making In final form, the general recommendation offered by CAP is as follows: Administering the annual state academic assessments in their current form is likely not practical in the circumstances of next school year. That is why, starting now, states must work with their test vendors and technical advisory committees to identify what is feasible regarding the statewide annual assessment (p. 5). CAP notes six critical questions: How can the assessment be condensed in content and length and still provide useful results? How can assessment results be provided in a timely manner - in 30 days or fewer - so that their results can affect the current school year? How can the assessment be cognizant of digital and connectivity equity concerns and be administered under different scenarios, including at home, at school, or virtually at an off-site location? How must the assessments be adjusted to accommodate the needs of students with disabilities and English-language learners? How can schools be supported in using high-quality curricula that are aligned to statewide standards and assessments? What state assessment policies must be revised to allow for these changes? (pp. 5-6). These threshold issues are not resolved. What CAP calls for in 2021 bears little resemblance to the mandated achievement testing of the last two decades. Assessment, as CAP sees it, must be directed toward new purposes. Among those new purposes is impact analysis - basically causal analysis using non-experimental data. According to the CAP report, the new assessment instruments and strategies being advocated are supposed to evaluate the impacts of the pandemic, alternative methods of delivering instruction, the disadvantages of inadequate internet connections for poor children, "the efficacy of recovery funding for schools," and a raft of other influences. Even attempts at mounting controlled experimental studies to answer questions of this type have produced equivocal findings; but this fact is ignored in the report. It is as though by enumerating grand goals for a mandated assessment the case is made for denying waivers from the assessment. That these goals are unattainable undercuts the argument for insisting that the assessments go forward in 2021. CAP argues that assessments are necessary to do what assessments have never done. The report does not take account of the long development and validation timelines required by the test industry. No attention is given to the increase in home schooling and various choice mechanisms that would invalidate any data used for long-term baselines or measures of progress. VI. Validity of the Findings The CAP report is of little relevance in the current debate regarding continuation of federally mandated assessments in 2021. However, when read as a critique of the current form of education assessment, there is value in its position. The report clearly sees current methods of assessment falling short of having utility for teachers and administrators. Test data, uninformed by the circumstances under which they have been collected, are of little value. If assessment is to contribute to the development of informed policy, test results must be contextualized by the addition of a great deal of data on students' schools, homes, and communities. The assessments imagined by the report do, in fact, "... need to capture multiple aspects of student well-being, including social-emotional needs, engagement, and conditions for learning ...." (p. 1). Of course, this is true but is never done. The sheer cost of such an ambitious plan relegates it to the dead letter file. Before the virus, our urban areas suffered great financial deficits. With Covid-19, schools will find hiring nurses, counselors, and teachers to be more imperative than expending funds on a testing system of little proven value. The report seeks to reinforce its recommendations for a different and better form of assessment by appealing to the need for greater equity in public education noting that "[t]he annual statewide assessment provides critical data to help measure equity in education" (p. 2). It is an arguable point whether state testing has exacerbated inequities among racial, ethnic, and socio-economic groups in the delivery of schooling. Test-based school improvement strategies have been common since the basic skills movement in the 1970s. Likewise, any number of researchers have examined multiple measures to tease out relevant variables. Yet, the achievement gap remains. Instead of a boon, the model assessment envisioned by CAP may be a bane. Test results are used by realtors and home-buyers in crypto-redlining that leads to greater segregation of public schools (Note 9). School choice has contributed to resegregation of public education by offering charter schools to white families seeking to flee diverse schools, is perversely sold as an "equity issue." Indeed, one can argue that mandated assessment in 2021 would be even more unfair than in the past. Poor children are likely to suffer the greatest loss in opportunities to learn and hence show the greatest deficits in test performance. It is an article of faith unsupported by history that these deficits would prompt greater efforts at remediation. One can imagine instead increased use of "retention in grade" as a result of lowered test performance, a practice shown repeatedly not to be in the interest of the future success of those retained (Note 10). To the extent that the nation needs to know how badly the COVID-19 pandemic affected school learning, that question may be addressed by the 2021 National Assessment of Educational Progress (NAEP) - at least in so far as NAEP's methods permit such causal analysis. The NAEP's Governing Board voted 12 to 10 in favor of administering NAEP in 2021. That ten NAGB Board members demurred is in itself noteworthy. And there is some sense to moving forward with NAEP. NAEP tests fewer than 1 in 1,000 students in grades 4, 8, and 12. Its disruption to the curriculum is minimal. The results might help clarify the extent of the devastation of the coronavirus, though a clear and convincing answer is anything but guaranteed. VI. Usefulness of the Report Paradoxically, the shortcomings of the report highlight the case for suspending the federally mandated state assessments for the 2020-21 school year. It is already apparent only weeks into the academic year that the current school year will operate under unusual circumstances: start times will differ not only among states but among districts within states; the suspension of face-to-face instruction that occurred in spring 2020 could well take place again in spring 2021; methods adopted by districts to cope with the pandemic will differ substantially. A promised benefit of spring 2021 assessments is the parsing of the effects of these multiple influences on test data. Unfortunately, this has been tried countless times under far more favorable circumstances and has not proven successful. An augmented assessment simply interferes with the schools' orderly recovery and strengthening from the multiple traumas of the past year. While it is hoped that serendipitous advantages will come to light, schools need vast latitudes - not expanded mandates from the federal government. Such mandates are not only tone deaf, they are disruptive and ill advised. For reasons contained in this CAP report, in fact, states should not have to request waivers of annual testing from the USDOE. The Department should announce suspension of the mandate without delay. Notes Jimenez. L. (September 10, 2020). Student Assessment During COVID-19. Retrieved September 18, 2010 from https://www.americanprogress.org/issues/education-k-12/reports/2020/09/10/490209/student-assessment-covid-19/ Chambers, J. (2020, September 3). DeVos rejects waiver requests for state assessments. Detroit News. Retrieved September 23, 2020 from https://www.detroitnews.com/story/news/education/2020/09/03/michigan-schools-still-need-give-students-state-exams/5708833002/ Smith, M.L., Heinecke, W., & Noble, A J. (1999). Assessment policy and political spectacle. Teachers College Record, 101(2), 157-191. Cole, H., Hulley, K., & Quarles, P. (2020). Does Assessment Have to Drive the Curriculum? Forum on Public Policy. Retrieved September 23, 2020 from https://files.eric.ed.gov/fulltext/EJ864817.pdf Prince Charles made headlines regarding the National Curriculum and examinations in Great Britain: “I want to encourage teachers to enrich their teaching despite the straitjacket of assessment.” And he continued, “More frequent exams mean that the time for learning has shrunk and that leads to defensive teaching.” Quoted in Peterson, B. (2002). Testing reigns in Britain but resistance is growing. Rethinking Schools Online. Retrieved October 12, 2020 from https://rethinkingschools.org/articles/e-s-e-a-watch-2/ McNeil, L.M. (1988) Contradictions of school reform: Educational costs of standardized testing. London: Routledge. Dawson, P., Bearman, M., Boud, D.J., Hall, M., Molloy, E.K., Bennet, S. & Joughin, G. (2013). Assessment Might Dictate the Curriculum, But What Dictates Assessment? Teaching & Learning Inquiry: The ISSOTL Journal, 1(1), 107-111. See generally the blog VAMboozled: A Blog About Teacher Evaluation, Accountability, and Value-Added Models (VAMs). Retrieved October 12, 2020, from http://vamboozled.com/ DeRoche, T. (2020). A fine line: How most American kids are kept out of the best public schools. Los Angeles: Redtail Press. Shepard, L. A. & Smith, M. L. (Eds.) (1989). Flunking grades: Research and policies on retention. London: The Falmer Press.