2000
Are Data Enough?
Review of Chubb and Moe's Politics, Markets and America's Schools
Gene V Glass
Dewayne A. Matthews
Arizona State University
Politics, Markets, and America's Schools is one of those rare books of educational research that breaks through into the press and public debate of the day. In this case, the subject is the timely one of the effects of school organization on student learning. The book reports on a research project involving an analysis of several national data bases, particularly High School and Beyond (HSB), to attempt to determine what factors lead to high levels of academic performance in schools. Chubb and Moe conclude that problems of academic performance in the schools will not be solved by any of the changes brought on by the school reform movement. The problems are a direct and inevitable result of the structure of American public schools, specifically their control through democratic processes. The solution is autonomy—building-level autonomy of principals and teachers freed of the dead hand of bureaucratic regulation from government and from school boards.
Chubb and Moe's Argument
The authors have reached this conclusion through a comparison of the academic performance of students from public and private schools, and have attributed the better performance of private school students to structural differences between the two types of school. (The authors speak favorably of the Coleman report of 1981 on public and private schools). However, the book attempts to go beyond the simple conclusion that students seem to learn better in private schools to examine the effects of structural differences among schools and their effects on student learning. Their basic premise is far from revolutionary, namely that public schools suffer from excessive levels of bureaucratization and politics. More important, they suggest that excessive bureaucracy is the proximate cause of problems in the schools, and that politics are the ultimate cause of the over-reliance on bureaucracy. The bureaucracy cannot be changed unless the underlying political structure is changed.
Chubb and Moe contend that the results of education are inherently difficult to measure. Hence, only those who are in direct contact with the learners can know what is happening in the schools and judge the effectiveness of teaching. Reliance by bureaucracies on top-down hierarchical management results inevitably in conflict: Effective bureaucracy is commonly built around rules that specify appropriate behavior, rewards, and sanctions that encourage such behavior, and monitoring to ascertain whether goals are being met, whether rules are being followed, and whether the rules and incentive system need to be adjusted. All are rendered highly problematic in education, because good education and the behaviors conducive to it are inherently difficult to measure in an objective, quantifiable, formal manner. The measurement problem makes it difficult or impossible for education administrators to know what they are doing—and their controls, as a result, threaten to be ill suited to the ends they want to achieve. (p. 36)
Because the public schools are governed, funded, and directed through the political process, the interests of parents and students receive no more weight than the interests of any other group. Indeed, Chubb and Moe argue that parents and students will have less influence than others because of certain structural features of the political process, namely the inherent power of organized over disorganized groups. The many demands on schools that result from this political process will be accommodated by the educational system through the establishment of regulations, guidelines and monitoring procedures. This political process leads inevitably to highly bureaucratic modes of organization and management. Rigidly hierarchical bureaucracies are not conducive to effective learning because they do not promote or allow the effective use of professional personnel, particularly those who are in direct contact with students, namely teachers.
Private schools are different, allegedly, because they are insulated from the political process. The crux of Chubb and Moe's argument is the basic distinction between democratic and market control. Public schools are democratically governed through the political process. Private schools are not; the owners are free to run the school as they wish. (Chubb and Moe may fail to take into account the considerable public influence over private schools through state regulations, certification requirements, and the like; though comparatively, the distinction is probably still valid.) However, to attract students the owners of private schools must be responsive to the needs and desires of parents and students. Private schools are, therefore, relatively free to concentrate on that with which parents and students are presumably most interested and concerned, student learning.
Their analysis of the differences in student achievement between independent and public schools suggests to Chubb and Moe that the root cause of poor performance in schools is found in their governance. Chubb and Moe therefore recommend that the present system of public school governance be scrapped in favor of a market-driven one in which parents have primary control over the schools. This recommendation is, however, only academic; Chubb and Moe acknowledge the impossibility of its adoption. Although their recommendation for reform draws more heavily than most on academic theory (of organizations and political bodies, in this case), it is not an unfamiliar proposal; it represents neither discovery nor invention nor new ideas. Why, then, does this book appear now?
The Statistical Study
Chubb and Moe claim uniqueness for their arguments about reform of the organization of schools, and they may be justified. More than most recommendations for school reform (perhaps Coleman or Goodlad are the visible exceptions), their argument grows out of the quantitative analysis of empirical data, specifically data on student achievement, students' families and school organization. Whether the data compel the argument, or even whether the data are up to the task of suggesting policy is a question we will address here.
Chubb and Moe's claim to empirical backing for their policy recommendations rests on a causal argument, namely, that certain aspects of school organization cause student achievement. The causal argument is pursued via the High School and Beyond (HSB) data set and an analysis plan that fits systems of linear equations to the data under specified constraints. The familiar data base comprises 20,000 cases and hundreds of items from questionnaires administered to students, teachers and school principals. Achievement tests were administered twice, first in 1980 and again in 1983 when the student cohort had reached the Senior year. About 100 items tested performance in reading, vocabulary, writing, math and science. Questionnaires probing classroom and school organization, personnel policies and the like were administered to teachers and administrators in 1983-84. In justifying their causal claims, Chubb and Moe recap the standard criticisms of structural equation modelling; it works (i.e., determines causes) if two conditions are satisfied: 1) all third variables are present and accounted for, 2) the direction of causal influence (from putative cause to putative effect) is known a priori or controlled by fixing temporal priority. Chubb and Moe give their work good marks on both counts.
They attempt to cope with the third variable problem "...by allowing student achievement to be influenced by many of the kinds of variables—for example, family SES and student ability—that also ought to predict whether students select their schools" and by using gain in achievement from grade 10 to 12 as the dependent variable. (p. 112) The third variable problem doesn't yield to such modest exertions as scoring a handful of questionnaire items dealing with students' families and their social-economic background. If it did, we in educational research would be adrift in reliable, well-established causal relationships, and James Coleman and many others would live much less controversial careers. Cronbach has elaborated the conditions under which non-experimental data approach the validity of randomized experiments in establishing causal claims: perfectly reliable measurement of exhaustive measures of differences among the different levels of the putative independent variable (Cronbach, 1979). Chubb and Moe's data set comes nowhere near solving the "third variable problem." The twelfth grade achievement variable surely is contaminated with a goodly amount of influence from unmeasured and unreliably measured differences between students and schools. Chubb and Moe have surely attributed some of this influence—perhaps a great deal, no one can know—to their favorite independent variables. We certainly do not fault them for failing to turn surveys into experiments; we only wish that they were quicker to acknowledge that they can not do so.
Even assuming that the remaining difference in student achievement is associated only with organization autonomy differences, how can one be confident that autonomy leads to improved achievement? Can it not be that schools with students who learn at a faster pace are granted greater freedom, either by design or as a result of greater constituent satisfaction? Chubb and Moe acknowledge that in fact student achievement and school organization may bear reciprocal causal relationships to each other—sometimes one causes the other, other times vice-versa, or the causal influence runs in one direction in circumstances A and in the opposite direction in circumstances B. "Organization may be both cause and effect." (p.113) How do they unravel this mystery, a mystery that plagues most attempts at causal modelling that lack longitudinal observations? "We do not wish to pretend that we have a solution to this [ambiguous direction of causality] problem—for we do not—but we do believe we have a workable method of analysis that keeps the ... problem in clear view. Despite all we have said about the problem of reciprocal causality, we believe that the key influences on student achievement tend to run in one direction. We believe that school control affects school organization more than the other way around, and that school organization is primarily a cause of student achievement and not a result of it." (p. 114)
Chubb and Moe state this article of faith, that the problem of ambiguous direction of causality can be solved by willing it away, in a disarmingly direct and simple way—as if one were asserting that the chances of radio wave disruption causing sunspots were too small to be taken seriously. But the ambiguity in the HSB database, for this particular assertion, will not be dispelled so simply. It is equally obvious to some observers closer to American education than Chubb and Moe that high and low student achievement (even that amount left over after imperfect partialing out of pre-achievement scores and a few questionnaire items about family) prompt organizational response. Indeed, precisely the finding on which Chubb and Moe hang their entire proposal for school reform—that organizational autonomy is related to high achievement—is likely to arise from a causal influence of achievement on organization: low achieving schools prompt managers at all levels to intervene to solve the problem of poor performance; high achieving schools are spared the kind of meddling that well-intentioned persons from the state agency to the school building are prone to offer. Is there anything in the data set that lends credibility to one direction of causal influence over the other? Indeed there is. Causes precede effects in time, at least for the notion of causality still used in accounting for human behavior. The HSB surveys measured student achievement in 1980 and 1983; school organization was measured in 1983-84. In view of this sequence, a bit more modesty in making uni-directional causal claims seems called for.
Nor does Chubb & Moe's analytic attempt to unravel the ambiguous causal direction problem engender confidence. They attempt to study the influence of student achievement on school organization by reversing the regression and entering the former as an independent variable and the latter as outcome. This analysis dissolves in a confusing inconsistency, of which far too little is made. Essentially, the continuously measured achievement variable is reported to be not significantly related to school organization while a dichotomized measure of degree of student achievement gain (below vs above average) is reported to be related to school organization (p. 175). A secondary data analysis may be required to straighten out this anomaly.
In the face of the authors' enthusiasm for their findings, even the careful reader—and surely the media and other second-hand consumers of this research—quickly loses sight of the fact that these sweeping recommendations are based on statistical results where the model accounts for only 5% of the variance in the dependent variable of student achievement; we repeat: the multiple R in these analyses is less than .25. Of course, any regression coefficient must be quite small in these circumstances, and that coefficient for the School Organization variable, while significantly non-zero on 20,000 cases, is tiny. One implication of this result is that enormous changes in a school's position on the organization variable will be predicted to yield very small changes on the achievement variable. A school that moves from the 5th percentile to the 95th percentile on autonomous organization would be expected (assuming all problems in causal inference are resolved in Chubb and Moe's favor) to climb a month or so in grade equivalent units on a standardized achievement test. Research and Policy
Chubb and Moe have such confidence in the results of their statistical analyses that they recommend the creation of an entirely new system of public education for the nation. This approach of "policy by regression coefficient" raises some serious questions. Does the largest beta prevail in determining policy? By acting as if policy flowed from statistical analysis of achievement scores, they practice a brand of social science that, while not value-free or value-neutral, is at least value-insensitive. Chubb and Moe's recommendation to remove the bureaucracy from the schools raises value questions that they don't address. Some, but hardly all, of the democratically generated bureaucracy that they wish to strip out of the public schools was created to protect the rights of students. Clumsy though they may be, the rules and regulations often stand as a safeguard against callous and unfair treatment of children, particularly those who suffer handicaps or are ethnic minorities. Have we reached an enlightened state in this country where those safeguards can be dispensed with for the sake of teachers' and administrators' autonomy? Some will doubt it.
Chubb and Moe do a service by raising the issue of governance as it relates to efforts to make schools more effective, but they give scant serious attention to the broad context of American education that they imagine their study reforming. Any effort toward greater autonomy for teachers or toward school-based management will have to recognize the reality of educational governance today, and be responsive to its demands. Developing an understanding of the policy environment in which schools operate is difficult for most educators, as reflected in Chubb and moe's decision to dismiss it wholesale. However, local school boards, state boards of education, and legislatures are the very messy environments in which educational policy will continue to be made. Any benefits that could accrue from more autonomy for educators will only be gained when policy-makers are convinced that it is in the best interest of the public, as defined through their constituencies, to let go of central control. Chubb and Moe, unfortunately, do not make that case.
Policy-makers are, however, greatly preoccupied with issues of autonomy. In the aftermath of the school reform movements of the early 1980s, policy-makers are experimenting with various approaches to enhance educator autonomy. Whether called teacher empowerment, site-based management, or school restructuring, such approaches are the new darlings of both educators and policy-makers interested in educational improvement. Chubb and Moe establish an apparent link between autonomy and school performance, but the direction of influence is ambiguous at best. What is needed is a greater understanding of the ways through which autonomy affects school performance. Case studies and other more narrowly focused research into schools could help develop an understanding of these relationships that could guide both educators and policy-makers in determining the appropriate role of autonomy in school improvement. Some contributions in this regard have been made by S.R. Glass (1997) in her qualitative study of autonomy in public and private schools. Chubb and Moe opt out of this effort by their assertion that school effectiveness is pre-ordained by governance structures which cannot be changed. The widespread experimentation in school restructuring would suggest that most educators and policy-makers do not agree.
Politics, Markets and America's Schools is the research legitimation that the school choice movement has been waiting for. The book has been heralded as an important contribution to knowledge. The authors expected to be attacked by researchers, and they have been. John Witte, a political scientist at the University of Wisconsin at Madison, has attacked Chubb and Moe's data, their analytic techniques and their conclusions. "To suggest that we know enough from High School and Beyond to overthrow the public school system in the United States and replace it with a choice system is sheer madness," Professor Witte charged hyperbolically in an interview published in Education Week (November 14, 1990, Vol. 10, Number 11, p.20). Witte claimed that the data in their original form were flawed beyond repair and that certain transformations performed by Chubb & Moe exaggerated the achievement gains to be expected from school reorganization. But that will hardly matter; in the political battles over school choice, technicalities about empirical research will be brushed aside and Chubb & Moe will be cited as authority by one side and decried as bogus on the other. Will the book raise the level of the debate, introduce new perspectives, lead to better thinking? No, not by itself; it is not that kind of book. It is rather a polemic wrapped in numbers.
Cronbach writes in Designing Evaluations of Educational and Social Programs about two contexts in which social researchers imagine themselves being when they present their work: the context of command and the context of accommodation. The former is a dream of omnipotence in which supremely powerful decision-makers issue directives that others follow. The context of accommodation is the reality of policy-making in American society; compromises are struck between competing interests, sometimes, it is to be hoped, in light of the social researchers' models, findings and ways of thinking. Social research benefits and grows more useful, we believe, when its creators recognize the reality of the context of accommodation. Chubb and Moe, ironically since they are political scientists, act as if they were addressing the non-existent commanders of the American educational system.
This book has received an uncommon amount of attention in the popular press. Its authors have appeared on the Op/Ed page of prominent newspapers to give capsule versions of their position. Famous persons praise the work on the dust jacket; Chester Finn calls the book "...the most eagerly awaited education book of the year, and very likely destined to become the most influential." Why is such attention showered on a rather ordinary regression analysis of a data base that was constructed by the government nearly a decade ago? Chubb and Moe offer the school choice movement the legitimacy that empirical research can confer. Research is today a language of legitimate authority, and political positions are strengthened when it can be cited. The voucher and school choice interest groups have cited Chubb and Moe with glee, as if the long awaited experiment had suddenly proved the rightness of their cause. It is a mark of the maturity of educational research that its findings are so eagerly sought. It is a mark of its undeniable limitations that the findings of educational research still have about them as much of the character of political rhetoric as they have the character of scientific discovery.
References
Cronbach, L.J. (1979). Designing Evaluations of Educational and Social Programs. San Francisco: Jossey-Bass.
Glass, S.R. (1997). Markets and myths: Autonomy in public and private schools. Education Policy Analysis Archives. 5, 1. [Entire issue. Available online at http://epaa.asu.edu/epaa/v5n1.html ]
This review was originally published in 1990 in the Educational Researcher. It was slightly revised as published here. Dr. Matthews is currently at the Western Interstate Commission on Higher Education; his email address is dm@wiche.edu. Glass is still at Arizona State University where his email address is glass@asu.edu.
(5/26/2000)
No comments:
Post a Comment