Not For Sale: Scholarly Communications in Modern Society

1999

Not For Sale: Scholarly Communications in Modern Society

Gene V Glass

In the past thirty years, scholarly publication has seen the emergence and cresting of one technological revolution, and, within the past five years, the emergence of a second. The first harmed the quality of scholarship and restricted access to it. The second technological revolution promises to repair the damage.

The first revolution is the advent of PC based "desktop publishing." The second is PC mediated telecommunications (a.k.a. the "internet").

My father was a printer. In the late 1960s his union struck against the early incursion of computers into the composing room. He never worked another day as a printer; when he retired in 1972, the International Typographical Union was broke--he lost all his retirement. His was the first trade completly wiped out by the computer revolution. I was publishing two books in the late 1960s: one with Prentice-Hall, another with a University Press. Both were thoroughly and rigorously reviewed by anonymous colleagues at the request of the publishers--because publication then was a major financial investment. One was set into type by printers like my father in North Ireland.

In 1974, I witnessed the emergence of the first technological revolution. While editing a book of papers for SAGE Publications I visited Beverly Hills CA and saw how mini-computers in a small office above a drug store on Beverly Drive were changing the face of scholarly publication. Books could now be printed with the expectation that they would sell well under 1000 copies and make a profit.

Today, scholarly publication is in the hands of commercial publishers like Elsevier, Springer, Kluwer, SAGE et al., who publish literally thousands of books and journals each year with profits--by some estimates--of over 40%. And, the stuff is scarcely reviewed at all. More junk is being published today in the name of scholarship and fewer and fewer institutions (libraries) and individuals can afford it. (Notice how Kluwer--to pick one typical example operates. Last week I received a MS from a secretary to an editor whom I'd never heard of who was responsible for the journal Policy Sciences. Please review the MS, it said, and telefax your review to us within three weeks and return the MS by mail. No return envelope was enclosed. If you can not meet the deadline, the letter warned, please pass the MS on to a colleague who could review it! What gaul! And all this from a publisher of dozens of scholarly journals which it sells at a rate of $200 for 300 pages a year to libraries and $50-60 to individuals.) The motive is clear; the means are obvious; the opportunity abounds.

Commercialization is ruining scholarly communication. The solution is simple: computer-mediated telecommunications of scholarly work controlled by scholars working in universities--not university presses--making work freely available to everyone as a public service. I'm talking about the internet, if you didn't recognize it.

I have published for seven years, completely by myself--no secretary, no graduate assistant, no budget--a peer refereed journal in education policy analysis that anyone can access for free on the internet. And many people do; 1,000 persons a weekday from all over the world (Malaysia, Korea, Bulgaria, Brazil and on and on). The journal competes with three other journals in its field whose combined subscriptions total about 4,000. One article in my journal has been downloaded 25,000 times in three years. A national survey of home schooling was published in my journal on March 23 and will surpass 5,000 downloads this week.

I have edited three journals on paper, going back to 1968 with RER. None of them has had peer review even remotely approaching the quality of what my editorial board gives me on my electronic journal. (If Ken Strike will forgive me for divulging that which could not possibly do him any harm, his article in my journal two years ago received 14 peer reviews in two weeks; he said it was one of the most rigorous and helpful experiences of his publishing career.)

Twice I have turned down offers from foundations to support my policy journal. I could not accept the money and continue to argue that commercial publishers have no right to charge what they do for journals and books. We, the scholars who write, can and should take control of our reviewing and publishing, and make the fruits of our labors freely accessible.

Note: Summary of my remarks at Session 6.31 "A Dialog About Electronic Forms of Scholarly Communication" of the American Educational Research Association Annual Meeting; Monday, April 19, 1999

2017

Who Runs Our Public Schools?

Gene V Glass

There was a time when they were run by educators. But all that has changed.

Public education, circa 1945

I was entering Kindergarten in Lincoln, Nebraska, a typical school in a typical town in a typical state. My classmates were all white, as were my teachers, who were all female. The superintendent of schools & principals conferred with teachers about curriculum. Teachers exercised discretion in their choice of books and materials. No one was looking over their shoulders to see that in the first week of February they were teaching Standard #1285 – in fact, there were no standards, and no tests, just books and activities and recess.

At the risk of appearing senile, I cannot fight back the temptation to further compare my experiences of 70 years ago with children’s experiences today. Some might think that Lincoln, Nebraska, was a backward village in the boondocks at the end of WWII. In fact, it was – and still is – a fairly forward-looking town with a very credible university. The college of education at the University of Nebraska was called Teachers College, the name borrowed, I feel certain, from Teachers College at Columbia University. Now Teachers College, Columbia, was the seat of Deweyan progressive education. Many of my teachers would have been trained in an institution that emphasized Dewey’s ideas, and so would have been the very few student teachers to whom I would have been exposed. I recall a nice young woman from the university giving me an individual IQ test when I was in the 4th grade. Nothing much was made of it; she was probably satisfying some requirement for a class.

I don’t wish to romanticize the experiences of my elementary school days, but I think I can summarize their essence in a single comparison. School then was a refuge from life at home and in the neighborhood, which could be harsh and stressful. Today, the tables are turned. For children lucky enough to have loving families, home is a refuge from the stress of life in school. Much has been lost.

Things would soon change. Brown vs Board of Education told the nation that there were two public school systems: one for Whites and one for Blacks; and this had to change, because they were not equal. Brown vs Board was perhaps the first expression of the Civil Rights movement, which was to impact the nation in profound ways over the next 50 years.

Brown vs Board of Education signaled the start of trends in demographics and federal policy that were to radically alter the middle class support for public education. In the past, public schools were paid for by property taxes as they are largely today, and no one objected because we were all a nation of upwardly mobile middle class White families ramping up their fertility – in advance of the discovery of oral contraceptives. It is amazing to note – in light of the contemporary whine about taxes – that in 1955, the highest marginal tax rate of persons making over $1,000,000 a year was 81%. That rich people today complain about 28% marginal tax rates says something about whom they imagine their taxes are supporting: not their children or grandchildren, but other people’s.

The Powell Memorandum

At mid-century, business and government operated fairly independently for the most part and with only a few noteworthy exceptions (e.g., Teapot Dome, 1922). But by 1970, all of this was to change in very profound ways.

Lewis F. Powell, started out as an inconspicuous lawyer in a Richmond, Virginia, law firm. His primary occupation in the years leading up to appearance on the national scene was representing the tobacco industry. For a time, he served on the Virginia State Board of Education. In 1971, the US Chamber of Commerce asked Powell to address the question of the relationship of business to government, a topic that few had written about in the past. Writing in his memo to the Chamber, Powell had this to say: “Business must learn the lesson . . . that political power is necessary; that such power must be assiduously cultivated; and that when necessary, it must be used aggressively and with determination—without embarrassment and without the reluctance which has been so characteristic of American business.” How strange this strikes us in the 21st Century.

The Powell Memorandum went on to outline multiple avenues of influence by which business could influence government: the creation of the profession of Lobbyist; the creation of think tanks (AEI, Cato Institute, Heritage Foundation, Manhattan Institute, et al.); and multiple means of pressuring universities.

And Lewis Powell did not disappear into the shadows after delivering his memorandum to the Chamber of Commerce; no, indeed. He was appointed to the U.S. Supreme Court by Richard Nixon in 1972 and served there for 15 years.

The Corporations & the Rich

Spurred on by the Powell memorandum, influence of corporate America on public education began in earnest in the late 1970s and early 1980s. Reagan gave “education reform” a huge boost when he declared in 1983 that K-12 public education was a failure. His remedies? Vouchers and abstinence education. Reagan's policies set the stage for the next 30 years of change in America. Huge cuts in tax rates for the rich have resulted in a wealth gap in America that is reminiscent of feudal societies in medieval Europe. In 1960, the average CEO’s salary was 20 times the average salary of the worker in the CEO’s company; today, it is several hundred times greater.

While the rich have been getting richer and the poor have been getting poorer over the past 40 years, the poor have been expanding their ranks through heightened fertility and immigration. America today is a bimodal population of young Hispanics & African-Americans and older White people, with an increasingly hollowed out middle class.

I have detailed this evolution in Fertilizers, Pills & Magnetic Strips. And so we find ourselves in the contemporary predicament: Both corporations and a shrinking and aging White middle class are withdrawing their support of public institutions of all types that increasingly are serving other people’s children.

Three Sources of Power Over Public Education: Who Really Runs Our Public Schools?

There are three major wielders of power that increasingly are determining what goes on in public schools: 1) Corporations whose only goal is increased earnings and higher share value by lowering taxes, as well as corporations who seek income directly by supplying services or even running public schools; and 2) super rich dilettantes who think they know better than educators how schools should be run. 3) A third source of power is the courts. I won’t have much to say about them other than that their force can be considerable, but that it can be blocked for years by politicians and special interests.

Rupert Murdoch (of FOX News & Wall Street Journal) sounded the charge a few years ago when he announced to his fellow capitalists that K-12 public education in American was a half-trillion dollar plum waiting to be picked. Among those who heard the bugle call was the UK publishing giant, Pearson PLC (Public Limited Company).

John Fallon, 52, CEO of Pearson, heads a company with $8.2 billion in revenues that is trying to reinvent education. “It doesn’t matter to us whether our customers are hundreds of thousands of individual students and their parents in China, or thousands of school districts in America,” says Fallon. “What we’re trying to do is the same thing—to help improve learning outcomes.” Fallon waxes philosophical as he shares his view of the future: “This idea of the shift from sort of inputs to outcomes, I think, is one that is now becoming all-pervasive in pretty much every area of public policy… .So why should education be any different?” Such utterances betray nothing so much as a naiveté about public policy.

Pearson is bent on controlling every element of the education process, from teacher qualifications to curriculum to the tests used to evaluate students to the grading of the tests to, increasingly, owning and operating its own learning institutions – like Connections Academies (the online virtual charter school company), which it acquired about five years ago.

Consider a recent example of Pearson’s reinventing public education. Pearson landed the contract to redo the GED exam. They decided that the exam needed more thought questions. They rewrote the exam and suddenly the failure rate jumped from about 15% to over 60%. Pearson is a major pusher of the Common Core, for which it stands ready with the textbooks and the tests to sell to states as soon as they adopt the new standards.

According to some stock analysts, Pearson PLC supplies 60% of the U.S. testing market. Expenditures for assessment by U.S. public schools have grown by 60% (from $1.6 billion to $2.4 billion) from 2010-11 to 2012-13. And the rate of growth may well be accelerating.

The behavior of Pearson PLC in landing big contracts with states has raised some eyebrows. Texas signed a half-billion dollar five-year contract with Pearson for assessment services. The corporation played a very common game of creating a non-profit foundation, completely controlled by the parent corporation, to do its business of enticing potential customers. Education officials in states that were considering how to implement the Common Core State Standards would be treated to luxurious trips to far away places like Brazil and Singapore. New York State AG Schneiderman investigated these junkets and Pearson paid a nearly $8 million fine in two years ago. The corporation folded the non-profit foundation when its mode of operation became widely known.

Corporations and ALEC

Pearson was embarrassed when its non-profit foundation was exposed as a briber of education officials. There are dozens of other corporations who do not wish to have their financial contributions to politicians and public officials out in the open. Thanks to Citizens United – perhaps the most incredible ruling of the US Supreme Court of the last 50 years – corporations now hide their bribes through a non-profit organization known as the American Legislative Exchange Council, ALEC.

ALEC is a non-profit organization funded by large corporations and member dues. Its members are almost exclusively Republican state legislators who pay a nominal fee to join and corporations who pay very large fees. At ALEC’s regular meetings in attractive places like Florida and southern California, legislators are “educated” on some of the issues currently on the minds of the corporations. These issues deal with things like the sentencing of law breakers, sale of tobacco, energy policy, and even education. Drafts of bills are passed out with the expectation that they will be introduced in the legislators’ home states. In 2012, 1,000 bills dealing with education written by ALEC were introduced in state legislatures, and 100 of them were adopted as law. Most dealt with charter schools, tuition tax credits and testing.

ALEC is a major force behind privatizing state prisons and supplying them with prisoners. It causes bills to be introduced providing for mandatory minimum sentences and “three-strikes” sentencing. About 40 states passed versions of ALEC's Truth in Sentencing model bill, which denies parole to prisoners convicted of violent crimes. The U.S. incarcerates a higher percentage of its population than any country in the world.

In 2010, ALEC arranged meetings between the Corrections Corporation of America and Arizona legislators such as Russell Pearce to write Arizona SB 1070, which would ensure CCA's immigrant detention centers a steady supply of “clients.” (Source: Wikipedia, “Private Prisons,” accessed January 28, 2015.) The newly elected Governor of Arizona, Doug Ducey, filed his coming year budget two weeks ago. K-12 public education, which has suffered the deepest cuts percentagewise of any state in the nation since 2008, was cut another $17 million. Private prisons are slated for an increase of $52 million, and a bill to create a new 3,000 inmate prison operated by Corrections Corporation of America has been introduced into the Arizona legislature. A couple years ago, the Arizona legislature repeal a law that required the state to perform a cost comparison between state-owned and private prisons before making any purchases. It is as though the very letter of the laws are being written by corporation lobbyists.

The Declining Influence of Unions

Concurrent with the changing demographics of the U.S. over the past 60 years has been the declining role of unions in American politics.

There was a time when labor unions were very powerful. My father joined the International Typographical Union as a 18 year-old in 1927 after completing his apprenticeship as a type-setter at the Lincoln Journal. The ITU wielded enormous power until about mid-century because it could shut down newspapers, the primary means of communication at the time. But the revolutionary changes in business of the second half of the 20th Century have decimated traditional labor unions. Outsourcing to labor in the global economy and the role of computers have reduced the price and the power of American labor. My father struck against computers entering the composing room at the newspaper in 1963 and never worked another day at his trade.

The only labor unions left with any clout are those that deal with human services that can neither be outsourced nor computerized. (And the Teamsters shudder at the prospect of driverless trucks.) Foremost among the few remaining labor unions are those that enroll public employees. Consequently, public employee unions have become the target of tax slashing groups. Teachers associations are being recast as collections of incompetent slackers.

The Rise of the Elected State Superintendent of Schools

Another manifestation of how control of public education has slipped away from the hands of educators into the hands of politicians is the rise of the phenomenon of elected State superintendents. In 13 states, the state superintendent of education is an elected position. The increasing politicization of this office has had unfortunate consequences for professional education that will be obvious to anyone who thinks about it even momentarily.

Of the 13 elected State Superintendents, 3 are Democrat (I’m guessing CA, OR and WA), 6 are Republican and 4 are “nonpartisan” (presumably Republicans when you look at the remaining states). This politicizing of a role that historically either did not exist or was occupied by educators represents just one more stage in the removal of control of education by professionals. That politicians are so susceptible to the influence of moneyed interests further raises suspicions that private corporate power in strengthening its grip on public schools.

The 21st Century: The Rise of the Super Rich Education Reformers

This brings us up to the 21st Century and the entrance of the Superrich Ed Reformers. I shall focus on just two: Bill Gates (actually the Bill & Melinda Gates Foundation) and the Koch Brothers. A by-product of these persons’ ascent to among the richest humans on earth is an overweening attitude that they know more than anyone else about most things. This over-confidence in their own abilities has produced in both of them the Shoe Button Complex.

Warren Buffett (the Sage of Omaha and another of the world’s richest persons) had a partner named Charlie Munger as a long-time partner at Berkshire-Hathaway. Munger’s grandfather had managed to corner the market on shoe buttons back around 1900. The grandfather exercised a virtual monopoly over their production and sale. Emboldened by his business acumen, the old man grew to believe that he not only knew more than anyone else about shoe buttons but that he knew more than anyone else about anything—and he preached and proclaimed at length on such. Munger and Buffett named the syndrome the Shoe Button Complex, and they encountered it frequently in their dealings with successful business practitioners.

Buffett had another insight that he embodied in the principle of the Circle of Competence. He knew what he knew, and he didn’t stray far from what he knew. Step outside your circle of competence, and you are on dangerous ground. Back in the great internet boom of the 1990s, people repeatedly asked Buffett which internet stock he was investing in. Buffett reminded them that in the history of the 20th Century the US has known more than 1,000 automobile manufacturers: Essex, Lafayette, Marquette, Ajax, Mercer, etc. How, he asked, am I supposed to pick out which one of these dozens of internet companies is going to be successful? Instead, reasoning that in the future people are highly likely to wear underwear, he bought stock in Fruit of the Loom.

But Buffett was not foolproof. Twenty-five years ago, he phoned Bill Gates and invited him to Omaha, and he sat him down and told him it was time to start giving away his money. And to underline his advice, he told Bill and Melinda that he was leaving his $50 billion estate to their foundation because they were better able to make good philanthropic decisions than he. It may have been one of the few times Buffett ventured outside his Circle of Competence. So what has Gates done with his and Buffett’s money? He has spent billions in Africa attempting to lift backward nations and he has spent hundreds of millions attempting to reform K-12 education in America. How has he done? Not particularly well. He recently admitted that his Africa policy to push genetically modified crops on farmers has been a failure.

Bill Gates’s insight into what public education in America needs is obviously not insignificant, given his influence over education policy at all levels today. Gates’s worldview is all about global economic competition. When he was asked to testify to the House Education Committee back in the 1990s, he spent the bulk of his time talking about visas for engineers he wanted to import from India and China to work at Microsoft. It is safe to say that Gates’s conception of young people is that of future employees of transnational corporations, and the purpose of education is to prepare them to make money for the firm.

Gates speaks absolute simplistic nonsense when he says that we can’t improve education if everybody is aiming at different goals and if we don’t have standards to hold educators responsible for attaining and if we don’t have tests to tell us if we have reached those standards. Excuse me, but this is pure poppycock, and if it were not coming out of the mouth of one of the world’s richest men, it would be laughed out of the room. To Gates, America needs the Common Core and “data,” by which he means tests, tests, and more tests. Perhaps he should take a course in qualitative research methods and expand his concept of “data” just a bit.

The Bill and Melinda Gates Foundation has sunk hundreds of millions of dollars into creating and pushing the Common Core State Standards and Value Added Measurement (VAM) of teachers, principals, schools, states and colleges of education, surely.

All that I have to say about the Common Core State Standards is that:

1. They should not be common to all states;
2. They are not a plot by the federal government to take over 15,000 school districts – contrary to the opinion of our the new Arizona State Superintendent of Public Instruction and many TEA partiers;
3. They would if implemented further deskill teachers and contribute to the de-professionalization of teaching.

VAM (evaluating teachers and others by the gains in the students’ test scores) is an invalid monstrosity that has failed in history going back to Ireland in the late 1800s. I can’t imagine that reformers like Bill Gates intend to turn teachers into minimum-age clerks, but their conception of an efficient and effective education system threatens to do just that. “Teachers” in name only who download their curriculum from the state education agency or Pearson PLC, monitor students’ time online, proctor tests, and enter data into data bases so that administrators can make data-driven decisions.

Will Gates soon declare his U.S. education reform policy a failure? He has backed off on testing the Common Core any time soon, and states are backing out of adopting the CCSS under threat of US Department of Education punishment. A nationwide opt-out of standardized testing movement is gaining speed.

The Koch Brothers

Charles and David control the second largest privately owned company in the United States. They are a couple of good old boys from Wichita, Kansas, who, according to Mitch McConnell, essentially own the Republican party. (http://www.politicususa.com/2014/08/27/leaked-audio-tape-mitch-mcconnell-admits-koch-brothers-running-republican-party.html ) They are major bakers of the TEA Party and their money makes possible the existnc of such right-wing think tanks as The Heritage Foundation and the Cato Institute. The Kochs have announced that they plan to spend $900 million on the 2016 election.

The Kochs spent hundreds of millions of dollars in the 2014 mid-term elections, and almost certainly turned one U.S. senatorial race outcome from Democrat (Mark Udall) to Republican (Cory Gardner). The fact that Gardner has repeatedly introduced bills in the Colorado legislature to declare a fertilized egg a person and to outlaw the sale of contraceptives helps to describe the politics of the Koch brothers.

Many have wondered why an issue like the Keystone pipeline is such a hot political topic when it will create an insignificant number of permanent jobs and when its sole function is to move Canadian raw petroleum from the northern oil sands to Texas refineries for eventual sale on the world market. Approving the pipeline has been a do-or-die issue with the Republican Party for a few years now; and quickly after assuming majority power in the Senate in 2015, they pushed through a bill that authorizes the construction and operation of the pipeline. The Kochs are the largest owners of land in the Canada oil sands.

These two fellows are exercising an enormous influence over U.S. politics and they have public education in their sights. For some years now, the Koch brothers have been contributing to the campaigns of school board candidates. As a matter of fact, it has been a strategy of conservative political movements to start promising politicians in school board elections as a try-out for higher office.

In a recent school board election for the Jefferson County (Colorado) schools, two seats were won by very conservative candidates. Their campaigns were financed primarily with Koch brothers’ money. The votes had scarcely been counted when the business of the board erupted into a series of political battles with parents, teachers and community members. Conservatives now held a majority on the board. Proposals were discussed for implementing value-added measurement (VAM) of teachers for salary and tenure purposes, and a superintendent search was favoring an administrator from a neighboring district who had rammed VAM done his teachers association’s throat. But the issue that generated the strongest public reaction was the board’s foray into curriculum. Led by the newly elected board members, an investigation concluded that the Advanced Placement History course textbook did an inadequate job of instilling loyalty and a sense of pride in America’s accomplishments. Changes were ordered. The board passed a resolution that ordered the classes to promote "patriotism and ... the benefits of the free-enterprise system" and that they should not "encourage or condone civil disorder."

For some teachers, students and parents, an attack on the AP History curriculum was a bridge too far. Students reacted first. They walked out of school, or more accurately, they refused to attend. Then they took to the streets with placards, and demonstrated against the intrusion of the board into the AP curriculum. They were eventually joined by parents and teachers. After the walk-out appeared on the local news, the board backed down.

Other Super Rich Ed Reformers

• The Walton Foundation (big backers of charter schools)
• The Broad Foundation (Training administrators to love the market)
• John Fisher (son of the GAP creators investing in charters)
• David Welch (Silicon Valley billionaire who funded the Vergara case)

What Can An Educator Do to Take Back Control of Public Education?

Of course, it is an exaggeration to say that public education today is being run by corporations, politicians, and the super-rich who own them. But it is no exaggeration to point out the growing influence of the forces that seek to reap private gain from this half-trillion dollar sector of the economy. Is it too late for professional educators to take back control of the schools? Not much. Face it. Politicians are for sale in the Age of Citizens United, and educators are not the ones with the deep pockets. Diane Ravitch has opined that students – walk outs – and retired teachers are in the best position to resist these forces.

Among the Many Things Wrong With International Achievement Comparisons

2012

Among the Many Things Wrong With
International Achievement Comparisons

Gene V Glass

The Brown Center for Education Policy of the Brookings Institution released a report just a couple days ago with the jaw-breaking title “HOW WELL ARE AMERICAN STUDENTS LEARNING? With sections on predicting the effect of the Common Core State Standards, achievement gaps on the two NAEP tests, and misinterpreting international test scores.” The report is penned by Tom Loveless, a researcher from whom one might have expected nothing but analysis aimed at proving the abject failure of American public education. In this instance, he has disappointed any followers seeking such a message, and has in fact produced a fairly balanced analysis of the many pitfalls in basing policy on numbers.

My attention was drawn to the section on “misinterpreting international test scores,” since I have long felt that these international assessments are a mess of uninterpretable numbers providing a full-employment program for psychometricians, statisticians, and journalists. Loveless took a close look at PISA (Program for International Assessment). He concluded that policy makers, educators, journalists, and the public in general often arrive at “dubious conclusions of causality” based on the results of such assessments. Of course. Just recall the grand exodus to Japan in the 1980s when our nation was discovered to be “at risk” and the Japanese economy was booming, just before the Japanese economy tanked. And what did emissaries discover as the secret to education excellence? Jukus (privately operated “cram” businesses), high suicide rates among young people who were subjected to immense high stakes pressure, and an economy about ready to go into the dumpster. Now all eyes are on Finland. The whole scene is reminiscent of the IRA (International Reading Assessment on the 1970s) that showed that the top nation in the world on reading was …are you ready?...Italy. Italy?? That one sent people to Rome for a few months until it was discovered that the attempt at random sampling in the assessment was never so badly compromised as it was in Italy.

Loveless also pointed out that the numbers (averages of a nationwide sample of students) on which rankings of nations are based are frequently so close that there is no “statistical significance” to the differences. True, but let’s ignore this problem so as not to be diverted into an alley of dry mathematical mumbo jumbo.

But wait a minute. There is something far more wrong with these international assessments and comparisons than anyone seems interested in talking about. Think! A reading test that compares students in dozens of countries. The obvious question is “In what language is the test written?” And the obvious answer is “In the language of that nation.” But who is drawing the obvious conclusion? How in heaven’s name can you construct a reading test in dozens of different languages (English, Hungarian, Norwegian, and yes, Finnish) and be confident that the test is equally difficult in all of these languages? Well, the answer is that you can’t. It should be perfectly obvious to anyone who thinks about it for more than five minutes that it is impossible. And all the ministrations and obfuscations of the companies and consultants who make or supplement their living off of such stuff do not change that fact.

Let’s take a look at some of these results. I have excerpted some data from the 2003 PISA Reading test for 15 year-olds. They are merely illustrative and it’s of no consequence that it is a small subset of the complete results.

2003 PISA Reading 15 year-olds
Finland	543
Canada	528
Liechtenstein	525
Sweden	514
Hong Kong	510
Norway	500
Japan	498
Poland	497
France	496
USA	495
Germany	491
Austria	491
Hungary	482
Spain	481
Italy	476

So there is the USA a point or two or three below France and Poland and Japan and whoever, and a point or two above Germany and Austria. This is the kind of statistical insignificance that Loveless was talking about. However, even to take seriously the kinds of differences like 19 points between the US and Sweden ignores the question before us: How do you write a reading test in English and then translate it into Swedish (or vice versa) and end up confident that one is not intrinsically more difficult than the other? I insist that the answer to that question is that you can’t. And to claim that one has done so merely sweeps under the rug a host of concerns that include grammatical structure, syntax, familiarity of vocabulary, not to mention culture of the students taking the test.

Now the keepers of the PISA tests have produce a lengthy—almost 40-page–Appendix to a report that in which they claim to have solved the problem of producing equivalent translations by the assiduous application of the finest psychometric theories. ( I don’t believe it. Forget about DIF analysis, i.e., Differential Item Functioning which only tosses out a few items that show really large differences in difficulty between two forms and ignores consistent though small differences.) What the PISA technical manual omits are any examples of reading test items in two or three different languages so that we might scrutinize the results of all this fine theory. (Ironically, one keeper of the items declined to release a few examples to me even though that person was my own doctoral student some 30 years ago.)

So let’s look at an example of our own. Tom Sawyer setting up Huck Finn to whitewash the fence.

Tom appeared on the sidewalk with a bucket of whitewash and a long-handled brush. He surveyed the fence, and all gladness left him and a deep melancholy settled down upon his spirit. Thirty yards of board fence nine feet high. Life to him seemed hollow, and existence but a burden.

And now, a translation into German (since I majored in German as an undergrad some 50+ years ago):

Tom ist auf dem Bürgersteig mit einem Eimer von Tünche und einer langbehandelten Bürste erscheinen. Er hat den Zaun vermessen, und alle Freude hat ihn verlassen und eine tiefe Melancholie aud sein Geist geberuhigt wird. Dreißig Höfe von Ausschusszaun neun Füße hoch. Leben, zu dem ihn Hohlraum gescheinen hat, und Existenz aber eine Last.

Now we could observe multiple difficult choices that would have to be made in translating the English to the German that would surely affect the ability of a student to comprehend a sentence, phrase or the entire passage. Just a few: “whitewash” being Tünche in German is a relative obscure word? Is it equally obscure in American English or Canadian English (who are prone to speak of Scotch tape as “cello”); Melancholie might just as well be translated as Traurigkeit, depending on the local preferences for Latinate vs Germanic roots, such preferences still being strong in certain locales; etc. And what should we do with the 30-yard long fence? Translate it as a fence that is 27.432 meters long?

Bottom line: I believe that the differences in difficulty produced by the vagaries of translating a reading test across several languages are at least as large as many of the differences among average PISA Reading test scores, the latter differences being the stuff of media accounts as well as learned papers on school reform.

To bolster my belief, along comes an actual piece of education research addressed to precisely the translation question in international reading comparisons. A recent article in the Scandanavian Journal of Educational Research by Inga Arffman carries the title “Equivalence of translations in international reading literacy studies,” (Vol. 54, No. 1, 37-59). The paper summarizes a study that examined the problems encountered in translating texts in international reading assessments. And in spite of the fact that Arffman is a faculty member of the University of Jyväskylä in Finland—which has every motive possible to believe that PISA Reading assessments are the most valid tests in the history of psychometrics—the conclusion of the research is that “..it will probably never be possible to attain full equivalence of difficulty in international reading literacy studies….” Amen.

2000

Are Data Enough?
Review of Chubb and Moe's Politics, Markets and America's Schools

Gene V Glass
Dewayne A. Matthews
Arizona State University

Chubb, John E. and Moe, Terry M. (1990). Politics, Markets and America's Schools. Washington, D.C.: The Brookings Institution. Pp. 318 + xvii.

Politics, Markets, and America's Schools is one of those rare books of educational research that breaks through into the press and public debate of the day. In this case, the subject is the timely one of the effects of school organization on student learning. The book reports on a research project involving an analysis of several national data bases, particularly High School and Beyond (HSB), to attempt to determine what factors lead to high levels of academic performance in schools. Chubb and Moe conclude that problems of academic performance in the schools will not be solved by any of the changes brought on by the school reform movement. The problems are a direct and inevitable result of the structure of American public schools, specifically their control through democratic processes. The solution is autonomy—building-level autonomy of principals and teachers freed of the dead hand of bureaucratic regulation from government and from school boards.

Chubb and Moe's Argument

The authors have reached this conclusion through a comparison of the academic performance of students from public and private schools, and have attributed the better performance of private school students to structural differences between the two types of school. (The authors speak favorably of the Coleman report of 1981 on public and private schools). However, the book attempts to go beyond the simple conclusion that students seem to learn better in private schools to examine the effects of structural differences among schools and their effects on student learning. Their basic premise is far from revolutionary, namely that public schools suffer from excessive levels of bureaucratization and politics. More important, they suggest that excessive bureaucracy is the proximate cause of problems in the schools, and that politics are the ultimate cause of the over-reliance on bureaucracy. The bureaucracy cannot be changed unless the underlying political structure is changed.

Chubb and Moe contend that the results of education are inherently difficult to measure. Hence, only those who are in direct contact with the learners can know what is happening in the schools and judge the effectiveness of teaching. Reliance by bureaucracies on top-down hierarchical management results inevitably in conflict: Effective bureaucracy is commonly built around rules that specify appropriate behavior, rewards, and sanctions that encourage such behavior, and monitoring to ascertain whether goals are being met, whether rules are being followed, and whether the rules and incentive system need to be adjusted. All are rendered highly problematic in education, because good education and the behaviors conducive to it are inherently difficult to measure in an objective, quantifiable, formal manner. The measurement problem makes it difficult or impossible for education administrators to know what they are doing—and their controls, as a result, threaten to be ill suited to the ends they want to achieve. (p. 36)

Because the public schools are governed, funded, and directed through the political process, the interests of parents and students receive no more weight than the interests of any other group. Indeed, Chubb and Moe argue that parents and students will have less influence than others because of certain structural features of the political process, namely the inherent power of organized over disorganized groups. The many demands on schools that result from this political process will be accommodated by the educational system through the establishment of regulations, guidelines and monitoring procedures. This political process leads inevitably to highly bureaucratic modes of organization and management. Rigidly hierarchical bureaucracies are not conducive to effective learning because they do not promote or allow the effective use of professional personnel, particularly those who are in direct contact with students, namely teachers.

Private schools are different, allegedly, because they are insulated from the political process. The crux of Chubb and Moe's argument is the basic distinction between democratic and market control. Public schools are democratically governed through the political process. Private schools are not; the owners are free to run the school as they wish. (Chubb and Moe may fail to take into account the considerable public influence over private schools through state regulations, certification requirements, and the like; though comparatively, the distinction is probably still valid.) However, to attract students the owners of private schools must be responsive to the needs and desires of parents and students. Private schools are, therefore, relatively free to concentrate on that with which parents and students are presumably most interested and concerned, student learning.

Their analysis of the differences in student achievement between independent and public schools suggests to Chubb and Moe that the root cause of poor performance in schools is found in their governance. Chubb and Moe therefore recommend that the present system of public school governance be scrapped in favor of a market-driven one in which parents have primary control over the schools. This recommendation is, however, only academic; Chubb and Moe acknowledge the impossibility of its adoption. Although their recommendation for reform draws more heavily than most on academic theory (of organizations and political bodies, in this case), it is not an unfamiliar proposal; it represents neither discovery nor invention nor new ideas. Why, then, does this book appear now?

The Statistical Study

Chubb and Moe claim uniqueness for their arguments about reform of the organization of schools, and they may be justified. More than most recommendations for school reform (perhaps Coleman or Goodlad are the visible exceptions), their argument grows out of the quantitative analysis of empirical data, specifically data on student achievement, students' families and school organization. Whether the data compel the argument, or even whether the data are up to the task of suggesting policy is a question we will address here.

Chubb and Moe's claim to empirical backing for their policy recommendations rests on a causal argument, namely, that certain aspects of school organization cause student achievement. The causal argument is pursued via the High School and Beyond (HSB) data set and an analysis plan that fits systems of linear equations to the data under specified constraints. The familiar data base comprises 20,000 cases and hundreds of items from questionnaires administered to students, teachers and school principals. Achievement tests were administered twice, first in 1980 and again in 1983 when the student cohort had reached the Senior year. About 100 items tested performance in reading, vocabulary, writing, math and science. Questionnaires probing classroom and school organization, personnel policies and the like were administered to teachers and administrators in 1983-84. In justifying their causal claims, Chubb and Moe recap the standard criticisms of structural equation modelling; it works (i.e., determines causes) if two conditions are satisfied: 1) all third variables are present and accounted for, 2) the direction of causal influence (from putative cause to putative effect) is known a priori or controlled by fixing temporal priority. Chubb and Moe give their work good marks on both counts.

They attempt to cope with the third variable problem "...by allowing student achievement to be influenced by many of the kinds of variables—for example, family SES and student ability—that also ought to predict whether students select their schools" and by using gain in achievement from grade 10 to 12 as the dependent variable. (p. 112) The third variable problem doesn't yield to such modest exertions as scoring a handful of questionnaire items dealing with students' families and their social-economic background. If it did, we in educational research would be adrift in reliable, well-established causal relationships, and James Coleman and many others would live much less controversial careers. Cronbach has elaborated the conditions under which non-experimental data approach the validity of randomized experiments in establishing causal claims: perfectly reliable measurement of exhaustive measures of differences among the different levels of the putative independent variable (Cronbach, 1979). Chubb and Moe's data set comes nowhere near solving the "third variable problem." The twelfth grade achievement variable surely is contaminated with a goodly amount of influence from unmeasured and unreliably measured differences between students and schools. Chubb and Moe have surely attributed some of this influence—perhaps a great deal, no one can know—to their favorite independent variables. We certainly do not fault them for failing to turn surveys into experiments; we only wish that they were quicker to acknowledge that they can not do so.

Even assuming that the remaining difference in student achievement is associated only with organization autonomy differences, how can one be confident that autonomy leads to improved achievement? Can it not be that schools with students who learn at a faster pace are granted greater freedom, either by design or as a result of greater constituent satisfaction? Chubb and Moe acknowledge that in fact student achievement and school organization may bear reciprocal causal relationships to each other—sometimes one causes the other, other times vice-versa, or the causal influence runs in one direction in circumstances A and in the opposite direction in circumstances B. "Organization may be both cause and effect." (p.113) How do they unravel this mystery, a mystery that plagues most attempts at causal modelling that lack longitudinal observations? "We do not wish to pretend that we have a solution to this [ambiguous direction of causality] problem—for we do not—but we do believe we have a workable method of analysis that keeps the ... problem in clear view. Despite all we have said about the problem of reciprocal causality, we believe that the key influences on student achievement tend to run in one direction. We believe that school control affects school organization more than the other way around, and that school organization is primarily a cause of student achievement and not a result of it." (p. 114)

Chubb and Moe state this article of faith, that the problem of ambiguous direction of causality can be solved by willing it away, in a disarmingly direct and simple way—as if one were asserting that the chances of radio wave disruption causing sunspots were too small to be taken seriously. But the ambiguity in the HSB database, for this particular assertion, will not be dispelled so simply. It is equally obvious to some observers closer to American education than Chubb and Moe that high and low student achievement (even that amount left over after imperfect partialing out of pre-achievement scores and a few questionnaire items about family) prompt organizational response. Indeed, precisely the finding on which Chubb and Moe hang their entire proposal for school reform—that organizational autonomy is related to high achievement—is likely to arise from a causal influence of achievement on organization: low achieving schools prompt managers at all levels to intervene to solve the problem of poor performance; high achieving schools are spared the kind of meddling that well-intentioned persons from the state agency to the school building are prone to offer. Is there anything in the data set that lends credibility to one direction of causal influence over the other? Indeed there is. Causes precede effects in time, at least for the notion of causality still used in accounting for human behavior. The HSB surveys measured student achievement in 1980 and 1983; school organization was measured in 1983-84. In view of this sequence, a bit more modesty in making uni-directional causal claims seems called for.

Nor does Chubb & Moe's analytic attempt to unravel the ambiguous causal direction problem engender confidence. They attempt to study the influence of student achievement on school organization by reversing the regression and entering the former as an independent variable and the latter as outcome. This analysis dissolves in a confusing inconsistency, of which far too little is made. Essentially, the continuously measured achievement variable is reported to be not significantly related to school organization while a dichotomized measure of degree of student achievement gain (below vs above average) is reported to be related to school organization (p. 175). A secondary data analysis may be required to straighten out this anomaly.

In the face of the authors' enthusiasm for their findings, even the careful reader—and surely the media and other second-hand consumers of this research—quickly loses sight of the fact that these sweeping recommendations are based on statistical results where the model accounts for only 5% of the variance in the dependent variable of student achievement; we repeat: the multiple R in these analyses is less than .25. Of course, any regression coefficient must be quite small in these circumstances, and that coefficient for the School Organization variable, while significantly non-zero on 20,000 cases, is tiny. One implication of this result is that enormous changes in a school's position on the organization variable will be predicted to yield very small changes on the achievement variable. A school that moves from the 5th percentile to the 95th percentile on autonomous organization would be expected (assuming all problems in causal inference are resolved in Chubb and Moe's favor) to climb a month or so in grade equivalent units on a standardized achievement test. Research and Policy

Chubb and Moe have such confidence in the results of their statistical analyses that they recommend the creation of an entirely new system of public education for the nation. This approach of "policy by regression coefficient" raises some serious questions. Does the largest beta prevail in determining policy? By acting as if policy flowed from statistical analysis of achievement scores, they practice a brand of social science that, while not value-free or value-neutral, is at least value-insensitive. Chubb and Moe's recommendation to remove the bureaucracy from the schools raises value questions that they don't address. Some, but hardly all, of the democratically generated bureaucracy that they wish to strip out of the public schools was created to protect the rights of students. Clumsy though they may be, the rules and regulations often stand as a safeguard against callous and unfair treatment of children, particularly those who suffer handicaps or are ethnic minorities. Have we reached an enlightened state in this country where those safeguards can be dispensed with for the sake of teachers' and administrators' autonomy? Some will doubt it.

Chubb and Moe do a service by raising the issue of governance as it relates to efforts to make schools more effective, but they give scant serious attention to the broad context of American education that they imagine their study reforming. Any effort toward greater autonomy for teachers or toward school-based management will have to recognize the reality of educational governance today, and be responsive to its demands. Developing an understanding of the policy environment in which schools operate is difficult for most educators, as reflected in Chubb and moe's decision to dismiss it wholesale. However, local school boards, state boards of education, and legislatures are the very messy environments in which educational policy will continue to be made. Any benefits that could accrue from more autonomy for educators will only be gained when policy-makers are convinced that it is in the best interest of the public, as defined through their constituencies, to let go of central control. Chubb and Moe, unfortunately, do not make that case.

Policy-makers are, however, greatly preoccupied with issues of autonomy. In the aftermath of the school reform movements of the early 1980s, policy-makers are experimenting with various approaches to enhance educator autonomy. Whether called teacher empowerment, site-based management, or school restructuring, such approaches are the new darlings of both educators and policy-makers interested in educational improvement. Chubb and Moe establish an apparent link between autonomy and school performance, but the direction of influence is ambiguous at best. What is needed is a greater understanding of the ways through which autonomy affects school performance. Case studies and other more narrowly focused research into schools could help develop an understanding of these relationships that could guide both educators and policy-makers in determining the appropriate role of autonomy in school improvement. Some contributions in this regard have been made by S.R. Glass (1997) in her qualitative study of autonomy in public and private schools. Chubb and Moe opt out of this effort by their assertion that school effectiveness is pre-ordained by governance structures which cannot be changed. The widespread experimentation in school restructuring would suggest that most educators and policy-makers do not agree.

Politics, Markets and America's Schools is the research legitimation that the school choice movement has been waiting for. The book has been heralded as an important contribution to knowledge. The authors expected to be attacked by researchers, and they have been. John Witte, a political scientist at the University of Wisconsin at Madison, has attacked Chubb and Moe's data, their analytic techniques and their conclusions. "To suggest that we know enough from High School and Beyond to overthrow the public school system in the United States and replace it with a choice system is sheer madness," Professor Witte charged hyperbolically in an interview published in Education Week (November 14, 1990, Vol. 10, Number 11, p.20). Witte claimed that the data in their original form were flawed beyond repair and that certain transformations performed by Chubb & Moe exaggerated the achievement gains to be expected from school reorganization. But that will hardly matter; in the political battles over school choice, technicalities about empirical research will be brushed aside and Chubb & Moe will be cited as authority by one side and decried as bogus on the other. Will the book raise the level of the debate, introduce new perspectives, lead to better thinking? No, not by itself; it is not that kind of book. It is rather a polemic wrapped in numbers.

Cronbach writes in Designing Evaluations of Educational and Social Programs about two contexts in which social researchers imagine themselves being when they present their work: the context of command and the context of accommodation. The former is a dream of omnipotence in which supremely powerful decision-makers issue directives that others follow. The context of accommodation is the reality of policy-making in American society; compromises are struck between competing interests, sometimes, it is to be hoped, in light of the social researchers' models, findings and ways of thinking. Social research benefits and grows more useful, we believe, when its creators recognize the reality of the context of accommodation. Chubb and Moe, ironically since they are political scientists, act as if they were addressing the non-existent commanders of the American educational system.

This book has received an uncommon amount of attention in the popular press. Its authors have appeared on the Op/Ed page of prominent newspapers to give capsule versions of their position. Famous persons praise the work on the dust jacket; Chester Finn calls the book "...the most eagerly awaited education book of the year, and very likely destined to become the most influential." Why is such attention showered on a rather ordinary regression analysis of a data base that was constructed by the government nearly a decade ago? Chubb and Moe offer the school choice movement the legitimacy that empirical research can confer. Research is today a language of legitimate authority, and political positions are strengthened when it can be cited. The voucher and school choice interest groups have cited Chubb and Moe with glee, as if the long awaited experiment had suddenly proved the rightness of their cause. It is a mark of the maturity of educational research that its findings are so eagerly sought. It is a mark of its undeniable limitations that the findings of educational research still have about them as much of the character of political rhetoric as they have the character of scientific discovery.

References

Cronbach, L.J. (1979). Designing Evaluations of Educational and Social Programs. San Francisco: Jossey-Bass.

Glass, S.R. (1997). Markets and myths: Autonomy in public and private schools. Education Policy Analysis Archives. 5, 1. [Entire issue. Available online at http://epaa.asu.edu/epaa/v5n1.html ]

This review was originally published in 1990 in the Educational Researcher. It was slightly revised as published here. Dr. Matthews is currently at the Western Interstate Commission on Higher Education; his email address is dm@wiche.edu. Glass is still at Arizona State University where his email address is glass@asu.edu.
(5/26/2000)

Definition of Meta-Analysis, from Oxford English Dictionary

Oxford English Dictionary

meta-analysis,             n.

1. Philos. Analysis of the grounds and assumptions on which a theory, explanation, or account is based; an instance of this.

   1953 Philos. Rev. 62 462 We are told that the definition of cardinal number is ëone of Russell's best known accomplishmentsí.., yet there is nothing in the meta-analysis to indicate why it should be.

   1953 Jrnl. Philos. 14 114 Philosophically speaking, Mr. Levy's work is one of the clearest and most extended meta-analyses of the functional approach.

   1965 Philos. Q. 15 369 While we cannot analyse further what a philosophical explanation or account consists in, for the acceptability of such an analysis will itself rest on some undefined notion of what is reasonable or makes sense, it does not follow that we must suspend our philosophical activities until such a meta-analysis is produced.

   1991 Philos. & Phenomenol. Res. 51 715 My principal reservations..concern not the basic theory itself.. but..his meta-analysis, i.e., his claims about the presuppositions, implications, and significance of that account.

2. Statistics. Analysis of data from a number of independent studies of the same subject (published or unpublished), esp. in order to determine overall trends and significance; an instance of this.

   1976 G. V. GLASS in Educ. Res. Nov. 3/2 My major interest currently is in what we have come to call..the meta-analysis of research. The term is a bit grand, but it is precise and apt... Meta-analysis refers to the analysis of analyses.

   1987 Brit. Med. Jrnl. 5 Dec. 1434/1 Wald et al have concluded from a recent meta-analysis that people exposed to passive smoking have a small but definite increased risk of developing lung cancer. 1993 Guardian 30 July 18/8 We now have a group of scientists who specialize in undertaking meta-analysis, the analysis of all the trials on a particular subject looking at the results as though they were of one.

Are Demographics the Nation's Destiny? School Administrator, June 2008. Pp. 38-39.

Policy for the Unpredictable. (1979). Educational Researcher, Vol. 8, No. 9, Pp. 12-14.

Primary, Secondary, and Meta-Analysis of Research

1976

Gene V Glass. (1976). Primary, Secondary, and Meta-Analysis of Research. Educational Researcher, 5(10), 3-8.

Evaluating Effects in a Quasi-Experimental Design

Politics of Teacher Evaluation

1993

Politics of Teacher Evaluation

Gene V Glass
Arizona State University

Barbara A. Martinez
California State University, Los Angeles

Abstract. We sketch the current practice of teacher evaluation in the U.S. from our ordinary experience. We present a fragment of a case study that reveals how teachers—elementary school (K-8) teachers in this instance— view the experience of being evaluated, and how feelings of illegitimacy flow from the experience. Finally, we indicate certain considerations from political theory that bear on the problems of improving teacher evaluation. The Current State of Things

Elementary school teachers are evaluated differently from how secondary school teachers are evaluated; and secondary school teachers are evaluated differently from college professors, who further underline the differences between themselves and their public school colleagues by not even wishing to be called teachers. Nothing seems to account for these differences so clearly as does what we might loosely refer to as the politics of evaluation. We often learn something interesting about the organization and politics of education when we contrast how it is pursued at its different levels.

College professors are usually evaluated by their peers and superiors yearly for raises and less often for promotion; but in spite of what might be claimed (by the president of the college when addressing parents, for an instance), they are seldom evaluated qua teachers. It is common today for students to fill out simple forms, rating scales, at the end of a semester: "Instructor was organized," "Instructor knew the subject," "Instructor graded fairly." Typically, these student ratings count for little. The better the university, the less teaching is weighed in the balance that sways toward research and publication; and most colleges aspire to be like the better universities. Extraordinarily bad student ratings will be used to terminate an untenured faculty member if that person's research is poor; but the administration will swallow bad ratings when a strong researcher receives them.

Secondary school teachers are evaluated sporadically. Peer evaluation is non-existent in America's schools, and administrators seldom venture into a high- school teacher's classroom. Their presence would be viewed with suspicion by the teacher; the legitimacy of their place there would be questioned (silently or behind closed doors if not publicly). Administrators appear to concede that secondary school teaching involves specialized knowledge (of chemistry or mathematics), and that a specialist may be needed to recognize good teaching.

Elementary school teachers are treated substantially differently. Principals visit their class once or more each year. Indeed, principals regard these visits as a responsibility of their position. Teaching is observed, occasionally a check-list is filled in; lesson plans may be inspected. Much is written in school personnel manuals about evaluating teachers on the basis of their stude nts' achievement test scores; but the threat is an empty one that nonetheless has the power to shape instructional styles and choice of content (Glass, 1990).

The legitimacy of the elementary school principal's presence in the classroom for the purpose of evaluating the teacher is less likely to be questioned openly. Everyone knows elementary school subject matter after all, and an instructional leader is an expert in the general techniques of effective instruction, or so it is widely believed.

Each level is distinguished by a different balance of teachers' professional autonomy on the one hand, and the exercise of administrative authority in a democratic bureaucracy on the other. At the elementary level, professional autonomy is difficult to discern and administrators are seen fulfilling the dictates of the duly elected school board to insure that teachers are competently delivering instruction to the students. At the secondary school level, teachers enjoy more autonomy to structure their classes and curriculum as they judge appropriate; administrative authority is exercised seldom and usually only in crises. College teachers enjoy autonomy granted by a three hundred-year tradition of academic freedom; no administrator dares to cross the threshold of the lecture hall.

Some have sought to reform teacher evaluation by attempting to alter the balance between these two forces. Art Wise and Tamara Gendler (1990), in The New Handbook of Teacher Evaluation, distinguished seven purposes and separate functions of teacher evaluation: preservice, selection, certification, "beginning," tenure, merit and school improvement. There is much that can be said about the politics of each of these separate phases of teacher evaluation (most of which would center on the politics of higher education and of the labor movement). Wise, for example, focuses on licensure and recommends a state board licensure system for teachers like that for physicians and lawyers. Such licensing might confer prestige on the profession and with prestige may come autonomy. But one might wonder whether medicine is well served by doctors or justice by lawyers. John McNeil (1981), in the Handbook of Teacher Evaluation, acknowledged the deep conflicts that surround this phenomenon in the school, but recommended new forms that scarcely differ from established practice and that fail to separate incompatible purposes for evaluating teachers. Armiger (1981), of the New Jersey Education Association, recommended guidelines for teacher evaluation that would give teachers more power within a system still "owned" by the bureaucracy.

It is our contention that the problems with teacher evaluation do not stem principally from the conflict between professional autonomy and bureaucracy (although these forces are apparent, they can not be changed without changing the political context), but from the perceived illegitimacy of the democratic bureaucracy in which the evaluation is embedded. Our argument will benefit from a portrayal of how teacher evaluation is experienced in the work lives of teachers.

Teacher Evaluation in Nocam

Nocam Elementary School is a K-8 school of 800 students located in the heart of a city of over two million people. Its attendance area is about a third Anglo and half Hispanic, with a smattering of children of many different cultures. Nocam is one of three elementary schools in the school district. It has a full-time principal and is closely linked to the Superintendent's office through the efforts of a curriculum specialist who has assisted Nocam is a major overhaul of its language arts curriculum. "Whole language" teaching, cooperative learning and non-graded organization have come to Nocam. In the course of pursuing a larger study focused on school reform, the second author conducted numerous interviews with Nocam teachers and administrators and observed classes, board meetings and teachers meetings. On the following pages, Nocam teachers speak of the way in which their work is evaluated by their superiors.

District policy mandated that all teachers be evaluated once a year, despite the fact that there was no merit system of pay. Teacher evaluations were conducted by the district personnel director, except in the case of "new" teachers. "New" teachers, those who had been employed by the district for less than three years, were evaluated by their immediate supervisor, the school principal. Both new and veteran Nocam teachers viewed the evaluation process as "a joke," regardless of who the evaluator was. As one teacher explained,
"...some man is going to come into my classroom, who has never been in my classroom all year and evaluate me on how good a teacher I am, by [observing] a twenty minute lesson [and]checking things off? That's impossible. I couldn't evaluate my students that way."

Teachers generally shared this opinion,
"...it's a scheduled appointment, they will be in your room at 10:30 and you have to have the handbook and the detention notices and the homework notices... they want to see homework, they want to see discipline records and it has to all be clearly posted, your discipline plan and everything...."

Few of the veteran teachers were intimidated by the evaluation process; many, however, found the process coercive and demeaning. As one teacher explained,
"I never see that personnel director except when he comes into make an appointment to do the observation... and you can't talk to him then. And then the next time that I see him is when he is handing me back my evaluation. And the thing is s o arbitrary...it's 'you're an A teacher, you're a B teacher, you're a C teacher, and you fail.' I don't need to know if I'm an A teacher or a B teacher, I don't care. I care about whether or not I'm improving. They don't have enough respect for me...the amount of work that I have done and the amount of dollars that I have put in...to give me something that would actually help me improve. Instead they give me something that makes me work first of all to put something together for [the evaluator] to keep, then they want to evaluate my classes."

Few teachers found the evaluation process informative or instructive. Teachers complained that the process failed to provide them with any insights as to how to improve their teaching. Frustrated by the procedure, teachers did not feel that the exercise was meant to help them improve their teaching,
"...it's not meant to improve [teaching] although that's the letter of the law, that the teacher evaluation systems are to improve teachers...to improve instruction. But it does not do that. In fact if it does anything I think it doesn't improve my instruction because I'm ticked for two days before I have to do it and I'm ticked for two days after I have to do it."

"New" teachers were a bit more anxious about the process, perhaps because they were evaluated by their principals. Like the veteran teachers, however, new teachers found the evaluation process much more burdensome than helpful. The evaluation experience described by Ms. Clark, a "new" K-3 Project teacher, was not unusual:
"[The principal] kept saying ahead of time 'don't worry, I really need to see what's going on in your class'. I thought okey. And it was a time when I thought it was a good lesson, except for one kid, and this kid has been documented s exually abused, well he pooped in his pants during the lesson. The principal got hysterical with me, he was like 'I spent forty minutes in this room and I've seen nothing of value happening', and he left. That was my first evaluation. ... But I decided that I was going to talk to the principal [about my evaluation] and I said [to him], 'I'm a first year teacher and you can't just walk in my room and spend almost an hour and tell me nothing of value is going on. I'm not leaving until you tell me what good you saw. So he decided not to count that [evaluation] and to do it again. So for the next one I prepared the kids...."

Ms. Anderson was coached by her fellow teachers as to what this principal liked to see, the curriculum he preferred, and the practices he approved of. With this information in mind both she and her students practiced what they would do for the next evaluation:
"...we practiced, we rehearsed what we would do when [the principal] walked inandItoldthekidsifwediditright,wegotasurprise. And[other teachers] took the worst kids [to their classes that day] so that it wouldn't be bad. "It was awful...like the kids did these little work-sheets and they sat there and we had practiced what the work-sheets would look like, so they sat there and did them without talking. It was so awful...it was really hollow...we played the game. [The principal] told me I did a good job and I thought [to myself] 'you don't know anything.' I've learned part of the principal's game. I can do it when I have to, I've done it."

Stories about "putting on a show" for both evaluators and administrators were common among Nocam teachers:
"The kids know how to act for the administrators. We bribe them [to act a certain way] when administrators are there. Then [the administrators] leave and we go back to our real way of working and of teaching."
"...when the district people come into our class, I have to act a certain way, to put on a show.... I train the kids to act the way the administrators expect them to act ... even if the way they act [and the things we do] are not developmentally appropriate."

The administrators' ability to evaluate accurately teacher's performance was questioned by many teachers. Nocam teachers were of the opinion that their administrators did not really understand the pedagogical techniques nor the theoretical underpinnings of the techniques which were the basis of the K-3 Project,
"I don't think they have any idea what [whole language or developmentally appropriate practice] look like. They think they are very supportive of whole language, but it's only as long as kids are sitting at their desks being really quiet."
"...they don't understand how children learn and they come in and what can you tell them when you talk [with them] in the classroom for five minutes. Of course it looks like chaos...but learning is going on...you have to be there for a while to really get a grip on what is happening with the children...they don't understand [developmentally appropriate practice]. [The principal] evaluates you at a desk, a file cabinet between you and the reading group, he's not listening to the kinds of questions I'm asking the children or, you, know the communication skills going on. He's watching the behavior problems, and you are always going to have some. He's counting how many crayons the kids have on the floor."
"They are not very supportive in the teaching methodology way, but more picky, and you have to do this and you have to teach from this book and you have to cover so much and you try to slip your own things in between without getting caught... You understand that [administrators] don't know much about [teaching] and you try to take it with a grain of salt."
"...there are all kinds of politics on why we are getting raked over the coals for this and that, but if a principal does not understand what you are doing in the classroom before the evaluation starts, or if they don't agree with what you are doing, how can they evaluate you fairly ..."?

In addition to their not being very well grounded in the nontraditional models used by the K- 3 Project teachers, many Nocam teachers were of the opinion that administrators didn't spend enough time in classrooms to accurately assess their teaching ability:
"... they come in and what can you tell them when you talk [with them] in the classroom for five minutes ... you have to be there for a while to really get a grip on what is happening with the children...."
"...I don't really think he has a perfect understanding of what it is ... of what exactly it is, because he doesn't come in our classrooms and hang out for an hour or two for a few days a year."

Nocam teachers also had concerns about the evaluators' ability, or lack thereof, to provide them with practical guidance and relevant assistance:
"[The administrators] can't sit and discuss whole language theory with you .... if they can't discuss the concepts with you how can they tell if what you are doing is right or wrong, and how can they help you improve upon it?"
"... sometimes the things that they ask us to do don't particularly correspond with what we are trying to accomplish [in terms of teaching]."
"... they don't understand how children learn....they don't understand developmentally appropriate practice...I can't get much guidance from them...."
"[The principal] says 'what can I do to help you', but I feel that there is nothing he can do to help me because he doesn't have any knowledge to give me ...."
"I know they are busy [but] they need to spend more time in the classrooms with us. They come in once in a while, they make me nervous because they don't come in often enough for me to feel like they are friends, like I can ask them for help...I don't even know if they know what I'm doing in here."
"...They might be supportive when you are explaining [the methodology] to them, the cooperative learning and the learning centers, but then they come into our rooms and see the movement and it's 'wait a minute, you didn't say kids were going to be talking to each other and moving around the classroom, you said cooperative learning.’ "

Nocam teachers lacked guidance and direction. Even their direct supervisors—the school principals—lacked the appropriate pedagogical theory, and therefore were not a source of assistance or guidance. As one teachers explained:
"I've really had to depend just on myself with all this. It's like I've been left out on this island, all alone ... no guidance, no support, no validation ... it's been pretty much a sink or swim situation ... I still don't know which I'm doing...."

Nocam principals viewed their role primarily as that of "facilitator." Although these administrators encouraged their kindergarten through third grade faculty to "use whole language, cooperative learning, and developmentally appropriate practices," neither they nor the district superintendent provided teachers with concrete suggestions for implementing the techniques or improving their teaching. This task, according to the principals, was "left to the experts," who were brought in to provide in-service training throughout the school year.

The K-3 Project teachers managed to convince the district administration that it was unfair for teachers to be evaluated on their use of traditional teaching techniques when the K-3 Project relied so heavily on the use of nontraditional approaches. After three years of requesting, and largely in response to the requests of the K- 3 Project coordinator, Nocam teachers were given a choice of being evaluated based on the standards of the "traditional teacher evaluation" instrument or based on the standards of a "whole language evaluation" instrument, recently developed by the district administration.

Both instruments assessed the same general categories of performance: classroom management, communication skills, instructional capabilities and materials, planning and organizational skills, compliance with school policies, and professional qualities. The whole language instrument, however, was much more extensive than the traditional instrument. The traditional evaluation instrument contained a total of five criteria per category which teachers could "exceed," "meet," or against which they could be judged "average" or "failed to meet." The whole language instrument contained twenty different criteria per category, which teachers could "exceed," "meet," or with respect to which they could be found "adequate" or "inadequate."

Technically, teachers had a choice in the evaluation matter; practically they did not. Neither the district evaluator nor the principals were adept in using the nontraditional instrument. As a result, though a fair number of teachers requested that the new instrument be used, only one teacher was actually assessed with it. The process was described by the teacher as "a disaster."

According to this teacher, the district evaluator did not have the new forms in his evaluation package when he came to evaluate her, nor did he know what was on the forms. The teacher had to supply the evaluator with the new forms. The evaluator didn't understand the stated criteria; he had the teacher explain to him how the new criteria related to the old, so he would know what to look for during the evaluation session. The teacher described thus:
"It was just a disaster.... [The district evaluator] couldn't sit and discuss whole language theory with me .... he doesn't know a thing about developmentally appropriate practice ... and cooperative learning ... forget it .... he [kept] looking for my assertive discipline program. That's not my priority .... I had to explain the entire process to him, what to look for, what was appropriate and why. I'm sure he learned a lot, if he paid any attention, but for him to evaluate me, what a joke."

Whether they knew how to use the nontraditional evaluation instrument or not, the district principals avoided using it. The principal at one of the other two elementary schools in the district went as far as to tell his teachers that it was his choice which instrument was used, not theirs. And he chose to use only the traditional instrument.

One teacher decided to "check this out with the district." She was informed that the teachers did indeed have the right to choose. When she shared this information with her principal he "got upset at me for questioning his authority and he told me that he was going to talk with the district people himself. In the meantime, he used the traditional instrument to evaluate me."

The Nocam district superintendent maintained that the district allowed teachers "the freedom to use what they think is appropriate" in teaching their classes. He also believed that someone needed "to make sure that what they feel is appropriate is in line with ... our curriculum philosophy... [that it is] highly matched to what we expect kids to be tested on." A number of tools were developed to assist Nocam administrators in monitoring curriculum and instruction.

The Nocam case makes one thing clear. Even at the elementary school level where it might be expected that administrative evaluation is most defensible, it is viewed by teachers as illegitimate. Principals are seen as uninformed about curriculum and unable to spend the time to understand the circumstances of the class in such a way that they could help improve it.

Bureaucratic evaluation of teaching at the secondary and college levels is seen as an affront to professional autonomy and as being even less legitimate than elementary school teacher evaluation. It can not be argued that what was seen in Nocam is somehow an outgrowth of the special circumstances of poverty or ethnic minority culture. Similar experiences are widely spread in the American educational system and, perhaps, elsewhere. (Beery, 1992)

Legitimizing Teacher Evaluation

It is our contention that the principal problem with teacher evaluation is that it is viewed as lacking legitimacy by the persons who are the object of the evaluation, the teachers themselves.

Consequences of Loss of Legitimacy

Teacher evaluation viewed as illegitimate by teachers themselves generates nothing but dissembling, passivity, and feelings of alienation and powerlessness. (Glass, 1990) School boards through administrators have a legitimate interest in how instruction is conducted, but it is not an overriding interest nor does it follow that their interest is served by direct participation of the principal in the evaluation of teachers.

Where legitimacy is lacking, one can expect little more than passive compliance. Is it a matter of concern that an evaluation system is imposed from the administrative hierarchy and not seen as legitimate by the teachers who are being evaluated? One line of argument answers "No." Suppose that the system imposed is so comprehensive and well designed that it encompasses most of what teachers should be expected to perform. One might argue then that it is irrelevant whether the teachers "like" what it imposes, since if they conform to its vision of what a good teacher is, they will ipso facto teach well. This argument is similar to questions debated under the topic of "teaching to the test" in educational assessment. Some maintain that if a test is good enough, then teaching to it will only result in good education. Similarly, if a teacher evaluation system is good enough (i.e., defines a good model of what a teacher is to be), then complying with it—even if that compliance is "false" or pretended in some sense—will result in the teacher being a good teacher.

The difficulty with the "teaching to the test" argument is that the kinds of test generally used in assessments are rather pale reflections of a good education. Likewise, many of the bases of teacher evaluation systems are weak or impoverished models of what good teaching is. Checklists of teaching acts or "elements" reduce teaching to a few general principles of instruction, and divert attention from concerns of curriculum. A teacher can be a good teacher under such surveillance while teaching shallow or false knowledge. Some believe that this is little concern at elementary grades since "there is no discipline" (in the academic sense) at that level. "Elementary school teachers have no discipline, they just teach"; or "teaching is their discipline." Others are shocked to hear that the teaching of reading or la nguage or mathematics is believed by some not to raise technical and intellectual questions as complex and sophisticated as the teaching of calculus to high-school students.

Ways of Seeking Legitimacy

Legitimacy can be bestowed in at least two ways: by appeals to widely accepted scientific or technological knowledge or through the appeal to the authority of legitimate political institutions or arrangements.

The attempt to legitimate the standard practice of teacher evaluation by appealing to science and technical-rationality fails for a couple of reasons. First, there is no widely respected science of teaching and learning. Common sense or practical and tacit knowledge of teaching usually succeed as well as systems that profess to be based on research. Second, most efforts to reform teacher evaluation start from an assumption that all parties with a direct interest in improved education share a consensus on what good education is. From false assumptions of consensus come technical-rational attempts to manage teachers. We begin from a different starting point. Schools are micro-political units where teachers, administrators, parents, students and even society far removed from the classroom seek to realize their interests.

These interests often conflict. Without agreement on ends, mechanical and technical solutions fail. Third, school administrators who are vested with the authority to evaluate teachers as instructors generally lack knowledge of the subject matter being taught. Their role as evaluator consequently strikes the teacher as superficial, then illegitimate. (See Scriven, 1992, for a discussion of what limits subject matter specificity does and does not place on teacher evaluation.)

An example may help illustrate how a nonspecialist's lack of subject matter knowledge can invalidate the type of evaluation that focuses on general acts of teaching. A junior high school English teacher is reviewing a lesson on nouns. He writes on the board "A noun is a Person, Place or Thing," leaving space beside each for examples. Turning to the class, he invites anyone who can illustrate a noun as the name of a person to step to the board and write. A student is congratulated for writing "singer" beside Person, as is a second student who writes "school" beside Place. The third volunteer writes "kitchen" beside Thing only to be told politely by the teacher that a kitchen is a place, not a thing. The checklist that the observer of this episode was filling out had categories only for the commonly identified important elements of teaching: previews lesson, clarifies goals, provides for active participation, reinforces correct responses, and the like. This teacher even scores points for correcting mistakes quickly. This kind of evaluation of this act of teaching misses the important point of what actually took place. One need not be a grammarian to sense that something is seriously wrong with this lesson. Of course a kitchen is a "thing," as in "I have remodeled my kitchen." And it may function as a place in other uses. Nor are Person, Place and Thing mutually exclusive and exhaustive categories. Where does the "unicorn" reside, grammatically speaking, and what about "truth"? The point is that this teacher is teaching a shallow or false point and this consideration should override all other questions. Indeed, his grasp of grammar has led him to confuse a student (probably more than one) and draw that student into a publicly embarrassing situation, where her valid understanding of language usage is labeled "wrong." Where in the evaluation of this teacher is it noted that the teacher has a responsibility to understand and continue to learn the subject being taught? Some will argue that it is impossible for a principal or a principal's deputy to know all the subject matter taught by all the teachers in the school. Indeed it is; and lacking that understanding, it is questionable to what degree the principal can serve as the guide for the teacher's efforts to become a better teacher.

If technical-rationality cannot confer legitimacy on teacher evaluation, then it remains for political arrangements to do so. Modern political institutions are bureaucratic democracies, with one distinguished from the other by the balance of democracy and bureaucracy.

Appeals to the scienc e of teaching or to technical-rational arguments about what ought to be taught and how can not hope to justify a particular form for teacher evaluation within the hierarchical bureaucracy of contemporary schools. Legitimacy for some form of teacher evaluation must be found in a new set of political arrangements that will be viewed as legitimate by the teachers themselves.

Seeking New Political Arrangements

Three evaluation theorists have addressed the systemic political problems that have led to the current state of teacher evaluation.

MacDonald. "Democratic evaluation," as envisioned by Barry MacDonald (1974), addresses the tension between power concentration and power diffusion in liberal democracies by opting for radical power diffusion. MacDonald focuses more on the role of the evaluator than on such aspects of the evaluation as the criteria, data and the like. He saw the evaluator as an information broker among interested parties. The evaluator stops short of making recommendations; information, ultimately owned by those from whom it is collected, is presented to those persons with legitimate interests in what is being evaluated. Decisions flow from some unspecified process of democratic discussion among interested parties. "MacDonald's evaluation approach intentionally includes diverse interests, allows people to represent their own interests, and is based on an idea of mutual consent." House (1980, p.150) A direct, rather than a representative form of democracy, is being imagined by MacDonald. The limitations of direct democratic participation in complex, mass societies are obvious. However, one is casting a small net when the object of an evaluation is a teacher and a classroom. The range of interests to capture and bring to consensus is narrow. After all, juries reach consensus even when the stakes are high.

Strike. Kenneth Strike has pursued an examination of political forms and their relevance to education. He contrasts two quite different approaches to achieving democratic governance. The first is John Locke's legislative majoritarian democracy. Its operations are familiar to all; its assumptions are less obvious. Naturally free and equal humans are to be granted equal sovereignty, which is exercised by voting for representative government. With the consent of the governed, sovereignty is placed in the legislative body. Legislatures exercise sovereign control through hired managers who follow the policy direction set down by the legislative body. Exigencies will call for clearer rules and policies; in time, rules will accrue and the modern bureaucratic democracy will emerge. Citizens may not regard every rule as legitimate, but the stability of the institution rests on a wide-spread belief in the legitimacy of the process by which the representatives are first chosen and then formulate the rules. "We can see a legislature as a means to vector interests more than as a means for making and judging the merits of practical argument. Majorities may be seen as formed more by a process of combining and reconciling interests than by a process that seeks the better argument." (Strike, 1993, p. 16). Current practice in teacher evaluation is embedded in this context of legislative bureaucratic democracy. It has failed to engender among teachers a belief in its legitimacy. Most writing to date about the politics of teacher evaluation have assumed no changes in the basic nature of democratic institutions and as a result offer suggestions that tinker with the balance of power between democratic bureaucracies and professions. An opposing view of democratic institutions grows out of the attempts of Jurgen Habermas to justify liberal democracy. Habermas argues for the legitimacy of a communitarian democracy in which social norms are justified by uncoerced a rgument among equals in an ideal speech community. To Habermas, a social choice is "discursively redeemed" when it has the consensus of a community of citizens and that consensus was reached in open and undominated discourse. Argument—not votes—legitimates choices and actions for the good of the community.

Strike recognizes a Utopian character to Habermas's notion of the discursive redemption of policy choices in an ideal speech community. As a practical matter, sovereignty will have to be located in a representative body and conflicting interests "vectored" to a solution when consensus is impossible. But he can not back away from Habermas's ideas without trying to answer the question, "How might we make bureaucratic democratic institutions more Habermasian?"

Kemmis. While Strike may wish to cook the Habermasian omelet without breaking the Habermasian eggs, Stephen Kemmis does not hesitate to recommend the Habermasian ideal. To Kemmis, teacher evaluation would be one particular aspect of what he calls "emancipatory action research" (Carr Kemmis, 1986; Kemmis, 1993a; Kemmis, 1993b). "When schools— teachers, students, principals and others—are forced to change on the basis of outside evaluations and the crude coercive powers of the state, however, they frequently resist, passively if not actively. And that, it seems to me, just produces still further administrative demands for surveillance, regulation and control.... I believe that the evaluation processes I have attempted to develop—as well as some of the practices associated with 'responsive', 'illuminative' and 'democratic' evaluation—did (and do) contribute to the development of less irrational, less unjust and less satisfying forms of social life. Though some of those perspectives have no particular inclination to justify themselves against the criteria of critical social theory or critical social science, in practice they do seem to offer increased opportunities for what Habermas describes as "communicative action"—action oriented towards mutual understanding and unforced agreement...." (Kemmis, 1993, pp. 46-47)

Conclusion

Teachers view the evaluation to which they are subject as being illegitimate. They do not recognize the authority of those who perform the evaluation; they do not accept it as valid and defensible. Legitimacy can be conferred by democratizing the process of teacher evaluation, by removing it from the context of hierarchical bureaucracy in which it now resides, and by carrying it out in a new context. Some theorists offer justifications for this rearrangement of the politics of teacher evaluation. As yet, it is unclear how the reform would be played out in its essential details, e.g., who would participate in the evaluation of teachers, what information would be relevant and how it would be obtained, with whom authority would lie to call for an evaluation, and the like.

References

Armiger, M.L. (1981). The political realities of teacher evaluation. Chp. 16 (pp. 292-302) in Millman, J. (Ed.) Handbook of Teacher Evaluation. Beverly Hills, CA: SAGE Publications.

Beery, B.F. (1992). Master teacher conceptions of relationships between teacher evaluation and excellence in teaching performance. Doctoral dissertation. Tempe: Arizona State University.

Carr, W. & Kemmis, S. (1986). Becoming critical: education, knowledge and action research. London: Falmer Press.

Darling-Hammond, L., Wise, A.E. and Pease, S.R. (1983). Teacher evaluation in the organization context: A review of the literature. Review of Educational Research, 53, 285- 328.

Glass, G.V. (1990). Using student test scores to evaluate teachers. Chapter 14 (pp. 229-240) in Millman, J. & Darling-Hammond, L. (Eds.). The New Handbook of Teacher Evaluation. Newbury Park, CA: SAGE Publications.

House, E.R. (1980). Evaluating with validity. Beverly Hills, CA: SAGE Publications.

Kemmis, S. (1993a). Action research and social movement: A challenge for policy research. Education Policy Analysis Archives, 1(1). Retrieved May 1, 2005 from http://epaa.asu.edu/epaa/v1n1.html .

Kemmis, S. (1993b). Foucault, Habermas and evaluation. Curriculum Studies, 1(1), 33-52.

MacDonald, B. (1974). Evaluation and the control of education. Norwich, England: Centre for Applied Research in Education.

Martinez, B.A. (1993). How educational reform is compromised: A critical investigation. Doctoral dissertation. Tempe: Arizona State University.

McNeil, J.D. (1981). Politics of teacher evaluation. Chp. 15 (pp. 272-291) in Millman, J. (Ed.) Handbook of Teacher Evaluation. Beverly Hills, CA: SAGE Publications.

Scriven, M. (October 1992). Should teacher evaluation be subject-matter specific? TEMP Memo #10. Kalamazoo, MI: Center for Research on Educational Accountability Teacher Evaluation, Western Michigan University.

Scriven, M. (September, 1991). Duties of the Teacher. Memo from the Center for Research on Educational Accountability Teacher Evaluation, Western Michigan University. (Also see Scriven, M. (1988). Duty-Based Teacher Evaluation. Journal of Personnel Evaluation in Education.)

Strike, K.A. (1993). Professionalism, democracy and discursive communities: Normative reflections on restructuring. AmericanEducationalResearchJournal,30(2),255- 275.

Strike, K.A. (1990). Is teaching a profession? How would we know? Journal of Personnel Evaluation in Education 4(1990), 91-117.

Strike, K. A. (1991). The Moral Role of Education in a Liberal democratic Society. In G. Grant (Ed.),Review of Research in Education (Vol. 17, pp. 413-483). Washington, D.C.: American Educational Research Association.

Strike, K.A. (1991). Humanizing Education: Subjective and Objective Aspects. Studies in Philosophy of Education 11(1), 17-30.

Strike, K.A. (1990). The ethics of educational evaluation. Chp. 21 (pp. 356-373) in Millman, J. & Darling-Hammond, L. (Eds.). The New Handbook of Teacher Evaluation. Newbury Park, CA: SAGE Publications.

Strike, K. & Bull, B. (1981). Fairness and the legal context of teacher evaluation. Chp. 17 (pp. 303-343) in Millman, J. (Ed.) Handbook of Teacher Evaluation. Beverly Hills, CA: SAGE Publications.

Wise, A. Gendler, T. (1990). Governance issues in the evaluation of elementary and secondary school teachers. Chp. 22 (pp. 374-389) in Millman, J. & Darling- Hammond, L. (Eds.). The New Handbook of Teacher Evaluation. Newbury Park, CA: SAGE Publications.