Gene V Glass Archives: An Archaeological Dig for VAM

The following is an edited log of an asynchronous online discussion of the Tennessee Value Added Assessment System (TVAAS) for assessing teachers. This discussion took place in late 1994 and early 1995 on an internet LISTSERV known as EDPOLYAN, which was housed at the Arizona State University College of Education.

The TVAAS was one of the first techniques proposed for measuring the value that teachers added to students' learning, namely, VAM systems. "Value," in this context was taken to mean pre-test to post-test gains in class averages on standardized tests. TVAAS was developed by an agricultural statistician at the University of Tennessee, William L. Sanders, and first published in a little known journal in 1994, just months before this discussion took place. Sanders occasionally participated in the discussion that follows, but it will soon become apparent that he was not happy with the direction the discussion took. He died in 2010.

The log of the discussion was edited in an attempt to add clarity and reduce redundancy. Each participant in the discussion was given the opportunity to review the edited log and make corrections.

The reader may want to know that TVAAS and VAM more generally became the object of a great deal of attention from the political world and from academics studying statistics and teacher evaluation. VAM made appearances in the accountability plans of several states during the Obama administration -- largely because it had won the approval of Secretary of Education Arnie Duncan. Post-Obama, VAM made a few appearances in court, where decisions largely enjoined its use. Suffice it to say, by 2020 VAM is little used and has few friends.

Anyone with a continuing interest in VAM will learn much from Audrey Amrein-Beardsley's blog Vamboozled.

===============================================
Date: Fri, 9 Sep 1994 12:57:52 EDT
From: Scriven@AOL.COM

"Do you really want to see teaching become an even higher-turnover profession?" No, and it's because of the bad teacher evaluation systems now in use that we lose many good teachers, so I'm trying to improve the evaluation systems. Piece work is not the best way; it's just a counter-example to the idea one can't use individual rewards.

I do include the TN Value Added System as one of the new and promising efforts, and I'd like to hear what you see as its flaws.

The reason for rewarding teachers as individuals is that nearly all of them work as individuals; the Deming approach is fine for situations where you only have groups as the work unit.

----------------------------------------------

Date: Fri, 9 Sep 1994 23:17:35 -0500
From: SHERMAN DORN

For those of you unaware of the Tennessee Value Added Assessment system (mandated by state law), I'll try to describe it briefly: you take the standard scores from the state normative test for each child, subtract from it the standard score for the child from the previous year, and use the gain score (essentially a putative change up or down a normal curve) in a statistical equation with district, school, and teacher as explanatory variables. (For statisticians out there, I know Bill Sanders, the agronomist who created VAA [I guess he got the idea from measuring corn yields or something similar], uses a mixed-model equation. I don't know, however, which are the fixed-effect variables and which the random-effect variables.)

The problems with VAA? Well, I can think of a few:

a. The intrinsic worth of the standardized tests are questionable, at best. For test security, test questions and structure are confidential. Which means we can't judge what, in fact, the things are measuring. Not to mention the usual shenanigans that occur in schools with high-stakes tests.
b. The tests for different grades (first, second, etc.) have different norming populations that are not equivalent: some people are retained in every grade, there's some differential migration, more kids in higher grades are certified in special ed (and thus likely excluded from the norming population) -- with the end result that a standard score of 500 in first grade (putatively at the mean of a normal curve) cannot mean the same thing for ANY OTHER GRADE. Ergo, subtracting standard scores is patent nonsense. (Essentially, having noncomparable norming populations does two things: shifts the mean around, and changes the standard deviation, but in ways we just can't know without having access to the original norming group.)
c. A gain score is a questionable basis for statistical analysis.
d. Since the standardized tests are taken at least one, and in some years almost two, months before the end of the year, gain scores conflate the effects of two different teachers.
e. VAA may seriously underestimate the effects of prior knowledge, social background, etc. I know of no analysis to date of the implications of model misspecification for VAA -- as far as I know, Sanders just plugs in last year's scores, this year's scores, the districts, schools, and teachers, and lets the program rip, without reference to race, sex, gender, or even the possibility of a nonlinear relationship between last year's score and this year's. Would the effects be different if you put in sex, race, economic class, perhaps a square of last year's scores, in the equation? I bet no one knows.
f. VAA is not an evaluation system accessible to teacher understanding. It is as abstruse and anxiety-provoking as anything one could imagine. An incentive for better teaching? Absolutely not, from my experience working with teachers. An incentive for absolute panic? You bet.

The reason for rewarding teachers as individuals is that nearly all of them work as individuals; the Deming approach is fine for situations where you only have groups as the work unit.

The question is not whether people work as individuals or groups, but whether the *work* is individual or group. And, unless I'm wrong, the socialization and education of children is not done by an individual but by lots of people. Is the third-grader who reads poorly the fault of the third-grade teacher? Perhaps a bit, for plenty of third-grade teachers will refuse to teach a child who cannot read. But it is certainly the fault of prior teachers. Same for a high-achieving child in a subject.

And the status quo of individual teaching does not justify the use of individual incentives; we need to compare individual merit-pay schemes with more collective incentive systems combined with a change in teaching structure. The fact that teachers do not work together now very often is not a justification for disincentives from working together forever.

-----------------------------

Date: Thu, 27 Oct 1994 13:22:16 LCL
From: "William L. Sanders"

Last week, a response to one of Scriven's comments by Dorn (9- 9-94) was called to my attention.

I am writing in response to this totally erroneous and distortive description of the Tennessee Value-Added Assessment System (TVAAS). Never in my 28+ years working in the academic arena have I read or seen such a blatant misrepresentation of the truth--a misrepresentation which, I hope, is based upon a lack of knowledge instead of a deliberate attempt to sabotage. I want to response to this gross misrepresentation point by point.

First, I am not an agronomist; I am a statistician. For many years, I have had the responsibility for the Statistical and Computing Services Unit within the Agricultural Experiment Station. The University of Tennessee being the Land-Grant university of Tennessee, not unlike Iowa State or Purdue or NC State, etc. has supported statistical consulting and research through its Experiment Station for many years. I am also an Adjunct Professor in the Department of Statistics which at UTK is in the College of Business. I let my professional record and research speak for itself. My research interests are and have been in the areas of: 1. experimental design and 2. statistical mixed models. I came into the educational research arena in the early '80s quite by accident; but due to an apparent need to solve many of the statistical problems which educational types were citing, this has become in the past 5-6 years my primary focus.

What is so distressing about Dorn's comments is the blatant lack of knowledge about our work and methodology. He claims to know me; as I write this, I would not know Sherman Dorn if he walked into my office. I have made over 200 presentations conceptualizing this approach; if I met him during and after one of those presentation, I certainly do not remember it. If he is so committed to 'trash' this approach at least he should have the common decency to make accurate and objective academic arguments without relying on petty slurs and innuendos.

Second, his attempt to describe the modeling approaches are totally inaccurate. We do not even calculate simple gains. For example, we use the whole observation vector for each child over all subjects and grades. In fact, we have a 'small' article later presented at an American Statistical Association meeting that demonstrates how this approach is superior to traditional multivariate approaches in that the whole observational vector is not lost due to one missing value for a variable.

But the most telling comment of his ignorance of the TVAAS process was when he indicated that he did not know what was 'random' in the model. The process is built on the formulation of the late C.R. Henderson, a Cornell animal breeder who was named a fellow in the American Statistical Association for his pioneering work in this area. Henderson's development of the concept of best linear unbiased prediction (BLUP), has been shown to be related to other shrinkage estimator concepts by David Harville at Iowa State and others (i.e. many bayesian concepts, kalman filtering from the engineering sciences, some of the hierarchical linear model concepts, etc.) . However, we have found that Henderson's formulations have tremendous computing advantages over some other equivalent alternatives. In fact, while I was a consultant to SAS Institute, Inc., as they were planning and implementing their rather new MIXED procedure, I strongly recommended that they use this formulation in their development and manuals -- a recommendation which was accepted.

As we apply these approaches in the context of the estimation of the teacher and school effects on the academic growth of populations of students, we take advantage of the prior knowledge of the distribution of the variance-covariance structure among populations of teachers, as well as the variance-covariance structure among students. By so doing, we solve the following problems:

2. teachers changing assignments
3. modes of instruction (team teaching, departmental instruction, self contained classrooms)
4. use of dissimilar indicator variables over time.
5. combinations of different quantities and qualities of information
etc., etc., etc., etc., .......

Dorn indicated that "since the standardized tests are taken at least one, in some years almost two, months before the end of the year, gain scores conflate the effects of two different teachers". This is another example of his lack of knowledge of the process. We have developed a process which we call the 'stacked block concept' which enables the partitioning of these effects. This concept is not totally dissimilar to the recovery of interblock information from incomplete block designs.

I invite any of you, including Dorn, to call and we will supply you with as much detail and information as you wish.

As to his concern with regard to model specification, we have done a tremendous amount of computer simulation to evaluate just that concern, and we can demonstrate to any reasonable person that this process is very robust against rather severe mispecification.

We welcome legitimate and qualified criticism. For the past four years, we have developed and implemented this process for the whole state of Tennessee. Presently, the data base contains over 3 million records longitudinally merged for the entire state's student population. To provide these estimates for all schools (and next year all teachers grades 3-8 for the entire state), the solution to tens of thousands of equations is required. We are doing this computing on a dedicated RS-6000 workstation. Because of this work crunch, our writing has lagged; however, we do have a sufficient amount of written material to enable informed comment. Informed comment, we welcome; hatchet jobs like Dorn just pulled, have no place on the internet. I describe his recent message on the board as worst than political dirty tricks. That kind of activity is not consistent with the academic reputation and tradition of one of Tennessee's treasures, Vanderbilt University.

======================================================

Date: Fri, 28 Oct 1994 07:41:07 -0500
From: SHERMAN DORN

I will reply at some length to Prof. Sanders' comments later, but just a few points here:

1. I have searched in several places for peer-reviewed descriptions of the Tennesee Value-Added Assessment System (TVAAS), as well as looking at the text of the legislation in Tennessee Code Annotated. Until a member of the staff at the center producing the assessments told me last week of an article in the *most recent* Journal of Personnel Evaluation in Education, I have been unable to locate a single article about it through ERIC or the other periodical indices I have searched. What I described in September was the result of information I had at the time -- which, by the way, was more than what teachers and principals are given to understand about TVAAS. The legislation was passed several years ago; the fact that I have been unable to locate peer-reviewed articles on the subject using abstract indices in 1994 is not evidence of laziness or misrepresentation on my part, I believe.

(It is not evidence of either on the part of TVAAS center staff, either, just not evidence about my actions.)

2. I stated very clearly that I *knew* of no attempts to gauge the results of model misspecification and that I thought it would result in wide discrepancies -- and by this I mean discrepancies in the rankings given schools. My question about what are the random or fixed effects stands: the fact that the TVAAS uses a mixed-model methodology which SAS now uses does not tell us whether the specific models used in calculating TVAAS make sense in the real world. I know SAS has that model, and I have read the article in American Statistician which Sanders co-authored and which is cited in the legislation as criteria for acceptable statistical models. The fact that Sanders' mixed-model methodology is a good general approach to mixed models (where the effects of some things are fixed and others are random, with the possibility of interactions among the levels) is irrelevant to the question of whether the model MEANS anything.

3. Prof. Sanders' statistical expertise does not answer the policy-related questions which I raised at the time -- or the additional ones I have now.

Quickly re-reading my comments in September, I regret implying that Sanders' expertise was in agronomy and not in statistics only. I will confirm that Sanders left me a message several days ago, and I have sent an e-mail message explaining why I have not returned his call. (I did so yesterday, before I saw his message. The two may have passed ... rather, the fact that we did not see the other's message does not imply anything except that the Internet is not instantaneous.)

(Yeesh, please mentally delete "only" from "in statistics only." I forgot to use the "/edit" command for this, and can't re-edit lines above.)

As mentioned above, a TVAAS staff person has pointed out to me a recent article, and I will wait until I receive it and read it before I comment further. I have also recently become aware of Sanders' presentation to the CREATE folks at Western Michigan State last year and (according to one educator I know) of presentations about TVAAS this summer. Unfortunately, the CREATE gopher site at Western Michigan is not yet up-and-running, and the campus' gopher does not have a working faculty directory. (CREATE is one of the federally-sponsored research centers for education, and although I forget what the acronym stands for, it is about assessment and evaluation.)

====================================================

Date: Fri, 28 Oct 1994 12:02:53 MST
From: Gene Glass

Dear Professor Sanders:
I like statistics; I made the better part of my living off of it for many years. But could we set it aside for just a minute while you answer a question or two for me?

Although I have read little about the Tenn Value Added Assessment system, I gather that it is a means of measuring what it is that a particular teacher contributes to the basic skills learning of a class of students. Let me stipulate for the moment that for your sake all of the purely statistical considerations attendant to partialling out previous contributions of other teachers' "additions of value" to this year's teachers' addition of value have been resolved perfectly--above reproach; no statistician who understands mixed models, covariance adjustment and the like would question them. Let's just pretend that this is true.

Now imagine--and it should be no strain on one's imagination to do so-- that we have Teacher A and Teacher B and each has had the pretest (Sept) achievement status of their students impecabbly measured. But A has a class with average IQ of 115 and B has a class of average IQ 90. Let's suppose that A and B teach to the very limit of their abilities all year long and that in the eyes of God, they are equally talented teachers. We would surely expect that A's students will achieve much more on the posttest (June) than B's. Anyone would assume so; indeed, we would be shocked if it were not so.

Question: How does your system of measuring and adjusting and assigning numbers to teachers take these circumstances into account so that A and B emerge with equal "added value" ratings?

====================================================

Date: Sat, 10 Dec 1994 20:55:23 CST
From: William Sanders

The process is built on the formulation of the late C.R. Henderson, a Cornell animal breeder who was named a fellow in the American Statistical Association for his pioneering work in this area. Henderson's development of the concept of best linear unbiased prediction (BLUP), has been shown to be related to other shrinkage estimator concepts by David Harville at Iowa State and others (i.e. many bayesian concepts, kalman filtering from the engineering sciences, some of the hierarchical linear model concepts, etc.) . However, we have found that Henderson's formulations have tremendous computing advantages over some other equivalent alternatives. In fact, while I was a consultant to SAS Institute, Inc., as they were planning and implementing their rather new MIXED procedure, I strongly recommended that they use this formulation in their development and manuals -- a recommendation which was accepted.

1. fractured student records
2. teachers changing assignments
3. modes of instruction (team teaching, departmental instruction, self contained classrooms)
4. use of dissimilar indicator variables over time.
5. combinations of different quantities and qualities of information
etc., etc., etc., etc., .......

We do not calculate simple gains. For example, we use the whole observation vector for each child over all subjects and grades. In fact, we have a 'small' article later presented at an American Statistical Association meeting that demonstrates how this approach is superior to traditional multivariate approaches in that the whole observational vector is not lost due to one missing value for a variable.

{Some have worried} that "since the standardized tests are taken at least one, in some years almost two, months before the end of the year, gain scores conflate the effects of two different teachers". We have developed a process which we call the 'stacked block concept' which enables the partitioning of these effects. This concept is not totally dissimilar to the recovery of interblock information from incomplete block designs.

We have done a tremendous amount of computer simulation to evaluate just that concern, and we can demonstrate that this process is very robust against rather severe mispecification.

Question: But is there any way that you or someone on your staff can explain in some sort of intuitive way how the numbers you are crunching have anything to do with children's learning, and teachers' teaching -- and how you can know that for certain. I am not a statistician, but I do know something about the relationship between math and the physical world, and I know that just manipulating formuli will not always produce results that reflect what occurs in the physical world. How can you know that what you are measuring is what you think you are measuring? Especially since it seems unlikely we can directly test your results to see whether you have "it" right; since there is no "it" apart from your results that we can directly observe as verification.

Answer and further introduction: The Tennessee Value-Added Assessment system was developed on the basis of three assumptions: Fair and valid assessment is necessary to program improvement; such assessment models can be constructed; and it is reasonable to expect that children grow academically at a rate commensurate with their peers if effective instruction is taking place. None of these assumptions is original with us, but the way TVAAS addresses them is new.

Because it uses metrics that have long been suspect, at least in part due to the fact that previous methodology was incapable of rendering the necessary degree o fairness and validity, TVAAS is likewise looked upon with suspicion. That's to be expected. However, TVAAS in not plagued by the flaws of previous models, and it may be that a real paradigm shift is in order here. This is why:

1. TVAAS utilizes the scaled gain scores students make over time in order to model their learning patterns. In this way, it is possible to note when the normal pace of academic growth deviates. Obviously, this cannot be accomplished by figuring simple gains for each child. TVAAS uses the entire observational vector for each child across time. By using all the available data, TVAAS estimates the effects of educational entities without having to calculate the gain for individual kids. The records of students with fewer data points are weighted more lightly than those of students with complete sets in the calculation of educational effects, but the data of students with incomplete sets are not ignored.
The advantage of following growth over time is that the child serves as his or her own "control." Ability, race, and many other factors that have been impossible to partition from educational effects in the past are stable throughout the life of the child. By taking the child where we find her; by aggregating the gains of cohorts of students in school systems, schools, and classrooms; and by constructing covariance matrices among teachers in a school, schools in a system, and systems in a state, we can fairly and reliably attribute educational effects.
2. The scaled gain scores are derived from the norm-referenced items (the CTBS/4) of the Tennessee Comprehensive Assessment Program (TCAP). TCAP is administered state wide to all students in grades two through eight and ten. Scores from science, math, social studies, language arts, and reading furnish the data for TVAAS. The Education Improvement Act, the legislative package which identifies TVAAS as the model for educational accountability in Tennessee, mandates that it derive its data from "fresh, non-redundant, equivalent tests," insuring that "teaching to the test" is minimized and perhaps eventually extinguished due to lack of effectiveness.
Of course, there is a very vocal group that excoriates any use at all of standardized test data. We do not contend that the CTBS/4--or any other assessment tool, standardized or otherwise--can accurately depict the totality of a child's learning experience. However, we believe that the information it does provide is valuable. On the other hand, if better measures are found, TVAAS can easily utilize them in addition to or instead of the CTBS/4, so long as they provide linear measures with appropriate statistical properties.
3. TVAAS does not prescribe any particular model for effective instruction. Teachers are free to teach as they see fit. As long as their students make appropriate gains, teachers can assume that they are meeting the needs of their pupils, regardless of the method they choose.
4. TVAAS encourages appropriate instruction for all students. We all know that students enter the classroom with various levels of preparedness. It is not expected that all students will perform at the same level of competence or will achieve the same outcomes from a year of instruction. It is assumed, however, that each child will achieve the normed gain for his or her age and grade. Four pilot studies and three years of state wide reports have verified that this is a reasonable assumption since gain has been shown to be unrelated to level of ability, racial make-up of the student body, or socio-economic indicators such as percentage of students who receive reduced-price and free lunches. What this means is that if each child is taught according to his or her ability and level of preparedness, normal gains are to be expected. We think this attention to the needs of the individual child is crucial. Schools, systems, and teachers who do best under TVAAS are those who address the needs of all their students.
5. TVAAS can and does furnish far more than the sterile reports schools are used to receiving as a result of standardized tests. Each school in Tennessee will receive, beginning this year, a report broken down by student achievement level, indicating the gains of students in four to five divisions, ranging from low to high. Average gains for each achievement level in each tested subject and grade will be provided. This allows schools to pinpoint areas in which they are doing well and areas that need attention. For instance, a pattern of high gains for low achievers and low gains for high achievers might suggest that a school was doing well by its slower students but might not be offering enough enrichment or accelerated classes for its high achievers. Schools can also assess their programs in specific subject areas and grades. All of this is essential to effective program improvement.
6. An outgrowth of TVAAS is an enormous data base of educational information. Currently, there are over three million pieces of merged data in the data base. This unique resource allows for an unprecedented perspective on educational phenomena. It is already yielding significant findings and promises to continue to enlighten research on a tremendous range of educational issues for many years to come. The University of Tennessee Value-Added Research and Assessment Center hopes to form research collaboratives to investigate the findings deriving from the TVAAS data base as well as educational research questions originating from other sources.

{We wish to} encourage you to think about collaborative research projects. I don't think I'm stretching it when I say that the TVAAS data base is an educational treasure, totally unprecedented in educational research. Consider the fact that we are talking about a state wide population of students and teachers--not a sample. We want it to be used.

Question: You talked about the question I sent, but did not quite answer it, however. Remember I am not a statistician, so it may be difficult for you to answer what I am asking, but it is the question of how you know that your method of analysis really does isolate and "attribute educational effects." For example, suppose that second grade teachers teach material in such a way that second graders seem to make normal gains, but that in actuality the second grade teacher is teaching "surface tricks" or algorithms that make it difficult for a child, in third or fourth grade, to learn the material there because they were not given an important kind of understanding then. Would TVAAS likely point to second grade as the problem? Also, I can see that TVAAS can give EVIDENCE for problem educational areas; do you claim it is CONCLUSIVE evidence or just an indicator that some specific investigation may need to be done in what seems to be a problem place? And, from what you said, it makes it seem (correct me if I am wrong) that teachers are not held accountable for making the kinds of gains with culturally deprived kids that teachers would make in culturally advantaged areas -- unless some teachers in culturally deprived, similar areas also made much better gains. Did I understand this correctly? What I am getting at here is how the assessment is used with regard to evaluating rewarding/ punishing/weeding out teachers or improving teaching methods, etc.

On re-reading your post about TVAAS I noticed one particular sentence I want to ask you about, the one that says teachers, schools and systems who do best under TVAAS are those that address the needs of all their students. This is the kind of thing I was asking about --independent verification that the people TVAAS says are best ARE best. What led you to know that this correlation you mention exists? And would not that independent kind of verification work in place of, or better than, TVAAS? What I am getting at is how do you know that the teachers your methods say are doing the best ARE the teachers who are doing the best?

Further question (Gene Glass): I gather that TVAAS is a means of measuring what it is that a particular teacher contributes to the basic skills learning of a class of students. Let me stipulate for the moment that for your sake all of the purely statistical considerations attendant to partialling out previous contributions of other teachers' "additions of value" to this year's teachers' addition of value have been resolved perfectly--above reproach; no statistician who understands mixed models, covariance adjustment and the like would question them. Let's just pretend that this is true.

Now imagine--and it should be no strain on one's imagination to do so-- that we have Teacher A and Teacher B and each has had the pretest (Sept) achievement status of their students impeccably measured. But A has a class with average IQ of 115 and B has a class of average IQ 90. Let's suppose that A and B teach to the very limit of their abilities all year long and that in the eyes of God, they are equally talented teachers. We would surely expect that A's students will achieve much more on the posttest (June) than B's. Anyone would assume so; indeed, we would be shocked if it were not so.

So the question is: How does your system of measuring and adjusting and assigning numbers to teachers take these circumstances into account so that A and B emerge with equal "added value" ratings?

Answer: > You talked about the question I have, but did not quite answer it, >however. Remember I am not a statistician, so it may be difficult for you to >answer what I am asking, but it is the question of how you know that your >method of analysis really does isolate and "attribute educational effects." >For example, suppose that second grade teachers teach material in such a way >that second graders seem to make normal gains, but that in actuality the >second grade teacher is teaching "surface tricks" or algorithms that make it >difficult for a child, in third or fourth grade, to learn the material there >because they were not given an important kind of understanding then. Would >TVAAS likely point to second grade as the problem?

More than likely. Although it is true that there are algorithms that improve test taking skills to some degree, large gains over all subject areas attributable to such "tricks" have not been documented. As to whether we can isolate teacher effects, the answer is yes. I am not a statistician, either. I am a theoretician who interprets TVAAS to non-statisticians. Therefore, I will tell you what this model does without telling you how. If you want to know how, re-ask and I will restate.

TVAAS, in determining teacher effects, the first reports of which will be issued in 1995, uses the gains of at least three years of a teacher's students as data. This information is entered in a covariance structure that includes the performance of these students under their previous and subsequent teachers, as well. Since we can follow students over time, we can examine whether one teacher is injecting an artificial high. If a teacher's students make tremendous gains above those achieved under their previous teachers and then "fall off the table" the year after, we may suspect that something bogus is happening. The software is engineered to call such data to our attention. You have to understand that we are talking about whole classes of students for three or more consecutive years and that, usually, these students are disbursed to many subsequent teachers, not just one who is offering substandard instruction. At this point you may be getting your first glimpse at how revolutionary TVAAS is. These data are computed simultaneously for all students in grades two through eight in Tennessee, along with the covariant matrices for teacher, school, and system effects. As long as a child takes TCAP, we can follow him or her from school to school, system to system, and teacher to teacher.

> Also, I can see that TVAAS can give EVIDENCE for problem educational areas; > do you claim it is CONCLUSIVE evidence or just an indicator that some >specific investigation may need to be done in what seems to be a problem >place?

Excellent question, and the answer is that TVAAS supplies indicators that something is going well or ill in specific areas, at least in so far as certain subject-specific achievement in specific grades. TVAAS furnishes this information to schools, systems, and, next year, to teachers for grades 3-8 in science, social studies, math, language arts, and reading.

> And, from what you said, it makes it seem (correct me if I am wrong) that > teachers are not held accountable for making the kinds of gains with > culturally deprived kids that teachers would make in culturally advantaged > areas -- unless some teachers in culturally deprived, similar areas also > made much better gains. Did I understand this correctly?

Absolutely not. As I told you, where a kid starts does not influence the expected gain for that child. Deal is, all those schools in poorer areas where the students enter disadvantaged can look good for once if they achieve normal--not to mention extraordinary--gains. This is really important. Some of our inner city schools are, for the first time, able to demonstrate that they are bringing their kids along much faster than our suburban showcases. In the past, because scale scores labelled these kids "substandard," no matter how much progress they made, they always compared poorly to kids from enriched environments who started out miles ahead of them. Now, that's not an educational effect. That's environmental. If we're trying to see what schools are doing, we have to look at growth. Some of our higher scoring schools and systems are finding out that, unless their students make gains, even if they scale well above the norm, they don't do well under TVAAS because they're failing their kids by letting them coast. Our best systems achieve appropriate gains even in their very top scorers. TVAAS was developed to insure that EVERY child, regardless of ability, achieved academic gains normal for their peer group.

> What I am getting at here is how the assessment is used with regard to >evaluating rewarding/punishing/weeding out teachers or improving teaching >methods, etc.

Hmmm. Well, there's a complex question. In Tennessee, there are state and local evaluation models for performance evaluation of teachers. Those involve a combination of principals, supervisors, and state evaluators, depending on which cycle it is and what the teacher requests. We also have a Career Ladder which is a series of performance evaluations, dialogs augmented with "evidence" which is something akin to a portfolio, professional development initiatives, and leadership surveys as well as principal surveys. The EIA (Education Improvement Act) states that TVAAS cannot be the sole reason for dismissal of a teacher, and, while school and system reports are public record, teacher reports are not and are provided only to the teacher and "appropriate administrators," under the law. So far, and for as far as we can see, TVAAS is not part of a teacher's formal evaluation. It is, instead, a tool for self- and program evaluation.

You previously asked how we knew whether the teachers we were evaluating as best really were the best. Well, once again, the first teacher reports don't come out until next year. However, in pilot studies done in the early eighties, when principals were asked to predict whether their teachers would be in the top, middle or bottom third as assessed by the prototype of TVAAS, they were abe to predict the bottom third of teachers in all subjects; there was good discrimination between the top and average group of math teachers; but they had no clue as to who their top and average language arts teachers were. That's very anecdotal, to be sure. Interesting, but anecdotal. The reason I mention it is to say that TVAAS is an indicator. Professional knowledge, performance assessment, TVAAS--all of these are ways of knowing, and although they are correlated, I contend that their emphases are different. Performance assessment concentrates upon process. Professional knowledge of a teacher by an administrator may also be process of a different sort. TVAAS is product oriented. We look at whether the child learns--not at everything s/he learns, but at a portion that is assessed along the articulated curriculum, a portion each parent is entitled to expect an adequately instructed child will learn in the course of a year. Of course a child will learn more, and we can use that information in TVAAS, so long as it furnishes linear metrics with appropriate statistical properties (I've said that before). Right now, we've merged the data for all the kids that have taken ACT and/or PLAN in the last 3 years and are about to enter data on the first year of state wide writing assessment in grades 4, 8, and 10. We're also in the process of developing subject-specific high school tests which have to be on-line by 1999. We're going to be examining the relationships among all of these data to determine whether some offer information not available from others, what the intercorrelations might be, and, of course, we'll be on the look out for the unexpected, too. Wanna help?

Response to this answer:

I did want to comment quite favorably on one of the things you said TVAAS does --pointing out schools and districts that are "coasting" on what I would call culturally advantaged students, that is, those districts who have "good" students to work with because of their home environments, etc. and who don't add as much to their education as they could, or develop the potential those students have. I am in an area where three of the perennially top four school districts (in the state) based on SAT etc. scores do not, I believe, do much to develop their students potentials or cultural advantage nearly as much as they should and could. I have argued for nearly twenty years that the school district that finishes in one of the top five each year, but has a much "lower" socio-economic mix of students with much less culturally advantaged background is probably the best school district of the bunch because it does more with the students it gets. So I was really excited to see that is something you seek as well.

Rick Garlikov (dems042@uabdpo.dpo.uab.edu)

=========================================================================

Date: Mon, 12 Dec 1994 13:11:16 CST
Question from Sherman Dorn:

First, I want to divide TVAAS as a research tool from TVAAS as an evaluation tool. There is no doubt that a database with several million records is an invaluable resource. In addition, I can't see anything inherently wrong with the mixed-model methodology as described, and quite a bit good. As a tool of policy, I have some additional qualms about the effects of a high-stakes evaluation instrument as the data and law currently exists. But for now I'm only going to talk about TVAAS as research and statistics. Besides, I hold the state legislature responsible for policy. So, my thoughts:

1) I think there are several good reasons to enter the prior year's scale score as a covariate with the current year's score as the dependent (okay, the matrix of scores, with the matrix of scores from the prior years as covariates). First, as Gene Glass mentioned on EDPOLYAN's mailing list, a good bit of research suggests that kids who do well on an omnibus test like the CTBS would expect to have higher gains. Second, the cross-sectional nature of norming on the CTBS, at least for the data as it currently is in TVAAS, makes gains in scale scores ambiguous in meaning: (a) gains between grades in 1989 may conflate gains one might expect with differences in cohort experiences; (b) the norming in 1989 did not represent what one would expect from truly longitudinal cohorts -- due to differences in populations, as well as grade retention, for example, the grade 3 norming population is not the grade 2 norming population aged a year. The two are incomparable. Both of these issues change the scale scores in a linear fashion, so entering the prior years' scores as covariates solves the problem. (Solving Gene Glass' conundrum means that one assumes a linear relationship between first set of scores and second set of scores, but that's much more tenable than assuming an expected gain that's constant across the distribution of first sets of scores.)

2) There are a few potential problems I can identify with the procedure TVAAS uses to include information from kids who are not completely observed (e.g., not taking the language arts test in fourth grade). From what I gather -- and correct me if I'm wrong -- the gain score in an unobserved domain is imputed from a combination of information about the location of the child (system, school, and teacher) and information taken from the child's scores in observed areas (i.e., the deviation from the gain one would expect for a child in that particular system, school, and teacher), and then the score is given a lower weighting in the algorithm. First, I assume that this is multiple rather than single imputation. Beyond that, it appears that you're assuming that observation of a single score is random (and thus it is reasonable to assume that a child's deviation on an unobserved test score would be the same as on the child's observed test scores). Have you tried other models -- specifically, that school officials may try to steer kids into being absent on days when the test they're weakest on is being conducted? For kids in special education, that bias is almost certainly true and is explicit in the ability of school officials to exclude kids with disabilities from tests. Finally, I'm concerned about the implications of weighting kids' scores differently --this might end up making kids who school officials expect to do well being more important in the TVAAS system, and thus giving an incentive for teachers to teach to kids who are already doing well. If the weights are something like .95 versus 1.00, that's pretty minor, but more severe weighting might give some perverse incentives.

3) I would hope that the TVAAS staff would urge the state legislature to revise the state code regarding program and personnel evaluation to eliminate the exclusion of students with disabilities from the core of the system. I've already heard reports of this creating some perverse incentives for schools to ignore the needs of students with disabilities, and it's part of why I asked about retained students. TVAAS staff are not responsible for state policy, but they can go a long way toward convincing legislators of the need to revise the statute so individuals with disabilities are not ignored in program and personnel evaluation.

The first two items are really the gist of the matter, and I'm curious as to your thoughts about them.

******************************************************

Answer from TVAAS (Tennessee Value-added Assessment System)

I don't have time to go into all your points right now, but I would like to point out, as I did in my last message, that special ed kids are included in the assessment of schools and systems. We could easily include them in teacher assessment, as well, but are precluded by law from doing so.

A point that is very salient here: all of our research has shown that Dr. Glass, if he indeed attributes higher gains to higher scorers, is mistaken. Gain is not predicted by achievement level. Low scoring students are as likely as high scoring ones to achieve normal gains, above-normal gains, and below-normal gains. Eliminating potential low scorers is ineffective in manipulating gain scores for cohorts of students. That's one of the good things about TVAAS--your low scorers must be attended to AND ALSO you must provide challenging materials for your high scoring students.

Please remember, too, that we do not merely compare one year's scores with another. The scores kids make are modelled into a description of what learning is normal for each child over time. Although we don't model the actual learning curve for individuals, it is a good metaphor for what happens deep within the calculations. If we have, say, ninety kids who drop from their normal learning curve in various years but under the same teacher or, conversely, if they rise under a certain teacher and gain from there the next year, we know something about that teacher. (If they rise and then show very low gains the next year, we know something else about the teacher in question.) As you know, in Tennessee schools, it is highly unusual, except in very small schools, for students to remain together from grade to grade--they tend to be dispursed among teachers. Therefore, within the model, we must also take into consideration the interaction among students among teachers among schools among system over time and calculated simultaneously.

Further, I must question how much "strategic cheating" goes on, especially to the point of asking students to stay home from the test. We are able to spot such wholesale manipulation of the testing situation as well as many other forms of cheating, should they occur. The law specifies immediate and harsh penalties for those engaging in such practices. If anything, the EIA has cut down on such practices, and it is very rare that such anomolies occur according to our best knowledge and that of the SDOE's Department of Accountability.

After you read this stuff, would you please reask the questions I didn't get to? I'll be right here.

**************************************************

Sherman Dorn:
If the data on Tennessee kids shows equivalent expected gains for low-performing and higher-performing kids, that would be very interesting -- and important -- research. (I still prefer using prior scores as covariates, because of the non-equivalence of norming populations in different grades -- unless it makes no difference in the order of estimated effects.)

That finding may not hold, however, for kids in special education. Because their participation in TCAPs is at local discretion, you may be getting a selection effect. (You may not be getting a selection bias; you just don't know without 100% participation.) If some system(s) mandates participation for kids labeled LD, MR, or SED, that would be a possible way of examining the issue (assuming the system has some way of accommodating their needs appropriately and that accommodation is held fairly constant across the years for a specific child). (LD=learning disabled, MR=mental retardation, SED=severe emotional disorder.)

Besides, my concern is less with conscious manipulation than the fact that high-stakes testing can feed people's prejudices and lead to perverse incentives regarding students with disabilities. Because of the local decisions about testing participation for individuals with disabilities, teachers and building principals may write off some kids in *hopes* of improving test scores for others. (The hope doesn't even have to be rational.) Assistant Commissioner Cannon acknowledged this week in a public forum that he's heard stories of this type from across the state; I've heard the same thing locally from special ed teachers. It's very disconcerting when in a group of kids you know, the vast majority were missing at least one TCAP score in reading or math -- when this test is the major instrument of personnel evaluation. Without their mandatory participation in *some* form of assessment, their needs will come after those of nondisabled students; it's just the incentives their exclusion (either mandatory or discretionary) creates.

But, then again, this issue of incentives is a policy matter.

*************************************

TVAAS:
Whether special ed student are required to take the test is something over which we have no control. As I said before, their gains are, by law, a factor in the determination of school and school system effectiveness, so it is certainly in the best interest of principals and superintendents to meet the needs of these students.

I can do nothing whatever about superstition and wrong-headedness of individuals. I can only hope, and with good reason, I think, that systems such as TVAAS will provide an incentive for real improvement for all children. That is the basis upon which it was created.

Mr. Cannon may have "heard of" such instances. We have all "heard" of a lot of things, some of which have basis in reality and some of which don't. Document such instances and they will be examined by the Department of School Accountability. We want such things stopped as much as you do.

As for finding gains unrelated to level of achievement, yes, that is an interesting finding, isn't it. And our n's are huge. We invite and encourage use of the TVAAS database for educational research. Perhaps you have a point you would like to examine?

************************************************

(The following message also came to me off-list from someone else, and it relates to this matter. I have asked TVAAS for clarification, since I am confused about what the law or policy is regarding special ed students' taking, or being exempted from taking, the test that TVAAS uses to make its evaluations. Rick Garlikov)

I read your posting about TVAAS with great interest and have followed up to get some information about students with disabilities in the system. I learned the following.

The Reform Act provided that *all* students with disabilities were to be exempted from testing in the TVAAS. That has been changed to allow local option as to whether or not they are included. There are no consistent criteria applied to make that decision - it is up to the local school and the assessment team. (The TN special education program includes the gifted and they are not exempted from testing under the Act).

If students with disabilities are included in the testing, there will be 2 different reports issued - one with their scores included and one without.

I understand the TVAAS is causing a lot of consternation especially among teachers. As I understand it, the first year of the system, the *Report Card* was issued only for the school systems and in year two, it included data for each school in each system. This is now year three and the data will be specific to each teacher in the system as well at the end of this year.

I look forward to finding out more about this topic in your future postings. ***************************************************

Rick Garlikov (dems042@uabdpo.dpo.uab.edu) ========================================================================= ========================================================================= Date: Tue, 13 Dec 1994 06:49:48 -0600 From: SHERMAN DORN

I can clarify somewhat the situation regarding students with disabilities. The 1992 law establishing TVAAS says nothing about students with disabilities except in the section on "estimates of teacher effects" -- i.e., reports on individual teachers. *There* the legislature excluded the scores explicitly from the calculation of teacher effects.

According to the Tennessee Value-Added and Research Center, they inclu

de any tests from individuals with disabilities in the calculation of school and school system effects. That is, if a person took a test. However, it is up to the discretion of local officials as to whether individual students with disabilities will take the battery of standardized tests that are the basis of the Tennessee Value-Added Assessment System. Tennessee only has a limited number of accommodations possible in the administration of the TCAPs. In part because of those limited accommodations, in part because the TCAPs are a VERY LONG battery of tests (schools only administer one part per day), in part because of real or imagined fears on the part of local officials, many *many* students with disabilities either do not take the tests at all or only take a few sections. (I could call the Division of Accountability to ask about the actual numbers, though it may take a while for me to get around to it during the workday.)

In addition, as I wrote to the Tennessee Value-Added and Research Center, the way they explain their imputation procedure it looks as though they assume all missing scores are missing at random, and then they weight the missing scores less than observed scores. I think that a random assumption is poor for students with disabilities, and that (depending on the weighting factors) lower weights for the missing scores *still* gives incentives for local officials to exclude individuals from some tests.

(By the way, there is a group which does research precisely on the inclusion of students with disabilities in assessment systems. It's the National Center for Educational Outcomes, located in the Dept of Special Education at the University of Minnesota. They have a slew of reports available through ERIC.)

Sherman Dorn =========================================================================

Date: Tue, 13 Dec 1994 20:08:35 CST From: Rick Garlikov *************************************************************************

Regarding TVAAS and Students with disabilities: From Sherman Dorn:

According to the Tennessee Value-Added and Research Center, they include any tests from individuals with disabilities in the calculation of school and school system effects. That is, if a person took a test. However, it is up to the discretion of local officials as to whether individual students with disabilities will take the battery of standardized tests that are the basis of the Tennessee Value-Added Assessment System. Tennessee only has a limited number of accommodations possible in the administration of the TCAPs. In part because of those limited accommodations, in part because the TCAPs are a VERY LONG battery of tests (schools only administer one part per day), in part because of real or imagined fears on the part of local officials, many *many* students with disabilities either do not take the tests at all or only take a few sections. (I could call the Division of Accountability to ask about the actual numbers, though it may take a while for me to get around to it during the workday.)

TVAAS:

As for who gets tested among special ed students, I really don't know, and I'm going to have to seek that information from State Department sources. As I've said before, the EIA says that special ed students are excluded from the TVAAS assessment of individual teachers, but their scores are included in the assessment of schools and school systems. I checked with John Schneider who works with the raw data, and he says that the data we receive distinguishes special ed students only by the time they spend in special ed classes per week. These students fall into one of four groups depending upon the amount of time they spend in special ed., but we don't use that data because whether they are included or not is a question of "any" or "none," not "to what degree." Therefore, gifted are included or excluded along with the rest of special ed kids since they are not identified as gifted on the data collection sheets; and all special ed, regardless of the hours are treated in the same manner. =========================================================================

Date: Thu, 15 Dec 1994 19:32:19 CST From: Rick Garlikov Subject: Re: TVAAS #3

From TVAAS:

I need to answer a couple of points I haven't addressed in the replies below.

On Tue, 13 Dec 1994, Rick Garlikov wrote: > ************************************************************************* > Regarding TVAAS and Students with disabilities: > >From Sherman Dorn: > I can clarify somewhat the situation regarding students with disabilities. The > 1992 law establishing TVAAS says nothing about students with > disabilities except in the section on "estimates of teacher effects" -- > i.e., reports on individual teachers. *There* the legislature excluded the > scores explicitly from the calculation of teacher effects. >

> According to the Tennessee Value-Added and Research Center, they > include any tests from individuals with disabilities in the calculation of > school and school system effects. That is, if a person took a test. > However, it is up to the discretion of local officials as to whether > individual students with disabilities will take the battery of > standardized tests that are the basis of the Tennessee Value-Added > Assessment System. Tennessee only has a limited number of > accommodations possible in the administration of the TCAPs. In part > because of those limited accommodations, in part because the TCAPs are > a VERY LONG battery of tests (schools only administer one part per > day), in part because of real or imagined fears on the part of > local officials, many *many* students with disabilities either do > not take the tests at all or only take a few sections. (I could > call the Division of Accountability to ask about the actual numbers, > though it may take a while for me to get around to it during the > workday.)

I spoke to State Testing today about the administration of TCAP to special ed students. They said that whether a special ed student is required to take the test is not based on the decision of "officials," as such. In other words, special ed testing is not decided on a group basis. Instead, whether or not a special ed student is tested is a decision based on the recommendations of that student's M-team, a group composed of the student, his or her parent or guardian, the student's teachers (some or all), guidance and/or social/psychological support personnel, and, sometimes, an administrator. This team also decides whether any modifications in the testing need to be implemented such as individual administration, provision of large-type editions, provision of a reader, etc.

> > In addition, as I wrote to the Tennessee Value-Added and Research > Center, the way they explain their imputation procedure it looks as > though they assume all missing scores are missing at random, and then > they weight the missing scores less than observed scores. > I think that a random assumption is poor for students with > disabilities, and that (depending on the weighting factors) lower > weights for the missing scores *still* gives incentives for > local officials to exclude individuals from some tests.

Not at all. If a student is missing all scores, that student doesn't appear in the computations. If we have some data on a student and he or she does not take the test for some reason in some year, we do not assume a "generic" score. The missing data is specific to that child and is a projection based upon past performance. Furthermore, as subsequent data are collected, the "missing" score is modified to be a better estimation of what *that* student would have scored and all computations incorporating that score are also modified to reflect the new estimate.

=========================================================================

Date: Fri, 16 Dec 1994 06:56:25 -0600 From: SHERMAN DORN

Rick Garlikov asks:

>Why does TVAAS use (or bother to make) projected scores at all? Why not >just say something like "With 91% of the students taking the exam, the >scores are ....." What is the purpose of projected scores? And don't >projected scores defeat the purpose of taking the exam?

There are two general reasons why one would impute scores for any statistical purpose. (Impute means to plug in numbers for the missing values.) One is technical: you are underestimating the variance of a parameter if you only rely on complete records. The other is common-sense: using only complete-data records does not give the whole picture. You may be getting a biased picture by only including records of individuals with all tests.

As I understand it, the proper way to do imputation is to decide on a few a priori models of censoring (or the pattern in which data is missing) and then test each of those models through a process called multiple imputation. I'm going to ignore multiple imputation and only talk about the models.

Some models of missing values are essentially random: in the case of test scores, you may reasonably assume that some kids are going to be sick on the day of the test, and that such illnesses will be distributed randomly. On the other hand, some patterns are *not* random, and that is what I believe to be the case with individuals with disabilities. I am fairly sure that, for students in special education, they are much less likely to take a subject test (even with accommodations) if they are performing very low in that subject.

Now, what are the models that TVAAS uses? According to the article by Sanders and Horn in the Journal of Personnel Evaluation in Education (1994), the model for all student scores is partitioned into school system effect, school effect, teacher effect, and child deviation from the other effects (and maybe an error term; I'm at home while I type this). As I read the article, the imputation model plugs in a score for a particular subject that is a combination of school system effect, school effect, and teacher effect for *that* subject and child deviation *on other subjects and other years* (i.e., observable data). That seems to me to be a random model of censoring; it's fine if the child's deviation is randomly distributed across subject areas; it is also fine if missing values are randomly distributed (and thus you wouldn't expect the deviation for any missing value to be greater or worse than deviations for observed values).

However, if my supposition is right, it is a poor model of censoring for individuals with disabilities, because the child's deviation is likely to be nonrandomly distributed, and you should expect much lower values for deviations in the missing scores. As I understand it, the TVAAS imputation model could be overestimating the scores for individuals with disabilities.

Sherman Dorn

========================================================================= =========================================================================

Date: Fri, 16 Dec 1994 07:41:22 -0600 From: SHERMAN DORN

The Tennessee Value-Added and Research Center writes:

>I spoke to State Testing today about the administration of TCAP to >special ed students. They said that whether a special ed student is >required to take the test is not based on the decision of "officials," as >such. In other words, special ed testing is not decided on a group >basis. Instead, whether or not a special ed student is tested is a >decision based on the recommendations of that student's M-team, ...

I stand corrected. I suspect that in some cases, while an individualized education plan (or IEP) decided by an M-team may be the document that *should* govern the taking of tests, parents and guardians may be pressured to agree to certain testing omissions. (IEP meetings are well-known in special education to be frequently concerned with compliance with federal law and less with good programming.) Would it be possible for the Tennessee Value-Added and Research Center to flag schools or school systems where the pattern of test score omissions seems particularly out of line (and this needs to be done by hours of service)? That would be a valuable service.

>> I think that a random assumption is poor for students with >> disabilities, and that (depending on the weighting factors) lower >> weights for the missing scores *still* gives incentives for >> local officials to exclude individuals from some tests.

>Not at all. If a student is missing all scores, that student >doesn't appear in the computations. If we have some data on a student >and he or she does not take the test for some reason in some year, we do >not assume a "generic" score. The missing data is specific to that child >and is a projection based upon past performance.

As I describe in my response to Rick, the notion of random imputation is not isolated to plugging in the mean value. Technically, I believe that TVAAS uses an "ignorable response model," which I think is inappropriate for individuals with disabilities.

=========================================================================

Date: Fri, 16 Dec 1994 17:16:29 EST From: Anne Louise Pemberton

Sherman, I find the problem with the disabled far less disturbing than you fear AS LONG AS the assigned scores have some basis on a/some former scores for the individual, and that efforts are made to arrange for "make-up tests" for missed tests if/when an avoidance pattern emerges on an individual student. ...

What concerns me is the application of the data is "rating" schools and especially individual teachers. If the tests are only administered every 3 years rather than annually, you are measuring ONLY the aggregate of the 3 teachers. I'm uncomfortable with the idea that you can magically/mathematically deduce exactly what each of the 3 accomplished individually. Perhaps I would feel more comfortable if there was additional data input from the teachers themselves, and others who physically observe the daily goings on. Things like noisy mechanical equipment, insufficient textbooks or supplies, room too hot/cold, dark/bright, etc. etc likewise limit the options a teacher has in maximizing her/his effort. Are these factors rated and reported? (Factoring out is still working with "unproven" factoring).

I'm not a statistician, but wonder if we have instruments and measurements for physical plants at the individual classroom level. Is it possible to rate an instructional facility? Is TVAAS doing it now? Should they be? Would a line on the report say, for instance: Mrs. Sunshine, Rm 323, Tiny Elementary.1993 Student gain .79 grade/age level, in a 1.38 physical facility.

=========================================================================

Date: Fri, 16 Dec 1994 16:58:58 CST From: Rick Garlikov

Regarding the question of why have projections about missing test scores:

TVAAS: That's a very important question, so I asked Bill to frame an answer. I am forwarding it to you, having added only the meanings of BLUE and BLUP for the benefit of the non-statisticians.

---------- Forwarded message ----------

One of the major advantages of the mixed model process is that it solves the 'fractured record' problem.

All students do not have complete test records for various reasons: illness, absenteeism, movement from the state etc., etc.. To complete longitudnal analyses, only those students with complete records would be included if traditional multivariate statistical processes were deployed. However, with the mixed model process all student records are included, yet still provide BLUE* of the fixed effects and BLUP* of the random effects. To achieve this, the prediction and inclusion of the missing values of the record for each student is NOT necessary. Rather BLUE and BLUP come directly from the solution of the mixed model equations. Perhaps what has generated some confusion on this point are our teaching examples which we have presented in some of our short courses. In these examples, we asked attendees to visual a situation in which we would use all other information to predict the missing value for each student, then 'plug in' those predictions with appropriate weights to obtain a conceptual approximation to BLUE.

As to Sherman's concern of differential weighting of special ed. students, even if some systems are 'playing' games, then the effect would be observed on the estimate of means, and should have trivial effect on the estimated gains for schools, systems, and teachers.

BLUE: Best linear unbiased estimator BLUP: Best linear unbiased predictor

=========================================================================

Date: Sat, 17 Dec 1994 09:10:28 CST> From: Rick Garlikov

It seems to me that Tennessee teachers probably would be worried about at least four things regarding the Tennessee Value-added Assessment System: (1) accuracy of the assessments, (2) administrative uses of the assessments, (3) meaning of the reports issued, and (4) significance of what is tested.

With regard to (1), the point that Sherman makes about projections, and my earlier question of independent verification of the statistical procedures used, it seems to me a test is readily available: withhold the grading of a randomly selected significant sample of students who DID take the test, make the projections TVAAS thinks they can make to whatever degree of accuracy about what those test scores probably are, then grade the tests and see how close the projections are. This would be an empirical test of a statistical methodology.

Regarding (3),
I would like to see sample reports for districts, schools, individual teachers, to see what kinds of things TVAAS says -- i.e., what the reports actually look like. I am interested in the intelligibility of the reports and what sort of form they take. If you could supply examples of a report, please, or some sample passages from a report.
Rick Garlikov (dems042@uabdpo.dpo.uab.edu)

=========================================================================
=========================================================================

Date: Mon, 19 Dec 1994 17:25:09 +0000
From: Harvey Goldstein

I have come in on what I gather is the tail end of a discussion of missing data in the analysis of TVAA system data to produce estimates of school effects. Apologies therefore if the issue has been discussed already, and also because I'm from a different educational system but one where we have had quite a lot of debate about value added analysis using longitudinal data.

I have two questions. The first refers to Rick Garlikov's message of Dec 16th which I don't really understand. If you are missing a test score which is the dependent variable in the particular model being used then the traditional procedure is to omit the case, here presumably the student. This is somewhat inefficient, however, (and ignoring the issue for now about bias) and there are a range of imputation procedures (e.g. multiple imputation a la Rubin etc) which attempt to make use of whatever is available. The same can be done for the predictor variables when missing. I do NOT think, however, that this is a standard by-product of the mixed model. What is a by-product of that model is when you have what is often referred to as a repeated measures design where response measurements on some occasions are missing...then a mixed model approach, as apposed to the traditional multivariate one, allows you to ignore just the missing occasions without throwing out each individual subject. As I understand it, however, the value added model is not a repeated measures one, but an analysis of 'outcome' scores, adjusting for prior achievement...or have I missed something important.

A second question....in the UK the value added debate has been looking at problems with the sampling errors (standard errors) of value added gain scores...it turns out that these are typically so large that you cannot make any statistically significant comparisons between most of your schools...only those at opposite extremes of a ranking. Is this also the case in Tenessee? If so what do you do about it when reporting?

=========================================================================

Date: Mon, 19 Dec 1994 17:45:47 -0500
From: Greg Camilli

I think that we probably aren't on the tail end of the debate on missing values. Regarding the TVAA procedures, we may be just beginning.

I have a few questions (not related to missing values). First, it was indicated earlier that the norm-referenced items used by TCAP are somehow related to the CTBS/4. In this regard, I'm wondering if items are sampled from the CTBS, or whether new items are being written at every assessment. The latter is suggested by the phrase "fresh, non-redundant, equivalent tests." My second question is regarding the CTBS scores per se. A number of different metrics are available from CTB: is TVAA using the IRT (developmental) score scale? Finally, there is the suggestion that the CTBS/4 scores are linear measures with appropriate statistical properties. I'm wondering how this was established. Is the contention here that the CTBS scale is known to be a linear metric? (More to follow after I receive the answer to this question.)

Perhaps I could also ask if my interpretation of the missing data procedure is correct. I understand that multiple imputation does not affect the values of "effect" coefficients. The models and methods used to obtain these bypass the estimation of individual scores. However, multiple imputation increases the standard errors of the estimated coefficients -- this simply refects the notion that with less than complete data, less is known about the parameter. Thus, multiple imputation is a post hoc adjustment to estimation. However, with multiple imputation one can always produce a "complete" set of data (where imputed values have replaced missing values) to expedite reporting or secondary analyses. (Please do not hesitate to correct errors in this formulation.)

=========================================================================

Date: Mon, 19 Dec 1994 19:39:31 -0600
From: Sherman Dorn

Harvey Goldstein writes:

>[An explanation of the difference between multivariate imputation a la Rubin and repeated measures >designs.]

Thanks -- no, no one has yet mentioned this. I will admit being relatively ignorant about repeated measures designs. I suspect that when Bill Sanders weighs in, TVAAS will be more akin to repeated measures.

>A second question....in the UK the value added debate has been looking at > problems with the >sampling errors (standard errors) of value added gain scores...it turns out that > these are typically so large that you cannot make any statistically significant > comparisons between most >of your schools...only those at opposite extremes of a ranking. Is this also the > case in Tenessee? If so what do you do about it when reporting?

I don't know if the Tennessee Value-Added and Research Center noted the standard errors when talking to the press: I know that the Nashville papers did NOT report either standard errors nor took account of that when reporting scores publicly.

In the 1983 report co-authored by Sanders on a feasibility study (using a forerunner of the TVAAS model) in Knox County, the vast majority of teachers fell within two times the standard error of the median, in all three grades and all subjects tested. In that case, it truly seemed to be true that the gain score methodology only distinguished the very worst and very best teachers from the mass of teachers.

=========================================================================

Date: Mon, 19 Dec 1994 19:58:04 CST
From: Rick Garlikov

In response to Anne Pemberton who thought the TCAP testing schedule was once every three years. No, it is every year, in the spring. TVAAS incorporates three years of data for all its assessments.

=========================================================================

Date: Tue, 20 Dec 1994 07:04:59 -0600
From: Sherman Dorn

Please accept my apologies for the prior post: I was unaware that the mailer would turn the attachment into gibberish.

The following are my thoughts about TVAAS and accountability after reading the articles and manuscripts available about it and after a lengthy exchange with a staff member of the Tennessee Value-Added and Research Center. The following are largely comments about the use of TVAAS for policy purposes (i.e., the evaluation of school personnel and programs). They are not comments about the research utility of the TVAAS database.

My concerns about TVAAS-as-evaluation focus around the following five issues:

The state legislature has confused the technocratic tool of TVAAS-as-evaluation with the fundamentally political task of program evaluation.
The state legislature locks program evaluation into a set of centrally-calculated, rather than grassroots-developed, statistics.
The state legislature locks educational program evaluation into summative, rather than formative, evaluation.
The creation of the TVAAS locks the state into a testing program (the Comprehensive Test of Basic Skills) that gives little useful feedback to teachers.
The state legislature has discriminated against individuals with disabilities in the exclusion of students eligible for special education from the TVAAS estimates of teacher effects.

The following will discuss each issue in turn.

The creation of TVAAS at the heart of educational evaluation demonstrates the legislature's confusion about the fundamentals of evaluation.
When the Tennessee legislature enacted TVAAS, it tried to put a statistical model (value added assessment) at the heart of evaluation. Essentially, it instructs the state board of education to gauge whether schools meet the "required rate of progress" on TVAAS in order to be in compliance with state policies (i.e., that schools should have a "mean gain for each measurable academic subject within each grade greater than or equal to the gain of the national norms" [Tennessee Code Annotated $49-1-601 (b) ]). If schools don't meet this rate of progress, the commissioner of education can place school systems on probation and can, if the rate of progress doesn't come up to speed, remove local board members and the local superintendent.
Similarly, the legislature allows TVAAS estimates of teacher effects to be used for formal personnel evaluation after three years of data exist.
Now, while I think that statistics are an important part of policymaking when used correctly, I do *not* think that it is intelligent to put a single mechanism at the heart of such decisions. Fundamentally, decisions about competence in administration (and teaching) are political. I suspect that any such decisions in Tennessee will eventually become a matter of political judgment, but even in that case educators will then see TVAAS through cynical eyes: "They said it was the numbers that are all, but look at who they decided to remove." It is much better to avoid policies that can easily be seen as hypocritical. We have enough of that in education as it is. If we are to have evaluation of school systems and teachers, it must be the *educators* who see the evaluation as legitimate. Will TVAAS encourage that? I'm not so sure; I have yet to see evidence of teachers' seeing TVAAS as acceptable in Nashville area schools. One principal told me that TVAAS made her school look good, but she didn't think it was a legitimate tool of evaluation.
And I even have some doubts about whether the judgments will be judgments and not just rote decisions based on cut-offs. Case-in-point: the state established a few hundred thousand dollars which the state board was supposed to distribute this year to schools who met certain goals, one of which included a 10% "dropout rate" or less. One of the schools it gave an award to was Hume-Fogg Magnet School in Nashville, which had an amazingly low dropout rate. Guess what? Magnet schools don't have dropouts; they send problem students back to their home schools. So setting up the criterion led to an unfair advantage for magnet schools, but the board decided it still had to follow the guidelines because they had established the goals explicitly. This may happen with TVAAS and probation decisions as well.
In short, setting a statistical mechanism at the heart of evaluation ignores the basic political fundament of evaluation and policy decisions. In the long run, it is not wise.
TVAAS as established by the state locks Tennessee into centrally-created, rather than grassroots-developed, statistics.
Essentially, the Tennessee Value-Added and Research Center does the statistical analysis and presents the information to the state by itself. With some added "help" by the media. (The Nashville Tennessean this year, by the way, printed two versions of the rank-ordering and scores of schools this fall. They had to print it a second time because they screwed up a lot of the scores the first time. Imagine what that does to teachers reading the paper on a Saturday, and then reading their school's scores the next day as different.) These numbers are not derivable by educators themselves and -- more to the point -- I have yet to meet a teacher which understands the information presented to him or her. This may be because I have a limited sample, or maybe it's biased towards teachers who don't have the proper information, or maybe teachers sense my skepticism and wouldn't speak to me if they really like TVAAS, but I suspect that it's because the numbers are created far away from them. They're not tangible, in the form that a test graded by them, or in the form of answers correct, would be. Again, this erodes the legitimacy of evaluation in the eyes of teachers -- and it's a problem with all centrally-administered tests.
In addition, the central creation of statistics also erodes the possibility of grassroots political activism to reform schools. If the core of Tennessee's reform system was -- to create a thought experiment -- the mandatory publishing by each local system of certain facts, and manuals distributed to local organizations and citizens of how to calculate their own statistics on a home computer, how might educational reform be different? The voters might *vote out of office* board members who didn't move to change things.
In reality, statistics are produced both by central administrations and locally. There should be a healthy mix, however, and the weighting of things towards central administration in TVAAS is unhealthy.
Let me give a contrast between Chicago school reform and the removal of school officials in East St. Louis. In Chicago, a broad alliance of people, from businesses to civil rights activists, were fed up with schools. They used a mix of statistical sources -- from standardized test scores to a report on high school dropouts created by Aspira, Inc. (a Hispanic civil rights organization) to demand first the ouster of the superintendent and then the wholesale reform of schooling in Chicago. The state legislature made the *explicitly political* decision to reform and restructure Chicago's public schools through a statute. The result (as far as the legitimacy of the reforms is concerned): the primary opponents of the reform were the school principals and administrators, many of whom lost jobs (and for the principals, lifetime tenure) through the reforms.
In East St. Louis, the state department of education decided to remove school officials in this largely-African American city for incompetence. In contrast with Chicago, the move was opposed by LOTS of different groups locally, with a commonly-voiced suspicion that some racial politics were involved in the decision to remove officials in this particular school system. The lesson, I think, is that you need pressure from both grassroots activists and others allied with them to create successful movements for school reform in the political arena. And I don't think you can get that grassroots activism without grassroots creation of statistics. People have to *feel* they know what's going on in schools. Numbers created far from them, which they cannot replicate, aren't enough. If they only have such numbers, the debate can turn superficial very quickly, when opponents can only spout numbers at each other instead of talking about what the numbers mean. In Chicago, a local civil rights group did its own research and created its own, alternative, dropout statistics. That is a far cry from what the state legislature has done in Tennessee.
The legislature has locked Tennessee into an evaluative model that is summative rather than formative.
When a state uses annual tests as part of an accountability mechanism, it is creating a summative evaluation system, where you make a decision about a program as a whole [do we remove the superintendent], instead of formative, where you can use the information to make programmatic decisions [what do we do now?]. The problem with annual tests is that, when you get the information back, the kids are gone. The teachers can't do anything to change their teaching methods for that group of kids.
(Okay, I suppose annual tests are formative if you're only looking at the style of the individual teacher. It is *not* formative, however, if you're looking at adjustments for the benefit of specific children.)
In contrast, there *are* formative evaluation systems that let teachers make instructional decisions many times during the year. Let me contrast TVAAS (or the CTBS, on which it's based) with two systems I'm familiar with, one in math and one in reading. The following table describes how often students take the test, whether the test provides information about subskills, and how many tests are needed before you can make a teaching change inductively:
```
            Frequency   Subskills?   Minimum to change
TVAAS/TCAP  annually       no            unknown
Math        weekly         yes               4
Reading     2x week        no                8
 
```
In the case of the *truly* formative evaluation systems, you can make a teaching change every month based on substantive, real data. If you want to make a teaching change on a particular group of kids based on TCAPs, you can't. In order to change teaching methods, you either need information about subskills during the year that's comparative across time, or frequent tests, or (preferably) both.
And that leads me to ...
The state legislature has locked the state into a testing system with little useful feedback.
Put simply, the state currently uses the Comprehensive Test of Basic Skills as its standardized test (or part of it, the part that's used for TVAAS). The only breakdown you can get from the test is by subject area: scores in math computation, math application, reading, social sciences, science (and maybe language arts and writing, I forget exactly). But if you're a teacher and wondering if your students aren't picking up division, you're out of luck.
And I suspect that, for a variety of reasons, such tests (without any item information coming back to teachers) will continue to be used in Tennessee, because the state is very keen on test security (and it interprets that broadly to include item information on feedback) and because the test used for TVAAS will always be standardized.
The current TVAAS discriminates against individuals with disabilities.
This discrimination comes in two forms: one is the explicit exclusion of individuals with disabilities from the estimation of teacher effects (and that's in the statute); the other comes from the frequent exclusion of individuals with disabilities from taking the standardized tests at all, and the implicit exclusion of individuals with severe disabilities from tests that they really can't take (and thus are excluded from any measure that would describe their progress from year to year).
First, this is a direct denial of benefits to individuals wtih disabilities. The state has decided that evaluation is a critical function of state government, as important to educational improvement as activities like building construction and textbooks. Thus, we should analyze the investment in this evaluation system in the same way we do for building construction and textbooks. Would it be acceptable for the state to create buildings that were only used by nondisabled students? Or for the state to purchase instructional materials that are only used or usable by nondisabled students? Absolutely not. That would be a clear violation of section 504 of the Rehabilitation Act of 1973.
Second, the exclusions create an incentive to overidentify students as disabled. If a teacher is worried about a student's progress, and knows that the student's progress will be part of the teacher's evaluation unless the child is certified in special education, the exclusion creates an additional incentive to refer the child rather than make one more attempt to accommodate her or his needs. (This will probably happen for children who are, in the opinion of the teacher, borderline.)
Third, the exclusions create an incentive to remove disabled students from regular classrooms. If a teacher thinks that a student is somewhat disruptive, and whose scores are unimportant anyway for TVAAS (or who probably won't take the tests), he or she may push to get the child out of the classroom "so that the other children can learn." This sort of action would clearly violate the right of disabled children to be in the "least restrictive environment" where they can be accommodated. Again, this would probably happen to children who teachers think are borderline acceptable as far as presence in the classroom is concerned.
Fourth, the exclusions create an incentive for regular education teachers to ignore the instructional needs of disabled students. If students are either unlikely to take the test or are by fiat excluded from the teacher's TVAAS responsibilities, the teacher may very well concentrate on those children "who matter." This could easily be a self-fulfilling prophecy as far as students with mild disabilities are concerned: regular teacher thinks child probably won't take the tests, teacher doesn't pay attention to child, child learns so little and whose confidence drops so much that the teachers convince the parents to exclude the child from the tests at the end of the year.
As far as these issues go, the state has at least three possible ways to eliminate this discrimination: (a) it could eliminate standardized tests and the TVAAS; (b) the gain scores of students who don't take the test could be arbitrarily set to zero; or (c) the state could invest in a parallel system of teacher supervision, as heavily funded as TVAAS, for individuals with disabilities.
Note that the elimination of the statutory exclusion of individuals with disabilities from TVAAS would *not* eliminate the discriminatory effects. First, some children will never take the standardized tests because of the severity or nature of their disabilities. Second, the exclusion of some children from testing at all retains all the perverse incentives described above.

=========================================================================

Date: Wed, 4 Jan 1995 16:21:37 CST
From: Rick Garlikov

This is *MY* response to a previous post; it is NOT a response from TVAAS. Rick Garlikov

Sherman Dorn said, then subsequently explained and argued for, the following:

"My concerns about TVAAS-as-evaluation focus around the following five issues:

"1.  The state legislature has confused the technocratic tool of
TVAAS-as-evaluation with the fundamentally political task of
program evaluation.
"2.  The state legislature locks program evaluation into a set of
centrally-calculated, rather than grassroots-developed,
statistics.
"3.  The state legislature locks educational program evaluation
into summative, rather than formative, evaluation.
"4.  The creation of the TVAAS locks the state into a testing
program (the Comprehensive Test of Basic Skills) that gives
little useful feedback to teachers.
"5.  The state legislature has discriminated against individuals
with disabilities in the exclusion of students eligible for
special education from the TVAAS estimates of teacher effects."

Regarding (1): I am not happy with the language used to express what Sherman's good examples show he means; and I will return to that after discussing what I understand him to mean. First, I don't think the state has at this point confused the issue, though there is a reasonable danger, in ways Sherman points out, that they will. TVAAS has already said in a previous post that their results are supposed to be used as INDICATORS that something is very good, or very wrong, about a school/district/teacher, not conclusive evidence. That leaves room for the kind of evidence to be given about the merit of schools/districts/teachers which IS reasonable to educators -- a concern that Sherman expressed in explaining (1). Keeping this system from degenerating into a system of rote cut-offs based on TVAAS numbers without use of judgment is crucial. But surely it should be possible to prevent such degeneration; and safeguards should be put into place to make certain it does not occur.

Sherman says "If we are to have evaluation of school systems and teachers, it must be the *educators* who see the evaluation as legitimate." I want to make certain this is not ambiguous; while educators must see evaluations as legitimate, they should not be the only ones who do. If educators are the sole arbiters of whether they are doing a good job or not, and which of them is competent or not, without having to give reasons that make sense to anyone else, I fear we would end up with the same kinds of problems we have with the AMA policing its own --which they tend not to do very much unless there is the most egregious and publicized ineptness, negligence, or malpractice. My experience with "ethics" or "licensing" boards in governmental, professional, and business organizations is that they end up with a set of guidelines they follow that are superficially ethical or professional sounding, but which, at any meaningful level, frequently have very little to do with normal notions of competence or morality. In fact, much of the public wants to know why some administrators cannot seem to identify incompetent or excellent teachers when left to do so on their own; and why some teachers are able to get degrees at all. While the criteria for competence in education certainly need to be reasonable to educators, they should also be reasonable to people who are not educators.

The reason why I am unhappy with the language of (1) is that I don't think program evaluation is fundamentally a political decision, though I think programs often are judged on political grounds (i.e., power, perhaps coupled with ideology), and perhaps even more often appear to be. That is true when evaluations are not reasonable or when they do not respond to valid reasons given in opposition to them. But normally, when people act in good faith to evaluate a program, they are trying to be reasonable, not simply asserting power. The fact that some programs are judged purely on political (power and ideological) grounds, and are therefore not really being reasonably evaluated at all, should not lead us to say that the proper way to do evaluations is politically. The proper way to do evaluations is based on reasons and judgment, not power and blind ideology.

Moreover, I thought TVAAS was established to judge how well policy goals are being met, not how worthy those policies goals are. From what has been said so far, TVAAS tries to ascertain how well certain subject content areas --as prescribed outside of TVAAS-- are being taught, not whether those subject content areas are important or whether "knowing" them in certain ways is important. It IS important that legislators, administrators, and the public understand that these other issues ARE crucial, and that TVAAS evaluations are at best no more meaningful than what they are (legislated to be) based on. The media should never report mere numbers or ratings -- though they will.

Regarding (2), I think what Sherman says is true and important here, but MORESO if instead of speaking only about the calculation of "statistics", the operating principle was about the expression of "reasons". Although statistics can give evidence, not all evidence is statistical in nature. Plus, as Sherman points out, debates in terms of "numbers" alone, rather than the reasonable meaning of those numbers, can quickly turn superficial. In Sherman's examples from the two different school districts, it seems to me the important features were whether the reasons given by a centralized authority were agreed with by the local residents, and vice verse. Without some sort of consensus, the issues do become politicized and embattled, turning on a struggle for power, rather than reasonable and peaceful resolution. I think that this way of understanding (2) links it closely with (1).

Regarding (3), I may be misunderstanding the point of TVAAS, but I thought it WAS established to give summative evaluations about teaching competence/progress, not formative evaluations about improving instruction. So that even if TVAAS cannot be used to improve instruction or pinpoint students' specific learning difficulties, that is not a point against it. It is important to identify problems or good resources, even if that identification does not pinpoint a remedy for the problems. The profession should try to provide the remedy, I would presume, not TVAAS.

Teachers are free to give tests which allow formative evaluations and instructional corrections. There is no reason the state needs to be giving these tests for teachers. The mission of TVAAS seems to have nothing to do with this aspect of teaching, so I don't see that as an actual objection against TVAAS.

Regarding (4), Sherman says,
            "the state currently uses the Comprehensive
            Test of Basic Skills as its standardized test
            (or part of it, the part that's used for
            TVAAS).  The only breakdown you can get from
            the test is by subject area:  scores in math
            computation, math application, reading,
            social sciences, science (and maybe language
            arts and writing, I forget exactly).  But if
            you're a teacher and wondering if your
            students aren't picking up division, you're
            out of luck."

But again, the purpose of TVAAS, I presume, is to evaluate how well teachers/schools/districts have done that aspect of their jobs involved in teaching content of whatever sort is measured on the Comprehensive Test of Basic Skills. And surely, shouldn't a teacher be able to figure out whether his/her students are "picking up division" or not without the state's having to do that for him/her. I take it that TVAAS is not about helping teachers/schools/districts do their jobs, but about suggesting, in some attempted objective way, to everyone how well or poorly they have done it.

=========================================================================

Date: Wed, 4 Jan 1995 20:12:39 -0600
From: Sherman Dorn

> This is *MY* response to a previous post; it is NOT a response > from TVAAS. Rick Garlikov

And here I was, worried that no one had read that post.

Regarding my concern that TVAAS will be used as a rigid evaluation tool:

>First, I don't >think the state has at this point confused the issue, though >there is a reasonable danger, in ways Sherman points out, that >they will. TVAAS has already said in a previous post that their >results are supposed to be used as INDICATORS that something is >very good, or very wrong, about a school/district/teacher, not >conclusive evidence.

With respect to the Value Added and Research Center (VARC) staff member who posted that information, I beg to disagree, based on the law. The relevant sections of Tennessee's education code (49-1-601 through 49-1-610, in the 1994 Supplement to Tennessee Code Annotated) clearly says that systems which perform subpar for several years are subject to probation. The code does mention dropout rates (undefined) as another criterion, and pulling local officials is at the discretion of the state commissioner of education, but there is no doubt what is the central force here: TVAAS statistics.

To be fair to the incoming governor, a Nashville paper reported yesterday that he was intending not to follow the provisions of the code regarding the removal of local educational officials. Of course, with the reputation local papers have for absolute accuracy, I'll still wait and see what happens. (But see what happens if the report is true: the governor would be explicitly ignoring a provision of the education code. Great.)

>Sherman says "If we are to have evaluation of school systems and >teachers, it must be the *educators* who see the evaluation as >legitimate." I want to make certain this is not ambiguous; while >educators must see evaluations as legitimate, they should not be >the only ones who do.

Agreed. But this makes it very clear that evaluation is a political task, in its underlying assumptions if nothing else.

>The reason why I am unhappy with the language of (1) is that I >don't think program evaluation is fundamentally a political >decision, though I think programs often are judged on political >grounds (i.e., power, perhaps coupled with ideology), and perhaps >even more often appear to be.

The fact that actions are political does not make them bad. You can decide, for example, to fire Ronald Jones on the basis that Jones' lowest-performing students didn't improve their ability to read at all during ten years in his class; but that has at its basis the political judgment that a teacher must pay attention to lower-performing students. This decision may anger parents of higher-performing students who loved Jones' teaching and how their children did. That's a political decision, not a technocratic one.

>Moreover, I thought TVAAS was established to judge how well >policy goals are being met, not how worthy those policies goals >are.

My contention is that TVAAS is an abdication of the legislature's need to make policy goals. It presumes that, by the installation of TVAAS, Tennessee's schools will improve. A technocratic solution.

Moreover, my concern is that TVAAS freezes the state into its current set of tests; it will be very difficult to change evaluation goals at this point. TVAAS is a policy flywheel.

Regarding summative versus formative evaluation:

>I thought it WAS established to give summative evaluations about >teaching competence/progress, not formative evaluations about >improving instruction.

Sanders and Horn's article in the Journal of Personnel Evaluation Education claims that one advantage of TVAAS is its formative potential, and its instructional-methods neutrality.

>Teachers are free to give tests which allow formative evaluations >and instructional corrections. There is no reason the state >needs to be giving these tests for teachers. The mission of >TVAAS seems to have nothing to do with this aspect of teaching, >so I don't see that as an actual objection against TVAAS.

This is a very interesting claim. I hope I am not misinterpreting, but is Rick suggesting that the state should evaluate program effectiveness to the extent of removing officials and firing teachers, but not encourage them to use formative evaluation?

Moreover, I think the high-stakes nature of TVAAS will drive other forms of evaluation from consideration, as teachers focus on the annual tests in the spring. Besides, there is NO GUARANTEE that a specific form of formative evaluation, even if it is tied to the curriculum, will mean students will perform better on the tests used in TVAAS. So it is, in fact, unfair to tell teachers, "well, you go train yourself in formative evaluation, use it to the best of your abilities, and hope that the students then do better on our annual tests. Oh, and no, the evidence you accumulate in your classroom doesn't count as much as the TVAAS."

In addition, formative evaluation is alien to most teachers, as far as guiding instruction is concerned. Getting teachers to use formative evaluation would require a very different type of support structure, and quite a bit of money. It conflicts with TVAAS simply because it would compete with the VARC for funding. (And the next governor has promised he won't raise taxes.)

Regarding the usefulness of feedback of the tests (and my claim that TVAAS freezes the state into using these tests): >But again, the purpose of TVAAS, I presume, is to evaluate how >well teachers/schools/districts have done that aspect of their >jobs involved in teaching content of whatever sort is measured on >the Comprehensive Test of Basic Skills. And surely, shouldn't a >teacher be able to figure out whether his/her students are >"picking up division" or not without the state's having to do >that for him/her.

I'll defer here to William Webster, head of the evaluation unit in the Dallas schools, who told a CREATE audience a few years ago that one of the requirements of top-down evaluation of teachers was that teachers be provided with explicit, helpful feedback.

>I take it that TVAAS is not about helping >teachers/schools/districts do their jobs, but about suggesting, >in some attempted objective way, to everyone how well or poorly >they have done it.

This is precisely what it is, and I think Tennessee law gives an unwise emphasis to one type of evaluation instrument.

=========================================================================

Date: Wed, 4 Jan 1995 20:52:42 CST
From: Rick Garlikov

Here is TVAAS's response to Sherman Dorn's first post on the policy aspect of TVAAS.

----------------------------Original message----------------------------

In response to Sherman Dorn's most recent critiques of the Tennessee Value-Added Assessment System (TVAAS), the UT Value-Added Research and Assessment Center offers these remarks as a complement to those presented by Rick Garlikov:

On Tue, 20 Dec 1994, Sherman Dorn wrote:

> Please accept my apologies for the prior post:  I was unaware that
> the mailer would turn the attachment into gibberish.
>
> The following are my thoughts about TVAAS and accountability after
> reading the articles and manuscripts available about it and
> after a lengthy exchange with a staff member of the Tennessee
> Value-Added and Research Center.  The following are largely
> comments about the use of TVAAS for policy purposes (i.e., the
> evaluation of school personnel and programs).  They are not
> comments about the research utility of the TVAAS database.
>
> My concerns about TVAAS-as-evaluation focus around the following
> five issues:
>
> 1.  The state legislature has confused the technocratic tool of
> TVAAS-as-evaluation with the fundamentally political task of
> program evaluation.

TVAAS is a statistical model for program evaluation. It is not a "technocratic tool," as Dorn so colorfully phrases it. It was not developed by the State of Tennessee but by Dr. Bill Sanders, a statistician, for the purpose of addressing problems previously encountered in using student achievement data in educational assessment. The State of Tennessee adopted it as the model for such assessment because TVAAS was able to supply valid, reliable, unbiased data based on student gains, data the State thought was important. The Education Improvement Act (EIA) allows the use of TVAAS data as a *component* of the evaluation of educational entities--teachers, schools, and school systems. It specifically states that TVAAS may not be the only source of data in such evaluations. Teachers are further protected in several ways. First, since the law mandates the use of at least three years of data for a formal evaluation (although fewer years may be used for informational purposes), only tenured teachers will receive TVAAS reports that can be used for assessment purposes. Second, the EIA specifically states that teachers may not be dismissed exclusively on the basis of TVAAS data. Third, teachers are evaluated by several different models including performance assessment by local supervisors and principals. These evaluations are mandated by the State. Fourth, teachers may elect to be evaluated intensively to achieve advanced Career Ladder status. Career Ladder evaluations include dialogs, presentation of artifacts, and repeated classroom observations.

TVAAS data is currently available only for grades three through eight in the subject areas of science, social studies, math, reading, and language arts. This means that many teachers do not have the advantage of TVAAS data. Subject-matter specific tests for grades nine through twelve are now being developed by CTB/McGraw Hill in conjunction with committees of Tennessee secondary school teachers who are devising questions in their areas of expertise. However, it is unlikely that TVAAS will ever assess ALL teachers in ALL subjects. Therefore, it will never be the ONLY means of assessing teachers or schools. TVAAS merely provides measures of student academic gains, surely a useful component of any educational assessment system.

As for Dorn's projections of what "might" happen, we see no basis in reality for such dire prophecies. The link he attempts to establish between the problems associated with fairly rewarding schools for improvements in the dropout rate and TVAAS is nonexistent.

Finally, we disagree with Dorn on the political nature of assessment. Assessment should be a scientific process rather than a political one. Although setting the criteria for assessment is a political act, assessment itself must be scientific in orientation, centered on the real meanings of the most common of assessment terms--reliability, validity, and fairness.

> 2.  The state legislature locks program evaluation into a set of
> centrally-calculated, rather than grassroots-developed, statistics.

The State of Tennessee has a vested interest in monitoring the health of one of its largest and most important endeavors--the education of its children. It has chosen TVAAS as a means of carrying out its oversight responsibilities because TVAAS fairly estimates the academic growth of students, an indicator recognized as important by teachers, parents, local administrators, community leaders, and state officials, alike. Other indicators--dropout rate, attendance, promotion rate, and graduation rate--are also used by the state for assessment purposes. As stated above, a variety of methods are used state-wide for the evaluation of teachers.

TVAAS is a sophisticated statistical process that is beyond the capacity of many to grasp mathematically. However, TVAAS data is reported in a manner that is comprehensible to any educator willing to look at the reports and graphs supplied, with explanations, specifically for the purpose of rendering the data USEFUL. Judging by reports from many schools and systems across Tennessee, TVAAS data is now extensively used for curriculum planning and development.

> 3.  The state legislature locks educational program evaluation into >
summative, rather than formative, evaluation.

For further clarification of Mr. Dorn's point, we quote him from a paragraph in which he discusses this point, further on in this same composition. Mr Dorn writes

"When a state uses annual tests as part of an accountability mechanism, it is creating a summative evaluation system, where you make a decision about a program as a whole [do we remove the superintendent], instead of formative, where you cn use the information to make programmatic decisions [what do we do now?]. The problem with annual tests is that, when you get the information back, the kids are gone. The teachers can't do anything to change their teaching methods for that group of kids. (Okay, I suppose annual tests are formative if you're only looking at the style of the individual teacher. It is *not* formative, however, if you're looking at adjustments for the benefit of specific children.)"

First, we would direct those following this topic to Rick Garlikov's excellent discussion on this point. We add only a few comments:

The State of Tennessee considers TVAAS to be a model for both summative and formative assessment. Education is an ongoing effort, and the end of a school year is simply a point in a continuum. TVAAS is used, as stated under (2) above, to assess how well students are progressing under current practices. If they are making expected gains, we assume that the teacher, school, or system is providing effective instruction. If not, then adjustments should be made. When systems are doing very poorly indeed, they are placed under supervision and are more closely monitored by the state. While under probation, they are expected to make progress toward acceptable standards--a formative process directed by many indicators, one of which is student gain.

TVAAS provides breakdowns of student gain data to schools and school systems by school, grades, subjects, and achievement levels of student (this last, upon request from the Department of School Accountability). With all this information, schools and systems can easily pinpoint problems and successes and make specific policy decisions based upon this knowledge. The data provide guidance as to "What we do now."

The State of Tennessee leaves the day-to-day adjustments in teaching strategies to the classroom teacher. In the last four years, Tennessee has conducted a concentrated effort to deregulate its schools, placing great emphasis on site-based management and school-based decision-making. In dispensing with the rules and regulations that have, in the past, served to dictate how schools may operate, Tennessee has placed the responsibility for program development, allocation of resources, the organization of schools, teaching strategies, and many other things in the hands of local administrators. Because this is so, TVAAS was adopted to monitor the educational process, not the "style," as Dorn puts it. The "style" in which education is conducted is not prescribed. The process of education can be completely individualized. TVAAS merely assesses the value added, the product of educational efforts, in the grades and subjects for which we have appropriate linear measures. As Garlikov points out, it was never designed to be otherwise. And as we have repeatedly stated, no one assessment model will suffice for all legitimate assessment purposes.

> 4.  The creation of the TVAAS locks the state into a testing program
> (the Comprehensive Test of Basic Skills) that gives little
> useful feedback to teachers.

As we have repeatedly told Dorn, TVAAS does not lock the state into any testing program whatever. TVAAS can use any assessment data that provides appropriate linear measures. In other words, TVAAS requires scalable data because it assesses progress over time.

TCAP was in place before TVAAS was ever adopted, and state-wide annual tests have been used in Tennessee, as in other states, for years. TVAAS simply utilizes data in a new way, furnishing far more useful information to teachers and schools than was ever possible in the past.

> 5.  The state legislature has discriminated against individuals
> with disabilities in the exclusion of students eligible for special
> education from the TVAAS estimates of teacher effects.

Please see Dorn's discussion of this item, below.

First, Dorn refuses to acknowledge, as we have repeatedly told him, that low-achieving students are at least as likely to achieve appropriate gains as their higher-scoring classmates (based upon three years of state-wide data), so there is no logical reason to exclude them from the tests.

Second, we find it most disturbing that Dorn attributes such unethical behavior to Tennessee teachers as to ignore students in need of their tutelage and to connive to have their parents classify them as special education and to encourage them to avoid testing.

Third, we have suggested to Dorn that if he has any real knowledge of such occurances that he refer them to the Department of School Accountability. There is a mechanism (to use Dorn's terminology) in place to insure that special education and other students are not exploited in this manner.

Fourth, Dorn is obviously unaware that the funding for TVAAS is miniscule and that supervisory personnel are already available in every system for the purpose of overseeing teacher conduct and, in particular, special education programs. Perhaps he believes that they are possessed of the same iniquities he attributes to teachers and cannot be trusted with their appointed duties.

In contrast, it is our supposition that teachers and administrators are dedicated to the same goal that the State of Tennessee espouses: the improvement of education for all students in order to maximize the potential of each of them. We sincerely hope that TVAAS is a means by which this goal can be more readily achieved.

We appreciate the opportunity to respond to these issues.

=========================================================================

Date: Wed, 4 Jan 1995 23:12:35 -0600
From: Sherman Dorn

The UT Value-Added Research and Assessment Center wrote a lengthy response to my long comments several weeks back about the Tennessee Value-Added Assessment System. (Please accept my apologies for the incorrect name of the center in other posts.) I will try to confine my remarks to central issues and matter of potentially discriminatory effects on children with disabilities.

Regarding evaluation and my claim of its political assumptions:

>        we disagree with Dorn on the political nature of
>assessment.  Assessment should be a scientific process rather than a
>political one.  Although setting the criteria for assessment is a
>political act, assessment itself must be scientific in orientation,
>centered on the real meanings of the most common of assessment
>terms--reliability, validity, and fairness.

This is the core of our disagreement. TVAAS assumes that assessment should fundamentally be a scientific process. I assume that evaluation's political nature will outweigh any claim to positivistic science, *especially* if it's asserted as such.

(Note: this does not mean that I am anti-empirical, or that VARAC staff are against using other materials for evaluation. This is an argument about the orientation of evaluation.)

Regarding the origins of statistics:

>The State of Tennessee has a vested interest in monitoring the health of
>one of its largest and most important endeavors--the education of its
>children.

The question here is NOT whether the state should produce any statistics, but whether its creation of statistics should dominate the debate. I think it's much healthier for public debate to include statistics from various sources and levels of government and often from outside government as well. I think the Educational Improvement Act puts TVAAS into an overwhelmingly dominant position.

Re: my claim of potentially discriminatory effects on individuals with disabili

>so there is no logical reason to exclude them from the tests.

Information on low-achieving students as a group is NOT information about individuals with disabilities. Moreover, my concerns deal with the incentives for flawed, stressed educators to try to slough off some of their responsibility

>        Second, we find it most disturbing that Dorn attributes such
>unethical behavior to Tennessee teachers as to ignore students in need
>of their tutelage and to connive to have their parents classify them as
>special education and to encourage them to avoid testing.

Most unethical behavior in institutional settings occur not because of conscious and evil conniving but because the circumstances and work habits combine to create incentives for unethical behavior. Besides, the existence of accountability frameworks such as TVAAS presumes the irrationality and untrustworthiness of teachers (else why would we need to hold them accountable?).

>        Third, we have suggested to Dorn that if he has any real
>knowledge of such occurances that he refer them to the Department of
>School Accountability.  There is a mechanism (to use Dorn's
>terminology) in place to insure that special education and other students
>are not exploited in this manner.

Such evidence would have to be very clear; the hearsay testimony from a non-tenure track academic about the supposed intent to pressure parents into removing their children from testing would not cut it. However, I do not have to make a hard case of specific instances in order to make a pretty good case that Tennessee educators tend to exclude kids with disabilities from testing. According to the National Center on Educational Outcomes, fewer than 50% of students with disabilities in Tennessee participated in state testing in 1992. Some states have much, much higher participation rates. My conclusion: Tennessee schools just don't try hard to include students with disabilities in testing.

>        Fourth, Dorn is obviously unaware that the funding for TVAAS
>is miniscule

Compared to the entire education budget, certainly. However, there is NO such comparable accountability system with quantifiable tracking of student outcomes for children with disabilities; when looked at in that frame, individuals with disabilities receive the short end of this sytem that the state legislature has put at the heart of educational reform in Tennessee. If quantifiable tracking is good for nondisabled students, why doesn't a system exist that will accommodate the requirements of students with disabilities?

This is a question for the state legislature; I assume that VARAC staff would have no objections to such a parallel system that overlaps TVAAS, and that they have a great incentive for Tennesse to put in place preventive measures against discriminatory effects. Regardless of our differences over educational evaluation, I assume we can agree on this.

=========================================================================

Date: Wed, 4 Jan 1995 23:10:05 -0600
From: Alan Davis

I find that I agree with all of Sherman Dorn's criticisms of the TVAAS evaluation system. All evaluation is political, because the process and findings of evaluation affect power and resources. The interests and assumptions of some are reflected in the questions the system are designed to answer and the test data selected to answer them; the interests and assumptions of others (teachers, in particular) are not. The fact that the system seems to rely on objective evidence of learning and is statistically sophisticated does not change the essentially political nature of the process; by cloaking the enterprise in the garb of science, teachers may be doubly intimidated and policy makers impressed, but nonetheless you have here a summative evaluation conducted far from the site of local decision making that is likely, in my view, to have pernicious effects despite the very sincere intentions of its designers to be fair.

I am open to the possibility that TVAAS may be a valuable tool for research. I believe that good teaching is teaching that results in significant learning for a broad spectrum of students. At the elementary level, I will go so far as to agree that the sort of standardized multiple choice measures designed by CTB will detect improvements in reading comprehension and mathematical conceptual and problem solving understandings that most of us want for our children. I have used regression residuals from hierarchical linear models of learning outcome measures regressed on pretests and SES to select individual teachers to study in my own research on effective teaching. The main problem, I believe, is that the multiple choice measures used in Tennesee are at the outset poor proxies for the tasks we really want kids to be able to perform in school, and when we attach consequences for teachers around socres on these measures, very bad consequences are likely to result.

At the same time, students' performance on the tests may have consequences for teachers. At the very least, someone is forming judgments about what teachers are good and what teachers are not so good based upon their students' performances on these tests. Linda McNeil, Lorrie Shepard, Mary Lee Smith, and others have documented the behavior of teachers under these circumstances. They concentrate on raising the scores rather than teaching to the broader goals of instruction, and the strategies they hit upon to raise scores involve adjusting instruction to model the format, content, and mindset of the multiple choice tests, sometimes to the point of teaching actual passages and items.

The problem, which I do not believe can be avoided, is that the validity of tests is inseparable from the uses to which they are put. A test that may be valid as a research tool to discover effective teaching will lose its validity when teachers suspect that the outcome has consequences for themselves.

=========================================================================

Date: Fri, 6 Jan 1995 19:22:04 CST
From: Rick Garlikov
Subject: TVAAS/Harvey Goldstein

Harvey Goldstein stated on Dec. 19, 1994, "..it turns out that these [standard errors] are typically so large that you cannot make any statistically significant comparisons between most of your schools...only those at opposite extremes of a ranking. Is this also the case in Tennessee? If so what do you do about it when reporting?"

From TVAAS:
Below are listed the mean gains for math with their standard errors for schools within one of the larger school systems in Tennessee. These means are three year averages and were calculated from the TVAAS mixed model process. This should give an idea of the sensitivity of the process.

            INTERMEDIATE SCHOOLS
 
        GRADE   SCHOOL    MEAN   STD. ERR
                RANK      GAIN
 
           3        1   71.596     4.931
           3        2   71.205     3.714
           3        3   67.038     2.624
           3        4   66.876     3.641
           3        5   62.734     3.427
           3        6   62.574     6.906
           3        7    62.14      5.17
           3        8   62.032     3.628
           3        9   61.713     3.534
           3       10   59.062      4.04
           3       11   58.096     3.337
           3       12   57.262     4.849
           3       13    57.22     2.321
           3       14   56.909     5.552
           3       15   55.062     3.866
           3       16   54.775     3.301
           3       17     53.8       2.4
           3       18   53.553     4.459
           3       19   53.452     2.813
           3       20    52.06     2.074
           3       21    51.24     1.411
           3       22   50.853     2.909
           3       23   49.298     3.716
           3       24   48.654     3.134
           3       25   48.269     4.603
           3       26   47.889     3.141
           3       27   47.884     3.714
           3       28   47.864     2.931
           3       29   47.574     5.398
           3       30   46.382      3.73
           3       31   46.281     1.664
           3       32   44.722     3.622
           3       33   43.843     2.533
           3       34   43.272     3.417
           3       35   42.965      2.47
           3       36   42.907     2.443
           3       37   41.844     2.803
           3       38   41.645     2.772
           3       39   41.018     1.859
           3       40   40.079     2.649
           3       41   39.492     2.958
           3       42   38.662     4.633
           3       43   37.784     2.807
           3       44    35.84     3.517
           3       45   33.597     4.541
           3       46    32.46     5.394
           3       47   31.586     3.868
           3       48   30.907     3.329
 
 
                 MIDDLE SCHOOLS
 
           7        1   25.894     0.971
           7        2   23.758     0.861
           7        3   23.134      1.17
           7        4   22.988     1.275
           7        5   22.738     1.347
           7        6    18.51     1.215
           7        7   18.185      0.99
           7        8   16.831     1.071
           7        9   16.268     1.075
           7       10   15.843     1.374
           7       11   15.394     1.055
           7       12   15.357     1.077

Additionally, there was a question concerning how TVAAS deals with missing scores. We will write a longer respose to this question later. But, briefly, it is more like the analysis of repeated measures. However, we do include all of the scores among subject-grade combinations. This is certainly sufficient and avoids the issues and problems associated with imputations.
Date: Sat, 7 Jan 1995 04:43:10 CST

Rick Garlikov:
Sherman Dorn, Alan Davis, and Mark Fetler all make important and cogent points about the nature of assessment and assessment tools. But I am arguing that characterizing these points by the term "political" is a mistake, and it is a mistake that severely weakens their message and the force behind that message.

There should be no doubt that assessment is not purely a matter of science, and that it is both a value-laden enterprise, and one subject to misleading scientific/mathematic measurements and the use of those measurements. But these issues are not "political" issues. ...

By using the word "political" to characterize the nature of the technical aspects of TVAAS, or any other scientific/mathematical/statistical sorts of assessment tools/indicators, you undermine your credibility with a government and public who believes, and I think rightly so, that surely there are some sorts of objectively recognizable contributions that teachers make or don't make to students' educations. To claim that all teacher evaluation, by its very nature, is "political" SEEMS to make the claim that there are no objective standards for discussing/judging teaching; and it makes it sound as though you are arguing that any judgment of teaching ability is worthless and unnecessary -- that no teacher is REALLY good or bad; they are all just popular or unpopular, so there is no fair way to judge any of them.

It seems to me that what you want to claim, especially if you go public, or lobby legislators with your points, is that there are many aspects, not only to good teaching, but even to the teaching of content, that are not reflected in the tests that TVAAS analyzes -- for the reasons that you give, and that the studies you cite show. And that though there is merit in the statistics that TVAAS produces (assuming that is true), those particular statistics are not what necessarily determine a teacher's value or contribution, any more than any one sport statistic determine's a player's value or contribution. People will understand and accept that (at least as a possibility). They will not accept your saying that scientific measurement or statistics is political any more than they would accept your saying that "batting average" in baseball is merely political. Neither of these things is "political" in any normal sense of the word. You are stretching the use of the word; and in doing so, you are losing most of your audience since most people will neither understand nor share your meaning. And they will think you are saying there can be no criteria of any sort for legitimately distinguishing good from bad teaching.

From: Rick Garlikov

The example Sherman gives of the firing/keeping of a teacher who teaches high-achievers well and low-achievers poorly as being a political decision seems to me to conflate a number of possibilities here. It is a political decision IF the wishes and needs of the aggrieved party are ignored simply because they are powerless to cause anything of consequence to the people making the decision. But it is not political if the teacher is replaced by a teacher who is able to help both groups of students. Nor is it political if a way is found to utilize the skills of the current teacher while also doing something to improve his/her skills with the other students, or to divide teaching assignments in other ways so that both sets of students get teachers who can help them the most. Not every decision is a political decision - -even if some of the results are the same as a purely political decision would be.

Now, since I don't understand what exactly Sherman is attributing to me with regard to the question of TVAAS's impeding the use of formative testing in teaching, I think there is some sort of misunderstanding. So let me try again.

Sherman had said that since TVAAS and the test it utilized did not give formative feedback in, say, students' long-division skills, it did not help teachers know whether their students were "getting" long division at a time that knowledge would be helpful to better teaching/learning. My response was meant to say that this did not matter --as long as TVAAS did nothing to actually impede good teaching --in this case, presumably, the use of formative testing by a teacher in order to figure out whether students were understanding or correctly doing long-division or not. The state does not need to train/foster/force/guide good teaching techniques; it just needs not to impede their usage. By analogy, serving good lunchroom food to teachers does not improve their skill at learning whether their students are improving at long-division either; but there are other reasons for serving good food. Similarly there are reasons for TVAAS apart from whether it serves to help teachers identify --as they are going along-- whether students are learning or not.

If the test that TVAAS uses measures skill in long-division, then presumably teachers will have a self-interest in teaching long division well, and in learning to use whatever methods will do that most reasonably. But the state, and certainly TVAAS, does not need to explain to teachers what those methods are. Especially if TVAAS is pointing out which teachers seem to be using good methods. An environment does not need to be helpful in ALL ways as long as it is not harmful in important ways.

Now, there will undoubtedly be teachers, just as there are students, who will try to find some shortcut, easy method to get good superficial results even if that means not doing what is best in the long run. But that seems to me to be a system problem only if the test or the statistical use of the test is so bad as to allow or reward, and therefore encourage, that. TVAAS is supposed to be able to pick this out even, not foster it. If Sherman has some reason to believe the test or the statistical package encourages shortcut, easy-way-out "teaching", then that is important to know. But his claim was that testing for ability in long-division at a time too late to correct inability, somehow precluded teachers from giving such formative testing in their everyday teaching. That just seems clearly false to me. The two things are at most unrelated, but I would suspect that generally something like TVAAS --if the test used is not vulnerable to teaching "toward" in a superficial way-- would encourage teachers to want to learn the best teaching methods, not the easiest bad ones. For Sherman's point to be successful, it seems there needs to be some evidence that TVAAS and the test it uses does, or likely will, adversely affect how teachers teach. The fact that some testing is high stakes testing, and the fact that some high stakes testing is counterproductive gives reason to be initially concerned; but it does not give evidence that TVAAS will actually be high stakes testing or that, if it is, it will impede rather than enhance good teaching.

Finally, it seems to me that the concern that TVAAS statistics will be misunderstood and misused by administrators, the legislature, the media, and the public, is best remedied by informing those groups in ways they understand and can use; not by trying to eliminate one kind of indicator from being used at all. Surely, though they resist it often, the media and the legislature can be made to understand some things -- and that one of them is that wise judgment needs to be based on all sorts of available evidence and not just on one statistic. As I argued in regard to my baseball analogy of a previous post, ordinary people do understand, when examples and reasons are given, that one statistic alone is not likely to indicate the overall value of a person's contribution to his/her profession. This is not a difficult concept to get even a legislator or a newspaper editor to understand, if it is explained in the right way. The legislature may even be persuaded to require certain kinds of clearly visible, audible, and understandable information about how to use TVAAS statistics whenever those statistics are reported in the media.

I think there are some important concerns about this enterprise that have not yet been raised, and I intend to raise them soon if no one else does; but I think they can be remedied too. I am not trying to defend TVAAS or the current use of it against all charges; but I think that, if the statistical methodology really is reliable and there are tests it can use which really do give some indication of what students collectively (or on average) have learned in a content area from a teacher, and if this information can be gathered relatively inexpensively and quickly, it is important to use the program and to simply be vigilant in keeping it from being misused.

From: Sherman Dorn

In Rick's response recently:

>For Sherman's point to be successful, it seems there needs
>to be some evidence that TVAAS and the test it uses does, or likely will,
>adversely affect how teachers teach.  The fact that some testing is
>high stakes testing, and the fact that some high stakes testing is
>counterproductive gives reason to be initially concerned; but it does not
>give evidence that TVAAS will actually be high stakes testing or that, if it
>is, it will impede rather than enhance good teaching.

This is an interesting counterpoint to the VARAC's argument that assessment should be scientific: don't we require "treatments" to be VERIFIED as both harmless and beneficial before we let them loose on the general public? There was no such experimentation with TVAAS (unless people consider it an ethical experiment to subject several hundred thousand children to the unknown effects of an untried evaluation system).

In reality, of course, there is little opportunity for such experimentation in policy, and it would misrepresent the purpose of policy (which is often to creat NEW systems -- and they're going to be new for someone at some point). But I am very concerned by the switch in burden of proof -- Rick is suggesting t despite evidence of some disturbing effects of high-stakes testing, we should le systems go ahead unless someone else can demonstrate concrete effects. This represents a rather high standard for opposing a new policy -- after all, we can demonstrate concrete effects until the policy is in place.

Also, let me answer Rick's questions about formative versus summative evaluation with an example: Suppose Renee Jones uses formative evaluation in her fourth-gr class for math and reading, and she responds in appropriate ways (changing instr when it seems appropriate, either for the entire class or parts of the class). the kids know everything fairly well, though three kids have only partially mast long division by the time of the annual tests. Later in the year, TVAAS gives h mediocre scores in math. How is she supposed to respond to this? Should she de that she should have concentrated on long division? Should she concentrate on o skills? Should she junk the formative evaluation system, since it obviously did help her? Remember, she has only three chances to improve her TVAAS scores in math before they can be used in personnel evaluation, and she has to make the decision for the ENTIRE year. And, the test for TVAAS doesn't give any extra information.

Now, I don't know whether TVAAS is consistent with other measures of school achievement -- thus far, the only study I am aware of that preceded the state's imposition of TVAAS was a mid-1980s test of the feasibility of the mixed- model methodology. I know of no empirical comparisons between TVAAS estimates of teacher effects and other measures of student gain. So let's choose another, more likely scenario:

Renee Jones does *not* use formative evaluation currently. For the past several years, she has tried various measures to "improve test scores," as her principal is constantly saying to the staff at her school. None have worked, a as Tennessee started to use the CTBS, her frustration has just increased with th lack of concrete feedback and the test scores that are at variance with her admittedly subjective judgment of the children's competence. Now comes TVAAS, and the stakes have taken a quantum leap upwards for her. The first set of TVAAS scores for her as a teacher this year looks lousy. She's very nervous, and doesn't really know what to do. She's considering several options: adopting a math program that's patterned on "Hooked on Phonics," with moderately expensive tapes but all planned out for the teacher; working with other teachers at her school in an "experiential math curriculum" (which they haven't yet desig spending most of her math time in the computer lab to try to teach using the CAI software she can spend her allotted class material funds for; or investing t time to learn a formative evaluation of math (which requires a lot of time and m test scoring and the tests). Again, of these FOUR options, she has to choose ONE for a single year. Also remember, she has a personal history of trying various measures to improve her test statistics and of the test scores being at variance with her own judgment of students' competence.

It is very likely, in my personal contacts with teachers and knowledge of resear and gut instinct, that Renee Jones will dismiss the formative evaluation system because (a) she has no reason to believe it will improve her test scores any mor than anything else she has either tried or is considering trying; and (b) it is resource-intensive, moreso than other possibilities. In this case (and this is assuming teachers are exposed to the idea of formative evaluation), the high-sta environment of TVAAS can drive a teacher away from a way of checking kids' performance that allows MORE flexibility for teachers. That is why I described TVAAS as in competition with formative evaluation, and why I believe states should instead be explicitly supportive of it.

__________

From: Alan Davis

Rick, Let me try to disentangle some of the arguments about the "political" nature of the TVAAS evaluation system.

My point that evaluation is a political act was not meant to be a criticism of TVAAS, or to argue that evaluation shouldn't occur. It was instead a recognition of the fact that when you decide to evaluate a publicly funded activity, such as teaching, the act of evaluation has consequences for the prestige, power, and resources of various actors. When one thinks about the information generated in the process of conducting an evaluation, one needs to distinguish the question, "Is this information accurate?" from the question, "Is this evaluation valid?" The latter question requires one to examine the interpretations that will be given to the data, the interests that will be served, and the consequences associated with providing limited data addressing a limited range of questions.

There are several problems with the validity of an evaluation system that compares teachers on the basis of the measured learning of students.

The system only credits learning measured on the test. It tells nothing about curiosity, work habits, or the social climate of the classroom.
The test is very limited. It correlates only moderately with the quality of students' written work, the ability of students to tell about what they have read, or the ability of students to conduct investigations, for example.
Students have little motivation to do well on the test. It is external to anything else they do in school.
Once teachers become aware that they are being compared on the basis of test scores, they are pressured to bring the scores up. To do this, they are likely to (a) stop spending time on things that are not on the test -- things like art, music, science projects, written stories and reports, and class discussions; (b) provide students more practice answering all sorts of things in the format of the test -- multiple choice; (c) teach the particular spelling words and vocabulary words that appear on the test in place of ones that are not; and (d) grow to hate the test and respond in a mode of resistance/compliance rather than in a pursuit of professional excellence.

Rick, as one who has in the past argued vehemently against the validity of multiple-choice assessment to evaluate student learning on this very list, I have a hard time understanding why you are intrigued by the use of the same tests to evaluate teachers. No one attempts to reduce the contribution of doctors or psychologists to a numerical score. Teaching is every bit as multi-faceted and context-specific, yet policy makers want to find a way to reduce teaching to a bottom line. I believe there is no way to do this that is actually beneficial to the education of children in the long run.

From: Rick Garlikov

Alan,
I agree with the concerns you list. And I have others along those lines, that I will express.

What I disagree with is calling what those concerns address to be the "political nature" of evaluation. (My mail got scrambled today, and I sent a response to Brian's post before getting yours; but I think it addresses this again. My point is not against your concerns; it is against calling the problems you discuss "political". I don't think that helps you make your case with anyone except those who happen to use "political" in this same, unusual way.)

TVAAS has made it quite clear that they are only providing one indicator of possible teaching problems or excellence. And they are apparently going to some lengths to keep tests from being predictable in certain ways. Sherman feels there is reason to believe that in spite of TVAAS intentions and explanations, forces are at work in the legislature and in ed administrative offices to misuse the results, considering them alone; and that teachers will be pressured into teaching in whatever ways they can "to the tests." Part of the reason for my wanting to discuss TVAAS was to see if there were not some ways of articulating what is going on, what should be going on, and what is likely to be going on, in ways that will be useful to the teachers and people of Tennessee to try to make sure there are pressures to have the "right" things going on regarding TVAAS and teacher evaluation in general. I AM concerned there is not enough of the right sort of information being given teachers, administrators, the media, and the public. But I think that can all be addressed and remedied.

I never thought teachers should be evaluated solely by means of a multiple choice test of their students. But I think it may be a good indicator of where to look for good or terrible teaching to confirm what TVAAS shows. When I was an academic adviser to freshmen and sophomores at Michigan, incoming freshmen were given Benno Fricke's OAIS test, which measured a number of things which were reported to advisers. I never saw one of those tests be wrong on a student, BUT I also never took any of those tests as a sure thing. They were merely to be used as an indicator of student academic interest areas and of students' motivation and emotional maturity levels. Low scores were simply red flags to be alert to certain kinds of possible problems developing. As an advisor I wanted to see other evidence and talk with the students to see whether the OAIS scores might be significant or not.

The same sort of thing should be done with TVAAS, and there is no reason that cannot be made clear to everyone. The OAIS scores were very helpful to good counselors and just because bad counselors might misuse them, that does not mean the test should not be given and scores reported.

The issues that need to be guarded against are those you bring up. But I think that can be done. And, if so, then TVAAS can be helpful in those cases where administrators in the past seem unable to detect or remedy problem areas on their own. It would be very difficult for a superintendent to continue to tell everyone it is not his system's fault they have poor GAIN scores, if other systems similar to his in all other respects have good GAIN scores. Further, one of the things I like about the TVAAS approach is that culturally advantaged districts with high achievement scores will not necessarily show the best GAIN scores --and in areas of Alabama I am familiar with, that would be a great thing, since many of the supposedly better systems rest on their students' backgrounds rather than the school districts' contributions to their education. They don't help students gain all that much; and that is important to point out.

Doctors and hospitals ARE partly assessed in certain kinds of numerical scores --mortality rates being one important one. But again, these are used as indicators, not sole factors.

I am not only opposed to grading students by multiple choice tests; I am opposed to grading them at all; and to grading them by means of any formal tests. I am opposed to grading teachers too. But TVAAS is NOT INTENDED to grade teachers or districts; it is intended to point to probable problems (and probable successes). I believe in assessment, not grades. And I believe that people have strengths and weaknesses which can be assessed in various ways. I try to assess my students' strengths and weaknesses so that I can help remedy their weaknesses, or so that I at least present the right sorts of material in the right ways --stuff they can assimilate and deal with in some feasible ways.

Education is one of the professions that has seemed rather weak in setting or maintaining high standards for itself. Many good teachers understand the incompetence of some of their colleagues who nonetheless receive tenure. Many parents, for good reasons, as well as those parents with bad reasons, think certain teachers are really incompetent. There seems to be a need for some sort of independent indication of who is doing a good job and who is not -- teachers and administrators.

Finally, one of the problems I have about testing students summatively is that it is not quite clear to me that a given test will be very reliable for a given student on a given day, for reasons I detailed a year ago in a long series of posts. But, what TVAAS is doing is weighing the results of hundreds of student tests involving a given teacher. It seems to me that averages will cancel out a number of individual factors, so that teachers will get some sort of average score, rather than an individual score. I don't want to give a kid a grade based primarily on a final exam; but I think I should be able to judge how well *I* have taught from the collective nature of 100 or 200 exams. When the median on the midterm in my second semester calculus course (when I was a freshmen) was 30, out of 84 possible points, the teachers of the 1500 students in the course knew something was wrong with how they had all taught that first part of the term. But it was not clear that any one student's test scores was definitely reflective of what s/he had learned or could do.

Anyway, what I am concerned about is ways of verifying TVAAS as a good indicator of good, bad, or average teaching; seeing what its limitations are in terms of what it can even identify probabilistically; and then trying to figure out how to state this so clearly and forcefully that legislators, media, and the public cannot misconstrue what the TVAAS results indicate, and how they need to be confirmed or disconfirmed. All this so that teachers do not feel threatened by the test itself, nor encouraged to try to take teaching shortcuts that merely shortchange kids. It may be that this cannot be done. But I don't accept that as the STARTING point in some a priori fashion.

=========================================================================

From: Rick Garlikov
Subject: Re: TVAAS and Dorn further

In response to Sherman's points. I was not as clear as I should have been about the "burden of proof" he discusses. There are things that TVAAS has done or tried to do to eliminate the kinds of problems high-stakes tests cause. Further, they have tried to be clear in their literature (though I think this could be much clearer yet) that TVAAS should not be high-stakes conclusive testing, but merely an indicator of problems. The question ought to be whether they have succeeded in keeping the tests used for evaluation from being high stakes tests or from being able to be predictably taught "to". It is not a question of burden of proof; it is a question of whether the whole program is relevantly similar to the kinds of high stakes tests, that do cause problems, documented in the literature.

As to "Renee Jones", it seems to me the rational thing for her to do would be to find out who has good test results and go talk with them about what they are doing that seems to work so well. It may be they do teach better than she, and that they can teach her how to do it. It may be that they use some trick she feels is unconscionable and short-changes kids. If so, TVAAS needs to be aware of this trick that skews their results. And they need to pay attention to the person reporting it.

But it may also be that Renee Jones just isn't a very good math teacher and isn't likely to ever become very good at it. Maybe she doesn't really understand math very well herself. What then? Do you want her to keep teaching math?

Finally, it boggles my mind that teachers --many teachers-- have no clue about what I guess is called "formative assessment" which to me just means doing whatever you can to try to monitor your students' knowledge and understanding as you go. I have gone over this ground before, but I just cannot see how one can be trained as a teacher, certified as a teacher, hired as a teacher, tenured as a teacher if one does not have at least some intuitive idea that one needs to monitor what kids are "getting" from one's instruction so that one can see whether modifications are necessary. Surely it does not take "resources" to teach this to ed students or for the state to expect it of teachers.

Of course there is some question whether what is tested is appropriate -- e.g., long division or whether long division ought to be part of the fourth grade curriculum. But TVAAS is set up to test whatever content area is specified by the curriculum; it is not their job to determine curriculum. I myself tend to suspect that the math curriculum is not too difficult for kids if it is taught correctly --which to me involves both practice AND deep understanding. But I am not certain primary grade teachers will have sufficient understanding themselves. I think finding the proper resources to teach math well will involve something other than finding resources for teaching about "formative assessment". It is about bringing people into classrooms who have enough math understanding to be able to teach math decently, even if they have the right teaching techniques and tools.
Rick Garlikov (dems042@uabdpo.dpo.uab.edu)

=========================================================================

From: Rick

Contrary to what Sherman says, teaching is done publicly. I count kids as being an audience that sees what is going on. And parents can observe, and also talk with teachers.

The important point he mentioned, however, is about ballplayers negotiating the stats that will be relevant to their contract bonuses, etc. That is only true within relevant limits -- generally the stats used have to be something that seem relevant to accepted productivity. Once in a while some screwy stat clause gets in and actually occurs, but generally the stat clause is about some obviously relevant achievement. And though teachers individually don't have the same luxury of contract negotiations baseball players do, they often have great collective clout both politically, and professionally --professionally where they help set standards that schools, teachers, and districts should be measured against. Although teachers sometimes fall victim to having irrelevant standards placed on their profession by outsiders, that is not always the case; and it doesn't have to be the case at all, I don't think.

From: "Fetler, Mark"

Rick, you said -

In reply-

Politics is not intrinsically good or bad. It is just the practice of resolving disputes or making decisions about the distribution of limited resources. I think what distinguishes evaluation from many other types of more basic or pure research is that evaluation is intended to serve policy makers. The decision to use evaluation is political, particularly if it affects policy decisions or the support for those decisions. Of course, there are always moral, ethnical, and relationship issues around the exercise of power. However, often one's perception of these issues is colored by the degree of benefit received. All this is true, in my opinion, even if the evaluation conforms to the most rigid technical standards and is "cleaner than a hound's tooth."

As to objective measures of teachers' contributions. I would submit that a really well designed, administered, scored, and reported achievement test is an objective measure -- of students' performance in answering certain questions. How that measure relates to teaching ability requires additional information, for example, about the students' abilities, opportunity to learn, etc.

From: Alan Davis

Rick,
First, I want to concur with Fetler and Stecher regarding the meaning of "political." You wrote that "In politics, people often use data to deceive" and went on to describe how politicians might intentionally mislead people through a selective interpretation or mis-interpretation of objective date. But as Mark and Brian pointed out, those of us who swim in the seas of policy analysis do not think of "political" as either good or bad. The term refers to processes affecting the ability to influence decisions -- processes having to do with power, in other words. Evaluation cannot escape being part of the the political process. Even the decision to evaluate has political consequences, even if no one ever reads the evaluation.

You long to introduce information into a politically neutral world, it seems to me, when you write, "Anyway, what I am concerned about is ways of verifying TVAAS as a good indicator of good, bad, or average teaching; seeing what its limitations are in terms of what it can even identify probabilistically, and then trying to figure out how to state this so clearly and forcefully that legislators, media, and the public cannot misconstrue what the TVAAS results indicate, and how they need to be confirmed or disconfirmed. Al this so that teachers do not feel threatened by the test itself, nor encouraged to try to take teaching shortcuts that merely shortchange kids."

It cannot be. As soon as I publish a list of teachers with their adjusted gain scores and my "clear and forceful" caveats about what these mean, some school board member will say, "These teachers at the bottom of this list should be put on probation" and a good many teachers will feel threatened by the test itself. I cannot conceive of any evaluative use of this information that would not put teachers under pressure.

From: Rick Garlikov

I have a few questions about TVAAS results, and the way scores are used and reported.

According to TVAAS results, "student gains were not related to the ability or achievement levels of the students when they entered the classroom." There are a number of possible causes for this, some good, some not so good. And I wonder whether TVAAS has any evidence which causes operate, and to what extent.

First of all, the evidence is somewhat counter-intuitive for individual students at least, if not for groups of students. It would seem that a "bright" kid with a good background in language, reading math, etc., curiosity, and motivation would learn a lot more than a kid with a disadvantaged background who is also perhaps a bit slower to pick up new concepts or understanding. Is the TVAAS result dependent in some way on averages for groups rather than direct comparisons of the "brightest high achievers" versus an average or "below average student"? That is, do a lot of bright high achievers lose motivation or slow down on gains for some other reason, in order to pull the "high-achiever" average down?

Or are you saying that with equal or really good teachers lower ability/achievement students can gain in some sense actually or proportionally as much as higher ability/achievement students? That a student who comes into third grade being facile with multiplication tables will not progress much further into math (e.g., division, long division, factoring, combining fractions through lowest common denominators, etc.) than a student who enters the third grade with no good grasp of either multiplication facts or an understanding of what multiplication is about?

Of course, one, I think bad, way this can happen is if high ability/achievement kids are ignored and simply not taught to their potentials, while teaching time, energies, and lessons are devoted primarily to lower ability/achievement students. This is the way many teachers teach, in fact, and it is the what some school districts seem to promote. Higher ability/achiever students are left to learn whatever they do on their own essentially, so their "gain" relative to their potential gain is low. This philosophy often stems from a reaction to an also bad, previous, philosophy where resources were channeled in helping the PERCEIVED best and brightest academically while letting other students slide more than would be helpful.

Which brings me to the question of whether the notion of "norms" does not serve as somewhat of a false standard for students of all levels of ability/achievement. Shouldn't the "standard" reflect, not some sort of mean, but some sort of reasonable potential, as gauged perhaps by what the best teachers, schools, or districts do, not what the average does. So that if one school or district consistently has significantly higher gain scores (along with other evidence that school is doing much for its students) with, say, the same relative budget) isn't that evidence such teaching is possible and that it ought to be the goal to aim for. Perhaps research evidence might demonstrate an even higher level to be reasonably possible under the typical school conditions. Shouldn't there be some sort of distinction made about how far from what is possible a school, district, or teacher does, not just how far from the average they do?

Of course, anyone can compute how far below the current best they are in regard to these particular gain scores, by subtracting; but I am not certain people bother to do that, or that if they do, they care about the result. If it were a reported number, it might be more of an incentive for schools to improve instead of trying to maintain or achieve merely the status quo average.

I raise this issue because I am concerned about teachers/parents/administrators having unreasonably low expectations. I am aware that there may be "curve setters" based on additional resources or circumstances not likely to be replicable in other systems. I am not after a standard that is out of reach, but the highest reasonably, or probably, attainable standards.

From: Gene Glass
Subject: TVAAS, Bright Kids, Gains, Fairness.

On Tue, 10 Jan 1995 14:28:52 CST Rick Garlikov said:

>I have a few questions about TVAAS results, and the way scores
>are used and reported.
>      According to TVAAS results, "student gains were not related
>to the ability or achievement levels of the students when they
>entered the classroom."  There are a number of possible causes
>for this, some good, some not so good.  And I wonder whether
>TVAAS has any evidence which causes operate, and to what extent.
>      First of all, the evidence is somewhat counter-intuitive for
>individual students at least, if not for groups of students.  It
>would seem that a "bright" kid with a good background in
>language, reading math, etc., curiosity, and motivation would
>learn a lot more than a kid with a disadvantaged background who
>is also perhaps a bit slower to pick up new concepts or
>understanding.

Indeed, Rick's Sense that something is not right here is quite understandable.

I originally asked TVAAS (whoever it is we are talking to) how it could be fair to compare the gains in achievement of students from one teacher to the next if the abilities (on average) of the students in the classes differed substantially. To this question, TVAAS long ago gave the incredible answer that it was an empirical fact that in their data there is no difference in the gains of bright children and slow children. Rick quoted the relevant sentences above.

This artefact is not because bright and slow children actually learn at the same rates; it is because the TVAAS system of calculating gains surely uses a form of least-squares estimation that forces the pre-year measures to be uncorrrelated with the post-year measures.

I can take a group of IQ equal 120 kids who gain 1.5 years in grade equivalent units in a school year and a group of 80 IQ kids who gain .75 grade equivalent units in a school year and calculate the residualized gain score for the entire group and as sure as God made little green apples, the "gain" score will be perfectly uncorrelated with the pretest score. This is no paradox, nor does it make much sense as a measure of true gain.

I continue to believe that TVAAS has no fair way of equating the teachers who are compared with respect to the ability of their pupils.

Rick, you are right. TVAAS did not give an answer that was responsive to the question, and thus succeeded for a time in finessing an important criticism of the system.

=========================================================================

From: Rick Garlikov

Alan Davis said:

>It cannot be.  As soon as I publish a list of teachers with their
>adjusted gain scores and my "clear and forceful" caveats about what these
>mean, some school board member will say, "These teachers at the bottom of
>this list should be put on probation" and a good many teachers will feel
>threatened by the test itself.  I cannot conceive of any evaluative use
>of this information that would not put teachers under pressure.

You are saying school board members will ignore the caveats and will not be challenged by anyone in doing so. I would think that if the caveats ARE clear and forceful, it is likely there would be a strong challenge if school boards were ignorant enough to act in the way you predict.

>On the other hand, I don't think any of us who are complaining about
>TVAAS would argue that teaching should not be evaluated.  How best to do
>that is an important discussion to have.  I am arguing that the
>evaluation should not involve students' gains on standardized tests, even
>though I believe that student learning is what teaching is mostly about.

That is an agument worth pursuing; and I hope we can pursue that. As I understand the reasons for that though, they are that in high stakes testing, teachers tend to teach "to" the tests, and in doing so take shortcuts, leave out other important elements of education, etc. But as I said in response to Sherman, since TVAAS and/or TCAP has taken steps (1) to discourage teaching to the test by changing the test each year, and (2) to prevent this from being high stakes testing, by making clear it is only one form of indicator or evidence about the quality of teaching, and that in content areas alone, it seems to me the question is whether they have succeeded or not, or could.
Why do you think they have not succeeded or that they cannot succeed?

From: Rick Garlikov

Mark,
TVAAS claims to be able to isolate learning from the sorts of factors you mention --ability, opportunity to learn, etc. The question is whether this claim will stand up. But it is not that TVAAS has ignored the sorts of factors you mention; and it is not that they were unaware of their potential influence. The only question in THIS regard is whether they have satisfactorily accounted for these factors or not.

From: Sherman Dorn

Rick Garlikov writes:

 
>    You can call looking for, and trying to compile and describe, relevant
>data a political act; but that just confuses a whole bunch of things under
>a designation that applies in only the most superficial way, if it can be
>said to apply at all.

Rick, am I incorrect in reading your argument as follows? "It is unfair to call statistics political when people try to gather them in good faith; things that a political are manipulated."

I think this is a narrow reading of politics, and it misses the point that often actions have political assumptions precisely WHEN people act in good faith. Good faith judgment, in my view, is irrelevant to whether there is a politics of statistics. Wiliam Alonso and Paul Starr edited a book, _The Politics of Numbers_, several years ago, which discussed the production of government statistics, their political assumptions, and implications. I remember some discussion of the political assumptions of some government statistics, but I don't recall either a conspiracy theory or a description of much crass political motivation.

From: Greg Camilli

I am reposting an earlier message regarding TVAAS. I hope these questions will fetch responses, though I suspect the TVAAS staff are pretty busy.

First, it was indicated earlier that the norm-referenced items used by TCAP are somehow related to the CTBS/4. In this regard, I'm wondering if items are sampled from the CTBS, or whether new items are being written at every assessment. The latter is suggested by the phrase "fresh, non-redundant, equivalent tests."

Second, a number of different metrics are available from CTB: is TVAAS using the IRT (developmental) score scale which was established with national norms? Is there a document I could read to get the specifics of how the tests are constructed and normed? If so, it would really save a lot of time.

Third, I'm wondering if my interpretation of the missing data procedure is correct. I understand that multiple imputation does not affect the values of "effect" coefficients. The models and methods used to obtain these bypass the estimation of individual scores. However, multiple imputation increases the standard errors of the estimated coefficients -- this simply refects the notion that with less than complete data, less is known about the parameter. Thus, multiple imputation is a post hoc adjustment to estimation. However, with multiple imputation one can always produce a "complete" set of data (where imputed values have replaced missing values) to expedite reporting or secondary analyses. (Please do not hesitate to correct errors in this formulation.)

Finally, I hope that Harvey Goldstein can find time to study and comment on the standard errors reported to us by Rick. Based on his previous post I expected to see larger values. BTW, I've been wondering what the standard errors mean. Usually, I have in mind that a sample is drawn from a population, and an effect (say gain score) is estimated from the sample data. The standard error then conveys how precise this estimate is (much like the "margin of error" that pollsters use). For TVAAS, what are the sample and population?

From: Rick Garlikov

Sherman Dorn said:

>Rick, am I incorrect in reading your argument as follows?  
>"It is unfair to call statistics political when people try 
>to gather them in good faith; things that
>are political are manipulated."

No, that is not what I mean. I am not talking about whether it is fair or unfair to call something political. I am merely trying to capture the standard sense of distinguishing political acts from non-political ones. Politics tends to involve promoting an ideology or vested interest, often through some sort of position of power. It does not have to involve deceit; and I am sorry I mentioned that aspect of it before. Politics is generally distinguished from (attempted) dispassionate, reasoned, unbiased judgment.

This distinction is embodied in such dichotomies as "political" vs "non- political" appointments; or in saying that the courts are supposed to be a non-political part of government. Or in distinguishing between political trials and other sorts of trials, even though the non-political trials may also have ramifications for many people's lives. It is supposedly a bad thing to "politicize" the process of appointing Supreme Court justices, and democrats and republicans both often agree someone is qualified to serve and is not a political nominee. Of course, many supposed non-political things are actually political; and many politically appointed judges tend to ignore their own political views --often disappointing those who appointed them to the bench. The difference has to do with whether one is pursuing a bias or whether one is open to evidence and reason in some at least relatively unbiased way.

The kind of things you, Mark, Brian, and Alan are discussing are value-laden, of course, but being value-laden is not necessarily the same as being political. A scientist trying to find an answer to some problem may have to choose among a number of research avenues; and s/he may make an educated guess of some sort about which might be the most beneficial to pursue. Ordinarily, merely in the confines of the lab, that kind of choice is not a political choice, even though it may involve expenditures of funds, directions of lots of people, etc. It is not the same kind of thing as choosing to pursue a line of research because that is where government funding is easy to secure or because it leads to promotion and tenure at the university, fame, or whatever. These latter I take to be political considerations about what research to pursue.

My point is simply that the average citizen makes a distinction between "political" decisions and "non-political" ones, and even though there may be some reason to argue his distinctions are sometimes fuzzy or erroneous, still there seems to be some sort of clear-cut cases; and if you say something is "political" in policy-analysts' technical sense of the term that is not political in the ordinary sense of the term, people will tend to dismiss what you are saying.

From: Leslie McLean

The purpose of this post is to focus on two aspects of the TVAAS that I feel have received too little attention: validity and standard errors. This is not to say that the political nature of any evaluation is not important or to take anything away from the discussion of formative vs. summative evaluation. My other point will be that we are getting mixed messages as to the purpose of TVAAS. Since I started to put this collection together, Greg Camilli has posted some related questions.

What follows is a selection from several postings concerning the TVAAS ((Alan Davies, Rick Garlikov, TVAAS, ...)), with comments from Les McLean added. Les's comments are enclosed in double parentheses: ((...)). Emphasis added by Les is signaled by *** ... ***

((Selections from Alan Davis's post of Wed. January 4. He amplified these on Jan. 9)) "The main problem, I believe, is that the multiple choice measures used in Tennesee are at the outset poor proxies for the tasks we really want kids to be able to perform in school, and when we attach consequences for teachers around scores on these measures, very bad consequences are likely to result.

Note first, that performance on the measures valued by TVAAS are not valued by students. The tests are not part of any instructional unit. No grade is associated with performance on them. Students have no intrinsic or extrinsic motivation to perform well on them, especially if they don't like mental puzzles for their own sake. *** The problem, which I do not believe can be avoided, is that the validity of tests is inseparable from the uses to which they are put. A test that may be valid as a research tool to discover effective teaching will lose its validity when teachers suspect that the outcome has consequences for themselves. ***" ((emphasis added--and stressed Jan. 9))

((Date: Wed, 4 Jan 1995 20:52:42 CST From: Rick Garlikov))

"TVAAS has already said in a previous post that their results are supposed to be used as INDICATORS that something is very good, or very wrong, about a school/district/teacher, not conclusive evidence. That leaves room for the kind of evidence to be given about the merit of schools/districts/teachers which IS reasonable to educators -- a concern that Sherman expressed in explaining (1). ((Emphasis in original))

.....From what has been said so far, TVAAS tries to ascertain how well certain subject content areas --as prescribed outside of TVAAS ***are being taught,*** not whether those subject content areas are important or whether "knowing" them in certain ways is important."

((Emphasis added--this is the clearest statement I could find that they equate quality of teaching with scores on standardized tests, scores that have been scaled and transformed in unknowable ways--see remark from TVAAS below.))

((Rick goes on to make it very clear,)) ".....I may be misunderstanding the point of TVAAS, but I thought it WAS established to give summative evaluations about teaching competence/progress, not formative evaluations about improving instruction. I take it that TVAAS is not about helping teachers/schools/districts do their jobs, but about suggesting, in some attempted objective way, to everyone how well or poorly they have done it."

((But then he seems to change his mind,)) ".....TVAAS is a statistical model for program evaluation. It is not a "technocratic tool," as Dorn so colorfully phrases it." ((When most people refer to program evaluation, they include content choice, teaching materials and teaching methods, and they distinguish this from 'teaching competence/progress'. Given the complexity of the 'model', Sherman Dorn ought to be allowed to refer to it as a 'technocratic tool'.))

....It was not developed by the State of Tennessee but by Dr. Bill Sanders, a statistician, for the purpose of addressing problems previously encountered in using student achievement data in educational assessment.

The State of Tennessee adopted it as the model for such assessment because TVAAS was able to supply valid, reliable, unbiased data based on student gains, data the State thought was important.

.....TVAAS merely provides measures of student academic gains, surely a useful component of any educational assessment system. .....TVAAS is a sophisticated statistical process ***that is beyond the capacity of many to grasp mathematically.*** However, TVAAS data is reported in a manner that is comprehensible to any educator willing to look at the reports and graphs supplied, with explanations, specifically for the purpose of rendering the data USEFUL. Judging by reports from many schools and systems across Tennessee, TVAAS data is now extensively used for curriculum planning and development.

((Yes, the mathematics is beyond most people's capacity--including most of the officials in Tennessee. How much beyond cannot be determined until they publish a full technical report--such as the one's ETS publishes about its surveys. The term, "mixed model" is quite inadequate to describe the scaling and multilevel model fitting that is going on. BUT--the data are being used for 'curriculum planning and development'????? THIS BRINGS ME TO MY MAIN POINT: what evidence is there that the "data" resulting from these complex statistical manipulations can be interpreted as INDICATORS of competent teaching--much less the quality of the curriculum? From the TVAAS, we learn of an even more ambitious outcome--they provide schools with gain scores FOR INDIVIDUAL STUDENTS. See what they say:))

......As we have repeatedly told Dorn, TVAAS does not lock the state into any testing program whatever. TVAAS can use any assessment data that provides appropriate linear measures. In other words, TVAAS requires scalable data because it assesses progress over time.

((Appropriate linear measures? ... scalable data? To anyone familiar with test theory since Lord and Novick (1968), these phrases suggest a very considerable locking in, even given Geoff Masters's extensions to IRT. The numbers the TVAAS are reporting back to schools are a long, long way from the item responses of the students to these multiple choice questions that Alan Davies quite rightly characterises as, " poor proxies for the tasks we really want kids to be able to perform in school". Even if we accept the items as valid indicators, we are entitled to ask for some careful checks that the scaled and imputed and estimated scores really do, "provide guidance as to 'What we do now'".

According to Sherman Dorn (8 Jan. Post), ))

"Now, I don't know whether TVAAS is consistent with other measures of school achievement -- thus far, the only study I am aware of that preceded the state's imposition of TVAAS was a mid-1980s test of the feasibility of the mixed-model methodology. I know of no empirical comparisons between TVAAS estimates of teacher effects and other measures of student gain."

((Concerning the size of the standard errors))

Below are listed the mean gains for math with their standard errors for schools within one of the larger school systems in Tennessee. These means are three year averages and were calculated from the TVAAS mixed model process. This should give an idea of the sensitivity of the process.

((A very few of the results are listed below for reference--the heading (Grade ...) has been edited to make sense. I echo Greg Camilli's wonderment at the size of these 'Mean std. Err.'s. Including all data from several years and using imputation procedures for the (inevitably) missing numbers all lead to larger standard errors.))

            INTERMEDIATE SCHOOLS
 
        GRADE     RANK   GAIN   MEAN STD. ERR.
 
           3        1   71.596     4.931
           3        2   71.205     3.714
           3        3   67.038     2.624
                ........................
 
 
                 MIDDLE SCHOOLS
 
           6        1   22.624     1.043
           6        2   20.176     0.824
           6        3   15.602      1.07
           6        4   13.943     1.099
           6        5   13.152     1.286
           6        6   12.521     1.051
           6        7   11.146     0.961
           6        8    9.897     1.194
           6        9    9.362      1.34
           6       10    8.465      0.998
                     ...................
 
           8       12    13.55      6.605
           8       13   13.261      1.738
           8       14    11.08      0.98
                 ......................

((The problem for those of us who have calculated, pondered and puzzled over such results as these, in national and international assessments, is that the reported standard errors are unbelievable (impossibly small). We can't say they are wrong, of course, because we lack the details of the calculations, but Harvey Goldstein has analyzed at least as much data and written several books and taken the lead in multilevel modeling (sometimes called, by others, hierarchical linear modeling), and his informed and experienced "opinion" is not to be taken lightly. The standard errors remind me of those Richard Wolfe found faulty in the first International Assessment of Educational Progress--the fault being that the estimates of error failed to include all the components reasonable people agree should be included. Moreover, the Std. Errors above are clearly proportionate to the mean scores, not a desirable outcome. The must be at least one error (three lines from the bottom of those displayed above). I, too, will leave to later a comment on the statement below from TVAAS, except to say that whatever it is they do is not "certainly sufficient":

"...Additionally, there was a question concerning how TVAAS deals with missing scores. We will write a longer respose to this question later. But, briefly, it is more like the analysis of repeated measures. However, we do include all of the scores among subject-grade combinations. This is certainly sufficient and avoids the issues and problems associated with imputations."

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

From: Leslie McLean

Your questions are much appreciated, Rick, as they bring out unclear points and (we hope) useful elaborations.

1. The problem with pairing student achievement with teacher competence is that the teacher, or even two teachers in succession, can only build on what the students bring to the teacher's class, and then they only add a small amount to the total experience of the students. In short, it is simply not fair to judge a teacher by what the students learn in a year unless you are able to bring in lots of contextual information, not all of it quantifiable. This argument has been made in many places and many ways, so I only espouse it. I conducted a province-wide (read: state-wide) survey of student achievement in mathematics and English in 1981 (mostly constructed-response items, thousands of them) in Ontario, and I produced distributions of classroom mean achievement, suggesting that as many as 20 percent of teachers were in need of some help. BUT my contract specified that the teachers not be "identifiable" (not just not identified--not identifiable), a sensible provision in view of the large-scale nature of the survey--a sample, but covering a larger area than Tennessee, and at least as diverse. We are not unaware of statistical developments here in the Great White North--just skeptical when the mathematics cannot be understood except by a half-dozen people who have not been in a classroom in more than 20 years.

2. What I am arguing is that whether the "math is sound" is not a matter of correct calculations. These multi-step scaling and estimation procedures involve important substantive decisions at every step--decisions often made by well-meaning programmers or statisticians unfamiliar with the context from which the item responses came. The programmers and statisticians cannot ask the educators because the math is so abstruse. In summary--the numbers the TVAAS report are so far removed from the test items that students answer that NO ONE can be very sure the numbers correspond to any reality.

It is this technical gap we worry ;about and ask to be bridged, and we see no bridge. Until there is a bridge, we cannot discuss how appropriate the process is.

=========================================================================

From: Leslie McLean
Subject: Les stands corrected

Apologies to Rick for attributing to him some text from TVAAS. I knew when I was putting it together that this was a danger, and I was not careful enough. Mea Culpa.

From: Rick Garlikov
Subject: Your TVAAS post

Les,
I was glad to see your post. Just wanted to make one small correction first. You accidentally confused one of my own posts with one that was passed on by me from TVAAS. I am in the odd position of having my name and return address on all the posts they send me to forward, so some of them are bound to be seen as MINE by accident. The part right after where you said "((But he seems to have changed his mind)) ....TVAAS is a statistical model for...." is from them, not me.

However, I am not quite sure about the point you want to make regarding the apparent inappropriateness of assessing teachers by their students' progress on certain kinds of multiple choice types of tests. Ignoring for the time what "program evaluation" tends to cover normally, isn't there some point in trying to assess how well students are in general learning the specific kinds of content "knowledge" the State has already said they want taught? I see TVAAS as supposedly trying to measure how well schools, teachers and districts have done in helping kids "gain" increased knowledge or skills over the course of a year. I cannot comment on the reliability or soundness or meaningfulness of the mathematics of all this, but it seems to me that if the math is sound, that the concerns Alan, you, and Sherman have is not with TVAAS as much as it is with the kinds of tests they are being asked to evaluate and report on. TVAAS seems to me to have been quite clear about the possible limitations of those tests in terms of "general education" of students and other aspects of schooling. Their's seems to be a quite narrow focus; and one that is reasonable. The problem is in making that either a prime matter of assessment of teachers, or if Alan and Sherman are right, ANY part of the assessment of teachers. I tend to think it is a reasonable PART of the assessment of teachers. But it surely should not be the only aspect of teacher evaluation; and TVAAS says that it should not. Do you disagree with their and my view of this? Do you think there is no place for standardized test results about certain parts of the curriculum in seeking INDICATIONS of teachers' competence in those areas?

From: Gene Glass
Subject: Re: TVAAS and norms -Reply

On Fri, 13 Jan 1995 07:55:49 -0600 Richard Swerdlin said:

>    Relatedly, an elementary principal in East St. Louis, Illinois
>once told me that he felt guilty over doctoring achievement test
>results (Stanford Achievement Test).  He was fearful of possible
>criticism, if results were below expectations.  Occasionally such a
>case does surface in various parts of the country.
>In conclusion, it is possible that the Texas news item is applicable
>elsewhere too.

More than merely possible...a predictable consequence of these "high stakes" systems. When a large district in Texas--some 10 years ago--instituted a scheme of paying principals bonuses (up to $15,000) for test score gains, a couple of princpals were discovered to have doctored their results. Large gains on the target ("high stakes") test were common while no gains on closely related tests were also common throughout the district. These are demeaning situations in which to place educators.

Would those who administer the TVAAS program be willing to stake their yearly raises on a "significant" score gain on National Assessment for the state of Tennessee? Or does this kind of "accountability" only apply to those under us?

The effects of these high stakes arrangements on teachers and curriculum have been much studied, and they are as one would expect: debasing the curriculum (as judged by teachers and educators), transformation of inquiry into "drill and kill," teaching the test and sometimes worse.

These are difficult matters to talk about because the effects of these programs place many teachers in circumstances in which they act unprofessionally. Then the politicians feel that their suspicions that teachers are unprofessional hacks are confirmed and they institute even further demeaning controls.

It is a sign of the political powerlessness of teachers that these "high stakes" testing systems exist. They are not applied to lawyers; they are not applied to physicians; they are not even applied to college professors.

And on this question of power plays. Does anyone else feel demeaned as I do by this arrangement that TVAAS has sucked us into for discussing these matters with them? We don't speak directly to them; our messages are carried to them as if to the great Wizard of OZ because they are too busy to be bothered by discussion--as if we who work here are mere slugs with nothing better to do with our time than nettle these great benefactors of society. There is greater combined statistical expertise among the participants in this forum than on the TVAAS staff, and those who are on the TVAAS staff have a professional responsibility to the discipline that extends beyond intermittent puffery and bluster.

From: Harvey Goldstein

I see that Les McLean and one or two others have taken up my query about how well schools can be separated taking the estimated standard errors into account. I don't yet know how the standard errors have been calculated, but based upon a table Sandra Horn sent me, I would say that the results (e.g. for grade 3 based upon a 3 year average in one of the larger school systems, are in line with our own results. What you do (roughly) is multiply each standard error by about 1.5, use this to place and interval (i.e. +-1.5 s.e.'s)about each gain estimate and judge whether two schools are significantly different at the 5% level by whether or not the intervals overlap. Most intervals do! BUT if you average over 3 years then you get smaller standard errors so fewer do. A particular problem with averaging over 3 years is that the data are even more out of date than using the latest 1 year data since they refer to a cohort who started x years earlier where x = years between intake measure and output measure + 2. Is such historical data of great use. One should at least be measuring trends over time. My own view is that value added procedures may have some use as a crude screen to detect highly 'overperforming' or 'underperforming' schools etc but are not diagnoses - such definitive judgements can only be made by studying what goes on in schools in more detail. A paper is shortly to appear in J. Royal Statist, Soc, A, which gives details of the interval setting procedure outlined above. Anyone who wants an advance copy please send me their FAX number.

From: Leslie McLean

This is an ASCII file, output from MINITAB, but it might arrive a mess or be so chopped up as to be unreadable. If you can read it, you see a simple analysis of the gains and standard errors Rick sent us from TVAAS. My observation that the gains were proportional to the standard errors does NOT seem to be true--within grades. If you lump all grades together, the correlation is over 0.5, but within grades (the correct plot, IMHO) it is essentially zero. Grade six shows a substantial NEGATIVE correlation, but there are only 12 observations.

What are these standard errors anyway? In a separate post to me, Greg Camilli points out that if all students are tested, then the "sampling error" has to be zero. What we need to make sense of this is, as I have said already, a technical report. How are they modelling the error in their multilevel models? What explanatory variables do they use? (Rick says that TVAAS says they allow for OTL ...) Do they include covariance terms? Is the "standard error" an estimate of measurement error? Just how much data is missing? etc... etc... etc...

 
 MTB > note     DATA ON GAINS AND STANDARD ERRORS BY GRADE FROM TVAAS
                A=Grade 3, B=Grade 4 ..., F=Grade 8
 
  stderr  -
          -          F                                A
          -
       6.0+
          -              B                        A
          -               C       A         A        A
          -             B    B        A     A     A         A
          -           B           A   B B       A
       4.0+          C     B  2 C2     B    B B  A A
          -              2      C   A CB 2AAAA       2A  A A
          -                 C B2 4C BBB     3B   A A
          -             CCCC C2CBBCCBA423   A A A        A
          -         C  CB  C 2 B2CC 3C  B3  B   A A
       2.0+                222  3C2B  BA       A
          -          F  F C CCCCC     B    A  A
          -   D  DDDD346EFF5  C  C 2
          -       D2  F E2  EE C
          -
            +---------+---------+---------+---------+---------gain
            0        15        30        45        60
 
 MTB > boxplot 'stderr'     (I believe at least one outlier is an
                             error--the one from Grade 8)
 
 
                      -------------
                ------I     +     I--------------      * *
                      -------------
           +---------+---------+---------+---------+---------stderr
         0.0       1.5       3.0       4.5       6.0
 
 MTB > info c11-c22   (The data, unstacked by grade--outliers in)
 
 Column   Name          Count
 C11      gain3            48
 C12      err3             48
 C13      gain4            48
 C14      err4             48
 C15      gain5            48
 C16      err5             48
 C17      gain6            12
 C18      err6             12
 C19      gain7            12
 C20      err7             12
 C21      gain8            14
 C22      err8             14
 
1
 MTB > plot 'err3' 'gain3'
 
          -
  err3    -
          -                                      *
          -
       6.0+
          -                                *
          -       *               *             *
          -              *        *        *              *
          -         *                   *
       4.0+       *                      *   *
          -           *      * ** **            2*   *   *
          -      *                **     *  *
          -             *** 2     *  * *             *
          -                  2*         *  *
       2.0+                *          *
          -                     *    *
          -
          -
            ----+---------+---------+---------+---------+-----gain3
               30        40        50        60        70
 
 MTB > corr 'err3' 'gain3'
 
 Correlation of err3 and gain3 = 0.188
 
 MTB > plot 'err6' 'gain6'
 
      1.40+
          -     *            *
  err6    -
          -                         *
          -
      1.20+                   *
          -               *
          -
          -                           *  *
          -                        *                   *
      1.00+                *
          -                     *
          -
          -
          -                                       *
      0.80+
          -
          -
            --------+---------+---------+---------+---------+-gain6
                  5.0      10.0      15.0      20.0      25.0
 
1
 MTB > corr 'err6' 'gain6'
 
 Correlation of err6 and gain6 = -0.598
 
 MTB > plot 'err8' 'gain8'
 
  err8    -
          -             *
          -
       6.0+
          -
          -
          -
          -
       4.0+
          -
          -
          -
          -
       2.0+
          -            *                *
          -              *  *        2*           *   *   *
          -   *                  *                *
          -
            --------+---------+---------+---------+---------+-gain8
                 12.5      15.0      17.5      20.0      22.5
 
 MTB > note   NOT MUCH POINT IN CALCULATING PEARSON R HERE, EH?
 
 MTB > copy 'err8' 'gain8' c51 c52;
 SUBC> omit 'err8'=6.0:8.0.           (get rid of the outlier)
 MTB > name c51 'err8trim'
 MTB > name c52 'gain8trm'
 MTB > plot c51 c52
 
      1.80+
          -            *
  err8trim-
          -
          -
      1.50+
          -                             *
          -
          -                                           *   *
          -
      1.20+                           *
          -                          *            *
          -              *  *        *
          -
          -   *                  *
      0.90+
          -                                       *
          -
            --------+---------+---------+---------+---------+-gain8trm
                 12.5      15.0      17.5      20.0      22.5
 
 MTB > corr c51 c52
 
 Correlation of err8trim and gain8trm = 0.039
 
 MTB > NOTE    STILL SEEMS TO BE AN OUTLIER (A DIFFERENT ONE)
 MTB > note       NOT PURSUED.

From: Rick Garlikov
Subject: TVAAS responses

From TVAAS:
All this is to say that I am looking at about a dozen posts regarding TVAAS tonight, alone. I don't know whether we can take on the philosophical discussions, simply because we don't have the time, but there are others among you who offer alternate viewpoints, eloquently. I honestly think we are only going to have the time to write responses about the model, what it does and how it does it, although I will provide all the answers I can about how TVAAS is being used by teachers, administrators, and others.

In closing, I would like to direct you to our article, "The Tennessee Value-Added Assessment System (TVAAS): Mixed-Model Methodology in Educational Assessment," in the _Journal of Personnel Evaluation in Education_ 8:299-311, 1994. It may answer some of your questions about the model.

Thanks for your interest in TVAAS and for your patience.

From: Sherman Dorn

The following are excerpts from the 1992 Tennessee state act creating TVAAS and the relevant sections ($$, if you'll excuse the ASCII notation):

$49-1-601.

(a) There shall be performance goals for each school district which shall include, but not be limited to, determinations based on the current status of each local school system as determined through the value added assessment provided for in $$ 49-1-603 -- 49-1-608.

(b) The goal is for all school districts to have mean gain for each measurable academic subject within each grade greater than or equal to the gain of the national norms.

(c) If school districts do not have mean rates of gain equal to or greater than the national norms based upon the TCAP tests (or tests which measure academic performance which are deemed appropriate), each school district is expected to make statistically significant progress toward that goal. The rate of progress within each grade and academic course, necessary to maintain compliance with $$ 49-1-209, $$49-1-210, and this part, will be established after two (2) years of consecutive testing with tests adopted for each grade and subject, as provided in $$49-1-603 -- 49-1-608. Schools or school districts which do not achieve the required rate of progress may be placed on probation as provided in $49-1-602. If national norms are not available then the levels of expected gain will be set upon the recommendation of the commissioner with the approval of the state board.

[$$49-1-209, -210 deal with administrative oversight by the state commissioner and board of education; it does not specify directly anything about TVAAS or specific items of accountability.]

(d)All schools within all school districts are expected to maintain appropriate levels of school attendance and dropout rates. The 1991-1992 school year is the base year for measuring levels of attendance and dropout rates. Schools which do not maintain appropriate levels, as set by the state board on the recommendation of the commissioner, may be placed on probation, as provided in $49-1-602.

(e) There is a rebuttable presumption that if a school or school district has not achieved the goals pursuant to subsection (c) or maintained attendance and dropout rates pursuant to subjection (d), it is out of compliance with the requirements of $$49-1-209, $$49-1-210, and this part and subject to probation as provided for in $49-1-602.

$49-1-602. ...

[lots of administrative detail about probation, then]

(c) ... If after two (2) consecutive years a system remains on probation, the commissioner is authorized to recommend to the state board that both the local board of education and the superintendent be removed from office. If the state board concurs with the recommendation, the commissioner shall order the removal of some or all of the board members and/or superintendent and shall declare a vacancy in the office or offices....

[procedures for filling the vacancies follow here]

$49-1-603.

(a) Value added assessment means:

(1) A statistical system for educational outcome assessment which uses measures of student learning to enable the estimation of teacher, school, and school district statistical distributions; and

(2) The statistical system will use available and appropriate data as input to account for differences in prior student attainment, such that the impact which the teacher, school and school district have on the educational progress of students may be estimated on a student attainment constant basis. The impact which a teacher, school, or school district has on the progress, or lack of progress, in educational advancement or learning of a student is referred to hereafter as the "effect" of the teacher, school, or school district on the educational progress of students.

(b) The statistical system shall have the capability of providing mixed model methodologies which provide for best linear unbiased prediction for the teacher, school and school district effects on the educational progress of students. It must have the capability of adequately providing these estimates for the traditional classroom (one (1) teacher teaching multiple subjects to the same group of students), as well as team taught groups of students or other teaching situations, as appropriate.

(c) The metrics chosen to measure student learning must be linear scales covering the total range of topics covered in the approved curriculum to minimze ceiling and floor effects. These metrics should have strong relationship to the core curriculum for the applicable grade level and subject.

$49-1-604.

[This section refers to several published articles, including one co-authored by Sanders in American Statistician, February 1991.]

$49-1-605.

[This section provides for annual estimates of school and district effects.]

$49-1-606.

(a) On or before July 1, 1995, and annually thereafter data from the TCAP tests, or their future replacements, will be used to provide an estimate of the statistical distribution of teacher effects on the educational progress of students within school districts for grades three (3) through eight (8). A specific teacher's effect on the educational progress of students may not be used as a part of formal personnel evaluation until data from three (3) complete academic years are obtained. Teacher effect data shall not be retained for use in evaluation for more than the most recent five (5) years. A student must have been present for more than one hundred fifty (150) days of classroom instruction per year or seventy-five (75) days of classroom instruction per semester before that student's record is attributable to a specific teacher. Records from any student who is eligible for special education services under federal law will not be used as part of the value added assessment.

(b) The estimates of specific teacher effects on the educational progress of students will not be a public record, and will be made available only to the specific teacher, the teacher's appropriate administrators as designated by the local board of education, and school board members.

[The following sections detail additional matters regarding test security and requiring fresh test items annually.]

________________________________________________

From: Sherman Dorn
Subject: Re: Laying down the law

>Please let there be a historian, investigative journalist, snooper or
>other seeker after truth who documents how this legislation came to be
>enacted in its present form!

I am an historian, but I have not had time to research this. (I moved to TN in late 1993, well after the relevant events.)

I gather (and the TVAAS folks can correct me) that Governor Ned McWherter decided to enact a wholesale reform of educational finances (including a plan to dramatically increase spending on education over several years) in 1992. The state senator who chaired the education committee (Albright is his name, I believe, though he was defeated in the primary last year) was very active in inserting the TVAAS language into the bill as an accountability mechanism. My assumption is that he, and other legislators, didn't want to spend a lot more money without some check on the "product." Very reasonable; I just disagree with how they went about it.

More detailed materials are at the state archives downtown in Nashville. Someday I may have a chance to consult the committee and whole-body legislative records.

From: Rick Garlikov
Subject: Re: TVAAS legislation

From TVAAS:

----------------------------Original message------------
We would like to thank Sherman Dorn for providing sections of the legislation regarding educational accountability in Tennessee. We would like to comment on a few sections. We are deleting the sections upon which we have no comment to save your time and space but refer you to Dorn's original post for the balance.

On Sat, 14 Jan 1995, Sherman Dorn wrote:

> The following are excerpts from the 1992 Tennessee state act creating
> TVAAS and the relevant sections ($$, if you'll excuse the ASCII
> notation):
>
>         $49-1-601.
>
>         (a) There shall be
> performance goals for each school district which shall include, but not
> be limited to, determinations based on the current status of each local
> school system as determined through the value added assessment
> provided for in $$ 49-1-603 -- 49-1-608.
>
Please note that the performance goals are not limited to TVAAS findings.
 
>         (b) The goal is for all school districts to have mean gain for
> each measurable academic subject within each grade greater than
> or equal to the gain of the national norms.
>
>         (c) If school districts do not have mean rates of gain equal
> to or greater than the national norms based upon the TCAP tests
> (or tests which measure academic performance which are deemed
> appropriate), each school district is expected to make statistically
> significant progress toward that goal.  The rate of progress within
> each grade and academic course, necessary to maintain compliance
> with $$ 49-1-209, $$49-1-210, and this part, will be established after
> two (2) years of consecutive testing with tests adopted for each
> grade and subject, as provided in $$49-1-603 -- 49-1-608.  Schools
> or school districts which do not achieve the required rate of progress
> may be placed on probation as provided in $49-1-602.  If national
> norms are not available then the levels of expected gain will be set
> upon the recommendation of the commissioner with the approval
> of the state board.

Please note that, in this and in all other sections of the law pertaining to placing schools and systems on probation, the language is permissive, not directive. Schools and systems MAY be placed on probation; superintendents and school board members MAY be removed. There is no language in the law that states that these actions MUST take place.


> [$$49-1-209, -210 deal with administrative oversight
by the > state commissioner and board of education; it does not specify
> directly anything about TVAAS or specific items of accountability.]
>
>         (d)All schools within all school districts are expected to
> maintain appropriate levels of school attendance and dropout rates.
> The 1991-1992 school year is the base year for measuring
> levels of attendance and dropout rates.  Schools which do not
> maintain appropriate levels, as set by the state board on the
> recommendation of the commissioner, may be placed on probation,
> as provided in $49-1-602.
>
>         (e) There is a rebuttable presumption that if a school or school
> district has not achieved the goals pursuant to subsection (c) or
> maintained attendance and dropout rates pursuant to subjection
> (d), it is out of compliance with the requirements of $$49-1-209,
> $$49-1-210, and this part and subject to probation as
> provided for in $49-1-602.
>
>         $49-1-602.  ...
>
> [lots of administrative detail about probation, then]
>
>         (c) ... If after two (2) consecutive years a system remains
> on probation, the commissioner is authorized to recommend to
> the state board that both the local board of education and the
> superintendent be removed from office.  If the state
> board concurs with the recommendation, the commissioner shall
> order the removal of some or all of the board members and/or
> superintendent and shall declare a vacancy in the office or offices....

Please note all of the safe-guards in this section. Both the Commissioner of Education and the State Boards of Education would have to agree to this very drastic action. It cannot happen "automatically."

==================================================================

From: Harvey Goldstein

Re Les Mclean's message of 13th about standard errors. He quotes Camilli as stating that the standard errors given are 'sampling errors' and that if all students are tested then these are zero. I am confused! The usual standard errors quoted in this context are those relating to the accuracy of the estimated school effects where there is a conceptually infinite population of students of whom those measured (whether they are all those in the school at a particular time or not) are a random sample. If they are not this then what are they?

From: Gene Glass

On Mon, 16 Jan 1995 15:10:34 -0500 Greg Camilli said:
Harvey Goldstein raised questions about what Greg Camilli said about "standard errors" thusly:

>>Re Les Mclean's message of 13th about standard errors.
>>He quotes Camilli as stating that the standard errors
>>given are 'sampling errors' and that if all students are
>>tested then these are zero. I am confused! The usual
>>standard errors quoted in this context are those
>>relating to the accuracy of the estimated school effects
>>where there is a conceptually infinite population of
>>students of whom those measured (whether they are all
>>those in the school at a particular time or not) are a
>>random sample. If they are not this then what are they?

Greg answered with a hypothetical conversation between an educator and a statistician. I think Greg exposed some key problems with this notion of standard errors, and it is no more a problem with TVAAS than it is a problem with most applications of inferential statistics in education.

Harvey asks, in effect, what is wrong with regarding standard errors as being measures of the accuracy of samples as representations of "conceptually infinite populations" from which the samples might "conceivably have been drawn at random."

After more than thirty years of calculating, deriving, explaining and publishing "standard errors" and their ilk, I have come to the conclusion that I don't know what they mean and I doubt seriously that they mean anything like what they are protrayed as meaning.

Consider this: if the population to which inference is made is one that is conceptually like the sample, then the population is just the sample writ large and the "standard error" is much larger than it ought to be. If you show me 25 adolescent largely Anglo-Saxon boys who love sports and ask me the population from which they could conceivably have been sampled, I'll conceive of an "infinite" population of such boys. If no population has actually been sampled and all I know about the situation before me is the sample, then I will conceive of a population like the sample. This is surely the very opposite of inference and standard errors are surely beside the point.

Consider something even more troubling: I present you with a sample-- Florida, Alabama, Tennessee, South Carolina. N=4. I calcualte the state high school graduation rates, average them and calculate a standard error. What is the population? States in the Southern U.S.? Fine; that's certainly conceivable, even if not "infinite." But suppose that someone else conceives of "States in the U.S." Well, that's conceivable too. But it is surely ridiculous to think that these four states can be used to infer to both of these conceivable populations with equal accuracy (standard errors). Or to make matters worse, suppose that I suddenly produce a fifth "state": Alberta. Now it raises the question whether the conceptual population is "geo-political units in North America"-- or the entire Western Hemisphere.

I can't imagine that there is much wisdom in attaching a number accurate to two decimal places when we can't even be certain whether it is referring to an "inference" to the Southern U.S., North America or the Western Hemisphere.

Now, if you think I am playing with your head and will suggest a way out of this dilemma that rescues the business of statistical inference for us, let me assure you that I have no solution. In spite of the fact that I have written stat texts and made money off of this stuff for some 25 years, I can't see any salvation for 90% of what we do in inferential stats. If there is no ACTUAL probabilistic sampling (or randomization) of units from a defined population, then I can't see that standard errors (or t-tests or F-tests or any of the rest) make any sense.

Does any of this apply to TVAAS? Just this. If one is worried about "stability" (in any of the many senses in which the word could be interpreted) then why not simply compare teachers' scores across all years for which data are available. That would answer in very straightforward ways whether the ranking of teachers jumps around wildly for whatever reasons or is relatively steady.

(I hasten to add that I don't approve of such things as ranking teachers with respect to their students' test scores.)

From: Harvey Goldstein

Well, I enjoyed Greg Camilli's imaginary conversation, but of course the reality is that standard errors are not things statisticians invented to make life difficult. Most non-statisticians have little difficulty in understanding that if you only have a measurement on 1 student there ain't much to be said about the rest. The bigger the sample the more confident you become that what you have observed is a good guide to what you would get on repeated samples with also suitably large numbers...assuming of course that you adopted a sensible randomly based sampling strategy.
Now we come to the philosophical bit. Social statisticians are pretty much forced to adopt the notion of a 'superpopulation' when attempting to generalise the results of an analysis. If you want to be strict about things then the relationship you discovered between parental education and student achievement back in 1992 from a sample of 50 elementary schools in Florida can only give you information about the physically real population of Florida schools in 1992. Usually we are not interested just in such history, but in rather more general statements that pertain to schools now and in the future...we may be wrong of course and that is why we strive to replicate over time and place etc. BUT the point is that, getting back to value added estimates for a school, if we want to make a general statement about an institution we do have to make some kind of superpopulation assumption....what we happen to observe for the students we have studied is a reflection of what the school has done, and would have done, for a bunch of students, given their measured characteristics such as initial achievement. The more students we measure the more accurate we can be and that's why we need an estimate of uncertainty (standard error).

From: Harvey Goldstein

Gene Glass also takes me to task on standard errors and raises the interesting question of when a sample should be considered as having a reference population and when not. There is no general answer...it depends on what you want to do. As I said in my response to Greg, I cannot easily see how you can have empirical social science without assuming that the units (people, schools etc) you happen to have measured are representative (in the usual statistical sense) of a (yes) hypothetical population whose members exhibit relationships you want to estimate. Such populations must (I think) be hypothetical because they have to embrace the present and future as well as the past when the data were collected. The issue is therefore the general philosophical issue and not a statistical one - statisticians simply try to provide tools for making inferences about such populations.

From: Greg Camilli

Harvey reponded to a previous post with the following:

>Well, I enjoyed Greg Camilli's imaginary conversation, but of
>course the reality is that standard errors are not things
>statisticians invented to make life difficult. Most
>non-statisticians have little difficulty in understanding that
>if you only have a measurement on 1 student there ain't much
>to be said about the rest. The bigger the sample the more
>confident you become that what you have observed is a good
>guide to what you would get on repeated samples with also
>suitably large numbers...assuming of course that you adopted a
>sensible randomly based sampling strategy.

Smaller is better, I agree. Another issue is whether it is the correct standard error, and still another is whether the SE has a meaningful referent. If the sample consists of all kids in the system, how can imagining a larger group possibly create more information. If I want to understand the behavior of my three car (I wish), how would it benefit me to imagine I had a fourth? IMO, this is not a statistical issue at all. Population has always been a heuristic device.

>Now we come to the philosophical bit. Social statisticians are
>pretty much forced to adopt the notion of a 'superpopulation'
>when attempting to generalise the results of an analysis. If
>you want to be strict about things then the relationship you
>discovered between parental education and student achievement
>back in 1992 from a sample of 50 elementary schools in Florida
>can only give you information about the physically real
>population of Florida schools in 1992. Usually we are not

Your option is to generalize to physically unreal schools.

>interested just in such history, but in rather more general
>statements that pertain to schools now and in the future...we
>may be wrong of course and that is why we strive to replicate
>over time and place etc. BUT the point is that, getting back
>to value added estimates for a school, if we want to make a
>general statement about an institution we do have to make some
>kind of superpopulation assumption....what we happen to
>observe for the students we have studied is a reflection of
>what the school has done, and would have done, for a bunch of
>students, given their measured characteristics such as initial
>achievement. The more students we measure the more accurate we
>can be and that's why we need an estimate of uncertainty
>(standard error).

I think you said in your ensuing post that this really isn't a statistical issue. Generalizing beyond known populations is risky business, and requires more than statistical knowledge. This was the focus of the long and interesting dialogue between Cronbach and Campbell. Standard errors have something to do with the precision of estimates. Perhaps they convey something about how well a model fits certain data. You might want to argue, on this basis, that the model is likey (or not) to generalize; but model fit at one instant does not *logically imply* model fit one second later.

The standard errors will apparently be used to measure whether statistically significant progress is being made by schools that fail to meet the standard (whatever that turns out to be), so it is important to be clear about what SEs mean. I find it fascinating that they are being used as policy tools with legal implications. In this regard, it is important to understand what drives the SEs. I'm guessing that missing data will add to SEs (it really would be helpful if the TVAAS staff would respond), and am sure that unit size will decrease SEs. Thus, standard errors for schools will typically be smaller for districts than for schools than for teachers than for students. As far as I can tell, only certain districts are required to make statistically significant progress; this may turn out to be a pretty easy criterion to satisfy.

From: Leslie McLean
Subject: The law demands std. errors

The discussion of standard errors has gotten so involved that a look at the Tennessee legislation should tell us where standard errors are needed and what interpretations reasonable people ought to be able to put on them. Below, the text is from Sherman Dorn's post and Les McLean's comments are in upper case letters.

(b) The goal is for all school districts to have mean gain for each measurable academic subject within each grade greater than or equal to the gain of the national norms.

HOW WILL ANYONE DECIDE WHETHER THE MEAN GAIN IS GREATER THAN OR EQUAL TO THE GAIN OF THE NATIONAL NORMS? PUBLICATION OF "STANDARD ERRORS" MUST MEAN THAT AN ERROR BOUND WILL BE ESTABLISHED AROUND THE NATIONAL NORMS--PERHAPS 1.5 TIMES THE MEDIAN STD. ERROR PER GRADE--ONE "HARVEY", OR 2.0 STD. ERRORS--ONE "DORN".

OK, GANG, THE VEIL IS LIFTED FROM OUR EYES--THERE IS NO SUCH THING AS "STATISTICALLY SIGNIFICANT PROGRESS" WITHOUT STANDARD ERRORS AND THE ASSUMPTION OF SAMPLES FROM SOME POPULATION.

Schools or school districts which do not achieve the required rate of progress may be placed on probation as provided in $49-1-602. If national norms are not available then the levels of expected gain will be setupon the recommendation of the commissioner with the approval of the state board.

YO, COMMISH! I DO NOT ENVY YOU YOUR TASK.

(a) Value added assessment means:

(1) A statistical system for educational outcome assessment which uses measures of student learning to enable the estimation of teacher, school, and school district statistical distributions; and

I COULD WRITE A RATIONALE FOR A "STATISTICAL SYSTEM" THAT DID NOT NEED STANDARD ERRORS, GIVEN THAT THEY TEST ALL THE STUDENTS. IT WOULD CONTAIN CAREFUL, MODERN DESCRIPTIVE STATISTICS THAT WOULD GLADDEN JOHN TUKEY'S HEART.

(a) On or before July 1, 1995, and annually thereafter data from the TCAP tests, or their future replacements, will be used (NOTICE THE 'WILL'-- THE LANGUAGE IS NOT JUST PERMISSIVE HERE) to provide an estimate of the statistical distribution of teacher effects on the educational progress of students within school districts for grades three (3) through eight (8).

HERE WE ARE AGAIN--THESE GAINS ARE TO BE INTERPRETED AS "TEACHER EFFECTS". PEACE, TVAAS, BUT I DO NOT BELIEVE THAT ANYONE'S MODELS AND TECHNIQUES ARE YET GOOD ENOUGH TO ISOLATE THE TEACHER EFFECT FROM ALL THE OTHER EFFECTS ON STANDARDIZED TEST SCORES IN SCHOOLS WITH ALL THEIR COMPLEXITY. NEXT TO THIS CONCERN--IT IS A CONCERN ABOUT VALIDITY AND IS NOT VAGUE OR COMPLEX--THE DEFINITION AND ESTIMATION OF STANDARD ERRORS IS TOO SMALL A MATTER TO TAKE OUR TIME.

From: Gene Glass

I quite agree with Les McLean that set over against questions of the validity of TVAAS "teacher effects" the matter of "standard errors" is of lesser importance.

Here's a validity question. I have asked it twice here--so that TVAAS could address it, but have not seen an answer.

A passage from the law makes it highly likely that what TVAAS does is measure pre-year achievement and partial (via some form of least-squares estimation) it out of post-year achievement for each teacher's class. Although this makes the resulting "gains" uncorrelated with pre-year achievement, it does not make them uncorrelated with pre-year ability-- abiltiy and achievement not being the same thing.

Consequently, teachers working with classes of different average ability can not be said to be evaluated fairly.

(This short discussion doesn't even raise very serious matters of "errors of measurement" in any pre measure that will also lead to a breakdown in "equating" the teachers at the start of the year. Nor does it raise questions of how a third grade teacher can teach skills or understandings that won't come to fruition on tests until the fourth and fifth grade. Does the TVAAS model really detect this and allocate proper credit to the source of the knowledge? And so on through similar concerns about validity.)

From: Sherman Dorn

The TVAAS staff noted, accurately, that the 1992 legislation is inclusive rather than prescriptive in how it describes the role of TVAAS in evaluation. Nothing forbids the state from using additional statistics as part of evaluation, either on the system level or for personnel evaluation. Nonetheless, I think it is clear from the text of the law that TVAAS was always intended to serve a central (and, from my reading, THE central) role in program evaluation, and that the threat of probation and public censure from TVAAS is the major stick in the accountability system in Tennessee. Consider these aspects of the legislation:

        (1) Only value added assessment is rigorously defined in the
        legislation, through citations to peer-reviewed
        literature and definitions of threshold levels for students'
        inclusion in teacher effects.  (By contrast, the law talks about
        attendance and dropout rates without ever defining them, or
        how to apportion responsibility for students.)
 
        (2) The only defined standard in the law (that of meeting
        national norm gains where available) is directly related to
        TVAAS.
 
        (3) Only three topics of measurement (TVAAS, attendance,
        and dropping out) were mentioned in the legislation.

Now, it may be that school systems, and the Board of Education and new Commissioner of Education, will create all sorts of additional statistics to be part of program and personnel evaluation. But from where I sit on this drizzly night, I think that's unlikely -- and, moreover, the legislation as it stands currently sends a very powerful signal, one that reinforces the high-stakes nature of the current set of annual tests in Tennessee.

From: Harvey Goldstein

Les McLean's comments have inspired some more thoughts.
In the simplest value added model, an outcome score is regressed on an input score so that generally each school will have a different regression line - perhaps with varying slopes but in the basic model with parallel slopes so that schools can then be ranked on the resulting regression intercepts. (The actual analysis is a bit more complex but this simple model captures the essence). We find, typically, that the variation among these intercepts is relatively small compared to the residual variation of student scores about the regression lines for each school (5% - 30% depending on which educational system you are studying). In addition, the regression itself will account for quite a lot of the variation in outcome...maybe as much as 50-60%. This means that there is a substantial remaining variation (among students) unnacounted for and it is this residual variation which determines the standard error values. Thus, e.g. if this residual variation was zero, we would exactly predict each schools (relative) mean and the standard error of that prediction would be zero. This would mean also that once we knew each student's input score (and anything else we were able to put into our regression model) and the school that student was in we would have a perfect prediction of the student's outcome. Of course, we are nowhere near that situation and it is this uncertainty about the individual prediction that translates into uncertainty about the school mean (think of the mean roughly as the average of the student residuals about the regression line for each school). If you took another bunch of students with exactly the same set of intake scores you would NOT therefore expect to get the same set of outcome scores - this is what the uncertainty implies - nor the same mean for the school. In the absence of being able to predict with certainty we have to postulate some underlying value for each school's mean (otherwise we are pretty well lost) which we can think of as the limit of a series of conceptual allocations of students to the school. Thus an estimate of uncertainty, conventionally supplied by calculating the appropriate standard error,is important if you want to make any inference about whether the underlying means are different and, more importantly, to set limts (confidence intervals e.g.) around the estimated difference for any two schools or around the difference between a school's estimate and some national norm. Hence my original remark some time ago that when you did just that you found that most institutions could not statistically be separated, and I suspect also for TVAA that very many cannot statistically be separated from a National norm, whether they are actually above or below it.
It would be good to hear from the TVAA people on this issue.

From: Rick Garlikov

Since Sherman and others read the Tennessee law he quoted the way they do, it IS extremely important for the state of Tennessee to make clear that is NOT the way it should be read.

That being said, it is either through different experiences or just a mere quirk of psychology, but *I* don't get the same meaning out of what Sherman quoted as he does, and as others seem to. Even TVAAS, in their commentary on the law, did not point out what I think is the single most important word -- "rebuttable". I don't have the language available at this time, but the phrase was something like "It is the rebuttable presumption that a school district should have [average gain scores...]..." and that if they don't, AND don't seem to be able to AND if there doesn't seem to be any compelling reason (i.e. rebuttal) why they don't, THEN ....

Further, the consequence then is that the state superintendent (or whoever -- I forget) MAY recommend to the state school board that people be removed from offices, and the State School Board has to concur with that recommendation.

If Tennessee is anything like Alabama in these sorts of matters, it would take an act of God for all these things to happen in a way that would actually end up in somebody's removal -- regardless of how abysmal their district's test scores are. There would be a bunch of excuses given that would count as "rebuttal" whether they were or not.

So, not only do I not see the language's saying what Sherman sees, but I don't see the POLITICAL reality's being what Sherman sees, though I understand his expectations sometimes are met -- and though I see that WITHOUT explicit and REPEATED policy statements to the contrary there can be various sorts of pressures to have tests drive the curriculum and instruction in undesirable ways. Where Sherman and I disagree is whether tests will drive the curriculum even if there is a clear policy STATEMENT to the contrary.

From: Leslie McLean

First: Please dissociate names of persons from std. errors (Harveys and the like). When I think of it, I wouldn't like to be named for an error either. Consider this one: -1 = One Les.

Second: Harvey Goldstein's exposition on standard errors (17 Jan, "Standard Errors: yet again") may have been more than some wanted, but I found it instructive and thought-provoking. If you deleted without reading, reconsider--it gets at the heart of the matter of TVAAS.
While still wanting to retain the concept of the sample from some (unspecified) population, Harvey's main lesson for us was to highlight the crucial role of the model adopted by the statistician in estimating scores--gain scores, in the case of TVAAS. A model is a formula that the statistician considers a reasonable try at relating the desired quantity, the 'gain' in achievement (not directly measurable because of nuisances such as social class and prior learning) to aspects of schooling, such as teacher competence.

Advised by statisticians with wide experience outside of education (and maybe in education--we have not been told), the policy-makers decide to give the statisticians their head and to accept their estimate of 'gain', knowing that the formula will be complex and the procedures well beyond the understanding of all but a very few. The statisticians make a persuasive case that their formulae and their procedures will provide the policy-makers with an estimate of gain that will distinguish the bad teachers from the poor from the average from the good from the excellent. "National norms" are invoked, unspecified, but responsibility given to the Commissioner of Education to provide norms if the national government lets the side down.
All this tedious repetition is needed to give a context for Harvey Goldstein's description of standard errors. In essence (correction, Harvey, please, if needed) the errors are S&E, not SE--errors of Specification & Estimation, not of sampling. A 'specification' error is made when our model, our formula, does not accurately link the target (the gain) with the data (the item responses or scale scores plus proxies for prior learning and social class and the like). We ALWAYS made a specification error--the only question is how large. If we limit ourselves, as in the TVAAS, to linear models, and we try to estimate gains across big, complex societies such as states, the error can be huge--and there is not consensus how to estimate the size of the error. Here is a source of error.
Even though they do not sample students and schools, sampling cannot be avoided--people are absent, times of testing vary, the tests cannot possibly cover all the content (hence content sampling), items are omitted, test booklets get lost, some teachers do not cover the material on the test, ..., and so on and so on and so on. This is why we do not use a very simple formula:
Gain = (Avg. score end - Avg. score beginning)
After all, when we test everyone, and when the goal is to measure gains by these students this year in these places with these teachers, who needs an error term? With well-constructed tests, the measurement errors will cancel out when we calculate school and class means. Oh--there is measurement error in individual pupil scores, but we can report that (from the test publisher's manual) and besides, these scores don't count in the student's grade--the teacher does not get them in time, and even if they do they do not use them.

Ok, so I seem to have lost the tenuous thread of the argument--NOT SO! We have learned over the years that the simple formula is more likely to mislead than to lead--to distort our view of gain rather than to clarify it. Raw score comparison tables (called 'League Tables' in the UK, after the rankings of sports teams), however compelling they seem, are statistically invalid, immoral, racist, sexist and stupid. Apart from those few flaws, they are fine. But would Tennessee put up with such poor procedures? Not on your life--scaling, imputation, hierarchical linear models and prayer are brought into play. Here is another source of error.

Pick up the tread again, the two of you who are still reading. All this talk of standard errors and models and politics keeps coming back to one key aspect: VALIDITY. Do those numbers represent gains in achievement? The formulas and procedures are complex enough that evidence is needed. Even if they do, how accurate are they--and I mean how much do they tell us about better learning, class-by-class, teacher-by-teacher; or has the TVAAS traded in science for voodoo? Without a better explanation, the use of these scores to label teachers as competent or incompetent seems a lot like sticking pins in dolls.

It is possible to validate the numbers--but it would take a lot of thinking, a lot of hard work and maybe 0.01 of the budget of TVAAS.

From: Greg Camilli

Harvey Goldstein wrote:

>In the absence of being able to predict with certainty
>we have to postulate some underlying value for each
>school's mean (otherwise we are pretty well lost) which
>we can think of as the limit of a series of conceptual
>allocations of students to the school.

I think we're lost when we accept statistical inferences based on data that weren't observed, and moreover, do not exist conceptually. If "all the students in the school" doesn't really have that meaning, then we are playing a game with language.

>Thus an estimate
>of uncertainty, conventionally supplied by calculating
>the appropriate standard error,is important if you want
>to make any inference about whether the underlying means
>are different and, more importantly, to set limts
>(confidence intervals e.g.) around the estimated
>difference for any two schools or around the difference
>between a school's estimate and some national norm.
>Hence my original remark some time ago that when you did
>just that you found that most institutions could not
>statistically be separated, and I suspect also for TVAA
>that very many cannot statistically be separated from a
>National norm, whether they are actually above or below

If we can get away from the superpopulation for a moment, we can begin to analyze what drives the standard error. It certainly isn't sampling error; nonetheless, it is a quantity that exists in a real sense. As you've implied above, SEs have something to do with model fit. Thus, we should be interested in those things that cause models to fit more loosely to the data. District size is certainly one factor; but correlation of effects within the model will also inflate SEs. Effects like teachers within schools, teachers with school, schools with district might be some examples. As Gene implies, separating these effects may take some doing.

From: Greg Camilli

Rick G.:
>  Greg says that his main concern is "how testing programs shape
>school culture and curricula."  Greg, cannot school culture and curricula
>also shape testing programs?  And couldn't the dominant cause and effect
>relationship flow in that direction, more than in the direction of the
>tests driving curriculum and instruction?  If not, why not?

For a number of reasons, testing programs usually affect schools, and not the reverse. First, testing programs are *intended* to shape curricula. Second, large-scale tests are mainly written to assess only skills that cut across curricula, perhaps with the message "what should be taught." Thus, the individual character of schools and districts can get lost. Maybe you mean "But isn't it possible that they affect testing program?" to which I would answer maybe it happens somewhere, in some superpopulation, but not where I come from. Perhaps in an ideal world.

Maybe you ask "Wouldn't it be possible to change testing programs so that they address the needs of schools" to which I would reply "Yes." But I hasten to add that there is a distinction between local and global testing. I'm sure a lot of really good district assessment programs are in place. If you imply that far from being agents of change, global testing programs may themselves need to be reformed, then I would also agree.

From: Rick Garlikov
Subject: Response from Bill Sanders, TVAAS

From TVAAS and Bill Sanders. Regarding TVAAS:
Several recent queries have dealt with the model(s) used in TVAAS. Anyone interested in learning specifically how the model--in particular, the teacher model--is defined and how the estimates are obtained, there is an explicit definition in the paper cited in an earlier post, "The Tennessee Value-Added Assessment System (TVAAS): Mixed-Model Methodology in Educational Assessment" (Sanders and Horn, _Journal of Personnel Evaluation in Education_ 8:299-311, 1994) on pages 305-309. For those of you interested in how standard errors are obtained, property 2(c) on page 306 of the same article details that information. If you are unable to obtain a copy of this article, you may write to the UT Value-Added Research and Assessment Center, P.O.Box 1071, Knoxville, TN 37901-1071.

To Gene Glass: I hope that reading this article will give you a more accurate picture of what we are doing. After you have looked at it, you may want to restate your questions.

To Leslie McLean: Your plots of standard errors as calculated make no sense. Middle schools in the example school system we provided have more students than intermediate schools in almost every case. Thus, their standard errors tend to be lower. Middle schools also have smaller expected nominal gains. Therefore, your attempt to show a relationship over grades is nonsense.

To Harvey Goldstein: Thank God for you and your insights. I am glad to learn that you in the UK are obtaining comparable sensitivities using similar approaches. As evidenced by the estimate mean gain and the relatively small standard errors, clearly many schools can be distinguished as deviations from the average school within a district. Even though this is not the objective of TVAAS, it does show relative sensitivity. To those of you worrying about conclusions reached from a specific type of test, let me share with you recent findings (as yet unpublished) resulting from the merging of data from assessment instruments other than TCAP into the master database at UTVARAC. We have recently completed merging the 10th grade PLAN (previously known as the PACT) and the 12th grade ACT scores for the last three years and the Tennessee Writing Assessment data into the master database. This database is now comprised of more than three million records. What we have found is that the differences among school systems in the scores of 10th and 12th graders are huge, even after holding 8th grade achievement level constant. Further, the findings are relatively consistant regardless of the test data used. Additionally, writing Assessment data is being analyzed in conjunction with the more traditional forms of testing to evaluate how much unique information is available from the writing assessment. This work will continue, and we will be writing and reporting more of it in the next several months. To those of you who are concerned about the understanding of the TVAAS reports by Tennessee educators, let me share observations based on scores of phone calls and numerous presentations across our state. Educators' understanding and attitudes vary widely. In those systems in which superintendents, supervisors of instruction, and/or principals have worked to learn and share the diagnostic value of the analyses, positive atitudes and progressive plans are leading to improved academic growth opportunities for thousands of Tennessee youngsters.

As educators strive to improve gains, they have identified the following practices as some of the primary impediments to be overcome: 1) excessive re-teaching; 2) failure to communicate over grades; 3) the enormous effects of building change (see Sanders, W. L., et al.(1994). Effects of Building Change on Indicators of Student Academic Growth, Evaluation Perspectives_. 4.3-7); 4) "lock-stepping" instruction to the detriment of many high achieving students, etc. etc.

Finally, for those of you who have complained about a timely response to technical matters relating to TVAAS, if you could have forwarded a magic cure for flu to me two weeks ago, then my responses would have been far more punctual.

As I stated in my original response to Dorn, we will respond to legitimate criticism from all of you to the best of our ability. That still stands. However, I do not feel it is necessary for me to rewrite and post to this forum those things which have been previously published and rigorously reviewed. We will furnish citations, instead.

From: Greg Camilli

Les,
I think your distinction between SE and S&E is a clear and elegant statement. It is a must-read for anyone interested in how statistical models are likely to behave in policy contexts. I'd like to throw in two additional cents:

1. Because some statistical models are complex, and understood by few, it is ironic that this initially evokes more (rather than less) credibility. The downside (or upside depending on how you look at it) is that when a small crack in the model's facade appears, the public and policy makers can be very unforgiving. A relatively small equating anomaly in a New Jersey state test nearly caused the demise of the testing program. Moreover, when such a crack can be patched, it creates an atmosphere in which technical personnel are *less* motivated to diagnose future problems.

I think TVAAS is certain to encounter a related problem with its "linear metric." How is it, the press may ask, that gains are so much larger in the earlier than the later grades? Does this mean that students aren't learning very much in high school? Moreover, because the standard errors are likely to be different across districts, larger districts might have to achieve smaller gains to be consistent with the law. Does this imply different standards for different districts? (I recognize that larger districts have to pull up more kids to achieve a SE's worth of gain -- but I'm not sure this type of argument would wash since a SE may be only a baby step toward the national average.)

2. The "natural" sample that exists on any given day does, I suppose, give rise to a superpopulation of the sort that Harvey Goldstein writes of. However, this is not the population about which most people think of when evaluating gains since, as Bill Hunter points out, it is not a random sample from the school's student body.

From: Bill Hunter

Per Greg Camilli:

> 2. The "natural" sample that exists on any given day does, I suppose,
>    gives rise to a superpopulation of the sort that Harvey Goldstein
>    writes of. However, this is not the population about which most
>    people think of when evaluating gains since, as Bill Hunter points
>    out, it is not a random sample from the school's student body.

I need to clarify a bit. I think it is not the case that a sample of convenience "gives rise to" or "implies" a population of any sort (unless one chooses to regard the sample _as_ a population). As far as I can tell this thinking is exactly backwards--samples derive their meaning and existence from populations: I cannot see that the reverse order has any meaning at all. I also question the utility of Harvey G.'s conception of such samples as samples from a population in time. This _might_ make sense in a time/space of great stability, but I see little reason to believe that children four or five years from now will have experiences of the world (especially the world of information) that is comparable to children of today (or five years past). The kinds of changes that required revision and re-norming of intelligence tests every 15 or 20 years half a century ago now take place in five years or less--probably about the same time scale that would be required to conscientiously develop and renorm the test.

Moreover, I think it is not just that such a sample is not a random sample from some _specific_ population (as Greg suggests above), but that it is not a random sample of ANY population for two reasons: 1) the process of selection did not insure equal and independent likelihood of selection for all members of the population and, more importantly, 2) no population was specified (to which the above process was not applied) (Sorry about the double negative. I'll have to stop watching Seinfeld.)

From: Leslie McLean
Subject: Error plots: clarification

On January 18, Bill Sanders wrote (via Rick Garlikov--and along with many other topics):

>To Leslie McLean:  Your plots of standard errors as calculated make
>no sense.  Middle schools in the example school system we
>provided have more students than intermediate schools in almost
>every case.  Thus, their standard errors tend to be lower.  Middle
>schools also have smaller expected nominal gains.  Therefore, your
>attempt to show a relationship over grades is nonsense.

It was indeed the point I was making--that the plot (or correlation) over grades made no sense. That is why I argued that the within-grade correlations were the ones to look at--and that they were around 0.0. BTW, if means in a table are based on widely different Ns, you would do your readers a good turn to say so, don't you think? Your remark that "middle schools also have smaller expected nominal gains" is ambiguous and interesting. In what sense "expected"; in what sense "nominal"?

From: Sherman Dorn

>The Tennessee's 1992 Educational Improvement Act lists several
>references pertaining to the technical aspects of TVAAS's "mixed
>model methodology" for value-added learning assessment.  In addition
>to the citation coauthored by Professor Sanders (posted 1/18/95) are
>the following:

According to both Sanders & Horn (1994) and MacLean et al. (1991), the Henderson article is the root of the mixed model Sanders and MacLean developed. (I also think Sanders mentioned this in a previous post.) I would hope that someone else, preferably with some more appropriate statistical background than I, would take the time to read the Sanders & Horn, MacLean et al. (American Statistician, 1991), and perhaps Henderson or the SAS Stat chapter on their PROC MIXED in order to help us here.

Conceptually, I gather the rationale for using a mixed model on gain scores is as follows: using raw scores is unfair to evaluate teachers because of initi capacity of students to perform on tests. Using just gain scores is better, but one needs to view various "effects" as random because of (a) individual students' varying responses to a teacher, and (b) measurement error. According to Sanders and Horn, the mixed model solves the problem of regression to the mean and gain scores through the viewing of effects as random. They also claim that the calculus behind TVAAS is able to use partially-censored student records, eliminating the problems with complete-data analysis when, for example, Joey had the flu on the day his school was conducting the math tests and thus all scores for him would be thrown out.

My statistical concerns about this are less with the model (though I've tried to puzzle through the matrix algebra) than with how it connects with real-world schooling and learning. Gene noted a long time ago his supposition that one would expect low-performing students on tests to have lower gains than students who initially perform highly on the tests. TVAAS folks have vigorously disputed this, claiming that their records indicate low-performing students can expect, on average, equivalent gains to higher-performing students. I am assuming, though I have not asked, that this information is individual student scores, rather than an ecological view through teacher or school effects measured against average initial scale scores.

My concern is the use of gain in SCALE scores, which are derived from a cross-sectional 1989 norming of the test used for TVAAS -- the CTBS. To put it simply, the third grade norming population is not the equivalent of the second grade norm population aged a year. (Not only do age cohorts' experiences differ, but flunking students and migration change the age-grade composition of grades.) Thus, we really don't know whether a scale score of, say, 700 on the third-grade CTBS would be the same thing as if the 1989 second grade had been tested a year later in the third grade. Probably both the mean and the standard deviation would be different, which means that scale scores would undergo some linear transformation. For this reason, as well as Gene's concern, I have suggested to UTVARAC staff that they use previous year scores as an independent variable, with this year's scores as the dependent variable, rather than the gain score approach. According to them, adding a variable and rerunning the entire system would not be very onerous at all. Thus far, UTVARAC staff have not responded to this suggestion.

=========================================================================

Sherman Dorn:
Now, as an historian, I would be remiss if I didn't point out that Tennessee already had high-stakes testing well before 1992 and the Educational Improvement Act. And TVAAS is not, in itself, a test. It is a statistical system for analyzing test scores. My concern is that TVAAS' existence will cement in place any mutual distrust between teachers and policymakers that currently exists, and exacerbate organizational problems that TVAAS cannot respond to. I should note, also, that with Lamar Alexander as governor in the 1980s, Tennessee was at the forefront of the earlier wave of school reform, with higher graduation standards, the career ladder program, and the first wave of high-stakes testing. As with the others, I doubt that TVAAS will facilitate much improvement, in its current form at the center of educational evaluation.
Sherman Dorn
Vanderbilt University

=========================================================================

From: Gene Glass
Subject: Glass Replies to William Sanders about TVAAS

On Wednesday, January 18th, William Sanders wrote the following in response to my repeated requests that he address, among other questions, the matter of how the TVAAS deals with ability differences in students who happen to appear in particular teachers' classes:

"To Gene Glass: I hope that reading this article will give you a more accurate picture of what we are doing. After you have looked at it, you may want to restate your questions." (Sanders)

I don't appreciate being fobbed off in this manner and I am in an even more atrabilious mood for having traipsed to the library to copy the article referred to (Journal of Personnel Evaluation in Education 8:299-311, 1994). It contains no adequate information on the question I asked. Indeed, it contains six superficial pages on teacher evaluation and a four-page statement of a completely unextraordinary mixed effects linear model. (I too, Mr. Sanders, can write mixed effects models--Glass & Hopkins, Statistical Methods in Education & Psychology, 2nd Edition, 1984, pp. 465-474; so don't imagine that I am intimidated when someone flashes four pages of equations in my face instead of being responsive to my inquiry.)

The question is whether the TVAAS system takes adequate account of differences in student, and hence, school class ability (as measured by generally accepted ABILITY tests). An examination of Sanders and Horn's article reveals clearly that it does not; I repeat, it does not. In my opinion, the implementation of the system is unfair to teachers for this reason, and very likely other reasons.

The Sanders and Horn article makes a few references to student ability:

1. An early 1980s application of the method "rendered the following findings: .... 5. Student gains were not related to the ability of achievement levels of the students when they entered the classroom." (p. 300) The documentation for this peculiar finding (which if taken seriously would, of course, imply that bright and dull students make the same achievement gains in a school year--a palpable and self- contradictory absurdity) is an unpublished report (Working Paper No. 199) from the UT College of Business Administration (McLean & Sanders, 1984). (It is an irony in need of clarification that the McLean with whom Sanders has collaborated on some statistics articles is not the McLean (Les) who has here taken serious issue with the TVAAS approach.) I don't consider unpublished working papers as adequate documentation for so extraordinary an assertion. They are unreviewed and unpublished. So reference to the article is no more helpful than is repeating the unsubstantiated assertion made weeks ago here that student ability is unrelated to teacher effects in the TVAAS system. (How, I might ask, does TVAAS imagine that the positive correlation between ability and achievement arises in the world if more able students do not learn at a faster rate than less able ones?)

2. In the section of the paper entitled "Problems of using Student Achievement Data in Educational Assessment," Sanders and Horn write: "Since random assignment of students to teachers is usually not practiced and seldom is possible, simple means of class achievement test scores are seriously biased by many factors other than teacher influences that affect student learning. Travers (1981) listed (1) teachers influences, (2) parental influences, (3) genetic endowment, (4) other school influences, and (5) availability of materials as being some of the most important factors that determine the rate of student learning." (p. 304) The reference to Travers (1981) is to a chapter (referenced by page numbers or title) in the Handbook of Teacher Evaluation edited by Jason Millman, not "Millmen" as Sanders and Horn report.

3. Sanders and Horn go on to write at the bottom of page 304 that "Obviously, any system that will fairly and reliably assess the influences of teachers on student learning must partition teacher effects from these and other factors." The reference to "these and other factors " being to both Travers's list and Bingham et al.'s unremarkable assertion that student test scores are influenced by many things including family characteristics, personal characteristics and the like. They go on: "However, it is a hopeless impossibility for any school system to have all the data for each child in appropriate form to FILTER (emphasis added) all of these confounding influences via traditional statistical analysis." (p. 304)
We learn two things here: 1) that Sanders and Horn will soon tell us that they are not going to employ any measures of these student background characteristics in their system and that they regard their statistical analysis as more than merely "traditional."

4. The next paragraph delivers the news: "Using a different approach, the three studies conducted by Sanders indicate that these influences can be FILTERED (emphasis added again) without having to have direct measures of all of the concomitant variables. By focusing on measures of academic gain, each student serves as his or her own 'control'--or in other words, each child can be thought of as a 'blocking factor' that enables the estimation of school system, school, and teacher effects on the academic gain with the need for few, if any, of the exogenous variables." (305) This is an incredible and patently ridiculous claim. It says that "gain scores" gotten by correcting posttest achievement scores via least-squares mixed model estimates from pretest achievement scores can be safely assumed to control for "exogenous" (merely, in this context, "not measured") variables such as social class, race, intelligence, culture and many others. It is said that pretest achievement scores will be the "filter" through which these exogenous influences operate on the posttest scores, hence, the student background characteristics can be ignored. If this is an assumption of the system, it is clearly contestable and nearly certainly false; if it is a belief about what accounts for variation in achievement scores, it is incompetent.

5. They go on: "In an attempt to partition the teacher and school effects from the partial confounding with class ability level, the well-known linear model techniques of analysis of covariance and ordinary multiple regression have been suggested by Millman (1981) and others. The obvious intent was to adjust differences that exist among students to enable a fairer evaluation of teachers. However, if these simple approaches are applied, and even if all of the concomitant data were available, still unanswered is the well-known problem of regression to the mean of the teacher effects that would provide unfair rankings of teachers with varying quantities of student achievement records." (p. 305) Millman (spelled correctly this time) is cited but the bibliography contains only the citation "Millman, J (ed.). (1981). Handbook of Teacher Evaluation. Beverly Hills: SAGE." I presume that Sanders and Horn are referring to a specific chapter in the Handbook that Millman edited, though they don't cite one. I also assume that the chapter in question is in fact Millman's own chapter on using student test scores to evaluate teachers. When Millman and Darling- Hammond edited the Second Handbook on Teacher Evaluation, Jay Millman asked me to write the chapter on Using Student Test Scores to Evaluate Teachers. I will be happy to send an email copy of this chapter to anyone who requests it. As those who know me might guess, I had nothing good to say about the practice. But back to Sanders and Horn--the last sentence quoted above beginning with "However, ..." is simply unintelligible to me. What is clear, however, is that far from lacking "all of the concomitant data," the TVAAS has NONE of the concomitant data.

It is clear in the Sanders and Horn piece that they believe that their mixed model estimation procedure solves problems of unmeasured exogenous variables (ability, social class, race and the like) and provides a fair comparison among teachers: "If the problem (of estimating teacher effects) is viewed not as a fixed-effects problem but rather as a mixed-model problem with both fixed and random effects, then much established theory and methodology exist that offer solutions to many of the problems that have been cited as reasons for not doing educational outcome assessment from student achievement data." (p. 305) This is absolute nonsense. If information on intelligence of the class of students, their family life and the like is not measured and included in the model, it does not somehow magically appear compliments of solving the normal equations. The only way that such background influences can be assumed to enter the TVAAS system is through the "filter" of pretest scores; to think that this "filter" is sufficient to correct for the background characteristics is simple fantasy.

It is clear from reading the TVAAS responses to questions posed here over the past few weeks and from reading their latest published exposition of the method that they have no appreciation of the validity question whatsoever. Nor do they appear to have studied and taken into account the psychometric literature on these problems, which is abundant and well-known: Harris, C.W.(ed.) (1963) Problems in Measuring Change. Univ of Wisconsin Press; Cronbach, L.J and Furby, L. (1970). How do we measure "change"-- or should we? Psychological Bulletin, Vol. 74, 68-80. Cronbach, L.J. (1982) Designing evaluations of educational and social programs. Jossey-Bass. These are a minimum set of references for grappling with these problems.

From: Greg Camilli

Communication does seem to be a problem lately. Rick doesn't understand our differences (or those with Tom and John). I don't understand them, and Rick's latest communication is even more baffling to me. I also don't understand Harvey's position on superpopulations, though doubtlessly generations of statisticians assume this as axiomatic. (As for William Sanders, I wouldn't classify his post as serious communication at all. I will, however, read the articles in good faith.)

In a strange way, Rick and Harvey are saying something similar. Harvey talks about superpopulations; these are entities that don't exist, except in the imagination. Yet it is contended that it is a "reality in the sense that further batches of students are samples from it. How else would you make sense of anything?" A lot of people have sought to answer this question, among them Alan Birnbaum who paraphrased the likelihood principle as the "irrelevance of outcomes not actually observed." He went on the write of the "immediate and radical consequences for the everyday practice as well as the theory of informative inference.' As for the superpopulation, it exists in ones mind as a vehicle for generalization. But generalization itself requires more worldly knowledge. For example, consider the standard error of statistic calculated from a poll during an election. You might say a population exists, but only for a limited amount of time. Experience with the rate of change in public sentiment (and the way the question is asked) is required for a valid generalization. Happily, however, we are in full agreement on the role of specification error, as masterfully articulated by Les. (Because William Sanders thanked God for Harvey, one assumes he also agrees on this point. I'm also thankful for Harvey's participation.)

Rick has a world in mind where he suspects that the rest of us presume that "of course teachers will try to psych out the test, and they ought to do that for their own personal protection." Some of us "might even think that this is the best way to get good scores on such tests or that it is natural for many teachers to believe that -- even if they feel it is an improper way to teach. I have been assuming that it is NOT the best way to get test scores, and it is NOT the best way to teach." And further "In any case, it seems to me to be a dishonorable and a very strange for a profession as a group to acquiesce to a bad policy in such a way that they spend more effort coping with it than trying to correct it. You make it sound like the legislature, the media, and the public do not care whether tests are predictable or not. I don't think it is idealistic to think that the public's or media's interest is that unconcerned about this whole thing."

Rick, the educational profession cares that teachers do not teach to tests because it dilutes the content of the curriculum. Most people in testing programs also believe this. It seems to me that you are arguing with someone who thinks tests should be used to dumb down the curriculum. I believe as you believe, but not all you believe. I also think tests can be good or bad, taught to or not taught to, and that some may cheat while others struggle to maintain integrity. And I think that one can label anything as political. It's a term that is often used in lieu of a sensible explanation of how something has come to pass, and what can be done about it. Most people think this way. (There, I've done it, I've created a superpopulation.)

When I read your messages, it strikes me that you are reflecting on how people are thinking and the language they are using, rather than the content of the messages. The blanket characterization about educators is out of line, in my opinion. I think we are trying to expose and correct bad policies, that's what this discussion is about, and it is not without precedent. In 1988, John Cannell published an article "Nationally Normed Elementary Achievement Testing in America's Public Schools: How All 50 States Are Above the National Average." Mr. Cannell is a doctor (MD) not a measurement specialist. This article (whose gist is given in the title) stirred widespread attention inside and outside the academic community. The effect he noted is now widely called the "Lake Woebegone" effect. Anyone familiar with this article, knows the public is interested in good testing practice. The problem has been researched extensively, in both universities and testing companies. The results are clear: testing programs, whether the test is good or bad, can have unintentionally harmful effects.

Perhaps the communication problem results from a "type" mismatch of our arguments based on experience with your arguments based on a more technical sorting of language and logical form. Finally, I'm not bemoaning TVAAS. I don't yet know how well the program works, but I do ask for information and I am skeptical. Moreover, the TVAAS staff can probably learn more from our skepticism than warm congratulations on a job well done. They too are involved in a sorting language (say test scores) and logical form (say statistical model) and could benefit from our experience.

From: Michael Scriven Scriven@AOL.COM

Aha! At last the battle has been joined. I look forward to a response by Bill S. However, I do hope that both parties, or at least the spectators, can keep in mind the need to go forward with the basic question: is there a feasible method to get at least an approximate estimate of the extent to which teachers are contributing to student learning?

Whether Bill has got it or not, there's a worthwhile task here-getting the best feasible model. If it's not very good, that isn't important. As long as it's better than failing to bring in outcomes to teacher evaluation, and not too expensive, then it's worth having. Remember that even now no teacher is at risk on the basis of the TVAAS results alone: there has to be other confirmatory data. (Sure, we need to look at whether that gets biased by knowledge of the statistics results, but if not, it can easily be done independently.)

So I hope we can keep our collective eyes on the goal of getting an idea of whether any errors here are correctable to the point where we have a useful device for correcting the appalling alternative of judging teaching without reference to outcomes.

Michael Scriven

From: Greg Camilli
Subject: CTB Scale scores

I thought that some of you might want to take a look at some statistics regarding the metric of the scores that TVAAS uses. Below, I've given the mean, median and standard deviation of the IRT metric for fall reading comprehension as reported in the CTBS/4 Technical Bulletin 1 (1989).(I hope this isn't too far out of date.)

Grade    mean    median     STD
 
1        473     481        84.3
2        593     606        81.1
3        652     657        59.6
4        685     694        53.6
5        707     714        48.6
6        725     730        43.8
7        733     738        43.6
8        745     750        43.1
9        760     764        38.6
10       770     774        39.6
11       776     780        38.2
12       780     782        38.0

If you plot these data by grade, some interesting possibilities emerge. For example, one wonders why students below average gain as much as students above average. The explanation I see is that there is much less room for growth at higher grade levels, but this is a function of the scoring metric. A transformation of scale might lead to different results.

From: Sherman Dorn

Michael Scriven writes:

>Whether Bill has got it or not, there's a worthwhile task here-getting the
>best feasible model. If it's not very good, that isn't important. As long as
>it's better than failing to bring in outcomes to teacher evaluation, and not
>too expensive, then it's worth having.

This assumes that (a) there is no alternative to TVAAS to bring outcomes into teacher evaluation, and (b) those of us who criticize the present place of TVAAS in Tennessee official policy [and the potential for similar statistical systems elsewhere] are therefore against judging teachers by what students learn. Neither is true. (Counterexample: go read the issue of Exceptional Children, vol 52 [1986], on formative evaluation.)

Also, the state of Tennessee has spent millions over the past ten years, including the money spent to develop TVAAS, in having students complete annual high-stakes tests. I would call that rather expensive.

Furthermore, the state of Tennessee has, through its promotion of high-stakes testing, demonstrated to its teachers the profound mistrust policymakers have of them, further eroding the legitimacy in teachers' eyes of ALL attempts to judge them by students' outcomes. I would call that rather expensive.

>Remember that even now no teacher is
>at risk on the basis of the TVAAS results alone: there has to be other
>confirmatory data. (Sure, we need to look at whether that gets biased by
>knowledge of the statistics results, but if not, it can easily be done
>independently.)

What you've written reads to me as the following: "I don't care if TVAAS is a bad model. It's a model. Besides, it's a model that doesn't really matter. So let's go on working with it." This is beginning to sound like the kettle defense. ("I never touched the kettle; I only borrowed it; it was broken when I first took it.") I hope that's not what you meant.

From: Leslie McLean
Subject: Validity, again (no apology)

As I have come to understand the TVAAS, it goes like this:

These numbers are measures of gain in student achievement over a school year, and our PROCEDURE entitles you to read them as measures of teacher competence--the larger they are, the better the teacher, and v.v.

Many of us, based on lots of experience, are extremely skeptical of this claim, and so far the explanations have only increased our skepticism. The procedure is SO questionable (from test content to scaling method to model-fitting and adjustment) that any interpretation such as I have given above should be withheld pending supportive evidence--talk about "meaning and values in measurement and evaluation"!

As many posts have argued, the TVAAS numbers become the only evidence to get attention, not because the TVAAS folk say so but because the sheer size and bulk and prestige of the enterprise drive out other measures. Messick ended his speech with a quotation from one Liam Hudson (1972), from his book, The Cult of the Fact (p. 125). Hudson argued that social science:

     ...should be pictured not as a society of good men and true,
     harbouring the occasional malefactor, but rather, as one in
     which everyone is searching for sense; in which differences
     are largely those of temperament, tradition, allegiance and
     style; and in which transgression consists not so much in a
     clean break with professional ethics, as in an unusually high-
     handed, extreme or self-deceptive attempt to promote one
     particular view of reality at the expense of all others.

More strenuous efforts would appear to be required to avoid such a transgression in Tennessee.

From: Leslie McLean
Subject: More on validity (with apologies)

All the posts about statistics, including Gene Glass's response to Bill Sanders, may have left some of you wondering whether it all matters, or what it means. Michael Scriven reminds us that "outcomes" (by which he means what students learn) must not be left out of the data when the competence of teachers are assessed. He gives us a version of what I heard Patrick Suppes say many years ago, "Anything worth doing is worth doing badly". Suppes (a mathematical logician by trade) was quite serious, but his context was the very early days of computer-assisted instruction (remember CAI?). The context today is VERY different--high-stakes testing programs with real consequences for teachers and school officials.

Peace, Michael, but I do not think we can advocate any measure of teacher effectiveness that has not been tested and validated against several different ways of judging teacher competence. Patrick Suppes was working at the leading edge of research, with no serious consequences for anyone if he was wrong. He made many contributions, and quite a bit of money, and no one should be critical of his entrepreneurial spirit and his creativity. Neither should we confuse the context of the late 1960s with the legislated accountability of the middle 90s. Oh, I know, you work with Dan Stufflebeam all the time and you know all about the middle 90s. Your post does not reflect this knowledge. (As you will see, I do not believe that we yet have a measure, or set of measures, valid enough to include in the evaluation of teacher competence.)

So what about all this statistics talk? Gene Glass has indicated his reservations about TVAAS's "mixed models" (perhaps I understate the case). What ALL the regulars on Edpol should consider (those that have not already done so) is that statistics has moved on well beyond "mixed models", in response to the complexity of social groupings--ESPECIALLY students within classes within schools within districts within states (within countries!). Since Gene has listed his excellent book (with Hopkins), let me cite my own contribution to the post-mixed-model literature, an application of multilevel models (with the essential contribution of Harvey Goldstein): McLean, et al., (1988) The reliability of the oral examination in internal medicine. Journal of the Royal College of Physicians and Surgeons of Canada, ... ...). The study is in education (of medical specialists) and is in the mainstream of generalizability theory.

There have been two main channels of development and two major tributaries to better understanding of school achievement:

Murray Aitken and Nick Longford's early work, parallel to Harvey Goldstein's (e.g., 1986, J. Royal Stat. Soc. B, 149: 1-43). Tributary.
Harvey's book, the first, Multilevel Models in Educational and Social Research. New York: Oxford U. Press, 1987. Harvey's group at the Univ. of London Institute of Education offers computer software (ML2 and ML3) and training. Mainstream.
Bryk and Raudenbush, at about the same time: Bryk et al. (1986) An introduction to HLM: Computer program and user guide. University of Chicago. (HLM: Hierarchical Linear Models--see previous Edpol posts). B&R and Co. offer computer software for fitting two-level models and do extensive training. Mainstream.
Nick Longford and Murray Aitken's later work, extending their earlier models and offering computer programs. Tributary.

So what? We have been led to believe that the TVAAS utilise the latest developments in statistical models in order to produce estimates of gain that truly reflect what students are learning. Yet the published sources cited fail to support this claim--in fact the sources fall quite far short. Either Dr. Sanders and his associates are hiding their light under a bushel or their claims are not supported--these seem to me the only alternatives. Throughout the interchange on Edpolyan (I hesitate to call it a discussion, given the reticence of the TVAAS folk), a nagging suspicion has grown among those of us who struggle with test scores and models thereof that the TVAAS structure is a house of cards. Dr. Sanders will no doubt regard this as another bit of "nonsense" from me, but his contributions have yet to add to the "sense". Given the salience of the system in Tennessee, we all hope to have the sense revealed to us.

From: Scriven@AOL.COM

Of course, anything worth doing should be done as well as we can, and I'm counting on you, Gene, and Bill S to get the best measure we can. But it seems to me you're talking unrealistic standards, Les, when you say that we don't yet "have a measure... valid enough to include in the evaluation of teacher competence." One must look at the validity of the measures we currently use in order to see whether TVAAS is "valid enough".

Short of finding a teacher in flagrante delicto with a student, what we use is a sorry bunch of variables, ranging from the process variables observed on a visit to the classroom, some of them vaguely correlated with learning gains via process/outcome research (none of these are legitimate), through reports from parents, noise heard through the classroom walls, all the way to evidence of enrolment in post-grad studies.

I think TVAAS still looks like a better addition to that pile than most of the stuff in it at the moment.

In any case, I'm suggesting that we should judge it in terms of whether it enables us to do better, not just point out that it doesn't do as well as would be desirable.

From: Scriven@AOL.COM
Subject: Re: Dorn on Scriven

No, Sherman, what I'm saying isn't much like your version of it. If someone has a better student-outcome measure of teacher merit than TVAAS, let's show it's better, not just point out imperfections in TVAAS. I know of many allegedly better efforts, but none that hold up under the kind of heavy fire that TVAAS is getting here. If you have a better one in mind, try explaining it here and let's see. In any case, whether you do or not, the standard for criticism of it and TVAAS has to be whether it's better than nothing in the outcome dimension, not whether it's flawless.

As to your claims that (i) I'm assuming that all critics of TVAAS are "against judging teachers by what students learn" and that (ii) in pointing out that there's a safety net for teacher evaluation against errors in the TVAAS model I'm saying the TVAAS is of no value, your logic in imputing them to me seems pretty far-fetched. In any case, I did not and do not intend any such implications.

I'm trying to get us to keep reasonable standards in mind, not ideal ones; that doesn't seem to be an effort that deserves the kind of overkill attack you've launched on the suggestion.

By the way, I thought we were trying to avoid being condescending to each other on this board. It seems to be pretty condescending to say to me "Go read X" (and you'll see how wrong your assumptions are), rather than give reasons here and now. I could throw references at you, too, but I try to get the points across with a summary of reasons.

From: Sherman Dorn
Subject: TVAAS

Michael Scriven writes:

>If someone
>has a better student-outcome measure of teacher merit than TVAAS, let's show
>it's better, not just point out imperfections in TVAAS. I know of many
>allegedly better efforts, but none that hold up under the kind of heavy fire
>that TVAAS is getting here. If you have a better one in mind, try explaining
>it here and let's see. In any case, whether you do or not, the standard for
>criticism of it and TVAAS has to be whether it's better than nothing in the
>outcome dimension, not whether it's flawless.

There are two pieces to this "better than nothing in the outcome dimension." One piece is what you have explicitly described -- trying to include information about student learning in teacher evaluation. Gene's and Les' criticism get at that, and Gene has described at least one alternative.

A second piece, however, deals with the legitimacy of the tool within school cultures. If something (or a set of different somethings) is so bad that it poisons the atmosphere for further attempts to include student outcomes in teacher evaluation, then, yes, I'd argue that it can be possible to be much worse than nothing in the outcome dimension. It is very difficult from the outset to get teachers to pay attention to all their students, and the umpteenth attempt at accountability will just be treated as crying wolf.

The basics of formative evaluation, or curriculum-based assessment, as described in special education journals and as I've described a few months ago, is for a teacher to test students frequently and make instructional decisions based on whether an individual student is meeting a pre-specified goal. Yes, this depends on competent teaching, appropriate selection of goals (from what I gather, many teachers are timid about goals), and tests that are decent and relatively easy to conduct and score appropriately.

But there are several advantages: the information on the student shows frequently-gathered information in a situation that, after a few times, should be immune to practice effects, can be adjusted to be sensitive to student progress for very low-performing children, is much more likely to be seen as legitimate information by teachers than annual test scores, and can be used for evaluation of the teacher (who should be selecting appropriate goals, assessing appropriately, and making decisions in response to student progress or lack thereof). This type of assessment for students was developed in a context of individualized programming (for special education students), but there are also forms appropriate to whole class assessment, and you could make parallel decision rules -- create tests that ask students to perform tasks related to curriculum for the entire year, assess frequently, and respond instructionally to what you see. (And, at the teacher evaluation level, see if the teacher is responding to the information.)

>As to your claims... I did not and do not intend any
>such implications.

I will certainly accept that, and mea culpa if it read as too personal a comment. It did, however, seem rather strange to be arguing that poor methods are better than nothing in a policy context. Maybe I could accept that if TVAAS were used as evaluation in a single or a few school systems. But this is something that affects several hundred thousand children, and thousands of teachers.

In response to Les McLean, Michael Scriven writes:

>Short of finding a teacher in flagrante delicto with a student, what we use
>is a sorry bunch of variables, ranging from the process variables observed on
>a visit to the classroom, some of them vaguely correlated with learning gains
>via process/outcome research (none of these are legitimate), through reports
>from parents, noise heard through the classroom walls, all the way to
>evidence of enrolment in post-grad studies.

As an historian, I see the causal relationship here in a very different way. Yes, teachers are not evaluated on what children learn, but that's not primarily because we have bad outcomes measures. We have bad outcomes measures because schools are not designed to pay attention to outcomes. Even when standardized testing spread in the 1980s, it was frequently not designed to assess teachers but rather punish students for not learning (e.g., the proficiency test I had to pass in California in order to graduate). Daniel Calhoun writes, in THE INTELLIGENCE OF A PEOPLE, of the ways in which our views of intelligence, and ways of judging what children have learned, have typically been a way of blaming children (or adolescents or adults) for not learning. List subscribers here in the past week have volunteered anecdotes of how tests supposedly designed to help children by making teachers accountable instead have created pressures which (I believe) have no business in a school. I don't think it's because people have not yet come up with the perfect statistical tools to turn those test results into good outcomes measures. I think it's because schools have a tendency to sort and blame when the heat's on. With this tendency, I'd rather not rely on something like the TVAAS to get teachers to pay attention to students.

From: Rick Garlikov
Subject: Re: Validity, again (no apology)

Les, TVAAS has given at least one non-mathematical argument to demonstrate the reasonableness of their approach; that is that their numbers match the evaluations supervisors make. And although Harvey doesn't see any purpose or point in my previous request about this matter, it seems to me that one of the implications of TVAAS's claims is that they could quite accurately predict the test scores of students, given sufficient information about that student's past performance relative to his/her classmates, and given the information how those classmates have done on a test the student in question does not take, or has not yet taken. I suggested a trial in which TVAAS makes such predictions based on tests that are taken but withheld from them until after the predictions. Harvey thinks this is nonsense. But it seems to me to be a way of demonstrating whether TVAAS has some sort of statistical power or not, without arguing about the mathematics. As I said in my first post about all this, in science math is only a guide, not a proof of how the real world behaves. Even in physics, they have to do the experiment to try to confirm, not that the math is right, but that it is the right math. I think Gene would agree with this in those cases where he thinks math is applicable at all. And, like Gene, I would agree that there are far more important aspects to evaluating teachers than what can be described or computed mathematically. But TVAAS agrees with that. Their focus is only about one aspect of teaching. They are, to use my baseball analogy, only computing batting averages; they are not claiming batting averages are the only measure of the value of a baseball player. And they are using every means at their disposal to try to make that point abundantly clear.
But with regard to the math, I think a non-math means of demonstrating the reasonableness of the method is perhaps more important than mathematical arguments about the math itself. Especially for those, like me, who have no way of following the arguments Gene, Harvey, Greg, Les, et al, can make, but which don't get to the heart of whether even the most impeccable math applies or not.

From: Sandra P Horn
Subject: Response to Glass from Wm. Sanders

Response to Glass's Jan. 19 post from William Sanders:

Sanders: "To Gene Glass: I hope that reading this article will give you a more accurate picture of what we are doing. After you have looked at it, you may want to restate your questions."

Glass: "I don't appreciate being fobbed off in this manner and I am in an even more atrabilious mood for having traipsed to the library to copy the article referred to (Journal of Personnel Evaluation in Education 8:299-311, 1994)."

Sanders: There was certainly no intent on my part to fob you or anyone else. What has been disturbing to me (and still is), is how you or anyone else could assume to know what we are doing in TVAAS without examining relevant materials and then write criticisms based upon your own assumptions in such declarative tones. Now that at least you have looked at the paper to which we referred, we can begin to discuss your criticisms from the perspective of the model(s) which we are using.

Glass: "It is said that pretest achievement scores will be the "filter" through which these exogenous influences operate on the posttest scores, hence, the student background characteristics can be ignored. If this is an assumption of the system, it is clearly contestable and nearly certainly false; if it is a belief about what accounts for variation in achievement scores, it is incompetent."

Glass: "The only way that such background influences can be assumed to enter the TVAAS system is through the "filter" of pretest scores; to think that this "filter" is sufficient to correct for the background characteristics is simple fantasy."

Sanders: Observe that we fit the model (including all teachers over all subjects over all grades simultaneously within each school system) to the entire observational vector available for each student with the appropriate variance-covariance structure within the r-matrix. By so doing, one can view each student vector to be "like" an incomplete block with the analogy to the analysis of incomplete block designs. After obtaining teacher and school effects this way, what justifies our claim that most of the socio-economic confoundings have been filtered?

We do ex post facto analyses relating the teacher or school effects to variables that have been accepted by some (at least) to proxy socio economic status. We have found the following based upon the state-wide analysis. There is no relationship between the school effects and: (1) the percentage of students receiving free and reduced lunches in the school; (2) the racial composition of the student body; (3) the location of the building as to urban, suburban and rural.

Also, the relationship between the school effects and the school means are extremely low as can be seen in the following table. {No we do not have state-wide a measure of student ability. However, in our early studies we found that the inclusion contributed virtually nothing}.

Simple coefficients of correlation between school effects (math) and the mean for each school (3 year averages).

      Grade       r     N (no. of schools)
        3       .056        898
        4       .081        888
        5       .101        865
        6       .163        673
        7       .105        509
        8       .078        510

With relationships of this magnitude, it certainly is easy to show numerous examples of schools across the entire spectrum of mean achievement scores who have obtained excellent gains.

The statements that we have made are not mere assumptions but rather come from the results of the data analysis!! However, if needed, the modeling process which we deploy does not in any way preclude the use of other covariables. Let me restate that, to date, we have not found the need for these additional variables. If, in the future, using other test data, a need is found to insure fairness, then the models will be expanded to include them.

Glass says, "How, I might ask, does TVAAS imagine that the positive correlation between ability and achievement arises in the world if more able students do not learn at a faster rate than less able ones?"

One of the major findings from the state-wide analysis of the data is that most (not all) of our school systems do not have the curricular stretch to allow the highest achieving students to express gain commensurate with the gain obtained by the average and below average students. However, in other systems, we find this to be not true! In those systems, students of all achievement levels are making satisfactory gains.

Glass: It is clear from reading the TVAAS responses to questions posed here over the past few weeks and from reading their latest published exposition of the method that they have no appreciation of the validity question whatsoever.

That is a heavy charge. Let me share several additional findings.

1. As part of the second study (data from Blount County), the principals were asked to forecast whether each teacher would profile in the top, mid or bottom third of the Blount County distribution. The principals forecasted about 90% of the bottom profiling teachers, and could distinguish between the top and mid groups of math teachers; however, they could not distinguish between the top and mid reading and language arts teachers.

2. As was mentioned in the most recent post, we now have merged the writing assessment data into the master data base. Even though the studies are not complete, it appears that we could substitute this data for the language arts data from TCAP without appreciably changing the rankings of schools within a system.

3. We have data from the 10th grade TCAP and from the 10th grade PLAN tests. The data could be interchanged with virtually no change in the rankings of systems and schools.

4. When we have had direct knowledge of change in educational practice, then we have observed change in the effects. For instance, Knox County, which has a middle school system, has had severe retardation in the gains for 6th grade (the first year of the middle school). This past year a major effort was launched to improve communication between feeder schools and receiving schools such that instruction could be provided earlier in the school year commensurate with where the feeder schools had left off the previous year. After this effort, the Knox County 6th grade gains improved appreciably.

6. One of the administrators at Tennessee State University (TSU) gave us list of several schools in which they had assisted in developing a "hands on approach" to teaching science. In all of these schools, we found that the cumulative gains (gains summed over grades) were greater than 100% cumulative norm gain.

7. For those schools that are offering pre-algebra in the seventh grades, their gains are considerably higher than those which are not.

Does any of this speak to validity?

Cost

Some have been addressing this question. Let me mention a cost that needs to have the most consideration. That is the cost of denying human beings a chance to be competitive in later life because they were unfortunate enough to have attended schools and systems that were not offering a competitive academic program.

To me the purpose of all formal education is to provide as many human beings as possible, as many choices in life as possible. As I mentioned in the previous post, we found that tremendous difference exists among our school systems in mean ACT scores, when considering students that were at the same place academically as measured with the 8th grade TCAP tests. For example, considering only the top quartile of eighth graders, the top two or three systems will have mean ACT math scores of 27 while other system means will be 18. Why? Because most of these low averaging systems do not offer accelerated math programs, no AP courses, etc., etc. TVAAS is not just about profiling teachers, but rather is an attempt to measure those influences that are either accelerating or impeding the academic progress of populations of students.

Many can quibble about our models, can quibble about the tests and testing, can argue that the taking of the blood sample will do more harm to the patient than the illness. However, any reasonable analysis of the totality of the data will confirm that this process is fair and indeed has begun to bring pressure on school officials to improve the educational process.

No system, school or teacher is being asked to do more in Tennessee than is presently being done by educators working under similar circumstances. The extreme variation in effectiveness that exists among systems, schools and teachers is THE problem in public education in Tennessee. Attempts to attribute these findings solely to the testing or the statistical methodolgy will require extreme contortions of logic to evade the mountains of evidence to the contrary.

William L. Sanders
Director and Professor
University of Tennessee Value-Added Research and Assessment Center

From: Sandra P Horn
Subject: Re: Les McLean on TVAAS

There have been several messages similar to this one that originated from Les McLean. I would like to respond to it as a member of the TVAAS team, but not as a spokesperson for the team or for the UT Value-Added Research and Assessment Center.

On Sat, 21 Jan 1995, Leslie McLean wrote:

> Gene Glass's hypothetical procedure for local use of test scores in teacher
> evaluation ended with a modest claim--for a modicum of validity.  The
> post arrived just as I finished re-reading an ancient manuscript--Samuel
> Messick's presidential address to Div. 5 of APA in August 1974, "The
> Standard Problem: Meaning and Values in Measurement and Evaluation".  My
> it has stood up well! It contained reference to an even more ancient
> scroll, Lee Cronbach's writing on validity (with Meehl), going back to 1955
> (and thus more than 40 years's old).  The phrase that deserves emphasis
> here in 1995 is that we do not validate tests, but "AN INTERPRETATION OF
> DATA ARISING FROM A SPECIFIED PROCEDURE".  (That's from Cronbach's
> chapter in Ed. Measurement, edited by R.L. Thorndike, 1971, p. 447).
>
>    As I have come to understand the TVAAS, it goes like this:
>
>    These numbers are measures of gain in student achievement over a
> school year, and our PROCEDURE entitles you to read them as measures of
> teacher competence--the larger they are, the better the teacher, and v.v.
>

Then you have not been reading the posts we have sent you, or you are selectively ignoring whole sections. TVAAS uses student data longitudinally over at least three years. So far, reports have only been issued for schools and systems. Teacher reports will be issued for the first time this year. Teachers are only asked to attain normal gains for their students. Standard errors are employed so that a range of scores fall within the definition of "normal gain." We are working closely with teacher and administrator representatives to insure the privacy of individual teachers and to develop meaningful, useful reports for them.

>    Many of us, based on lots of experience, are extremely skeptical of
> this claim, and so far the explanations have only increased our
> skepticism.  The procedure is SO questionable (from test content to
> scaling method to model-fitting and adjustment) that any interpretation
> such as I have given above should be withheld pending supportive
> evidence--talk about "meaning and values in measurement and evaluation"!

We are attempting a dialogue with those of you interested in TVAAS. We didn't realize there were so "many" of you, since most of the posts have come from a handful of people. If you have questions, we continue to attempt to address them. What is it you find questionable, specifically? Asserting that the model is questionable certainly should be withheld pending supportive evidence. Talk about "meaning and values in responsible discourse"!

>   As many posts have argued, the TVAAS numbers become the only evidence
> to get attention, not because the TVAAS folk say so but because the sheer
> size and bulk and prestige of the enterprise drive out other measures.

These "many posts" have originated from a very few individuals who may not be representative voices. TVAAS has not by any means driven out other forms of assessment. The Tennessee Writing Assessment has just recently come on line. Performance evaluation of teachers continues, apace, and is the primary means of teacher evaluation in Tennessee. As we have stated repeatedly, TVAAS currently assesses the effects of educational practice on the outcomes students portray in five subject areas in grades 3 through 8, hardly the bulk of education in Tennessee.

> Messick ended his speech with a quotation from one Liam Hudson (1972),
> from his book, The Cult of the Fact (p. 125).  Hudson argued that social
> science:
>      ...should be pictured not as a society of good men and true,
>      harbouring the occasional malefactor, but rather, as one in
>      which everyone is searching for sense; in which differences
>      are largely those of temperament, tradition, allegiance and
>      style; and in which transgression consists not so much in a
>      clean break with professional ethics, as in an unusually high-
>      handed, extreme or self-deceptive attempt to promote one
>      particular view of reality at the expense of all others.
>
>
> More strenuous efforts would appear to be required to avoid such a
> transgression in Tennessee.

This is the part I consider insulting. Categorically, we have never broken with professional ethics. We communicate with educators on every level, within our state and across America, Edpolyan being only one example, seeking input, reaction, responsible criticism, and an understanding of what educators need in order to most effectively use TVAAS data.

TVAAS was created for the betterment of education. That is its purpose. There are no ulterior motives, and no one here stands to profit from some demonic plot to crucify teachers. Nor are we self-deceiving. We are fully cognizant of how achievement data have been misused by media and others in the past. TVAAS is an attempt to change that. Could it be that perhaps you are stuck in the old paradigm, that because this model uses data from standardized tests that you are making assumptions based on past usage rather than openly examining this new way of analyzing and reporting outcomes?

We have remained civil when those with whom we seek to communicate sink to such off-handed insults as these and when others have accused us of everything from arrogance to statistical ignorance rather than question their own assumptions or attempt to work collegially toward solutions.

It is not high-handed to refuse to engage in this type of low insinuation. Furthermore, it is not high-handed when our responses do not come fast enough to satisfy your schedule. We have tried to explain why there are delays but, repeatedly, this is interpretted as a personal issue rather than a logistical one. Is there a reason why we must waste time in this manner?

I am not willing to abandon my belief in the ability of people of good will to solve society's problems. After all, that's what brought me to the TVAAS team in the first place. I just find it very discouraging to find so little good will. It never occurred to me that this discussion would devolve into paradigm wars.

We have two discussions going on here: the philosophical aspects of educational assessment and the statistical propriety of the TVAAS model, both of which are vital. I don't foresee much progress being made on either front unless we base the discussions on belief in the good will of our partners in seeking solutions to the problems in education and, in the absence of evidence to the contrary, in the competence of our fellow discussants in their respective fields.

Perhaps this bickering is normal and I just don't know how to play this game. Forgive me. I am new and perhaps naive. But I, for one, find it counterproductive and saddening whenever, in any circumstances, people do not treat each other with respect, especially when egos preclude the open exchange and consideration of ideas.

Sandra Horn

From: Sandra P Horn
Subject: Re: Camilli's q's re: TVAAS

Dear Greg, Sorry it has taken so long to get back to you on these important questions. In regards to the CTBS/4, which is used in the Tennessee Comprehensive Assessment Program (TCAP) tests that provide the scaled data for TVAAS:

1. You asked us some time ago about how the TCAP was scored. It is scored by pattern (IRT) scoring in accordance with CTS's three-parameter statistical model.

2. You also asked what form of the CTBS/4--Benchmark, Battery, or Survey--was used. The full battery is used for the math, language arts and reading tests. Science and social studies are tested with the survey versions.

3. There has been considerable discussion on the issue of "teaching to the test." John Covaleski points out that the degre to which a test is capable of being "psyched out"--by this, we understand that he means "predicted"--by stakeholders has a great deal to do with how valid the results will be. We agree that this is true. The current TCAP tests, both the norm-referenced subtests and the criterion-referenced subtests, are comprised of 70% new items every year, the remaining 30% being drawn from a bank of repeat items from all previous years' tests. CTB certifies that the new items are equivalent to the items they replace in scoring properties. The purpose of requiring new items is specifically to discourage the educationally indefensible--and ineffective--practice of teaching to the test. Those of us involved with TVAAS anticipate that TVAAS will show that students who are fortunate enough to have teachers who develop concepts and encourage the discovery of connections do far better on all forms of assessment than those under teachers who waste instructional time trying to "psych out" the test. If, indeed, this turns out to be the case when the teacher reports are issued later this year, perhaps the preferred strategy for improving test scores will more often center around improving instructional strategies that meet the needs of each individual student, and the teaching to the test strategy will be abandoned in the face of evidence of its ineffectiveness.

We appreciate your continued interest in TVAAS.

Sandra Horn

From: Alan Davis

Rick recently argued that teaching to high stakes tests probably would not have adverse consequences if the tests were good and if teachers "knew that it is good teaching that gives the best chance for improved test scores."

One of the principles of high stakes testing, documented by Linda McNeil in Texas and supported by other research (see Eva Baker at UCLA) is that things that are tested push out curriculum that is not tested. So when Jane Armstrong (Education Commission of the States) and I visited school districts in Virginia in a study of elementary science instruction, teachers in schools where standardized test scores were low in Reading and Math informed us that they didn't teach much science at all, because science either wasn't tested or the science scores weren't of muc interest to parents. About the same time, the Michigan Association of Science Teachers pressured the state to start testing students in science as part of the Michigan Educational Assessment Progam (MEAP) so that teachers would begin teaching science -- it had been shoved out by reading and math.

The consequence of this is that everyone wants their subject tested, but to test everything that is important for kids to do in schools becomes a matter of overkill. Most will agree on the central importance of reading and math, but most will also agree that kids should learn other things in school that are less readily measured. In high stakes states, reading and math (and language arts, defined by the tests as editing sentences containing mistakes) push out other instruction.

From: Leslie McLean
Subject: Hommage a Sandra P. Horn

Bravo, Sandra Horn, for your spirited responses on behalf of the TVAAS! No insult was ever intended in my postings, but since insult was felt, please let me apologize.

I was particularly grateful for your clarification that you had not yet reported scores by teacher as gains-attributed-to-teacher-competence, but that you would shortly do so. Some of us (perhaps even you) forget sometimes that this is a listserv discussion and not a debate in the Tennessee legislature. We anticipate events--always a risky venture--but that is the way of the scholar--however bothersome it may be to the people who are on line for actions and justifications every minute. As unlikely as it may seem, Sandra, I have been the person who had to draft responses for the Minister (read: Chief elected official) on occasion. I would like to share with you sometime my reply to the school district official who wrote to the Minister to complain about our "Perception Bag", an unstructured set of curriculum materials on perceptions--including bits of paper soaked in scents--a sick drunk, a new Buick, ...

Please let no one ever read my posts as suggesting that the TVAAS staff are not concerned with the welfare of teachers and students. You have made your case well and I would call attention to it.

I not only read your posts, I save them (most of them)! For example, here is an excerpt:

Sanders:
Observe that we fit the model (including all teachers over all
subjects over all grades simultaneously within each school system)
to the entire observational vector available for each student with
the appropriate variance-covariance structure within the r-matrix.
By so doing, one can view each student vector to be "like" an
incomplete block with the analogy to the analysis of incomplete
block designs.  After obtaining teacher and school effects this
way, what justifies our claim that most of the socio-economic
confoundings have been filtered?
  
     We do ex post facto analyses relating the teacher or school
effects to variables that have been accepted by some (at least) to
proxy socio economic status.  We have found the following based
upon the state-wide analysis.  There is no relationship between the
school effects and: (1) the percentage of students receiving free
and reduced lunches in the school; (2) the racial composition of
the student body; (3) the location of the building as to urban,
suburban and rural.
  
     Also, the relationship between the school effects and the
school means are extremely low as can be seen in the following
table.  {No we do not have state-wide a measure of student ability.
However, in our early studies we found that the inclusion
contributed virtually nothing}.
  
  Simple coefficients of correlation between school effects (math)
and the mean for each school (3 year averages).
 
      Grade       r     N (no. of schools)
        3       .056        898
        4       .081        888
        5       .101        865
        6       .163        673
        7       .105        509
        8       .078        510
 
With relationships of this magnitude, it certainly is easy to show
numerous examples of schools across the entire spectrum of mean
achievement scores who have obtained excellent gains.
 
     The statements that we have made are not mere assumptions but
rather come from the results of the data analysis!!  [end of clip from
TVAAS post]

Thank you for the reports of your data analysis, which support what I noted, that you are concerned about teachers and students. It appears that some of us have not clearly communicated our concerns, since our concerns antedate all the data analysis reported above. We are concerned, and our concerns are not laid to rest, by descriptions such as, "we fit the model simultaneously ...over all grades ... to the entire student vector". How long must a student be in a teacher's class in order to be counted amongst that teacher's "achievements"? (Or do students not move in and out of classes in Tennessee--in many Ontario classrooms, there are more than 50% "in's and outs" in a year. And you fit the model over three years! I agree--the incomplete block design/metaphor is apt, and please forgive some of us if we sometimes lose patience and feel "fobbed off" by simplistic explanations.

Please do accept, Sandra, that those few of us who are trying to stay in communication with you on these very important but technical issues are people who also care about teachers and students--and colleagues who are charged with implementing complex systems. Some of us have likely been at it longer than you have; not that this makes us wiser, but it does give us a fairly wide context. We listen to wise colleagues such as Michael Scriven who remind us that teacher evaluation must somehow include consideration of student achievement. Our context, however, includes some distressing and sceptisism-inducing experiences with systems that begin with IRT scaling of multiple choice items by three-dimensional models and carry on to models of exceptional computional complexity to results said to have simple interpretations. If you could share our experiences, I would hope you would not feel an insult at our questions. Let us keep up the dialogue.

From: Sandra P Horn
Subject: TVAAS and moving students

Dear Les, Thanks for your kind post. It made me feel a lot better. I have always assumed that all of us were participating here with the common aim of hashing out direction and means for insuring the best education possible for our children. It's a point I felt needed to be be made again, though-- that mutual respect and open inquiry are the means by which our aim can be achieved.

As to your question on how moving students affect teacher reports, the answer is that students must be in a teacher's classroom a minimum of 150 days during the school year in order to be represented in that teacher's cohort. So not only are teachers protected from being held accountable for students who enter late, they are also not held accountable for students with excessive absences (students have a 180-day school year).

Please continue to ask.

From: Gene Glass
Subject: Why TVAAS Exists. Yesterday, Sandra Horn wrote:

>   TVAAS was created for the betterment of education.  That is its
> purpose. There are no ulterior motives, and no one here stands to profit
> from some demonic plot to crucify teachers.  Nor are we self-deceiving.

We can certainly credit the motives of Sandra Horn and William Sanders without accepting this somewhat oversimplified explanation of what caused TVAAS to be. Those of us who have studied the politics of similar efforts across the past 25 years have reason to doubt that TVAAS is solely an expression of a popular will to better education--even if it is far short of a profit-making daemonic plot.

An alternative account of the creation of TVAAS holds that it was proposed by traditional foes of public-education funding in the Tennessee Legislature as the compromise required to pass career ladder legislation. The career ladder legislation provided big financial benefits for teachers. "If teachers are going to get big payoffs like that, then they better prove they are adding value to students' lives." Such is the political rhetoric of accountability.

I think it is fair to say that the legislative genesis of TVAAS bore all the markings of suspicion toward teachers and schools and hostility toward increased education expenditures. This doesn't doom it to fail statistically or politically, but it is relevant context for understanding how it will operate, how its strengths and weaknesses will be seen and what its future is.

From: William Robert Saffold
Subject: More TVAAS Questions

I have been "lurking" for some time now, but I have yet to see many of my basic questions about TVAAS answered, or addressed sufficiently. I would love to hear (on-list) from the TVAAS staff or anyone else who can help me understand exactly what is going on with value-added assessment. Following are some observations and questions:

1. Dr. Sanders claims that scores are consistent from year to year, but the variations in gains (or losses) are actually quite large. Following are a few actual district report cards in various subjects. I have chosen small, medium, and large districts just to show the variation in gains. I have included the explanatory material at the end of this section:

___________________________________________________________________
Hollow Rock--Bruceton Special School District--804 students
 
Language--Estimated Means
 
Grade           2       3       4       5       6       7       8
USA Norm     667.0   696.0   707.0   724.0   739.0   749.0   757.0
1991:        681.2   705.7   706.4   729.9   734.5   745.5   757.1
1992:        698.5   701.6   723.4   737.6   757.1   738.0   763.0
1993:        692.0   693.6   708.7   732.0   741.0   753.4   760.4
1994:        673.9   695.2   711.3   725.9   737.3   761.9   763.5
 
Language--Estimated Gains
 
Grade           3       4       5       6       7       8       %CUMGAIN
USA Norm:     29.0    11.0    17.0    15.0    10.0     8.0
1992:         20.4    17.7    31.2    27.2     3.5    17.5      130.6
1993:         -4.9     7.1     8.7     3.4    -3.7    22.3       36.5
1994:          3.2    17.7    17.2     5.3    20.9    10.1       82.6
 
3 Yr Avg       6.2R*  14.1G   19.1G   12.0R    6.9R   16.6G      83.2
Std Error      2.8     2.5     2.1     2.0     2.2     2.3
 
______________________________________________________________________
Campbell County School District--6,410 students
 
Science--Estimated Means
 
Grade           2       3       4       5       6       7       8
USA Norm:    655.0   690.0   709.0   732.0   745.0   756.0   765.0
1991:        663.7   688.5   697.0   708.5   723.9   733.1   752.6
1992:        672.9   694.7   707.5   719.5   724.0   743.2   761.2
1993:        660.5   683.8   709.6   713.5   738.5   740.6   755.8
1994:        673.6   698.0   706.6   723.1   716.4   736.0   757.0
 
Science--Estimated Gains
 
Grade           3       4       5       6       7       8       %CUMGAIN
USA Norm:     35.0    19.0    23.0    13.0    11.0     9.0
1992:         30.9    19.0    22.5    15.5    19.3    28.1      123.0
1993:         10.0    14.9     6.0    19.1    16.6    12.5       72.8
1994:         37.5    22.7    13.5     3.0    -2.6    16.4       82.4
 
3 Yr Avg      26.5R*  18.9Y   14.0R*  12.5Y   11.1G   19.0G      92.7
Std Error      1.5     1.3     1.3     1.2     1.2     1.1
 
____________________________________________________________________
Chattanooga City School District--20,159 students
 
Social Studies--Estimated Means
 
Grade           2       3       4       5       6       7       8
USA Norm:    652.0   691.0   713.0   735.0   745.0   749.0   761.0
1991:        664.7   686.1   717.3   738.9   736.7   737.8   754.2
1992:        669.1   693.9   718.4   743.9   740.8   750.5   756.7
1993:        645.8   690.4   721.4   726.2   729.3   751.5   761.2
1994:        642.8   674.7   705.9   733.6   739.6   747.4   754.0
 
Social Studies--Estimated Gains
 
Grade           3       4       5       6       7       8       %CUMGAIN
USA Norm:     39.0    22.0    22.0    10.0     4.0    12.0
1992:         29.3    32.1    26.5     1.9    13.5    18.9      112.0
1993:         21.0    27.2     7.6   -14.5    10.6    10.5       57.3
1994:         28.5    15.1    12.1    13.2    17.6     2.4       81.6
 
3 Yr Avg      26.3R*  24.8G   15.4R*   0.2R*  13.9G   10.6R*     83.6
Std Error      0.9     0.7     0.6     0.6     0.6     0.6
____________________________________________________________________
Mixed Model Analysis using Scall Scores from Norm-Referenced Section of the
TCAP
 
G=  Green Zone:  Estimated mean gain equal to or greater than national norm
Y=  Yellow Zone:  Gain below national norm by 1 std error or less
R=  Red Zone:  Below norm by more than 1, but no more than 2, std errors
R*=  Ultra-Red:  Below norm by more than 2 std errors
 
NG=  Negative Gain:  no percent-of-norm calculated

Slight variations noticed in this year's estimates when compared to last year's estimates are a function of the fine tuning of the methodology. The scores for previous years will change slightly as the most recent data are incorporated, refining previous estimates.

[I HAVE TYPED THIS ALMOST VERBATIM FROM THE 1994 TVAAS REPORT--I hope that I have typed all the numbers correctly. Apparently, there is a computerized copy of the data available, but the TVAAS staff have fixed it so that no data can be manipulated or copied electronically. I would love to hear their explanation/justification for this.]

___________________________________________________________________

Dr. Sanders' position seems to be that all variations are the result of educational practice (unless someone else proves otherwise). The huge variations from year to year would seem to undercut his position. If the same teachers are teaching (presumably) in much the same way each year, what accounts for the variation in gains? Could the measurement instrument have anything to do with it? Or could the model just be misspecified?

2. Dr. Sanders (through Sandra Horn) claims that"

"When we have had direct knowledge of change in educational practice, then we have observed change in the effects. For instance, Knox county, which has a middle school system, has had severe retardation in the gains for 6th grade (the first year of middle school). This past year a major effort was launched to improve communication between feeder schools and receiving schools such that instruction could be provided earlier in the school year commensurate with where the feeder schools had left off the previous year. After this effort, the Knox County 6th grade gains improved appreciably."

The following are gain scores for 6th graders in Knox County:

 
                Math    Reading         Language        SocS    Science
USA Norm        19.0    18.0            15.0            10.0    13.0
1992:           12.5    15.1            14.1             1.1     5.3
1993:            9.7     7.8             5.2            -7.0    12.1
1994:           16.6    13.9             4.3            13.2     6.3
 
3 Yr Avg        12.9    12.3             7.9             2.5     7.9
Std Error        0.3     0.3             0.3             0.4     0.4

Sixth graders experienced drops in gains in 1993, and recoveries in some gains in 1994--but improvement in gains scores is not even across all subjects tested. Dr. Sanders provides nothing more than anecdotal evidence to prove that any increase in gain scores is due to educational effort--and he fails to mention the drops in language arts and science. Comparisons between gain scores in 1992 and 1994 show virtually no overall improvement. Gain scores in 1993 look more like the exception than the rule. THe change in gain scores could be, among many other potential factors, merely natural year-to-year variations.

3. I keep reading that TVAAS differentiates between district, school, and teacher effects on student learning. Does the student gain at the classroom level equal the teacher's effect? Does the student gain at the school level equal the school's effect? Does the student gain at the system level equal the system's effect?

4. Dr. Sanders admits that we don't even have a measure of student ability. How can we know that a student's innate ability has no effect on gain? (And if ability--or lack thereof--doesn't matter, why are special education students excluded from teacher assessment?) Dr. Sanders believes that low-achieving students (who may or may not be low ability students) can make "satisfactory gains" consistent with the national norm gain---but a low-ability student may not be able to do so, at least at the same rate as a higher-ability student (and the implicit assumption in the TVAAS model seems to be that all students should learn at the same rate--the national norm gain). Thus teachers might have to work much harder with a group of low-ability students in order to achieve acceptable gain scores. This discriminates against teachers with low-ability (as opposed to low-achieving) students.

5. Will the TVAAS staff please provide information about the correlation between the results of the norm-referenced test and the criterion-referenced test? It would help to know how well the norm-referenced test matches the Tennessee curriculum. We could have some objective evidence that the norm-referenced test sufficiently matches the curriculum if it can be shown that students score roughly the same on the two tests---at all grade levels and in all subjects.

Thanks, Bob Saffold
Vanderbilt Institute for Public Policy Studies

From: Sandra P Horn
Subject: To Saffold Re: More TVAAS Questions

I have only this minute read the post from which this exerpt is taken, so I can only reply to one item at this time and it is the one below:

On Wed, 25 Jan 1995, William Robert Saffold wrote:

> 2.  Dr. Sanders (through Sandra Horn) claims that"
>
> "When we have had direct knowledge of change in educational practice, then
> we have observed change in the effects.  For instance, Knox county, which
> has a middle school system, has had severe retardation in the gains for 6th
> grade (the first year of middle school).  This past year a major effort was
> launched to improve communication between feeder schools and receiving
> schools such that instruction could be provided earlier in the school year
> commensurate with where the feeder schools had left off the previous year.
> After this effort, the Knox County 6th grade gains improved appreciably."
>
> The following are gain scores for 6th graders in Knox County:
>
>                 Math    Reading         Language        SocS    Science
> USA Norm        19.0    18.0            15.0            10.0    13.0
> 1992:           12.5    15.1            14.1             1.1     5.3
> 1993:            9.7     7.8             5.2            -7.0    12.1
> 1994:           16.6    13.9             4.3            13.2     6.3
>
> 3 Yr Avg        12.9    12.3             7.9             2.5     7.9
> Std Error        0.3     0.3             0.3             0.4     0.4
>
> Sixth graders experienced drops in gains in 1993, and recoveries in some
> gains in 1994--but improvement in gains scores is not even across all
> subjects tested.

You are absolutely correct, and the fault is mine that this example was included in Bill Sanders' post.

When Bill wrote out his response to questions regarding validity, he forwarded it to me for a read-through. He also sent another post that told me to remove the item regarding Knox County schools because upon checking the figures, he had found that the findings were not consistant across all subjects, although improvements had occurred in some. However, there were two items regarding Knox County, and I removed the other one. I then sent the post to Edpolyan without returning it to Bill for checking. I just assumed I'd deleted the right thing. Upon reading this post, I discovered my mistake.

As for why these "anecdotal" examples were provided, it was in answer to those who find statistical validity "invalid" in terms of educational assessment.

Saffold raises several questions that will required detailed responses. We will check through his figures to be sure they are correct, although I assume they are. However, I am sometimes mistaken in my assumptions (see above), so we will check them for typos. We will attempt to answer each point in detail.

I apologize to all of you and to Bill Sanders for my mistake.

Sandra Horn

From: Greg Camilli
Subject: Re: Validity, again (no apology)

Rick Garlikov and Michael Scriven have suggested on more than one occasion that any test scores are fallible; surely that is no reason to abandon them in the assessment process. The problems with the TVAAS program may be

The scale of the scores may have anomalies (more variance in grade 1 than in grade 12),
The model may not take into account student background (e.g., family income, neighborhood, preschool, parents' education),
The model and its technical properties (e.g., standard errors) may be understood by only a very few,
*Some* test scores may be corrupted by the unscrupulous, other less wittingly,
The testing program may encourage some to teach the test rather than the educational skills,
The population actually tested may not be well understood (e.g., the effects of absences and exemptions.

When I think about it, the differences between this program and many other assessment programs is not that great, especially because the high stakes provisions seem to be loosely coupled with legal consequences. I don't remember reading in the material that Serman provided whether there are graduation consequences for seniors (like there are in other assessment programs.) However, I have reacted to some of the more extravagant claims made for the model.

Suppose it were the case that many of these claims were untrue (e.g., disentangling teacher and student effects). And suppose we agreed that test scores reflected the actual level of attainment for a group of children that we would describe at length. Thus, a child is tracked, and an unadjusted growth curve is plotted, or a teacher is tracked. We don't pretend the information is unconfounded with uncontrolable sources of variation.

Given this scenario, I have a couple of policy questions:

Is this information useful? If so, how?
Is it what the "public" or state adminstrators want? (As opposed to what you, the reader, want.)
Are the *model's* claims essential for its role as a high stakes test?

I ask these questions in good faith; they arose from consideration of TVAAS, but extend far beyond. Can *large-scale* testing programs can be used to evaluate teachers and students?

From: William Robert Saffold
Subject: TVAAS & Legitimate Inquiry

Kathy Bolland's response to my post underscores the reason I have refrained from entering the discussion until now. Debates of this nature always seem to end up forcing people into one of two positions: either they deny that there are any valid and/or cost-effective means of evaluating teachers, or they accuse those who raise questions about assessment of trying to let poor teachers off the hook. I categorically reject this false dichotomy. Raising legitimate questions about a new and poorly-understood model of assessment hardly strikes me as an attempt to "kill the messenger." Failing to raise basic questions about model construction and interpretation of results strikes me as an abdication of our responsibility as educators and citizens.

My point about the TVAAS and student abilities was this: There are differences in student abilities and those differences have an impact on value-added scores. If the TVAAS does not take student abilities into account, then it is more favorably disposed to some teachers than to others. If students were randomly assigned to teachers, this point might be moot--but we know that this is not the case. If you were a classroom teacher, would you want to take the risk that the TVAAS was unfairly weighted against you? Or would you raise the questions in hopes that the issue might be remedied? I choose the latter course.

Both Kathy and Rick seem to believe that I reject the value-added scores themselves as indicators of student gain. This is not so. What concerns me is that gains (or losses) at the classroom level might be ascribed solely to teachers--without any acknowledgment that there are other factors at work here. Of *course* it is important to know that low-ability kids (or high-ability kids for that matter) are not reaching their potential--but is it reasonable to assume that *all* children, regardless of their natural abilities, will achieve the same amount of gain (i.e., the national norm gain) in each subject each year? And is it reasonable to tell all teachers, regardless of the types of children they teach, that each and every class of children must meet or exceed that national norm gain or they have failed as teachers? The answer to both questions may be "Yes"--but I (and, I suspect, many others) would like to see some evidence to support this position.

I do not see these questions as an attempt to shift the blame for poor student academic gains away from teachers and on to the model itself. Quite the contrary--I think the model may very well be incredibly useful. As I told Sandra Horn in an off-list post, I am sincerely interested in the TVAAS. My questions are meant to enlighten both myself and others as to the way the model really works. I hope they will be regarded in that light.

From: Sherman Dorn
Subject: TVAAS

I've been sitting on this for a few days, in part because of scheduling and a cold, but also because I've been wanting to think about a few posts. Rick Garlikov writes:

> (3) if administrators and the media and public cannot be made to
>understand how to use these numbers as an indicator for this one narrow
>aspect of schooling, I don't see how the kind of thing Gene and Sherman
>describes is possible for them to apprehend and use at all.  It seems to
>me that what TVAAS is advocating is even easier for, say, administrators
>to understand than what Gene or Sherman is advocating.

This bewilders me. In part, this is because I think what I wrote and what Gene Glass wrote were very simple and easy to understand. Maybe that's my bias. But in addition, from a political standpoint, simplified statistical explanations may very well encourage oversimplified policies. People who don't understand TVAAS numbers except as "well, they mean something, and here's the ranking of your school" WILL take the numbers as absolute, which is what NO ONE ON THIS LIST has advocated. When TVAAS numbers were provided the print media this year, Nashville papers printed the rankings of schools without any indication of which differences in rankings meant something, even to the TVAAS statisticians.

In response to Gene Glass, William Sanders writes:

>     We do ex post facto analyses relating the teacher or school
>effects to variables that have been accepted by some (at least) to
>proxy socio economic status.  We have found the following based
>upon the state-wide analysis.  There is no relationship between the
>school effects and: (1) the percentage of students receiving free
>and reduced lunches in the school; (2) the racial composition of
>the student body; (3) the location of the building as to urban,
>suburban and rural.

This is provocative, but it is ecological reasoning. You cannot extrapolate from an analysis of system characteristics to student characteristics. To demonstrate the irrelevance of a variable to TVAAS, UTVARAC should see if adding individual characteristics such as race, sex, family income, or the presence of a disability would change the rankings of effects on various levels (teacher, school, or system). Or, if one merely wants to look at previous student scores, UTVARAC should be able to calculate bivariate plots/regression equations for each grade, pairing up the scale scores from the previous year to the current year (for example, fourth-grade math computation test scale scores to those students' math computation test scale scores for the prior year, when they were in third grade). What do the plots look like? What are the unstandardized regression coefficients? If Sanders is right, the plots should look very linear and (as importantly) the regression slope should be just about 1 (i.e., kids with high prior scores should be gaining the same as kids with low prior scores).

Later in the same post, Sanders writes:

>     The statements that we have made are not mere assumptions but
>rather come from the results of the data analysis!! However, if
>needed, the modeling process which we deploy does not in any way
>preclude the use of other covariables. Let me restate that, to date,
>we have not found the need for these additional variables.  If, in
>the future, using other test data, a need is found to insure
>fairness, then the models will be expanded to include them.

This assurance seems rather unusual for someone producing publicly-consumed statistics. In cases of controversy or doubt, the most reputable statistical bureaus in the country produce alternate sets of statistics with explanations of their assumptions. For example, the Census Bureau provides alternative population projections (labeled simply high, medium, and low) for the near-term future, and explain what assumptions led to each series. I believe that the Congressional Budget Office also provides alternative budget projections, and the Labor Department provides scores of alternative statistics one can choose from. Even the Department of Education, in its annual report on dropout/graduation statistics, is beginning to include alternatives to their three dropout "rates."

I see no reason why TVAAS could not produce alternative analyses with different variables added. It seems like the responsible thing to do, since this is one of the most controversial issues related to TVAAS.

Later yet, in response to Glass' question about general concerns with validity, Sanders made several claims about TVAAS:

>2.  As was mentioned in the most recent post, we now have merged
>the writing assessment data into the master data base.  Even though
>the studies are not complete, it appears that we could substitute
>this data for the language arts data from TCAP without appreciably
>changing the rankings of schools within a system.
 
>3. We have data from the 10th grade TCAP and from the 10th grade
>PLAN tests.  The data could be interchanged with virtually no
>change in the rankings of systems and schools.

These two bits of information suggest that the relative rankings would not depend on the tests mentioned. It still assumes that these high-stakes tests are the appropriate venues for judging student performance.

>4. When we have had direct knowledge of change in educational
>practice, then we have observed change in the effects.

This is a rather broad generalization. In order to confirm the thesis of improvement in effects if and only if changes in practice, you would have to have definitions of "change in educational practice" and then ask the question of every school in the state. And even this statement is not precisely true. For when William Saffold questioned the evidence of this in Knox County, Sandra Horn acknowledged the discrepancy:

 
>When Bill wrote out his response to questions regarding validity, he
>forwarded it to me for a read-through.  He also sent another post that
>told me to remove the item regarding Knox County schools because
>upon checking the figures, he had found that the findings were not
>consistant across all subjects, although improvements had occurred in some.

Please correct me if I am wrong about this: Bill Sanders writes that they have observed changes in effects whenever they "have had direct knowledge of change in educational practices." Knox County is one of those systems where UTVARAC has knowledge of such changes. Knox County does not show such uniform improvement. Sanders asked Horn to delete the Knox County data from the response but, and again correct me if I am misreading this, he did not revise the broader claim to reflect his new reading of the data.

Considering that he acknowledges that Knox County does not fit his prior statement, I would like to know what his current claim about this form of validity is.

From: Leslie McLean

Reply to William Sanders regarding hierarchical linear models (and multilevel linear and non-linear models) and his kind invitation to visit Knoxville.
Alas, Dr. Sanders, it is mid-crunch-term and I am up to my fingertips in students, on- and off-line. Your invitation is much appreciated, and I hope some of your fellow correspondents on EDPOLYAN will be able to take you up on your generous offer. As soon as classes end here (mid-April--eat your heart out American colleagues), I'm off for an isolated cottage on Prince Edward Island (off Canada's East Coast) to finish a book about teachers and teaching with my dear friend and colleague, Prof. Johan Aitken. I hope you will keep up the dialogue with Harvey Goldstein and others who can appreciate the work you are doing. I have read the paper in J. Pers. Eval. in Educ. and agree with Gene that it does not answer (as it could not) all the important questions. I have not seen the Amer. Statistician article. My position is still this:
When it comes to the validity of numbers reported to schools, teachers and students (not to speak of local newspapers, who often get the numbers whether you send them copies or not):

The details are everything: items, scaling, regression models, hierarchical models, treatment of missing data, rules for inclusion/exclusion, reporting formats and details (with caveats, whether ignored or not), follow-up meetings in districts/schools to discuss interpretations and releases to the press.
The content of the tests is important, but not as important as the way it is tested: the pedagogy suggested by the form(s) of high-stakes tests. NOWHERE is this more important than in the testing of language, and nowhere are the tests weaker than in this area. Most standardized (read 'published') tests CANNOT (and hence do not) test competence/proficiency in first language according to the best current pedagogy. In short, the "Reading" scores predict other reading scores but not whether students can read their textbooks--and especially not whether they can and do read books/newspapers/magazines. The scores and their non-linear scaled transformations tell us something about the level of literacy of students, but they do not, and cannot, tell teachers how to help the students who attain low scores. Since I make this claim, I make the corollary claim that gain scores are equally unhelpful to teachers. I offer some simple rules in this enormously complex enterprise:
1. ANYTHING that demeans and diminishes respect for teachers is BAD.
2. Test scores published in newspapers demean and diminish respect for teachers, though this is almost never the purpose of the publication.
3. The validity of a test score diminishes by the cube of the distance of the test constructor from the classroom.
4. The credibility of a derived achievement test score increases by the square of the number of terms in its predictive equation, multiplied by 10 times the number of times "exp" appears in the formula (whether direct or implied by 'log').
5. The correlation between credibility and validity is -0.14161828. Corollary: The credibility of scores arrived at by item-response models is approaching infinity so this may diminish the accuracy of the above correlation estimate.
6. The demand for published test scores is insatiable, esp. IRT scores.
7. Discussion on public forums such as EDPOLYAN is pointless.
----------------------------------------------------------------
From: Harvey Goldstein
Returning after a few days absence I find another 100 or so mesages about TVAA which I will slowly work my way through. However a few people have referred to the 'mixed model' used by TVAA so it might be worth trying some clarification of this (though when I thought I was trying to be helpful about standard errors it seemed to create more confusion than clarification!) The mixed model, elaborated by Henderson, Harville and others and the subject of the American Statistician paper (1991) by (Robert) McLean et al is in essence just the ordinary multiple regression model where the coefficients are allowed to have (random) distributions across the units defined in the model. Thus, e.g., if we have a model with students and schools identified, we can take a coefficient in the model, say a pretest score coefficient and specify that it varies randomly across schools and then estimate its variance. If we have several of these we can also estimate covariances among them. I include here the general linear model extension to ordinary multiple regression which allows factors, such as gender, or, say, type of school to be included (as dummy variables). Then you could see whether the gender coefficient (representing boy-girl difference) varies from school to school. In the TVAA case the mixed models of interest are known as hierarchical models or multilevel models because, as I have described in my example, they model a nested or hierarchical structure where students are grouped within achools and you have between student and between school variation. As I understand it (and the TVAA group and I are corresponding about the technical details) the basic TVAA model is what is known as a repeated measures model where the same students are repeatedly measured and you formally model measurements grouped within students, themselves grouped within schools. This is a 3-level structure. The advantage of this formulation is that you can isolate the influences at each level and study the factors which may explain the variations. As a by-product you can also estimate 'value added' scores, for example for teachers or schools - together with standard errors(!).
In the mid 1980's a number of investigators (Aitkin and Longford, Bryk and Raudenbush, my group in London)worked out efficient computational procedures that allow very complex and large datasets to be analysed efficiently and have led to important new insights. At least 3 software packages exist and the big groups (eg SAS) are also now getting in on the act. Ita Kreft did a review in 1990 and has an update due out soon in the Americal Statistician of these packages. The journal of educational and behavioural statistics is about to produce an issue on this also and there is a large literature about, with applications in education and elsewhere, including a few expository texts. If anyone is interested I can supply some introductory references, and the recent book by Bryk and Raudenbush would be a good start (hierarchical linear models, Sage, 1992). The Mclean et al article is not in my view a good introduction and also doesn't mention any of the important multilevel literature since 1986.
In short, 'mixed models' are neither obscure nor really difficult for anyone with a basic understanding of multiple regression - and there are efficient, publicly available software packages which are being used by large numbers of quantitative social, biological and medical scientists. Multilevel models have come to be recognised as the basic statistical technique in school effectiveness studies and there is a growing literature there too.
From: Sandra P Horn
Subject: W. R. Saffold's Questions
William L. Sanders offers these responses to W. Robert Saffold's questions about TVAAS:
Saffold states: "I HAVE TYPED THIS ALMOST VERBATIM FROM THE 1994 TVAAS REPORT-- I hope that I have typed all the numbers correctly. Apparently, there is a computerized copy of the data available, but the TVAAS staff have fixed it so that no data can be manipulated or copied electronically. I would love to hear their explanation/justification for this."
These files are available in ASCII format for easier transfer to other computers. The ASCII formatted data is available on request from the UTVARAC. Many school systems have already availed themselves of this opportunity.
Saffold: "Dr. Sanders' position seems to be that all variations are the result of educational practice (unless someone else proves otherwise). The huge variations from year to year would seem to undercut his position. If the same teachers are teaching (presumably) in much the same way each year, what accounts for the variation in gains?"
There are 138 school systems in our state. Each system gets a report for each of five subjects. Of the 690 reports, there certainly are many examples of considerable difference from year to year. Yes, I attribute most of the differences (above and beyond the 'noise' in the entire process) to changes in educational practice. Some of the changes may be known by the locals, some may be easily identified, while others may be much more subtle and much harder to identify. Let me share some of the reports which I have received from educators within systems that have been addressing some of the more obvious differences.
Among other things, they attribute the variations to
1. Less than desirable communication across grade levels. Pretend that two years ago that a system/school had retarded gains in math from second to third grade. Assume that the high achieving students in the third grade were permitted to progress at a more rapid pace. But assume that the fourth grade instruction was 'locked-in' and was not extended from where the third-grade faculty 'left-off'. The gain from third to fourth could show some dramatic change between adjacent years.
2. Changes in the structure of the school day. Consider a school in which last year the gains in science were low but the gains in language arts were quite high. If the school decides, as a result of this, to reallocate time so that more time is spent on science at the expense of another subject area, both subjects may show a change from one year to the next.
These are just two of many examples of practices that could result in large changes between adjacent years. However, the school and system main effects are considerably larger than the school*year and system*year interactions.
Saffold asks: "Could the measurement instrument have anything to do with it?"
The test forms are different but equivalent each year. The distributions for each subject and grade over the five years of testing are extremely similar. The means have gone up slightly in some subjects, but the over-all variances are virtually identical. These distributions are based upon the 55-56,000 records obtained from each grade-subject combination each year. The simple r's are about .7 between scores in adjacent grades for each subject.
"Or could the model just be misspecified?"
No model is perfect. However, I can demonstrate that the school and teacher effects are not related to indicators of ses and prior and post levels of achievement.
3. Saffold notes, "I keep reading that TVAAS differentiates between district, school, and teacher effects on student learning. Does the student gain at the classroom level equal the teacher's effect?"
The teacher model is:
```
Y(ijklm) = mu(ijk) + Year*grade*subject*teacher(ijkl) + e(ijklm)
where  i= ith year j= jth grade k=kth subject l=lth teacher
       e(ijklm)=mth student score within i,j,k,l
 
var Y  is ZGZ' + R
 
In this specific model the mu(ijk) is deemed to be the fixed part
of the equation, while year*grade*subject*teacher is the random
part.
 
The general form of the mixed model equations that we use is:
  -                                     -  -   -   -           -
 |   X'*INV(R)*X   X'*INV(R)*Z           | | b |   |X'*INV(R)*Y|
 |                                       | |   | = |           |
 |   Z'*INV(R)*X   Z'*INV(R)*Z  + INV(G) | | u |   |Z'*INV(R)*Y|
 |-                                      -  - -     -          -
 
 The G-matrix contains the covariances among teachers over
grades,subjects and years.
 
The R-matrix contains the covariances among student scores over
all years, subjects and grades.
 
The Y-vector contains the scale scores (not gains) for all
students over all subjects over all years and over all grades for
a school system.
 
The X-matrix contains all fixed effects, either continuous or
discrete variables.
 
The Z-matrix contains all random effects.
 
THE B-VECTOR IS BLUE IF G AND R ARE KNOWN.
THE U-VECTOR IS BLUP IF G AND R ARE KNOWN.
  Harville has called the u's , 'the realized value of the random
variable', a label that I personally like.
 
If G and R are estimated from the data, then B and U are often
referred to as empirical BLUE  and BLUP.  These estimations are
usually completed with REML, or as in our case with close
approximations for G and R for computing reasons.
```
Upon inspection, it is easy to see that the number of equations to be solved for a large system like Memphis will number into the tens of thousands.
The U's sum to zero for each grade and each subject. These are 'shrinkage' estimates and give a direct measure of each teacher's effect as the deviation from the system mean for each subject each year. By choosing estimable functions correctly, these estimates can be scaled as 'gains'. Several points: with little information the estimates are 'pulled' close to the average. This adds considerable protection against someone having an unfair estimated effect due to chance; in fact, an estimate cannot be judged different from average unless the effect is extreme and until considerable information is accumulated. The sensitivity of this process accrues because we are using the considerable correlation structure that accumulates by simultaneously considering all records for each student and fitting all teacher effects simultaneously (a comment that McLean felt that I was fobbing him with).
(Note: We encode Z in various ways to accommodate various forms of teaching, ie. self-contained classroom, departmentalized instruction, team teaching, teaching across grades, changing assignments over time, etc. Also, we have developed a procedure that we call our stacked-block concept, which I won't go into here, that adds considerable improvement to the sensitivity of the estimates).
(Second note: To all of you HLM modelers, I am fully aware of these models. I recognize that they are a sub-set of the general form of mixed model theory and methods. I don't agree with a recent post of McLean that there is a new theory and methodology that surpasses mixed models. They have been developed to deal with a different set of problems when covariables are included at different places on the hierarchy).
Saffolf asks, "Does the student gain at the school level equal the school's effect? Does the student gain at the system level equal the system's effect?"
The school model is like the teacher model except it is included as a fixed effect.
4. Saffold says, "Dr. Sanders admits that we don't even have a measure of student ability."
We don't have a DIRECT measure of student ability. However, the evidence strongly supports that by including all of the covariance structure that an unbiased measure of the teacher effect is obtained. The state-wide analysis clearly shows that the group of students in Tennessee which are not making as much gain in most of our systems is NOT the low achieving students. Rather it is the HIGH achieving students. Some of my colleagues are nearly through with a manuscript that will document this fact.
Saffold asks, "(And if ability--or lack thereof--doesn't matter, why are special education students excluded from teacher assessment?)"
That part of the EIA was suggested by others.
Saffold says, "Dr. Sanders believes that low-achieving students (who may or may not be low ability students) can make "satisfactory gains" consistent with the national norm gain---but a low-ability student may not be able to do so, at least at the same rate as a higher-ability student (and the implicit assumption in the TVAAS model seems to be that all students should learn at the same rate--the national norm gain)."
No. The implicit assumption is that all students eligible to take the TCAP tests can achieve gains at least equal to the norm gain if they are provided effective instruction from where they enter the classroom, academically.
Saffold says,"Thus teachers might have to work much harder with a group of low-ability students in order to achieve acceptable gain scores. This discriminates against teachers with low-ability (as opposed to low-achieving) students."
I find it interesting that you and others have expressed concern about teachers of low achieving (or low ability) students being penalized by the TVAAS process. In fact, most of the concerns that we have received have come from teachers, principals, etc. within systems and schools with a disproportionate number of high achieving students. In fact, many educators who are working with populations of students with lower achievement (and lower abilities) have expressed to me that they are delighted with TVAAS because for the first time there is some documentation for the public to see that their students are making creditable progress.
5. Saffold asks, "Will the TVAAS staff please provide information about the correlation between the results of the norm-referenced test and the criterion-referenced test?"
We have not done a thorough analysis of this. The state- wide system criterion referenced data are available from State Testing.
"It would help to know how well the norm-referenced test matches the Tennessee curriculum."
That question has been recently raised and a thorough review of that issue has been completed. The correlation is extremely high in all subject areas. If you want more information, I would contact someone on the staff of the State Board of Education.
We hope that this answers the points you have raised.
From: Sherman Dorn
Subject: Assessment
In a post responding to me on the comparative reform questions I've raised, Rick Garlikov raised an important question about assessment:
```
>     Finally, Sherman, in previous posts, does not seem to expect
>the same problems with internal assessments that he tends to
>expect from external ones.  I find that odd.  As I said in a
>previous post, if it is acceptable for sixth grade teachers to
>assess what their new students have been taught previously, why
>is it not acceptable for the state or for private assessment
>enterprises to monitor as students go along, in order to report
>to parents?
 
```
I see nothing fundamentally wrong with reporting progress data to parents. I see everything wrong with the way that the state legislature in Tennessee established TVAAS. The devil is in the details in this matter.
Consider, for example, whether TVAAS establishes a high-stakes environment, and whether teachers see it as a relatively non-pressured form of feedback (which is how UTVARAC staff argue it should be used ideally). The legislature created TVAAS' legal framework in a way that made it very clear to teachers what the point was. In 1991 and early 1992, then-governor Ned McWherter was trying to craft a major education financing reform (including a state income tax to fund it) in response to a successful finance equity suit brought by districts around the state. There is quite a bit of evidence that TVAAS was seen as the accountability "bait" to get legislators to go along with the tax hike. The state consulted with Bill Sanders about TVAAS, and the chief sponsors of the legislation quickly accepted an amendment by state senate education chair Ray Albright to include TVAAS in the bill.
The state commissioner of education at the time, Charles Smith, stated publicly that value-added assessment would give Tennessee the best accountability mechanism in the nation, and it would be part of a carrot-and-stick approach to education reform. Lamar Alexander (then Secretary of Education) approved of McWherter's bill precisely because of TVAAS as an accountability mechanism. Several legislators tried to amend the bill to remove everything EXCEPT TVAAS from the reform bill. Others tried to amend the bill to include explicit cut-offs at which administrators would be removed or to make teacher effect estimates public. (Sanders is on record as having opposed the latter type of amendments.)
In the end, McWherter did not get his state income tax, but did get a half-cent hike in the state sales tax and most of what he proposed plus TVAAS.
All of this was reported in the Tennessee Education Association newsletter at the time. I don't think this could have escaped notice among most public school teachers that many legislators may have voted for the increased funding for schools only because there was TVAAS attached, or what the proposed amendments represented.

Gene V Glass Archives

Thursday, October 6, 2022

An Archaeological Dig for VAM

No comments:

Post a Comment

Me & Saul

Report Abuse