Newark Star-Ledger warns against New Jersey plan to use 'value added'
A column run on September 15, 2011, in the Newark Star-Ledger challenges plans by New Jersey governor Chris Christie to use so-called "value added" measurements to determine how well teachers in New Jersey's public schools are doing. The column, based on a recently published book by the former head of statistics for the Educational Testing Service is causing quite a stir in educational circles as the new school year begins.
Braun: Christie misses the mark on grading teachers, author says
Published: Thursday, September 15, 2011, 6:30 AM Updated: Thursday, September 15, 2011, By Bob Braun/Star-Ledger Columnist Follow
The former chief statistician for the Educational Testing Service, or ETS, has written a book, "Uneducated Guesses," in which he raises significant questions about whether student test scores can be used to evaluate teachers.
Gov. Chris Christie has been touting his plans for education overhaul, including the use of student test scores to evaluate teachers. It's the first full week of school, a traditional time for politicians to roll out proposed changes.
It's also the week a new book on education, Howard Wainer's "Uneducated Guesses," was released by the Princeton University Press. It raises significant questions about the premise on which much of Christie’s crusade is based — using student test scores to evaluate teachers.
"It sounds like a good idea if you say it fast," says Wainer, a Pennington resident who teaches statistics at the Wharton School at the University of Pennsylvania. ``But then you have to look at the evidence."
Wainer is no anti-test ideologue, no apologist for teacher unions. Much the opposite: For many years, he was principal research scientist at the Educational Testing Service, the people who brought us the SAT and Advanced Placement tests. He believes in tests.
What he doesn't believe is that you can use test scores for purposes for which they were not designed — like judging the value of teachers based on changes in their students’ test scores, a so-called "value-added model" or VAM.
"It appears," he writes "that the more you know about VAM the less faith you have in the validity of the inferences drawn from it."
Wainer’s brief is science, not politics. He says he would be willing to accept a teacher evaluation model using student scores if someone would provide him with evidence it worked. Meanwhile, he has problems. The first is the impossibility of drawing what he calls a "causal inference" — based on changes in student scoring — that those changes were caused by the teacher.
John Munson/The Star-Ledger
Harold Wainer, the former chief statistician for the Educational Testing Service, has written a book titled "Uneducated Guesses," in which he argues student test scores cannot be used to evaluate teachers.
Let's say the students in Mr. Jones’ class score an average of 12 points higher on some test. "It is a slippery slope from description to causality, and it is easy to see how the users of such a result could begin to infer that the children gained points ‘because’ they had Mr. Jones. To be able to draw this inference, we would need to know what gain the students would have made had they had a different teacher."
Wainer’s book contains funny riffs on the misuse of statistics. A study, for example, on the most dangerous profession based on age at death. Want to guess? Student. Think about it — students who die do so at a young age so, obviously, they have the lowest life expectancy.
But that hardly makes studying a dangerous profession; it makes the clumsy use of statistics dangerous to believe.
Christie’s office declined comment on Wainer’s book and comments about teacher assessment. The state Department of Education issued a statement saying it was using student performance measures in 10 pilot districts "before rolling out statewide in 2012.’’
It said such measures would be used in only "50 percent of a teacher’s evaluation." "We believe that teachers should never be evaluated on a single consideration such as test scores, much less a single test, but on multiple measures of student learning."
Wainer says the decision has already been made to roll out statewide use next year. Such a short pilot period, he says, "is worthless at testing the viability of the whole enterprise.’’
The education department statement also says New Jersey will not use an evaluation model tried without much success in Tennessee but rather one in Colorado, the so-called Colorado Growth Model, or CGM. Wainer sees no difference in the two and contends the experience in Colorado has been "remarkably unstable.’’
What is certain about the new plans is that they will be challenged, starting with the first teacher who loses a job or an increment. Years of litigation have brought us the law of tenure, and years of litigation will determine the use of student scores.
