MEDIA WATCH: Tests are '...more sensitive to how [they] rank students than to measuring what they have learned...' New York Times article adds to challenges to performance based teacher evaluations

A major article in the July 29, 2012 New York Times challenges the tests being promoted by the U.S. Department of Education and most states (including Illinois, which has "Performance Counts") about the reliability of the use of tests to evaluate teachers, schools and students. Once again, researchers have demonstrated that supposed "standardized" tests are flawed measures for the jobs corporate accountability experts are asking them to do. The July 29, 2012, article, based on recent Texas research, shows again what education historian Diane Ravitch was talking about at the AFT convention in Detroit. The URL for the Times article is

A SERIOUS DESIGN FLAW IS SUSPECTED IN STATE TESTS, New York Times -- July 29, 2012, By Morgan Smith

In 2006, a math pilot program for middle school students in a Dallas-area district returned surprising results.

The students’ improved grasp of mathematical concepts stunned Walter Stroup, the University of Texas at Austin professor behind the program. But at the end of the year, students’ scores had increased only marginally on state standardized TAKS tests, unlike what Mr. Stroup had seen in the classroom.

A similar dynamic showed up in a comparison of the students’ scores on midyear benchmark tests and what they received on their end-of-year exams. Standardized test scores the previous year were better predictors of their scores the next year than the benchmark test they had taken a few months earlier.

Now, in studies that threaten to shake the foundation of high-stakes test-based accountability, Mr. Stroup and two other researchers said they believe they have found the reason: a glitch embedded in the DNA of the state exams that, as a result of a statistical method used to assemble them, suggests they are virtually useless at measuring the effects of classroom instruction.

Pearson, which has a five-year, $468 million contract to create the state’s tests through 2015, uses “item response theory” to devise standardized exams, as other testing companies do. Using I.R.T., developers select questions based on a model that correlates students’ ability with the probability that they will get a question right.

That produces a test that Mr. Stroup said is more sensitive to how it ranks students than to measuring what they have learned. That design flaw also explains why Richardson students’ scores on the previous year’s TAKS test were a better predictor of performance on the next year’s TAKS test than the benchmark exams were, he said. The benchmark exams were developed by the district, the TAKS by the testing company.

Mr. Stroup, who is preparing to submit the findings to multiple research journals, presented them in June at a meeting of the Texas House Public Education Committee. He said he was aware of their implications for a widely used and accepted method of developing tests, and for how the state evaluates public schools.

“I’ve thought about being wrong,” Mr. Stroup said. “I’d love if everyone could say, ‘You are wrong, everything’s fine,’ ” he said. “But these are hundreds and hundreds of numbers that we’ve run now.”

Gloria Zyskowski, the deputy associate commissioner who handles assessments at the Texas Education Agency, said in a statement that the agency needed more time to review the findings. But she said that Mr. Stroup’s comments in June reflected “fundamental misunderstandings” about test development and that there was no evidence of a flaw in the test.

After a lengthy back and forth at the meeting, the committee’s chairman, Rob Eissler, suggested a “battle of the bands” — a hearing where the test vendors and researchers traded questions. Mr. Eissler, Republican of The Woodlands, said recently that he found Mr. Stroup’s research “very interesting” and that he was weighing another hearing.

Mr. Stroup’s research comes as opposition to high-stakes standardized testing in Texas is creating an alliance between parents, educators and school leaders who wonder how the tests affect classroom instruction and small-government conservatives who question the expense and bureaucracy they impose.

This is not first time the use of standardized test scores in Texas has been questioned. In 2009, the state implemented the Texas Projection Measure, a formula that critics said allowed schools to count students as passing who did not. After outcry from lawmakers, the state dropped the measure in 2011.

State Representative Scott Hochberg, Democrat of Houston, led the charge against the measure and has since proposed legislation aimed at reforming the role of standardized testing because of data showing that a student’s test score on the first year highly predicted it for the next.

“I have for a long time said that the accountability system doesn’t give us all the information that the numbers are used to generate,” Mr. Hochberg said, adding that basing accountability “more on the kid’s history than the specifics of what happened in the classroom that year may make us feel good but it doesn’t give us any true information.”


Add your own comment (all fields are necessary)

Substance readers:

You must give your first name and last name under "Name" when you post a comment at We are not operating a blog and do not allow anonymous or pseudonymous comments. Our readers deserve to know who is commenting, just as they deserve to know the source of our news reports and analysis.

Please respect this, and also provide us with an accurate e-mail address.

Thank you,

The Editors of Substance

Your Name

Your Email

What's your comment about?

Your Comment

Please answer this to prove you're not a robot:

1 + 4 =