Sections:

Article

Data-Driven Analysis: How Illinois school officials rigged the ISAT Reading Test for political purposes

Susan L. Zupan - February 04, 2010

About The Reporter: I am a twenty-plus year, elementary school teacher in the Chicago Public Schools (CPS). I am a proud and long-time subscriber to and presently a reporter for Substance, a member of CORE (Caucus of Rank and File Educators), and PURE (Parents United for Responsible Education).

NARRATIVE

I’ve been teaching one elementary grade-level for three years now, but prior to that I taught a different grade-level for five years, in my present prekindergarten through eighth grade school. Little did I realize how much my teaching abilities in reading would diminish from the grade level switch, that is, “according to the Illinois Standards Achievement Test (ISAT) data.”

Over the past two years, I witnessed with concern some of my students’ previous ISAT reading scores deteriorate with me at their next grade level as I apparently made them “less intelligent.” And I appeared to be such a great success when I taught my previous grade level!! All this from “the data” — even though it seemed to me that the students just kept progressing and advancing along, working as hard as I was working.

So, as a teacher, I began to wonder: What is going on?

Thanks to Mr. “Data-Driven” Huberman, the latest in a too-long line of non-educators in charge of education in the City of Chicago, I have put together data that I believe might shed some light on what might be happening to me/my students, as well as other teachers/students in CPS within the Illinois State Board of Education (ISBE) system.

Two things triggered my research project. One, I, along with the faculty of my school, received information via one of the many sheets of deep-dive data-driven diggin’analysis given to us from the “experts in charge of CPS” that the “value-added” calculations of certain grade levels in our school amounted to negatives. I thought: Whew! What does one do in a case like that? Crawl in a hole?

Two, as part of an assignment for professional development at the beginning of the school year, the intermediate and upper grade teachers were required to compile our own data, tracking student ISAT test scores over time.

Two grade levels (third and fourth) did not have any data to track for purposes of comparison, so they actually did not have any assignment due. (Thankfully, no one asked these teachers to compare their students’ ISATs to any previous DIBELS data from earlier grades; this would have been an exercise in futility, given that all the time spent on DIBELS as part of CPS’s primary grade reading program, still following George Bush’s national model, has nothing to do with students actually reading anything for comprehension.)

I decided (after crawling out of the psychological hole hearing from the “experts” that hard-working teachers are a negative to the education of their students threw me into) to do a further data-driven research project of my own. I was curious.

As is my habit anyway, I had already taken my students' scores from the previous grade(s) and compared them to their newer scores at the end of their (unfortunate?) sojourn with me as their teacher. I do this every year, but for the last two, since my grade level switch, I have seen the same, overall pattern — the scores in reading generally rise — but not at “the expected levels.” I recalled this seemed to happen at my new grade level as a whole, not just my classroom; I also vaguely recalled this happening for other grade levels as well.

So, I decided to go backward then forward to see where my new students in this new school year just might already stand at this stage in the ISAT game. Lo and behold! The students were already greatly losing ground! Among other things, their incoming ISAT reading scores indicated that XX% met/exceeded the standards upon entry to my classroom, yet the fall CPS Reading Benchmark Assessment scores (not that I have any faith in that particular assessment instrument) now predicted a very worrisome percentage of students who would no longer meet/exceed the ISAT standard for this year. All this in just the first 5-6 weeks of the new school year!

Of course, all this made me wonder… maybe, just maybe, were the kids and I just plain normal and their previous year’s ISAT scores INFLATED??? My reluctance to consider this possibility for the past two years was due to two things:

One, I never questioned the legitimacy of the scores from these, it seemed to appear, always high-scoring grade levels (whereas others always seemed to score lower) because I knew that I (and the teams I either worked with or socialized with) did not cheat, therefore an otherwise obvious reason that scores might be inflated was never considered;

Two, I did not like the idea that I/we might not have been the “great” teachers the scores appeared to indicate. (In a nutshell: naiveté and vanity.)

But now, armed with my suspicions, and stuck with these new and not-improved scores at my different grade level, I had to see if I was alone in the CPS/ISBE education universe or in good company.

METHOD

I used a rare copy of the CHICAGO PUBLIC SCHOOLS 'Proposed BUDGET 2009-2010.' The document lists all elementary schools. I selected every 10th school in the “Neighborhood Schools” section. Since they are listed in ABC order by the first name of the school, I thought – how much more random could I get? I chose only schools with student populations of grades PreK or Kindergarten through grade 8, both for comparison to my own school and for a sense of school continuity. If the 10th school in the selection was a high school or grade 4-8 school, etc., I simply took the very next school that fit the criteria but continued counting by 10 from the actual 10th school listed. I ended up with a list of 45 schools for the first of what became five research quests.

QUEST/DATA SET #1

The first thing I did was simply look on the ISBE website and note whether or not the percentage of students who met/exceeded the standards in reading increased or decreased in the randomly-selected schools from grade three (2008 scores) to grade 4 (2009 scores) and from grade seven to grade eight. The basic question was: What happened to the scores of the same students when they moved from one grade level (3 and 7) to the next (4 and 8)? Yes, students transfer in and out of schools, but it was good enough for my purposes.

My results were as follows: For the same third graders taking the test in fourth grade, 24 of the schools showed increases in the percentage of students meeting/exceeding the standards, while 18 showed decreases. That’s 57% for the increases to 43% for the decreases. For the same seventh graders taking the test in eighth grade, 35 of the schools showed increases in the percentage of students meeting/exceeding the standards, while 7 showed decreases. That’s 83% for the increases to 17% for the decreases.

Note: Though I started with 45 schools, the number dropped to 42 because I was still figuring out how to navigate the ISBE website/data; I simply couldn’t find three schools, and one turned out to be a new school with no data available. In fact, with all 45 schools, the percentages are as follows: grade 3 → 4 +58%/-42%; grade 7 → 8 +84%/-16%.

My suspicion/hypothesis was that the 3rd → 4th grade numbers would drop more than they did, but the 7th → 8th grade results had me curious for more. Quite frankly, they were (and are) unbelievable.

QUEST/DATA SET #2

I took the 45 schools from the method described above. I eventually found the two missing schools and selected the next one in the listing to replace the school with no data – and decided to look at what happened to all the students’ ISAT reading scores from one year (2008) to the next (2009) – 3rd to 4th, 4th to 5th, 5th to 6th, 6th to 7th, and 7th to 8th.

After further checking and rechecking, my results are as follows for the averages of the percentages of the same students increasing or decreasing in “meets/exceeds the ISAT reading standards” from one year to the next:

Grade 3 → 4 increase 1.4% 58% of the schools increase; 42% decrease

Grade 4 → 5 decrease .8% 53% of the schools increase; 47% decrease

Grade 5 → 6 increase 12.1% 91% of the schools increase; 9% decrease

Grade 6 → 7 decrease .8% 47% of the schools increase; 49% decrease; 2 schools remained the same

Grade 7 → 8 increase 7.7% 84% of the schools increase; 16% decrease

I certainly acknowledge(d) the possibility (if not certainty) of human error! But, at the time, I felt I was on to something.

QUEST/DATA SET #3

I went back to the Proposed BUDGET and selected every 7th school with the same method described above to see if the pattern would be replicated. I ended up with 65 schools this time. I have not had time to thoroughly double-check my own calculations; they may be published (as here) when I or others have time to do so. However, the results were very similar and clearly showed the same pattern as demonstrated above for the previous 45 schools.

QUEST/DATA SET #4

I was on a roll. Then it occurred to me to look at the data already compiled for the Subregions, District, and State – right in line next to the data by grade levels for the individual schools on the ISBE Report Cards. (Duh.) I calculated the year-to-year data (2008-2009) from the totals given for the “Subregions” (Chicago neighborhood schools), District (all Chicago Public Schools, including selective enrollment, charters, etc.), and State. Triple-checked by me (but see the human error statement above at the end of Set #2), my results are as follows:

Subregion District State

3 → 4 +3.3 +4.2 +2.1

4 → 5 -.6 -.2 +.3

5 → 6 +12.5 +12.8 +6.4

6 → 7 +1.8 +2.2 -1.5

7 → 8 +8.1 +8.1 +5.9

At this point, the pattern was/is indisputable. The number of students from third grade meeting/exceeding “the ISAT standards in reading” slightly increases in their next (fourth grade) year, but close to half of the schools register a decrease; the number of students from fourth grade meeting/exceeding “the ISAT standards in reading” drops when those same students take the test in their fifth grade year of elementary school; the number of students from fifth grade meeting/exceeding “the ISAT standards in reading” skyrockets in sixth grade, particularly in Chicago with close to all of the schools registering increases in the number of students who meet/exceed the standards; however, when sixth graders from one year take the test for their next seventh grade year, the number of students who meet/exceed “the ISAT standards in reading” nosedives; and finally, the number of students who meet/exceed “the ISAT standards in reading” from seventh grade now in eighth grade increases nicely for a grand elementary school finish.

In other words, the composite percentages for the number of students “meeting/exceeding the ISAT standards in reading” are nice and high for Chicago’s “Bridge” grades (3rd, 6th, and 8th). If they were lower, CPS would have to spend more money to enforce its policy of requiring all students who do not meet a certain (very low) standard to attend summer school, test again, and either pass or be retained based on the scores of the retests. (Conveniently, summer school is not funded by CPS for any other grades.) However, in quite an opposite fashion, lest anyone draw a very wrong conclusion from this data that such high-stakes testing works, the composite percentages for the high-stakes 7th grade test are comparatively low – a test that has traditionally been used to determine which students (without political clout) are allowed to enter Chicago’s selective enrollment high schools, i.e. Whitney Young & Co.

Note: The (non-double-checked by me/see human error statement above) data for the Math ISAT showed a similar pattern, just much less dramatic.

I am not alone in the CPS/ISBE education universe after all. In fact, I think my students and I do damn well on a RIGGED TEST.

QUEST/DATA SET #5

The last data I crunched was time-lapsed. I calculated then averaged one grade-level to the next grade-level for the District (CPS) and State over time from 2006. In 2006, for the first time, the ISAT reading test was given to all students in grades third through eighth; prior to that, one cannot make the same grade-to-grade comparisons due to differing grade levels taking or not taking the reading test. The (double-checked by me/see human error statement above) results are as follows, suggesting that these politically-expedient test results have been going on since, I fuzzily recall, about the time Arne Duncan became CEO of the CPS:

DISTRICT STATE

3 → 4 +4.9 +1.8

4 → 5 -2.2 -1.0

5 → 6 +12.2 +6.9

6 → 7 +5.4 +1.1

7 → 8 +11.8 +7.9

True experts, not self- and media-proclaimed experts who have no experience in the given field, need to verify the information/data presented above. (Substance has someone working on this.) In the meantime, the following are the concerns, questions, and comments of the reporter:

• An independent investigation should take place. Who might have done this to the students and teachers in the state of Illinois, aka the city of Chicago?

• Until an independent investigation is completed, a moratorium on any “standardized” testing, along with any related “high stakes” uses of these tests (failing students, merit pay, school closings/turnarounds, etc.), from ISBE or CPS should take place. And the nation should take heed.

• Transparency is demanded. Let the public see these “standardized tests.” Doesn’t the term imply valid and reliable movement by students through their grade levels in even chunks? Can you spell n-o c-o-n-f-i-d-e-n-c-e?

• The test-takers, aka students, along with their teachers/principals/school communities, have been held accountable long enough. It is now high time to hold the test-makers and their political cheerleaders accountable.

• Did someone in CPS/ISBE confuse “No Child Left Behind” with “Leave the Non-Bridge Grade-Level Children in Chicago Behind to Make Mayoral Control of the Public Schools Look Good”?

• If the politicians of Illinois had legislated merit pay (read: Race to the Top) starting in 2006 based on students’ test scores, it appears that in Chicago/Illinois elementary school teachers of grades 3, 6, and 8 would have suspiciously hit the jackpot, leaving the teachers of the non-bridge grades behind.

• Huberman’s super-dooper-goober-calculating-pooper-scoopers don’t know about or suspect this? Really? Every elementary school in CPS has been presenting them - via performance management, local school, data-crunching teams - data-driven “concerns” and data-driven “celebrations.” No patterns have arisen?

• Why did CPS/ISBE feel the need to give a “pilot ISAT Test” to 3rd, 6th, and 8th graders recently in December? What is the real what and why of CPS’s new policy for selective enrollment schools? In other words, are the above-demonstrated patterns of test score calculations now being rigged to change with this year’s round of tests? Or next year’s possible “new” test (see Chicago Tribune 01-29-10)?

• What is the real what and why of ISBE wanting to “scrap” (a term used for junk) the ISAT Test? In a cash-starved, supposedly flat broke system in a cash-starved, supposedly flat broke state? Different politics now demands different tests (with different results?) whatever the costs? The 3rd through 8th grade tests the children of Illinois have taken for the past four years now somehow aren’t good enough? Explain, please. Schools have been closed, children have been uprooted, employees have lost their jobs – due to tests now determined to not be somehow appropriate for Barack Obama’s Flunkin’ Duncan national demands?

• Where was/is Senator Meeks and his… gang on the Illinois Senate Education Committee as this was allegedly happening? Participating? Or sleeping at the watchdog helm? • Where was/is the Chicago “mainstream” media, that is, besides cheerleading all things related to “education reform” especially via “test scores,” and swallowing whole and regurgitating all numbers presented to them by all the CEOs of the CPS over these many years? • What might this mean for the education policies of the President of the United States and his non-educator education secretary, whose previous incarnation was the politically-appointed CEO of the CPS underlying this alleged data scandal? Such leadership for our nation’s most precious resource, its children?!

I believe our country, following Chicago’s example, is in a Race to the Top of SHAME. 

Copyright 2006-2020 by Substance News. All rights reserved.