MEDIA WATCH: You can't call it a failure if you ignore it... QUESTION. When is 'Merit Pay' not news in Chicago? ANSWER. When the scores didn't go up during the $26 million 'TAP' program
Clearly, one of the mottos of the Emanuel administration must be: "If at first you fail, keep doing it over and over." Or perhaps there is something there going all the way back to the kings of pre-revolutionary France, about learning nothing and forgetting nothing. But, then, the same must be said about Chicago's corporate media and reporters, who will be reporting with a straight face the latest iterations of "merit pay" (now rebranded as "differentiated compensation") from the Obama and Emanuel administrations, even though the previous iteration of the exact same thing, from the Bush and Daley administrations, failed.
Not just had problems. Failed. So naturally Chicago's hyperactive mediagenic mayor in 2012 is pushing forward with two merit pay programs (one for principals; another for teachers) even though all the evidence from previous years points to the fact that locally the program failed (and nationally such programs fail). When you're working from a Hollywood style script and one of your biggest financial backers is Stephen Spielberg, why let a little reality get in the way?
Five and six years ago, one of the big news stories in Chicago was the fact that the U.S. Department of Education was giving more than $20 million to CPS for a merit pay program that involved the Chicago Teachers Union, then under the leadership of Marilyn Stewart. The TAP (Teacher Advancement Program) merit pay program, which was supposed to provide bonuses for everyone in the school when scores went "up", was also a feature of the July 2008 convention of the American Federation of Teachers (AFT), which was held in Chicago at Navy Pier. There was on Chicago dog-and-pony show announcing the merit pay results, featuring Arne Duncan, Mayor Richard M. Daley, and CTU President Marilyn Stewart. But then — silence.
Then Chicago Teachers Union President Marilyn Stewart joined Chicago Mayor Richard M. Daley, Chicago Schools CEO Arne Duncan, and U.S. Secretary of Education Margaret Spellings on the stage of Chicago's Westcott Elementary School to sing the praises of the merit pay program (called TAP) that Chicago was utilizing with the union's support at the time. Substance photo by George N. Schmidt.Now the silence continues, even though a study has found that merit pay, at least as far as the TAP program was concerned, didn't raise student test scores. So, naturally, Chicago Mayor Rahm Emanuel is ignoring the facts, and his hand-picked Board of Education is proposing that everything from principal contracts to most years of the upcoming teacher contract feature, you might have guessed it: MERIT PAY.
There was no shortage of media hype for Chicago's merit pay program between roughly 2006 and 2009, when the TAP program was being funded by the federal government and hailed not only locally but nationally as an innovative way to improve inner city schools. The announcement of the federal dollars for the program was made at a major press conference held by Mayor Richard M. Daley at Orr High School during the "Principal for a Day" event in 2007. In July 2008, the American Federation of Teachers devoted a large part of its opening fanfare to a summary of the Chicago merit pay program. Later that year, at the same time Arne Duncan was being considered for the post of U.S. Secretary of Education, the city hosted a major media event at Westcott Elementary School where then U.S. Secretary of Education Margaret Spellings, surrounded by dignitaries including Arne Duncan, Richard M. Daley, and Marilyn Stewart, gave out the first TAP program checks to the principal, teachers, and others.
And then there was silence, which continues to this day.
Well, you probably won't see the following story anywhere in Chicago's corporate media as the Rahm Show continues following a brief interruption (when the Mayor went to Peoria to avoid questionns about why he was stripped of the G8), but the study is fairly conclusive:
Teacher Retention Up, But Scores Flat, Under Chicago Reform Project, By Stephen Sawchuk, Education Week, posted on March 6, 2012 5:44 PM
The Chicago Teacher Advancement Program increased mentoring and improved teacher-retention rates in some participating schools compared with a similar, nonparticipating set — but didn't appear to raise student achievement, according to a study released today by Mathematica Policy Research.
It is the final report on the Chicago TAP program, looking at all four years of the program's implementation in the Windy City.
Arne Duncan handed the checks to Richard Daley who handed the envelopes to Margaret Spellings who passed them along to the teachers and principal of Westcott Elementary School during the December 11, 2008 media event promoting the TAP merit pay program in Chicago. Substance photo by George N. Schmidt."Chicago TAP was only partially successful in achieving its goals," researchers Allison Siefullah and Steven Glazerman conclude. "The program can be credited with improved retention outcomes for some of its schools, but it did not have a noticeable positive impact on student achievement over the four-year rollout in Chicago."
U.S. Secretary of Education Arne Duncan was the head of Chicago schools when the district applied for the Teacher Incentive Fund grant that financed the program.
Officials at the National Institute for Excellence in Teaching—the nonprofit organization that oversees TAP—said that the effort was never fully implemented in Chicago, as in other cities.
"It was a small group of schools, over different area superintendents, so I think it was a fragmented approach from the beginning," said Gary Stark, the president and CEO of NIET. "I do want to applaud the teachers and the teachers' union for participating, they absolutely they came together to give this thing a chance. The collaboration was really strong, but the implementation wasn't there, and I don't think there was a sense it was going to take root or hold."
CTU officials Connee Fitch-Blanks (of the Quest Center) and Mary McGuire (then recording secretary) were among those cheering the TAP program at Westcott Elementary School on December 11, 2008. Substance photo by George N. Schmidt.TAP is a complex initiative that knits together professional development, advancement roles for "master" and "mentor" teachers, an evaluation system, and performance-based compensation. It's being used in a number of places across the country and has expanded significantly under the TIF grant program.
Hybrid Study
The Mathematica researchers used a hybrid of a randomized experiment and a quasi-experimental method to analyze the project.
For the randomization, the researchers were able to randomly assign cohorts of schools, 34 in all, to begin TAP either in 2007 or 2008, and again in 2009 or 2010. The planned delays in implementation allowed researchers to compare schools with one year of TAP under their belts with those with none, and again to compare schools with two years of implementation with those with just one.
For a longer-term look at the data, researchers also compared all the participating TAP schools with a group of some 200 other non-TAP schools of similar size and demographic characteristics, such as achievement, poverty, number of novice teachers, and proportion of special education students.
The results were as follows:
• Student achievement effects. The randomization analyses found no significant differences in student achievement between TAP schools in their first year of implementation and non-TAP schools, nor did an additional year of TAP implementation seem to affect test scores.
During the July 2008 convention of the American Federation of Teachers, then CTU President Marilyn Stewart was hailed by the national union to the union's progressive stances on such things as merit pay, which was featured during the dramatic opening of the convention. Despite this, Stewart (above) covered her face whenever Substance asked her about the TAP and other programs during the convention. Substance photo by George N. Schmidt.Similarly, the quasi-experimental analysis, which looked at cohorts in all four years of implementation, didn't find that scores had gone up in the schools with more years of TAP. In addition, researchers looked at this data using a variety of different lenses to see which effects were "robust," or showed up under all of them. An increase in science scores appeared in one calculation, but not in the others.
Overall, the sample size of TAP schools was small, and a handful either closed, reconstituted their staff, dropped the program, or never implemented the program over the course of the four years. But the researchers said the fact that the experimental and non-experimental data came to the same conclusion about student achievement allows for more confidence in the result.
• Teacher retention. In a bit of good news, the program did seem to have an impact on teacher-retention rates, though this seemed most pronounced for the cohort that began implementing the program in 2007. The retention rate over three years for this group was 67 percent, compared with 56 percent of teachers in non-TAP schools. In layman's terms, teachers were 20 percent more likely to stay in the TAP schools over that period compared to a non-TAP school.
• Teacher support. Teachers in the TAP schools did receive more mentoring, meeting more frequently with their mentors for a total of three hours of contact compared with about an hour and a half for teachers in the control group. Mentor teachers were more likely than non-TAP veterans to provide other teachers with literacy strategies, help in setting instructional goals, preparing lesson plans, and modeling lessons, among other things.
Alterations Made
Chicago officials made some changes to the TAP model before implementing it and were beset with other technical problems that affected how it was instituted. For instance, under TAP, part of compensation is based on an individual value-added measure. But Chicago had technical issues producing teacher-level estimates and never produced them.
In addition, the bonus payouts to teachers were somewhat lower than initially planned or the ones used in other TAP programs, averaging $1,000-$2,500 rather than the planned $2,000-$4,000. And finally, the district weighted each of the TAP components differently in each year of the program's operation.
NIET officials ended up parting ways with the district in year three of the study, based on these and other alterations.
The Chicago program "is not representative of the national TAP system model that has been implemented in hundreds in schools from South Carolina to Texas and Louisiana," said Kristan Van Hook, the senior vice president for public policy for NIET. "As a result, Mathematica's report should not be mistaken as an evaluation in any way of the national TAP system's proven effectiveness."
Chicago Public Schools didn't immediately return a call seeking comment.
Finding Answers
While the results will surely be disappointing for supporters of differentiated pay, they are no picnic for people supportive of induction or professional development, either. Given all the issues with the compensation element, this project put a lot more emphasis on the other pieces.
A handful of other nonexperimental studies have found benefits for TAP schools, but no randomized study has yet found that the program increases student achievement.
The question of TAP effectiveness aside, the study also raises a lot of conceptual questions. When a school intervention program doesn't have the results desired, who bears the responsibility? Is it a problem with the design of the program, with the fidelity of its implementation, or some other factor?
Comments:
By: Jean Schwab
TAP
Our school had TAP and the extra money was good and working with the other teachers was valuable. It was interesting that after a year of working together, we still had trouble deciding, when viewing a teaching video, who was the most effective teacher. The structure of lessons did not provide for any variation from the lesson plans or teachable moment. Sometimes we were provided with the exact words for certain lessons.
We also had people observing us and evaluating about weekly.
By: Kati Gilson
TAP
Our school participated in TAP. I spent an hour a week, away from my preschool students, in "PD". Most of it on extended response and some on math "estimation". My teacher assistant was left alone with my students. I have a half-day program, so losing a teacher for an hour a week is a problem. Ironically I was already doing extended response and estimation with my students. The only benefit for me was learning more of what goes on in K-2. Limited benefit for my students. The $ were nice but the PD received was not geared toward Pre-K. Another advantage was I did get to inform other grade teachers what preschoolers do and how they learn and why we do things the way we do. That was beneficial to them.