The loneliness of the Long-Distance Test Scorer
[Editor's Note: The following article appeared in Monthly Review, December 2010, http://www.monthly review.org/101201 dimaggio.php. Numerous Substance reporters have long recommended a subscription to Monthly Review as a method of ending obsessive anger with corporate media. MR provides a healthy contrast from the USA to plutocratic propaganda disguised as "news"].
Pearson's corporate results (above, for 2009) and the need to get test contracts renewed year after year had more to do with how tests are scored than anything to do with student performance, according to Dimaggio. Note that sales and profits from "education" go up dramatically at Pearson (as at the Washington Post and other companies doing test prep and testing) during the years since "student performance" measurements via so-called "standardized" tests have become the driving force for the public schools of the USA, Standardized testing has become central to education policy in the United States. After dramatically expanding in the wake of the No Child Left Behind Act, testing has been further enshrined by the Obama administrations $3.4 billion Race to the Top grants. Given the ongoing
debate over these policies, it might be useful to hear about the experiences of a hidden sector of the education workforce: those of us who make our living scoring these tests. Our viewpoint is instructive, as it reveals the many contradictions and absurdities built into a test-scoring system run by for-profit companies and beholden to school administrators and government officials with a stake in producing inflated numbers. Our experiences also provide insight into how the testing mania is stunting the development of millions of young minds.
I recently spent four months working for two test-scoring companies, scoring tens of thousands of papers, while routinely clocking up to seventy hours a week. This was my third straight year doing this job. While the reality of life as a test scorer has recently been chronicled by Todd Farley in his book Making the Grades: My Misadventures in the Standardized Testing Industry, a scathing insiders account of his fourteen years in the industry, I want to tell my story to affirm that Farleys indictment is rooted in experiences common throughout the test-scoring world.1
Wait, someone scores standardized tests? I thought those were all done by machines. This is usually the first response I get when I tell people Ive been eking out a living as a test-scoring temp. The companies responsible for scoring standardized tests have not yet figured out a way to electronically process the varied handwriting and creative flourishes of millions of third to twelfth graders. Nor, to my knowledge, have they begun to outsource this work to India. Instead,
every year, the written-response portions of innumerable standardized tests given across the country are scored by human beings — tens of thousands of us, a veritable army of temporary workers.
I often wonder who students (or teachers and parents, for that matter) picture scoring their papers. When I was a student, I envisioned my tests being graded by qualified teachers in another part of the country, who taught the grade level and subject corresponding to the tests. This idea, it turns out, is as much a fantasy as imagining all the tests are being scored by machines.
Pearson Chief Executive Officer Marjorie Scardino (above) has been cited as one of the 20 most powerful female executives on earth, according to the business press, despite the fact that her testing division has been wracked with scoring scandals (that have even lost lawsuits) for years. The drive to score quickly and replicate lucrative state testing contracts since the advent of No Child Left Behind (continued "on steroids" in "Race To The Top") has led to widespread "juking" of the stats (who use the term for data manipulation from the HBO series "The Wire").Test scoring is a huge business, dominated by a few multinational corporations, which arrange the work in order to extract maximum profit. I was shocked when I found out that Pearson, the first company I worked for, also owned the Financial Times, The Economist, Penguin Books, and leading textbook publisher Prentice Hall. The CEO of Pearson, Marjorie Scardino, ranked seventeenth on the Forbes list of the one hundred most powerful women in the world in 2007.
Test-scoring companies make their money by hiring a temporary workforce each spring, people willing to work for low wages (generally $11 to $13 an hour), no benefits, and no hope of long-term employment — not exactly the most attractive conditions for trained and licensed educators. So all it takes to become a test scorer is a bachelors degree, a lack of a steady job, and a willingness to throw independent thinking out the window and follow the absurd and ever-changing guidelines set by the test-scoring companies. Some of us scorers are retired teachers, but most are former office workers, former security guards, or former holders of any of the diverse array of jobs previously done by the currently unemployed. When I began working in test scoring three years ago, my first team leader was qualified to supervise, not because of his credentials in the field of education, but because he had been a low-level manager at a local Target.
In the test-scoring centers in which I have worked, located in downtown St. Paul and a Minneapolis suburb, the workforce has been overwhelmingly white — upwards of 90 percent. Meanwhile, in many of the school districts for which these scores matter the most — where officials will determine whether schools will be shut down, or kids will be held back, or teachers fired — the vast majority are students of color. As of 2005, 80 percent of students in the nation’s twenty largest school districts were youth of color. The idea that these cultural barriers do not matter, since we are supposed to be grading all students by the same standard, seems far-fetched, to say the least. Perhaps it would be better to outsource the jobs to India, where the
cultural gap might, in some ways, be smaller.
Many test scorers have been doing this job for years — sometimes a decade or more. Yet these are the ultimate in temporary, seasonal jobs. The Human Resources people who interview and hire you are temps, as are most of the supervisors. In one test-scoring center, even the office space and computers were leased temporarily. Whenever I complained about these things, some coworker would inevitably say, Hey, it beats working at Subway or McDonald’s.
True, but does it inspire confidence to know that, for the people scoring the tests at the center of this nation’s education policy, the alternative is working in fast food? Or to know that, because of our low wages and lack of benefits, many test scorers have to work two jobs — delivering newspapers in the morning, hustling off to cashier or waitress at night, or, if you’re me (and plenty of others like me) heading home to start a second shift of test scoring for another company?
Company communications with test-scoring employees often feel like they have been lifted from a Kafka novel. Scorers working from home almost never talk to an actual human being. Pearson sends all its communications to home scorers via e-mail, now supplemented by automated phone calls telling you to check your inbox. After the start of a project, even these e-mails cease, and scorers are forced to check the project homepage on their own initiative to find out any important changes. Remarkably, for a company entrusted with assessing students’ educational performance, messages from Pearson contain a disturbing number of misspellings, incorrect dates, typos, and missing information. Pearsons online video orientation, for example, warns scorers that they may face civil lawshits from sexual harassment. Error-free communications are rare. I was considering whether this was a fair assessment, when I received a message from Pearson with the subject Pearson Fall 2010. The link in the e-mail took me to a survey to find out my availability — for the spring of 2011.
Communications at scoring centers are hardly better. For example, test-scoring jobs never have a guaranteed end date. If you ask a supervisor when a job is going to be completed, you will get a puzzling response that we dont know how many papers are in the system, so we cant say when well be done. This response persists, even though its pretty easy to calculate how many fifth-graders there are in Pennsylvania and how long it will take to grade their papers, given our scoring rate. If we are lucky, we get twenty-four-hours notice before being told that a project is about to end and we should seek other work. Two hours notice is more common. In general, scorers are given no information beyond what is absolutely necessary to do the job.
What is the work itself like? In test-scoring centers, dozens of scorers sit in rows, staring at computer screens where students’ papers appear (after the papers have undergone some mysterious scanning process). I imagine that most students think their papers are being
graded as if they are the most important thing in the world. Yet every day, each scorer is expected to read hundreds of papers. So for all the months of preparation and the dozens of hours of class time spent writing practice essays, a students writing probably will be processed and scored in about a minute.
Scoring is particularly rushed when scorers are paid by piece-rate, as is the case when you are scoring from home, where a growing part of the industrys work is done. At 30 to 70 cents per paper, depending on the test, the incentive, especially for a home worker, is to score as quickly as possible in order to earn any money: at 30 cents per paper, you have to score forty papers an hour to make $12 an hour, and test scoring requires a lot of mental breaks. Presumably, the score-from-home model is more profitable for testing companies than setting up an office, especially since it avoids the prospect of overtime pay, the bane of existence for companies operating on tight deadlines.
But overtime pay is a gift from heaven for impoverished test scorers; on one project, I worked in an office for twenty-three days straight, including numerous nine-hour days operating on four to five hours sleep—such was my excitement about overtime.
Yet scoring from home also brings with it an entirely new level of alienation. You may work on a month-long project without ever speaking to another human being, never mind seeing the children who actually wrote the papers. If you do speak to another person, it’s at your own expense, since calling the supervisors at the test-scoring center takes time, and might cut into the precious moments you spend scoring (especially when you have to wait fifteen minutes for someone to answer, as happens routinely on some projects).
The piece-rate system also leads to some sinister math; I have often wondered how much money I lose for every trip to the bathroom, and debated taking my laptop there with me. And since you are only guaranteed employment until the papers run out, you are in a race against all your phantom coworkers to score as many papers as you can, as fast as possible. This cannot be good for quality, but as long as the statistics match up and the project finishes on time, the companies are happy. I did receive some automated warnings from Pearson that I was scoring too fast, while simultaneously receiving messages on the Pearson website to the effect that, We re way behind! Log in as many hours as you can and score as much as possible!
No matter at what pace scorers work, however, tests are not always scored with the utmost attentiveness. The work is mind numbing, so scorers have to invent ways to entertain themselves. The most common method seems to be staring blankly at the wall or into space for minutes at a time. But at work this year, I discovered that no one would notice if I just read news articles
while scoring tests. So every night, while scoring from home, I would surf the Internet and cut and paste loads of articles—reports on Indian Maoists, scientific speculation on whether animals can be gay, critiques of standardized testing—into what typically came to be an eighty-page, single-spaced Word document. Then I would print it out and read it the next day while I was working at the scoring center. This was the only way to avoid going insane. I still managed to score at the average rate for the room and perform according to quality standards. While scoring from home, I routinely carry on three or four intense conversations on Gchat. This is the reality of test scoring.
There is a common fantasy that test scorers have some control over the grades they are giving. I laugh whenever someone tells me, Make sure you go easy and give the kids good grades! We are entirely beholden to and constrained by the standards set by the states and (supposedly) enforced by the test-scoring companies. To ensure that test scorers are administering the correct score, we receive several hours of training per test, and are monitored through varying quality control measures, such as random validity papers that are pre-scored and that we must score correctly. This all seems logical and necessary to ensure impartiality—these are, after all, standardized
tests. Unfortunately, after scoring tests for at least five states over the past three years, the only truly standardized elements I have found are a mystifying training process, supervisors who are often more confused than the scorers themselves, and a pervasive inability of these tests to foster creativity and competent writing.
Scorers often emerge from training more confused than when they started. Usually, within a day or two, when the scores we are giving are inevitably too low (as we attempt to follow the standards laid out in training), we are told to start giving higher scores, or, in the enigmatic language of scoring directors, to learn to see more papers as a 4. For some mysterious reason,
unbeknownst to test scorers, the scores we are giving are supposed to closely match those given in previous years. So if 40 percent of papers received 3s the previous year (on a scale of 1 to 6), then a similar percentage should receive 3s this year. Lest you think this is an isolated experience, Farley cites similar stories from his fourteen-year test-scoring career in his book, reporting instances where project managers announced that scoring would have to be changed because
our numbers don’t match up with what the psychometricians [the stats people] predicted. Farley
reports the disbelief of one employee that the stats people know what the scores will be without reading the essays.2
I also question how these scores can possibly measure whether students or schools are improving. Are we just trying to match the scores from last year, or are we part of an elaborate game of juking the stats, as it’s called on HBO’s The Wire, when agents alter statistics to please superiors? For these companies, the ultimate goal is to present acceptable numbers to the state education departments as quickly as possible, beating their deadlines (there are, we are told, $1
million fines if they miss a deadline). Proving their reliability so they will continue to get more contracts.
As Farley writes, Too often in my career the test results we returned had to be viewed not as exemplars of educational progress, but rather as numbers produced in a mad rush to get things done, statistics best viewed solely through the prism of profit.3 It seems to me that what the companies would tell us, if they were honest, would be something like, Hey guys, your scoring doesn’t really matter. We just want to give the same scores as last year, so that there’s no
controversy with the state and we get more contracts and make more profits—so no matter what you learned in training, just try to forget it. States and local governments, meanwhile, play their own version of this game, because it looks good for them when politicians can claim that test scores are going up. Witness the recent controversy in New York City, where the percentage of students passing the math exam rose from 57 percent in 2006 to 82 percent in 2009, before
plummeting back down to 54 percent in 2010 (along with a 43 percent passing rate in English) after the standards were reviewed.4
As test scorers, we never know what the numbers we are
assigning to papers mean, or where we fit in this
elaborate game. We are only responsible for assigning
one score, on one small part of a test, and we do not
even know whether the score we assign is passing or
failing—that information is never divulged in training.
We never hear how the students fared. Whether Marissa
will be prevented from going to seventh grade with her
friends because one of us, before our first cup of
coffee kicked in, decided that her paper was a little
more like a 3 than a 4, we will never know. Whether
Marissa’s school will be closed or her teachers fired
(to be reborn as test scorers next spring?) remain
mysteries to the test scorers. And yet these scores can
be of life-and-death importance, as seen in the recent
suicide of beloved Los Angeles middle school teacher
Rigoberto Ruelas, Jr. Upon learning that he ranked as
less effective on the LA Times teacher performance
rating scale—based solely on test scores—Ruelas took
his own life.5
Even if the scoring were a more exact science, this
would in no way make up for the atrocious effect on
creativity wrought by the mania for standardized
testing. This impact has now been documented. According
to one study, creativity among U.S. children has been
in decline since 1990, with a particularly severe drop
among those currently between kindergarten and sixth
grade.6
While test scorers and students might be separated by
age, geography, race, and culture, we share one bond:
standardized testing puts us to sleep. In the face of
the crushing monotony of the hundreds of rote responses
fostered by these tests, scorers are left to fight
their own individual battles to stay awake. In any
test-scoring center, by far the most essential job is
done by the person whose sole responsibility consists
of making coffee for hundreds of workers, many of whom
will consume four to six cups a day to survive. In my
mind, I see a hideous symmetry between test scorers’
desperate attempts to avoid dozing off, and the sleepy,
zombie-like faces of the students as they prepare for
and take these tests.
Of course, these students only exist in my imagination.
Just as test scorers are never allowed to know the
effects of our scores on students, we never get a
chance to meet them, to see how they have developed as
writers, thinkers, or human beings, or to know what
life in their communities or families is like. All we
see is a paper on a screen. And after reading hundreds
of monotonous papers each day, it’s not uncommon to
start to feel a bitter distaste for the undoubtedly
beautiful youth of America and the seeming poverty of
their creative thought.
I remember reading, for twenty-three straight days, the
responses of thousands of middle-schoolers to the
question, What is a goal of yours in life? A
plurality devoted several paragraphs to explain that
their life’s goal was to talk less in class, listen to
their teacher, and stop fooling around so much. Its
asking too much to hope for great literature on a
standardized test. But, given that this is the process
through which so many students are learning to write
and to think, one would hope for more. These rote
responses, in themselves, are a testament to the
failure of our education system, its failure to
actually connect with kids lives, to help them develop
their humanity and their critical thinking skills, to
do more than discipline them and prepare them to be
obedient workers—or troops.
While we test scorers might be prone to blame these
children for the monotony of their thoughts, it’s not
their fault that their imaginations and inspirations
are being sucked out of them. No points are given for
creativity on these tests, although some scorers have
told me that, until recently, a number of states did
factor creativity into their scores. Ironically,
scorers are often delighted to see papers that show
individuality and speak in their own voice, and often
reward them with higher scores, though, judging by the
papers I’ve read, it appears as if students often
explicitly are told not to be creative. Yet even if
creativity were considered, it would not likely do much
to change the overall character of the writing—and
education—engendered by an emphasis on standardized
testing. As Einstein put it, “It is a miracle that
curiosity survives formal education.”
An entire education policy that thrives on repetition,
monotony, and discipline is being enacted, stunting
creativity and curiosity under the guise of the false
idol of accountability. What is more, this policy has a
differential impact, depending on students’ race and
class. As Jonathan Kozol explains,
In most suburban schools, teachers know their kids are
going to pass the required tests anyway—so No Child
Left Behind is an irritant in a good school system, but
it doesn’t distort the curriculum. It doesn’t transform
the nature of the school day. But in inner-city
schools, testing anxiety not only consumes about a
third of the year, but it also requires every minute of
the school day in many of these inner-city schools to
be directed to a specifically stated test-related
skill. Very little art is allowed into these
classrooms. Little social studies, really none of the
humanities.7
Seeing the results of this process is demoralizing to
test scorers, and you can feel it in the scoring
centers. Even though you can move about freely, use the
bathroom when you need, and talk to one another, the
room I was in this spring was almost always completely
silent. On every project, as the weeks go by, the
health of many scorers deteriorates, making me curious
as to whether the relentless, soul-crushing monotony of
the papers has an actual physical impact on those
forced to read them.
To be fair, these papers arent a total wash. There is
often wisdom in them, even on standardized tests. The
chasm between rich and poor is at times felt in the
writing itself, as some students come from unimaginable
privilege, while many more endure heartbreaking
experiences in foster homes. The papers are also a
testament to the persistence of racism, describing
teenagers kicked out of stores or denied service or
jobs because of the color of their skin. And it would
be wrong to think of test scorers as a down-and-out
bunch—many of us do this job in order to avoid having
to get other ones that would keep us from our creative
endeavors, or from traveling or pursuing other
life-enriching possibilities. A number of test scorers
I’ve met over the past three years are authors,
artists, photographers, or independent scholars, and
it’s common to see postings for book releases and other
events featuring the work of test scorers on bulletin
boards in the break room.
In the error-filled Pearson training video, Marjorie
Scardino says, Most of the people who work at Pearson
work with a passion and an intensity, because they
think know are doing something important. But I’ve
never gotten the sense from my coworkers that they
"think know" what they’re doing is helping kids or the
education process. If the Obama administration asked
test scorers whether the solution to this country’s
education system would be more standardized testing, I
think most of them would laugh. I’ve never gotten the
sense from my coworkers that they feel that what
theyre doing is helping kids or the education process.
Unfortunately, the joke is on us, as the Obama
administration pushes for even more high-stakes
standardized testing. I didn’t know whether to laugh or
cry back in April, when all workers at my test-scoring
center were asked to fill out a form allowing the
company we were working for to get a tax break for
hiring us. This tax break came via the Obama
administration’s HIRE Act, which was supposed to
provide subsidies for companies creating jobs. Never
mind that we were all going to be hired anyway, because
this is seasonal employment. Or that this money was
subsidizing temporary jobs with no health care and no
hope for transitioning into long-term employment—jobs
which, in a better world, would not exist.
While these companies brazenly collect what can only be
described as corporate welfare checks, hundreds of
thousands of teachers are being laid off, as
governments cut funding to education. Maybe next year,
some of them will get paid $12 an hour (or $10, if they
flood the market) to score tests taken by students
stuffed into even bigger classes, and help
“impartially” decide which schools will be shut down,
and which of their colleagues will be laid off. Equally
bad, the fanaticism surrounding accountability via
testing, which claims it will result in higher-quality
teachers, is doing nothing of the sort. Referring to
the test-intensive No Child Left Behind Act, Kozol
says, “By measuring the success of teachers almost
exclusively by the test scores of their pupils, it has
rewarded the most robotic teachers, and it’s driving
out precisely those contagiously exciting teachers who
are capable of critical thinking who urban districts
have tried so hard to recruit.”8
As a friend of mine was saying his goodbyes to the
coworkers in his room at the end of this year’s scoring
season, his seventy-year-old supervisor, a veteran
test-scoring warrior, uttered the words I imagine many
test scorers hope to hear: “I hope I never see you here
again.” This is a measure of the cynicism with which
many test scorers approach the industry, recognizing
that it is fundamentally a game, which too many people
are forced to play—but “hey, it beats working at
McDonald’s or Subway!” Yet amid all the hopes of
escaping the industry, these test-scoring companies are
successfully expanding and are now hoping to get their
hands on billions in “school turnaround” money handed
out by the Obama administration and state governments.
Pearson, for example, has “formed the K-12 Solutions
Group, and…is seeking school-turnaround contracts in at
least eight states…[claiming it] could draw on its
testing, technology and other products to carry out a
coherent school-improvement effort.”9
The big test-scoring companies will undoubtedly be
called on to furnish their supposed “expertise” in
developing and scoring the new generation of more
complex tests envisioned by Secretary of Education Arne
Duncan. The Obama administration just gave two groups
of states $330 million in grants to develop these new
national tests, with the stated aim of assessing more
critical thinking skills and providing better feedback
to students and teachers. But rather than addressing
the problems outlined above, it seems more likely that
this move will only transfer the absurdities in current
state tests to a national level, with the danger that
they will take on an even greater legitimacy. In fact,
given that Duncan’s proposal involves even more tests,
it is likely to make matters worse.
If scoring is any indication, everyone should be
worried about the logic of putting more of our
education system in the hands of these for-profit
companies, which would love to grow even deeper roots
for the commodification of students’ minds. Why would
people in their right minds want to leave educational
assessment in the hands of poorly trained, overworked,
low-paid temps, working for companies interested only
in cranking out acceptable numbers and improving their
bottom line? Though the odds might seem slim, our
collective goal, as students, teachers, parents—and
even test scorers—should be to liberate education from
this farcical numbers game.
Notes
↩ Todd Farley, Making the Grades: My Misadventures in
the Standardized Testing Industry (San Francisco:
Polipoint, 2009). ↩ Todd Farley, “A Test Scorer’s
Lament,” Rethinking Schools (Winter 2008/2009). ↩ Todd
Farley, “Standardized Tests Are Not the Answer: I Know,
I Graded Them,” Christian Science Monitor, October 28,
2009. ↩ Sharon Otterman, “Confusion on Where City
Students Stand,” New York Times, August 28, 2010. ↩
Alexandra Zavis and Tony Barboza, “Teacher’s suicide
shocks school,” Los Angeles Times, September 28, 2010,
http://articles.latimes.com. ↩ Po Bronson and Ashley
Merryman, “The Creativity Crisis,” Newsweek, August 10,
2010. ↩ Matthew Fishbane, “Teachers: Be subversive
(Interview with Jonathan Kozol),” Salon.com, August 30,
2007. ↩ Ibid. ↩ Sam Dillon, “Inexperienced Companies
Chase School Reform Funds,” New York Times, August 9,
2010. | Top |
____________________________________________
PortsideLabor aims to provide material of interest to
people on the left that will help them to interpret the
world and to change it.
Submit via email: labor@portside.org
Submit via the Web: http://portside.org/submittous3
Frequently asked questions: http://portside.org/faq
Sub/Unsub: http://portside.org/subscribe-and-unsubscribe
PS Labor Archives: http://portside.org/archive
Contribute to Portside: https://portside.org/donate
By: Susan Ohanian
long-distance test scorer
When the author sent me this article, I wouldn't post it. He participated in an enterprise that harms children for THREE YEARS. This self-serving article in no way measures up to the kind of analysis I expect to find in Monthly Review.