Bunkum Awards replace Jerry Bracey's 'Rotten Apples' in helping detoxify nonsense in so-called 'education research'

[Editor's Introduction and thank you. Without spending too much time and space on a lead in, Substance is happy to provide our readers with the annual "Bunkum Awards" from our colleagues and friends at the University of Colorado and Arizona State University. We will continue working on our own local Chicago Bunkum Awards, beginning with the special award for the "data driven drivel" Chicago has faced since our current CEO Ron Huberman purchased his "Performance Management" software from the Dell computer company and was able to foist it off on most Chicago (and all of Chicago's education writers except at Substance) for a year. Chicago still stands at the top of the heap in following bunkum in its claims, but now that former Chicago CEO Arne Duncan and the Chicago Boys are heading up the U.S. Department of Education, as the award below shows, nonsense in the form of "research" will be coming quickly and fast out of Washington. George N. Schmidt, Editor, Substance, February 16, 2010].

Five "Honorees" of Bunkum Awards Announced for their Contributions to Sub-Par Education Research, February 15, 2010... High-Production Values and Eye-Catching Charts and Graphs Can Never Replace Strong Methodology and Sound Research Practices

BOULDER, Colo. and TEMPE, Ariz. (February 15, 2010) -- State education agencies and local school districts are increasingly asked to make evidence-based decisions about school reform initiatives, often assuming that all evidentiary claims are the result of high-quality research. Unfortunately, much of the evidence offered in policy debates is based on research reports that have bypassed the quality control mechanisms of academic research.

In an effort to help education policy makers separate the wheat from the chaff, expert third party reviews are provided by the Think Tank Review Project, a collaboration of the Education and Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University. Each year the reports identified by experts as the worst of the worst are awarded a "Bunkum." The Think Tank Review Project today announced five "honorees" for 2009.

While the social science of the winning reports was sub-par, they typically had very high production values, glossy paper, multi-color printing, and artful layouts. "Given the bibliographies, footnotes, charts and tables, policymakers or laypeople may be forgiven for thinking that these honoree reports are based on the highest quality research. We hope that our expert reviews have helped to correct that impression," said EPIC director Kevin Welner.

The 2009 Bunkum Award honorees:

The Time Machine Award. Weighted Student Formula Yearbook 2009

Authored by Lisa Snell, and published by the Reason Foundation.

In a truly breathtaking innovation, the report enters its time machine and attributes positive reform outcomes to policy changes that had not yet been implemented.

The Data Dodger Award — How New York City's Charter Schools Affect Achievement

Authored by Caroline M. Hoxby, Sonali Murarka & Jenny Kang, and published by the National Bureau of Economic Research.

New York City's charter schools might genuinely be improving student outcomes; however, this study — because of the information it withheld and its methodological shortcomings — does not and cannot resolve the issue.

The Misdirection Award: Keep Our Eyes Off What Works — Reroute the Preschool Juggernaut

Authored by Checker Finn and published by the Hoover Institution.

The report misdirects readers from a mountain of empirical, peer-reviewed and widely accepted evidence, and instead cherry-picks a few weak studies to critique proposals for universal preschool.

The Innovations in Promoting Alternative Certification Award — An Evaluation of Teachers Trained Through Different Routes to Certification: Final Report

Authored by Jill Constantine, Daniel Player, Tim Silva, Kristin Hallgren, Mary Grider & John Deke, and published by the U.S. Department of Education, Institute of Education Sciences.

The authors report 'no evidence' that traditionally trained teachers provided better student scores than alternatively trained teachers. The report does not bother to set forth caveats to the 'no evidence' conclusion, but there should, in fact, have been many, many caveats — including small sample size, sampling methods, and a failure to distinguish the treatments. Also interesting: the study actually included many analyses that found traditionally trained teachers outperformed their alternative route counterparts. It's just that the authors chose not to fully report and acknowledge these findings in the report's conclusions.

The Annual Friedman Foundation Johnny One-Note Award — Multiple Reports by the Friedman Foundation

Multiple authors, all published by the Friedman Foundation.

The Friedman Foundation has, over the past three years, cloned the same study on the cost of drop-outs in at least seven states, a tax credit voucher report in at least six states, and opinion polls on school choice in 15 states. Amazingly, all these reports lead to the same conclusion: vouchers and other forms of school choice will save money and improve student outcomes. The basic technique used by Friedman researchers is to take the same report, change the name of the state, plug in some state-specific data, vary the title a bit, and come up with the predetermined conclusion.

This year's honorees were selected following expert third-party reviews of research reports published by think tanks and other research organizations. Reports reviewed by the Think Tank Review Project are carefully selected. Every day the web sites of prominent think tanks are visited to identify new research publications for possible review. If a report is deemed of sufficient importance, it is then assigned for review to an independent scholar with expertise in the area of inquiry.

A complete analysis of this year's Bunkum Award winners can be found at:

About the Bunkum Awards

The term 'bunkum,' meaning essentially 'nonsense,' came about because of a long-winded and pointless speech given in 1820 on the House floor by Congressman Felix Walker of Buncombe County, North Carolina. The Bunkum Awards help to highlight nonsensical, confusing, and disingenuous reports produced by education think tanks.


Nikki Rashada McCord

Associate Director

Education and the Public Interest Center (EPIC)

University of Colorado at Boulder

(303) 735-5290 


Add your own comment (all fields are necessary)

Substance readers:

You must give your first name and last name under "Name" when you post a comment at We are not operating a blog and do not allow anonymous or pseudonymous comments. Our readers deserve to know who is commenting, just as they deserve to know the source of our news reports and analysis.

Please respect this, and also provide us with an accurate e-mail address.

Thank you,

The Editors of Substance

Your Name

Your Email

What's your comment about?

Your Comment

Please answer this to prove you're not a robot:

1 + 3 =