MEDIA WATCH: Research challenges use of PISA test results to show how poorly students in the USA are doing...
A new study by researchers at the University of Colorado raises serious questions about the integrity of claims widely reported in the media in the United States of America that American students constantly "lad behind" their peers in other countries. The study is based on a critique of the PISA exams and deserves to be widely understood, although given that most American media have by now a vested interest in the narrative about how American public schools are "failing," some truth in reporting remains unlikely.
The research report follows here:
From News from the NEPC: PISA Country Rankings Misleading, Friday, October 30. 2015., PISA Country Rankings Misleading
Brief highlights flaws in the international testing to drive educational policy Contact:
William J. Mathis, (802) 383-0058, wmathis@sover.net
Martin Carnoy, (650) 725-1254, carnoy@stanford.edu
URL for this press release: http://tinyurl.com/pszpfjf
BOULDER, CO (October 30, 2015) - For 15 years, journalists, advocates and policymakers have cited scores on international tests, such as the Program for International Student Assessment (PISA), to conclude that American student achievement "lags woefully behind" other nations, threatening our future and suggesting an urgent need for education reform.
A brief published today by the National Education Policy Center at the University of Colorado Boulder explores such policy analyses and claims around PISA as well as a second prominent international test, the Trends in International Mathematics and Science Study (TIMSS).
In International Test Score Comparisons and Educational Policy: A Review of the Critiques, Stanford education professor Martin Carnoy focuses on four main critiques of analyses that use average PISA scores as a comparative measure of student achievement.
The ranking is misleading, Carnoy asserts, because:
* Students in different countries have different levels of family-not just school-academic resources;
* The larger gains reported on the TIMSS, which is adjusted for different levels of family academic resources, raise questions about the validity of using PISA results for international comparisons.
* PISA test score error terms -- the difference between measured achievement and actual achievement -- are considerably larger than the testing agencies acknowledge, making the country rankings unstable.
* The Shanghai educational system is held up as a model for the rest of the world on the basis of data on a subset of students that is not representative of the Shanghai student population as a whole.
Professor Carnoy also assesses the underlying social meaning and education policy value of international comparisons. First, he describes problems with claims that average national math scores are accurate predictors of future economic growth. Second, he explains that using scoring data in this manner has limited use for establishing education policy, due to the lack of causal inference analysis. This is the well-known "correlation is not causation" problem.
Third, there is a conflict of interest arising from the Organization for Economic Cooperation and Development (which administers the PISA) and its member governments acting as a testing agency while simultaneously serving as data analyst and interpreter of results for policy purposes.
Fourth, Carnoy questions the usefulness of nation-level test score comparisons with regard to countries such as the United States with such diverse and complex education systems. The differences between states in the U.S. are so large that employing state-level test results over time to examine the impact of education policies would be more useful and interesting.
Despite such compelling critiques of international testing, Carnoy concludes that these tests will not go away, nor will they stop being wrongly applied to shape educational policy. However, he says, "there are changes that could be made to reduce misuse." He concludes with five policy recommendations, including reporting test results by family academic resource subgroups of students with different levels of resources.
In a companion report released today by the Economic Policy Institute (EPI), Carnoy and co-authors Emma Garc�a and Tatiana Khavenson provide detailed analyses explaining how and why comparisons using data at the level of U.S. states are more useful than comparing the U.S. with other countries for understanding and improving student performance.
Find International Test Score Comparisons and Educational Policy: A Review of the Critiques by Martin Carnoy on the web at:
http://nepc.colorado.edu/publication/international-test-scores .
--------------------------------
The National Education Policy Center (NEPC) is housed at the University of Colorado Boulder School of Education. Its mission is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information about the NEPC, please visit http://nepc.colorado.edu/.
