European Scholars: PISA Rankings Are “Utterly Wrong” and “Meaningless”

This from Diane Ravitch:

Professor Svend Kreiner, a prominent statistician and psychometrician at the University of Copenhagen in Denmark and Dr. Hugh Morrison of Queens University in Belfast have published studies blasting the reliability and validity of the PISA league tables. They describe PISA’s rankings as “useless,” “utterly wrong,” and “meaningless.”

According to TES (London),

“Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, said that an inappropriate model is used to calculate the Pisa rankings every three years. In a paper published this summer, he challenges their reliability and shows how they fluctuate significantly according to which test questions are used. He reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.

“Dr Hugh Morrison, from Queens University Belfast in Northern Ireland, goes further, saying that the model Pisa uses to calculate the rankings is, on its own terms, “utterly wrong” because it contains a “profound” conceptual error. For this reason, the mathematician claims, “Pisa will never work”.

“The academics’ papers have serious implications for politicians, including England’s education secretary Michael Gove, who justified his sweeping reforms by stating that England “plummeted” down the Pisa rankings between 2000 and 2009.

“The questions used for Pisa vary between countries and between students participating in the same assessment. In Pisa 2006, for example, half the students were not asked any reading questions but were allocated “plausible” reading scores to help calculate their countries’ rankings.

“To work out these “plausible” values, Pisa uses the Rasch model, a statistical way of “scaling” up the results it does have. But Professor Kreiner says this model can only work if the questions that Pisa uses are of the same level of difficulty for each of the participating countries. He believes his research proves that this is not the case, and therefore the comparisons that Pisa makes between countries are “useless”.

“When the academic first raised the issue in 2011, the OECD countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point. But Professor Kreiner’s new paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis.

“He is sceptical about the whole concept of Pisa. “It is meaningless to try to compare reading in Chinese with reading in Danish,” he said.

“Dr Morrison said that the Rasch model made the “impossible” claim of being able to measure ability independently of the questions that students answer. “I am certain this (problem) cannot be answered,” he told TES.”

To read Dr. Kreiner’s studies, google his name.