The leaning tower of PISA rhetoric


The U.S. is doing terribly! China is beating us! It’s all the other guy’s fault!

It must be PISA time again. The once-every-three-years international standardized test has released its 2012 scores and rankings. Scores in the US are down, and our ranking remains middle of the pack, trailing much of the rest of the developed world. This has prompted a wave of consternation, much of which isn’t helpful. We’ll take a look at some of the common themes.

The U.S. is flailing.
Scores in the U.S. were down in 2012 (after having increased between 2006 and 2009). We can be embarrassed to have been bested by the likes of Estonia, Slovenia, and Latvia, and we share statistically equivalent company with Norway, Sweden, Russia, Italy, Spain, and Hungary, among others. Leading the charts are Asian jurisdictions — Shanghai, Singapore, South Korea, Japan — with some small European countries like Switzerland and Liechtenstein for flavor.

Then again, the U.S. has always had mediocre test scores. Yes, we live in an increasingly globalized world, so there is some reason for concern that this will matter more than it has in the past. Still, our internal equity problems should probably be a bigger deal to us than persistent worries about our global competitiveness.

China’s much smarter than we are.
There’s a regular bit of sleight-of-hand that happens when we discuss PISA scores, in which the stellar scores from Shanghai are generalized to all of China. This, it turns out, is like generalizing the scores from 2012 Manhattan to the United States in the early twentieth century. Roughly two-thirds of China remains rural, and the top of the education reform to-do list is increasing attendance and access. These are issues we grappled with for much of the last century, but which we have since largely transcended.

Shanghai specifically is a city of college-going elites which excludes low-income migrant families from the schools assessed by PISA. It is in no way representative of China as a whole.

If only we did This Thing I Like, we’d be at the top of the charts, too!
All of the top-scoring and gain-making countries have national curricula, and we don’t! We should adopt one as soon as possible! Oh, hold on a second… pretty much all of the low-scoring and declining countries have national curricula, too. Hmm.

Say it with me: “Correlation is not causation.” Figuring out what thing, or two things, or five dozen things a country has done that impacted their PISA scores is much more complicated [PDF] than seeing what the folks on top are doing.

There are probably some useful lessons we can draw from a careful analysis of PISA data and the contexts and policies that shape those scores. We probably can’t do it in the two or three days after the scores have come out. Breathe, think, and be careful not to read too much into the early punditry.