Joining up the dots II

“Like Henry”:https://crookedtimber.org/2005/03/29/joining-up-the-dots/ and “Max”:http://maxspeak.org/mt/index.html, I got a bit of a laugh out of the “Left Business Observer’s plots”:http://www.leftbusinessobserver.com/FreedomIndex.html of the Heritage Foundation’s “Freedom Index.” I think the LBO are right to be skeptical of the index. But maybe the scatterplots they show sell it a bit short.

The _Observer_ article finds that the correlation between each country’s 1996 Freedom Score and its 2003 per capita GDP (in PPPs) is a derisory -0.02. There’s no association. To give the FI a better shot, they also check the relationship between the _change_ in a country’s freedom score and overall GDP growth between 1997 and 2003 (in 1995 US Dollars). The association is a little better, with a correlation of 0.33. LBO says “that’s still very far from impressive; the improvement in the index can explain statistically less than 10% of GDP growth – ignoring the fact that growth itself probably explains some of the improvement in the index. An analysis of the second chart by the naked eye would conclude the relationship is as good as random.”

Like I say, I don’t disagree with the basic point they’re making, but that comment about eyeballing the data caught my, uh, eye. I wondered whether plotting the raw numbers rather than the rankings (for the freedom score and GDP) would make a difference to how the chart appeared. It seemed like the way the scales are constrained for rankings might make the association look weaker on a scatterplot. I don’t have the original data, so just to illustrate the idea here are some plots of random variables (n=125, like the LBO plots) with the rank correlations constrained to be about 0.33 or so. (Also available as “a PDF”:http://www.kieranhealy.org/files/misc/val-rank-comparison.pdf.)

The top panel shows a scatterplot of the rank of each variable. As you can see, the association is pretty poor, and if you were just looking at the data points you’d probably say the relationship was as good as random. But a “loess smoother”:http://www.itl.nist.gov/div898/handbook/pmd/section1/pmd144.htm in the plot nevertheless picks out a positive trend line. This is surprising on the face of it. But now look at the bottom panel, which shows a scatterplot of the same data with the _raw_ values plotted rather than their ranks. It’s still noisy, and the association isn’t terrifically strong, but the trend line now looks more reasonable. (The lines aren’t identical, by the way: they’re separately fitted to the data in each panel.) Anyone eyeballing the data would say there seemed to be a weak positive relationship between the variables. The correlation is slightly higher (r=0.4), but not that much higher. It’s interesting to me that one panel looks so much more like white noise than the other, even though their correlations are comparable and the smoothed regression line picks up the same trend in both cases.

I don’t think this lets Heritage off the hook, though I’d be interested in seeing the association between the raw scores on the freedom index and the growth numbers. The broader issue is the plausibility of the measure: Heritage says it’s constructed by rating countries using a 1-to-5 scale for each of 50 variables. That’s an awful lot of variables. Why score them on a 1-to-5 scale, anyway? Wouldn’t more fine-grained measures be available in many cases? I’d be interested to see how much redundancy there is across these measures, and also how they aggregated them. Collapsing all that information (whether good or bad) into a single index is a tricky business. Did they just take the average of all 50?

On the other hand, we might say it’s actually to Heritage’s credit that there appears to be at best a weak association between the Freedom Index and growth. With that many variables to play with, and so many decisions to make about how to combine them, they could have cooked the numbers all sorts of ways until they got the kind of predictions they wanted. Or maybe they did, except they calibrated it to a different outcome or something.