Joining up the dots II

by Kieran Healy on March 29, 2005

“Like Henry”: and “Max”:, I got a bit of a laugh out of the “Left Business Observer’s plots”: of the Heritage Foundation’s “Freedom Index.” I think the LBO are right to be skeptical of the index. But maybe the scatterplots they show sell it a bit short.

The _Observer_ article finds that the correlation between each country’s 1996 Freedom Score and its 2003 per capita GDP (in PPPs) is a derisory -0.02. There’s no association. To give the FI a better shot, they also check the relationship between the _change_ in a country’s freedom score and overall GDP growth between 1997 and 2003 (in 1995 US Dollars). The association is a little better, with a correlation of 0.33. LBO says “that’s still very far from impressive; the improvement in the index can explain statistically less than 10% of GDP growth – ignoring the fact that growth itself probably explains some of the improvement in the index. An analysis of the second chart by the naked eye would conclude the relationship is as good as random.”

Like I say, I don’t disagree with the basic point they’re making, but that comment about eyeballing the data caught my, uh, eye. I wondered whether plotting the raw numbers rather than the rankings (for the freedom score and GDP) would make a difference to how the chart appeared. It seemed like the way the scales are constrained for rankings might make the association look weaker on a scatterplot. I don’t have the original data, so just to illustrate the idea here are some plots of random variables (n=125, like the LBO plots) with the rank correlations constrained to be about 0.33 or so. (Also available as “a PDF”:

The top panel shows a scatterplot of the rank of each variable. As you can see, the association is pretty poor, and if you were just looking at the data points you’d probably say the relationship was as good as random. But a “loess smoother”: in the plot nevertheless picks out a positive trend line. This is surprising on the face of it. But now look at the bottom panel, which shows a scatterplot of the same data with the _raw_ values plotted rather than their ranks. It’s still noisy, and the association isn’t terrifically strong, but the trend line now looks more reasonable. (The lines aren’t identical, by the way: they’re separately fitted to the data in each panel.) Anyone eyeballing the data would say there seemed to be a weak positive relationship between the variables. The correlation is slightly higher (r=0.4), but not that much higher. It’s interesting to me that one panel looks so much more like white noise than the other, even though their correlations are comparable and the smoothed regression line picks up the same trend in both cases.

I don’t think this lets Heritage off the hook, though I’d be interested in seeing the association between the raw scores on the freedom index and the growth numbers. The broader issue is the plausibility of the measure: Heritage says it’s constructed by rating countries using a 1-to-5 scale for each of 50 variables. That’s an awful lot of variables. Why score them on a 1-to-5 scale, anyway? Wouldn’t more fine-grained measures be available in many cases? I’d be interested to see how much redundancy there is across these measures, and also how they aggregated them. Collapsing all that information (whether good or bad) into a single index is a tricky business. Did they just take the average of all 50?

On the other hand, we might say it’s actually to Heritage’s credit that there appears to be at best a weak association between the Freedom Index and growth. With that many variables to play with, and so many decisions to make about how to combine them, they could have cooked the numbers all sorts of ways until they got the kind of predictions they wanted. Or maybe they did, except they calibrated it to a different outcome or something.

{ 1 trackback }

Crooked Timber » » A mess of pottage
04.16.05 at 10:23 pm



DeadHorseBeater 03.29.05 at 4:24 pm

As pointed out in the other comment thread, shouldn’t we really be looking at some kind of multivariate regression here?
Even the folks at Heritage must think that there are other things relevant to GDP levels or growth than their FI. Omitting variables both causes bias (could be either way) and loss of precision (makes correlations weaker).


Kimon 03.29.05 at 11:04 pm

deadhorsebeater: That’s what Sala-i-Martin did in “I just ran four million regressions” (NBER W6252). He found a significant correlation with a few of his variables, including “rule of law”, “political rights”, “civil liberties”, and “degree of capitalism” (all positive).


John Emerson 03.30.05 at 12:10 am

Possibly if Heritage had used fewer variables (ie. removed some of their pet ideas) they would have got something more meaningful. Which I guess is the point of what the other guys were saying.


Daniel 03.30.05 at 12:33 am

I don’t think the top plot looks like white noise at all; certainly less so than the LBO scatterplot. It’s got two “empty corners”.


Jonathan Dursi 03.30.05 at 1:40 am

Ok, well, my sense of fun runs that way, so I did what I could and the data is at . From the free worldbank data I could only get GNI/capita (PPP) back to 1999. I only considered those countries with freedom index data for 2005, 2003, and 1999, and GNI/capita(PPP) data for 1999-2003. A graph of fractional change in GNI/capita (PPP) 1999-2003 compared to the mean value over that period vs. 1999 freedom index is shown at . It’s late, so it wouldn’t shock me at all if I got some of the numbers wrong.


JStat 03.30.05 at 3:06 am

The Freedom Index , like most survey data, is poor. This is primarily because you are attempting to treat the measure of a construct that is purely ideologically based into a meaningful , continuous variable from which you can derive information. For example we know that there is a real measurable difference between an individual that is 5 feet tall and one that is 6 feet tall. We also know that heights in general conform to a distribution and that probabilities can be calculated for height ranges based on the normal distribution and that if we would compare this with real data we would get similar results to the theoretical ones. But what meaningful difference is there between a country that is 5 or 4.5 on the “Freedom Index”, what is a .1 difference , a .2 difference? The construct itself , “freedom”, is one that I believe cannot be measureable mathematically (those of you who know measure theory help me out here). The more of the story is: Crap begets crap.


Travis Thomas 03.30.05 at 3:31 am

This also does not take into account the level of development in these countries. China can afford a lower degree of economic freedom and still attain high economic growth because they’re playing catch-up (and as the July/August 2004 Foreign Affairs article “The Myth Behind China’s Miracle” noted) and seeing relatively little in the way of purely domestic development and instead relying on already efficient methods of production developed in freer countries (a level of efficiency that would not arise in a command economy) being imported by foreign firms.


John Quiggin 03.30.05 at 3:37 am

A particularly interesting feature of the Heritage Index is that big government is good for economic freedom


DeadHorseBeater 03.30.05 at 4:37 am

Yeah, I forgot about the Sala-i-Martin paper.
Best. Title. Ever.
All I could remember was this one Barro paper I once read.


dsquared 03.30.05 at 5:41 am

FWIW, the correlation in Jonathan’s dataset is 0.08 (ie next to no relationship) between 1999 score and 1999-2003 growth, unless I have made a spreadsheet error which I surely have. Eyeballing the data, it’s notable that the real growth stars of the period 1999-2003 were Turkmenistan, Nicaragua, Albania, Latvia and Lithuania (of which Latvia and Lithuania had better than average EF scores) while the worst performers were Argentina, Uruguay, Uzbekistan, Paraguay and Brazil (of which Uruguay had a better than average score; all the rest worse). The average for the whole sample was 3.146 (low scores good); the top quintile by growth rates was 3.425 and the bottom quintile by growth rates was 3.248. Thus telling you that the low scores were associated with being a dull developed country in the middle of the distribution, which was pretty much free information anyway.


derrida derider 03.30.05 at 6:00 am

When there is no a priori reason for assuming a linear relationship and normally distributed error then use of non-parametric measures (ranks, in this case) is pretty well mandatory, at least as long as you’re using simple regression.

The very low rank correlation is likely to be a far better indicator of the true relation than an OLS regression (which of its nature must make strong implicit assumptions about functional forms and error structures).


dsquared 03.30.05 at 6:00 am

Indeed I had made a spreadsheet error, and I now reproduce results which I think are right.

Correlation between:

1999 score and growth 1999-2003: 0.14
change in score 1999-2003 and growth 1999-2003: -0.25
1999 score and growth relative to average: 0.12
change in score and growth relative to average: -0.25

Remember that lower scores mean more freedom, so the correlations between the 1999 score and 1999-03 growth have the “wrong sign”. I conclude that there is basically no relationship here, other than that possibly Doug is right in saying that the influence of growth on freedom seems to have a bit of explanatory power over the drift in the Freedom scores.


nnyhav 03.30.05 at 8:21 am

> Best. Title. Ever.

Would have been, were it “I just ran four million regressions, and all I got was this stupid t-score.”


Michael Mouse 03.30.05 at 9:52 am

Worth noting that weak evidence of a weak correlation is no evidence of causation (still less direction of causation) at all.

I’d guess that third variables like “degree of armed conflict” might predict both FI and GDP better than either predict each other.


snacknuts 03.30.05 at 11:55 pm

martin wolf came up with something similar last week…


“What, then, do the charts show about the link between government spending and economic performance? There is none, is the answer.”

“What does indeed matter is the efficiency with which money is both raised and spent. But tax levels are only one of many determinants of economic performance. Far more important are: the quality of institutions, particularly of public administration and the judiciary; the security of property; the probity and public spiritedness of politicians; the soundness of money; the quality of education, health and infrastructure; and the extent of arbitrary regulation of economic activities.”

In making the decision on what to put into the public sector and how much to spend on it, we have to place substantial weight on underlying social and political values. But we must also ask, first, what we must do through the government (defence and law and order, for example); second, what we want, for reasons of social solidarity, to do through government (provision of basic incomes for all, of universal education and of basic health services, for instance); third, whether we wish to pay for services through general taxation or user fees; fourth, what is the least costly way of raising revenue; and, finally, whether we want services to be paid for and provided by government or merely paid for by government and provided by competitive private suppliers.”

“What matters here, as elsewhere, is not what you do but the way that you do it.”

also fwiw :D


Comments on this entry are closed.