Andrew Gelman links to an interesting piece by Dave Armstrong on the Heritage Foundation Economic Freedom Index. I have my own issues with some of the subtleties of the index, but the approach taken is a good example of how a quantitative approach can sometimes miss the qualitative point altogether. In the analysis of the components of the economic freedom, the assumption of unidimensionality is made.
“Unidimensionality - that there is only one underlying source of variation. Here we are assuming that each indicator is an imperfect indicator of Economic Freedom and that once we take account of economic freedom, there is only random variation left over.”
By making this assumption, he can then decide that factors that aren't correlated to the common factor of the other variables do not actually measure economic freedom. In the subsequent analysis, fiscal freedom, a measure of taxes, and government spending are found to be uncorrelated with the common factor of variables. This is where a quick qualitative check would have been helpful. The amount of money the government controls and taxes from the private sector is a priori related to economic freedom. Instead of assuming that government spending and fiscal freedom are not related to economic freedom, the assumption of unidimensionality should be reassessed.
The new index created with the unidimensionality assumption that takes out government spending and taxes could be very interesting as a rule of law index, it just isn't an index of economic freedom.
The quantitative approach does yield some useful insights, as the article does make a good point about statistical significance of the change in rankings. The United States index score is still well within its historical range, so the United States's move off of the list of most free countries is not really statistically significant. The most relevant part of the ranking change is that over the ten year period certain countries have become more free, but that is a relative and not absolute US decline.