In the practical community of professional journalists writing about political events, the term “open data” is hardly ever in circulation. And yet, to those who are doing the best work it’s an invaluable tool. David Simon succeeded in turning the idea that information age journalists need to learn to “do more with less” into a national joke, but the underlying concept makes perfect sense. The very same information technology revolution that’s undermined the business models of traditional newspapers has done an enormous amount to increase the productivity of working journalists. Open data is an enormous part of that.
Especially for those of us who want to do informed commentary on economic issues, the FRED database and associated tools that the Federal Reserve Bank of St Louis has compiled is invaluable. Its companion set ALFRED that let’s you compare different iterations of the same data series as agencies revise their estimates is, if anything, even more amazing. For example, it took me about fifteen minutes to throw together a chart comparing current GDP estimates for the critical Q3 2007–Q3 2009 period to those available to policymakers in 2009. Debates about the adequacy of the policy response to the recession should be informed by the reality that the economic shrinkage began a full quarter earlier than was contemporaneously known and that the decline during the winter of 2008-2009 was much more severe than people realized.
This basic National Income and Product Account data has always in some sense been available, but the internet and the determination of the FRED team have made it much more available than ever before. And it makes a difference, as FRED outputs are a regular feature on my blog, on Joe Weisenthal’s policy writing at Business Insider, on Ezra Klein’s Wonkblog, on Paul Krugman’s blog for the New York Times, and wherever else on the internet serious economic policy discussion is taking place.
In debates on the value of open data, some put what I think is undue weight on a distinction between commercial and civic activity that the case of journalism tends to undermine. The New York, clearly, is a commercial enterprise that’s also primarily controlled by a founding family that sees it as serving some civic functions. Krugman, personally, is paid for his work but it beggars belief to imagine that he’s driven by purely pecuniary motivations. And journalists of all kinds are dependent, on one level or another, on non-compensated contributions from quoted sources, experts used for background, or freely available data sources. An civic-minded person might want to write for or be quoted in a commercial publication precisely because the engine of commerce is a powerful motive to widely disseminate information.
The fundamental issue is that as the marginal cost of transmitting information falls ever closer to zero, two things happen simultaneously. One is that it becomes increasingly difficult to internalize the value of information-production because the facts (or “facts”) once unleashed into the world tend to spread beyond the control of the producer. The second is that for that very same reason, information becomes more socially valuable. Governments are ideally situation to serve as producers of these goods. In the U.S. debate this is widely acknowledged in the special case of basic scientific research, where there’s a strong bien pensant consensus that subsidies are socially and economically valuable. But the issue has nothing in particular to do with science. As the marginal cost of information distribution falls, market systems increasingly fail to produce it at an optimal level. Governments should step in wherever it seems feasible to do so. The push for “open data” is best viewed as, like scientific research, a particular case of this general principle.
Whether this will actually lead to better politics in the end has more than a little to do with the question of to what extent political decisions are actually driven by information. I’m somewhat skeptical on this score that they are. But even if they aren’t, all you need to believe is that some important decisions of some sort are driven by information to conclude that more production and more open dissemination of data of all kinds is of enormous pote