The Market for Predictions

Andrew Gelman and John Sides have a “very good piece”:http://bostonreview.net/BR34.5/ndf_election.php at the _Boston Review_ on the reasons why journalists and pundits got so much about the 2008 presidential election wrong, with responses by Rick Perlstein, Mark Schmitt and others. In their response to the response, John and Andrew say:

bq. Will these efforts get political scientists invited to Joe Scarborough’s kaffeeklatsch? Probably not. The media ecology fetishizes novelty in reporting and certainty in commentary. And yet the academic study of elections shows that what is certain is almost never new, and what is new is almost never certain. We might only bore Fox & Friends with our scholarly qualifications and caveats, or simply look foolish trying to present our research in soundbites.

This has interesting overlap with this “review article”:http://www.nationalinterest.org/Article.aspx?id=22040 by Philip Tetlock on political economic forecasting in _The National Interest._

bq. To appreciate why, it is helpful to think like a game theorist—to take the forecasters’ perspectives and consider the incentives confronting them. Both private- and public-sector prognosticators must master the same tightrope-walking act. They know they need to sound as though they are offering bold, fresh insights into the future not readily available off the street. And they know they cannot afford to be linked to flat-out mistakes. Accordingly, they have to appear to be going out on a limb without actually going out on one. That is why (with the interesting exception of Bueno de Mesquita), they so uniformly appear to dislike affixing “artificially precise” subjective probability estimates to possible outcomes—the only reliable method we have of systematically tracking accuracy across pundits, methods, time and contexts. It is much safer to retreat into the vague language of possibilities and plausibilities—things that might or could happen if various difficult-to-determine preconditions were satisfied. The trick is to attach so many qualifiers to your vague predictions that you will be well positioned to explain pretty much whatever happens. China will fissure into regional fiefdoms, but only if the Chinese leadership fails to manage certain trade-offs deftly, and only if global economic growth stalls for a protracted period, and only if . . . And if you venture specific policy recommendations—such as invading Iraq or deregulating financial markets—make sure to leave yourself the fallback position: “Well, of course, my recommendation was fundamentally sound, but how was I to know that the idiots in charge would implement things so badly. If only they had . . . ”

bq. Having mastered this subtle balancing act, why should these private- and public-sector pundits open their reputations and livelihoods to the unpredictable risks of competing against each other in level-playing-field forecasting exercises?

As Andrew and John say, there is a premium on bold, ‘innovative’ claims in the market for pundits, but as Tetlock suggests, there is also some room for artful hedging, so that apparently unambiguous predictions turn out on closer examination to be carefully hedged. Myself, I’d add three qualifications.

First – that the bold, innovative claims can’t be too bold, or too innovative. They certainly shouldn’t challenge basic precepts of American public debate such as the awesomeness of US world leadership. Even better if you can dress mutton as lamb, by presenting old chestnuts as if they were exciting, contrarian insights (a good half to two thirds of the average US policy journal consists of articles trying to pull off this trick, and usually failing).

Second, that the incentives to engage in careful hedging may be considerably weaker than Tetlock suggests. There aren’t any very good repositories of public memory in the arena of US public debate – pundits who have made grand pronouncements that turn out to be horribly wrong usually aren’t called on it. Here, blogs have served as a modest corrective, reminding op-ed writers of past confident assertions that they would likely have preferred to have remained buried. However, the emphasis here should be laid on the word ‘modest.’ Amity Shlaes, Max Boot et al. still hold nice fellowships at the Council on Foreign Relations, despite the scorn and ridicule that was rightly heaped upon their heads.

Third, that in any event, there are herding effects – the costs of making bold, confident assertions that turn out to be wrong are much lower when lots of other prominent people are making the same public mistake. In the event that the pundit is called on his mistake later, he can not only point to the fact that others made the same mistake too but can likely rely on those others to help circle the wagons against the nay-sayers. I still haven’t read a good intellectual history of the Iraq war debate in the US, but these three factors all seem to me to have played a significant role in producing the horrible public debates that we saw in 2002-2004. I “mentioned”:https://crookedtimber.org/2009/08/20/brandin/ Diego Gambetta’s wonderful book on signalling among criminals in my last blogpost – it seems to me that there is scope for a similar book about signalling and strategic interaction among US foreign policy, journalistic and political elites interested in coordinating with each other, accruing resources etc. This could perhaps really get at the underlying pathologies of political debate in this country.