A working paper by students at Hamilton College out yesterday has the laudable aim of auditing the predictions made by political pundits in order to see whether they are any use or not. Unsurprisingly, it finds that Paul Krugman is the most useful columnist and that a bunch of hacks I’ve never heard of are the worst (it also, wonderfully, gives the success formula for prognostication as “avoid law school and adopt a liberal philosophy). Below the fold, a few points on a subject which many readers will know is dear to my heart.
First, this is an amazingly praiseworthy exercise. In one of the more successful posts on my personal blog, I noted that “the vital importance of audit” is one of the things they drill into you in business school. In general, people who don’t systematically keep track of how projections turned out, tend to get the quality of predictions they deserve. However, having said that, I would make two fairly significant qualifications to the methodology involved, from the perspective of someone who basically makes a living out of making predictions and assessing those of others.
First, not all bets are equal. The performance metric used by the Hamilton team judges pundits on the basis of a pass-fail record of their predictions, equal weight given to each. But in the real world, people make predictions with a greater or lesser degree of certainty. You can be right materially less often than 50% of the time (“worse than a coin flip” or “ugly” by the paper’s classification) and still make a lot of money if you bet big when you are reasonably confident of being right, and bet small when you are less sure. This fact is what makes card-counting in blackjack work, for example – a card-counter still wins fewer than half the hands by number, but wins big bets and loses small ones. Similarly, it is axiomatic in backgammon that no amount of skill in moving the pieces will make up for poor play with the doubling cube. In principle, since the HC dataset contains a variable for the degree of certainty with which a prediction was expressed, the skill metric could be adjusted to take this into account – I suspect this would end up further polarising the scores but not necessarily changing their rank order.
Second, staying wrong is worse than being wrong. Predicting the future is hard, which is why so many people do it so badly. Predicting the recent past ought to be a lot easier, which is why it’s somewhat more surprising that it is also, in general, done quite badly. To use a nautical analogy, while it’s obviously great to be able to forecast the weather, it is often just as usual to have a nimble crew that can quickly trim your sails when the wind has changed. Philip Tetlock’s original study on prediction and pundits suggested that pundits often made mistakes by being too keen to predict change and underestimating the likelihood of the status quo, but this isn’t wholly a bad quality – hanging on to an outdated view of the world is often really very dangerous, and the risk/reward tradeoff is usually asymmetric[1]. I think it would be harder to measure this in quantitative ways, but it is highly worth bearing in mind.
So, although the development of even rudimentary forms of audit is a great boon to the democratic public (and probably a lot more so than yet another inconclusive study of “media bias” one way or the other), I think it needs to be taken with two caveats. The biggest villain is not the guy who gets it wrong. The people who will cost you money and reputation over the long run are first, the guy who says he’s more certain than he really is, and second, the guy who won’t admit he’s wrong when he knows he is. Hang on to your wallet out there.
[1] As Bob McTeer said ” Fortunately, most of the time the question is whether to zig or not to zig. When you are contemplating a zig, a zag is usually not even a consideration.”
{ 28 comments }
someguy 05.03.11 at 6:36 pm
https://crookedtimber.org/2004/08/19/the-correct-way-to-argue-with-milton-friedman/
Henry 05.03.11 at 6:57 pm
I’d point out that this isn’t the first such exercise – my late colleague Lee Sigelman did something similar in the mid-1990’s for the pundits of the McLaughlin group, a particularly obnoxious shoutfest of the period. Alan Hirsch’s _Talking Heads: Political Talk Shows and Their Star Pundits_ is purely qualitative, and very dated, but has some good material.
michael e sullivan 05.03.11 at 7:15 pm
“Similarly, it is axiomatic in backgammon that no amount of skill in moving the pieces will make up for poor play with the doubling cube.”
Just to be a completely pedantic nit, your backgammon analogy isn’t quite right. I happen to be a very good, though well short of world class backgammon player. I would probably be favored versus world best players or bots in a long session if they were forced to let some novice or novice level algorithm make all their cube decisions. But an average player might not be. Switch it up, let the world class player make all the cube decisions, but make a novice player move the checkers, and they’d almost never beat someone like me in a long session, and they’d be a big dog to anybody who understood the basic fundamentals of strategy and tactics (i.e. an average rated player at any major online site).
cube decision errors tend to be much bigger than checker errors, but there are so many more non-trivial checker play decisions that normal players give up much more equity in play than in cube decisions.
Your point is still relevant though. The cube makes it possible to win when you aren’t winning, so to speak. One very good way to handicap a backgammon game is for one player to start with the cube on their side of the board. If I can first cube, but my opponent cannot, I recall a calculation suggesting that that alone would be enough advantage to put me even with players ~150 ELO points ahead of me (the spread between average avid online players and world-class is about 5-600 points).
H 05.03.11 at 7:16 pm
Yes, but… Pundits make predictions not just to predict, but to influence. So a conservative pundit in this period (around the 2008 election) would “predict” that conservatives were going to do well, in order to encourage fellow conservatives. (“Don’t give up and stay home, there’s still a good chance our side can win.”) My guess is that if you did an identical study in the period September 2009 to December 2010 (around the 2010 election), conservative pundits would be “more accurate”.
NBarnes 05.03.11 at 7:28 pm
Actually, it seems to me that there’s a fair amount of debate on the zig vs zag front on economic policy, both in the US, the UK, and the continent. There’s strong support for everything from the fairly radical supply-side semi-austerity of Paul Ryan’s budget in the US, along with strong austerity in the UK, to counter-cyclic Keynesianism.
Substance McGravitas 05.03.11 at 7:43 pm
And? I’m still interested in the ones who are more accurate. Set your study period for any time you like.
Satan Mayo 05.03.11 at 7:47 pm
I’d point out that this isn’t the first such exercise – my late colleague Lee Sigelman did something similar in the mid-1990’s for the pundits of the McLaughlin group, a particularly obnoxious shoutfest of the period.
I actually discovered earlier this year that The McLaughlin Group is still being created and, now and then, broadcast! Approaching its 30th anniversay.
chrismealy 05.03.11 at 7:56 pm
How should you score people who can’t manage to predict the present?
Martin Bento 05.03.11 at 8:21 pm
Looking over the paper quickly, it looks like there is no consideration of the apparent or actual likelihood of the prediction. I can predict the sun will rise tomorrow with astounding accuracy. Predicting that Osama bin Laden would be killed by US armed forces in a suburb in Pakistan in 2011 should count for much more than predicting, in 2002, that he would be killed somewhere by someone sometime in the next 9 years. But both predictions were fulfilled last week, and it looks like this study would score them equally. I realize that quantifying probabilities of complex political events (unlike things like elections, which have discreet well-defined outcomes that are usually extensively polled) will inevitably involved some arse-pluckery. But a study like this seems meaningless without it, so make your best guess, tell us explicitly what it is, so we can judge it, and tickle the ole’ posterior.
Andres Corrada-Emmanuel 05.03.11 at 10:35 pm
Tongue in cheek, I would like to suggest that having bad pundits can sometimes be as good as having good ones. One can use the “George Constanza” decision algorithm in the case of binary choices – do the opposite. The real utility of rating pundits is not so you can ferret the bad ones out by not listening to them or censoring them. Their bad opinions can also improve the collective decision of all pundits if you can figure out how to discount them properly. That is, knowing that a pundit is bad can be just as good for making final decisions as them being good.
Alex 05.03.11 at 11:01 pm
Looking over the paper quickly, it looks like there is no consideration of the apparent or actual likelihood of the prediction. I can predict the sun will rise tomorrow with astounding accuracy.
Counterargument: there are plenty of things that were highly predictable and indeed predicted, but that a lot of predictors didn’t predict because they didn’t consider them to be acceptable things to predict. Punditry would be better if it didn’t quite often argue that this time, the sun wouldn’t rise tomorrow, and in any case it hadn’t really yesterday.
John Emerson 05.03.11 at 11:25 pm
“Pundits make predictions not just to predict, but to influence.”
And? I’m still interested in the ones who are more accurate. Set your study period for any time you like.
Gene, you’re not in the target audience. They don’t care what you think.
Generally the conservative hacks predicted poorly or miserably, whereas the liberals predicted rather well. What this tells me is that the hapless liberals are reality-based, whereas the conservatives know how to use the media. The important thing is not what is, but what can be done with what is.
“Philosophers have only interpreted the world in different ways. What is crucial, however, is to change it.”
These were in the middle: Tom Friedman 6, Andrea Mitchell 5.5, David Broder 4.5, Joe Lieberman 4, Sam Donaldson 3.5. All I can say is that Andrea Mitchell is more liberal than I had thought, and that Sam Donaldson is more conservative than I’d thought.
John Emerson 05.03.11 at 11:26 pm
“Pundits make predictions not just to predict, but to influence.â€
And? I’m still interested in the ones who are more accurate. Set your study period for any time you like.
Gene, you’re not in the target audience. They don’t care what you think.
Generally the conservative hacks predicted poorly or miserably, whereas the liberals predicted rather well. What this tells me is that the hapless liberals are reality-based, whereas the conservatives know how to use the media. The important thing is not what is, but what can be done with what is.
“Philosophers have only interpreted the world in different ways. What is crucial, however, is to change it.â€
These were in the middle: Tom Friedman 6, Andrea Mitchell 5.5, David Broder 4.5, Joe Lieberman 4, Sam Donaldson 3.5. All I can say is that Andrea Mitchell is more liberal than I had thought, and that Sam Donaldson is more conservative than I’d thought.
The media are organized for the prupose of deception. That should have been factored in.
Tom T. 05.04.11 at 1:04 am
Political actors are presumably going to tend to predict election victory by their party’s candidate. Including predictions of elections would thus seem to favor partisans of the winning side. It would be interesting to compare this data with data showing which side was more accurate in 2004, for instance.
Note that George Will is listed as having made 17 predictions, of which 19 were correct.
And, really, any survey that finds Hank Paulson to be a successful predictor must necessarily be highly suspect.
John Emerson 05.04.11 at 1:20 am
Damn. I got that one way wrong.
Sev 05.04.11 at 2:17 am
#6 “Pundits make predictions not just to predict, but to influence.”
‘And? I’m still interested in the ones who are more accurate. Set your study period for any time you like.’
Krugman is frequently exasperated by the divergence between these two values- IOW it appears there are some negative correlations between accuracy and influence, particularly where VSPs and their policy preferences are concerned.
Xarici 05.04.11 at 9:01 am
You can’t account for black swanns. We put entirely too much faith in pundit soothsaying.
ajay 05.04.11 at 9:12 am
any survey that finds Hank Paulson to be a successful predictor must necessarily be highly suspect.
Presumably this means that he only ever went on the record in print about things he was very sure about. You could, and I think this is Martin’s point, rack up a very good score by being a very boring pundit. But you wouldn’t get a job because no one would want to print articles that said “CHINA WILL CONTINUE TO GROW NEXT YEAR” and “ARAB DISSATISFACTION WITH ISRAEL SET TO PERSIST IN 2012”. But you’d get your articles into print, boring or not, if you happened to have a very famous name like “Treasury Secretary Hank Paulson”.
dsquared 05.04.11 at 9:33 am
I probably should have mentioned that the outperformance of liberal pundits, like the predominance of liberal professors, is probably at least partly a self-selection effect because someone with rightwing politics who is good at predicting the future can usually think of something more remunerative to do than opinion journalism.
Michael Sullivan 05.04.11 at 10:29 am
Dsquared@19:
Why wouldn’t that self-selection effect apply to liberal potential pundits as well?
chris 05.04.11 at 1:43 pm
@20: Liberals are less likely to highly value their own personal wealth compared to other goals advanced by a career in punditry, such as fame? Some of those more remunerative career paths are unethical by liberal standards but conservatives have no problem with them? Liberals consider accurate journalism a public service and value it beyond its financial rewards on that basis but conservatives scorn the idea of public service (at least when it doesn’t involve violence)?
That’s just off the top of my head, there may be more.
dsquared 05.04.11 at 1:47 pm
More reasonably, that liberals entering the workforce are more likely to have weak social ties to media and academic careers than to financial services.
ajay 05.04.11 at 1:57 pm
22: People in the financial services industry have not shown a terrific aptitude for predicting the future.
I think it may be more that someone who is intelligent and well-informed enough to predict the future accurately is also going to be intelligent and well-informed enough not to be a right-winger. Political beliefs aren’t an immutable part of your nature.
cerebus 05.04.11 at 4:25 pm
Presumably the Hamilton College Government department faculty and students tilt left which explains Broder and Lieberman (4.0, really? is this because he kissed Bush?) being characterised as right-of-centre. It’s like a right-tilting group placing Brooks and Parker around 6.0.
ScentOfViolets 05.04.11 at 6:21 pm
You might consider it a type of influence, but I’d guess that pundits also make predictions to legitimize a course of action. That changes the arrow of causality just a bit. One example would be pundits “predicting” the imminent collapse of SS and the need to act now by defunding it. This is, of course, what many elites already want to do. How convenient for them.
john b 05.05.11 at 4:06 am
Leiberman is characterised as right-of-centre because, erm, he is. Off the top of my head: massively pro-Bush, ran against the Democrat candidate on a platform that was to their right, endorsed John McCain in 2008 – exactly how right wing do you have to be to score ‘right of centre’ in cerebus-world?
cerebus 05.05.11 at 11:19 am
John, just look at voteview.com. His voting record puts him firmly in the Democratic caucus. His domestic views are bog-standard (American) liberal. Supporting the Bush-Obama national security state does not a rightwinger make. He only ran as an independent because the kossacks overreached.
ovaut 05.05.11 at 5:33 pm
It’d be interesting to have a bunch of experts ‘predict’ (in your sense of predicting the past) what the actual story of OBL’s killing is, and see how right they were in a hundred years when we know.
Comments on this entry are closed.