Comments on: The Wisdom of Crowds who don’t check facts

By: Kaus Hackula

Kaus Hackula — Thu, 23 Dec 2004 23:07:29 +0000

hey, I heard that if you get a crowd of several thousand people to throw darts at a board, and average the results, you get nearly a perfect bullseye.

By: Justin Wolfers

Justin Wolfers — Sun, 19 Dec 2004 21:51:18 +0000

John Quiggin’s response is interesting. It may adequately describe some of the excess hype in the media, but I don’t think that it describes any of the serious academic research on prediction markets at all… In my own writing, I’ve argued that prediction markets serve three purposes, providing incentives for 1. Truthful revelation; 2. Information gathering; 3. An algorithm for information aggregation No serious economist has ever said that they aggregate all information. Indeed, I’m also quite interested in figuring out which information aggregation mechanisms perform best in a variety of settings. (It won’t always be markets… Witness what happens to the odds on a horse race when Kerry Packer decides he wants to bet.) But John raises a very interesting approach to thinking about the wisdom of markets. The new information raised by the exit polls produced a change in beliefs. Presumably one merely needs to invert Bayes’ Rule to back out just how tight the priors of the markets were prior to the exit polls. One could then compare this with the degree of confidence that other predictive institutions might allow. That very strong (mis-)information (knowing how 10,000 people voted) led to a moderate revision of priors suggests that the market had already aggregate a fair bit of information. But this is clearly a glass half-full view, and I’m more than happy to hear that others view the same glass is half-empty. As long as we are looking at the same glass…

By: John Quiggin

John Quiggin — Sun, 19 Dec 2004 21:13:47 +0000

“Unless one thought that the markets already knew which way all Americans were voting, ” But, a lot of the discussion thus far has implied something pretty much like this. At least, it has implied that the markets had collective access to the kind of information that would produce tight priors, unlikely to be radically revised in the light of exit polls.

By: Justin Wolfers

Justin Wolfers — Sun, 19 Dec 2004 14:13:05 +0000

Let me make two very simple points about the response of markets on election day: 1. Sensible markets respond to information. If the election betting markets didn’t respond to the “news” in the exit polls, then I think we should indict them. (Unless one thought that the markets already knew which way all Americans were voting, then a “random” sample of such information is clearly news.) As such, efficient prediction markets should have responded to this news. And they did. 2. The remaining question is whether they responded appropriately. On this, it is harder to be sure. So how much should the markets have responded to this news? As always, when assessing the “wisdom of crowds” it is important to assess it relative to some alternative predictive institution. Two comparisons seem useful here. First, the S&P 500 moved almost in lock-step with Tradesports throughout election day. But you might object that this is simply comparing the foolishness of crowds with the foolishness of crowds. So the only other predictor I can think of comes from the standard statstical approach (which pollsters and others use). I believe that there were 13,660 people surveyed in the exit polls, although I don’t know how many had been polled in the early leaked numbers. This suggests a “standard error” on the exit poll of +/- 0.5%, or a 95% “confidence interval” around the polling estiate of +/- 1%. That is, given standard statistical methods (and the fact that it was not yet clear that the sample was demographically unbalanced), you would have thought that there was not a hope in hell that Bush could have won. So by this benchmark (which is really the next best predictive institution), the markets did pretty well in giving this news very little weight, as Bush only fell as far as 25-30 on election night. Personally, I market this up as one for the markets. (For those who are interested, Eric Zitzewitz and I present data on the performance of the markets every ten minutes through election night in a forthcoming paper “5 Open questions About Prediction Markets”. In that paper we analyse the performance of the markets in somewhat greater depth, and raise questions about when and where we might expect the markets to fail.)

By: nick

nick — Sat, 18 Dec 2004 21:26:59 +0000

I think we can establish a New Rule of Lazy Journalism: ‘Electronic markets will always be deemed to have predicted results more accurately, regardless of whether or not they actually did.’

By: Sebastian Holsclaw

Sebastian Holsclaw — Fri, 17 Dec 2004 18:52:35 +0000

It is an interesting case of markets reacting to bad information. Is this a well understood area? (It obviously isn’t for me). It is pretty clear that if you had taken the predictions of the toy-markets at one day before the election, they were very accurate. But choosing that time is arbitrary and may be influenced by our current knowledge of the outcome. We should have determined the cut-off point before hand.

By: Winner

Winner — Fri, 17 Dec 2004 17:23:02 +0000

Five p.m. Monday I submitted an entry to an election contest requiring predictions of the presidential winner in each state. Making my predictions, I relied heavily on the odds quoted at Tradesports. In only one instance, Wisconsin, did I, on the strength of my reading of the survey evidence, go against Tradesport’s odds. I predicted 49 of 50 state outcomes correctly. The one I missed was Wisconsin. Similar remarks apply to the Senatorial races. It’s true that when the exit poll news broke, the odds tilted heavily in Kerry’s favor. This is an iffy business, and the Best Evidence at ten a.m. on Tuesday was the exit polls. The crowd responded appropriately (as for a while did in-the-know White House insiders).

By: Giles

Giles — Fri, 17 Dec 2004 17:03:10 +0000

Rob I don’t think the exit polls themselves were manipulated – just that some one put “selected” and put outlier raw figures into circulation and that then everyone just ran with them. Who could do that – maybe just some data compiler at Zogby/Gallop who was meant to be putting it all together for the definitive poll, he selected a few outliers and emailed them out. And once things got going the balancing process broke down. And yes, if it was someone low down, 35K would be enough to take this risk. After all the only liability that they face is getting the sack from zogby/gallop or whoever.

By: Rob

Rob — Fri, 17 Dec 2004 16:26:58 +0000

Uhh, giles—the netowrks based their news reports on the exact same exit polls. The exit polls themselves would have to be manipualted to make what, $35k at most?

By: Bruce Cleaver

Bruce Cleaver — Fri, 17 Dec 2004 16:25:26 +0000

Rob brings up a great point – the markets are quite imperfect, albeit better than other methods. A one-eyed man in the land of the blind. The analogy for sports betting has an important twist to it – I think the job of the oddsmakers is to get 50% of the bettors to vote on Team A, and 50% to vote on Team B, not necessarily to predict which team will actually win and by how much. They are predicting what the market will predict, so to speak, in order to minimize their risk and maximize their vigorish.

By: Giles

Giles — Fri, 17 Dec 2004 15:32:02 +0000

Just to through a bit of conspiracy in – doesnt anyone else think the market was ramped. Both the polls and markets correctly predicted a Bush victory 2 days before hand. Then the market tailed down – perhaps because some one was doing some tactical selling (having gained the infomration on how the market would respond the month before when there was that sudden downward spike due to a big sell). Then on the day of the election “selected raw figures” that are favorable to a Kerry victory are leaked to the dimmer end of the Democratic blogosphere -e.g. wonkette. A panic ensues and Blogfeld buys Bush at 20 and makes a huge profit the next day. And since these markets seem to be relatively unregulated, Blofeld has little fear of being pursued by the SEC. Further on both sides of the spectrum, there’s little willingness to want to look into the matter. The right doesnt want to admit its markets aren’t purfect and the left doesnt want to admit that they were dupped. Anyway I think we should be told. Ba!

By: Rob

Rob — Fri, 17 Dec 2004 14:12:59 +0000

Ugh…I am feeling that people are really going to misunderstand what the “wisdom of crowds” provides. It can give us the best prediction of future events under given circumstances. But that doesn’t mean it is even a good prediction. Just compare the betting lines on NFL games to the final scores. Here we have a shorterm prediction where the outcome distribution is finite and the risks are fully understood and how many games actually end up pushes in a given week?

By: Angry Moderate

Angry Moderate — Fri, 17 Dec 2004 13:44:34 +0000

The other fact that is strangely neglected is that the betting markets rely highly on the polls, while the polls don’t (yet!) rely on the betting markets. So the betting markets are the wisdom of the crowd absorbing and slightly correcting on the margin the polling information. I’d like to run a (near impossible) experiment where a group of people were denied all polling information but granted all other information and see how they did in an elections market.

By: abb1

abb1 — Fri, 17 Dec 2004 11:36:58 +0000

I lost 200 bucks on the IEM, dammit. I still can’t believe 59 million Americans can be so stupid. It was a scam, I demand a recount.

By: Nick

Nick — Fri, 17 Dec 2004 11:16:23 +0000

Betfair’s markets were doing about the same during the night – from what I recall Kerry came in to about 1.25 at one point, while Bush was out past 5.