The reputation of exit polls was perceptibly if unfairly damaged by the US presidential election. But, as a writer in Fortune magazine points out, another predictor was unambiguously accurate. This was the electronic predictions market: the various websites allowing punters to place bets on the electoral outcome.As anyone who was watching the CT Election Night Special will know, this just isn’t true. The election markets, on the big day, were more or less exactly as bad at providing us with predictive information as were the exit polls. I think that we may have been the only place recording the intraday fluctuations on the prediction markets (which were massive), so maybe it’s important to summarise the facts.
The IEM was predicting a Kerry victory on the morning of the election; Tradesports wasn’t in the morning but started to do so during the day. When the exit poll data was published on Slate at 1600ET, both Tradesports and IEM reacted to it imediately; the price of the Bush contracts collapsed (about half an hour later, so did the Tradesports and IEM websites, under the load). As late as 1830ET, the markets had squarely plumped for Kerry, prompting me to write the following comment on our trader-talk thread:
Bush now down to 17+5 mid price on IEM. Surely to God he’s got more of a chance than that? Tradesports has him at 29 bid. For what it’s worth, Newsnight just suggested that Kerry’s lead in the early exit polls was attenuating as more information came in …
The Bush contracts recovered during the evening as it became clearer and clearer that the exit polls were wrong (by 2109ET, I was writing “Bush back up to 40% on Tradesports; anyone know anything we don’t?”, but there is simply no support in the intraday record for the contention that the markets had any information marginal to that contained in the exit polls.
The IEM and Tradesports contracts certainly did get it right by saying for most of the contract life that Bush would win, but so did the polls. Bush was ahead in the polls throughout the race, both nationally and in a preponderance of swing states; predictions of a Kerry victory like Ruy Teixeira’s were entirely based on a belief that don’t knows would break against the incumbent, that high turnout favoured Kerry, or various other ways of rationalising away the evidence in front of one’s eyes (a tendency we warned about). The best election predictions, as far as I can tell, came from Stuart “Dr Pollkatz” Thiel’s state-level poll correlation approach. As far as the question of “Collective Wisdom” is concerned, I stand by my assessment posted at 2:38am UK time (2138 ET), at which point I seem to remember that I was rather drunk:
“Bush keeps nudging up you know; more like 45% now. This election was 50/50 and remains 50/50, and we shouldn’t have got carried away by the “Wisdom” of fucking “Crowds of Halfwits Who Read Slate”. I still think that in a 50/50 jump ball, Bush wins by 20 electoral votes. Dark pessimism etc.”
Are there any examples of these kinds of markets where opinion polls have not been available? I suppose either because a) it’s a long time before the election (perhaps something like the London Mayor?) or b) because they are banned, e.g. in the last week of the Spanish election, or similarly (I think) in France.
Well, they do them on the Oscars and they work reasonably well, but I don’t know whether there are polls on the Oscars or not.
Betfair’s markets were doing about the same during the night - from what I recall Kerry came in to about 1.25 at one point, while Bush was out past 5.
I lost 200 bucks on the IEM, dammit. I still can’t believe 59 million Americans can be so stupid. It was a scam, I demand a recount.
The other fact that is strangely neglected is that the betting markets rely highly on the polls, while the polls don’t (yet!) rely on the betting markets. So the betting markets are the wisdom of the crowd absorbing and slightly correcting on the margin the polling information. I’d like to run a (near impossible) experiment where a group of people were denied all polling information but granted all other information and see how they did in an elections market.
Ugh…I am feeling that people are really going to misunderstand what the “wisdom of crowds” provides. It can give us the best prediction of future events under given circumstances. But that doesn’t mean it is even a good prediction. Just compare the betting lines on NFL games to the final scores. Here we have a shorterm prediction where the outcome distribution is finite and the risks are fully understood and how many games actually end up pushes in a given week?
Just to through a bit of conspiracy in - doesnt anyone else think the market was ramped.
Both the polls and markets correctly predicted a Bush victory 2 days before hand. Then the market tailed down - perhaps because some one was doing some tactical selling (having gained the infomration on how the market would respond the month before when there was that sudden downward spike due to a big sell).
Then on the day of the election “selected raw figures” that are favorable to a Kerry victory are leaked to the dimmer end of the Democratic blogosphere -e.g. wonkette. A panic ensues and Blogfeld buys Bush at 20 and makes a huge profit the next day.
And since these markets seem to be relatively unregulated, Blofeld has little fear of being pursued by the SEC.
Further on both sides of the spectrum, there’s little willingness to want to look into the matter. The right doesnt want to admit its markets aren’t purfect and the left doesnt want to admit that they were dupped.
Anyway I think we should be told. Ba!
Rob brings up a great point - the markets are quite imperfect, albeit better than other methods. A one-eyed man in the land of the blind.
The analogy for sports betting has an important twist to it - I think the job of the oddsmakers is to get 50% of the bettors to vote on Team A, and 50% to vote on Team B, not necessarily to predict which team will actually win and by how much. They are predicting what the market will predict, so to speak, in order to minimize their risk and maximize their vigorish.
Uhh, giles—the netowrks based their news reports on the exact same exit polls. The exit polls themselves would have to be manipualted to make what, $35k at most?
Rob I don’t think the exit polls themselves were manipulated - just that some one put “selected” and put outlier raw figures into circulation and that then everyone just ran with them. Who could do that - maybe just some data compiler at Zogby/Gallop who was meant to be putting it all together for the definitive poll, he selected a few outliers and emailed them out. And once things got going the balancing process broke down. And yes, if it was someone low down, 35K would be enough to take this risk. After all the only liability that they face is getting the sack from zogby/gallop or whoever.
Five p.m. Monday I submitted an entry to an election contest requiring predictions of the presidential winner in each state. Making my predictions, I relied heavily on the odds quoted at Tradesports. In only one instance, Wisconsin, did I, on the strength of my reading of the survey evidence, go against Tradesport’s odds.
I predicted 49 of 50 state outcomes correctly. The one I missed was Wisconsin.
Similar remarks apply to the Senatorial races.
It’s true that when the exit poll news broke, the odds tilted heavily in Kerry’s favor. This is an iffy business, and the Best Evidence at ten a.m. on Tuesday was the exit polls. The crowd responded appropriately (as for a while did in-the-know White House insiders).
It is an interesting case of markets reacting to bad information. Is this a well understood area? (It obviously isn’t for me). It is pretty clear that if you had taken the predictions of the toy-markets at one day before the election, they were very accurate. But choosing that time is arbitrary and may be influenced by our current knowledge of the outcome. We should have determined the cut-off point before hand.
I think we can establish a New Rule of Lazy Journalism:
‘Electronic markets will always be deemed to have predicted results more accurately, regardless of whether or not they actually did.’
Let me make two very simple points about the response of markets on election day:
1. Sensible markets respond to information. If the election betting markets didn’t respond to the “news” in the exit polls, then I think we should indict them. (Unless one thought that the markets already knew which way all Americans were voting, then a “random” sample of such information is clearly news.) As such, efficient prediction markets should have responded to this news. And they did.
2. The remaining question is whether they responded appropriately. On this, it is harder to be sure. So how much should the markets have responded to this news? As always, when assessing the “wisdom of crowds” it is important to assess it relative to some alternative predictive institution. Two comparisons seem useful here. First, the S&P 500 moved almost in lock-step with Tradesports throughout election day. But you might object that this is simply comparing the foolishness of crowds with the foolishness of crowds. So the only other predictor I can think of comes from the standard statstical approach (which pollsters and others use). I believe that there were 13,660 people surveyed in the exit polls, although I don’t know how many had been polled in the early leaked numbers. This suggests a “standard error” on the exit poll of +/- 0.5%, or a 95% “confidence interval” around the polling estiate of +/- 1%. That is, given standard statistical methods (and the fact that it was not yet clear that the sample was demographically unbalanced), you would have thought that there was not a hope in hell that Bush could have won. So by this benchmark (which is really the next best predictive institution), the markets did pretty well in giving this news very little weight, as Bush only fell as far as 25-30 on election night. Personally, I market this up as one for the markets.
(For those who are interested, Eric Zitzewitz and I present data on the performance of the markets every ten minutes through election night in a forthcoming paper “5 Open questions About Prediction Markets”. In that paper we analyse the performance of the markets in somewhat greater depth, and raise questions about when and where we might expect the markets to fail.)
“Unless one thought that the markets already knew which way all Americans were voting, “
But, a lot of the discussion thus far has implied something pretty much like this. At least, it has implied that the markets had collective access to the kind of information that would produce tight priors, unlikely to be radically revised in the light of exit polls.
John Quiggin’s response is interesting. It may adequately describe some of the excess hype in the media, but I don’t think that it describes any of the serious academic research on prediction markets at all…
In my own writing, I’ve argued that prediction markets serve three purposes, providing incentives for
1. Truthful revelation;
2. Information gathering;
3. An algorithm for information aggregation
No serious economist has ever said that they aggregate all information. Indeed, I’m also quite interested in figuring out which information aggregation mechanisms perform best in a variety of settings. (It won’t always be markets… Witness what happens to the odds on a horse race when Kerry Packer decides he wants to bet.)
But John raises a very interesting approach to thinking about the wisdom of markets. The new information raised by the exit polls produced a change in beliefs. Presumably one merely needs to invert Bayes’ Rule to back out just how tight the priors of the markets were prior to the exit polls. One could then compare this with the degree of confidence that other predictive institutions might allow.
That very strong (mis-)information (knowing how 10,000 people voted) led to a moderate revision of priors suggests that the market had already aggregate a fair bit of information. But this is clearly a glass half-full view, and I’m more than happy to hear that others view the same glass is half-empty. As long as we are looking at the same glass…
hey, I heard that if you get a crowd of several thousand people to throw darts at a board, and average the results, you get nearly a perfect bullseye.
À Gauche
Jeremy Alder
Amaravati
Anggarrgoon
Audhumlan Conspiracy
H.E. Baber
Philip Blosser
Paul Broderick
Matt Brown
Diana Buccafurni
Brandon Butler
Keith Burgess-Jackson
Certain Doubts
David Chalmers
Noam Chomsky
The Conservative Philosopher
Desert Landscapes
Denis Dutton
David Efird
Karl Elliott
David Estlund
Experimental Philosophy
Fake Barn County
Kai von Fintel
Russell Arben Fox
Garden of Forking Paths
Roger Gathman
Michael Green
Scott Hagaman
Helen Habermann
David Hildebrand
John Holbo
Christopher Grau
Jonathan Ichikawa
Tom Irish
Michelle Jenkins
Adam Kotsko
Barry Lam
Language Hat
Language Log
Christian Lee
Brian Leiter
Stephen Lenhart
Clayton Littlejohn
Roderick T. Long
Joshua Macy
Mad Grad
Jonathan Martin
Matthew McGrattan
Marc Moffett
Geoffrey Nunberg
Orange Philosophy
Philosophy Carnival
Philosophy, et cetera
Philosophy of Art
Douglas Portmore
Philosophy from the 617 (moribund)
Jeremy Pierce
Punishment Theory
Geoff Pynn
Timothy Quigley (moribund?)
Conor Roddy
Sappho's Breathing
Anders Schoubye
Wolfgang Schwartz
Scribo
Michael Sevel
Tom Stoneham (moribund)
Adam Swenson
Peter Suber
Eddie Thomas
Joe Ulatowski
Bruce Umbaugh
What is the name ...
Matt Weiner
Will Wilkinson
Jessica Wilson
Young Hegelian
Richard Zach
Psychology
Donyell Coleman
Deborah Frisch
Milt Rosenberg
Tom Stafford
Law
Ann Althouse
Stephen Bainbridge
Jack Balkin
Douglass A. Berman
Francesca Bignami
BlunkettWatch
Jack Bogdanski
Paul L. Caron
Conglomerate
Jeff Cooper
Disability Law
Displacement of Concepts
Wayne Eastman
Eric Fink
Victor Fleischer (on hiatus)
Peter Friedman
Michael Froomkin
Bernard Hibbitts
Walter Hutchens
InstaPundit
Andis Kaulins
Lawmeme
Edward Lee
Karl-Friedrich Lenz
Larry Lessig
Mirror of Justice
Eric Muller
Nathan Oman
Opinio Juris
John Palfrey
Ken Parish
Punishment Theory
Larry Ribstein
The Right Coast
D. Gordon Smith
Lawrence Solum
Peter Tillers
Transatlantic Assembly
Lawrence Velvel
David Wagner
Kim Weatherall
Yale Constitution Society
Tun Yin
History
Blogenspiel
Timothy Burke
Rebunk
Naomi Chana
Chapati Mystery
Cliopatria
Juan Cole
Cranky Professor
Greg Daly
James Davila
Sherman Dorn
Michael Drout
Frog in a Well
Frogs and Ravens
Early Modern Notes
Evan Garcia
George Mason History bloggers
Ghost in the Machine
Rebecca Goetz
Invisible Adjunct (inactive)
Jason Kuznicki
Konrad Mitchell Lawson
Danny Loss
Liberty and Power
Danny Loss
Ether MacAllum Stewart
Pam Mack
Heather Mathews
James Meadway
Medieval Studies
H.D. Miller
Caleb McDaniel
Marc Mulholland
Received Ideas
Renaissance Weblog
Nathaniel Robinson
Jacob Remes (moribund?)
Christopher Sheil
Red Ted
Time Travelling Is Easy
Brian Ulrich
Shana Worthen
Computers/media/communication
Lauren Andreacchi (moribund)
Eric Behrens
Joseph Bosco
Danah Boyd
David Brake
Collin Brooke
Maximilian Dornseif (moribund)
Jeff Erickson
Ed Felten
Lance Fortnow
Louise Ferguson
Anne Galloway
Jason Gallo
Josh Greenberg
Alex Halavais
Sariel Har-Peled
Tracy Kennedy
Tim Lambert
Liz Lawley
Michael O'Foghlu
Jose Luis Orihuela (moribund)
Alex Pang
Sebastian Paquet
Fernando Pereira
Pink Bunny of Battle
Ranting Professors
Jay Rosen
Ken Rufo
Douglas Rushkoff
Vika Safrin
Rob Schaap (Blogorrhoea)
Frank Schaap
Robert A. Stewart
Suresh Venkatasubramanian
Ray Trygstad
Jill Walker
Phil Windley
Siva Vaidahyanathan
Anthropology
Kerim Friedman
Alex Golub
Martijn de Koning
Nicholas Packwood
Geography
Stentor Danielson
Benjamin Heumann
Scott Whitlock
Education
Edward Bilodeau
Jenny D.
Richard Kahn
Progressive Teachers
Kelvin Thompson (defunct?)
Mark Byron
Business administration
Michael Watkins (moribund)
Literature, language, culture
Mike Arnzen
Brandon Barr
Michael Berube
The Blogora
Colin Brayton
John Bruce
Miriam Burstein
Chris Cagle
Jean Chu
Hans Coppens
Tyler Curtain
Cultural Revolution
Terry Dean
Joseph Duemer
Flaschenpost
Kathleen Fitzpatrick
Jonathan Goodwin
Rachael Groner
Alison Hale
Household Opera
Dennis Jerz
Jason Jones
Miriam Jones
Matthew Kirschenbaum
Steven Krause
Lilliputian Lilith
Catherine Liu
John Lovas
Gerald Lucas
Making Contact
Barry Mauer
Erin O'Connor
Print Culture
Clancy Ratcliff
Matthias Rip
A.G. Rud
Amardeep Singh
Steve Shaviro
Thanks ... Zombie
Vera Tobin
Chuck Tryon
University Diaries
Classics
Michael Hendry
David Meadows
Religion
AKM Adam
Ryan Overbey
Telford Work (moribund)
Library Science
Norma Bruce
Music
Kyle Gann
ionarts
Tim Rutherford-Johnson
Greg Sandow
Scott Spiegelberg
Biology/Medicine
Pradeep Atluri
Bloviator
Anthony Cox
Susan Ferrari (moribund)
Amy Greenwood
La Di Da
John M. Lynch
Charles Murtaugh (moribund)
Paul Z. Myers
Respectful of Otters
Josh Rosenau
Universal Acid
Amity Wilczek (moribund)
Theodore Wong (moribund)
Physics/Applied Physics
Trish Amuntrud
Sean Carroll
Jacques Distler
Stephen Hsu
Irascible Professor
Andrew Jaffe
Michael Nielsen
Chad Orzel
String Coffee Table
Math/Statistics
Dead Parrots
Andrew Gelman
Christopher Genovese
Moment, Linger on
Jason Rosenhouse
Vlorbik
Peter Woit
Complex Systems
Petter Holme
Luis Rocha
Cosma Shalizi
Bill Tozier
Chemistry
"Keneth Miles"
Engineering
Zack Amjal
Chris Hall
University Administration
Frank Admissions (moribund?)
Architecture/Urban development
City Comforts (urban planning)
Unfolio
Panchromatica
Earth Sciences
Our Take
Who Knows?
Bitch Ph.D.
Just Tenured
Playing School
Professor Goose
This Academic Life
Other sources of information
Arts and Letters Daily
Boston Review
Imprints
Political Theory Daily Review
Science and Technology Daily Review