Always nice to be able to test an idea in a live application … It’s worthwhile remembering in any discussion of “terrorism futures” that nobody was ever really proposing to offer contracts on any specific terrorist events; the proposed “Policy Analysis Market” (which claims on its website that it’s going to launch in March; sadly there is no currently existing futures market which allows me to bet that it won’t), was always going to be about betting on general indices of global political stability. For example, one might think it would be useful to have a futures market which gave finer-grained information about the risks to the US than the Department of Homeland Security’s Threat Level Indicator; not just whether today’s threat was “yellow” or “orange”, but whether the risk was growing or falling.
One might think that, but one would be wrong. In actual fact, it is possible to trade futures on the US Homeland Security indicator at Tradesports.com. So, since the threat level was raised from “Yellow” to “Orange” over the weekend we can go to the tape and see whether the traders there had any advance steer on this movement. Did they?
Did they, to put not too fine a point on it, my arse. Here’s the contracts page; a little playing around with Tradesports’ Javascript navigation reveals that, overnight, the “Yellow” contract gapped down from 86 to 12 while the “Orange” contract gapped up from 11 to 88. In the days immediately before the raise, Orange had been gradually declinign and Yellow had been gradually appreciating. I can only conclude that traders on the Tradesports contract had absolutely no additional information relevant to forecasting the Homeland Security threat level. I’d also point out that if any CT readers fancy having a punt that the threat will stay Orange for the remaining nine months of the year, they certainly should not buy Orange at 88; selling Red at 5 and Yellow at 10 gives you a better profit if Orange persists, plus a free bet on Blue and Green. It doesn’t say wonders about the efficiency or liquidty of the market that it has glaring anomalies in it like that (it’s also possible to bet on the Lord of the Rings film taking more than $116m over the weekend, which it did, and you can also have a look at a US Economic numbers book in which the probability of the Personal Incomes number being above 0.4% is quoted at 55-65, while the probability of it being above 0.7% is quoted 61-70. (Update: Or at least it was before I took out some sucker’s bid; if you want to fact-check my ass, email me and I’ll send you the screenshot).
The problem here is that the Tradesports financial and political books aren’t really all that deep or liquid, compared to their sports books. There just aren’t enough sharks like me preying on silly or late bids and offers to keep the market respecting basic pricing identities, let alone the millions of savvy traders “aggregating information” that the market-philes dream of. If you’re interested in the Democratic Nomination race, for example, it’s possible to buy Joseph Lieberman at 1.3% on the Iowa Electronic Market and then sell him short at 2.3% on Tradesports; you can also buy Gephardt in Iowa for 7.6% and sell him on Tradesports for 9%, among other intermarket arbitrages, but I for one could not be fagged registering for both sites in order to do so. I’ve argued in the past that an adaptation of Hayek’s critique of planning suggests that these playing-at-shops markets don’t work, but I now think that in practice, the real problem is simply going to be lack of interest. Every year, futures contracts are launched on actual things where there are genuine buyers and sellers on both sides, but fail to take off through lack of open interest. It’s much less likely that a purely speculative contract is going to attract enough liquidity for price discovery to be meaningful, and the Homeland Security performance suggests that the traders don’t have any useful information to aggregate.
Three endnotes: First, all price-related discussion above was clearly based on prices on my screen as I looked at them; it is likely that they will have changed by the time you read this. Second, Crooked Timber is not in the business of provding investment advice for betting on Tradesports, and any spread beeting CT readers carry out should only be done after full investigation of the risks involved. And third, I at least will no longer be entering into discussion on the subject of whether or not trading exchanges might “add something valuable to the process”, because the empirical evidence is that they don’t. Anyone wanting to convince me otherwise is going to need to cough up actual data; in particular, I reserve the right to gently mock anyone who claims that some hypothetical i>other idea futures market would be materially more liquid than Tradesports without giving a good reason why (endorsement by the government does not constitute a good reason). If I get time over the holidays, I will muse on why it was that so many people believed so passionately in this dead duck of an idea.
Finally, I am positive that Tradesports used to offer a “WMD found in Iraq” contract but I can’t find it for the life of me. I’ve sent them an email about it and will report back.
{ 23 comments }
phil 12.22.03 at 6:23 pm
Can’t find the find-Iraq’s-WMDs contract? Oh the irony.
Decnavda 12.22.03 at 7:20 pm
“a little playing around with Tradesports’ Javascript navigation reveals that, overnight, the “Yellow†contract gapped down from 86 to 12 while the “Orange†contract gapped up from 11 to 88.”
1. This is ONE annecdote, not two.
2. If the system is performing correctly, you should be able to find a counterexample in one of every 10 contracts listed at 10%.
No one here at CT has provided “empirical evidence” that these markets do not work, they have only provided single examples of times that events did not follow the probabilities. Of course there are such things, that is why they are probabilities, and not certainties. “Empirical evidence” in this context is a statistical study to determine, in the long run, whether the accual occurances track the probabilities. I have seen no such study indicating that they do not. Here is one study indicating that they do:
http://artificialmarkets.com/am/pennock-neci-tr-2000-168/
I am trying to track down another, more extensive, study I read about a year ago. The one I gave the link to here does not, for example, discuss the longshot bias which the Joseph Lieberman spread above is an example.
“If I get time over the holidays, I will muse on why it was that so many people believed so passionately in this dead duck of an idea.”
For me, it is because it is supported by empirical evidence. I have been trying to figgure out why so many people are so passionately opposed as to A) Not want to test the idea, and B) Not believe the empirical evidence.
Finally, I will admit that the empirical evidence I have seen and linked to supports the idea in the form of games. I have not seen a statisical evaluation of real-money markets, but I would like to. Most supporters claim that the theory predicts real money markets would be more accurate. I think there might be reasons that a play money game might be better than a real money market, but unlike aparently most people, I will reserve judgement untill I see actual evidence.
Decnavda 12.22.03 at 7:30 pm
As to my reasons a game might be better:
1. An inability to lose real money on such risky propositions might lead to greater liquidity.
2. Since everyone begins with the same amount, the only wealthy investors who can skew the probabilities with big spending are those who have demonstrated an ability to invest wisely.
IEM replicates some of this by limiting investors to $500. If I am right, IEM should be more predictive than TradSports.
But again, I would like to see actual evidence.
dsquared 12.22.03 at 9:44 pm
Dec: The Lieberman effect isn’t an example of a long-shot effect; it’s a pure an simple arbitrade, and the fact that it exists means that the markets for this sort of thing aren’t efficient. They might still in principle be useful even if they aren’t efficient markets, but they aren’t.
And it’s not a matter of the markets “failing to predict” something. It’s the price pattern; the way that they reacted to the news proves that they didn’t have any information at all; the move was a complete surprise to them.
Not meaning to be rude here, but your article is a piece of marketing material. Note as well that “they track the actual probabilities” is an incredibly weak standard”; all manner of things track the probabilities of all sorts of thing. Thomas Friedman’s opinions, for God’s sake “track probabilities in the real world”. The question is whether toy markets track these things any better than the alternatives, and even the paper you linked admits they don’t (the authors also use correlations to report a regression sampling scheme but that’s by the by) The game’s up on these toy markets. Give up on them.
Shai 12.22.03 at 10:39 pm
I suppose one could provide the same criticism of the hollywood stock exchange that Quiggin made about political races and polls. If both are contingent on significant public information, a generalization to this kind of market in general would be unwarranted where it doesn’t hold. (an aside: the difference between feb 18 and 19 oscar picks in that paper dec linked to looks pretty fishy to me, I wonder what’s going on there)
Thinking about the HSX specifically, I’d be interested to know how many movies come out of left field contrary to the relevant public indicators, and whether or not the HSX outperforms in any significant sense.
John S 12.22.03 at 11:42 pm
Either I’m misunderstanding the point of these markets or you are Dsquared (and I’m happy to admit it could well be me). I don’t see why such markets should outperform a poll. Such markets are supposed to ensure that superior information is revealed. If someone or someones have better knowledge than others about the future event then they can make money and change the odds by betting. But what superior information are individuals going to have about voting intentions across the US, unless they are unscrupulous and work for a polling agency and can use their knowledge before publication?
The example about the Homeland Security Indicator is different. What is Tradesports supposed to be doing? Predicting official policy or identifying the best official policy given available information? Tradesports clearly failed to predict official policy. But that presumes or assumes official policy is rational. Perhaps it’s official policy and not Tradesports that’s really wrong. Perhaps, given current information, there should have been no change to the Security Indicator. The reason I find these markets appealing is not because they can replicate what would have happened in the old ways, but that they assimilate more information and may therefore come up with a different conclusion.
For example, taking your position, if Tradesports had given strong odds prior to September 11 of a major terrorist event, while official intelligence sources were saying nothing, you would be pointing out with a huge grin how wrong Tradesports had obviously got it again.
James Surowiecki 12.23.03 at 12:34 am
What, exactly, is the critique here? That these markets will never be never deep or liquid enough to be useful, or that even if they were deep and liquid, they wouldn’t be useful because traders don’t have “any useful information.”
As far as the first point goes, that seems possible, but it seems hasty to draw any conclusions as long as participating in these markets is illegal in the United States (where presumably more good information about, say, the Homeland Security alert might have been located).
As far as the second point goes, I realize you’re not longer discussing the matter, but Wolfers’ and Leigh’s paper (and actually the link wasn’t even to the paper, but to a short summary of it) is not definitive “empirical evidence” that these exchanges don’t work. Set aside the fact that the Centrebet was not an “exchange.” Where is the evidence that the market had no aggregative properties? It picked 43 winners out of 47 regional races, in only three of which polls were available (and in one of those three the poll was wrong). And the market also called the over-under correctly in 12 out of 12 races. That performance may very well be statistically insignificant. But terming it evidence that these “toy” market don’t works seems hasty at best.
The evidence that sports betting markets are relatively efficient, and that parimutuel markets in particular do an excellent job of predicting race outcomes and victory probabilities is overwhelming. It’s perfectly possible that there is something about these markets — other than a lack of liquidity, which I think is a huge problem — that will guarantee their uselessness in contrast to other exchanges, but I’d like a coherent explanation of what that something is. It can’t be the absence of arbitrageurs, because there is no arbitrage in parimutuel markets, nor the absence of hedgers, because there are no hedgers among horseplayers.
dsquared 12.23.03 at 7:11 am
James: yes there are hedgers among horse bettors; they’re called “owners” and “stable hands”. I actually now wonder if you’ve ever been to a racetrack if you don’t know this one.
There are also arbitrageurs in pari-mutuel betting markets; I always keep an eye out myself for arbitrage opportunities when I happen to be passing the Tote booth and once or twice a year I see one.
And finally I reiterate; the question is not whether exchanges have “aggregative properties”. It would be fairly miraculous if they didn’t. It just seems to me that these information aggregative properties are no better than those of Thomas Friedman, which is a pretty low standard.
And finally, I don’t accept the argument that:
Perhaps it’s official policy and not Tradesports that’s really wrong
Tradesports tracked the official policy exactly; either it’s wrong now or it was wrong then. It appears to me that Tradesports has never outperformed the “naive forecast” of assuming that tomorrow will be the same as today.
robin green 12.23.03 at 11:17 am
Even supposing a terrorism futures market could work, which I don’t think it could, it would be a great moral hazard. There are already enough incentives in the world for certain parties (e.g. arms dealers) to act in ways which increase global tensions – we don’t need any more markets arising which could give the unscrupulous a _financial_ incentive to support terrorists (which would be easier to make a short-term profit on than promoting peace, I would have thought). This was, of course, where the moral outrage came from at the time of the DARPA leak/announcement.
James Surowiecki 12.23.03 at 1:52 pm
Daniel, the horse owners I know aren’t out there betting against their own horses in order to hedge their possible losses. And I don’t know any stable hands, but I can’t believe too many have enough money to affect the odds. And though I can’t tell whether this point is implicit in your argument or not, I also don’t think stable hands or horse owners are going to be systematically better at picking winners (or losers, for that matter) than normal bettors, either.
More importantly, I don’t understand how you can say there are arbitrageurs in pari-mutuel markets. In a pari-mutuel market, every bet on a horse, no matter when it’s placed, pays off at exactly the same odds. You can never get a better price on an investment than anyone else who’s already invested in it (or who will invest in it before the race is run) gets. Where is the possibility for arbitrage?
I agree with you that Tradesports totally failed on the Homeland Security. If the market’s offering a contract that tracks the alert, then the market should track the alert. But I really would like to hear whether you think the problem with “toy markets” is that they’ll never be big or deep enough to be accurate, or whether it is the project itself — that is, the attempt to set up markets to predict a wide variety of events/economic conditions/etc. that’s flawed.
You’re right that the question is whether a market is better at aggregating information than other alternative, including mechanical rules, etc. I’m actually not wedded to the idea of a market per se, since I don’t believe in the shark theory of why markets work, and think that any method of aggregation that asks large numbers of motivated people to make probabilistic judgments will do a good job of forecasting. Having said all that, there is no chance that Thomas Friedman could pick 43 winners out of 47 electoral races.
dsquared 12.23.03 at 3:15 pm
Daniel, the horse owners I know aren’t out there betting against their own horses in order to hedge their possible losses
I’m using “hedgers” in the sense of my own earlier article; people who take part in the markets as direct actions which form a part of their life (“tacit knowledge”) rather than people who bet based on opinions. This is the Hayekian argument which I came in with here; that Hayek’s concept of “tacit knowledge” is both valid, and more circumscribed than a lot of the market advocates think.
You’re off your cake, by the way, if you think that stable lads don’t have systematic information advantages; quite apart from anything, they will often be aware that a well-fancied horse is running lame. Getting decent tips on the stable’s other horses is one of the perks of being a racehorse owner, as far as I can tell from the lads I know who own a leg each of some donkey.
The arbitrage in pari-mutuel markets is (I’m assuming that they’re reasonably similar to the Tote on British racecourses) that the Tote offers separate pools on trifectas, exactas and such, and these odds aren’t always consistent with the win odds of the constituents. It’s usually a cheap/dear analysis rather than a strict arbitrage on the occasions I spot it, but these guys reckon that you can often construct genuine Dutch books.
http://www.maths.usyd.edu.au:8000/res/Nonlinear/EdeObr/2001-10.ps
In general, I think I have three objections to markets:
1. (To markets specifically) I simply don’t believe, empirically, that there is anything like enough of a pool of money out there interested in betting on policy questions to make the market work as a market.
2. (Generally on aggregative estimations) I think that there’s a party-trick element to all the situations in which these methods work; that actually diminishing returns have massively set in by the time you’ve asked the fourth or fifth person. I value tacit knowledge over opinion; that’s (paradoxically) why I like experts if they’re experts of the right kind; I think that the only thing that qualifies one to make sensible decisions about something is direct involvement with it in one’s own life.
3. (More generally) I don’t like attempts to have a “quick way” with human communication. I’m intrinsically suspicious of rinky-dink attempts to get simple answers to complicated questions, and extremely hostile to attempts to get reductive, numerical answers to complicated questions where the answers depend on probabilities of dubious existence. Even if I were to be convinced that policy markets were working as markets, I would still have very grave reservations about how one might interpret the prices.
Finally, on the wolfers work, you can get the full text of the paper here:
http://faculty-gsb.stanford.edu/wolfers/Papers/AJPS%20Three%20Tools%20Article.pdf
It seems that Centrebet was *not* acting as a book-maker on the Aussie election; it was taking a position with one side much more heavily backed than the other. Score one for collective versus centralised decision making, but given this, we can’t say that the 43 out of 47 marginals statistic reflects collective information rather than good oddsmaking either. In 43 of 47 cases, the favourite won the election, but that doesn’t necessarily mean that the favourite was the more heavily backed.
In general, I think that the Wolfers et al paper is highly flawed because they don’t appear to have observed that there is a difference between a bookie and a market, and that in particular, Centrebet does not appear to have been acting as a market in this case.
Jason 12.23.03 at 4:41 pm
1. Perhaps a valid opinion, although peoples propensity to gamble is astounding
2. Doesn’t this contradict 1? As for the expert comment. How do you test if an expert actually believes what they’re saying – get them to put their money where their mouth is. How do you judge a good expert from a bad one – one who puts their money in the right pile more often (ie., has more money and so can invest more in the market). I tend to disagree with decnavda’s opinion that games are better, because you want the people betting to have something real to gain/lose by guessing wrong, and also you want the better guessers to be a bigger part of the process (by giving them more money).
3. Setting up a market isn’t rinky-dink. People still have to do the analysis, and the ones that do good analysis will end up better off.
There isn’t much (any) difference between a bookie and a market, at least if you think NASDAQ is a market (since it buys/sells the shares itself, as opposed to the NYSE which acts as an intermediary).
dsquared, if you really believe that “the same as the day before” strategy is so much better than the market one, go bet your house on it. You may be right, and good luck to you.
James Surowiecki 12.23.03 at 5:37 pm
Daniel, you’re going to hate the book I’m writing. But these seem like reasonable objections (although I disagree with most of them).
1) may be true, but again I’d like to see what happens over time, and I do think that the fact that these markets are illegal in the U.S. makes a difference.
2) I don’t believe in the “direct involvement” theory, mostly because no one on his own has direct involvement with enough of the factors that influence the course of, say, a company’s future earnings to make me believe he should deserves a privileged position as “expert.” And I’m unconvinced there’s an easy way to identify experts or the informed beforehand. Diminishing returns may be a reality, I don’t understand why this is an argument against markets. Maybe there’s little be gained by asking more than four people, but there’s nothing to lose, either. (The forecast won’t get worse by asking more people, since the errors of the ignorant will be random rather than systematic.) And maybe the 10th or the 100th person you ask will have, not the answer, but a key piece of information that’s missing. So it seems to me to make sense to cast the net as widely, rather than as narrowly, as possible.
3) I’m all in favor of simple answers if the answers are right. The complexity of a method should not be a recommendation in favor of it. But the question of what numerical answers and probabilities mean in many of these cases is complicated, and does lend itself to facile conclusions of the sort I myself am often likely to reach. In the case of something like the would-be markets that PAM was going to set up in economic conditions in the Middle East, or internal corporate forecasting, though, the meaningfulness and the benefits of such predictions seems undeniable.
Finally, Wolfers probably should have been more explicit about the difference between Centrebet and a pure market. But in many cases, there is no theoretical difference. A point spread, for instance, is the weighted average of the estimates of all bettors in the market, assuming the bookie is acting rationally and ensuring that the same amount of money is wagered on both sides of the bet. The same as the price of an Arrow-Debreu contract or a stock price. Odds markets can work differently, but assuming the bookie is smart enough to avoid creating arbitrage opportunities for bettors, I think the differences are minimal.
I think you misread the stuff about Centrebet, though. My impression is not that it was deliberately taking a position, but that it didn’t move the odds fast enough to offset the money coming in on Howard, so that it got stuck with a losing position. More important, if Wolfers writes “the betting favourite” won 43 of 47 races, that must mean the candidate that was most heavily backed by bettors won the 43 races. What else could “betting favorite” in a two-way race mean?
pierre menard 12.23.03 at 7:41 pm
Why tradesports is not efficient but other futures markets might be:
tradesports is pretty small.
The Lieberman contract, for example, has a daily volume of $400,000. This is very small compared with the money traded on, say, NYSE on a given stock in a given day.
Matt Brubeck 12.23.03 at 7:56 pm
If the market’s predictions are truly worse, on average, than Thomas Friedman’s, then you can make a profit using Friedman’s predictions to guide your trades. Whenever there is some source of information that is consistently better than the market, then there is a profit-making opportunity. If your sources really are better predictors, why not make a profit from your superior knowledge?
I don’t expect the market to be better than the best information sources. I do expect it to distill information from various sources into a quantitative measure, weighted by traders’ actual confidence in each source.
(Of course there are better predictors out there, but there are worse ones too. The hard part is telling which is which.)
By the way, Pennock and Lawrence have some more papers on empirical study of prediction markets.
Anthony 12.24.03 at 12:23 am
Why the Tradesports “market” might have failed to predict the Orange Alert:
The people with actual useful information regarding the situation are (probably) not allowed to participate. Who in the US might have a better idea than the “general public” of changes in the real risk of terrorism? Government employees in the intelligence community. Is someone in a position like that going to be allowed to participate in Tradesports, especially on a pool for something which he has a possibility of making money off “insider information”?
There are financial actors who “speculate” with substantial sums of money on the occurence of future terrorism: Insurance companies. The insurers of the WTC are out some $3 to $6 billion, because they bet wrongly that nothing would happen to the WTC.
Unfortunately, that market does not have some of the crucial characteristics that an exchange like Tradesports might: there isn’t enough liquidity – insurance companies can’t just cancel or reprice a policy whenever they feel like it, and there isn’t a secondary market for insurance policies. There also aren’t a lot of players in it – there are only so many insurance companies. Finally, there isn’t really an aggregate measure of terrorism risk – the insurance companies who didn’t hold the WTC’s policies weren’t much affected by the 9/11 attacks.
dsquared 12.24.03 at 8:40 am
OK, replies to questions … I think I will end up needing to do a “Director’s Cut” post on D2D encapsulating all of these:
Jason: I’ve never had any problems telling a good expert from a bad one and it amazes me that so many other people claim that they have. And in any case, the ultimate decisions are going to be made by people (unless you really believe that we would commit blood and treasure to a war of aggression because of a price and volume pattern), so markets don’t actually solve this problem.
Your 1) and 3), and I’m trying to say this without being rude, are exactly the sort of question I’m no longer discussing because I regard them as having been falsified by the evidence. And finally note that the naive forecast is, empirically, the one which is prevailing on Tradesports, but that doesn’t mean you can make money out of this fact.
James: No, I’m sure I’ll like it because I’ve always liked your stuff. I’ll probably even give it a decent review if anyone asks me to. I’m not going to agree with it, though …
2) is just another way of saying that the stock market doesn’t know all that much about the companies it trades, and nor does it, and nor could it, and nor should it. The stock market knows a hell of a lot about general conditions of liquidity and risk tolerance, because that’s what it’s there for. It’s one of Hayek’s deepest insights that markets know about that market; that tacit knowledge is local in space and time. Your 2) is simply an explanation of why companies are managed by experts, not markets, and why companies which spend too much time worrying about what the stock market thinks, invariably come to grief.
I don’t understand why you think that the errors of the ignorant will be random rather than systematic and I’d bet dollars (on an exchange of your choosing) that you can’t support it. The errors of the ignorant will, systematically, underestimate the likelihood of long-shots and give excessive weight to recent evidence. Those are two really well-established results of behavioural finance. And that’s without getting into all of the Elias Canetti stuff about crowds.
3) is a genuine difference of opinion (as well as a gnarly piece of Hayek scholarship), and I may end up writing a book of my own on it when I retire. Suffice to say, here’s a gedankenexperiment for you; do you think that the Soviet Union could have set up a load of “forecasting markets” in order to facilitate the planning of a non-market economy?
I read “the betting favourite” as meaning “the horse with the shortest odds”, but I suspect that not much of importance turns on this point.
Pierre: I think the volume on the Liberman contract is $400K over the entire life of the contract rather than per day. $400k would actually be quite respectable daily volume in a small or midcap stock.
Matt: Thanks for the Pennock and Lawrence stuff, although I’ve already said that I don’t like their paper on the Hollywood Stock Exchange at all.
Although it’s true that whenever there is some source of information that’s better than the market, there is a profit-aking opportunity, it doesn’t follow that there is a “risk-free” opportunity. There were a couple of cases on Tradesports where the probabilities implied were mathematically impossible, but there was nothing I could do about it without taking a risk I didn’t want to take.
Anthony: If we believe that government employees in the intelligence community have better information than the general public, isn’t the policy maret strategy dominated by the strategy of simply asking those employees?
I also disagree that the WTC insurers “bet wrongly”. Underwriting isn’t a form of betting.
Dan Hardie 12.24.03 at 10:13 am
James Surowiecki said: ‘And maybe the 10th or the 100th person you ask will have, not the answer, but a key piece of information that’s missing.’
But James: surely that’s where an intelligence futures market, such as you favour, would fail, and where traditional intelligence-gathering and collating would have at least a chance of succeeding. If one guy in a futures market is bidding in a way justified by reality, and everyone else is bidding the other way, the price and volume indicators will ignore the one man with correct information. The possibility that this one guy is correct won’t even show up under a system of intelligence-analysis-by-futures-market. Whereas in a half-decent intelligence bureaucracy, if a hundred sources are saying ‘Pakistan is not moving troops towards the Indian border’, but then one source close to the action calls in with contrary information, that claim has to be further investigated.
And the fact that you have specific information about why the guy made his report (he heard his brother-in-law, who serves in a particular unit, was headed towards India; he was selling drinks to a lot of soldiers heading towards the border, or whatever) means that you have a head-start in what kind of supplementary investigation you have to undertake to verify the claim. The ‘information’ you get from volume and price trends is, however, so opaque that you don’t know how to start verifying it. Okay, people are buying the ‘Hezbollah to bomb Chicago’ stock, but on the basis of what knowledge? And if you don’t know that, how can you ask supplementary questions to see if they were misinformed?
(A bit sad that I am typing this on Christmas Eve, but if the evil boss class will insist on dragging me into work today…)
Jason 12.24.03 at 4:53 pm
I have trouble telling good experts from bad ones all the time, particularly if my measure of expertness is being able to predict the future rather than knowing the past. I keep thinking of sports writers and commentators, or political pundits. It is easy enough to remember a couple of cases where someone was “right on” or “way off,” but unless you’re actually keeping score, it is harder to remember the averages. This whole process is further complicated by the fact that you don’t know the extent of the belief of the expert based on simple predictions.
Having experts betting on their predictions forces them to decide just how strongly they believe in a particular outcome.
What I meant by 3 is that the people who bet are still doing the analysis, or at least as much analysis as they would do if asked for an opinion. The market is a nice way of aggregating these opinions.
The naive forecast being the prevaling one on Tradesports does mean that you can’t make any money on it … unless you have a better model, for example waiting until your good friend who is an undercover agent in AQ calls you up and tells you that tomorrow is the day. If you don’t have better information than the market aggregate then you’re best off not participating.
In a complete market (one with sufficiently many assets) there should be a (risk-free) profit making opportunity whenever the probabilities are mathematically impossible.
P.S. Aggregate is a little misleading, as the market doesn’t really add everybodies information, but average is also misleading. I’m not sure what the precise word should be.
Jason 12.24.03 at 5:00 pm
How isn’t underwriting betting? Insurance companies take some money now and have the risk of paying out later. Sure sounds like a bet to me.
I would also disagree with the fact that the insurance companies bet wrongly, but only because you can’t say an expert is bad because of one outcome. The insurers bet that the chance of the WTC falling down was x%. It fell down, but were they wrong with their estimate? Who knows, but insurance companies typically make substantial profits which means they’re pretty good estimators (and have good spreads because they work in an illiquid market).
James Surowiecki 12.24.03 at 6:48 pm
I’m going to break this into two posts, because it’s way too long, but:
We can’t prove that the errors of the ignorant (or of the sophisticated, for that matter, since they make mistakes, too) will be random, and it’s clear that in some cases, they aren’t. But random distributions are certainly more the rule in the world than the exception, so beginning with the assumption that error is randomly distributed (that is, in the case of a numerical estimate, people are as likely to miss high as to miss low) seems sensible. The empirical and experimental evidence also supports the idea that when judging matters of fact, a group’s average judgment is superior to that of the vast majority of its members, and often uncannily accurate. This is some of what my book’s about, and I won’t bore you with all the details, but here a couple of simple (and simple-minded) examples: the audience’s answer on “Who Wants to Be a Millionaire?” (on the show, the contestants had the option of asking the audience for help one time) was right 91% of the time. Alan Blinder’s and John Morgan’s study of how groups made decisions that were meant to mimic, crudely, the tasks of the Fed found that the groups outperformed their best members and came up with correct answers 85%+ of the time. And, there’s a large number of studies documenting this same phenomenon.
The point isn’t that you could ask the “Millionaire” audience what Microsoft’s stock price should be or what Jordan’s economy will look like next year and get a good answer. The point is that in all these studies errors are, in fact, relatively randomly distributed — otherwise, the groups would guess wrong. It’s not clear to me why we would think that, on the whole, errors in other cases would be systematically skewed.
Look again at betting markets. We know that bettors are biased — they favor certain teams, they favor home teams, etc. — but, as I mentioned in an earlier post (although that may have been on a different thread) the evidence is that in NFL betting, for instance, “in aggregate, market prices are very close to true values,” even though bettors are, as the author of that study writes, “irrational.”
The best evidence comes, as I’ve argued, from parimutuel markets. Consider the study (published in 1974) that Hoerl and Fallin did of all 1825 races run at Aqueduct and Belmont (they’re New York tracks) in 1970. Essentially, what they did was compare the objective frequency with which horses won to the subjective probability of their victories that the market had predicted. They wanted to know two things: were the subjective probabilities good predictors of the frequency and was the market’s ranking of the horses’ chances a good predictor of the order in which horses finished.
Here’s what they found: first, with only a few exceptions, the market’s ranking of the horses predicted exactly the order in which horses finished. This was true no matter how many horses were in the race (the sizes of the fields varied that year from five horses to twelve): the favorite finished first most often, the second-favored horse finished second, and so on, declining in “lock-step fashion.” They also calculated the average finishing position of the horses, and that number declined monotonically with the rank of the horse, without a single mistake.
As for how well the odds predicted the frequency of victory, to me the match looks bizarrely good, although perhaps a statistician would feel differently. In 312 races with seven horses, for instance, the favorite was predicted to win 33% of the time. It won 34% of the time. Second-place horse: 22% predicted, 21% real; third-place: 16%/16%; fourth-place, 12%/12%; fifth-place, 9%/8%; sixth-place, 6%/8%; and seventh-place, 3%/2%. The results were similar, though not quite as dazzling, in the other races.
The odds for win/place/show bets in a parimutuel market (Daniel, you may be right about arbitrage in exotics, I just don’t know) are pure dollar-weighted averages. So if bettors’ errors are systematic, I don’t see how the odds could be so accurate.
The longshot bias does seem to exist, although I don’t think we know whether it’s really a mistake or rather a conscious choice analogous to a lottery bet. The fact that the longshot bias is really most present in later races makes me think it’s the latter. As for the bias toward “recent evidence,” this doesn’t mean that errors would be systematic, because there’s no reason to think that everyone is looking at the same recent evidence or, more importantly, that everyone will interpret recent evidence in the same way. I have no doubt that errors are sometimes systematic — just look at a stock-market bubble — but to me the evidence suggests that, on the whole, they are not.
James Surowiecki 12.24.03 at 7:20 pm
Daniel, I’ve actually considered the Soviet question, and I’d say this: setting up forecasting markets would certainly have helped. Planning would still undoubtedly have failed, but I think that has more to do with incentives, the massive agency problems that a centrally-planned economy entails, and the negative effects of top-down control on local performance than with the inability of markets to aggregate information. Of course, the forecasting markets would have had to be so big and complex as to be unimaginable. And frankly I think the way the real market works is close enough so that I don’t see the point of messing with it. But I’m not convinced of the theoretical impossibility of it. As a concrete example, GM at one point employed as many people as worked in the entire Netherlands. If GM could perform a wide variety of tasks relatively efficiently (though eventually it stopped doing that), it’s not obvious why a country couldn’t. And I have no doubt that GM would have done better if it had used forecasting and prediction markets to formulate strategy and allocate resources.
Finally, with regard to the stock market, I don’t believe the stock market actually does know much about “general conditions of liquidity and risk tolerance,” or rather I think it knows more about individual companies than about those bigger questions. I’m with Paul Samuelson: I think the market is micro-efficient and macro-inefficient. I realize this directly contradicts CAPM and probably all of neo-classical finance theory, but this is an empirical, not a theoretical question. And I think the evidence is overwhelming: money managers do not attempt to find the efficient frontier. They invest in individual stocks, not in the market. And they try to beat the market by picking stocks. Certainly this is what had to be doing before CAPM, etc., was invented, since no one knew what an efficient frontier was.
The reason this matters is that if we’re trying to evaluate the efficiency of the stock market as a prediction machine, what we really need to look at is how well individual companies’ market prices predict their discounted free cash flow — not at the market as a whole. There’s been bizarrely little work on this, but of late there have been a few interesting papers. Cohen, Polk and Vuolteenaho (2002) argued that 75-80% of variation in firms’s book-to-market ratios “can be explained in terms of future variations of profits.” Even more interestingly, Jeeman Jung and Robert Shiller looked at a cross-section of U.S. stocks that have been traded continuously since 1926 and found, in the words of Shiller (no believer in efficient markets), “the price-dividend ratio is a strong forecaster of the present value of future dividend changes. So, dividend-price ratios on individual stocks do serve as forecasts of long-term changes in their dividends, as efficient markets assert.” (All quotes from Shiller’s paper “From Efficient Markets to Behavioral Finance.”) And this is the case despite all the well-known problems with the stock market as an aggregative mechanism, including most obviously the systematic lack of short-selling.
The Shiller study is of a small sample of stocks, and the results may be explained as the product of luck or something else we don’t understand, but they are at least suggestive. And they call into question the idea that people can only really know that which they have direct involvement with. I like Hayek’s tacit knowledge stuff, but when he writes things like, “All man’s mind can effectively comprehend are the facts of the narrow circle of which he is the center,” I think he’s overreaching.
David Lloyd-Jones 12.26.03 at 1:31 am
Daniel,
Surely a large part of conventional sports gambling as it exists at present constitutes a functioning “toy market.”
Sure there are a lot of people simply gambling, almost all of them losing. There are, however, serious numbers of people who are in the business of profit-taking on changing odds, i.e. making a bet and then coviering by making the opposing bet at changed odds before the event occurs.
This is toy marketing in opinion on the morning line — surely no different from what Tradesports hopes to develop in its business.
Comments on this entry are closed.