If it is a one-off forecast, of course, then one does not have any particular reason to believe that the whole set of forecasts will be biased above or below the actual forecast. If, however, forecasters have some error-correcting mechanism going on (and a reason to try to improve their errors) then the forecasts shouldn’t be systematically biased one way or another. We’re not talking about random guesses here, the forecasters are trying to get their values right.

Of course, at times the consensus forecast can have errors of the same direction for several periods running, sometimes that’s because people mistakenly identify a trend as a blip or vice-versa, and sometimes for the same reason that you sometimes get four heads in a row if you toss a coin.

And the other problem is – what’s the alternative? Picking the best model is a good idea if the errors are systematically biased, but you don’t always know what the best model is ahead of time.

I’ve constructed a forecasting model for “Other Persons” tax according to this method, and was able to come up with a model that averaged the results of multiple rough-models that got the direction of change right for both in-sample and out-of-sample data. Okay, from memory, I only had 13 data points in total, but the previous models couldn’t get the direction of change right for their in-sample data. At times the averaged forecast gave a higher error than one of the sub-models, but over 13 periods the averaged forecast errors were lower than the forecast errors of the best of the sub-models.

Do you know why they work? Hint: It’s not because of any mystical wisdom-of-crowds.

It’s for a very simple reason:

IT PAYS PEOPLE ANONYMOUSLY FOR TELLING THE TRUTH!

I’m a programmer by trade. Any programmer can tell you a tale where they can give an accurate estimate of project time, and management doesn’t want to hear it. In fact, the accurate estimate will likely be punished. They want to hear an inaccurate estimate, usually called “aggressive”

But provide a setting where the people who are right have a reward, and not a punishment – surprise, the information has a tendency to converge to the correct answer.

It’s a shame one needs market folderol. But I suppose if management can understand it, and the engineers profit, it’s OK. But don’t think it works because of any wisdom-of-crowds. It works in fact because of wisdom-of-experts.

Basically, there’s a complicated question-begging. Sometimes errors average out, and sometimes they don’t, but if you don’t know which is the case in any particular problem, there isn’t much justification for blindly averaging.

Anyway, when you next return to the subject of market prediction, I’d like to hear more about how you choose which hypothesis to prefer “on the balance of probabilities,” and also what will count as a “sufficiently surprising” prediction. It seems good to lay these things out as carefully as possible ahead of time, to minimize selection effects when good test situations arise in the future.

John;

I just want to stress this point.

You knew when you made the prediction that the polls had gotten it wrong. Markets would either get it right (and beat the polls) or wrong (and equal them). But observing markets getting if right doesn’t support the “markets outpredict polls” hypothesis – it just shows that markets and polls give different results. You hace to show that polls don’t beat markets with a greater frequency, your test says nothing about this.

There are four possible outcomes of interest:

(1) Polls Right, Markets Right; draw
(2) Polls Right, Markets Wrong; supports polls
(3) Polls Wrong, Markets Wrong; draw
(4) Polls Wrong, Markets Right; supports markets

You only consider (3) & (4). So long as Polls and Markets aren’t identical, markets will sometimes get it right when polls get it wrong – but that doesn’t show the “superiority of markets”; it just shows polls and markets aren’t identical.

Polls can also get it right when markets get it wrong – (2). The outcomes that support polls are unobservable. Your test is set up to count the hits but not the misses. Hence, my coin example. Tossing a coin will out-guess polls some of the time, but it isn’t superior to them because polls are going to outguess a coin more often. Your test, however, can’t establish this because you’re not looking for (2).

On the general approach, what I state in the post is a willingness to prefer the hypothesis “markets outpredict polls” over its contrary on the balance of probabilities. So I don’t require anything like 0.001 probabilities.

The German outcome was sufficiently surprising that, had the markets predicted it clearly, and well in advance of the polls, I would have shifted my prior beliefs. As it is, they didn’t and I haven’t.

I don’t know much about Bayesian probability, but in most scientific revolutions I’ve looked at, it has been the theory at least as much as the experiment that has convinced the relevant scientists. An experiment that equally supported GR and the much simpler Newtonian physics should be (and usually is) taken to support the simpler, null hypothesis (if only because, time and again, the simpler theory seems to more often to turn out to be correct, all else being equal). Only really beautiful new theories, like SR or GR, seem to be theoretically compelling enough to grab the mantle of the null hypothesis from the previous theory, such that ambiguous evidence could be taken to support both equally. But I don’t know how many other cases there have been where this happened I can think of, and GR’s immediate success was due (even among scientists) in large part to Einstein’s existing reputation. So I think the impact of the Rutherford experiment was quite possibly a mistake, and one that science actually makes relatively infrequently. In any case, the wisdom-of-crowds arguments are not remotely in the same league as GR in terms of theoretical appeal, and thus bear a much higher burden of proof.

Thinking a bit more about Bayes…based on the little statistics work I’ve done, I think the force of the null hypothesis depends a lot on where you happen to be. That is, for Rutherford, he sets out to test GR, and gets a result whose error bars overlap both the GR and Newtonian (N) predictions; chances are, were N correct, he could just as likely have gone on the other side of the N prediction, with no overlap of the GR number. So the fact that he did get a datum that overlapped both is, as a one-time-thing, slightly in favor of GR, all else being equal. But from my point of view, as anyone other than Rutherford at the time, it sure wouldn’t have made the papers on the other side of the N value. With the fame of Einstein, etc, there’s a huge selection effect. Who knows how many experiments produced N-leaning results before Rutherford came along? This may seem like an irrelevantly realistic issue, but I think it is an important part of Bayesian reasoning, which really is talking about the decisions one makes with a single instance. Rutherford might be right, if he had to bet his life one way or the other right then, to bet on GR, but no one else would be right to do that. Certainly I’m going to be equally skeptical of any data that emerges supporting the betting markets, though I suppose John Quiggin is in Rutherford’s position in that he proposed the test in question, and thus might be willing to bet his life (if forced) on the pro-market side if this particular market did an ambiguous bit better than the polls (since it could have done worse).

For me, since there’s a decent chance of the market accidentally landing on the correct prediction, and a high chance of that accident getting publicized, I’ll have to wait for a) lots of evidence, and b) a convincing theoretical defense of the superiority of markets. Perhaps one piece of evidence could do it if, say, the market predicted something with a poll-based probability less than .0001, but since elections are bounded quantities and polls are almost never off by that much, it is hard to imagine that scenario happening.

I agree this is a far better method when you can get it to work. And sometimes you can. I’ve heard that a London trader once made a fortune by having a system of carrier pigeons that reported the result of the battle of Waterloo to him before anyone else in London knew about it. The brother in the movie 50 First Dates provides another example.

However, there still remain a percentage of cases where one must make a prediction about the future without actually knowing what it is. This may be a small percentage, but this is the point where we must resort to inferior methods compared to plugging the right answer in, and it is when averaging forecasts comes into its own.

As for your person with the feet in the freezer and their head in the oven, assuming that the oven and freezer are at standard “on” temperatures, then their average temperature level is probably over 90 degrees celsius, which is much more uncomfortable than the normal range of between 0 to 40 degrees celsius. The evaluation of the utility provided by a world state is a separate issue from the accuracy of the forecast.

I disbelieve.

I am thoroughly unconvinced of the practice of putting someone’s head in an oven and feet in the freezer and saying the average personal climate is more comfortable than a typical day (and of course better than the indivdual parts!)

As a quick example, imagine two forecasters forecast the unemployment rate next year. One forecasts it will be 6.0%, the other that it will be 4.5%. It turns out to be 5.0%. The average forecast of 5.25% has an error of 0.25%, which is lower than the best forecast’s error of 0.5%.

This concept has been tested in real-life situations, where, unlike my constructed example, forecasts can all be statistically biased (I am not necessarily talking about ideological bias, statistical bias could arise from other neutral causes, e.g. a mistake by the statistics department so everyone’s plugging data into their models which was wrong in the first place). If you don’t know which way the forecasts will be biased ahead of time, it makes sense to go ahead and average. See http://www-marketing.wharton.upenn.edu/forecast/abstracts/combining.html for a reference.

This is a way in which markets can be better than the best professional forecaster. Participants do not need private information, they just need a wide range of different biases and a way of combining their forecasts (which can be a simple average or a weighted average such as provided by how much money you can bet on it) to improve the final forecast.

Intrade.com’s betting on the 2008 US Presidential election includes some bets I consider rather odd.

As regards the margin of error, this is the point at which Bayesian and classical inference typically part company. From a Bayesian viewpoint, the question is best posed “in the light of the available evidence, which theory is more probable”, not “can we reject the null hypothesis”.

As a matter of historical fact, Einstein’s theory was widely accepted on the basis of the single data point of starlight deflection, and those who accepted it were right to do so.