Merkel and the Markets

I think the polls were predicting a CDU win in the sense that they were predicting that the CSU wouldn’t be able to form a government, but I don’t think they were predicting any absolute majorities.

This shows the poll evolution. From just eyeballing it, a case can be made the polls weren’t wrong, and there was a very strong late surge

http://www.economist.com/displaystory.cfm?story_id=4401678

They went bust on the CDU – at the end of trading they still had them over 40% (the polls had begun to reflect the drop).

I read somewhere the claim that Wahlstreet’s prediction was slightly better, but havent verified so far.

Here is the best overview about the other polls:

http://www.wahlrecht.de/umfragen/index.htm

abb1;

I think the (theoretical) justification of markets is that they’re all based upon private information as well as public polls. This includes private polling for political parties; knowledge that something damaging about one of the candidate is going to come out; that snow on the day will encourage one lot of supporters to stay inside; right down to your voting intention.

http://www.zeit.de/online/2005/38/wahlstreet_ende

wahlprognosen

Parkett schlägt Umfragen

Have not tried to verify their claim. Here is an overview of the polls: http://www.wahlrecht.de/umfragen/index.htm

My impression is that this not about any private information, but rather about mystical ‘wisdom of the crowds’, i.e. composite gut feeling of a large number of individuals.

“Einstein’s theory of relativity was widely accepted on the basis of a single data point (perihelion shift of Mercury).

All you need is that a theory’s prediction should be both correct and highly improbable if the theory is false. Then apply Bayes’ theorem.”

This is complete nonsense.

There are many possible data points to look at concerning electronic markets – lots of events that they “predict” and which have poll data. Yet, among all those points, you choose one and say that will have “established” the superiority of markets over polls.

Incidentally Einstein’s theories weren’t confirmed by one data point, and to say so shows a gross ignorance of the history of science.

According to the article linked in nr. 14, the Wahlstreet beat all professional pollster’s predictions, but not by a dramatic margin. The Wahlstreet prediction, moreover, got it wrong in just the same way as polling institutes. It didn’t foresee the dramatic drop of the conservative party (CDU) in the actual result and it also didn’t predict that the three parties of the left would together gain a share of more than 50% of the vote. According to Wahlstreet, the CDU/FDP combination was at 49% of the popular vote and the combination of the three parties of the left slightly weaker.

In political terms the betting market expected the same result as the one that was predicted by the polling institutes and the pundits, i.e. a CDU/FDP share of the vote large enough for an absolute majority of seats in parliament.

Have to briefly concur with “a” here: by my memory of the history of General Relativity, the precession of Mercury’s orbit was already known well before Einstein developed the theory, and was cited as a pre-existing piece of evidence, much as the Michelson-Morely experiment was for Special Relativity. Furthermore, the experiment that supposedly did clinch it after the fact, the Rutherford measurement of the deflection of starlight by the sun’s gravity, was subsequently shown to be within the margin of error the apparatus, and thus was ambiguous in its support of General Relativity. So I think you need another paradigm case for your one-piece-of-evidence-is-enough view. I doubt such a case exists.

English-language betting on German politics can be found here: http://www.intrade.com/jsp/intrade/contractSearch/

Intrade.com’s betting on the 2008 US Presidential election includes some bets I consider rather odd.

“It is a reasonably well-established principle that the average of forecasts is better than the best forecast …”

I disbelieve.

I am thoroughly unconvinced of the practice of putting someone’s head in an oven and feet in the freezer and saying the average personal climate is more comfortable than a typical day (and of course better than the indivdual parts!)

Returning to General Relativity re John Quiggin’s reply in 23, it was certainly the case that many people came to accept GR’s truth based on Rutherford’s experiment. But as with SR, I think many, like Einstein, believed it purely on theoretical grounds, so that those “convinced” by the starlight experiment were largely a) already convinced on theoretical grounds, or b) didn’t know enough about the math of GR to go against the huge publicity of the Rutherford experiment. I haven’t read about anyone who had deep theoretical doubts but was convinced by the experiment, though that’s quite possible, since Rutherford exaggerated the results (unintentionally, I think) enough to make them convincing.

I don’t know much about Bayesian probability, but in most scientific revolutions I’ve looked at, it has been the theory at least as much as the experiment that has convinced the relevant scientists. An experiment that equally supported GR and the much simpler Newtonian physics should be (and usually is) taken to support the simpler, null hypothesis (if only because, time and again, the simpler theory seems to more often to turn out to be correct, all else being equal). Only really beautiful new theories, like SR or GR, seem to be theoretically compelling enough to grab the mantle of the null hypothesis from the previous theory, such that ambiguous evidence could be taken to support both equally. But I don’t know how many other cases there have been where this happened I can think of, and GR’s immediate success was due (even among scientists) in large part to Einstein’s existing reputation. So I think the impact of the Rutherford experiment was quite possibly a mistake, and one that science actually makes relatively infrequently. In any case, the wisdom-of-crowds arguments are not remotely in the same league as GR in terms of theoretical appeal, and thus bear a much higher burden of proof.

Thinking a bit more about Bayes…based on the little statistics work I’ve done, I think the force of the null hypothesis depends a lot on where you happen to be. That is, for Rutherford, he sets out to test GR, and gets a result whose error bars overlap both the GR and Newtonian (N) predictions; chances are, were N correct, he could just as likely have gone on the other side of the N prediction, with no overlap of the GR number. So the fact that he did get a datum that overlapped both is, as a one-time-thing, slightly in favor of GR, all else being equal. But from my point of view, as anyone other than Rutherford at the time, it sure wouldn’t have made the papers on the other side of the N value. With the fame of Einstein, etc, there’s a huge selection effect. Who knows how many experiments produced N-leaning results before Rutherford came along? This may seem like an irrelevantly realistic issue, but I think it is an important part of Bayesian reasoning, which really is talking about the decisions one makes with a single instance. Rutherford might be right, if he had to bet his life one way or the other right then, to bet on GR, but no one else would be right to do that. Certainly I’m going to be equally skeptical of any data that emerges supporting the betting markets, though I suppose John Quiggin is in Rutherford’s position in that he proposed the test in question, and thus might be willing to bet his life (if forced) on the pro-market side if this particular market did an ambiguous bit better than the polls (since it could have done worse).

For me, since there’s a decent chance of the market accidentally landing on the correct prediction, and a high chance of that accident getting publicized, I’ll have to wait for a) lots of evidence, and b) a convincing theoretical defense of the superiority of markets. Perhaps one piece of evidence could do it if, say, the market predicted something with a poll-based probability less than .0001, but since elections are bounded quantities and polls are almost never off by that much, it is hard to imagine that scenario happening.

“On the general approach, what I state in the post is a willingness to prefer the hypothesis “markets outpredict polls” over its contrary on the balance of probabilities.”

John;

I just want to stress this point.

You knew when you made the prediction that the polls had gotten it wrong. Markets would either get it right (and beat the polls) or wrong (and equal them). But observing markets getting if right doesn’t support the “markets outpredict polls” hypothesis – it just shows that markets and polls give different results. You hace to show that polls don’t beat markets with a greater frequency, your test says nothing about this.

There are four possible outcomes of interest:

(1) Polls Right, Markets Right; draw
(2) Polls Right, Markets Wrong; supports polls
(3) Polls Wrong, Markets Wrong; draw
(4) Polls Wrong, Markets Right; supports markets

You only consider (3) & (4). So long as Polls and Markets aren’t identical, markets will sometimes get it right when polls get it wrong – but that doesn’t show the “superiority of markets”; it just shows polls and markets aren’t identical.

Polls can also get it right when markets get it wrong – (2). The outcomes that support polls are unobservable. Your test is set up to count the hits but not the misses. Hence, my coin example. Tossing a coin will out-guess polls some of the time, but it isn’t superior to them because polls are going to outguess a coin more often. Your test, however, can’t establish this because you’re not looking for (2).

tracy (#27) – There’s a class of problems where, in general, many people will give answers all over the map – both very high and very low. I think it’s a poor trick to take something where low and high will cancel each other out, and then proclaim that averaging gives a better answer than the individual estimates. In a sense, that only works because you know the behavior of the samples beforehand with regard to the true value. But if you *don’t* know that behavior, guessing that it averages out, is nothing more than hoping the Central Limit Theorem of statistics is favoring you there. Which is again unimpressive in my view.

Basically, there’s a complicated question-begging. Sometimes errors average out, and sometimes they don’t, but if you don’t know which is the case in any particular problem, there isn’t much justification for blindly averaging.

Ah, “in-house” prediction markets for corporations are a special amusement of mine.

Do you know why they work? Hint: It’s not because of any mystical wisdom-of-crowds.

It’s for a very simple reason:

IT PAYS PEOPLE ANONYMOUSLY FOR TELLING THE TRUTH!

I’m a programmer by trade. Any programmer can tell you a tale where they can give an accurate estimate of project time, and management doesn’t want to hear it. In fact, the accurate estimate will likely be punished. They want to hear an inaccurate estimate, usually called “aggressive”

But provide a setting where the people who are right have a reward, and not a punishment – surprise, the information has a tendency to converge to the correct answer.

It’s a shame one needs market folderol. But I suppose if management can understand it, and the engineers profit, it’s OK. But don’t think it works because of any wisdom-of-crowds. It works in fact because of wisdom-of-experts.

Seth (#32) – the whole point of averaging is that the high and low values cancel out (including the high and low values that are quite close to the true value).

If it is a one-off forecast, of course, then one does not have any particular reason to believe that the whole set of forecasts will be biased above or below the actual forecast. If, however, forecasters have some error-correcting mechanism going on (and a reason to try to improve their errors) then the forecasts shouldn’t be systematically biased one way or another. We’re not talking about random guesses here, the forecasters are trying to get their values right.

Of course, at times the consensus forecast can have errors of the same direction for several periods running, sometimes that’s because people mistakenly identify a trend as a blip or vice-versa, and sometimes for the same reason that you sometimes get four heads in a row if you toss a coin.

And the other problem is – what’s the alternative? Picking the best model is a good idea if the errors are systematically biased, but you don’t always know what the best model is ahead of time.

I’ve constructed a forecasting model for “Other Persons” tax according to this method, and was able to come up with a model that averaged the results of multiple rough-models that got the direction of change right for both in-sample and out-of-sample data. Okay, from memory, I only had 13 data points in total, but the previous models couldn’t get the direction of change right for their in-sample data. At times the averaged forecast gave a higher error than one of the sub-models, but over 13 periods the averaged forecast errors were lower than the forecast errors of the best of the sub-models.