Collective Wisdom

“individuals make noisy guesses, which on average are unbiased and uncorrelated”

Isn’t this just assuming the conclusion? Why would you ever make these assumptions?

Also – isn’t there lots of evidence that a group of people will converge on the views of those group members who assert their views most forcefully and unbendingly?

Adam @1: There’s certainly a lot of evidence that a task-oriented group will converge on the views of the highest-status actor (typically white men), regardless of whether he/she is the most knowledgable or not. This person can be the loudest and most forceful, too, but the driving factor is social status, not volume.

As an aside, it always amuses me when behavioral economics reinvents 1950s social psychology. And then it depresses me.

Sometimes experts try to solve the wrong problem. Two examples spring to mind:

1.) Engineers called in to respond to complaints about a slow bank of elevators in an office tower didn’t solve the problem, but a non-engineer in their office, hearing them talking about it, did. He correctly deduced that the complaints were the problem, not the functioning of the elevators, and saved the engineers’ reputations by getting them to install full-length mirrors on the walls of the elevator alcove.

2) An article in NPR yesterday describes how turning a thorny problem in enzyme structure into a computer game led to a solution which had escaped molecular biologists for more than a decade. The gamers knew nothing about the science, they were solving a much simpler puzzle: The game challenges players to manipulate the structure of the protein so that it reaches the lowest energy state, which earns them the highest score. The function of the protein changes with the shape it takes.

Often, I think, the wisdom of crowds consists in the diversity of approaches to any given problem which it can contain. Many of these approaches will turn out to be silly, and many will follow already discarded conventional wisdom of some sort, but a few may prove to be just the ticket. As a corollary, this is probably why efficient propaganda is not a good idea, and democracy is.

@William Timberman: That NPR story was the first thing I thought of reading this as well. It seems to me that the “Wisdom of Crowds” absolutely depends on expertise in designing the question. Per the protein folding game, a poorly designed game would have created the exact same outcome for the gamers (i.e. a group of gamers would have “won”) but the exact wrong outcome for the scientists interested in protein structure (ie, an incorrect model for the protease structure). It seems that the two key components are 1) knowing what questions can be answered through this sort of model, and 2) knowing how to ask the question to get an unbiased, uncorrelated set of answers to sort. It is an interesting and I think complementary approach to expertise based, hypothesis driven research. The analogy that strikes me (as a total novice) is comparing straightforward computer programming (where you set up a list of commands to transform input into output) to parallel computing or even the fabled quantum computing.

I’m not sure anything described here falls outside of standard learning theory. I would think a good explanation could be found in some variation of Holland’s schema theorem. Diversity and good heuristics can trump expertise, sometimes, given enough diversity and time and adequate heuristics.

Besides, the whole reason that “ask the audience” works on Who Wants to Be A Millionaire is because while guesses may be random, the people who know the answer will distort the average. It is the non-Gaussian outcome of asking many random people that makes it work. Not that people are on the average right, but that ignorance can be credibly modeled as a Gaussian distribution in many realistic contexts, while knowledge shows a clear bias. I’m sure Cosma has a whole bibliography of citations on very unproblematic-seeming problems where ignorance cannot be modeled as a Gaussian distribution, at least if we consider modeling ignorance to be what a good Bayesian prior should do.

I would think Holland’s theorem is a much better basis for most of crowd-sourcing’s successes than assumptions of Gaussian-distributed error in amateurs’ guesses. After all, if we asked millions of Americans in 2003 how many nuclear weapons Iraq had, the average would have been radically incorrect. If we’d asked them to bet money on it, the experts would have bet a lot that Iraq had none, and the amateurs would have bet less to reflect their lack of confidence. The market average might have been more informative.

@ Paul Orwin and AcademicLurker

My intent wasn’t expressed clearly enough, I suspect. In any event, I agree completely that expertise is never irrelevant, especially as AL puts it here:

Anyone who takes “expert knowledge isn’t important” as the lesson of the NPR story is getting it very wrong.

The heroes here weren’t the gamers, but the people who thought of using the game to come up with the solution. More broadly speaking, it isn’t expertise per se that’s the problem, but the fact that acquiring the expertise in the first place tends naturally to narrow one’s focus. The more people who look at a problem, experts or not, the more likely a solution, or at least a fruitful re-ordering of approaches to a solution, will be encountered.

@William Timberman

I didn’t mean to imply that you were claiming that “expert knowledge doesn’t matter” is the take home lesson of the NPR piece. It’s just that, the internet being what it is, I’m anticipating a lot of that sort of thing in response to the story.

OCS @10 is also right. The correct answer wasn’t some weighted average of input from all the FoldIt players. It was the solution arrived at by a small number of players who are, for whatever reason, really good at this sort of task.

Crowds have no wisdom when most of them have no source of news but Fox.

“But all of these examples are somewhat artificial, because they involve decisions that are made in a social vacuum. ”

This actually isn’t accurate. Most mundanely, in the case of the ox-weight-guessers or the crowd on Who Wants to Be a Millionaire?, people are kibitzing, talking to their neighbors, offering up their guesses out loud. So there is no vacuum. What there is is the opportunity, and incentive, for people to tune others out if they wish. More importantly, one of the most striking examples of the wisdom of crowds, to my mind, can be seen at the racetrack, where the betting crowd (which in the US sets the odds on horses, without the help of bookies) does an exceptionally good job of forecasting the likelihood of victory for each of the horses in a race. The crowd does that even though bettors can see what other bettors are thinking (via the odds on the tote board). So the notion that, as some have interpreted the study that Yong is writing about, even the “mildest of social influence destroys the wisdom of crowds” is plainly wrong. Having said that, it’s clear that social influence is a serious problem, which is why so much of my book deals with it. But I think the big problem with social influence arises primarily not when people are hearing many different voices, all offering up a different opinion, but rather when there is one voice, or a small number of voices, that are given undue weight. That’s why diversity is so fundamental to good collective decision-making.

Aside from that, I think Henry’s point about the value of alternative aggregative mechanisms is exactly right.

@18

It is obviously much easier to corrupt, hijack, hotwire, pervert, or (supply pejorative verb for exercise of malign influence) a small number of “thought leaders” than a huge number of tightly networked thought contributors. Much of the recent economic misery is attributable to the simultaneous moral failure of multiple elite groups in finance, journalism, academia, and government. It turns out that the few thousand “thought leaders” could be cheaply bought. Beyond the evils of mob rule and corrupted elites lies the tantalizing possibility of Internet-based knowledge networks that are honest, reliable, and democratic.

“How we organize groups of people to come to useful conclusions is a different problem.”

“What this shows is that crowdsourcing can be effective if the way that the crowd interacts with the problem and each other is appropriately structured for the task at hand.”

When I read Surowieki’s book, this was pretty much what I thought it was about. There is no guarantee that a mob will get the right answer, but knowing that, under proper conditions, crowds are pretty good at generating answers, the challenge is to generate the proper conditions. This “wisdom of crowds” business suggests a natural resource, but not one that’s free for the taking. If, and only if, you build a proper water wheel can you use it as a source of power. Only if you build a proper system for extracting aggregate answers can you get information from crowds.

Remember the missing submarine story? Picking the right people, asking them separately where the sub is, that’s what made it possible to find the boat. Not the same as guessing the weight of an ox at all.

Where in this construction appears the miraculous, emergent phenomenon of a group mind

We have seen it emerge in rudimentary form in the Wikipedia editors, who, contrary to all pessimistic predictions, have created an unparalleled store of reliable public knowledge out of the undirected input of a vast group of knowledge contributors. Given a hospitable Internet medium, Wikipedia poves that the truth is a self-organizing phenomenon.

I’m reminded of when the Monty Hall problem was published in Parade magazine some 20 years ago and many many people, including a number of mathematicians, wrote in to say that the given answer was wrong (while in fact it was correct). I assumed that any decent mathematician would get the answer right (I myself was wrong at the time, I will say, but I am not a mathematician). This says only, I think, that in some cases it’s even difficult to define who has expert knowledge in an area, meaning that incorporating expert knowledge becomes even more problematic.

And part of the reason professionally trained mathematicians got it wrong may have been that if you set up the problem the way you were taught to set up the last similar homework problem you had, the Parade answer is wrong. Arguably, the problem as published was unspecified (didn’t say originally whether the MC knew where the prize was, so didn’t say whether he could take it away). So the lesson of the puzzle is either (a) real-world problems aren’t like your old undergraduate homework problems and you should look again and see whether the problem is underspecified, (b) in your textbooks things might have been fair but in the real world the guy in charge can take stuff away from you.

CT friend and general culture hero, Aaron Swartz.

The linked blog post, in which he says that he downloaded the whole Wikipedia archives to a cluster of computers, foreshadows recent events. He sure likes big piles of data, doesn’t he?

I don’t think wikipedia and other internet phenomena are in the “wisdom of crowds” category, but rather something like ‘credential-less enthusiasts’. Activists. Obsessive cranks. Not a random selection by any stretch of imagination.

How soon we forget. Wikipedia was widely derided, criticized, and banned from schools just a few years ago. Commercial encyclopedia publishers declared it could never rival a professionally produced product. Now, at over 3 million English articles and 18 million in all languages, nobody is laughing any more.

Wikipedia is a classic example of the reluctant acknowledgement of significant novelty by Internet skeptics, what was an unproven and dangerous nostrum on Friday afternoon becomes an unimportant commonplace on Monday morning. Nothing to see here, move along.

Picking the right people, asking them separately where the sub is, that’s what made it possible to find the boat. Not the same as guessing the weight of an ox at all.

During the second world war, some of the British research centres and industrial plants that invented a couple of things like jet engines and radar and computers held regular “Sunday Soviets” – i.e. meetings to discuss a particular problem that were officially just private, unofficial gatherings that anyone involved might chip into. Specifically, the point was that anyone could speak up without fearing hierarchical revenge. (The name was telling.)

If there’s owt on the ‘net like that it’s IETF. It is less like, well, asking a lot of yelling fucksticks which welfare provision to cut.

straightwood:
We have seen it emerge in rudimentary form in the Wikipedia editors, who, contrary to all pessimistic predictions, have created an unparalleled store of reliable public knowledge out of the undirected input of a vast group of knowledge contributors.

Get off it. Wikipedia articles vary wildly in quality; some are reliable, some aren’t. No one copyedits the damn thing — typos and grammatical errors abound, and some of the sentences aren’t even sentences. Management puts up a notice saying “this article needs more citations,” some one supplies the citations, but management never takes the notice down. The thing is not especially easy to edit in conformity with Wikipedia norms, especially for someone unfamiliar with the Internet and codes and all that — yes, straightwood, there are such people. In short, Wikipedia is a valuable resource in some ways (I gave it a small contribution on one occasion during one of their fund drives) but in other ways it’s a mess. It is not consistently reliable and by its own admission it does not rival professionally produced encyclopedias. (Students should not be allowed to cite it in papers, imo, and should be told that it must be used with even more skepticism than most sources.)

Anyone who has played mafia will realize that crowds can be damned funny things. One of my friends once convinced us all he was an inspector who got the wrong answer 50% of the time. We were all math people, and he successfully fooled us with an absurd story. Any one of us who thought about it harder would realize the lie, but because of the crowd dynamics we did not.

@36

I am properly chastened. In future, I shall address the narrow context of each thread and show the good discipline and admirable restraint of other commenters. The future needs no guardians, and portents of its approach will certainly not escape the watchful eyes of those officially entrusted with heralding its arrival.

Actually, the Monty Hall problem is even trickier. To get the answer right, you have to understand, not only that there is always a prize, but that Monty always opens a door without a prize. If he chose at random between the doors the contestant hadn’t picked, ending the game whenever the prize was revealed, then there would be no value to switching.

Since economists routinely get bagged here on CT, I’ll point out that this is a question competent economists (trained since about 1980 or up with the literature) always get right first off, and not-so-competent economists slap their foreheads when their error is pointed out.

Straightwood’s problem with subversion of expert elites has a corollary for the non-expert policymaker. Such policymakers have two traditional strategies; poll the experts or poll the most expert experts. These strategies result in policy pronouncements beginning either “Most experts agree that…” or “Leading experts agree that…”. If you can’t rely on either of these two traditional strategies because of expert subversion, most non-expert policymakers are bereft.

Note that this doesn’t imply that no experts are right; it only implies that those two strategies for getting the right answer from a group of experts no longer work. The non-expert policymaker has no other way of finding the guy(s) who know(s) the answer.

By the way, I just used Wikipedia to find out what IETF means.

#11 ” More broadly speaking, it isn’t expertise per se that’s the problem, but the fact that acquiring the expertise in the first place tends naturally to narrow one’s focus. The more people who look at a problem, experts or not, the more likely a solution, or at least a fruitful re-ordering of approaches to a solution, will be encountered.”

This calls to my mind the way in which chess programs through brute force sometimes find winning moves/sequences which are unlike those of a strong human player- the computer’s trial moves playing the role of the crowd which outguesses the heuristics based approach of the master.

A daily demonstration of the wisdom of crowds can be found in the cognitive dissonance effect observed on blogs associated with mass media opinion leaders. A good example is the experience of the hapless Joe Klein, a leading journalistic “authority” on American politics. For years, on the Newsweek blog, “Swampland, ” Klein has had many of his published nostrums shouted down by huge majorities of commenters on this blog. The reader commentary was generally informed and well argued, but it had no impact whatever on Klein’s status in the hierarchy of the commentariat.

What this phenomenon illustrates is the importance of cultural inertia in preserving the leadership role of those who have attained positional power, either through a personal brand franchise or a grant of precious shelf space in the media marketplace. Demonstrably mistake-prone and out-of-touch writers, like Klein, David Brooks, or Tom Friedman, are able to persist in their positions, even in the presence of overwhelming dissent from their own readers, as revealed for all to see in Internet discussion threads.

Thus, even unshakeable evidence of the documented superiority of any wisdom of crowds phenomenon will be insufficient to shift the cultural paradigm of “name brand” thought leadership. The mechanism by which such a shift may occur remains obscure, and is a worthy subject for academic research.

The article linked by Henry (at 48 above) argues, among other things, that different kinds of rules are appropriate for online projects with different goals: Wikipedia’s rules should be geared to encourage toleration of diverse pts of view, b/c its goal is “knowledge generation”; Daily Kos’s goal is collective action, thus it should have different rules. Quibble: is Wikipedia’s goal accurately described as ‘knowledge generation’? After all, Wikipedia explicitly bans any original research from appearing in its articles (though the ban may not be well enforced). This suggests that Wikipedia’s goal is perhaps better described as ‘knowledge dissemination’ rather than ‘knowledge generation’.

Sev @ 47

It’s an interesting analogy, but I don’t see it as how the most interesting examples of crowd-sourcing work, although I’ll admit that the distributed processing with the SETI PC widget from some years back was pretty clever and useful.

Under the right circumstances, such as a functioning representative democracy, or a cleverly constructed bit of querying machinery, as in the FoldIt example, what you have isn’t massive trial and error, rather it seems to be the controlled engagement of many minds, which each has its own history of perceptions, cognitions, its own memory pathways, etc.

If what is keeping you from finding a solution is a poverty of associations, crowd-sourcing definitely helps in a way that even massive repetition may not. And then, of course, there are dreams, which serve a similar function in single individuals. (I’m thinking here of Kekulé’s epiphany about the ring structure of benzene, and such-like reports from other theorists.)

I’ve always thought you shouldn’t let juries deliberate. Just make them larger and take a vote.

This was how they worked in Periclean Athens. The system gets a bad press because an Athenian jury condemned Socrates on this method and the European intelligentsia has never forgiven it for this. Personally, I think it’s a damn good idea and that the outcome of Socrates’ trial is a point in its favour.

Bloix, thank you. That’s the first natural language account of the Monty Hall problem that I’ve ever found convincing without qualification. Please can I nick it for future reference.

I’ll pass on the Socrates case, except to note that pretty clearly he got the verdict (and especially the penalty) he wanted, and whether you take Plato’s explanation or Xenophon’s depends on whether your opinion of Plato is as low as mine, but the condemnation of the generals after Arginusai is a far better indication of how hideous that system could be.

It is not surprising that Wikipedia is poor at dealing with politically contentious issues. What is surprising is that Wikipedia is excellent at everything else.

Several months ago I happened to look at the Wikipedia entry on Talcott Parsons. Lots of info, not well written, lots of typos and infelicities, with open arguments at the end of the piece about who did or did not attend his funeral (or was it memorial service). If you call that excellent, fine; I call it a mess. (Maybe this particular entry has been improved since I looked at it; chances are it hasn’t.) Wikipedia is very good on some things, but it is not “excellent on 90% of applications.”

The other reason why wisdom-of-crowds arguments are appealing is that most people find liberty, representation, and the respect of individual agency to be normatively good. If you think you have a right to an opinion, that the people involved in a decision have a right to be heard, and that their status as reasonable beings is valuable in and of itself, you’re unlikely to derive a theory of optimal public choice that leads towards dictatorship.

This is a little “duh” but worth remembering.

Meanwhile, Monty Hall. The problem I always have with it is that the formulation seems to suggest that you would be better off picking a door again no matter which door you pick second time around. You know, after Monty shows you a goat, that there are now one goat and one Lincoln remaining behind the two doors. (Monty, as Bloix and the Quig point out, always shows you a goat.)

But this tells you nothing about which is which. If you pick again, you have a 50% chance of the Lincoln as against 33% first time out. By definition, if the chance of winning the Lincoln is 50% for one door, it must be for the other, as it has to add up to 100%. So you end up with the conclusion that it is rational to change your mind, whatever you change it to. In fact you’re better off as soon as you undertake a second act of choice – even if you don’t actually shift. So what is the difference between thinking about it, then picking the same door again, and not thinking about it?

Obviously, if you picked the door Monty opened, you should change (unless you like one goat more than all the goats you could buy for the resale price of the Lincoln) and indeed it doesn’t matter which you pick. But I think that’s a reasonable exclusion.

Bloix #32 :

“I personally had a lot of trouble accepting that Marilyn Vos Savant was right, because I couldn’t understand how, after you’d chosen one curtain, Monty showing you a goat behind a different curtain communicated any information about the true location of the prize. And if Monty is not communicating information, then the odds from your point of view should not change from 1/3.”

Your intuition that Monty’s action is independent of whether your door contains the prize, and so the probability of your door containing the prize is unchanged by his action and remains 1/3, is correct. But then it follows that the complement event (your door does not contain the prize, i.e. the prize is behind the remaining unopened door) has probability 2/3 and you should switch.

As Henri #62 says, the result is probably easier to see in the case of a game with a larger number of doors… 52, or a thousand, or a million.