Selection bias

“The Economist”:http://www.economist.com/displaystory.cfm?story_id=1907893 has a long article asking whether or not companies are too risk averse to take proper advantage of new opportunities and a changing marketplace. Explaining the roots of corporate advantage is tricky stuff; conventional economic theory isn’t very good at telling us when efforts to innovate are going to be successful, and when they’re not. Economic sociologists do a slightly better job, but they still have difficulty in providing useful lessons for business people. Which opens the way for all sorts of cranks and quacks, who offer dubious nostrums for business success, with all the fervid enthusiasm of a 19th century medicine show charlatan. I’m referring of course to management “theorists.”

Now, I don’t want to dismiss them all out of hand – some interesting and serious work does get done at business schools. But management theory has more than its fair share of fakers, especially towards the populist end of the market. Grand, sweeping claims are made on the basis of very iffy and dubious research and case studies which aren’t chosen for any better reason than that they can be shoe-horned into the analytic boxes prescribed by the prevailing wisdom.

Which brings us back to the _Economist_ article. It gives a guided tour of business school research on creative ways to deal with changing circumstances. The survey gives the impression that the field is a mess: different management theorists shouting bland (but somehow mutually contradictory) business prescriptions at each other. For example, one major five year research program comes to the conclusion that successful companies are distinguished from their peers by four factors – flawless execution, a company culture based on aiming high, a structure that is flexible and responsive, and a clear and focused strategy. Bloody obvious stuff in other words, which any half way intelligent senior executive could jot down on the back of a beermat in five minutes.

Why is this stuff so bad? Two possible reasons I can think of. The first is that management theory is less interested in the patient accumulation of real knowledge, than in providing training-fodder for future managers. Probably not all that much that can be done about this – b-school professors get whopping salaries for training MBAs, and are likely disinclined to change their ways. But the second reason is a problem that they could address without too much difficulty – their research methodology is often just awful.

The _Economist_ article gives an apparent example of this in its discussion of two highly influential books – _Built to Last_ by Jim Collins and Jerry Poras, and _Good to Great_ by Jim Collins on his own. Now I haven’t read these books, so I’m going on the Economist‘s description of what they have to say – it could be that they’re innocent of all the charges laid below, and are serious pieces of research. They’re almost certainly a considerable cut above the _Who Ate My Cheese_ genre of populist business nonsense. But it still sounds as though these books commit a serious social-science sin – “selecting on the dependent variable.”

What does this mean? Social scientists tend to believe that if you want to find out if a causes b by studying different cases, you need to be quite careful in choosing the cases. For example, if you want to argue that risk taking leads to business success, you want to look at cases of firms that are risk takers, and firms that are risk averse, and you also want to have cases of firms that are successful, and cases of firms that are failures. If you only study successful risk-taking firms, you’re cooking the books. It could be that there are many more risk-taking firms that are failures out there than successes – but because you’ve only chosen to look at the successes, you have no way of knowing this. You can thus end up providing pretty bad advice.

As far as I can tell from the summary in the _Economist_ , _Built to Last_ makes a mistake of just this sort, when it tries to figure out the source of business success by looking

bq. at a small sample of companies (18) that had been persistently great over time. It suggested that endurance and performance were linked.

Apparently, so does _Good to Great_ which looks at companies which spectacularly outperformed the stock-market over a three year period, and argues that the source of their success is quietly determined CEOs who believe in high standards.

The problem is that studies which concentrate on successful firms can’t tell us anything very useful about the differences _between_ successful and unsuccessful firms, because they haven’t looked at the latter at all. We don’t know whether or not other companies, which also endured a long time, had below average performance. And very likely, many did (at least that’s the conclusion of another piece of research that _The Economist_ also cites, which sounds a little more satisfactory). Equally, companies that tanked may also have had quietly determined CEOs who believe in high standards (which in any event sounds like a very hazy and subjective judgement on the researcher’s part – how can you _really_ tell the one kind of CEO from the other).

Case-studies play a big part in business-school education; they’re a useful way to make students work through the consequences of real life decisions. And they can, indeed, contribute to our more general stock of knowledge, if scholars use certain methodologies (such as ‘process tracing’) carefully. But management theory not only relies heavily on case studies – it all too often uses vague and wuffly concepts, that reflect the pre-conceived biases of the researcher rather than grounded theories. Thus its rather extraordinary noise-to-signal ratio – it’s less suited to providing new insights than new jargon to be wielded by McKinsey consultants in order to terrify staid managers into submission.