(Initial bad temper warning: I am a little bit cross as I write this, because I think that the distribution of the paper on the Michelle Malkin website was both silly (because the paper has huge flaws that a mass audience can’t possibly be expected to understand) and rude (because at the time when he gave permission for it to be distributed, David was soliciting comments, seemingly in good faith, from the Deltoid community, aimed at improving it before distribution). The Malkin link has meant that this paper has metastatised and I will therefore presumably be dealing with cargo-cult versions of it by people who don’t understand what they’re talking about from now to the end of time. I see that Shannon Love of the Chicago Boyz website is claiming to have been “sweetly vindicated”, FFS. Ah well, the truth has now got its boots on, and big clumpy steel toe-capped boots they are too. C’mon boots, let’s get walking.)
At this late stage, does anyone believe that careful metaanalysis is going to reveal that the Lancet studies were totally wrong, and that the invasion and occupation of Iraq actually went really well? Apparently yes; David Kane of the Harvard Institute for Quantitative Social Science (who CT readers might remember from this rather embarrassing incident last year) does. He apparently intends to present this paper at the JSM in Salt Lake City on Monday, arguing that the 2004 Lancet study actually could not rule out the possibility that the death rate had fallen in Iraq.
My advice is, David don’t hand this paper out. If not for the sake of your own reputation, think of the four (! On a tiny little paper like this!) research assistants you credit in it. The paper is a disaster. As the comments thread at Deltoid gradually teases out, it’s full of silly mistakes (the author constantly fails to make a distinction between an estimate and its confidence interval) and is based on a fundamental misreading of the paper (in that it assumes that the relative risk rate was estimated parametrically using a normal distribution when it wasn’t). But one doesn’t need to go into the maths of the thing to understand what’s wrong with it.
The mathematical guts of the paper is that under certain assumptions, the addition of the very violent cluster in Fallujah can add so much uncertainty to the estimate of the post-invasion death rate that it stretches the bottom end of the 95% confidence interval for the risk rate below 1. From this, David Kane concludes that the paper was wrong to reject the hypothesis that the Iraq War had not made things worse.
Let’s back up and look at that again. Under David Kane’s assumptions, the discovery of the Fallujah cluster was a reason to believe that things might have gone better in Iraq. This clearly means that these were the wrong assumptions.
The statistical problem here is basically that people can’t come back from the dead. The Fallujah datapoint increases the uncertainty of the estimate, but it doesn’t increase it in both directions, because there is no way that you could find an “anti-Fallujah” (a datapoint which brought the overall average down by as much as real Fallujah brought it up), because such a place would need to have a negative death rate.
And looking at the charts in David’s paper, it’s clear to see that the reason why the left edge of his estimate of the risk ratio has been dragged below 1 is that a substantial part of the distribution of his Bayesian estimate of the post-war death rate is below zero (and an even more substantial part is in regions of positive but wildly improbably death rates like one or two per 100K). That’s all there is to it, CT readers; the majority of the rest of the Deltoid thread consists of three or four people trying to explain that the Roberts et al. paper doesn’t make the same mistake.
As I note halfway down the thread, this is actually a nice example of some of the cases where the distinction between a frequentist confidence interval and a Bayesian credible interval makes an importance difference. In an infinitely repeated series of trials, you might very well get a small number of very unlucky or wild results that showed death rates of 1 or 2 per 100K. So if you’re thinking about the confidence interval as the limit of the empirical distribution of the estimator in repeated trials, it makes sense to have it where it is. But if you’re a Bayesian and you regard the confidence interval as your subjective probability distribution over a random variable, then it doesn’t make any sense at all to have any material weight on these low-end numbers (if you were a conscientious Bayesian, of course, you would never get into this position as you’d have used a sensible prior distribution which put zero probability on death rates below zero). In general, there is a distressing trend among statisticians to use the branding “Bayesian” (and the ubiquity of “diffuse” priors which don’t rule out silly cases like this one) as an excuse for talking crap about confidence intervals, and I think this is an example of the genre.
If any readers are attending the JSM, I’d be interested in any reports of how it went down. Now I’m off to spend the weekend playing Wingnut Whackamole I suppose …
(PS: actually, the cargo cult explanations of why the Lancet study has allegedly failed to enact someone’s ideas of the rituals of science are really quite interesting from a sociological point of view. Although I do wonder about the incuriousness here about the actual facts. I mean, if the “scientific method” regularly threw up conclusions like “knowing what we do now, it is quite likely that the Iraq War was actually a success”, do you think it would be as popular as it actually is?)