Apparently I am on “mea culpa watch” from Tyler Cowen, Picture me at present pursing my lips and flapping my wrist in the international signal for “ooh! Get her!”. I have looked at the NEJM study, had a look at some of the online discussion of it, and I think that few of my friends and few of my enemies will be disappointed to learn that my response is not so much “mea culpa” as “pogue mahone”. In particular, see below the fold for a list of apologies not forthcoming, additional castigation, and new heretics who need to be squelched.
Right, kicking off with the abuse of the current study. Currently castigating …
Anyone headlining the 150,000 violent deaths figure as if it were the total.
No. That’s not what the study said. Anyone who is trying to claim “The Lancetissses said 650,000 but the true figure was only 150,000 OMGLOL!” or similar, is misrepresenting what the study says. This study found a big increase in the death rate in Iraq as a result of the invasion. They didn’t carry out an excess deaths calculation, but the change in the death rate is of the order of a doubling, which on the basis of their estimate of the pre-war death rate would indicate 400,000 excess deaths as an order of magnitude calculation.
There is a related mistake that people are also making at the moment, which is to try and talk down the Lancet total by claiming that “the apples to apples comparison” is between the Lancet’s estimate of excess deaths as a result of violence (of the order 600k) and the NEJM estimate of total deaths from violence (of the order 150k). This is a mistake, because it’s ignoring the fact that the NEJM study found a big increase in the non-violent death rate which isn’t in the Lancet study, and the question of the potential misclassification of deaths as violent or non-violent depending on who’s asking is a known issue – I seem to remember saying a few times in 2006 that the breakdowns by cause couldn’t bear the interpretative weight that people were putting on them. Lancet 1 got roughly the same balance between nonviolent and violent deaths as the NEJM/IFHS, but Lancet 2 didn’t and Lancet 2 is the only one that checked death certificates. It is hardly difficult to see why, given the atmosphere of fear of death squads in a lot of Iraq, people might want to pretend that a relative died of natural causes when he didn’t.
So, to make this comparison is to implicitly cherry-pick the lower increase in the violent death rate from the NEJM study, and the lower increase in the non-violent death rate from the Lancet study, giving a number that I would call “the Greengrocer’s 150k” (so called because it appears to be the result of trying to put apples to oranges, picking cherries and ending up with a lemon).
Anyone making bold assertions of the superiority of the NEJM study based on sample size
The NEJM does have a much bigger sample size than either Lancet study (9345 households in 1086 clusters versus 1849 households in 50 clusters). But the randomness of the selection was seriously compromised – 11% of the clusters were too dangerous to travel to, and they have their data filled in by extrapolation from the Iraq Body Count website.
This matters, of course, because if we’re thinking as Bayesians, the weighting we should put on the new study relative to the existing ones is inversely proportional to the uncertainty of the estimate. And the uncertainty of the current estimate is dependent on an assumption about how much inaccuracy this extrapolation introduced into the recipe. The authors write, fairly enough, that “Uncertainty in the missing cluster-adjustment factors was difficult to quantify, since we assumed that the excess risk of mortality in missing clusters in Baghdad and Anbar was normally distributed, with standard deviations of 0.2 and 0.1, respectively”, but if you plugged higher numbers into this guess, it would blow out the confidence intervals quite materially. Anbar is the real question mark here – it is a very big contributor to the death rate estimates in the Lancet studies, and it was one of the least well-covered areas in Iraq in terms of the media reports that IBC relies on.
Finally, this study and Lancet 2 cover the same time period (from the invasion to July 2006), but the fieldwork for the NEJM one was carried out later, and over a longer period of time. This means that more households disintegrated, more Iraqis left the country, and therefore the potential for undercount grew – and it grew quite sharply during this period as it was the absolute peak for violent death at the time. The survey notes that this lack of a stable population could quite seriously undermine the underreporting correction that they tried to make.
All these things are mentioned in the NEJM paper, by the way – I’m not Milloying about it here. Kieran is correct to say that this lower estimate shifts our central estimate of the excess deaths caused by the invasion. But there are plenty of people who seem to be suggesting that the correct response is to chuck Burnham et al away and latch onto this as the new definitive number, which is crap Bayesianism in the first place and doubly incorrect given the points made above.
Now onto the “apologies specifically not forthcoming” section …
Anyone who had a go at the first Lancet study and who is now pretending to be retrospectively vindicated by this one, despite the fact that the violent death rates measured in both are the same
Well I mean really. By the way, the link Tyler gives in his “mea culpa” post is to my very first post on the Lancet study, so he appears to be (perhaps accidentally) putting himself in this camp – either he hasn’t checked whether the NEJM study compared its estimate to Burnham et al (2006) or Roberts et al (2004), or he hasn’t read my blog post properly. I’d add that to have been sceptical of Lancet 1 (when it was the high number) but not to have a word of criticism for this study (now that it isn’t the high number) goes really badly for the old credibility.
Fred Kaplan and anyone else who reproduced the “dartboard” argument or “Kaplan’s Fallacy”
These two were idiotic at the time and since they were idiotic based on the maths, they were idiotic a priori and no empirical evidence at all could make them less idiotic. Even if Iraq had never been invaded, a Platonist would say that these arguments would still have existed in abstract space as hypothetical elements of the Form Of The Stupid.
Anyone who opined wisely on the existence of comprehensive central repositories of death certificates in Iraq
By the way, I don’t recall any fawning apologies coming my way on this subject; most of the people who made the asinine argument that L2 must have been flawed because of mythical Iraqi government statistics based on this mythical and wholly nonexistent centrally collated repository didn’t even post a correction. I suppose the idea is that any slightly ambiguous data point or small flaw immediately discredits a commentator on the Left, while no amount of disconfirming evidence ever merits so much as an apologetic harrumph from a commentator on the Right (cf, Moore, Michael; Kristol, William).
Anyone who made evidence-free accusations of fraud against Les Roberts, Gilbert Burnham, Riyadh Lafta et al
When we’re talking about “squelching heretics” or “castigating” people here, let’s just remember who was saying what about who, hey? Let’s get this straight. Falsifying interviews for a survey is serious scientific malpractice. Anyone guilty of it ought to be sacked from their job and publicly disgraced everywhere people respect science. If at any future date, Roberts, Burnham, Lafta or anyone else involved in either of the Lancet reports is shown to have falsified data, I will be right in the vanguard of those calling for the very severest of sanctions to be applied and I suspect Tim Lambert will be too (if you want to know how pissy I get when I feel like I’ve been played for a sucker, ask David Kane). Let’s be totally straight about this.
But if it’s a serious crime (and it is), then accusing people of it should only be done on the basis of evidence. If it’s not OK for me to say “Fred Bloggs is a bit of a hack because he’s reproducing this crap argument about the Lancet study which he plainly doesn’t understand”, then surely we can all agree that it’s much more not OK for Fred Bloggs to say “Well that devastating critique has just about wrapped it up for me, so I suppose that the Lancet team must have falsified their results in order to influence the US elections!”
Anyone trying to pretend that people who defended the Lancet studies against ill-informed criticism in some way “wanted” the death count to be higher and are “disappointed” by the IFHS survey
This is not so much a “no apology” as a heartfelt “Kiss My Arse”, with a side order of “Try Saying That To My Face, Sunshine”. This is and always was a pure, simple and disgusting insult. Anyone who ever did this, went straight on my shit-list and has been on a permanent 100% discount factor for their views on Iraq ever since. I’ve even lodged standing instructions with the Grice United Fund to make sure they don’t accidentally respect your opinions on my behalf. Which brings us on to the subject of …
In general, I just don’t agree with Tyler’s implicit view that there’s something illegitimate about making your case forcefully and not giving the kid gloves treatment to people who try to push weak, uninformed or fraudulent arguments against it (I’m glad to note that, revealed preference reveals, Alex Tabbarrok agrees with me on this one). Things which are definitely and provably wrong, do need to be squelched. We are not in relativist country here – there are some things which are true and some which aren’t, and “the government of Iraq has accurate mortality statistics, based on death certificates collected from around the country”, isn’t. Nobody benefits if people are allowed to pretend that this falsehood is true, and people who assert it without checking it don’t deserve very much in the way of condescending validation of their views.
Let’s not forget that, although we can all sit around and scratch our chins now saying “wwwelll, we all agree that thousands of people have died in Iraq, but …”, at the time when Lancet 1 came out, there were plenty of people (and not just right wing maniacs either) trying to claim that the Iraq Body Count figure was wildly inflated by antiwar political ideologues. This was the period of “all the good news that the MSM won’t report from Iraq”, remember?
A lot of the reason why the Lancet debates were so vicious was that so many people decided to join in them who clearly had no business getting involved (for lack of relevant knowledge), but who did so on the basis of the repetition of a small number of “seed” fallacies, like Kaplan’s Fallacy, the Cluster Sampling critique, the Mere Random Sample, the Mythical Death Certificate Repository, Kane’s Zombies and so on. Stamping on these, early and hard, before they take root, seems to me to be exactly the sort of thing that bloggers ought to be in the business of doing, if they care about the debate at all.