Comments on: Lancet roundup and literature review

By: Deltoid » Innumerate BBC

Deltoid » Innumerate BBC — Tue, 08 Nov 2005 09:41:02 +0000

[...] Of course, it isn’t far out of line with other studies—it agrees quite well with the ILCS survey. And while it is possible to criticise the estimate because the pattern of violence has been so uneven, such a criticism that was grounded in an actual understanding of statistics would be that the Lancet underestimated the death toll because of the exclusion of Falluja and the tendency for cluster sampling to underestimate deaths when they clump together. [...]

By: Deltoid » Hitchens’ crazed fabrication

Deltoid » Hitchens’ crazed fabrication — Fri, 16 Sep 2005 12:31:21 +0000

[...] Kaplan’s criticism of the Lancet study was demolished here and here. [...]

By: Deltoid » Shannon Love and Andy S take swipes at the Lancet study

Deltoid » Shannon Love and Andy S take swipes at the Lancet study — Thu, 15 Sep 2005 18:36:30 +0000

[...] Update: Sigh. Brendan Nyhan cites Kaplan’s badly flawed critique. [...]

By: dsquared

dsquared — Mon, 15 Nov 2004 20:52:34 +0000

Mike: In principle, you might be onto something here; they certainly did use unadjusted January 2003 population estimates (I think that these were estimates rather than census data) for calculating the sampling scheme, and on the face of it, I’d say that this would tend to oversample people who’d stayed in the dangerous zones (although it would by the same token undersample households who had fled the scene after losing a family member). Note that households would have to move from one governorate to another in order to affect the chances of being sampled; my guess is that this would mean that Baghdad would be inflated relative to rural areas, as this is the only town where I would guess there were lots of people who had family to stay with in another governorate. On the other hand, I think I would guess that, given the extraordinary consistency of the story across the non-Kurdish governorates, this effect might not be too large in magnitude. But I can’t rule out a larger effect; this is a bloody good spot on your part. This article is about to drop off the front page, but I’ll keep in touch by email if I think of anything else.

By: Dan Hardie

Dan Hardie — Mon, 15 Nov 2004 20:36:13 +0000

Sebastian, if the survey had asked Iraqi respondents to say if the dead were combatant or non-combatant, and computed combatant/non-combatant death rates on those basis, you’d be telling us that the Iraqis were lying to inflate the number of non-combatants killed. I note that: i) a complete disinterral of all corpses in Iraq, or even a disinterral of a large sample of the same, followed by their forensic testing for evidence of weapons use, seems a rather impractical way of doing things; ii) the Allawi government, with not a word of public protest from the Bush administration, has discontinued publishing the Iraqi Ministry of Health’s figures for corpses seen in Iraqi hospitals (figures of war wounded seen in Iraqi hospitals, which in my view would provide a strong basis for an extrapolation of war dead, are also not being published); iii) the Coalition forces are estimating Iraqi deaths but not publishing those estimates. Given these facts, could you please tell us what you would accept as a basis for responsibly estimating the Iraqi death rate?

By: Mike Dunford

Mike Dunford — Mon, 15 Nov 2004 19:05:42 +0000

Thanks for your analysis of the paper. It’s helped me clear up at least a couple of the reservations I had about the methodology. However, there is a point that concerns me which I didn’t see addressed, and I wonder if you could help me with it. At both the province level and the locality level, decisions on where to sample appear to have been made based on the available prewar census data. But during war, there are frequently population dislocations, and these dislocations are non-random. Whenever possible, people relocate from high-risk areas to safer areas. If this has been the case in Iraq, wouldn’t using the prewar census data carry a significant risk of resulting in the oversampling of high-risk areas and the undersampling of low-risk areas? And couldn’t that, in turn, result in an elevated estimate of the nationwide mortality? Thanks

By: dsquared

dsquared — Mon, 15 Nov 2004 18:54:34 +0000

The unemployment figures also came out during the election campaign. Perhaps the BLS ought to have split them up into “proper” unemployed, plus people who were in their opinion lazy, workshy or not trying hard enough. It’s certainly been suggested in the past to the medical statistics profession that they ought to break down AIDS numbers into the categories of “innocent victims” who got the disease through blood transfusions, plus gays and drug users, whose illness obviously matters a lot less. To their eternal credit, they have always resisted this pressure.

By: Antoni Jaume

Antoni Jaume — Mon, 15 Nov 2004 17:51:23 +0000

mr Holsclaw, that the USA never even tried to have a fiable and public statistic of deaths and other mishaps that happend on the Iraqis is enough to negate their innocence. Let’s be excessive: the Nazi thought they were right and so documented most of their crimes. That made Nurenberg processes that much easier. Their modal (not moral, after all Nazis claimed to do what was right) offsprings of nowadays, like Milosevic, take care that no official trace can come up to them. The USA are doing the same, so while soldiers can believe their indoctrination over fighting terrorism and terrorists, the officialdom knows it is a lie. DSW

By: Sebastian Holsclaw

Sebastian Holsclaw — Mon, 15 Nov 2004 07:53:28 +0000

“Not only is there no objective basis for the actual subjective adjustments that people make, but the entire distinction between combatants and civilians is one which does not exist in nature. As a reason for not caring that 98,000 people might have died, because you think most of them were Islamofascists, it just about passes muster. As a criticism of the 98,000 figure, it’s wretched.” Shorter d-squared: You shouldn’t criticize the ‘science’ just because they didn’t bother with the important part. Guess what, they tried to influence the election with their hack summary. Failing to discriminate between combatants and non-combatants is failure to discriminate between politically powerful and politically crass.

By: Joshua W. Burton

Joshua W. Burton — Sun, 14 Nov 2004 05:31:57 +0000

There is an interesting correlation between two of the critiques Daniel itemizes here: the inconsistency of the infant mortality data may tend to support the Lying Iraqis theory. Back in 2002, DD and I had a brief exchange in which I pointed out (1) that pre-war Iraqi infant mortality numbers were controversial, (2) that the UNICEF estimate was much higher than the CIA’s, (3) that the CIA estimate was much easier to reconcile with total Iraqi population numbers 1990-2001, and (4) that the mortality rate Saddam was attributing to the sanctions regime was implausibly high based on UNICEF numbers, and absurdly high based on CIA numbers. See original link at http://www.j-bradford-delong.net/movable_type/archives/000908.html . Anyway, two years later we have the same numbers, but everyone looking at them has changed sides. The Lancet study finds current infant mortality rates that are supposed to have doubled since the war, but remain far below UNICEF’s prewar estimate…and almost exactly in line with the CIA’s. It seems to me that one model which explains all the observed facts is that Iraqi infant mortality has fallen slowly from 60+ per mil to 50+ over the last 15 years, without much impact from sanctions or war. In this model, the CIA and Lancet have always been right in the present tense, and UNICEF and Lancet have both been misled by anecdote in the past tense, in each case in the direction the Lying Iraqis hypothesis would favor. That is, during the sanctions regime, UNICEF investigators on the ground were told about fake deaths (or real deaths of cousins resampled as siblings, or other bias), while today Lancet investigators are losing half the real deaths from before 2003. Since the Iraqi “story” used to favor a high mortality rate under the sanctions regime, and now favors a low past mortality rate for the same sample period, the facts that UNICEF heard 110 per mil, and that Lancet now hears 29, cannot be viewed in isolation from each other. Rather, they begin to make a fair Bayesian case for serial deception.

By: Donald Johnson

Donald Johnson — Sun, 14 Nov 2004 04:16:07 +0000

Heiko, the reasoning of the Pentagon official in the Oct 12 NYT story was strikingly similar to the reasoning of Americans in Vietnam who would bomb VC-held villages hoping to force the villagers to “choose” the American side. Your dismissal of the Nancy Youssef story as not credible was, well, not credible to me. I think you wish to see Iraq through rose-colored glasses and I find this frustrating, so I’ll drop out of this conversation.

By: abb1

abb1 — Sat, 13 Nov 2004 21:52:16 +0000

FWIW - some anecdotal evidence: Farnaz Fassihi, Middle East correspondent for the WSJ writes in her personal email:

… Iraqis like to call this mess ‘the situation.’ When asked ‘how are thing?’ they reply: ‘the situation is very bad.”

What they mean by situation is this: the Iraqi government doesn’t control most Iraqi cities, there are several car bombs going off each day around the country killing and injuring scores of innocent people, the country’s roads are becoming impassable and littered by hundreds of landmines and explosive devices aimed to kill American soldiers, there are assassinations, kidnappings and beheadings. The situation, basically, means a raging barbaric guerilla war. In four days, 110 people died and over 300 got injured in Baghdad alone. The numbers are so shocking that the ministry of health—which was attempting an exercise of public transparency by releasing the numbers—has now stopped disclosing them. [...] I heard an educated Iraqi say today that if Saddam Hussein were allowed to run for elections he would get the majority of the vote. …

By: dsquared

dsquared — Sat, 13 Nov 2004 21:46:37 +0000

Dan, what you’re doing is, effectively, multiplying the Fallujah death rate by the population of Iraq to get a number of deaths, then comparing it to the population of Fallujah. This is why you’re getting a number that looks too big relative to the population of Fallujah, and it’s why the authors threw the Fallujah outlier out; to avoid doing exactly this.

By: Dan M.

Dan M. — Sat, 13 Nov 2004 21:41:45 +0000

D^2 suggested that I used suspect technique in my analysis. In particular, he suggests we don’t know the sample size. It wasn’t given directly, that’s true. But, the authors were kind enough to give death rates with and without Fallujah, as well as the number of post invasion deaths in Fallujah. Thus, we have a direct means of checking my observations. “During the period before the invasion, from Jan 1, 2002, to March 18, 2003, the interviewed households had 275 births and 46 death The crude mortality rate was 5·0 per 1000 people per year…The crude mortality rate during the period of war and occupation was 12·3 per 1000 people per year…If the Falluja cluster is excluded, the post-attack mortality is 7·9 per 1000 people per year” and “When included, we estimate that the rate of death increased 2·5-fold after the invasion (relative risk 2·5 [95% CI 1·6–4·2]) compared with before the war. When Falluja was excluded, we estimated the relative risk of death for the rest of the country was 1·5 (95% CI 1·1–2·3).” Now, in the first quote, there is some ambiguity as to whether methods changed from including the invasion time to excluding it in the two surveys. Since the ratios for the two quotes are equivalent to within rounding, I’ll assume the same technique is used. We also have 1.48 years post invasion. Just doing arithmetic, I obtained the following deaths per 1000 for the whole invasion period All Iraq: 18.2 Fallujah: 6.5 Without Fallujah 11.7 All Iraq, no increase 7.4 Multiplying by the population of Iraq, (24.4 million) we obtain: All Iraq: 445k Fallujah: 159k Without Fallujah 286k All Iraq, no increase 181k By these calculations, we would get 105k excess, instead of 98k. This is consistent with the rounding errors in the numbers I used, so that’s not too bad. The paper also states that 53 were killed in Fallujah during the post-invasion period. Since 33 were killed in Aug-Sep. ’04, the authors would extrapolate (to within rounding) 33/53 * 159k during that period. That comes to 99k. We can even normalize down for my rounding error, and get 92k. So, as you pointed out, my 100k estimate wasn’t exactly the same as what the authors estimated. Still, it’s close. In short, even though I’m just a humble aging, plumber, :-) I can still look at numbers in a paper and see what conclusions are obtained. Typically, I calculate “back of the envelope” numbers using the author’s technique in my head to check out the author’s work. Now, that I’ve spent a bit more time on making direct calculations from the authors’ numbers; I’d argue that my estimate was close enough to prove my point. Dan M.

By: GKL

GKL — Sat, 13 Nov 2004 21:18:32 +0000

GKL wrote:

So I’m at a loss to explain the confidence (again, in the normal usage of the term) of the Lancet authors in the “perfect” 100,000 (or 98,000) number which lies midway between the endpoints of the CI

dsquared replied:

The authors have no such confidence; they have repeatedly said that they believe that the central estimate is an underestimate.

We may be talking past each other on this point, but when authors of a study describe a numerical estimate derived from their work in every press interview as “conservative”, they are indeed expressing confidence in it. In particular, they express confidence that the true number is unlikely to be lower (whatever the CI has to say about it). The explanation given for this is that the Fallujah “outlier” would have produced a central estimate twice as large. But the Fallujah data doesn’t just point to possible underestimation, it also raises question marks about the survey’s reliability as a nation-wide estimator of excess mortality. One would have thought that this would preclude the possibility of making confident claims about “100,000” being a “conservative” extrapolation because it excludes Fallujah. But it seems not.