Lancet roundup and literature review

yes there was and I’ve put it in now, thanks very much.

To be honest, the editor of the Lancet leaves a slightly funny taste in my mouth. By being so blatantly political, he gives a lot of succour to the people who are claiming that there were defects in the refereeing process, a point which I haven’t covered here because it would only matter if there were flaws in the actual paper, which there aren’t. (Some people, mainly people with no real idea about science, seem to regard “peer review” as magic oofle dust that has to be sprinkled on a piece of research to turn the words into Science). I agree with the Lancet guy that violent death is a genuine public health problem in Iraq and therefore the factors which cause it are a proper object of study for the Lancet, but, in any case, it was too late in the campaign to have ever made it as a real campaign issue, so why bother? God I feel like Fred Kaplan after all that.

You can count me among the innumerate (no pun intended), so my comment can immediately be discounted. However, I would be interested to know in which class of hackery you would put it.

According to the Guardian article, the study was carried out in “33 randomly-chosen neighbourhoods of Iraq representative of the entire population.” How much confidence do they have about this? In a war zone, and in a country closed to the outside world by sanctions for a dozen years, how accurately can one estimate demographic variables such as population density, total population, employment, illness, etc. Employment and illness might seem to be irrelevant, since the study is measuring violent death, but maybe if unemployment is high in one town, there are more people at home when the bombs fall. Maybe in one region, people are more likely to go to the hospital when they are sick. Seems like small variations in these variables could result in significant differences in the confidence interval.

But what do I know, I’m not a statistician. Yet let me say this: many people found this study bogus for the simple reason that extrapolating large numbers from small numbers seems to almost always be pure guesswork. The analogy that first occurred to me was crowd size estimates: the people who put on the rally estimate that 100,000 people were there, while the cops put it at closer to 20,000. Presumably they both have some rational basis for their estimate, but the results are so divergent that there seems to be no science involved at all.

Anyway, those are my thoughts on why people seem to have such a hard time with this, rather than (in most cases) an unwillingness to admit that the war could have killed people.

George; the procedure for ensuring a random and representative sample is detailed in the paper and it’s not hard to understand – it’s nuts and bolts stuff rather than anything specifically mathematical. It looks good to me and I haven’t seen anyone seriously criticising it.

I don’t agree that this is like estimating crowd sizes. As I say, I don’t like the extrapolated number, but the relative risk ratio is solid, and that’s the big thing for me. It would be very difficult to get this result from a random sample if the death rate had not risen.

Hi,
I was curious about something. Is the “cluster sampling method” anything like a Monte Carlo technique?
I am glad you have taken the time to debunk the hack critiques that many of us have seen. Personally, if a journal like Lancet publishes a study of this nature, I would be very careful in critiquing and rubbishing it.

First let me take a minute or two for some wild enthusiastic cheering. Good job DD. It more than makes up for my disappointment when you attacked Steve Landsburg not on his libertarian sins, but for that paper on quantum game theory.

Okay, two comments. First, why can’t major news organizations hire people like you to explain studies like this? (Consider that to be part of the cheering). Second, why can’t they manage to spend more than thirty seconds mentioning them? This story was a one or day wonder in the American press–one story, usually containing some dismissive remarks, and then that’s all the time we had for investigating the utterly irrelevant story about whether we’ve made things worse in Iraq.

Yet let me say this: many people found this study bogus for the simple reason that extrapolating large numbers from small numbers seems to almost always be pure guesswork.

George, this is the “mere random sample”:https://www.crookedtimber.org/archives/002677.html objection, and it’s not well-founded.

More generally, I’d agree with Daniel that most of the objections to the study either misunderstand the methods or just assert that because it doesn’t conform to an ideal design it can therefore be dismissed out of hand as garbage. This kind of objection is common in the first year of graduate social science programs, where people who have yet to do any empirical research find it very, very easy to trash the work of others who have. Later they tend to realise that the available methods really are very powerful, all things considered, and that even a modest empirical paper takes a lot of work to pull off. The fact that these guys did work like this under the circumstances they faced is really remarkable. Of course it’s not the last word on the subject, but it’s a real contribution with a striking finding. To write it off with malformed (or stock) objections is sheer ideology.

A reasonable piece about an unreasonable subject, though rather than “flypaper” I tend to see it as just another episode of “when innumerates attack”.

Fumento has been fun, though. I shall admit to writing an insulting letter to the editor in response to his column being syndicated by my local paper. One guess what happened when I CCd him.

Add another critique to the study – the model the researchers fit allowed for an increase (and only an increase, based on their write-up – they never explicitly write down their model) in the death rate after the invasion. Thus the lowest estimate for excess death they could come up with would be 0 – no chance for a decrease in deaths. I don’t have enough experience with survival analysis to know about how this will effect the size of confidence bands, but it is suspect. It’s also another example of the authors’ bias.

Mike, that’s simply not true. They did explain what their model was and it did allow for a decrease. They calculated a relative risk ratio associated with the intervention (that’s their model) and if that risk ratio had been less than 1 (it wasn’t) then this would have corresponded to a falling death rate. They in fact did observe a falling death rate in the Kurdish provinces and reported it.

I think George is in for a shock when he finds out how economic indicators here in America (e.g. unemployment) are calculated, since the extrapolations from data are just as dramatic.

But it should be obvious this kind of thing is basically sound. We regard it as a news story when opinion polls (which extrapolate from 1000 voters to 100 million) are off by 3 or 4%. If extrapolation was as bad as people are claiming here, getting within 10% would be miraculous, which it pretty clearly isn’t. If this study is off by 3 or 4% in its mean figures, or even 10 or 15%, that means the Iraq war has led (so far at least) to waaaay more deaths than it has prevented.

“…people have been lining up to take a pop at it despite being manifestly not in possession of the baseline level of knowledge needed to understand what they’re talking about.”

Gosh, that makes it like most blog comments, alas. It frequently drives me crazy, to be sure.

To take a random pop, the stuff early in this post about “mines” makes me wonder the relevance, given that an anti-tank mine is set off by magnetic field, or weight, and “tossing a rock” is irrelevant. So this, for one, seems to be evoking some sort of unreal idea of a “mine,” that then, when an example purporting to represent the real world, uses such a non-existent sense of a “mine,” makes me think whatever proceeds makes no sense, given the premise on a non-existent device in reality. Anti-personnel mines are even trickier in their variability, but given their imaginary use in this theoretical example, I’m uninterested in further pursuing the fantasy of an accurate conclusion. Any argument based upon things that don’t exist, though people popularly imagine them, is suspect, in my book.

Shannon, this objection is dealt with in my piece and (with much greater thoroughness) at Chris Lightfoot’s site. UNICEF did not “report” the 2002 rate as anything. They extrapolated a 2002 rate based on the 1999 research. And from then on in, you are citing Richard Garfield as an authority in an argument against Richard Garfield. You’re also exaggerating the extent to which a mortality rate of 29 is an outlier; given the very wide confidence intervals on the estimation process as a whole (and the substantial bias toward underestimation, which, as a “service” to your own readers you have consistently failed to acknowledge), it strikes me that a 1999 rate of about 100, a guesstimated fall of 40% and a confidence interval of +/- 20 would include the sampled figure of 29 at its lower end.

I see that you are no longer defending the clustering critique, for which thanks, but you seem to now be advancing a much less intellectually respectable critique that “Occam’s Razor” tells us that there are strange murky flaws in the methodology which you can’t identify but nevertheless insist are there. For which, not thanks. I see you are also continuing to attribute the worst of motives to people who produce results that you disagree with; you are presumably doing this because you are judging the behaviour of others by your own standards.

But it should be obvious this kind of thing is basically sound. We regard it as a news story when opinion polls (which extrapolate from 1000 voters to 100 million) are off by 3 or 4%

Exactly, and you never see people attempting to rubbish polls especially when the numbers give pretexts for other ideological conclusions. Last year’s EU survey about European opinions on the war in Iraq and about Israel was a good example of that.
Or the “moral values” election poll. Everyone has draw all sorts of conclusions but I haven’t seen anyone disputing actual numbers.

People do think there is a lot of coverage about Iraq because there’s always something in the news about it, but how much of it has been about civilian deaths? So, because they don’t hear or see it on tv, it must not exist. Civilian deaths in Iraq, oh, it must be some foggy concept that can be easily put aside. (Possibly with “anyway, Saddam would have killed more” – now that’s a scientific method the Lancet authors should learn something from!).

The combination of attacks on Lancet is also so self-contradictory. I remember the IBC website was considered as reliable as Indymedia and now it’s become more authoritative than a peer-reviewed medical journal.

I don’t even understand the argument about UNICEF infant mortality rates presumably proving the Lancet study is all wrong.

How can the criticism simultaneously be that Lancet underestimates those rates _therefore_ it must overestimate the war casualties?

For me the big negative about the Lancet survey is they had to do it in the first place – the Geneva Convention insists that occupying forces keeps track of these figures, of course there is little chance of the US doing this – rather they just assume that everyone blow up by a bomb was probably an insurgent anyway.

Oh, one other brief comment. Chris Williams pointed me to a poll done by the International Republican Institute which found (see the “Power Point” presentation linked at the bottom of that page) that 22% of Iraqis answered “yes” to the question,

“In the past year and a half, has your household been directly affected by violence in terms of death, handicap, or significant monetary loss? (Close family member, up to 4th degree)”

Now, suppose that there are ten people in an average “close” family group; then these figures imply that about 2% of individuals have been “directly affected by violence” (because each person who is killed or injured or whatever is known to about ten others). That gives a raw number of about 600,000 victims; we can’t, of course, tell how many of those were killed, how many injured and so forth (or who did the killing and injuring).

But the important point about this is that, whatever the difficulties of conducting an opinion poll in Iraq and the imprecision of the number which comes out, it gives an independent estimate of the number of casualties — and that estimate is of about the same magnitude as Roberts et al.’s estimate.

Chris: average size of a nonextended family in Iraq is six, according to IBC, so your numbers look pretty good.

Gary – anti-tank mines are now set off by magnetic triggers? Last I heard, it was a pressure plate or a tilt rod.

I think that you’re thinking of anti-ship mines.

Fred Kaplan of Slate suggested that because the confidence interval was very wide, the Lancet paper was worthless and we should believe something else like the IBC total

Does he cite a confidence interval for the Iraq Body Count? Because if not I have a hint for him: just because they don’t give one, it doesn’t mean the confidence interval is small or zero.

To their credit, the IBC people emphasise that the best their figure can do is set a minimum, and that it is likely that it is a gross underestimate.

Daniel, I’m not statistically educated, and the Lancet site doesn’t seem to allow a non-subscriber to read the actual study, so excuse me if these questions are addressed in the study or are simply ignorant.

1) Does the study distinguish between combatants and civilians (I can’t tell for sure from the posts and comments here)? During the first Gulf War, the US estimated Iraqi military deaths at 100,000. Is it reasonable to believe that Iraqi combat deaths from the current war would have been comparable or even higher? I.e., is it possible that the figure of 100,000 deaths reported by the Lancet is indeed under-reported but consists largely of combatants?

2) The CIA Factbook reports Iraqi population at 25.3 million and a death rate of 5.66%, which I think suggests that about 150,000 people might die in Iraq in an ordinary year. Does this mean that 100,000 excess deaths represents a 2/3 jump in the death rate (I may be misusing figures, so please correct me)? It strikes me that even a well-functioning city would have trouble finding morgue and grave space to deal with such a number of additional bodies, and so it seems that, under the chaotic wartime conditions in Iraq’s cities, this amount of excess deaths would have resulted in heaps of bodies or mass graves. If so, I would think that America’s opponents (whether the Baathists, Zarqawi, or whomever) would have every incentive to film these corpses and smuggle out the video much as they do with the beheadings. Obviously, this is not a critique of the study but simply uninformed intuition. Am I thoroughly off base, though; is it reasonable to conclude that these excess deaths were more or less absorbed by the Iraqi infrastructure?

I’m just asking because these questions occurred to me; I don’t mean to be combative toward the study or your defenses thereof, or to suggest that any number of deaths should be minimized or dismissed.

Well, obvious question is: If you get blown up while waiting in line to join the Iraqi police, or gunned down execution style on your way home from boot camp, why is it OUR fault?

Yes, the case can be made that we ought to be willing to accept more casualties on our side, in an effort to reduce civilian casualties. (Good luck selling that idea to the American people, by the way!) But the fact is, we already liberated Iraq, and now we’re fighting a force that’s trying to enslave the country again, and not only are THEY responsible directly for a lot of the deaths Lancet is complaining of, they’re indirectly responsible for all of them, because there wouldn’t be a war if not for their agression.

You know Brett, what a great argument!!!! If only the US capilulated to Osama 3000 people would still be alive. Obviously its the fault of the US! If the Israelis woudl just leave, they wouldn’t suffer terrorist attacks. Every death is their own fault! If only native americans surrendered even faster, they wouldn’t have been as fully wiped out!

Question for Brett,

If you hit a hornet’s nest with a stick, are you not at least partially responsible for the increase in stings?

We knew going in that there would be an insurgency, there was inadequate planning for it, and obviously the administration considered the losses that would come from an insurgency to be at an acceptable level. So even if a significant number of those deaths were due to the actions of insurgents, it wouldn’t show that the administration bore no responsibility for them.

Perhaps more important is that regardless of whether the cause of death was the bullet or bomb of an American or an insurgent, the numbers matter for determining whether this war satisfies the macroproportionality condition on standard versions of the just war theory. It is starting to look like they don’t. Even those of us on the left gracious enough to play along and imagine that there was somewhere in the shifting sands a just cause for this war and to set aside speculation about the purity of the President’s motives can cite the Lancet number as solid evidence that this war quite simply doesn’t pass the tests imposed by traditional just war theory.

That’s a very nice and clear way of putting it; it explains the “Outlier Theory” – this would be the case in which a 2.5% tail event had happened and the study had significantly overestimated.

Dsquared,

You are ignoring the dynamic issues, and discussing the study as if it were a cross-sectional (point in time) analysis.

Other studies which use this technique don’t push it nearly as hard, and apoligize for it more.

For example, Grein et al, conduct a very similar study, but it is superior because:
1. The recall period is shorter (less than half).
2. They are careful to inquire about those entering and leaving the household.

Also, their reference period is clearer, and I’d guess that in an urbanized society like Iraq, people enter and leave the household more often (schooling, military service, hospitals, etc.)than in the Angolan refugee camps studied by Grein et al.

Nonetheless, Grein et al acknowledge (referring to the same method used by the Lancet study) that “the WHO/EPI sampling technique (originally conceived for immunisation and nutrition surveys) has not been fully validated as a tool to estimate mortality and alternative methods have been suggested.”

This is the same point I’m making: the sampling technique is suited for cross-sectional studies, but not so well suited for longitudinal studies. This is especially the case for studies that are sloppy about including people before they entered or after they’ve left the household, as in the Lancet study.

To speak in jargon, you should take the issue of informative censoring much more seriously.

Dsquared,

The Lancet study does discuss the issue of people entering and leaving the household. As I argued in comments on your previous post, their discussion makes clear that they are doing it wrong.

The Lancet authors are basically attempting to exclude household members who leave the household, rather than include them, which is what other studies that use this technique do. This is very important, because there are many reasons to think that leaving the household could be associated with death.

The disclaimer in the BMJ article I cited isn’t about cluster sampling, it’s really about the definition of the sample universe. I’m prepared to believe that this was reasonably clear in the BMJ study, but not the Lancet study.

It isn’t well-defined who is supposed to be included in the Lancet study’s universe. This is an invitation to overcount the kinds of deaths the Lancet authors were looking for. This isn’t lying. The surveyors in the field may well have thought that by counting violent deaths to people whose membership in the study universe was unclear, they were collecting complete information and doing a good job.

And the death certificates don’t validate anything, since they were only collected from a small, non-random, subset of the respondents.

“In other words, many of the 100,000 may have perished anyway when the Ba’ath regime (“naturally”) crumbled, however long down the road.”

And if they didn’t die then, they’d have died eventually, so it’s all good, eh?

“how many lives is freedom from political repression worth?”

Depends if it’s my life or not, don’t it? (Well, it does to me).

This was getting to be a long comment, so I split it up. Both parts are important to what I am trying to say.

Part 1:

Brian W and John Q use the example of opinion polls to demonstrate how accurate statistical sampling can be. But that comparison only reinforces my skepticism, since in fact opinion polls are almost never right. Look at this table of various polls taken the week before the American election: http://www.realclearpolitics.com/polls.html. All are (I presume) reputable and statistically sound, and all purport to measure the same thing, yet there’s a spread of 4% between the highest and the lowest. In state polls, the spreads were higher. In other words, it’s not news if an opinion poll is 3% to 4% off – it’s news if it’s accurate. And this is in a population that pollsters know abundantly well; how much more difficult must accuracy be in a population we do not know well, and that has in fact recently been through enormous change?

Granted, this is not necessarily a critique of the study itself, since I gather that that uncertainty is reflected in the wide confidence interval. But then I fall back on something like the Fred Kaplan argument, which, despite Daniel’s thorough and probably accurate criticism, nevertheless seems to hint at something true: when the actual results might be anywhere in such a wide range, does the study really say anything useful for judging whether the war was a good idea or not?

And if they didn’t die then, they’d have died eventually, so it’s all good, eh?

Actually, no. We’re talking about excess deaths. Natural death would occur irrespective of the counter-factual.

Depends if it’s my life or not, don’t it? (Well, it does to me).

True, I could reframe the question to capture that effect. Suppose the US was overtaken by a nasty tyrant, and X amount of lives from the population, chosen randomly, were required to remove said tyrant via revolution. Suppose you are among the population from which the sample is chosen. How small would X have to be for you to support revolution?

Pacifist Response: 0
Pro-war nut response: 300 million

Anyway I don’t want to veer this thread off its discussion of the Lancet. My point was that the humanitarian case has taken a serious blow from the Lancet, but it’s not out the window just yet.

D’oh! Despite proofreading, I made a big goof in my arithmetic. The period of time between 1985 and the invasion was 18 years, not 9, so cut all my “status quo” numbers in half. Not that it derails my basic argument, but talk about innumerate…

Thank you. Well done. The Lancet piece is indeed a good piece of science.

Replicate this study 5 times and you’ll likely get close to a good number.

Best,

D

I discuss the confidence interval issue in my ZNet piece: 100,000 Iraqis Dead: Should We Believe It?
http://www.zmag.org/content/showarticle.cfm?SectionID=15&ItemID=6565
What I conclude is that a major reason the CIs are so large is that the authors engaged in a conservtive (in a statistical sense) analysis of their data and included the Kurdish region. In this region, there was a reversal of the pattern in the rst of the country: more deaths prior to invasion than after. If this region had been excluded, the CIs would have been narrower, and the point estimate (98,000) higher.

The fact that the authors did not exclude the Kurdish region is one piece of evidence that they were NOT seeking to maximize the estimate of casualties. This type of thing is one of the things we researchers look for when judging the quality of another’s study. This study was one of the best I’ve seen. Still, given its discrepancy with the estimates of others (Iraq Body Count, Project for Defense Alternatives), I tend to believe the “true” estimate is somewhat below 98,000, but still in the many tens of thousands.

Ragout,
Do you find the results of this survey to any great extent surprising? The IBC numbers plus the number of combatant deaths would give you a low ball at best rather less than an order of magnitude smaller than the Lancet numbers.

If they were really only after the sensation value they could have given estimates including the Falluja figures.

At best you are going to find holes in what is already known to be a net. Whatever its shortcomings it is the best figure we have to go on and it has been peer reviewed by a reputable journal. It doesn’t say anything very surprising beyond crystalising the effects of a gradually developing situation. I’m not sure why you think its burden of proof should be so high or that it fails to meet it.

Why do you think there are no official figures? Surely this number, whatever its true value is an important performance indicator? In any case you will convince more people if you can provide a persuasive case for a significantly different number rather than just trying to find ways this number might not be right.

I’m not positive, but I don’t think this article addressed the strongest arguement agains the Lancet study: the numbers are utterly ridiculous.
For 100,000 deaths to have occurred since the end of the war, that would be an average of around 175 a day (I don’t know the exact number, but its been discussed since the study came out). The 100,000 may have included additional infant mortality, and may have included additional ‘natural’ deaths that weren’t occurring under Saddam, but as I recall, it specifically said that ‘most’ deaths were occurring in violence, and ‘most’ of those were due to American (presumably unintentional) weapons. So at most 49,999 were additional infant mortality figures, (leaves ‘most’ due to violence), and at most 24,999 were non-American weapons (leaving ‘most’ due to Americans) So of 180 deahts per day, you’ve got 90 or so deaths per day due to violence, and 45 or so deaths per day due to American violence. 90 (or 45) per day, every day, for the last year and a half. Thus, in the last year and a half, when you have read in the papers about a bombing that killed 70 people, you were actually reading about a slow day! The news media has been writing headlines about the days when violence was actually lower than the norm! 100 people would have had to have been killed for a normal run of the mill day, I guess.
So maybe you should address the “these numbers are absurd” statistical test.

Steve

In light of this I’m still not sure where I stand on the humanitarian case for war. If Iraq is successful in 10 years, any ex-post justification of the war will still require pondering the question: how many lives is freedom from political repression worth?

A fair question, but it would be a better one if the case for war had actually rested on humanitarian grounds. It only really did ex post facto when the WMD and “terrorist links” arguments went into the crapper.

I’m not positive, but I don’t think this article addressed the strongest arguement agains the Lancet study: the numbers are utterly ridiculous.

That’s not the stronget argument against the study. If anything, it’s the weakest.

As Marc Mulholland has pointed out, 175 deaths/day is about ten times worse than the worst year in Northern Ireland. This doesn’t seem all that unreasonable, given that the British Army was not in the habit of calling in airstrikes on the Falls Road.

Forget the media reports. There is saturation coverage of the green zone of Baghdad. There is no good information about the dangerous bits of Iraq. As Tim points out above, journalists aren’t allowed to visit the morgue.

I’m not saying the 100K number is gospel; as I hope I’ve been consistent in saying, I don’t like linear extrapolation. But the entire problem is that we have literally no information of sufficient quality to allow us to gainsay it.

‘But the entire problem is that we have literally no information of sufficient quality to allow us to gainsay it.’
…Although we might well have had such information had Allawi not ordered his Ministry of Health to stop publishing the death figures they were collecting from the hospitals.

There are several clear problems with the Lancet report that have not been adaquately addressed by this blogger. I would like to address two. They are the infant mortality rate, and the clear, overwhelming overestimation of the violent deaths in the Fallujah area.

First, let me consider the infant mortality rate. Quoting the report:

First, the preconflict infant mortality rate (29 deaths per 1000 livebirths) we recorded is
similar to estimates from neighbouring countries.

The use of neighboring countries appears reasonable…given the tacit assumption that no better comparisons are available from Iraq itself.

However, as the initial article here mentioned, there was a study, sponsored by UNICEF. It is available at:

http://www.unicef.org/newsline/99pr29.htm

Some relevant numbers are:

24,000 households in the South/Center of Iraq (85% of the population, Arabs)
16,000 households in the North of Iraq (15% of the population, Kurds)

were randomly sampled.

This is far greater than the Lancet survey, which surveyed, roughly, 1000 households.

Second, the infant mortality rate was measured at:

108 per 1000 live births in the South/Center
not given in the North but said to have fallen like the under 5 mortality rate

The under 5 mortality rate was

138 per 1000 live births in the South/Center
72 per 1000 live births in North.

So, we have an under-5 mortality rate of 12.8%, and an estimated infant mortality rate of 10% for the entire country in 1999. This contrasts with the remembered estimate of 29 per 1000 live births pre-invasion in the Lancet study.

It is considered good technique to address previous literature in the field. This study stands out as important relevant literature. Yet, it was ignored. If these numbers were correct, then one would expect a death rate of 130/5 per year for children under 5. With infant mortality being the majority of this, we can estimate that 1006 under 5 year olds seen correspond to at least 1100 live births in the last 5 years. (This gives a birth rate that is slightly lower than the pre-invasion birth rate 226/year given in the Lancet study.

Using these UN figures, we would expect that there would be 32 under 5 deaths during the pre-invasion interval. The Lancet report gives only 12 deaths under 15…and doesn’t break it down by under 5 and over 5. This difference of 20 deaths over 14.6 months is absolutely critical. With that age cohort representing 39% of the total population, it also represents a difference in the under 15 contribution to the mortality rate of 6.4 deaths/1000. With an Iraq population of, approximately, 25 million, this translates to over 125,000 deaths per year…more than the 98,000 given in the Lancet.

Given this, the authors of the Lancet study have a duty to mention the UN study and then explain why they think the death toll showed an unprecedented decline between ’99 (Feb to May) and ’02. (Jan 02, to mid-Mar 03) AFAIK, this decline far surpasses any other three-year decline.

Since the author of the blog states that the start of the oil for food program is the cause of the improvement, let me address that. There are a couple of problems with this: first the oil for food program started in ’96. With infant mortality, in particular, it is hard to see why it would not have had an effect on infants born two years later. Second, during that time, the oil- for-food program was raided by Hussein for other purposes. One would need to provide evidence that this stopped in ’98 and ’99 in order to argue that the infant mortality rate fell tremendously.

The second suspect number is the Fallujah death toll in 8-04 and 9-04. Reading the graph, I obtain 33 deaths in two months in Fallujah. Given that the total sample rate is, roughly, 1 in every 3200 Iraqis, this can be extrapolated to more than 100,000 deaths in this area during those two months.

The city only has a population of 250,000. Since civilians were allowed to escape, why would they stay when people were dying at that rate? Further, how would it go unnoticed? As mentioned in the Lancet study, deaths must be taken care of quickly in Arab countries. Wouldn’t 100,000 requests for death certificates, 100,000 funerals, be noticed. Further, since there are usually >1 wounded per death in military actions of this type, wouldn’t nearly everyone else be wounded? Wouldn’t the hospitals and morgues be more than overwhelmed? Wouldn’t someone notice the smell of the rotting bodies, since there wouldn’t be enough able-bodied people to take care of the dead?

Yet, the authors simply glide over this point…stating that there was a low possibility that this outlier was improperly sanding Even decent technique would require this type of cross checking between sampling and more direct measurement.

Given these two examples of bad technique; given contrary data that appears to be much more solid, I don’t think the Lancet article can be taken seriously. My question is what sort of peer review goes on at the Lancet. It might be the same folks who reviewed Sokal’s paper. :-)

Dan M.

I searched for “infant mortality” and “UNICEF”, and obtained one reference to that study: it is below:

and the last estimate of under-five mortality was from a UNICEFsponsored demographic survey from 1999.11,12

So, it is true, that it was mentioned. But, it appears that you and I differ on what properly adressing a previous study, with far better statistics is. For example, they also quoted numbers on infant mortality. That wasn’t mentioned. So, it was only briefly mentioned in the introduction, and not considered as relevant data when the infant mortality rate was discussed.

Second, the consideration of the Fallujah data was poor. Their data indicates that 100k died in two months where there was minimal US activity in Fallujah. Even if you only take half of that, its still 50k.

Cluster sampling is a technique that is fraught with difficulties. Done perfectly well, it works. But, if there are _any_ indications that there are serious problems with the techniques, a serious researcher is oblidged to deal with them directly. What they did, instead, is just say it was an outlier, and gave results with and without it.

A scientist is supposed to search for every possible way his data could be wrong before publishing. Cross checking with other data sources should be done, whenever possible. A reasonable scientist, would have cross checked the Fallujah data, and known that there was a serious problem with the sampling. The authors wrote as though it was a minor, or modest problem at most.

So, to be accurate, I shouldn’t say they didn’t address these two issues; rather I should say that they didn’t adress them as one would expect in a serious piece of work.

If I implied the former, I apologize.

One other thing, it is possible that I miscalculated a number, since I don’t perform the same due diligance in posts as I do in professional papers that I write. If you can catch an important numerical error that I made, I’d appreciate you letting me know what my mistake was.

Dan M.

It’s hard to get access to the original Lancet study at present so am unable to look to see how the variance was calculated. Was it a binomial variance?

Daniel,

I wasn’t being critical of the lack of distinction between combat and civilian deaths. I can understand why it wasn’t done, and I was asking just to educate myself.

When I mentioned combatants, though, I wasn’t being clear; I principally had in mind the Iraqi army that opposed the US forces prior to the fall of Baghdad (whose status as combatants is clearer), rather than the insurgents. By referring to the first Gulf War, I meant to be asking whether, since US forces killed 100,000 of those troops in taking Kuwait, is it reasonable to believe that US forces would have killed some similar (or higher) number of Iraqi army soldiers in taking Iraq?

Note that this is question is not meant as an attack on the study. Rather, I’m suggesting that our experience with the first Gulf War supports the 100,000 figure.

Also, I’m not trying to say that 100,000 deaths is OK if and only if they were primarily Iraqi army soldiers. I do think it raises slightly different moral issues, but obviously, just as with the first Gulf War, one still has to ask how many deaths are too many.

(I’ve reposted [and slightly amended, hopefully making myself clearer] these musings from another blog thread which has sadly reduced to arguments pro/contra the war, a subject in which I long ago lost interest.)

I’m no statistician, but it seems to me that:

(a) 24.4 million (the population of Iraq) divided by 100,000 (the Lancet authors’ non-Fallujah estimate of excess deaths in the 18 months since the invasion) = 244; therefore 1 in 244 Iraqis has died, by this estimate, thanks to the war – about 4 in every thousand, 40 in every ten thousand;

(b) 7868 (the number of living Iraqi individuals in the Lancet’s sample) divided by 244 = 32, which implies that the figure of 100,000 deaths is extrapolated from 32 excess deaths discovered by the Lancet.

This appears to accord neatly with the fairly detailed Economist article at: http://economist.com/science/displaystory.cfm?story_id=3352814, which states that

Of the increase in deaths (omitting Fallujah) reported by the study, roughly 60% is due directly to violence, while the rest is due to a slight increase in accidents, disease and infant mortality.

21 is the number given for non-Fallujah violent deaths in Table 2 of the paper, and 21 is of course “roughly” 60 percent of 32.

If so, then “100,000” is the number of deaths one would discover in Iraq if the Lancet’s sample of 7868 happened to be perfectly representative of excess deaths in the country as a whole. In other words, it assumes that the 7868 were a perfectly random sample, and that the phenomenon of excess deaths due to the war was uniformly distributed among the population (the entire Iraqi nation). How confident (in the normal usage of the term) can anyone be that this “perfect-world” number of 100,000 is, in fact, the “most likely” number of excess deaths country-wide?

From what I understand of the issues involved, there’s nothing controversial about the survey’s falling short of perfection: this is to be expected from the fact that it used cluster sampling – as would any other survey of its kind – and that clusters differed from each other to some significant degree (we don’t know by exactly how much, because the bar charts in Figure 1 relate to Governorates, some of which had several clusters each), and the survey’s size and design explains the width of the confidence interval (CI) ranging from 8,000 to 194,000 excess deaths. But the CI deals with the purely random errors arising from the size and mathematical design of the survey and (unless I am mistaken about the authors’ methods) the inter-cluster results obtained: it doesn’t address measurement or sampling errors due to poor survey design (if any). In that respect the CI is “unintelligent” – it knows nothing about Kurdish areas or Sunni or the way in which the fighting was distributed, nor does it compensate for “recall bias,” “respondents lying,” or what have you.

So I’m at a loss to explain the confidence (again, in the normal usage of the term) of the Lancet authors in the “perfect” 100,000 (or 98,000) number which lies midway between the endpoints of the CI. Surely it should never have been promulgated as enthusiastically as it has been. I respect the position taken by dsquared (that the interval is more interesting, in that it shows an indisputably elevated relative risk of death post-war, especially from violence), but for good or ill it’s with “100,000 excess deaths” that the Lancet study is now associated, and this is down to no-one but the authors, who have promoted this number from the beginning.

Sorry about the duplicate post – please delete at will.

Heiko, coalition claims that they try to avoid civilian deaths don’t mean much when they then go on to use artillery, helicopters, and 500 lb bombs in cities. There is some evidence (apart from this Lancet study) that the coalition forces are killing more civilians than the insurgents–Nancy Youssef reported this in a Knight-Ridder story last Sept 25. (I mentioned that in an earlier thread to which I never returned, so I don’t know if anyone commented on it.)

There was also a story in the October 12 issue of the NYT (I think I have the date right) where a Pentagon official told the Times that they saw a good side to civilian casualties in Fallujah, because it might make the inhabitants have to chose between supporting the insurgents and supporting the coalition forces. Presumably they were supposed to chose the Americans dropping the bombs on their town. I suspect that this story gives us a glimpse of the real thinking that lies behind the use of heavy weapons in urban areas, but official government spokesmen aren’t going to be so naively open about it. Americans are extremely good at doublethink–we are perfectly capable of dropping bombs on cities and seeing the useful side of collateral damage while sincerely proclaiming our desire not to see any civilians killed.

Jack,

The numbers I find fairly plausible are the figures for the post-war violent death rate. I think the authors’ definition of a household is less problematic over shorter time periods.

They don’t actually seem to report the violent death rate specifically, but some calculations show it to be 1.8 per 1000 (6 per 1000 with Falluja). Which translates to 44,000 per year (152,000 with Falluja). So, they’re pretty big numbers.

In the US, very high murder rates are something like 0.7 per 1000 (in places like Detroit in the early 1990s). So living in Iraq today seems to be something like living in the worst neighborhoods of Detroit, DC, or New Orleans during a high crime year.

So, even though I don’t believe the study’s pre-war figures, I think these are pretty depressing and believable numbers.

What a shame that this study has become a symbol of pro-war vs anti-war (which I guess is largely because of the sensationalist headlines and the timing of its publishing). Imagine if it had been conducted at the request of the coalition forces. It would appear as a limited but worthwhile attempt to measure how well they were achieving one of their objectives (improve the lot of the Iraqi people) under difficult conditions and with admirable speed. Instead it has become a battle-ground in its own right.

Let me clarify a few things. Even if we assume that something is seriously wrong with the sampling (eg because prisons were excluded), the numbers for violent deaths are pretty overwhelming, 1 before to 21 after (ex Fallujah, and the sampling period and population size aren’t quite the same, so make that 1.4 to 21 or so).

Saddam wasn’t involved in mass killings at the time of the invasion, as he had been at previous times in Iraq’s history, and there was little fighting going on. Another study has been mentioned that instead looked at torture, and Shiites in the South did report that this did go on at similar levels as earlier in the period just before the invasion.

And, it appears, that the Lancet study does not count deaths in prisons, which would be where most of Saddam’s victims would have died in the time period in question (estimates of between 2000 and 10000 deaths per year in the pre-war period from direct killing in prisons by Saddam seem realistic to me).

Still, clearly the violent death rate has gone up substantially.

Secondly, the reason this study is getting so much flak is that its results were presented to the press in such a terribly politicised fashion. 100,000 innocent civilians, mostly women and children, killed by coalition forces was the message widely perceived as having been released to just be timed before the election.

And I must say that the authors, the Lancet and the press have gotten pretty little flak on this from the study’s defenders.

It’s the presentation of their findings that’s getting people like Worstall to immediately dismiss them.

However, the findings themselves are hardly surprising, or, where they are strong, add much to what we already know anyway, namely that violent death has increased substantially.

The findings shed relatively little fresh light on the question of attributing the deaths, are most killed either terrorists / those killed by terrorists, or are most of those killed innocent civilian victims of coalition bombing?

The Lancet study also appears very weak on the question of infant mortality (it finds little change in non-violent adult mortality).

Unicef presents data for both 2002 (102) and 2001 (107), WHO even gives confidence intervals for their numbers of under five mortality in Iraq in 2002.

Daniel wonders why one wouldn’t suspect that other numbers are undercounted, if infant mortality is.

On the other hand, if under five mortality in 2002 in Iraq was 131, undercounting this is nearly a prerequisite for being able to claim substantial increases. The Lancet study found a near doubling for infant mortality. If that applied to a true pre-war figure of over a hundred, between 1 in 5 and 1 in 4 of Iraq’s infants would have died after the war.

An infant mortality of 29 before the invasion, if true, would suggest that neither sanctions nor Saddam’s regime were any longer responsible for excess deaths compared to other countries. And it would be an unbelievably sharp fall, and a record low that had never been approached before by Iraq, and only achieved by neighbouring countries over a decade or more of improvement.

It matters enormously, what true infant mortality was pre-war. Only if it really was above a 100, there is enormous scope for improvement, and either people who believed sanctions to be the problem, or who layed the blame on Saddam, had a much greater humanitarian case.

If infant mortality was no different than in neighbouring countries, either argument (for lifting sanction or for removing Saddam Hussein) becomes much weaker.

I have now written to Unicef. It is vital that they publish good figures. And that means when they say 2002, they mean 2002, not 1994-1999, or say 102 and the true number is 29.

That’s just not acceptable, if they mean 1994-1999 they should say exactly that, and if they are so unsure about the number that it could be anything between 25 and 250, they should indicate that clearly as well, rather than list 102. Some error is ok, but not over a factor 3. That renders the number meaningless.

And we aren’t only talking Iraq here. Unicef and WHO publish numbers for all countries. I’d find it rather worrying, if those might (or might not be without them bothering to mention that fact) wild guesses based on outdated data.

GKL wrote:

So I’m at a loss to explain the confidence (again, in the normal usage of the term) of the Lancet authors in the “perfect” 100,000 (or 98,000) number which lies midway between the endpoints of the CI

The authors have no such confidence; they have repeatedly said that they believe that the central estimate is an underestimate.

Let me start with an area that I think we agree on. The Fallujah numbers are clearly wrong. They are not merely an outlier, they are not merely suspect, they have been falsified by other observations.

No, we don’t agree on that. In particular, I think that your reason for believing this about the Fallujah numbers (excerpted below) is, to be frank, nonsense.

The second suspect number is the Fallujah death toll in 8-04 and 9-04. Reading the graph, I obtain 33 deaths in two months in Fallujah. Given that the total sample rate is, roughly, 1 in every 3200 Iraqis, this can be extrapolated to more than 100,000 deaths in this area during those two months.

What you’ve done is taken the Fallujah death rate, multiplied it by the sampling rate for the whole of Iraq, and attributed it to Fallujah. This isn’t right. What you should have done would be to multiply it by the local sampling rate. We don’t have this, but we do know that there was one cluster in Fallujah, and that a cluster for this study is 33 households. So in other words, the Fallujah study found roughly one death per household in the immediate postwar period. That’s a hell of a lot, but it’s not at all impossible as an outlier in a heavily bombed neighbourhood.

Daniel wrote:

I think that the simple fact that we can say with 97.5% confidence that the war has made things worse rather than better…

You have just demonstrated that this is not a “simple fact”!

You go on and on at length showing how the central estimate could be an underestimate, which I am willing to concede here.

But having just demonstrated that the nonsampling error has shifted the central estimate lower, you still accept a 97.5% probability that war made things worse, even though that’s based on no underestimation–and only accounts for sampling variability.

If you really think 98K really is an underestimate, isn’t 97.5% too low?

If, in the face of your evidence of underestimation, you still want to stick to 97.5%, then aren’t you assuming that think there’s an equal possibility that nonsampling errors made 98K an overestimate?

You don’t seem to believe that in your writing above.

Either way, 97.5% is one very important piece of evidence, but not really a “simple fact”.

Nevertheless, thank you for your insight on this matter.

D^2 suggested that I used suspect technique in my analysis. In particular, he suggests we don’t know the sample size. It wasn’t given directly, that’s true. But, the authors were kind enough to give death rates with and without Fallujah, as well as the number of post invasion deaths in Fallujah. Thus, we have a direct means of checking my observations.

“During the period before the invasion, from Jan 1,
2002, to March 18, 2003, the interviewed households had
275 births and 46 death The crude mortality rate was
5·0 per 1000 people per year…The crude mortality rate during the period
of war and occupation was 12·3 per 1000 people per year…If the
Falluja cluster is excluded, the post-attack mortality is 7·9 per
1000 people per year”

and

“When included, we estimate that the rate of death increased
2·5-fold after the invasion (relative risk 2·5 [95% CI
1·6–4·2]) compared with before the war. When Falluja
was excluded, we estimated the relative risk of death for
the rest of the country was 1·5 (95% CI 1·1–2·3).”

Now, in the first quote, there is some ambiguity as to whether methods changed from including the invasion time to excluding it in the two surveys. Since the ratios for the two quotes are equivalent to within rounding, I’ll assume the same technique is used.

We also have 1.48 years post invasion. Just doing arithmetic, I obtained the following deaths per 1000 for the whole invasion period

All Iraq: 18.2
Fallujah: 6.5
Without Fallujah 11.7
All Iraq, no increase 7.4

Multiplying by the population of Iraq, (24.4 million) we obtain:

All Iraq: 445k
Fallujah: 159k
Without Fallujah 286k
All Iraq, no increase 181k

By these calculations, we would get 105k excess, instead of 98k. This is consistent with the rounding errors in the numbers I used, so that’s not too bad. The paper also states that 53 were killed in Fallujah during the post-invasion period. Since 33 were killed in Aug-Sep. ’04, the authors would extrapolate (to within rounding) 33/53 * 159k during that period. That comes to 99k.

We can even normalize down for my rounding error, and get 92k. So, as you pointed out, my 100k estimate wasn’t exactly the same as what the authors estimated. Still, it’s close.

In short, even though I’m just a humble aging, plumber, :-) I can still look at numbers in a paper and see what conclusions are obtained. Typically, I calculate “back of the envelope” numbers using the author’s technique in my head to check out the author’s work. Now, that I’ve spent a bit more time on making direct calculations from the authors’ numbers; I’d argue that my estimate was close enough to prove my point.

Dan M.

FWIW – some anecdotal evidence: Farnaz Fassihi, Middle East correspondent for the WSJ writes in her personal email:

…
Iraqis like to call this mess ‘the situation.’ When asked ‘how are thing?’ they reply: ‘the situation is very bad.”

What they mean by situation is this: the Iraqi government doesn’t control most Iraqi cities, there are several car bombs going off each day around the country killing and injuring scores of innocent people, the country’s roads are becoming impassable and littered by hundreds of landmines and explosive devices aimed to kill American soldiers, there are assassinations, kidnappings and beheadings. The situation, basically, means a raging barbaric guerilla war. In four days, 110 people died and over 300 got injured in Baghdad alone. The numbers are so shocking that the ministry of health — which was attempting an exercise of public transparency by releasing the numbers — has now stopped disclosing them.
[…]
I heard an educated Iraqi say today that if Saddam Hussein were allowed to run for elections he would get the majority of the vote.
…

There is an interesting correlation between two of the critiques Daniel itemizes here: the inconsistency of the infant mortality data may tend to support the Lying Iraqis theory.

Back in 2002, DD and I had a brief exchange in which I pointed out (1) that _pre-war_ Iraqi infant mortality numbers were controversial, (2) that the UNICEF estimate was much higher than the CIA’s, (3) that the CIA estimate was much easier to reconcile with total Iraqi population numbers 1990-2001, and (4) that the mortality rate Saddam was attributing to the sanctions regime was implausibly high based on UNICEF numbers, and absurdly high based on CIA numbers. See original link at

http://www.j-bradford-delong.net/movable_type/archives/000908.html .

Anyway, two years later we have the same numbers, but everyone looking at them has changed sides. The Lancet study finds current infant mortality rates that are supposed to have _doubled_ since the war, but remain far _below_ UNICEF’s prewar estimate…and almost exactly in line with the CIA’s.

It seems to me that one model which explains all the observed facts is that Iraqi infant mortality has fallen slowly from 60+ per mil to 50+ over the last 15 years, without much impact from sanctions or war. In this model, the CIA and Lancet have always been right in the present tense, and UNICEF and Lancet have both been misled by anecdote in the past tense, in each case in the direction the Lying Iraqis hypothesis would favor. That is, during the sanctions regime, UNICEF investigators on the ground were told about fake deaths (or real deaths of cousins resampled as siblings, or other bias), while today Lancet investigators are losing half the _real_ deaths from before 2003.

Since the Iraqi “story” used to favor a high mortality rate under the sanctions regime, and now favors a low past mortality rate for the same sample period, the facts that UNICEF heard 110 per mil, and that Lancet now hears 29, cannot be viewed in isolation from each other. Rather, they begin to make a fair Bayesian case for serial deception.

mr Holsclaw, that the USA never even tried to have a fiable and public statistic of deaths and other mishaps that happend on the Iraqis is enough to negate their innocence. Let’s be excessive: the Nazi thought they were right and so documented most of their crimes. That made Nurenberg processes that much easier. Their modal (not moral, after all Nazis claimed to do what was right) offsprings of nowadays, like Milosevic, take care that no official trace can come up to them. The USA are doing the same, so while soldiers can believe their indoctrination over fighting terrorism and terrorists, the officialdom knows it is a lie.

DSW

Thanks for your analysis of the paper. It’s helped me clear up at least a couple of the reservations I had about the methodology. However, there is a point that concerns me which I didn’t see addressed, and I wonder if you could help me with it.

At both the province level and the locality level, decisions on where to sample appear to have been made based on the available prewar census data. But during war, there are frequently population dislocations, and these dislocations are non-random. Whenever possible, people relocate from high-risk areas to safer areas. If this has been the case in Iraq, wouldn’t using the prewar census data carry a significant risk of resulting in the oversampling of high-risk areas and the undersampling of low-risk areas? And couldn’t that, in turn, result in an elevated estimate of the nationwide mortality?

Thanks

Mike: In principle, you might be onto something here; they certainly did use unadjusted January 2003 population estimates (I think that these were estimates rather than census data) for calculating the sampling scheme, and on the face of it, I’d say that this would tend to oversample people who’d stayed in the dangerous zones (although it would by the same token undersample households who had fled the scene after losing a family member). Note that households would have to move from one governorate to another in order to affect the chances of being sampled; my guess is that this would mean that Baghdad would be inflated relative to rural areas, as this is the only town where I would guess there were lots of people who had family to stay with in another governorate.

On the other hand, I think I would guess that, given the extraordinary consistency of the story across the non-Kurdish governorates, this effect might not be too large in magnitude. But I can’t rule out a larger effect; this is a bloody good spot on your part. This article is about to drop off the front page, but I’ll keep in touch by email if I think of anything else.