Post-Invasion Deaths in Iraq

by Kieran Healy on January 10, 2008

A new study estimates violence-related mortality in Iraq between 2003 and 2006:

Background Estimates of the death toll in Iraq from the time of the U.S.-led invasion in March 2003 until June 2006 have ranged from 47,668 (from the Iraq Body Count) to 601,027 (from a national survey). Results from the Iraq Family Health Survey (IFHS), which was conducted in 2006 and 2007, provide new evidence on mortality in Iraq.

Methods The IFHS is a nationally representative survey of 9345 households that collected information on deaths in the household since June 2001. We used multiple methods for estimating the level of underreporting and compared reported rates of death with those from other sources.

Results Interviewers visited 89.4% of 1086 household clusters during the study period; the household response rate was 96.2%. From January 2002 through June 2006, there were 1325 reported deaths. After adjustment for missing clusters, the overall rate of death per 1000 person-years was 5.31 (95% confidence interval [CI], 4.89 to 5.77); the estimated rate of violence-related death was 1.09 (95% CI, 0.81 to 1.50). When underreporting was taken into account, the rate of violence-related death was estimated to be 1.67 (95% uncertainty range, 1.24 to 2.30). This rate translates into an estimated number of violent deaths of 151,000 (95% uncertainty range, 104,000 to 223,000) from March 2003 through June 2006.

Conclusions Violence is a leading cause of death for Iraqi adults and was the main cause of death in men between the ages of 15 and 59 years during the first 3 years after the 2003 invasion. Although the estimated range is substantially lower than a recent survey-based estimate, it nonetheless points to a massive death toll, only one of the many health and human consequences of an ongoing humanitarian crisis.

150,000 violent deaths in three years is a lot. You’ll recall that the _Lancet_ study estimated about 655,000 excess deaths, which is a lot more. The two numbers aren’t directly comparable because excess deaths due to violence are only one component of all excess deaths (e.g., from preventable disease or other causes attributable to the war). Deaths due to violence rose from a very small 0.1 per 1000 person years in the pre-invasion period to about 1.1 per 1000py afterwards, or 1.67 adjusting for estimated underreporting. This is where the authors get their 151,000 number. The overall death rate rose from about 3.2 per 1000 person years to about 6, an increase of just over 2.8. Depending on whether you use the raw or adjusted estimated rate of violent death this would work out to an overall excess death total of just under 400,000 or just over 250,000. (But this is just a back-of-the-envelope calculation, as the overall death rate isn’t reported.)

The discussion section questions the _Lancet_ result while emphasizing how difficult this kind of work is, given the appalling circumstances:

Recall of deaths in household surveys with very few exceptions suffer from underreporting of deaths. None of the methods to assess the level of underreporting provide a clear indication of the numbers of deaths missed in the IFHS. All methods presented here have shortcomings and can suggest only that as many as 50% of violent deaths may have gone unreported. Household migration affects not only the reporting of deaths but also the accuracy of sampling and computation of national rates of death.

The IFHS results for trends and distribution of deaths according to province are consistent with what has been reported from the scanning of press reports for civilian casualties through the Iraq Body Count project. The estimated number of deaths in the IFHS is about three times as high as that reported by the Iraq Body Count. Both sources indicate that the 2006 study by Burnham et al. considerably overestimated the number of violent deaths. For instance, to reach the 925 violent deaths per day reported by Burnham et al. for June 2005 through June 2006, as many as 87% of violent deaths would have been missed in the IFHS and more than 90% in the Iraq Body Count. This level of underreporting is highly improbable, given the internal and external consistency of the data and the much larger sample size and quality-control measures taken in the implementation of the IFHS.

At present, there are no better methods available to provide more accurate estimates of the death toll due to the humanitarian conflict in Iraq in the wake of the 2003 invasion. Rapid small-scale surveys of households are likely to yield unreliable estimates. Surveys of a large number of respondents with carefully prepared household interviews and multiple methods for collecting data on mortality still run into reporting problems because of the insecurity, instability, and migration associated with the conflict situation. The clustering of violent deaths may further affect uncertainty related to sampling, even though more than 1000 clusters were selected for the IFHS. It is unlikely that more accurate estimates of the death toll during the post-invasion period can be obtained by conducting more household surveys with recall questions on mortality. On the basis of press reports, the Iraq Body Count is also affected by considerable underreporting but is likely to be a valuable way to monitor trends over time. Further investment in such mechanisms is justified, especially if ways can be found to assess the level of underreporting and the consistency of the reporting mechanisms over time. Other methods, such as systematic reporting by mortuaries and hospitals and the strengthening of vital registrations with the use of sentinel sites, will also need to be explored.

Here at CT, Daniel was the one most involved in defending the _Lancet_ study against its detractors, and he’s well able to speak for himself. But a word is probably in order to those on a mea culpa watch. A study like this gives us good reason to substantially revise our estimate of the total number of excess deaths downward. The Burhnam et al estimate of excess deaths looks like it was too high, assuming that the new survey is basically reliable. It’s good that the IBC effort and Burnham et al have been supplemented by new work. (Again, though, the 151,000 number is not an estimate of excess deaths.)

All of this is separate from the question of whether many or most of the reasons offered by earlier critics of the _Lancet_ study were any good. Those who just said “this number just seems too high, I don’t believe it and want more data,” and left it at that, look a lot better than those who showed themselves ignorant of the methods used to calculate the estimates even as they tried to undermine them. The latter group should bear in mind that essentially the same cluster-sampling methods are used in the new study as the old, and the new survey was subject to many of the same constraints in accessing violent regions of the country. Those who simply floated accusations of fraud without any independent evidence at all look as bad as ever. Latching onto the new number just because it’s pleasingly lower doesn’t make any more sense than rejecting the old number because it was unpleasantly high for you.

More importantly, as the paper’s discussion makes clear, the main challenge facing those doing this sort of research is that there is a war going on, and wars kill a lot of people, bring about the dissolution of households, and compel very large numbers of people to flee the region. All of this makes the machinery of statistical science rather difficult to apply. None of the available numbers look any good, both on their own and given what they imply about what’s happening in Iraqi society. If you find yourself really delighted that a war of choice has resulted in the deaths of a population the size of Jersey City, or maybe Oakland, instead of one the size of Baltimore, you probably need to rethink your priorities.

{ 1 trackback }

It’s the count, not the numbers, that counts « Butterflies and Guns
01.18.08 at 1:08 pm

{ 215 comments }

1

christian h. 01.10.08 at 9:19 pm

There’s a good piece on this at Lenin’s Tomb, with links to (among others) a comment on this by Roberts. The discrepancy here, of course, should be resolved through scientific debate; and likely, it will be in the scientific community. Not in the public eye, though – our ruling class media will make sure of that. As Len writes, they can defend the murder of 151,000 people more easily than they can defend the murder of 600,000.

2

Justin 01.10.08 at 9:20 pm

Legitimite question – since the data in this report is a year and a half old, is there any way of extrapolating to determine the likely excess deaths throw the end of 2007?

3

Justin 01.10.08 at 9:32 pm

BTW, some comparables:

Vietnam War ~ 1.7 million deaths
Khmer Rouge ~ 1.65 million deaths
Rwandan genocide ~ 800,000 deaths
Iran-Iraq war ~ 500,000 deaths
Vietnam post-war ~ 430,000 deaths
Kurdistan ~ 300,000 deaths (includes Kurds in Anfel)
Sudan genocide ~ 200,000 deaths (since 2003)
Bosnia genocide ~ 200,000 deaths
Iraq under Sudan Hussein (1979-2003)
~ 120,000 (excluding Kurds in Anfel)
~ 300,000 (total)

4

dsquared 01.10.08 at 9:43 pm

I must confess I don’t understand Tyler’s point. Another study doesn’t make any of the old methodological criticisms valid, and doesn’t make it OK to accuse people of politically motivated fraud without evidence. (I’d also note that 11% of the clusters selected for this study were too dangerous to visit and the data was adjusted from ratios derived from the IBC estimates alone. The authors think they’ve corrected for this bias but they might not have).

5

Hidari 01.10.08 at 9:46 pm

Yes and isn’t it interesting that this report is being reported (even by the ‘liberal’ press) as a complete demolition of the two Lancet studies, whereas of course it could be interpreted as totally demolishing the figures of the (Western media based) Irag Body Count?

The other problem with comparing the situation in Iraq with the figures quoted by Justin above is that all those wars are now finished. The situation in Iraq is clearly not, and, equally clearly, will still be being fought for years to come. These figures ain’t going to go nowhere except up.

6

dsquared 01.10.08 at 9:47 pm

also I think Les Roberts puts it rather simply: “We reported the death rate went up 2.5 fold, the Iraqi government now claims that it only doubled”. I’ve said from day one that the risk ratio is the important number rather than the product of that ratio and the population of Iraq, and looking at it in those terms very much demonstrates the extent to which these two surveys corroborate one another.

7

Chris Bertram 01.10.08 at 9:50 pm

Justin’s list of “comparables” isn’t much use here, because he gives no indication if those are “Iraqi Body Count”-type numbers, total violent deaths, total excess deaths. Probably, some of those totals are one thing, and some of the others, another.

Marko Attila Hoare, who knows his stuff on Bosnia, has an “analysis”:http://greatersurbiton.wordpress.com/2008/01/04/what-do-the-figures-for-the-bosnian-war-dead-tell-us/ of an IBC-type number of 97K for Bosnia (unfortunately, he litters it with some rather dreary political asides, but it is otherwise a helpful piece.)

FWIW, I’d be surprised if the total excess deaths figure for Iraq isn’t substantially higher than the comparable figure for Bosnia, simply because the infrastructural collapse in Iraq seems much more severe, and the the impact on things like childhood mortality would be consequently greater.

UPDATE: Actually, this comment of mine is, on reflection pretty stupid. I just looked up the population of Bosnia and the population of Iraq (4 and 27 million respectively). If something like 97k (IBC-style) and 200K (total) is a credible figure for Bosnia, then much higher totals for Iraq surely suggest themselves. But I’d better bow out and leave the field to the statistically competent.

8

mpowell 01.10.08 at 9:53 pm

Lets not give the hysterics too much ground. This new report does not ask for a 3 times reduction in the excessive deaths estimated by the lancet report. The lancet report gave a total excessive death count at 650,000 and this new report estimates the total excessive deaths coming in at just over 400,000. Given the error in both studies, this is a pretty reasonable discrepancy.

Regarding the violent death discrepancy, its possible that in this new study there is more reporting bias regarding nature of death. Certainly, different people are asking different questions. And when you narrow down the studies to types of deaths, the error bounds are going to increase further. Finally, I think you’ll see the least amount of reporting bias if you just consider total deaths.

I was one who looked at the lancet reports, thought the distribution as to who was getting killed looked funny and the number a little high but saw it as a pretty clear indication that a lot of people were dying. This new data comes as some reassurance regarding both the distribution and the total. It may also illuminate some shortcomings in the lancet study, but that doesn’t mean the lancet study was a bad study- just not perfect.

9

dsquared 01.10.08 at 10:39 pm

The Burhnam et al estimate of excess deaths looks like it was two to three times too high, assuming that the new survey is basically reliable

for values of “two or three” equal to 1.5, I think – the new study has lower violent deaths but higher non-violent deaths.

10

Kieran Healy 01.10.08 at 10:44 pm

Yeah this is a mistake, it should be 1.5 to 2 (depending on the which rate of violent death you pick). I’ll fix it.

11

Donald Johnson 01.10.08 at 11:13 pm

Justin, what is the basis for the numbers killed in Iraq under Saddam? I’ve seen some of them before and incidentally, the estimates I’ve seen just for the Anfal campaign vary from 50,000 to 180,000.

My point being that people should be a little cautious about all the commonly reported numbers one sees in the press, unless you happen to know what method was used to derive them.

12

Donald Johnson 01.10.08 at 11:19 pm

“Recall of deaths in household surveys with very few exceptions suffer from underreporting of deaths. None of the methods to assess the level of underreporting provide a clear indication of the numbers of deaths missed in the IFHS. All methods presented here have shortcomings and can suggest only that as many as 50% of violent deaths may have gone unreported. Household migration affects not only the reporting of deaths but also the accuracy of sampling and computation of national rates of death.”

I asked this over at Deltoid and am too impatient to see if anyone will answer it there–what does this mean? Are the authors of the paper saying that their own violent death estimate might be off by a factor of two because of unreported deaths, or did they take that into account in their CI? (I’m not sure how you’d include uncounted deaths in a CI, but I’m a layperson and one who hasn’t read the paper yet.)

13

roger 01.10.08 at 11:22 pm

After the Lancet report, there was a flurry of reports all upping the total from the IBC number. This is one of them. The trend is towards the Lancet numbers, although depending on the details of this one, it might be towards the lower bound. I hope there is another Deltoid discussion once the study is published.

14

Donald Johnson 01.10.08 at 11:43 pm

Apparently they did take into account the possible underreporting–


On the basis of simulations that took into account survey sampling errors and estimated probable uncertainty in the adjustment factors for missing clusters, in the level of underreporting, and in projected population figures, we estimated that there were 151,000 violent deaths in Iraq during this period (95% uncertainty range, 104,000 to 223,000).”

Sounds odd to the non-expert, but I’ll take their word for it. What about the 2 million who’ve left the country–couldn’t they be people whose families suffered a larger-than-average violent death rate?

15

michael 01.10.08 at 11:49 pm

“The two numbers aren’t directly comparable because excess deaths due to violence are only one component of all excess deaths (e.g., from preventable disease or other causes attributable to the war).”

One of the more mystifying points of the Lancet study is that the reported number of excess deaths due to non-violent causes went DOWN, and the number that study reported was almost exclusively the result of violent deaths. An AP summary of the 2006 Lancet study says “Their conclusion, based on interviews of households and not a body count, is that about 600,000 died from violence, mostly gunfire.” So, while you’re correct the new 150k number is not comparable to the 660k number. It is comparable to the Lancet’s claim of 600k. So there goes Kieran’s primary point above.

This lead to one of the better critisms of the earlier study. While in most warzones violent deaths certainly go up, the vast majority of excess deaths are non-violent in nature, the result of indirect effects of war– inturruptions in food and clean water supply, decreases in sanitation, lack of medicines and access to medical care, etc. While it is possible that the US invasion actually led to an improvement in these services, while provoking extreme levels of violence, that type of result is unprecedented in modern warfare, and highly unlikely.

This current study provides a number that is more in line with measurements of Iraqi deaths through other methodologies, uses better staticstics (more clusters, etc.), and provides results more in line with historical patterns in other conflicts. The boosters the the earlier Lancet report DO owe everyone a mea culpa.

16

John Emerson 01.11.08 at 12:49 am

I want David Kane.

17

John Emerson 01.11.08 at 12:50 am

Abstract: “While this new study definitively proves that the Lancet study was fraudulent, it too is fraudulent.”

18

christian h. 01.11.08 at 12:56 am

As for the discrepancy in violent deaths reported, one should note that the WHO study was done by employees of the Iraqi government. It is quite possible that household would not want to report deaths as violent to government officials.

The fact that Western media reported a much higher proportion of deaths in Bosnia than in Iraq is not surprising, I’d say: journalists weren’t targets in Bosnia; the Western media and governments had a vested interested in reporting high numbers of deaths, whereas in Iraq they don’t (or is this a “dreary political aside”?).

19

SG 01.11.08 at 1:11 am

Donald, I think they got their final figure of 151000 (and the attendant rate ratio) from doing a series of computer simulations making various assumptions about the things they didn’t know. So they made assumptions about the level of underreporting, from memory they gave it a proportionate value and a normal distribution. But they pepper their paper with warnings about exactly how dire and dubious those assumptions are, and make very clear that their pre-war death rate seems very low (70% lower than neighbouring countries for Women!) Presumably their assumptions in this regard have the biggest effect on that estimate of violent deaths. But then again it could be that the underreported pre-war death rate balances the under-reported post-war death rate, and the really sensitive assumption in the simulations is the death rate in the missing clusters. I bet there’s a very interesting sensitivity analysis floating around somewhere about that, but it would not be the sort of document one would put up on the NEJM website, I presume.

My guess would be that the migration figure would represent a type of non-random missing data, since the people who migrated would be those who were most likely to suffer high death rates. This type of missing data is a big problem in these studies unless it is a very small proportion, and I don’t think 2/24 is so small. I can’t think how (theoretically) a lower population remaining affects estimates, but I have vague memories that it is only negligible, but no doubt the theory assumes that missing data is random. I would guess this type of question can only be answered by surveying refugees and doing simulations.

20

Michael 01.11.08 at 1:19 am

“The two numbers aren’t directly comparable because excess deaths due to violence are only one component of all excess deaths (e.g., from preventable disease or other causes attributable to the war).”

One of the more mystifying points of the Lancet study is that the reported number of excess deaths for most types of non-violent causes went down, and the number that study reported was almost exclusively the result of violent deaths. An AP summary of the 2006 Lancet study says “Their conclusion, based on interviews of households and not a body count, is that about 600,000 died from violence, mostly gunfire.” So, while Kieran is that correct the new 150k number is not comparable to the Lancet’s 660k number. It is comparable to the claim of 600k. So there goes his primary point above.

This leads to one of the better critisms of the earlier study. While in most warzones violent deaths certainly go up, the vast majority of excess deaths are non-violent in nature, the result of indirect effects of war– interruptions in food and clean water supply, decreases in sanitation, lack of medicines and access to medical care, etc. While it is possible that the US invasion actually led to an improvement in civil services, while simultaneously provoking extreme levels of violence, that type of result is unprecedented in modern warfare, and highly unlikely.

This current study provides a number that is more in line with measurements of Iraqi deaths through other methodologies, uses better statistics (more clusters, etc.), and provides results more in line with historical patterns in other conflicts. The boosters the the earlier Lancet report do owe everyone a mea culpa.

21

Barry 01.11.08 at 1:42 am

“Sounds odd to the non-expert, but I’ll take their word for it. What about the 2 million who’ve left the country—couldn’t they be people whose families suffered a larger-than-average violent death rate?”
Posted by Donald Johnson

That’s what I’d bet.

22

Barry 01.11.08 at 1:49 am

One of the interesting things I’ve noticed is that the critics don’t seem to know that there have been other surveys.

James Wimberly, over at Mark Kleiman’s Blog ‘The Reality-Based Community’ had a summary up back in September, comparing four surveys and the IBC project results.

23

scott cunningham 01.11.08 at 1:53 am

Does the new data contain any information about the demographic characteristics of the people who have died since the invasion? For instance, have men been more likely to die than women? Also, how long before a census can be conducted in a region like this? Obviously not now, but what about in other countries? How long, for instance, before other countries were able to get census surveys into the field, after a war?

24

Donald Johnson 01.11.08 at 2:14 am

Thanks for the responses.

One other thing. From memory, this new study seems to be in reasonably close agreement with Lancet1 if you look at the same time period. Lancet1’s midrange violent death toll was about 60,000 (excluding the Fallujah outlier), which for 17 months is about 120 per day. Which is essentially identical to the violent death rates in this study for the same time period–the paper says

” On the basis of population estimates shown in Table 2 of the Supplementary Appendix, the IFHS data indicate that every day 128 persons died from violence from March 2003 through April 2004, 115 from May 2004 through May 2005…”

If there hadn’t been a Lancet 2 paper, we’d all be talking about how this confirmed Lancet1 and how silly the critics of that first paper (including Iraq Body Count) look.

25

SG 01.11.08 at 2:16 am

good point Donald, but you need to bear in mind that Lancet 1 has been shown incontrovertibly to be a fraud, by the time honoured scientific method of yelling loudly, and italicizing occasional points.

So I think this means this paper must be a fraud too…

26

Kieran Healy 01.11.08 at 2:17 am

My guess would be that the migration figure would represent a type of non-random missing data, since the people who migrated would be those who were most likely to suffer high death rates.

Yes. It’s not that people who answer forget that someone died (though even then they may not in fact report it, for various reasons including who is asking). The problem is more that deaths can lead to the dissolution of households, which are then not available for sampling, or to the migration of households, which again takes them out of the sample frame in a way that’s systematically correlated with the outcome of interest. This messes up your estimates and your confidence in them.

27

Ragout 01.11.08 at 2:22 am

In other Iraq mortality news, the National Journal recently quoted Richard Garfield as saying, “The study in 2006 suffered because Les was running for Congress and wasn’t directly supervising the work as he had done in 2004.”

Richard Garfield is a Columbia University epidemiologist and coauthor of the first Lancet study. I was pretty surprised when he wrote a letter to the Lancet criticizing the second Lancet study. I was even more surprised to read in the National Journal that he was originally a coauthor of the second study, but asked that his name be removed from the article.

So yeah, I think some mea culpas are in order.

28

Barry 01.11.08 at 2:38 am

“…read in the National Journal …”

Anybody citing the National Journal does owe a mea culpa.

29

Ragout 01.11.08 at 2:43 am

Wow, now the Lancet defenders are accusing the National Journal of making up quotes? This just gets more and more ridiculous. Or maybe Barry is just confusing the National Journal with the National Review?

30

joseph duemer 01.11.08 at 3:03 am

An Event in the Capital

Imagine an ordinary person –
a middle-aged man, nondescript –
walking along a street in the capital.
Imagine an improvised bomb
exploding behind him, just far enough
that he is only knocked down,
falling forward onto his hands & knees.
He hears the splatter of gunfire
& sirens converging on the wreckage,
but the man gets up & continues
to the market, dusting himself off
as he walks. He is not surprised.
Returning home in the afternoon,
he feels his neck stiffening,
a headache forming behind his eyes
& despite the heat lies down
to rest. Later, in the early evening
he is awakened by someone
at the door. His daughter-in-law’s voice.
Her friend who has walked her here
retreats, ashamed to be marked with
news of the man’s son killed
that morning in the Street of Fountains.
It is only then that the man –
ordinary, middle-aged, nondescript,
sweaty from sleeping in the heat –
notices that his ears are still ringing
from the explosion earlier
in the Street of Fountains. With his arm
carefully around her shoulders,
he takes her into the kitchen, then goes
to draw a basin of water so
they can wash their faces, both now
beyond all forms of government.

31

rodeo 01.11.08 at 3:17 am

These stats are not much of a use except for being just stats– most likely to be perceived/lambasted by the mainstream/Hitchens types as “unscientific”, “moral idiocy”, “controversial findings” etc.

The Lancet’s Slant by Hitchens
http://www.slate.com/id/2151607/

32

David Kane 01.11.08 at 3:30 am

Did someone call for me? Hello Crooked Timber! I have been banned from Henry and dsquared’s threads, but, since Kieran has not banned me, I assume he does not object to my contribution. Kieran writes:

A study like this gives us good reason to substantially revise our estimate of the total number of excess deaths downward. The Burhnam et al estimate of excess deaths looks like it was too high, assuming that the new survey is basically reliable.

Exactly correct! Last year, my estimates were:

If I had to bet, I would provide much wider confidence intervals than either the Lancet authors or most of their critics. Burnham et al. (2006) estimate 650,000 “excess deaths” since the start of the war with a 95% confidence interval of 400,000 to 950,000. My own estimate would center around 300,000 and range from 0 to 1.2 million. Obviously, no one is really interested in my estimate — derived as it is from reading the literature and associate debates — but I thought it reasonable to be upfront about my prior beliefs.

In retrospect, I should have placed more weight on the informed estimates of people like Jon Pedersen. He estimated violent deaths at 100,000 (1/6th of the L2 estimate) and that sure matched up nicely with the 150,000 from IFHS. So, my new estimate is 150,000 (at first glance, this new paper seems much better than L1 or L2) with a confidence interval of 0 to 500,000.

It would be interesting to read the current estimates of folks like Kieran and dsquared. Note that I wrote a month ago:

Where is the debate going? I sometimes worry that, like so many other left/right disputes, this will never be resolved, that we will never be sure whether or not the Lancet articles were fraudulent. Will these estimates be the Chambers/Hiss debate of the 21st century? I hope not. Fortunately, other scientists are hard at work on the topic, reanalyzing the data produced in L2 and conducting new surveys. Both critics and supporters of the Lancet results should be prepared to update their estimates in the face of this new evidence. If independent scientists publish results that are similar to those of the Lancet authors, then I will recant my criticism. Will Lancet supporters like Lambert and Davies do the same when the results go against their beliefs? I have my doubts.

I should not have doubted Kieran’s willingness to update his estimates. My apologies! dsquared, on the other hand, is acting about how I suspected. Is there any new information that would cause him to doubt the L2 results?

Because more stuff is coming! What’s most interesting about IFHS is how they went out of their way to attack L2. They didn’t need to do that. They could have been much nicer. They could have spun the story as Roberts and dsquared would like to. Instead, they go for the jugular, as much as you can in the NEJM. They highlight how their confidence interval rejects the L2 range by more much more than 100,000 deaths. They don’t just argue that they are right. They argue that L2 is very, very wrong.

Who is right? Time will tell. Did everyone catch how Horton was shoving the L2 authors off the sled in the National Journal article?

Today, the journal’s editor tacitly concedes discomfort with the Iraqi death estimates. “Anything [the authors] can do to strengthen the credibility of the Lancet paper,” Horton told NJ, “would be very welcome.” If clear evidence of misconduct is presented to The Lancet, “we would be happy to go ask the authors and the institution for an official inquiry, and we would then abide by the conclusion of that inquiry.”

Hardly a ringing endorsement! Perhaps Richard Horton knows/suspects that something is not right with the L2 data . . .

Where is this going? The wheels of science grind slowly, but they grind very fine indeed. If the data underlying L1/L2 is fake, then the Lancet papers will be the most important scientific fraud of the decade. Think that is impossible? Think again.

What can the Crooked Timber community do? Act like scholars and scientists. (As Kieran does in this post.) Keep an open mind. Consider all the evidence. Look at the underlying data. Study the statistical models. Replicate the results. Make our findings public.

One small step would be for dsquared to allow me to publish comments in his threads on this topic. But perhaps open discussion and debate is not what he is looking for.

33

SG 01.11.08 at 3:30 am

Perhaps you should read more carefully Ragout. The conclusion of Garfield’s letter is this:

Without an explanation for the high availability of death certificates, one could assume that the reporting error is of the same size as the sampling error (±30%). This assumption still yields at least a five-fold higher number of violent deaths than the passive surveillance mortality numbers. If the death certificates are valid and the availability above 90%, it seems better to monitor mortality by compiling data from the local agencies that issue these certificates than by doing further dangerous household surveys.

i.e. he thinks that even if there are major reporting errors in the death certificates, the violent death rate is still 5 times the IBC rate, and he advocates suspension of further dangerous surveys if the death certificates are accurate. It’s hardly a call for a mea culpa.

34

Ragout 01.11.08 at 3:46 am

SG,

Yes, Garfield has been careful to say that mortality in Iraq went up a lot after the invasion, just not nearly as much as found in the second Lancet study.

When a coauthor disassociates himself from a study and writes a letter alleging “substantial reporting error” in his former coauthors’ study, that’s an awfully big mea culpa.

35

Justin 01.11.08 at 4:11 am

Sorry, here is the source I was using – though my own knowledge and history led me to adjust the numbers a little from this website, since their use of the median isn’t very scientific.

http://users.erols.com/mwhite28/warstat2.htm

36

Donald Johnson 01.11.08 at 4:31 am

Thanks Justin. I like that guy’s website and have recommended it myself to a friend once–I much prefer it to Rummel. But one thing this controversy has hammered into me is that one should take death toll estimates with a grain of salt, unless you know how they were calculated. One thing that drives me nuts is when someone contrasts Iraq Body Count numbers with vague claims that Saddam caused the death of millions. Not that you did that, but others have.

37

David Kane 01.11.08 at 5:12 am

Those who simply floated accusations of fraud without any independent evidence at all look as bad as ever.

How so? A year ago, I thought that there was a 50% (or whatever) chance of fraud. You thought that there was a 1% chance (or some small non-zero number). We now have a second study that disagrees dramatically with L2. Doesn’t that we mean that we should update our estimates of the probability of fraud? (Certainly, if this study had come out with a figure closer to 600,000, I would have updated my probability of fraud downward.)

Consider the extreme case. Another survey comes out with a 100,000 — 200,000 confidence interval. And then another study. And then another one. Each written by different teams, using different methods. After enough cases, isn’t it fairly clear that the truth is much more likely to be 150,000 than 600,000. And, once we agree on that, it becomes much more likely that the reason that L2 came up with 600,000 is that the data was faked? Or incompetently gathered? What other reason could their be for their sample being so different than all the other studies?

38

nick s 01.11.08 at 5:23 am

Doesn’t that we mean that we should update our estimates of the probability of fraud?

Only once you give this new study your patented analysis technique: i.e. hint at fraud, do some bullshit calculations, then go running to Michelle Malkin and any other receptive innumerates.

After all, what study is to be taken seriously now until it receives the Kane imprimatur? Please, outline the magic confidence interval methodology you plan to employ on this new study. Or is there, as we suspect, a 99% chance that you won’t subject this one to the same ‘rigorous’ ‘oversight’?

(I’d actually think #32 a bad parody of Wavy Davy: it doesn’t have enough exclamation marks.)

39

Tim Lambert 01.11.08 at 5:30 am

Yes, ragout, the National Journal misrepresented Garfield. I know this because I asked him. Didn’t you read my post?

40

Steve Sailer 01.11.08 at 7:06 am

As I blogged in 2006 about the Lancet study, the violence level in Iraq was so awful that it was unlikely that the study could have been carried out as randomly as described:

“Or it could be that the interviewers got in contact ahead of time with neighborhood leaders to see if their presence would be welcome to reduce their chances of being killed. (That’s not good random surveying hygiene, but are you going to blame them?) Then, in a neighborhood where the local big shot wanted their presence, he might have passed the word around to aggrieved families to get ready to tell their stories to the interviewers when they showed up. This could cause a bias upward in the number of deaths reported.

“The more I think about the mechanics of carrying out the survey on the street without getting killed, the more I suspect that the Iraqi interviewers didn’t actually implement the purely random survey design that the American professors from MIT and Johns Hopkins dreamed up for them. It would be nuts to to let luck determine which streets you’d choose, as the report claims they did. You’d want to only go where you knew you’d be safe. Then you’d tell the Americans you did exactly what they told you to do.”

I have not looked closely at the new study to see if it managed to overcome these problems.

41

SG 01.11.08 at 7:09 am

And, once we agree on that, it becomes much more likely that the reason that L2 came up with 600,000 is that the data was faked? Or incompetently gathered? What other reason could their be for their sample being so different than all the other studies?

This kind of thing really is nasty, David. You’re still braying about fraud with your only evidence being that some other paper used the same methodology and got a different result. Is this how we’re to do science now? Everytime a paper gets a different result, all the others must have been fraudulent?

42

lenin 01.11.08 at 7:49 am

Sorry to come in on this a bit late, but there’s a bigger problem with the 200,000 figure for the Bosnia ‘genocide’ than Chris Bertram suggests. This is that the figure is for the total deaths during the war, on all sides, and even that is likely to be an overstatement. The total deaths, civilian and military, on all sides, are now estimated at approximately 100,000.

To my knowledge, in this war, one episode has hitherto been characterised by the US-sponsored ICTY as genocide and that is the Srebrenica massacre. If people wish to characterise that as genocidal, I won’t object, although I think it’s mistaken to inflate terms in this fashion. However, the characterisation of the whole war simplistically as a genocidal process is an overhang of Western propaganda. Further, if it’s offered in comparison with Iraq, I would maintain that the Bosnian Serb leadership did not bear sole and exclusive responsibility for the war in the same way that the US bears sole and exclusive responsibility for the war on Iraq. This is far worse than anything that happened in the former Yugoslavia.

43

Chris Bertram 01.11.08 at 8:10 am

Lenin, I didn’t suggest that there was a problem with the 200,000 figure for Bosnia. As for the points your make about “on all sides” and “civilian and military”, you could do worse than look at Hoare’s breakdown of the 97k ibc-style estimate. Hoare’s post contains, his blog usually does, some rather annoying political sideswipes in boilerplate “decent” style which detract from his central point. But he does know his stuff on the facts.

[We shouldn’t, though, divert this thread into a discussion of Bosnia.]

44

Ray 01.11.08 at 8:15 am

David – “After enough cases, isn’t it fairly clear that the truth is much more likely to be 150,000 than 600,000.”

This is the kind of thing that convinces me David Kane is just another hack. The 150,000 is a figure for violent deaths, the 600,000 is a figure for excess deaths (as stated repeatedly above). Skating over that difference to say “150k not 600k! 600k must be a fraud!” is a pretty obvious sign of dishonesty.

45

lenin 01.11.08 at 8:42 am

I appreciate that you don’t wish to sidetrack the discussion, but Hoare’s discussion refers to this finding:

83.33% of civilian deaths in the Bosnian war were Muslims

Those are the figures for “killed and missing” Bosniaks given in the RDC’s ‘IBC-style’ (actually, it is not an IBC-style study at all, since it is a meticulous record of all known dead and missing not a compilation of deaths reported in two separate English-language media services), but this is the first I’ve seen of that particular study. It may be a minimum, but this study, which is also not an IBC-style study but an estimate, would suggest that it’s close to the toal – yet it breaks the ratio of deaths down differently. Of the approx. 100,000 deaths reported in Bosnia and Herzegovina from 1992-1995, it reports 55,261 civilian deaths, of which 38,000 were Muslims and Croats, and 16,700 Serbs. Of the dead troops, 28,000 were Muslims, 14,000 Serbs and some 6,000 Croats. Among the Muslims and Croats, some of the victims at least are those of the two-year war between the Bosnian forces and the Croatian forces. Unless one starts with the narrative of Serb-driven genocide already in place, as the Western media had already done as the war began with the very eloquent voice of Bernard-Henri Levy conveying Izetbegovic’s desire that we should see it that way, then one alternative upshot is that the distribution of civilian deaths reflect the relative balance of forces.

46

Chris Bertram 01.11.08 at 9:35 am

Lenin, I’ve just been looking at that study. I see the 55,261 total, but I don’t see the breakdown you give.

UPDATE: That’s because the breakdown you give is not, in fact, contained in the article you link to. It may be possible, by extrapolation from the numbers in that article to arrive at such a breakdown (I haven’t tried), and it may be that the same authors have given such a breakdown elsewhere. But it isn’t there in the article. However, the exact same breakdown is given on a number of dodgy websites (e.g freerepublic), on your blog, Lenin, and on Wikipedia.

47

Borwnie 01.11.08 at 10:18 am

This is the kind of thing that convinces me David Kane is just another hack. The 150,000 is a figure for violent deaths, the 600,000 is a figure for excess deaths (as stated repeatedly above). Skating over that difference to say “150k not 600k! 600k must be a fraud!” is a pretty obvious sign of dishonesty.

As I understand it, the 150k versus 600k comparison is perfectly valid. The Lancet II assessment of violent death is 601,027. This is a like-for-like, although I accept you can’t get from there to “Lancet II must be a fraud”.

48

dsquared 01.11.08 at 10:34 am

No it’s not valid, or at least not unless you also acknowledge the existence of a big increase in the nonviolent death rate that wasn’t in the Lancet study – otherwise you’re implicitly cherry-picking the violent deaths from this one and the non-violent deaths from the Lancet and coming up with an apple/orange St. Clements’ cocktail of a total deaths number that isn’t supported by either study. The question of classification of deaths as violent or non-violent depending on who was asking was one that was raised a lot during the original discussion of Lancet 2.

49

John M 01.11.08 at 11:01 am

It seems that everyone is agreed that whatever the precise margins involved the new study is convincing evidence that the overall death toll in Iraq is significantly lower than had at first been feared by most commentators here on CT and elsewhere. I am just a tiny bit surprised that there isn’t a single expression of relief about that. Surely everyone hoped that the original Lancet estimate was wrong even if they suspected that it wasn’t? You might almost get the impression that some people around here are a bit disappointed that there aren’t more dead after all.

50

abb1 01.11.08 at 11:04 am

I don’t see much of a point in comparing the number of casualties caused by a civil war with that caused by an unprovoked aggression.

Civil war is pretty much an act of god, one could as well include influenza epidemic or tsunami. Accident statistics is one thing, criminal statistics is another.

51

Ray 01.11.08 at 11:08 am

I find it hard to work up any enthusiasm for ‘only’ 150,000 violent deaths in 3 years.

52

Borwnie 01.11.08 at 11:48 am

No it’s not valid, or at least not unless you also acknowledge the existence of a big increase in the nonviolent death rate that wasn’t in the Lancet study – otherwise you’re implicitly cherry-picking the violent deaths from this one and the non-violent deaths from the Lancet and coming up with an apple/orange St. Clements’ cocktail of a total deaths number that isn’t supported by either study.

I was responding dierctly to commenter “ray” who mistakenly assumed the 600k figure in David Kane’s comparison related to all excess death in Lancet II. This led commenter “ray” to allege “dishonesty” on the part of David. I thoguht it was worthwhile pointing out that this charge was based on “ray’s” mistaken interpretation of the 600k figure.

I don’t think I need to apologise for responding to the actual point made by the commenter I was actually adressing. The idea that I must cover all other bases in any response or stand accused of “cherry-picking” is simply preposterous.

FWIW, my take on the latest is study is mostly likely not what you imagine or infer in your previous comment. It merely confirms what a difficult business this is. Lancet II and the NEJM study can’t even agree pre-war mortality rates.

Lancet 2
Pre-invasion mortality rates = 5.5 per 1000 people per year

NEJM
Pre-invasion mortality rates = 3.17 per 1000 per year

If the Lancet estimate for pre-invasion mortality is the more accurate and substituted for the NEJM figure, the NEJM increase in mortality drops significnatly (3.17 repalced by 5.5):

NEJM
Increase in mortality rate post invasion = 6.01.3/5.5 = 1.09

Conversely, if the NEJM figure for pre-invasion mortality is nearer the truth and included in the Lancet calculation, the increase in mortality shoots up (5.5 replaced by 3.17):

Lancet II
Increase in mortality rate post invasion = 13.3/3.17 = 4.2

If two accountants both manage to reconcile the bank for a company, but one reports a couple of million adidtional debits and credits in the process, you get a qualified audit report.

Unless hard evidence comes to light of fundamental flaws in one or other of the studies, isn’t it simply the case that all bets are off?

53

Z 01.11.08 at 12:05 pm

the new study is convincing evidence that the overall death toll in Iraq is significantly lower than had at first been feared by most commentators here on CT and elsewhere

I can’t vouch for this, but doesn’t the total number of excess deaths computed by the IFHS study is in the 400,000 (less excess in violent deaths but more in non-violent deaths than in Burnham and al.)? If this is correct, then yes, I am slightly relieved that the war killed 400,000 people and not 600,000 but mostly my reaction is one of horror and disgust.

54

Ray 01.11.08 at 12:09 pm

Mea culpa (there had to be one)- the Lancet’s figure for excess deaths was 650k, not 600.
But if the Lancet says most of the excess deaths were violent, while the IFHS says that only half of the excess deaths were violent, then maybe 150 and 600 are not the numbers to look at.

55

dsquared 01.11.08 at 12:15 pm

Just to confirm that the “you” in my #48 was a generic “you” because I’ve seen this comparison made a lot of times, rather than a specific accusation at Brownie, who I will have many opportunities to insult on other matters in future and thus can afford to apologise to on this one.

John M: concern troll much? The word “significantly”, I do not think it means what you think it means.

56

Hooverism 01.11.08 at 12:18 pm

“the new study is convincing evidence that the overall death toll in Iraq is significantly lower than had at first been feared by most commentators here on CT and elsewhere”

When I read the original debate here on CT, I got the impression that many commenters were not full of fear that the numbers were so high. They seemed strangely satisfied. Far be it from me to ascribe motives, but I wondered whether they were keen to use the figures to criticise the war…

Regards,

H.

57

dsquared 01.11.08 at 12:20 pm

[in retrospect, this was a breach of CT comments policy – dd]

58

David Kane 01.11.08 at 12:20 pm

dsquared writes:

the new study has lower violent deaths but higher non-violent deaths.

Hey, dsquared! Any chance that I will be allowed to participate in your future Lancet threads? Or are you only interested in having a discussion with people who agree with you? Just asking!

And, not that I speak on behalf of the Lancet-skeptic community, but we don’t really care about non-violent deaths. The (statistically insignificant) non-violent death increase in L2 was plausible. I have never seen anyone deny this. As Michael Spagat writes:

The main problem with the comparison highlighted by the L2 authors [in asserting that L2 validates L1] is that it is of all excess deaths, not just violent deaths. All suggestions of possible bias in L2 that we know of, sampling or non-sampling, pertain to violent deaths. The available facts simply do not support a claim that L1 and L2 suggest very similar numbers of violent deaths. By persistently conflating non-violent deaths with violent deaths the L2 authors have obscured this essential point

Exactly right. Now, given all the time in the world, we could have a fun-filled discussion of what the non-violent death rate in Iraq was, is and will be. But no Lancet skeptic really cares. We just think that the violent death rates in L1 and L2 are ludicrously high and that the IFHS results support our position.

And, if you push this line, you need to explain not just why I am wrong, but why the IFHS authors are wrong. (They clearly discount the increase in non-violent deaths as being a function of reporting biases. That is, if you did the same survey in the US, you would “see” a dramatic increase in mortality because people are much better at recalling deaths in 2006 than those in 2002, whatever the cause.) Anyway, good luck arguing that the IFHS authors don’t know their what they are doing. They clearly do.

59

Daniel 01.11.08 at 12:23 pm

Any chance that I will be allowed to participate in your future Lancet threads?

No.

Or are you only interested in having a discussion with people who agree with you?

I’m interested in having a discussion with anyone, subject to a basic threshold level of honesty which, in my assessment, you do not meet.

Just asking!

Just telling.

60

Borwnie 01.11.08 at 12:27 pm

ray,

I wouldn’t necessarily disagree. My point is that even if the NEJM non-violent deaths made up the entire shortfall to match Lancet’s total excess deaths, what would this prove? We’ve arrived at the same number, yes, but via two entirely contradictory means. The violent and non-violent components in at least one study must be wrong (because both can’t be right), but we don’t know which? The two studies don’t corroborate each other just becasue the totals are similar when each of the equivalent components in the two studies are miles apart.

Sure, it could be that we simply have a death classification problem across the two studies. Of gargantuan proportions.

61

stuart 01.11.08 at 12:29 pm

Which at worst would be like the President of the US pointing at the 30k civilian deaths on IBC at one point in time to point out how ‘few’ deaths there are to try and justify his invasion, despite anyone knowing the methodology of that site realising it is a massive undercount. Either side of a political argument are always likely to promote any reasonably credible studies that support their argument. In reality the truth (or the best approximation we will ever get to it), if it is ever know, is only ever found out by historians, because these things will be over long before reliable statistics and studies can be fully carried out.

62

Daniel 01.11.08 at 12:29 pm

Kane:

They clearly discount the increase in non-violent deaths as being a function of reporting biases. That is, if you did the same survey in the US, you would “see” a dramatic increase in mortality because people are much better at recalling deaths in 2006 than those in 2002, whatever the cause

Study:

Overall mortality from nonviolent causes was about 60% as high [sic] in the post-invasion period as in the pre-invasion period. Although recall bias may contribute to the increase, since deaths before 2003 were less likely to be reported than more recent deaths, this finding warrants further analysis.

This has been episode XXV in the series “Why David Kane Is Banned”.

63

Daniel 01.11.08 at 12:32 pm

60: I think this is wrong. Whether someone’s dead or not is a fact and is verifiable; what they died of is an opinion and much more subject to obfuscation.

64

Ragout 01.11.08 at 12:36 pm

Tim Lambert,

No, your post does not demonstrate that the National Journal misrepresented Garfield, it just demonstrates that Garfield thinks that the invasion caused an awful lot of deaths. I’d be more inclined to take your word for it if you reported facts that don’t fit your conclusion, like the fact that Garfield asked for his name to be removed from the Lancet II study.

65

David Kane 01.11.08 at 12:37 pm

Good to know. With luck, Kieran will originate most of the Lancet discussion so that the CT community of readers might hear from all sides in this important debate.

And, by the way, I am still working on a reply to Kieran’s post from 2006 on my fraud discussion. My rough draft of that reply is here.

But now we have new data. I have not looked closely at the IFHS response rates, but I suspect that they went back more than once to each household, as opposed to L2’s single visit. And I suspect that they were willing to talk to any adult in the house, as opposed to L2 insisting on speaking with the head or spouse. If the IFHS response rates (especially on their first attempt) are much lower than L2, doesn’t that suggest that not everything is wonderful with L2’s 98% response rate, and especially their 99% contact rate? I think so.

That debate will play out over the coming months. Stand by for more fun.

66

David Kane 01.11.08 at 12:44 pm

#62 Huh? I claim something and you quote the part of the paper that supports my claim (and that I was thinking of) and then this is a reason for banning me. I don’t get it.

67

Marko Attila Hoare 01.11.08 at 12:48 pm

To respond to Lenin’s point above, the Iraq war is only ‘far worse than anything that happened in the former Yugoslavia’ if one conveniently blames the Americans for 100% of the deaths in Iraq, but blames the Serb forces in Bosnia only for those they actually killed.

According to the Lancet report, 31% of deaths in Iraq were the work of the Coalition forces, 24% the work of ‘others’ and 45% unknown.

http://www.thelancet.com/webfiles/images/journals/lancet/s0140673606694919.pdf

In Bosnia, according to figures of the Research and Documentation Centre, as I show in my article cited above, at least 86% of civilian deaths were the work of the Serb forces, and not more than 14% the work of Croat and Bosnian/Muslim forces combined.

http://greatersurbiton.wordpress.com/2008/01/04/what-do-the-figures-for-the-bosnian-war-dead-tell-us/

In Bosnia, 39,685 civilians were killed. In absolute terms, this is much less than the 151,000 civilians killed in Iraq according to the WHO report. As a share of its population, however, Bosnia’s losses were much higher than Iraq’s have been.

For the sake of comparison, we can compare these figures to the most scientific study of deaths in East Timor under the Indonesian occupation:

http://www.ictj.org/static/Timor.CAVR.English/07.2_Unlawful_Killings_and_Enforced_Disappearances.pdf

According to this study, of 18,600 East Timorese civilians unlawfully killed between 1974 and 1999, 70% were killed by the Indonesian security forces and its collaborators, and 30% by the East Timorese Resistance.

I’m not in the business of ranking these wars according to which was worst. But contrary to what Lenin says, there is absolutely no basis whatever for minimising what happened in Bosnia in relation to Iraq, or to East Timor.

68

Daniel 01.11.08 at 12:50 pm

Kane:

They clearly discount the increase in non-violent deaths as being a function of reporting biases

Study:

Although recall bias may contribute to the increase […] this finding warrants further analysis

Kane:

Huh? I claim something and you quote the part of the paper that supports my claim

Me:

Episode XXVI.

69

Borwnie 01.11.08 at 12:52 pm

I think this is wrong. Whether someone’s dead or not is a fact and is verifiable; what they died of is an opinion and much more subject to obfuscation.

“Much more”? Like 4 or 5 times more? Why would the families of the deceased in one study’s clusters be that much more confused about how their relatives died than the families in the clusters used in the other study? I can easily imagine scenarios that could lead to *some* confusion about how loved ones died, but why should this be reflected in one set of clusters and not the other, assuming both are representative?

Genuine question.

70

Donald Johnson 01.11.08 at 12:57 pm

#66—They didn’t “clearly discount” the increase because of recall bias. They said recall bias might contribute to the increase.

71

Matt McIrvin 01.11.08 at 1:02 pm

The 95% confidence interval on the Lancet 2 study went down into the 400,000s range, if I recall correctly, so an excess-death number in that range could even be consistent. Many people seized on the median value as if it were some sort of confident pronouncement, but they did report a claimed uncertainty that was pretty wide.

Personally, when talking about these numbers I’ve always qualified them with statements like “Even if that’s high by a factor of 10…”

72

abb1 01.11.08 at 1:11 pm

#69, for example: someone was stabbed and died three weeks later from sepsis that could’ve been easily treated if antibiotics were available – is this a violence-related mortality?

73

Kieran Healy 01.11.08 at 1:16 pm

With luck, Kieran will originate most of the Lancet discussion so that the CT community of readers might hear from all sides in this important debate.

With luck, it will be clear to everyone that I have no interest in facilitating discussion with people who have repeatedly shown that they are not interested in arguing in good faith.

74

Ragout 01.11.08 at 1:18 pm

I think this is wrong. Whether someone’s dead or not is a fact and is verifiable; what they died of is an opinion and much more subject to obfuscation.

Since the Lancet II study asked for death certificates, it would be useful to know how the cause of death recorded on them matched up with the reported cause of death. Too bad the Lancet II researchers didn’t bother with this sort of cross-check.

75

Daniel 01.11.08 at 1:21 pm

Why would the families of the deceased in one study’s clusters be that much more confused about how their relatives died than the families in the clusters used in the other study?

Basically because it’s not a matter of “confusion”, as I say it’s a matter of “depends who’s asking”. A lot of people were murdered by militias, who everyone is scared of. A lot of the people who were murdered, might have been at least semi-attached to militias themselves. If Philip Tattaglia chucked Luca Brasi into the river and then sent his family a fish through the post, then the answer to “how did Luca die?” is going to be “he fell in the river” if the government is asking, “Tattaglia killed him” if Michael Corleone’s asking and one of the two, who knows which, if the questioner is a research team from Johns Hopkins University.

76

Daniel 01.11.08 at 1:24 pm

74: What are you talking about, Ragout? Yes they did. It’s on the second page of the Burnham et al study:

“At the conclusion of household interviews
where deaths were reported, surveyors requested to see a copy of any death certificate and its presence was recorded. Where differences between the household account and the cause mentioned on the certificate existed, further discussions were sometimes needed to establish the primary cause of death.”

It is a bit of a pity that the IFHS survey didn’t ask for death certificates, although one has to recognise that they were carrying out the survey for their purposes, not my personal convenience.

77

Tim Lambert 01.11.08 at 1:24 pm

The violent death rate in L1 is less than that in the IFHS. But David Kane says that it is ludicrously high. Looks like Kane will be out smearing the IFHS folks next.

78

David Kane 01.11.08 at 1:29 pm

Tim, Can you provide your analysis of the claim that the violent death rate is higher in IFHS than L1? Don’t forget to include Anbar in both. It’s part of Iraq, after all. (I could be wrong about this. Again, I have just skimmed IFHS.)

79

Barry 01.11.08 at 1:35 pm

No, Tim, my money is on him praising the IFHS study. For strictly scientific reasons, of course.

Daniel: “74: What are you talking about, Ragout? Yes they did. It’s on the second page of the Burnham et al study:”

Looks like Ragout goes in the cage with Kane.

80

Borwnie 01.11.08 at 1:45 pm

Daniel/abb1,

I understand completely. As I said, I can easily imagine scenarios that could either lead to confusion about how the dead died and/or elicit less-than-accurate responses to the question of how they died. But again, assuming the clusters in both studies are representative, that confusion should be reflected in the results of both. There’s no good reason why more people should be dying from sepsis after three weeks in one study and not the other.

The distribution of violent and non-violent deaths in each study is certainly influenced by who asks the question, how it is asked and the genuine confusion about the exact circumstances of death on the part of the interviewees, but these factors cannot explain why that distribution is turned on its head from study to study….again, assuming both sets of clusters are representative.

And for the record, and despite what might be thought, I have no interest in advancing the merits of one study over the other. Insofar as I have a position on this stuff at all, it’s only that I think anomalies – or flat out contradictions, if you prefer – such as the violent/non-violent distribution from study to study, undermine the credibility of all such studies in what are effectively war-zones.

81

Ragout 01.11.08 at 1:46 pm

Daniel,

The fact is that they didn’t record the cause of death listed on the death certificate. There was no way for the survey’s supervisors, never mind other researchers, to verify that death certificates were ever examined. The Lancet researchers simply didn’t perform the kind of routine quality control checks conducted by most survey researchers, such as those conducting the ILCS Iraq survey or the IFHS survey.

And I wonder how these discussions about the cause of death were conducted in Kurdistan, given that Lancet II employed no Kurdish-speaking interviewers?

82

Daniel 01.11.08 at 1:48 pm

but these factors cannot explain why that distribution is turned on its head from study to study

well, basically L1 and IFHS seem to be pretty consistent, but L2 (which was the only one in which death certificates were systematically checked) is different.

#79: no, this could easily have been an honest mistake and Ragout (although he disagrees with me on a number of issues relevant to the surveys and has done since L1) has a pretty good track record of honesty. You really have to make an effort to get banned.

83

Ragout 01.11.08 at 1:48 pm

Barry,

Are you going to back up your claim that the National Journal fabricates interviews?

84

Tim Lambert 01.11.08 at 1:50 pm

ragout, the National Journal claimed that Garfield had rejected the Lancet number and reduced his estimate of deaths to 250,000. I asked him if this was true and he said that it wasn’t. The only source for your alleged fact is the National Journal who have proved themselves to be biased and unreliable.

85

Kieran Healy 01.11.08 at 1:53 pm

Tim, Can you provide your analysis of the claim that the violent death rate is higher in IFHS than L1? Don’t forget to include Anbar in both. It’s part of Iraq, after all. (I could be wrong about this. Again, I have just skimmed IFHS.)

Tim, I just took a quick look and my garage needs tidying up. If you could pop over and do that for me as well, that’d be great.

86

Daniel 01.11.08 at 1:56 pm

Ragout, you appear to be operating quite a bit of a double standard here;

There was no way for the survey’s supervisors, never mind other researchers, to verify that death certificates were ever examined. The Lancet researchers simply didn’t perform the kind of routine quality control checks conducted by most survey researchers, such as those conducting the ILCS Iraq survey or the IFHS survey.

given that the ILCS and IFHS surveys didn’t check death certificates at all. I am not a big fan of this nitpicking Milloyish approach at the best of times (I’ve noted above that the Iraqi Health Ministry is quite sensible to concentrate its research efforts on things that will help improve the health of the Iraqi people rather than settle weblog arguments for me), but at least apply it consistently.

And I wonder how these discussions about the cause of death were conducted in Kurdistan, given that Lancet II employed no Kurdish-speaking interviewers?

I am hazarding a guess here that most Iraqi Kurds also speak Arabic.

87

abb1 01.11.08 at 1:58 pm

Borwnie, I imagine different studies could use different sets of criteria to classify death as violent or non-violent. For example, one study might consider a death violent only if it occurred within 24-hour interval following the injury and non-violent otherwise, while another study might just ask a family member – ‘was it a result of violence?’. I’m just guessing here.

88

Tim Lambert 01.11.08 at 1:58 pm

Sorry David, but Anbar doesn’t belong in the comparison. In L1, the Fallujah cluster was so different that you couldn’t get a reliable estimate if you included it. The IFHS wasn’t able to properly survey Anbar.

It’s par for the course with you that by L1 you don’t mean the estimate that the authors gave and everyone else means.

89

John Emerson 01.11.08 at 2:02 pm

A year ago, I thought that there was a 50% (or whatever) chance of fraud.

I did not seriously want to see Kane on this thread. I was joking, on the assumption that he had been banned. Fie on Kieran’s namby-pamby liberalism.

High among the reasons why I feel that Dsquared was justified in banning him was Kane’s hasty accusation that fraud was involved in the Lancet article. This is only one of the several reasons why I have concluded that Kane is an ill-intentioned guttersnipe. He helped turn a potentially reasonable and useful discussion of methodology, and of the facts in Iraq, into a mudslinging contest.

From a scientific point of view, the collection and analysis of data in a war zone like Iraq’s is very difficult, and when the discussion is invaded by operatives with a political agenda, discussion becomes impossible. My main criticism of all these studies is their naive, wishful belief that in the present embattled world a scientific study of the facts on the ground might contribute to the political resolution of the debate over the Iraq War.

For now, I have concluded that there were lots and lots of excess Iraqi deaths after the invasion, even if these are balanced against Saddam’s victims. (High ex recto estimates of the numbers of Saddam’s victims are somehow not subjected to the same critical scrutiny as estimates of the post invasion estimates.)

The differences between the reasonable high and low estimates are considerable, maybe a third of an order of magnitude, but we’ll know better after the war, if it ever ends. For right now, the “lots and lots” estimate looks pretty robust. I don’t think that much really rides on this, though Kane obviously wants people to think that something does.

90

Borwnie 01.11.08 at 2:06 pm

Abb1,

Yep, that’s possible I suppose. I don’t know how likely, though, given there is also a sizeable discrepancy in the violent/non-violent distribution between L1 and L2, where I assume (and I know how wrong that could be) the same methodology was employed.

Likewise, what happens when the response to the question “How did your relative(s) die?” is “We don’t know”. I imagine there are more than a few cases of people leaving home in the morning and simply never coming back. Are the studies consistent in how they classify these cases of unknown cause of death?

91

John Emerson 01.11.08 at 2:06 pm

I am just a tiny bit surprised that there isn’t a single expression of relief about that.

Oh, come on.

92

Ragout 01.11.08 at 2:10 pm

Tim Lambert,

We merely have to read Garfield’s published letter to the Lancet where he writes that his former coauthors’ study contains “substantial reporting error,” to know that it is you who are misrepresenting Garfield’s views, not the National Journal.

93

John M 01.11.08 at 2:14 pm

“Oh, come on.”

You’re right, of course, I’m not really surprised. It is something of a relief to me though.

94

Ragout 01.11.08 at 2:15 pm

I am hazarding a guess here that most Iraqi Kurds also speak Arabic.

That is not my understanding. It’s worth noting that both the ILCS and the IFHS survey instruments were translated into both Arabic and Kurdish.

95

Neil 01.11.08 at 2:19 pm

Wikipedia:

Most Kurds are bilingual or polylingual, speaking the languages of the surrounding peoples such as Arabic, Turkish and Persian as a second language.

96

John Emerson 01.11.08 at 2:20 pm

John M, to you the new study isn’t not 200,000 dead (or whatever), but 400,000 not-dead (or whatever, compared to the old study).You are easily made happy, John M. I envy you! Things still look pretty bad to me.

And, by the way, I am still working on a reply to Kieran’s post from 2006 on my fraud discussion. My rough draft of that reply is here.N’i>

Still working, a year later? It didn’t take Melville that long to write Moby Dick.*

*Now the motherfucker is going to go to Wiki to find out how long it really did take, and come back here shouting “Truth is important”.

97

Neil 01.11.08 at 2:22 pm

Most Kurds are also bilingual in the lingua franca of the country in which they live, for example, Arabic in Iraq.

http://www3.baylor.edu/~Charles_Kemp/kurdish_refugees.htm

98

David Kane 01.11.08 at 2:23 pm

Tim writes:

It’s par for the course with you that by L1 you don’t mean the estimate that the authors gave and everyone else means.

Here is the opening sentences of the methods section of L1.

The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja.

The authors don’t give an estimate of 2.5 for all of Iraq? I am making this up?

The issue before us is: How does the 151,000 estimate of violent deaths in all of Iraq from IFHS compare with L1? Now, since the surveys use different terminology over a different time scale, we will not be able to make an exact comparison. But, L1 reports 73 violent deaths in all of Iraq post-invasion compared to 1 pre-invasion (from the phantom US bombing runs, no doubt.) (See Table 2.) Speaking very roughly each excess death in the sample corresponds to 3,000 or so deaths in the population. So, for all of Iraq, there were around 200,000 violent deaths in L1 through September 2004. (I am obviously skirting over the, in this context, unimportant distinction between violent deaths and excess violent deaths.)

IFHS estimates 151,000 violent deaths through June 2006. Relative to IFHS, the L1 estimate is ludicrously high.

Again, this is just back of the envelope, but I wanted to help Tim clean out his garage, following Kieran’s kind suggestion.

Now, Tim might argue that we need to exclude Falluja for this that or the other reason. Fine. If the L1 authors had just dropped Falluja from all of their analysis (or included it everywhere or done both), I wouldn’t have objected so much. But they picked and chose. Yet, in this context, you don’t get to play that game. The IFHS authors estimate 151,000. That is their number. You can either try to get numbers out of L1 that are comparable to that (as I do above). Or you can claim that such a comparison is impossible. But you can’t just claim the comparison works for the subset of the IFHS data that you want to look at.

99

John Emerson 01.11.08 at 2:26 pm

David, when will Moby Dick be finished? Aren’t you wasting your time here? Do you have the same editor as Jonah Goldberg?

100

Ragout 01.11.08 at 2:30 pm

Daniel,

I am also not interested in nitpicking the Lancet study. It’s been trashed now by so many highly respected researchers that it’s clear that it has little credibility. See the NEJM article, which states flat out that “the 2006 study by Burnham et al. considerably overestimated the number of violent deaths.” Or see the many letters published in the Lancet criticizing the Lancet II study, including one by a former coauthor. Or see the critical article in Nature. Or see critical statements by Iraq mortality experts like Beth Osborne Daponte or Jon Pederson, who says thinks the Lancet numbers are “high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much.”

101

John Emerson 01.11.08 at 2:43 pm

Ragout, that may all be true, and eventually most people here may end up agreeing. Or it may not be true.

I’m not familiar with your contributions specifically on this topic, but David Kane’s hasty accusations of fraud poisoned the debate here on Crooked Timber. The discussion about the Lancet study is continuing, and in the end I presume that there will be a consensus. (I am not conceding that the critics are right. I’m just saying that I trust that ultimately the questions will be answered, and that keeping Kane out of the discussion will help us get a a good answer sooner.)

As for me, I still think that my “lots and lots” estimate is robust, and it’s precise enough for my purposes.

And when I see 100,000 or 200,000 dead people, I see 100,000 or 200,000 dead people. I don’t see the people who are no longer thought to be dead. In this context, glass-half-full thinking is creepy.

102

Ragout 01.11.08 at 2:47 pm

Neil,

“Most Kurds are also bilingual” doesn’t mean “all Kurds,” or even “60% of Kurds.” Nor does it tells us how fluent in Arabic they are. I suspect that there’s a reason why high-quality surveys in Iraq translate their questionnaire into Kurdish.

103

Neil 01.11.08 at 2:51 pm

Quite right, ‘most’ doesn’t mean 60%. It’s obviously vague, but it implies a high percentage; 60% would be too low. To call someone bilingual, too, is vague, but implies relative fluency.

104

Ragout 01.11.08 at 2:53 pm

John Emerson,

It’s true that it doesn’t make much difference to our judgements about the Iraq war whether 600,000 have died or “only” 100,00. But it’s still worth debating methods for conducting mortality surveys, because lots of these surveys are being conducted all around the world.

On the other hand, it isn’t all that worthwhile debating Lancet II anymore, because a consensus has pretty much emerged in the epidemiological community that its estimates are very unreliable.

105

Daniel 01.11.08 at 2:53 pm

I am also not interested in nitpicking the Lancet study

this assertion would have more credibility if it hadn’t come at the end of a long trail of posts doing just that.

On your more general assertion:

1) You don’t appear to mention the studies that have corroborated the Burnham et al estimate, which is odd
2) I agree with Tim L that you are substantially overstating your case with respect to Richard Garfield
3) the NEJM article disagrees with the conclusion of the Lancet article and thus notes that on the basis of its results Burnham et al is an overestimate, but does not even mention the methodology of Burnham et al, let alone criticise it (quite rightly, of course).

106

Daniel 01.11.08 at 2:54 pm

On the other hand, it isn’t all that worthwhile debating Lancet II anymore, because a consensus has pretty much emerged in the epidemiological community that its estimates are very unreliable

Ragout, this is a very specific and very extreme claim which you cannot possibly substantiate (particularly, if your post #100 was your best shot at doing so, it’s pathetic).

107

John Emerson 01.11.08 at 2:55 pm

In bilingual areas, “most” usually means “almost all men and many women”, and 51% doesn’t as “most” at all — I think that 60% is at the low end of “most”.

I doubt that there are any populations of any size in Iraq without a lot of Kurdish speakers.

But you fling all the shit you have in the hope that some of it will stick.

108

Tim Lambert 01.11.08 at 2:57 pm

Here’s what L1 says:

“We estimate that 98 000 more deaths than expected (8000–194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.”

Notice how they don’t give a number for an estimate that includes Falluja. Only a David Kane could read that sentence and claim that the L1 estimate wasn’t 98,000.

109

John Emerson 01.11.08 at 2:59 pm

104: Yes, as I said, it’s a good thing that the people who understand these things are debating the various studies. I’m just saying that Kane isn’t contributing anything to that debate here, and I doubt that you are either.

110

dalazal 01.11.08 at 3:05 pm

L2: 655,000 excess deaths, 2.4 fold increase in death rate post-invasion

NEJM: 430,000 excess deaths, 2 fold increase in death rate post invasion

L1: 98,000 excess deaths for Mar03-Sep04

NEJM: 99,500 excess deaths for Mar03-Sep04

How can that boost any claim of L1 and L2 fraud is beyond me.

111

Tim Lambert 01.11.08 at 3:06 pm

ragout says:

We merely have to read Garfield’s published letter to the Lancet where he writes that his former coauthors’ study contains “substantial reporting error,” to know that it is you who are misrepresenting Garfield’s views, not the National Journal.

Well let’s do that shall we?

The existence of a substantial reporting error is supported by the finding of low child mortality. The study population only reported 54 non-violent deaths in those younger than 15 years, and 1474 births—ie, an under-15 mortality of 36 per 1000 births. This is a third of the estimated preinvasion under-5 mortality.3 Since nothing indicates that child mortality has decreased,4 the results suggest that fewer than half of child deaths were reported.

The reporting error Garfield mentioned was an undercount. Which ragout tries to pretend means that Garfield concluded that L2 was four times too high.

In other words, ragout is dishonest.

112

John M 01.11.08 at 3:25 pm

“You are easily made happy, John M. I envy you! Things still look pretty bad to me.”

And they look pretty bad to me too (I don’t think I claimed to be ‘happy’ about any of this). 150,000 violent deaths is a truly shocking thing to consider. But surely you can be shocked at the reality and still take some relief from the knowledge or belief that the number of dead is less than you had at first feared. Regardles of the horror it is surely good news by any dfinition that fewer died than had been feared? A casual reader of Crooked Timber could go away with the sense that many contributors are disappointed that the Lancet study has been discredited (if it has been) when surely they should be pleased to discover that hundreds of thousands of people that they had thought to have been killed, are, in fact alive.

113

dalazal 01.11.08 at 3:27 pm

Ragout (#104): “On the other hand, it isn’t all that worthwhile debating Lancet II anymore, because a consensus has pretty much emerged in the epidemiological community that its estimates are very unreliable.”

I’m sure you say this based on something. Do you mind sharing your sources?

114

Kieran Healy 01.11.08 at 3:27 pm

I imagine you’re concerned, john.

115

JP Stormcrow 01.11.08 at 3:28 pm

“Yes,” Rieux said. “And though the burials are much the same, we keep careful records of them. That, you will agree, is progress.”
The Plague

Indeed.

I am sure the next generation of Neocons are internalizing the lesson that if you can reduce the number of stable households capable of responding to a survey after an intervention you can lower the “death rate”.

116

John Emerson 01.11.08 at 3:53 pm

But it’s still worth debating methods for conducting mortality surveys, because lots of these surveys are being conducted all around the world.

I absolutely agree. But why has the Iraq survey got 10 or 100 times more attention than the Ruanda and Bosnia surveys put together? Why do the same people nit-picking the Lancet story throw out ex recto estimates of the number of Saddam’s victims?

The serious studies you allude to are very good, important things, but it’s not for me to participate in them because I’m unqualified. What’s going on here is something completely different — a tendentious effort to Ratherize the Lancet study. I’m happy to jump in on that one.

117

Neil 01.11.08 at 4:15 pm

I think ragout’s bad faith can easily be demonstrated by perusing this thread. Daniel wrote

I am hazarding a guess here that *most* Iraqi Kurds also speak Arabic.

Ragout said that this wasn’t his understanding (#94). I cited some sources backing up Daniel, and ragout then moved the goal posts, saying that *most* might mean less than 60%.

118

Donald Johnson 01.11.08 at 4:19 pm

“Why do the same people nit-picking the Lancet story throw out ex recto estimates of the number of Saddam’s victims?”

Well, most of the technical details about surveys are outside my range of competence, though I hang out on the fringes of this debate, but we all know the answer to this question.

On a related topic, we shouldn’t let the MSM get away with comparing apples to oranges. Not that we have any actual say in the matter, but I once saw a NYT article comparing the Iraq Body Count numbers with the survey numbers for the Congo War. It’s perfectly acceptable to cite Les Roberts when he’s talking about huge death tolls that can’t (or aren’t) blamed on the US.

That’s one of the complaints I’ve had about IBC. They’ve established a new level of evidence that has to be reached when talking about US-caused atrocities, but you can cite any damn estimate you want for anything else, the larger the better if it can’t be blamed on our country.

119

paul 01.11.08 at 5:43 pm

Joe Stalin had these folks pegged: “A single death is a tragedy; a million deaths is a statistic.

I would like to see Keiran’s closing sentence in a speech or otherwise distributed widely. Well said.

120

c.l. ball 01.11.08 at 5:55 pm

This is the kind of thing that convinces me David Kane is just another hack. The 150,000 is a figure for violent deaths, the 600,000 is a figure for excess deaths (as stated repeatedly above).

No, no.

Lancet II (2006) estimated excess, violent deaths at 601,027 (426,369 to 793,663 at 95% confidence interval) for the 40 month post-invasion period. The IFHS/NEJM study estimates all violent deaths at 151,000 (104,000 to 223,000) for the same period. Of course, Lancet found so few pre-invasion violent deaths that several of the authors considered nearly all violent deaths to be excess, and NEJM finds the violent death rate increased 10x pre and post invasion.

The location estimates are interesting. Lancet estimated 156,000 deaths in Baghdad; NEJM estimated 81,000. The major difference was the death estimates for high-mortality provinces — Lancet estimates many more there than NEJM has.It could be that Iraqi health ministry personnel were less likely to sample properly in the high-conflict provinces, and so under-counted there.

The daily death toll for the Lancet is 80% higher than the NEJM for the first year of the war, but takes off after that. So NEJM and Lancet disagree strongly over death rates after April 2004, when the Shia insurgencies really take off.

Remember that both Lancet and NEJM are not counting only combat-related violence but all violence, so criminal murders and shootings, revenge killings, and murderous brawls all get counted along with gun battles, airstrikes, and car bombs. In that context, the Lancet data seems less startling though still awful. The household surveys will pick up violent deaths that the media ignore because they are not major bombings or battles.

Let’s put it this way. IBC daily death tolls are on par with the US daily homicide toll (roughly 43 per day; 16,000+ per year). Does that seem plausible? NEJM has the daily Iraq toll at about 3 times the US toll? Does that seem plausible?

121

Justin 01.11.08 at 6:19 pm

cl ball, you obviously have to factor in for the fact that Iraq is less than the 10th the size of the United States.

122

mq 01.11.08 at 6:35 pm

Well, the U.S. has a population ten times larger than Iraq, so the NEJM study would be a rate thirty times higher than the U.S., not three times higher. But thirty times higher still seems not at all unreasonable on the face of it — the U.S. is a very peaceful country.

Doing survey based estimates of mortality in the midst of a civil war, while millions of refugees are fleeing the country, will always be fraught with error. But it now seems clear that by the end of 2007 there were many hundreds of thousands of excess deaths in Iraq due to the U.S. invasion. 200,000 seems quite low for total excess deaths, when you count the non-violent ones (read some of the anecdotal stuff on what happened to Iraqi medical and hospital services in 2004-2007). 400-500,000 seems quite reasonable, based on multiple surveys now.

I suppose a half a million civilian dead conclusively proves that the warbloggers have been right all along.

123

christian h. 01.11.08 at 6:38 pm

c.l. ball, it’s not just that [i]t could be that Iraqi health ministry personnel were less likely to sample properly in the high-conflict provinces, it is in fact the case that many clusters in Anbar province in particular, and in high-violence areas of Baghdad were not sampled. The NEJM study’s authors corrected for this by weighting the distribution of violent deaths over the country according to IBC numbers. This is a serious problem, since IBC are derived from media reports, and therefore should be undercounting more severely precisely in the high violence areas. I’d like to see the numbers obtained if you take the NEJM results and weight with respect to the geographic distribution obtained in Lancet 2.

124

c.l. ball 01.11.08 at 6:48 pm

Re 121, sure but how much worse would you expect overall violence to be in Iraq than the US? 50x, 100x, 200x? At 100x, divide by 11 to account for the pop. differences (300m v. 27m roughly), so 9*43 gets you 391 deaths per day, x 30 days per month over 40 months, at we are within the lower bound of the Lancet confidence interval.

The Lancet finding sounds implausible because we hear 600,000 or 500 per day and can’t get our heads around such numbers. How could we miss an average of 500 each day — so some days would have thousands of deaths and some maybe a score? But we’re not missing them in a context of stable government but in a context of widespread absence of government. Nil effective policing, criminal gangs, undemocratic local rulers, short-tempers under stress and privation, and add on the invasion and insurgency battles, ethnic murders, and terrorist attacks and you get a lot of dead people really fast.

125

John Emerson 01.11.08 at 6:51 pm

It wouldn’t be unusual for the wartime death rate to be 100x or more the peacetime death rate. Wars are like that.

126

Uncle Kvetch 01.11.08 at 6:55 pm

A casual reader of Crooked Timber could go away with the sense that many contributors are disappointed that the Lancet study has been discredited (if it has been) when surely they should be pleased to discover that hundreds of thousands of people that they had thought to have been killed, are, in fact alive.

I deliberately set fire to an apartment building where 30 people lived. I thought I’d killed half of them, but now I’ve learned that only 13 died, not 15.

Joy!

127

seth edenbaum 01.11.08 at 7:04 pm

The WHO:
“…estimates total excess deaths (through June 2006) at about 393,000. The Lancet study pegged it at 655,000.

…estimates total post-invasion violent deaths at 151,000. The Lancet study said the number was 601,000.”

128

Tim B 01.11.08 at 8:11 pm

I suppose that, on the other hand, this new study does confirm something that’s becoming increasingly clear with time – namely that, contrary to what some people were telling us at the time, the sanctions on Iraq were an extremely humane policy, which, far from killing people, was actually saving lives relative to neighbouring countries:

http://content.nejm.org/cgi/content/full/NEJMsa0707782

The pre-invasion rates of adult mortality from any cause per 1000 person-years were 2.0 for men and 0.8 for women in the IFHS, with a relatively small proportion of deaths attributed to violent causes. In a regional comparison for 2002, a study by the WHO estimated that in Syria and Jordan, the rates of death for adults were 4.2 for men and 2.8 for women. In Iran, the rates were 4.7 and 2.9, respectively.

So while we’re rightly disgusted by Western foreign policy as regard the war in Iraq, maybe we could just take one moment to congratulate the framers of that other piece of foreign policy, the sanctions regime – kudos to Bill Clinton, Maddy Albright and all those others who framed it.

129

Barry 01.11.08 at 9:08 pm

“I am sure the next generation of Neocons are internalizing the lesson that if you can reduce the number of stable households capable of responding to a survey after an intervention you can lower the “death rate”.”

Posted by JP Stormcrow

Not really, I imagine. They’re just relearing the old lesson that lying about what’s going will buy enough time so that only the historians will still be interested. And the locals, as well, but they don’t count to the Empire.

130

Donald Johnson 01.11.08 at 9:38 pm

Was #128 serious? It’s hard to tell sometimes.

Anyway, on the off chance that it was, weird as that seems, most people agree that Oil for Food greatly decreased the effects of the sanctions on mortality. It’s doubtful that the sanctions actually improved life expectancy–if so, then those free traders have really been selling us a bill of goods.

131

David Kane 01.11.08 at 10:07 pm

Here’s what L1 says:

“We estimate that 98 000 more deaths than expected (8000–194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.”

Notice how they don’t give a number for an estimate that includes Falluja. Only a David Kane could read that sentence and claim that the L1 estimate wasn’t 98,000.

The L1 estimate for all deaths was 98,000. The question is: What does L1 tell us about violent deaths? We need something from L1 to compare to the 151,000 figure from IFHS. Given that we do not have access to the code, we do not know what the L1 authors would say. They do not provide an estimate for violent deaths. So, we need to look at the (limited) raw data that they supply, make some assumptions and do the best we can. That’s what I do above. Back of the envelope, using the L1 supplied data, there were 200,000 violent deaths in Iraq. What estimate do you think that L1 data implies? {Please show your work.)

132

Donald Johnson 01.11.08 at 10:49 pm

Excluding Fallujah, there were about 21 violent deaths in the Lancet1 survey. As a rough estimate, each death represents 3000 real deaths, so there were about 60,000 deaths in L1. That’s crude, but those were the numbers at stake.

This was understood by everyone on both sides of the issue back in the good old days, when the debate was between those who claimed that 60,000 violent deaths by Sept 2004 was ridiculously high, given Iraq Body Count’s much lower figures. At the time they were estimating something like 15,000 (this was later updated to a slightly higher number, around 18-19,000, I think).

David wishes to turn this into part of his morality play about the evil Lancet1 authors, but the Lancet skeptics in 2004-2006 rejected the possibility that the IBC number was 3-4 times lower than the total number of violent deaths. That’s what the argument was about. And the IFHS paper now supports a violent death rate that was rejected as far too high by Lancet skeptics in the 2004-2006 period.

133

ScentOfViolets 01.11.08 at 11:08 pm

What seems to be disputed about the older study? If we assume that the violent death excess totted up by the newer one is correct, and if we assume that table 3 in the report is correct, then I get:

[(6.01-3.17)/(1.09-0.10)]*150,000~400,000

Since this is in agreement with the lower bound on the confidence interval in the Lancet study (~400,000), how can anyone say these findings differ in any significant way? Is there something wrong with my math?

134

Donald Johnson 01.11.08 at 11:46 pm

Was that a violent death interval you are calculating, scent? Because it’s not the one in the paper.

135

christian h. 01.12.08 at 1:31 am

Is Kane claiming that Lancet 1 estimates all deaths at 98,000, but he gets (from their numbers) 200,000 violent deaths? Now I’m only a mathematician, but that seems like a strange contention.

136

ScentOfViolets 01.12.08 at 1:55 am

Um, no. I figured like this, let P be the population of Iraq (assumed to be more or less constant; it’s not, but close enough.) Let PoV be the rate of post-invasion violent deaths and PrV the rate of pre-invasion violent. So then the excess deaths due to violence is given by:

3(PoV-PrV)P=150,000, and solving for P, the population, we get:

P=150,000/[3(PoV-PrV)].
Now let PoA be the rate of all post-invasion deaths, PrA be the rate of all pre-invasion deaths. Then the excess of all deaths will then be given by:

3(PoA-PrA)P=E.

Using our substitution for P:

3(PoA-PrA)*150,000/[3(PoV-PrV)]=[(PoA-PrA)/(PoV-PrV)]*150,000=E.

Now we just substitute the values given in the table to get:
[(6.01-3.17)/(1.09-0.10)]*150,000~400,000

Note that the assumptions are a)that the number of excess violent deaths is understood to be 150,000, and b)the rates in the table are accurate. I am most specifically _not_ calculating the number of excess violent deaths, I am taking it as a given, something that everyone agrees upon, and saying that _if_ this is true, _then_ the overall excess must be on the order of 400K, which is in agreement with the lower bound of the Lancet estimate.

I don’t see any obvious mistakes here. Can anyone spot any flaws in the reasoning? Note that the three in the above formulas come from the number of years the study covered, and it doesn’t really matter since it cancels out at the end. Also, since the number in the square brackets is just a ratio, I didn’t bother to convert from thousands of person years – the units cancelled there too.

137

Dan 01.12.08 at 2:51 am

ScentOfViolets (133), that should be:

[(6.01-3.17)/(1.09)]*151,000=393,000

151000 is an estimate of violent deaths, not excess violent deaths, which means that it’s based on 1.09, not 1.09-0.10.

138

ScentOfViolets 01.12.08 at 3:02 am

Okay. This is a number I came up with earlier on a simplifying assumption that 1.09 and 1.09-0.10 were about the same, and, as you say, it’s still approximately 400,000, the lower range of the Lancet study. So how can anyone say that these two surveys don’t agree with each other?

Are there any other errors? Frankly, this seemed like a no-brainer calculation, and two crow that the studies are wildly disparate means that not only do you have to be a hopeless partisan, you have to be an innumerate hopeless partisan.

139

Donald Johnson 01.12.08 at 3:10 am

They disagree on the violent deaths, but overlap on the excess, though I don’t think the paper actually makes any claims on the excess deaths.

140

SG 01.12.08 at 3:21 am

Leaving out the little “please show your work” comment, which is too smug for words, this piece of mendacity and bigotry by David Kane is really fucking stupid:

The L1 estimate for all deaths was 98,000. The question is: What does L1 tell us about violent deaths? We need something from L1 to compare to the 151,000 figure from IFHS. Given that we do not have access to the code, we do not know what the L1 authors would say. They do not provide an estimate for violent deaths. So, we need to look at the (limited) raw data that they supply, make some assumptions and do the best we can. That’s what I do above.

Here we have a paper (L1) which gives an estimate of excess deaths but no estimate of violent deaths; and another paper (IFHS) which gives an estimate of violent deaths but no estimate of excess deaths.

But Kane’s “woe is me” tone is simply dripping with contempt for those evil lancet authors who couldn’t furnish him with BOTH counts. Why doesn’t he reserve that tone for IFHS? Why is it L1’s fault that Kane has to do “the best we can”, not IFHS’s?

Kane you are a right twat.

I just know that if IFHS had collected death certificates and L2 hadn’t, Kane would be all over L2 because of it. So would Ragout. But instead the opposite occurred, so Kane ignores it and continues to pretend that the IFHS data (which he hasn’t seen) is better. And Ragout complains about the level of oversight of the death certificates

You guys really are beyond the pale.

141

SG 01.12.08 at 3:26 am

Oh and dsquared, I vehemently disagree with your characterisation of Ragout’s honesty. The misrepresentation of Garfield’s letter to the Lancet is pretty cheap, but the way Ragout misrepresented the survey instruments over at deltoid was just nasty. Ragout acted as if the survey outline, called the “Template”, was a draft, and the portion of the questionnaire for recording deaths was the final copy. That way Ragout was able to accuse Tim Lambert of misleading his readers by not posting the final version, and accuse the Lancet authors of not collecting demographic details, in one fell swoop. A Trollish twofer.

I say Kane and Ragout are both liars.

142

Dan 01.12.08 at 4:01 am

One hidden assumption in that calculation (to get 393,000) is that the amount of underreporting is the same across the board.

The IFHS study calculates a post-invasion violent death rate of 1.09 (per thousand per year), before correcting for underreporting. But the authors note that underreporting of deaths is common in household surveys, because of factors like a household breaking up after a death. They use a something called the “growth balance method” (though they note that it is not entirely appropriate for Iraq, since it assumes a stable population) to estimate that the level of completeness in the reporting of death was 62% (that is, that 62% of deaths could be found by their sampling method and 38% would be missed). Then, for reasons that I don’t understand, they use 35% as their estimated level of underreporting (which means 65% as the level of completeness), and estimate the post-invasion violent death rate at 1.09/0.65=1.67 after correcting for underreporting. Then they translate this rate, 1.67, into the death toll of 151,000.

The rest of the death rates that they report in the paper do not get translated into number of deaths or corrected for underreporting, but anyone can do those calculations using the numbers that they report for the post-invasion violent death rate. If you take any death rate referring to the post-invasion period and multiply it by 151000/1.67, then you are simply translating it into estimated number of deaths (March 2003 through June 2006). If you instead multiply it by 151000/1.09 (as most people have been doing), then you are correcting for underreporting (assuming a 65% level of completeness) and translating the rate into number of deaths. Thus, 393,000 is the estimated number of excess deaths, March 2003 through June 2006, after correcting for underreporting. (6.01-3.17)*(151000/1.67) = 257,000 is the estimated number of excess deaths, March 2003 through June 2006, without correcting for underreporting. And 393,000 simply equals 257,000/0.65.

I’m guessing that it’s better to include the correction for underreporting, as people have been doing, although I don’t know enough about this sort of research to be sure. I think we may want to correct for underreporting when comparing death rates with the Lancet studies, as borwnie does in #52. Assuming the 65% level of completeness applies across the board, the estimated IFHS mortality rates are 3.17/0.65 = 4.88 pre-invasion and 6.01/0.65 = 9.25 post-invasion, which is more in line with the Lancet2 estimates of 5.5 and 13.3.

How sensitive are these results to the estimate of underreporting? With a 65% level of completeness, the estimated number of excess deaths is already equal to the lower bound of the Lancet2 confidence interval (393,000-943,000). With a 58% level of completeness, the IFHS estimate of pre-invasion mortality rate would equal the Lancet estimate, 5.5, and the estimated number of excess deaths would reach 446,000 (since 3.17/5.50 = 257000/446000 = 0.58). With a 51% level of completeness, the excess death estimate reaches 500,000 (since 257000/500000 = 0.51). If the level of completeness was only 39%, then the IFHS excess death estimate would reach the Lancet2 estimate of 655,000 (since 257000/655000 = 0.39). Which of these numbers are plausible? I don’t know, but the IFHS authors gave their 95% confidence interval for the level of completeness as 50%-80%, which suggests that 500,000 excess deaths should be inside the confidence interval.

143

Dan 01.12.08 at 4:21 am

Is it possible to figure out the 95% confidence interval for that 393,000 excess deaths estimate? In Table 4 of the supplementary materials, they do give confidence intervals for the (uncorrected for underreporting) pre-invasion mortality rate of 3.17 (2.70-3.75) and the post-invasion mortality rate of 6.01 (5.49-6.60), but I don’t know how to turn these into a single CI for the post-invasion excess mortality rate (the difference of the two, which has a point estimate of 6.01-3.17=2.84). Are the errors in the two quantities even independent?

Looking only at one cause of uncertainty, the level of underreporting (which they estimate at 35% and give a 95% CI of 20%-50%), we can create a (too small) 95% CI for the number of excess deaths. The lower bound is 257000/0.8 = 321,000 [which is (6.01-3.17)*(151000/1.67)/0.8], the upper bound is 257000/0.5 = 514,000, and the point estimate is 257000/0.65 = 395,000 (which differs from the 393,000 that everyone has been reporting only by a rounding error).

So the 95% confidence interval for number of excess deaths is wider than 321,000-514,000. Can anyone take this further?

144

ScentOfViolets 01.12.08 at 7:23 am

So I’m guessing there’s nothing wrong with the calculation from the tables then? If so, I think that’s all you really need to say, no need to fiddle around any more with the confidence intervals. After all, the original claim wasn’t that the Lancet study was off by ten percent or twenty percent; the claim was that it was off by a whopping 300 percent.

So not only are the findings of the two studies compatible, even if they disagree by as much as say, 30% (either way!), then they are still much closer to each other than either one is to the original administration estimate. That was on the order of, what, 50K? In which case, if this latest study is supposed to be the definitive, then we’re talking not 50 percent off, or 300% off, but a whopping 800% off.

The bottom line is, this study vindicates the Lancet survey, and torpedo’s the original administration figures.

145

troll 01.12.08 at 11:32 am

Tm Lmbrt:

f y dny Rgts ccstns bt n f th L2 c-thrs skng tht hs nm b wthdrwn frm th pblctn myb y cld nlghtn s s t why h dd s. Dd h smply jst wk p n th mrnng nd rndmly dcdd ths ctn? Myb h wntd th thrs t gt mr f th lmlght nd h ws bng gnrs?

Spclt lttl hr, Tm? Wlk rnd th prk wth s nd gv s yr hnch why.

146

Donald Johnson 01.12.08 at 4:07 pm

Oh, if it’s a question of the Bush Administration’s figures, I could flip coins or read the entrails of road kill and get better numbers.

147

David Kane 01.12.08 at 5:33 pm

Open-minded Daniel Davies has banned me from his thread, so I will just note here that Kane’s Zombie Army is on the march here.

And, by the way, isn’t it pathetic for Davies to ban me from a thread in which he criticizes me by name? I can understand a policy in which you ban someone from a thread in which you do not mention him, as Henry does to me. But to prevent someone from replying to a direct attack seems, I don’t know, sort of Little Green Footballish, don’t you think?

148

John Emerson 01.12.08 at 6:30 pm

We’re reconsidering Little Green Footballs around here. Without entirely agreeing with our adversaries, we can stil learn from them. We’re Fascists, you know.

149

nick s 01.12.08 at 6:43 pm

But to prevent someone from replying to a direct attack seems, I don’t know, sort of Little Green Footballish, don’t you think?

We already know what you’re going to say, Wavy. We’ve even stored up a little cache of exclamation marks here [!!!!!!!!!!!!!!!!] in case you run out. And it’s not as if there aren’t several other places you seem able to peddle your wares.

150

rodeo 01.12.08 at 8:29 pm

Gross Distortions, Sloppy Methodology and Tendentious Reporting
http://counterpunch.org/andrew01122008.html

Almost five years into the destruction of Iraq, the orthodox rule of thumb for assessing statistical tabulations of the civilian death toll is becoming clear: any figure will do so long as it is substantially lower than that computed by the Johns Hopkins researchers in their 2004 and 2006 studies. Their findings, based on the most orthodox sampling methodology and published in the Lancet after extensive peer review, estimated the post-invasion death toll by 2006 at about 655,000. Predictably, this shocking assessment drew howls of ignorant abuse from self-interested parties, including George Bush (“not credible”) and Tony Blair.

Now we have a new result complied by the Iraqi Ministry of Health under the sponsorship of the World Health Organization and published in the once reputable New England Journal of Medicine, (NEJM) estimating the number of Iraqis murdered, directly or indirectly, by George Bush and his willing executioners at 151,000–far less than the most recent Johns Hopkins estimate. Due to its adherence to the rule cited above, this figure has been greeted with respectful attention in press reports, along with swipes at the Hopkins effort as having, as the New York Times had to remind readers, “come under criticism for its methodology.”

151

Donald Johnson 01.12.08 at 8:53 pm

The Cockburn article is overly harsh, (there’s no need to be as mean to the NEJM authors as David Kane is to the Lancet people) but it raises most of the points people have mentioned here.

David, the proper attitude to take about this is that we don’t know which is right–L2 or the NEJM paper. The truth might even lie in-between the two. The NEJM article at least has had the effect of making it respectable in mainstream society to admit that the IBC number is several times too small and as dsquared pointed out, it shows that the Iraqi government doesn’t know what the death toll is.

152

engels 01.12.08 at 9:05 pm

Oh, look it’s David Kane and he’s comparing this website to Little Green Footballs. Now that’s not something that happens every single fucking day

153

abb1 01.12.08 at 9:13 pm

If you ignore the parts of Cockburn’s article where he accuses NEJM and WHO of being imperialist running dogs, there’s still some substantial criticism there. Their estimates for the Sunni areas where they couldn’t go seems like a big one.

154

Donald Johnson 01.12.08 at 9:25 pm

“Their estimates for the Sunni areas where they couldn’t go seems like a big one.”

Yeah, that’s one of the points made either at this blog or over at Tim Lambert’s.

155

David Kane 01.12.08 at 11:12 pm

Donald Johnson insists that “the proper attitude to take about this is that we don’t know which is right—L2 or the NEJM paper.” Don’t tell me. Tell the NEJM authors. As I outline here, they are amazingly critical of the L2 estimate. They don’t think that the truth lies between the two estimates. They think that L2 is very, very wrong. Indeed, I can’t recall the last NEJM paper that was so critical of previous work. (Examples welcome.)

If you’d like to explain why the NEJM authors don’t know what they are talking about, be my guest. (Actually, discussing the strengths and weaknesses of both studies is exactly what we should be doing. But the first step is recognize what the NEJM authors actually claim.)

156

Donald Johnson 01.12.08 at 11:21 pm

Replace NEJM authors with Lancet2 authors, David. It’ll save me the trouble of typing a reply.

Here’s a bonus, though–it’s common in science for two different groups to derive two different answers and to have each side vehemently certain that they are correct. Really, you could have thought of this yourself.

157

David Kane 01.13.08 at 12:24 am

Once again, we have a disagreement on empirical facts. You assert that “It’s common in science for two different groups to derive two different answers and to have each side vehemently certain that they are correct.” True. But it is not that common to have such vehement disagreement in the elite literature. (Such examples do, of course, occur. Latest in economics was Hoxby in latest AER.) But they are rare. I have trouble coming up with a recent example in epidemiology, a case where one group was so critical of another. Can you?

Again, the fact that the NEJM authors are so critical of L2 does not mean that they are right. Or wrong. It is just very, very unusual. Was there as extreme a piece published in the NEJM in 2007? Not that I recall.

The reason that this matters is that it is an indication of where the debate is going. There are lot of very serious folks who are now gunning for L2. Stand by more for more harsh criticisms.

And make sure reservations for Denver in August. The session that I am organizing for the Joint Statistical Meetings should be amazing. (Note that I am not presenting. I am just the chair.)

158

Walt 01.13.08 at 12:33 am

David, Lancet 2 could easily be wrong. The fact remains that your criticisms have been _more_ wrong. Let’s not lose sight of that fact.

159

SG 01.13.08 at 1:50 am

Kane, your charming little opus fails to find a single “crticism” of the L2 paper, merely points out that the NEJM authors think L2 is an overestimate. That’s not a criticism dear, it’s a dispute over numbers. A critique would be to find a flaw in their sampling methods or their statistics. The NEJM team didn’t do that, and in fact used the L2 microdata in their paper. How’s that a critique?

Your charming little hatchet job also neglects to mention that the NEJM paper had a higher response rate than L2. What was the phrase you used way back when? “I can’t find any credible studies with a 99%+ response rate”…?

That alone was basis for accusations of fraud. How come this time around no comment? Partisan very much?

BTW for anyone who isn’t aware, jc is the ultimate troll and you DEFINITELY should not reply to him.

160

John Emerson 01.13.08 at 2:39 am

Kane is the George Chuvalo of rational argument.

161

Donald Johnson 01.13.08 at 4:36 am

“Once again, we have a disagreement on empirical facts. You assert that “It’s common in science for two different groups to derive two different answers and to have each side vehemently certain that they are correct.” True. But it is not that common to have such vehement disagreement in the elite literature.”

You gotta be kidding. No, you probably aren’t. Amazing.

Cosmologists disagreed for years about the value of the Hubble constant. I once saw a book review in Science by an ornithologist who referred to the paleontologists who thought birds come from dinosaurs as spouting “paleobabble”. The asteroid theory on how the dinosaurs became extinct was highly entertaining in part because of how nasty it was. I’ve actually got a book on the subject. I could lengthen this list. What planet do you live on, David?

I don’t know much about epidemiology–perhaps we have that in common. But the fact that two sides disagree vehemently is not proof that one side or the other or either side is right in any area of science, independent of how frequently disagreements occur.

I feel like I’m earnestly explaining to someone that yes, on a clear cloudless day the sky tends to be blue.

162

Donald Johnson 01.13.08 at 4:37 am

“The asteroid theory on how the dinosaurs became extinct was highly entertaining in part because of how nasty it was. ”

How nasty the debate was, I mean. Condescension and contempt flung in all directions. I’m still not sure if there’s a consensus.

163

ScentOfViolets 01.13.08 at 5:48 am

I’m not terribly familiar with epidemiology myself, but . . . what about AIDS and HIV? For a time, it was considered reputable to question the link:

http://www.avert.org/evidence.htm

In closely related there was the mystery of what was killing honey bees, coral atolls (is that redundant?), the links between autism and mercury, aluminum and Alzheimer’s, BSD and prions. All of these subjects, as well as the explanatory opposing theories appeared in some pretty high-powered peer-reviewed journals, like JAMA, Nature and the like.

164

David Kane 01.13.08 at 5:57 am

I agree that those are good example. But read what I wrote! I claimed that “it is not that common to have such vehement disagreement in the elite literature.” Which part of “not that common” do you fail to understand? Let’s be specific. There are hundreds of articles in the NEJM every year. What percent are as critical of other peer-reviewed articles as IFHS is of L2? Certainly less than 10%, even 5%. My guess would be less than 1%, but we can quibble about that. This is what I mean by “not that common.”

165

snuh 01.13.08 at 8:11 am

so david, why aren’t you troubled by the high response rate for the NEJM paper?

166

SG 01.13.08 at 8:49 am

yes David, why aren’t you troubled by the high response rate?

And how can you claim IFHS disputes L2 when its point estimate of total excess death is contained in the confidence limits of L2?

167

SG 01.13.08 at 8:50 am

and do you have a cite for that 1%? Because otherwise we’re just talking hearsay.

168

sod 01.13.08 at 10:21 am

In fact, if I were Roberts, I would be very worried about the IFHS authors getting so pissed off

funny. David is on a crusade to establish the category of “getting pissed off” into the mainstream of academic discourse.

and he is obviously leading by example..

169

dsquared 01.13.08 at 1:18 pm

just to clarify – I actually allowed one of Kane’s comments to stand in my thread, for the reason stated. But when he started abusing the privilege to plug his website, I realised I’d made a mistake extending even basic courtesy to him.,

170

David Kane 01.13.08 at 2:21 pm

dsquared,

If you had told me that not citing my website was a condition for posting on your thread, then I would have abided by your request. Can I post in your next Lancet thread of I don’t link to my blog? (It is easy enough to just copy and paste the arguments from there to CT, if that is preferable to you.)

Or perhaps you prefer the sort of conversation which that thread degenerated in to without the presence of informed, dissenting views . . .

And I am working on something about response rates. But dsquared implies that I am a bad person if I link to it from a Crooked Timber comment. Quite the dilemma!

171

SG 01.13.08 at 2:30 pm

dsquared, if you are going to let Kane comment in your posts, I think you should let him, but delete any comment where he does not directly answer the response rate question:

David Kane, you [insert random dsquaredesque insults here], given that you were sure the Lancet survey was fraudulent because of its very high response rate, and given that the IFHS survey has a similar response rate, why do you think the IFHS is not?

He has shied away from answering this question about 10 times in the last 2 days. I think someone should bait him directly and watch him flail.

Then, given that this study’s estimates of all excess deaths fall within L2’s confidence intervals, and it confirms L1’s death rate, we can all heave a sigh of relief (for Roberts’ reputation), declare the whole debate settled, and get back to discussing the substantive issue of what to do with the fuckheads who started a war that has killed between 400,000 and 1.2 million people.

172

John Emerson 01.13.08 at 2:43 pm

When Kane is banned, none of the other Lancet-deniers show up either. We were waiting for them on Dsquared’s thread, but only a guy named “a” showed up, and he only wanted talk about The Surge.

Doesn’t this confirm the Kane Zombie Army theory? Perhaps Lancet-denial is a little cottage industry run out of Kane’s basement, or maybe a flying-saucer-cult type of youth group led by a charismatic leader.

As it was, without Kane we spent a very productive afternoon working on the theory and practice of liberal fascism.

173

SG 01.13.08 at 3:00 pm

Not really so productive John. Nobody used the phrase “you’ll be the first against the wall when the revolution comes.” And for not using that phrase, I vote you first against the wall.

Wait. Did I say “vote”?

174

Brett Bellmore 01.13.08 at 4:07 pm

“I don’t see much of a point in comparing the number of casualties caused by a civil war with that caused by an unprovoked aggression.

Civil war is pretty much an act of god, one could as well include influenza epidemic or tsunami. Accident statistics is one thing, criminal statistics is another.
Posted by abb1”

My God, that’s morally obtuse. Civil war as an act of god? Civil war is people fighting people, not a big wave drowning people. And, does this REALLY need pointing out: Civil war is just as capable as war between countries of involving unprovoked aggression on one side or both.

My criticism of the Lancet study is unchanged: You poll people in a war zone like Iraq, the people you’re asking questions of aren’t trying to recollect the truth, they’re trying to figure out what utterance is least likely to cause a death squad to visit their house in the middle of the night.

You can calculate error bars due to sample size, due to renormalizing, and all sorts of things. But there’s no mathematics for, “What should I say to keep my family from getting killed?” Pollsters need some more humility about the limits of their craft.

175

John Emerson 01.13.08 at 4:39 pm

I actually agree with Brett that the Lancet author’s error was thinking that Science could contribute much to the American political debate in this case.

Not mostly for the reasons Brett gave, but mostly because the American political discourse is so batshit crazy that for a significant demographic the Lancet study is much less authoritative than the prophecy in Revelation 12 telling us that a seven-headed dragon will rise from the sea.

176

ChiLois 01.13.08 at 4:40 pm

http://news.nationaljournal.com/articles/databomb/index.htm
Where does this write up ‘fit in’?

177

ChiLois 01.13.08 at 5:02 pm

Data Bomb By Neil Munro and Carl M. Cannon, National Journal

Wow, very inclusive article. This was first questionalble issue:
“Still, the authors have declined to provide the surveyors’ reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent — and repeatable — to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?”

Check out the rest – doesn’t look good for the Lancet studies.

178

abb1 01.13.08 at 5:10 pm

No, Brett, in a typical civil war millions of people living in the same geographical area are fighting each other. The reasons are usually structural and there’s nothing you can do about it; it’s a destiny.

Giving the order to invade is a completely different story.

Unless, of course, you’re a proponent of the hard-determinist view, in which case everything is destiny.

179

engels 01.13.08 at 6:21 pm

David, are you going to answer SG’s question or not?

180

John Emerson 01.13.08 at 6:28 pm

No.

181

David Kane 01.13.08 at 7:59 pm

My Zombie Army is hard at work on the response rate issue. More to come!

182

John Emerson 01.13.08 at 8:36 pm

And, by the way, I am still working on a reply to Kieran’s post from 2006 on my fraud discussion. My rough draft of that reply is here.

We should expect a rough draft of David’s work on the response issue in mid 2009.

183

dsquared 01.13.08 at 8:43 pm

David, you may consider the common English word “Banned” to have its normal meaning. For the record, I do not agree that Lancet discussion threads are worse without you.

184

engels 01.13.08 at 9:26 pm

David, if you’re having trouble getting started, why don’t you use your piece on Burnham et al as a template? You’d only have to change the title slightly: IFHS The Case for Fraud sounds pretty punchy to me. Then you could probably adapt a lot of the copy you wrote for the old one; or if you’re really pressed for time, just paste some it in verbatim. Like this:

I can not find a single example of a survey with a 99%+ response rates in a large sample for any survey topic in any country ever.

Or this:

We know very little about these Iraqi teams. Besides monetary incentives to give the … authors the answers they wanted, the Iraqis may have had political reasons as well. … How can anyone know that they are telling the truth?

See? It won’t take you very long at all once you get down to it.

185

John Emerson 01.13.08 at 9:30 pm

Engels, if you want something done right, you have to do it yourself. Just write it up and post it here under his name.

186

Donald Johnson 01.13.08 at 10:44 pm

“I claimed that “it is not that common to have such vehement disagreement in the elite literature.” Which part of “not that common” do you fail to understand?”

I fail to understand the part where it is at all relevant to any point you wish to establish. It’s common to have controversies in science–it’s not exactly surprising to find that it is difficult to measure mortality in a war zone. You informed me in your usual condescending tone that it is rare to find disagreements in the elite literature. Tell me how many papers have been published where people try to determine death rates inside a country plagued by death squads, terrorists, and an occupying power that isn’t terribly concerned about the innocence or guilt of the people it arrests. How common are disagreements under those circumstances?

187

Donald Johnson 01.13.08 at 10:45 pm

Sigh. First paraagraph was David, second was me.

188

John Emerson 01.13.08 at 11:13 pm

A country in which none of the players is at all interested in having accurate mortality statistics, and in fact all of them seem actively not to want them.

189

engels 01.14.08 at 12:57 am

How about this for a first draft?

A Case for Fraud [Generic Version]

David Kane, Harvard

The latest… survey of Iraqi mortality… has come in for criticism…

Assume, for a moment, that fraud occurred. How is it most likely to have happened? We can be fairly certain that the editors and authors did not do anything so crude as to lie about the numbers. If there is fraud, it derives from the Iraqi survey teams themselves…

We know very little about these Iraqi teams. Besides monetary incentives to give the… authors the answers they wanted, the Iraqis may have had political reasons as well… The field manager… could be the most honest and disinterested scientist in all the world. Or he could be a partisan hack. There is almost no way for outsiders to judge… How can anyone know that they are telling the truth?…

I can not find a single example of a survey with a 99%+ response rates in a large sample for any survey topic in any country ever… Assume, for a moment, that there are no such examples… If so, there are three possibilities.

1) The survey teams provided fraudulent data.

2) There is something different about this survey team or about Iraq at this time which makes this situation different from any other survey ever undertaken.

3) The high response rate is a once-in-a-life-time freak event. It would not be repeated even if the same survey team took another survey.

I do not think that 2 or 3 are very likely. Fraud in surveys, on the other hand, is all too common.

190

John Emerson 01.14.08 at 2:08 am

Wrap it up and send it in, Engels! (I mean David). Ya got a winner there!

191

Brett Bellmore 01.14.08 at 2:36 am

“No, Brett, in a typical civil war millions of people living in the same geographical area are fighting each other. The reasons are usually structural and there’s nothing you can do about it; it’s a destiny.”

Actually, I’m a “soft” determinist; We’re every bit as determinist as “hard” determinists, we simply regard the fact that the internal states of people play a part in determining their actions as essential to holding them responsible for those actions, rather than somehow exculpatory.

And I flatly refuse to describe the decisions of people as “acts of god”. Structural considerations be damned, they’re one by one deciding whether or not to murder.

192

SG 01.14.08 at 3:18 am

I see your point Brett but I have a suspicion it doesn’t hold in situations where civil wars start. Your neighbours start shooting you, you start shooting back.

I think what Abb1 is trying to say is that, if someone murders you, the arbitrary nature of their actions – and your complete lack of control over them – make it like an act of god. Similarly when people decide to start executing everyone whose surname starts with B. They may have made a conscious decision, but to you it might as well have been an act of God. And as a consequence your actions are out of your control to some extent.

193

David Kane 01.14.08 at 4:50 am

Another member of my Zombie Army is hard at work. Did anyone see how Rebecca Goldin described the case for fraud as “convincing?” Perhaps Kieran would like to read her the riot act as well.

194

SG 01.14.08 at 5:50 am

David, “Person A made a convincing argument that B may have happened” and “B may have happened” are different conclusions. You do know that don’t you?

Why would the data collectors lie about a violent death and then report they had not seen the death certificate? Wouldn’t they lie about both those things? Isn’t it more likely that the respondents lied about a violent death and refused to reveal a death certificate, or didn’t have it?

The obvious charitable interpretation of this is that the Lancet study may misattribute causes of death due to the survey respondents being unwilling to tell the truth. That’s what a reasonable observer would do (and what Ms. Goldin appears to be doing in your quote, without actually saying the National Journal are wankers).

But if the Lancet study did misattribute causes of death, that simply means that the violent death subcategory of the total excess deaths is also confirmed by the IFHS study. Which is why, I presume, you choose to blame the study authors for fraud.

195

SG 01.14.08 at 5:56 am

Also David, you do realise what the veiled implication of this paragraph is, don’t you?

If those looking for fault in the Lancet study only considered a few possible ways in which the data didn’t look random, then unusually distributed data is far more damning than if they considered many, many ways and found one.

It is suggesting that you found what you wanted to see.

196

Walt 01.14.08 at 6:19 am

I went and read Goldin’s post, and _shockingly_ David Kane misrepresented it. How do this keep happening to you, David? Are you just unlucky?

197

Brett Bellmore 01.14.08 at 12:05 pm

SG, the whole reason it’s called an “act of god” is because there isn’t a person acting. “God” stands in the place of that lacking person. Picking up a rifle, pointing it at somebody, and pulling the trigger, is definitively NOT “an act of god”. It’s an act of the person holding the rifle.

This isn’t a minor error, it’s standing the concept on it’s head.

“Your neighbours start shooting you, you start shooting back.”

Known as “self defense”, not “an act of god”.

198

abb1 01.14.08 at 1:06 pm

But it’s not person’s acting, when a million of persons are acting. It becomes a statistical certainty.

Take Bosnia. They were a part of Yugoslavia, everything was fine. Then a majority voted to secede from Yugoslavia while a significant minority strongly felt they shouldn’t secede. So, naturally, they started fighting. This is not a person’s doing; there’s no single individual or a group of individuals to blame here, it’s the matter of social, religious, ethnic and other dynamics.

199

Donald Johnson 01.14.08 at 1:18 pm

Apparently the National Journal claims about where missing death certificates occurred are wrong–

http://www.jhsph.edu/refugee/research/iraq/national_journal.html

Taking the NJ claims as true, though, I can think of reasons why interviewers might normally be more likely to remember to ask for death certificates for violent deaths (they’d be seen as more important) and why, in certain circumstances, they might not. Perhaps on one day they were just forgetful. Or maybe in some clusters there was an intimidation factor at work–they might have stumbled into a hotbed of insurgent or militia activity. This is all assuming it isn’t simply coincidence.

I noticed, btw, that in the National Journal article they used the 57,000 violent death figure for Lancet1 for the period up to Sept. 2004. IFHS said there were roughly 120 deaths per day throughout the first 3 years, so that would be, hmmm, about 60,000 deaths for the same time period. It’s almost as though the two studies agree, according to the National Journal article, though they didn’t know this, since the NEJM paper came out too late. Though somehow I suspect that if they’d published their article a few weeks later, someone might have told them to leave out that IBC vs. L1 comparison, preferably in favor of one that included the Fallujah outlier.

200

David Kane 01.14.08 at 3:42 pm

I “misrepresented” Goldin? Hmmm. Am I imagining that she used the words “tweaked” and “inventiveness” in discussing her concerns about the data?

201

Donald Johnson 01.14.08 at 3:53 pm

To be fair, Goldin says the evidence for “tweaking” is convincing and then she spends the final portion of her piece pointing out why it may not be convincing after all.

202

Walt 01.14.08 at 4:39 pm

David, did you _read_ Goldin’s post, or just the one sentence containing the words “convincing” and “tweaked”? In the rest of the post, she politely points out that any identified statistical anomoly is probably the result of data mining: if you look over any data set long enough you will find an anamoly. Any statistical inference that you do that does not correct for the fact that you looked at the data first to find an anomaly to test for is invalid, and unless you adjust your test statistics. Did you mention _any_ of that in your comment? What’s that? No you didn’t? Hmmm, I think that’s what colloquially known as “misrepresenting”.

David: There could be something wrong with the Lancet article; I’m prepared to hear a case to that effect. Maybe some day someone will make a case that Roberts et al are partisan and dishonest. But there is one participant in these debates who has already demonstrated themselves to be partisan and dishonest. That person is you, David.

203

SG 01.15.08 at 12:26 am

Brett, I know what the phrase “act of God” actually means. I’m merely pointing out that functionally (from the point of view of the dead person) the complete lack of control over the event means they can be seen as essentially equivalent. So when lots of people set out to try and kill you, it seems like an act of God even though clearly it is not. Sure it’s twisting the definition, but so what? You’re playing the editor because you want to pretend Abb1 is just another lefty who refuses to blame the invidivual.

David, the rebuttals of your opinions are coming so thick and fast you are losing track of them. You do know that “makes a convincing argument for A” and “proves A is true” are different, don’t you? You are aware that saying “some people think my nose was tweaked” and “my nose was tweaked” are different statements, don’t you?

204

John Emerson 01.15.08 at 12:40 am

See, we’re lost in the weeds now. That is what rational people call rational discussion.

But I’m not a rational person. I’m a liberal fascist! And I say, “Fuck this shit!”

And do I get invited to speak on NPR? No!

Does Jonah Goldberg? Yes!

Because he’s a rational person, and rational people speak rationally on NPR.

205

Someguy 01.15.08 at 5:55 am

Hey, Zizka! Did you ever find out if Dubya molested corpses in college?

206

Order of Magnitude 01.15.08 at 6:05 am

You all seem to ignore the uncanny timing of both Lancet study, shortly before US elections. The friendly treatment accorded to the papers (including the expedited review) at a minimum suggests collusion between the authors and the editors, with the intent of influencing the outcome of the elections.

207

Order of Magnitude 01.15.08 at 6:05 am

…Lancet studies …

208

SG 01.15.08 at 6:52 am

om, go read the comprehensive trashing of your theory over at Deltoid.

209

Donald Johnson 01.15.08 at 12:16 pm

#205–

Why would this matter even if it were true? Everyone who counts the dead in Iraq has a political motivation and hopes to have some kind of policy impact, even if not on a specific election. Are they supposed to do it because counting war dead is a hobby?

210

Alex 01.15.08 at 2:37 pm

phantom US bombing raids

I’m reasonably sure the McDonnell-Douglas F-4 Phantom II was not in use over Iraq; oh, that wasn’t what you meant?

Maybe this was it:

BAGHDAD, Jan 10 (Reuters) – U.S. warplanes dropped 40,000 pounds of bombs on more than 40 targets on Baghdad’s southern outskirts on Thursday in a major strike on al Qaeda safe havens, the military said in a statement.

The U.S. Air Force dispatched two B-1 bombers and four F-16 fighter jets, aiming at three large target areas in Arab Jabour…

211

Order of Magnitude 01.16.08 at 2:04 am

sg #208: The deltoid guy is obsessive in his defense of the Lancet studies and the people behind them.

#209: “everyone […] has a political motivation”
You see, the Lancet studies were not some liberal arts theoretical construct. They were published in a medical journal and assumed the facade of medical methodology. If you are employed (or stockholder or spouse of employee or stockholder, etc etc) and you write a medical article you have to disclose your ‘conflict of interest’. This is akin to liberal intellectuals having to post disclaimers such as “I subscribe to, and write in The Nation, donate money to left wing causes, volunteer for left political campaigns” or whatever. Now, after such a discloseure it would be funny to read said intellectuals’ “impartial” views” on things. We could start with CT.

So yeah, it’s OK to have a political view about Iraq, about the excess deaths there, or anything else, but in medicine you’d have to disclose those biases. They should have written an oped in the Guardian, instead of pretending to write for a medical audience — which they obviously were not.

Second, in medicine expedited reviews are rare. I have done peer reviews — many academic physicians do — and they are quite leisurely. Yet BOTH of these studies were timed to precede US elections, and at least one AFAIK got an expedited review. I’d like to see an audit of the L editor’s decision process to expedite this review.

212

Donald Johnson 01.16.08 at 3:58 am

In public health it’s assumed people are opposed to high mortality rates.

213

SG 01.16.08 at 4:24 am

oom, I have published in public health journals and have never had to declare a political interest. only financial. perhaps you think the Lancet authors were employed by the insurgency? (No actually, don’t answer that).

Also, expedited reviews are rare but not wrong or sinister, and occur for topical and important issues. Perhaps you aren’t aware, but the Iraq war is both.

Alex, I think that Kane is referring to a single death which apparently occurred due to coalition bombing before March 2003. This is the origin of the snarky “phantom bombing raids”. Kane seems to be the only person on earth unaware of the continuous bombing of Iraq that happened before March 2003. Given he doesn’t know that, mentioning types of plane will probably just confuse him further…

214

Walt 01.16.08 at 4:28 am

oom, I can only assume that you haven’t read the Lancet article. They are obviously written for a medical audience.

215

Order of Magnitude 01.16.08 at 6:45 am

walt #214: are you a physician?

Comments on this entry are closed.