More Burnham et al.

by Kieran Healy on October 16, 2006

Here are some comments from “Andrew Gelman”: on the “Burnham et al. paper”: People who’d like (or ought) to learn more about statistics could do worse than read Gelman and Nolan’s terrific Teaching Statistics: A Bag of Tricks. I am slightly embarrassed to admit that I am awaiting the publication of Gelman and Hill’s Data Analysis Using Regression and Multilevel/Hierarchical Models with a degree of anticipation that seems indecent (or unhealthy) to direct at a statistics textbook. (More about the book “here”: Note the blurb from a well-known blogger.)



Barry 10.16.06 at 8:06 pm

I posted condolences on his blog; the innumerates will be swarming in shortly.


C. L. Ball 10.16.06 at 9:55 pm

My stats is rusty, so is there any reason to be bothered by the fact that the Roberts, et al 2004 Lancet paper found that airstrikes were the main cause of violent death in the March 03-Sep-04 period but the Burnham, et al 2006 Lancet study found that gun-shots were the lead cause for the March 03-April-04 and the other periods as well?


P O'Neill 10.16.06 at 10:12 pm

I probably shouldn’t even pollute this thread with the following, but masochists should note that statistical expert Christopher Hitchens is disputing the Burnham et al application of the Central Limit Theorem in Slate.


rupes 10.16.06 at 11:49 pm

Another expert writing an opinion on the survey from a distinguished Epidemiologist

Francesco Checchi, an epidemiologist at the London School of Hygiene and Tropical Medicine, looks at the lambasting a new report on Iraq deaths has got from hostile governments. He has worked on mortality surveys in Angola, Darfur, Thailand and Uganda, and written a publication “Interpreting and using mortality data in humanitarian emergencies” for the Humanitarian Practice Network.


Daniel 10.17.06 at 2:08 am

Note that Hitchens’ argument appears to rely upon pretending that a) the years 1995-2000 did not happen or b) that oil-for-food did not benefit the Iraqi people.


Anatoly 10.17.06 at 2:46 am

What I’m really curious about is whether any of the CT authors are going to provide, in any of the posts inevitably yet to appear on the Lancet study, an intelligent reply to the Iraq Body Count press release. So far, quietly ignoring it (after several comments provided a link to it in the spirited discussion of John Quiggin’s post) seems to be the thing to do.

Which is really a pity, because to me it seems to shatter the study’s credibility rather dramatically. But what do I, neither a statistician nor an expert on Iraq, know? Next to nothing; that’s why it’d be helpful for hear the CT authors’ take on this.


snuh 10.17.06 at 3:06 am

forget innumeracy, and start with comprehension. according to hitchens, academic journals are now taken to hold not only the views expressed in the studies they publish, but also the views that may be expressed in letters they happen to receive:

In December 1995, the Lancet published another equally disturbing document, this time a letter to the editor from Sarah Zaidi and Mary C. Smith Fawzi.

But it does seem, according to the Lancet, that hundreds of thousands of Iraqis were doomed to die, one way or another, in peace or in war, unless Saddam was left unmolested.

also, at the end of the article, the following editorial note appears:

Related in Slate: …Fred Kaplan discredited an earlier report.


rupes 10.17.06 at 3:35 am

Anatoly (#6)

I agree – I’d like to see some discussion of it too. Although their may be some picque (“we do the reprting round here!”), IBC are hardly war justifarians so their comments are hardly biassed and should be treated with respect.

The only substantive discussion I have seen on it is from Lenin’s Tomb

That seems a little overblown in its language, but does have some good points.

A lot of IBC’s logic is a mix of the “I don’t believe it so it isn’t true” or circular “it wasn’t in the press so it didn’t happen, and the proof it never happened was that no-one reported it”.

A significant remark in The Lancet: The Lancet report notes in its conclusion that: “Other than Bosnia, we are unable to find any major historical instances where passive surveillance methods (such as morgue and media reports) identify more than 20% of the deaths which were found through population-based survey methods.”

On the other hand…

I too would like more discussion and maybe that will help resolve the different opinions.


Harald Korneliussen 10.17.06 at 3:39 am

anatoly: the bits of the IBC press release I read made exactly the same mistakes as I’ve seen explained dozens of places already, like assuming that death certificate issuing information proceeds flawlessly upwards in the bureucracy despite poor administration, corruption, lack of reliable computer systems and that ongoing civil war thing. I think they did the one about comparing crude mortality rates, too, forgetting that Iraq has a very young population compared to us. Age is still an important cause of death in all parts of the world)


Daniel 10.17.06 at 4:16 am

On the subject of the IBC press release, I have no substantial disagreement with Tim Lambert’s latest post. I’d note that the death rate by violence implied by the IBC total would be roughly the same as the murder rate of Jamaica last year.


Barry 10.17.06 at 6:41 am

C. L. Ball, probably due to sample size. As you will not recall, figures for sub-samples are less accurate than figures for the overall sample. This would be reflected in confidence interval widths (assuming that they gave them for those sub-rates).


Barry 10.17.06 at 6:41 am

Sorry, I meant to say ‘no doubt will recall’.


Barry 10.17.06 at 7:15 am

Anatoly, I’m very surprised that you accepted the IBC ‘rebuttal’. After all, it’s very well known among anybody with an interest in this, that the IBC counts *media reported* deaths. Therefore, the second that they compare the Lancet article rates to their own, as if they were comparable, you would have, of course realized that something was fishy.


tim 10.17.06 at 7:30 am

Do you have any figures for how the OFF programme benefitted the Iraqi people in terms of mortaity?


C. L. Ball 10.17.06 at 12:45 pm

Tim, if you got to the Lancet archives there was a study which found that mortality varied from province to province. The Kurdish areas had improved mortality and the Shi’ite areas had much worse mortality.


Anatoly 10.17.06 at 5:55 pm

Barry, I think IBC never stated that their counts are anything but underreported. The question is by how much.

Daniel, I didn’t see any argument in Tim Lambert’s latest, which is basically a large quote from a rather inane blogger. “Lenin’s Tomb” starts by impugning motives to IBC without any evidence; the rest of his post is a pooh-poohing of the IBC’s argument which amounts to “it’s an argument from incredulity, nyah nyah”.

Yes, IBC’s press report is mostly an argument from incredulity, but such arguments aren’t necessarily invalid or useless. Incredulity is hard to assess objectively, but IBC are making a good job of showing that Lancet’s figures imply conclusions that are too improbable to not have been noticed independently of the study.

The other part of Lambert’s latest is a quotation from a different blogger’s attack on IBC, Alon Levy’s Abstract Nonsense. Unfortunately, Levy’s engaging in, well, nonsense, because he’s quoting IBC’s earlier editorial, which was written before the new Lancet study, and not the recent IBC press release. When IBC talk about “the Lancet study”, in that editorial, they mean the 2004 study (and they reference it explicitly, of course), while Levy takes them to be talking about the 650,000 study. He quite literally doesn’t know what he’s talking about, and it’s embarrassing for Lambert to quote him approvingly without noticing this very basic mistake.

So, no satisfactory reply to the IBC press release yet (from what I’ve seen).


C.L. Ball 10.17.06 at 10:54 pm

Even if the Burnham et al estimate overstates the actual violent deaths, the question then becomes, By how much? The fact that no one else is doing serious research on this question and that the Iraqi government has supposedly gagged mortuaries from reporting monthly death totals (can someone confirm this?) makes me think that the Burnham study is closer to the truth than the US, UK, or Iraqi governments want to admit, even if the magnitude of the estimate seems implausible or too horrifying to want to believe it is plausible.


Walt 10.17.06 at 11:09 pm

After Galileo dropped those two balls from the Leaning Tower of Pisa to show that they really fell at the same rate, Anatoly would be the first to blog about how the argument from incredulity refuted the experiment. I mean really, how could two balls of two different weights fall at the same rate?


P O'Neill 10.17.06 at 11:12 pm

Going to Wednesday’s Wall Street Journal and seeing an article by “Steven Moore” on the Burnham et al study, I at first feared that they now declared that someone with an expertise in Laffer Curves could be an expert in statistics. But no, there’s another Steven Moore, who does seem to know a bit more about statistics (notwithstanding the IRI credentials), who makes the following claims inter alia:

— Number of clusters is too small, comparable only to a similar methodology for Kosovo but Iraq is much larger

— No demographic info collected from respondents

— Bizarre responses to seemingly simple questions put to the authors.

So worth a look.


Kevin Donoghue 10.18.06 at 3:18 am

So, no satisfactory reply to the IBC press release yet (from what I’ve seen).


What would a reply have to look like in order to be satisfactory to you? Basically the dialogue so far boils down to this:

Survey Data: the mortality situation Iraq is in the range you would expect to see during a major civil war.

IBC “Reality Check”: No, Survey Data, you’re wrong, we follow the media reports closely and it’s really not that bad at all.

How does one refute the argument from incredulity?


Anatoly 10.18.06 at 7:01 am


Something like this would do:

IBC: it’s incredible that at least 10% of male adult population died (in large areas; 7% overall) and this was not noticed by local media, international media, observers, Iraqi bloggers, officials, whatever. A decimation of the male adult population around you is an astounding event that doesn’t happen very often. Yes, Iraq is a hellhole, but according to all observers on the ground it doesn’t seem to nearly be such a hellhole as that.

An adequate reply to IBC: your argument from incredulity here is bull. Here (1,2,3,4) are examples of civil wars/other conflicts/whatever where the losses are now agreed to have been around 10% or more of adult male population, but at the time nothing of the kind was claimed by observers/agencies on the ground.

IBC: 800,000 people, conservatively estimated, received serious injuries and only 10% of them ever got any kind of reported hospital treatment, with hospitals largely continuing to function throughout the country? Considering that the hospitals are strongly motivated to report the injuries, to maintain funding? Can’t be.

Adequate reply to IBC: what nonsense. You’re factually wrong about the funding issue; here’s evidence that hospitals do not in fact report injuries on a large scale; here’s evidence that your estimate of number of wounded is seriously overblown; here’s evidence that in X% of the territory there’s no real access to hospitals; here’s historical evidence about other conflicts/civil wars in which the number of treated is in huge discrepancy with the agreed number of seriously wounded, despite working hospitals; etc. etc.

And so on, and so forth.


Kevin Donoghue 10.18.06 at 8:01 am


All of those points have been covered. Check posts here and at Deltoid and follow the links. To take just one of the niggles: why aren’t wounded people showing up at hospitals, or why is the Ministry of Health not reporting injuries if they are showing up? Look here and here for possible answers.


Anatoly 10.18.06 at 8:26 am


I looked at Deltoid and found that he endorses “rebuttals” based on inability to distinguish one IBC report from the other, and Lancet 2006 from Lancet 2004. The arguments are basically illiterate. The evidence is above in this thread. I’ve seen the two links you provide – they’re anecdotal, it’s not clear to how many hospitals in the country they apply, and the problem described there is explicitly stated to have begun in December 2005, so it would affect at most 7 months of the 40 covered by Lancet 2006. Moreover, if these horrible incidents really affected the population’s willingness to go to hospitals on a large scale, we’d see the difference in the statistics provided by the hospitals, between before December ’05 and after, would we not?


Barry 10.18.06 at 8:36 am

“How does one refute the argument from incredulity?”

Posted by Kevin Donoghue

Short of parachuting the incredulous into Iraq for a short but educational vacation, one points out their errors, and the fact that their ‘incredulity’ is either willful refusal to belief, or dumbf*ck stupidity. When the bystanders are sufficiently convinced that Mr. ‘My Gut Says Otherwise’ is indeed a liar or a fool, then one is done.

By now, it’s clear that there is no innocent denial of this study. There might be valid, genuine professional objections, but they don’t seem to be available on weblogs (they’re probably exchanged in e-mails between professionals).


Detlef 10.18.06 at 9:40 am


I´m not an expert in statistics.
But from what I´ve read in newspapers it looks like Iraq Body Count and the two-monthly UN reports badly underreport deaths.
In case of the Iraq Body Count, the reason is clear. They rely on media reports in English. I doubt there are that many reporters out there looking. So it is pretty clear that their number is too low.
The UN reports rely on Iraq Health Ministry reports. And it looks like Sadr´s people aren´t that interested in reliable numbers.

See for example:
UN report July/August 2006 (pdf-file)

The number of civilians killed by violence is calculated adding the number of casualties reported by the Ministry of Health, which includes reports from all hospitals in Baghdad and other Governorates excluding the Region of Kurdistan, and the reported number of bodies brought to the Medico-Legal Institute in Baghdad. Only a small fraction (between 5-6 %) of the latter figures may be attributed to causes other than violence. The Ministry of Health reported zero number of killed in Al-Anbar for July, which may indicate an under-estimation due to difficulties experienced in collecting information in that particular Governorate.

Death Squads in Iraq Hospitals

The takeover began after the last election in December when Sadr’s political faction was given control of the Ministry of Health. The U.S. military has documented how Sadr’s Mahdi Army has turned morgues and hospitals into places where death squads operate freely.

The chilling details are spelled out in an intelligence report seen by CBS News. Among some of the details of the report are:

* Hospitals have become command and control centers for the Mahdi Army militia.

* Sunni patients are being murdered; some are dragged from their beds.

* The militia is keeping hostages inside some hospitals, where they are tortured and executed.

* They’re using ambulances to transport hostages and illegal weapons, and even to help their fighters escape from U.S. forces.

Iraqi Hospitals Are War’s New ‘Killing Fields’

Authorities say it was not an isolated incident. In Baghdad these days, not even the hospitals are safe. In growing numbers, sick and wounded Sunnis have been abducted from public hospitals operated by Iraq’s Shiite-run Health Ministry and later killed, according to patients, families of victims, doctors and government officials.

As a result, more and more Iraqis are avoiding hospitals, making it even harder to preserve life in a city where death is seemingly everywhere. Gunshot victims are now being treated by nurses in makeshift emergency rooms set up in homes. Women giving birth are smuggled out of Baghdad and into clinics in safer provinces.

Insecurity, under-funding threaten children’s health in Basra

A concurrent shortage of doctors and nurses has also been reported in Basra. According to Abdullah, there are no reliable statistics on how many doctors, dentists, pharmacists and nurses have left the area, but unofficial data suggests that at least 200 health professionals have left since January. Health ministry statistics also suggest that an average of 30 doctors and nurses per month have left Iraq over the past year after being targeted by criminal gangs.

The emergency unit in the Teaching Hospital was closed for five months after a number of doctors were killed by unidentified attackers while working there. Now many doctors and nurses refuse to go to work, fearing for their lives.

UN report May/June 2006

25. The Medical Association in Mosul informed UNAMI HRO that, since April 2003, at least 11 doctors have been killed while another 66 have left the city. In Mosul, on 8 May 2006 at 18:00 hours, unknown gunmen arrived in two private cars to the Al-Zayzafon pharmacy, opposite Al-Khansa hospital in Al-Sukar district. The men took the pharmacist Fadhel Ezalddin Nidham and executed him in public before setting the pharmacy alight. Also in Mosul, on 15 May, in Garage Al-Shemal area unknown gunmen assassinated Dr. Adnan Abbas Al-Hashemy after he was leaving his private clinic. Two other doctors are said to have been killed the same week in Mosul.

27. As a result of the violence, many health workers continue to leave the country or to relocate to safer areas. In western regions of Iraq, where ongoing military operations have resulted in increased number of casualties, hospitals have reported lack of adequate supplies, military surveillance of medical facilities, intimidation and harassment of medical personnel. In general, health workers have failed to receive adequate protection during military operations and they have been unable to carry out their work in safety.

So your second point, hositals are largely functioning and reporting accurate numbers to the Health Ministry, looks pretty weak.
First, because of Shia militia influence as mentioned above. And second, because large numbers of doctors and nurses (the educated people) have fled, refuse to work, are dead.
Which means that the wounded are probably undercounted.

All in all, “Deaths are occurring in Iraq now at a rate more than three times than from before the invasion of March 2003,” Dr Gilbert Burnham, lead author of the study, said in a statement.
That statement doesn´t strike me as unlikely.


Anderson 10.18.06 at 10:40 am

argument from incredulity

Hm, so I can prove that the invasion of Iraq never actually happened?


Drm 10.18.06 at 11:43 am

Gelman’s main point that clusters are in effect the unit of measurent has not been picked up on as far as I can tell in this discussion. As he notes, there is little reason not to breakout the cluster means, etc in the paper or in attached supplementary data.

In any case, it’s not obvious whether or not 47 clusters is sufficient for a population as structured and complex as Iraq’s. The fact that Moore (WSJ oped) evidently favored similar cluster numbers for his own (apparently politically motivated) surveys is not encouraging.


Bobcat 10.18.06 at 12:31 pm

Here’s another skeptical paper on the Lancet Study:

It’s written by Jeffrey and Loring White. The latter “has had a long private- and public-sector career in the fields of mathematical modeling and scientific data analysis, specializing in the evaluation of high-uncertainty information.”

The methodological critiques are here:

“First, the study compares results from three fourteen-month periods after the invasion to results from only one fourteen-month period before the invasion—and that period is the one immediately preceding the invasion, when the Iraqi regime was preparing for war. How representative of violent activity in Iraq is that one preinvasion period? Would looking at three periods before the invasion improve the study’s credibility? Second, only two violent deaths were reported in the survey data during the fourteen months prior to the invasion. This seems questionable considering the regime’s prewar security operations and the low-level insurgency being waged in the south. Is this very low baseline of violence, in a regime known for violent activity, plausible? Third, it would be useful to know more about how representative the study’s random sampling process was. Where were the clusters of surveyed households located, and what were the actual numerical results for each cluster? If violent deaths are positively correlated with population density, would random selection of cluster locations proportionate to population size cause an upward statistical bias in the results (when projected over the country as a whole)? Finally, no mention is made of public access to the body of data collected. Public access would allow other experts to attempt replication of the results—a critical requirement for scientific credibility.”


Barry 10.18.06 at 4:33 pm

Anatoly: “Yes, Iraq is a hellhole, but according to all observers on the ground it doesn’t seem to nearly be such a hellhole as that.”

Prove that. You’ve asked for proof of this, that and the other thing; it’s your turn now.


SG 10.19.06 at 1:50 am

Regarding the hospitals, I don`t think you can use the argument of them being killing fields to counter the IBC comments, because in most cases the Lancet survey reported death certificates, which presumably have to be issued in the presence of the body.

The question about the role of hospitals can be initially answered with this simple question: In Iraq, do you need to go to a hospital to get a death certificate? Islam requires the body be buried quickly, so maybe local doctors can give death certificates?

If so, most questions about the role of Hospitals are answered and or irrelevant. If not, the only explanation for the discrepancy the IBC discusses seems, to me, to be that the hospitals aren`t reporting to the govt.

If any journalist in Iraq cared about the truth they could find this out very quickly.


Alex 10.19.06 at 7:16 am

Hell, I still find it hard to believe that the British government was stupid enough to invade Iraq for no apparent reason. Does it really pass a gut check that the Labour Party, the Foreign Office’s Middle Eastern experts, the General Staff, MI6, Defence Intelligence and more than half the media-political complex would get involved in a futile, murderous and open-ended guerrilla war and destabilise the whole Middle East?

So clearly it didn’t happen.


Brownie 10.20.06 at 7:26 pm

So clearly it didn’t happen.

That “whooshing” sound was your own irony passing over your head.

Comments on this entry are closed.