More Burnham et al.

I posted condolences on his blog; the innumerates will be swarming in shortly.

I probably shouldn’t even pollute this thread with the following, but masochists should note that statistical expert Christopher Hitchens is disputing the Burnham et al application of the Central Limit Theorem in Slate.

Note that Hitchens’ argument appears to rely upon pretending that a) the years 1995-2000 did not happen or b) that oil-for-food did not benefit the Iraqi people.

forget innumeracy, and start with comprehension. according to hitchens, academic journals are now taken to hold not only the views expressed in the studies they publish, but also the views that may be expressed in letters they happen to receive:

In December 1995, the Lancet published another equally disturbing document, this time a letter to the editor from Sarah Zaidi and Mary C. Smith Fawzi.

…

But it does seem, according to the Lancet, that hundreds of thousands of Iraqis were doomed to die, one way or another, in peace or in war, unless Saddam was left unmolested.

also, at the end of the article, the following editorial note appears:

Related in Slate: …Fred Kaplan discredited an earlier report.

anatoly: the bits of the IBC press release I read made exactly the same mistakes as I’ve seen explained dozens of places already, like assuming that death certificate issuing information proceeds flawlessly upwards in the bureucracy despite poor administration, corruption, lack of reliable computer systems and that ongoing civil war thing. I think they did the one about comparing crude mortality rates, too, forgetting that Iraq has a very young population compared to us. Age is still an important cause of death in all parts of the world)

C. L. Ball, probably due to sample size. As you will not recall, figures for sub-samples are less accurate than figures for the overall sample. This would be reflected in confidence interval widths (assuming that they gave them for those sub-rates).

Anatoly, I’m very surprised that you accepted the IBC ‘rebuttal’. After all, it’s very well known among anybody with an interest in this, that the IBC counts *media reported* deaths. Therefore, the second that they compare the Lancet article rates to their own, as if they were comparable, you would have, of course realized that something was fishy.

Tim, if you got to the Lancet archives there was a study which found that mortality varied from province to province. The Kurdish areas had improved mortality and the Shi’ite areas had much worse mortality.

Even if the Burnham et al estimate overstates the actual violent deaths, the question then becomes, By how much? The fact that no one else is doing serious research on this question and that the Iraqi government has supposedly gagged mortuaries from reporting monthly death totals (can someone confirm this?) makes me think that the Burnham study is closer to the truth than the US, UK, or Iraqi governments want to admit, even if the magnitude of the estimate seems implausible or too horrifying to want to believe it is plausible.

Going to Wednesday’s Wall Street Journal and seeing an article by “Steven Moore” on the Burnham et al study, I at first feared that they now declared that someone with an expertise in Laffer Curves could be an expert in statistics. But no, there’s another Steven Moore, who does seem to know a bit more about statistics (notwithstanding the IRI credentials), who makes the following claims inter alia:

— Number of clusters is too small, comparable only to a similar methodology for Kosovo but Iraq is much larger

— No demographic info collected from respondents

— Bizarre responses to seemingly simple questions put to the authors.

So worth a look.

Kevin,

Something like this would do:

IBC: it’s incredible that at least 10% of male adult population died (in large areas; 7% overall) and this was not noticed by local media, international media, observers, Iraqi bloggers, officials, whatever. A decimation of the male adult population around you is an astounding event that doesn’t happen very often. Yes, Iraq is a hellhole, but according to all observers on the ground it doesn’t seem to nearly be such a hellhole as that.

An adequate reply to IBC: your argument from incredulity here is bull. Here (1,2,3,4) are examples of civil wars/other conflicts/whatever where the losses are now agreed to have been around 10% or more of adult male population, but at the time nothing of the kind was claimed by observers/agencies on the ground.

IBC: 800,000 people, conservatively estimated, received serious injuries and only 10% of them ever got any kind of reported hospital treatment, with hospitals largely continuing to function throughout the country? Considering that the hospitals are strongly motivated to report the injuries, to maintain funding? Can’t be.

Adequate reply to IBC: what nonsense. You’re factually wrong about the funding issue; here’s evidence that hospitals do not in fact report injuries on a large scale; here’s evidence that your estimate of number of wounded is seriously overblown; here’s evidence that in X% of the territory there’s no real access to hospitals; here’s historical evidence about other conflicts/civil wars in which the number of treated is in huge discrepancy with the agreed number of seriously wounded, despite working hospitals; etc. etc.

And so on, and so forth.

Kevin,

I looked at Deltoid and found that he endorses “rebuttals” based on inability to distinguish one IBC report from the other, and Lancet 2006 from Lancet 2004. The arguments are basically illiterate. The evidence is above in this thread. I’ve seen the two links you provide – they’re anecdotal, it’s not clear to how many hospitals in the country they apply, and the problem described there is explicitly stated to have begun in December 2005, so it would affect at most 7 months of the 40 covered by Lancet 2006. Moreover, if these horrible incidents really affected the population’s willingness to go to hospitals on a large scale, we’d see the difference in the statistics provided by the hospitals, between before December ’05 and after, would we not?

anatoly,

I´m not an expert in statistics.
But from what I´ve read in newspapers it looks like Iraq Body Count and the two-monthly UN reports badly underreport deaths.
In case of the Iraq Body Count, the reason is clear. They rely on media reports in English. I doubt there are that many reporters out there looking. So it is pretty clear that their number is too low.
The UN reports rely on Iraq Health Ministry reports. And it looks like Sadr´s people aren´t that interested in reliable numbers.

See for example:
UN report July/August 2006 (pdf-file)

The number of civilians killed by violence is calculated adding the number of casualties reported by the Ministry of Health, which includes reports from all hospitals in Baghdad and other Governorates excluding the Region of Kurdistan, and the reported number of bodies brought to the Medico-Legal Institute in Baghdad. Only a small fraction (between 5-6 %) of the latter figures may be attributed to causes other than violence. The Ministry of Health reported zero number of killed in Al-Anbar for July, which may indicate an under-estimation due to difficulties experienced in collecting information in that particular Governorate.

Death Squads in Iraq Hospitals

The takeover began after the last election in December when Sadr’s political faction was given control of the Ministry of Health. The U.S. military has documented how Sadr’s Mahdi Army has turned morgues and hospitals into places where death squads operate freely.

The chilling details are spelled out in an intelligence report seen by CBS News. Among some of the details of the report are:

* Hospitals have become command and control centers for the Mahdi Army militia.

* Sunni patients are being murdered; some are dragged from their beds.

* The militia is keeping hostages inside some hospitals, where they are tortured and executed.

* They’re using ambulances to transport hostages and illegal weapons, and even to help their fighters escape from U.S. forces.

Iraqi Hospitals Are War’s New ‘Killing Fields’

Authorities say it was not an isolated incident. In Baghdad these days, not even the hospitals are safe. In growing numbers, sick and wounded Sunnis have been abducted from public hospitals operated by Iraq’s Shiite-run Health Ministry and later killed, according to patients, families of victims, doctors and government officials.

As a result, more and more Iraqis are avoiding hospitals, making it even harder to preserve life in a city where death is seemingly everywhere. Gunshot victims are now being treated by nurses in makeshift emergency rooms set up in homes. Women giving birth are smuggled out of Baghdad and into clinics in safer provinces.

Insecurity, under-funding threaten children’s health in Basra

A concurrent shortage of doctors and nurses has also been reported in Basra. According to Abdullah, there are no reliable statistics on how many doctors, dentists, pharmacists and nurses have left the area, but unofficial data suggests that at least 200 health professionals have left since January. Health ministry statistics also suggest that an average of 30 doctors and nurses per month have left Iraq over the past year after being targeted by criminal gangs.

The emergency unit in the Teaching Hospital was closed for five months after a number of doctors were killed by unidentified attackers while working there. Now many doctors and nurses refuse to go to work, fearing for their lives.

UN report May/June 2006

25. The Medical Association in Mosul informed UNAMI HRO that, since April 2003, at least 11 doctors have been killed while another 66 have left the city. In Mosul, on 8 May 2006 at 18:00 hours, unknown gunmen arrived in two private cars to the Al-Zayzafon pharmacy, opposite Al-Khansa hospital in Al-Sukar district. The men took the pharmacist Fadhel Ezalddin Nidham and executed him in public before setting the pharmacy alight. Also in Mosul, on 15 May, in Garage Al-Shemal area unknown gunmen assassinated Dr. Adnan Abbas Al-Hashemy after he was leaving his private clinic. Two other doctors are said to have been killed the same week in Mosul.
…
27. As a result of the violence, many health workers continue to leave the country or to relocate to safer areas. In western regions of Iraq, where ongoing military operations have resulted in increased number of casualties, hospitals have reported lack of adequate supplies, military surveillance of medical facilities, intimidation and harassment of medical personnel. In general, health workers have failed to receive adequate protection during military operations and they have been unable to carry out their work in safety.

So your second point, hositals are largely functioning and reporting accurate numbers to the Health Ministry, looks pretty weak.
First, because of Shia militia influence as mentioned above. And second, because large numbers of doctors and nurses (the educated people) have fled, refuse to work, are dead.
Which means that the wounded are probably undercounted.

All in all, “Deaths are occurring in Iraq now at a rate more than three times than from before the invasion of March 2003,” Dr Gilbert Burnham, lead author of the study, said in a statement.
That statement doesn´t strike me as unlikely.

Gelman’s main point that clusters are in effect the unit of measurent has not been picked up on as far as I can tell in this discussion. As he notes, there is little reason not to breakout the cluster means, etc in the paper or in attached supplementary data.

In any case, it’s not obvious whether or not 47 clusters is sufficient for a population as structured and complex as Iraq’s. The fact that Moore (WSJ oped) evidently favored similar cluster numbers for his own (apparently politically motivated) surveys is not encouraging.

Anatoly: “Yes, Iraq is a hellhole, but according to all observers on the ground it doesn’t seem to nearly be such a hellhole as that.”

Prove that. You’ve asked for proof of this, that and the other thing; it’s your turn now.

Hell, I still find it hard to believe that the British government was stupid enough to invade Iraq for no apparent reason. Does it really pass a gut check that the Labour Party, the Foreign Office’s Middle Eastern experts, the General Staff, MI6, Defence Intelligence and more than half the media-political complex would get involved in a futile, murderous and open-ended guerrilla war and destabilise the whole Middle East?

So clearly it didn’t happen.