Crooked Timber

John,

What are your thoughts on the change in non-violent/violent deaths distribution between Lancet 1 and Lancet 2 (see Lambert and DD’s “Death Certifciates” post below for contributions from Mike H specifically)?

I’m no statistician (no, really I’m not) but Mike H’s observations have been out there now for well over 3 days and I’m not seeeing any credible rebuttals. Which, of course, doesn’t mean there are none.

Skeptics use the gut-check argument from incredulity a lot. The combination of foreign-language problems, the difficulties of collecting information in war time, and (perhaps most important) successful message control by the military (and self-censorship by the media) altogether mean that American intuitions about Iraq are pretty much worthless. (The idea that the gut is a usable source of information is a dubious one in any case, but especially here.)

A compounding factor is that many Bush loyalists and super-hawks would be happy enough if every single Iraqi were killed. They just don’t want the biased liberal media to report about thing like that.

Here are the non-violent deaths as proportions of the survey totals:

Lancet 1 count: including Fallujah (69/142 [49%]) or excluding Fallujah (68/89 [76%]). 988 households were visited.

Lancet 2 count: (247/547 [46%]). 1,849 households were visited.

If I’ve understood the complaint correctly, it’s this apparent drop in non-violent deaths from 76% of the count to 46% of the count that’s causing difficulty. The two studies used somewhat different methodologies; the sample size and period were different. Death certificates were asked for only in some of the Lancet 1 interviews. Given this, and given the specialism of the maths, I think it would take a statistician with access to population data to interpret these figures meaningfully (pace John Quiggin above; I was busy writing this in the meantime). And my gut feeling is that I don’t see any special reason to assume that non-violent deaths would scale with violent deaths. Perhaps infectious disease epidemics caused in part by degraded sanitation would do it: I don’t see any reason to expect an increase in heart disease, which was said to be the leading cause of death pre-war. And if news reports are to be believed, sectarian gun killings seem to be up; is there any reason for these to lead to a concomitant increase in non-violent deaths?

OK, enough of my speculation. Contrary to what some commenters are suggesting, Lancet 1 had this to say about the non-violent death rate:

It is surprising that beyond the elevation in infant mortality and the rate of violent death, mortality in Iraq seems otherwise to be similar to the period preceding the invasion. This similarity could be a reflection of the skill and function of the Iraqi health system or the capacity of the population to adapt to conditions of insecurity.

And Lancet 2 had this to say:

… families might have misclassified information about the circumstances of death. Deaths could have been over or under-attributed to coalition forces on a consistent basis. The numbers of non-violent deaths were low, thus, estimation of trends with confidence was difficult.

These studies have known limitations which the researchers freely admit to, but that doesn’t render them worthless. If you want to discern specific mortality trends at a higher level of accuracy, I think you need to lobby for a better survey. And if you don’t get that from your government, I think you should assume that the news from Iraq is bad.

What astonishes me is that with 800-900 attacks a week, and with two airstrikes a day why would anyone think that a lot of people were not dying?

Surely, if you were a fiscal conservative you ought to be livid with the administration if they were telling the truth: the huge amount of money being spent on ammunition, expensive air-strikes every day. And yet nobody is dead?

Doesn’t it just seem more logical to agree that lots of people are being killed? Precisely as intended when we drop bombs on them.

Mr. Quiggin, I think the concern that brownie has raised is that the previous study had a vastly different proportion of deaths due to violence. I think the argument is that in the first study a very large proportion of excess deaths were due to non-violent causes, while the new study estimates pre invasion deaths to be largely due to violent causes. In response to Brownie I think the explanation is very simple. The original study had a very wide confidence interval, and in the cause of death data (which divides the main data set into smaller groups of counts) the confidence intervals would be even wider. The confidence interval around the number of non-violent deaths in the previous study is probably so wide that it cannot be considered to be statistically significantly different to the figures observed in the new study – i.e. while the point estimates look radically different, the imprecision of the old study prevents any conclusions from being drawn about this.

(I have read the critique of this aspect of the study, which was presented somewhere in comments to Tim Lambert’s blog I think, and this critique was constructed based on the point estimates of deaths only. One needs to consider the errors as well when comparing the old and new studies, and the old study had very wide confidence intervals).

The ony way to get around this problem would be to do a really big study – say, on the scale of the sorts of national surveys which occur in Australia, I think about 5000 households – so that all subgroup analyses (women vs. men, type of death categories by year, etc.) can be performed with narrow confidence intervals. If the governments of the coalition of the willing really cared about Iraqi lives they would have commissioned such a survey using these very techniques several years ago…

I’ve posted this elsewhere, but it’s relevant here.

Over the centuries civil wars have caused hideous carnage. 600,000 Americans died in the US Civil war, more than died in World Wars 1 or 2.

Little Bosnia, with its population of 5 million, managed to slaughter 100,000 in its three year war.

England’s Civil War killed 10% of the populace at the time.

The Spanish Civil War, spookily like Iraq in many ways, killed maybe half a million.

Etc etc.

Given that all other major civil wars have indeed caused deaths in the hundreds of thousands, and killed up to 10% of the populace or more, why should Iraq be different?

Why should Iraq, unique amongs civil wars, have a death rate of only 0.2% of the population, 50,000 people? Rather than the normal 2-10%? Why should Iraq get away with only 50,000 dead, when, looking at history, you should expect 500,000? What is so specially peaceful about Iraq that makes it such a fortunate place?

Is it the lack of widespread violence? Is it because all sides are showing admirable restraint? Is it because the Americans are brilliantly precise in their bombing?

I repeat, we expect 2-10% of a population to die in a civil war, that’s what happens. Yet amazingly enough, according to the war mongers, Iraq has got away with an astonishing 0.2%!!

Truly Iraq is a blessed place. The Lucky Country.

Brownie,

Probably the main reason why the statisticians aren’t rushing to respond to Mike H is that this is really difficult to explain without getting into sampling theory. Robert made a good stab at it in comments at Tim Lambert’s:

The basic issue is that small pieces of larger aggregates are often less well-estimated than the aggregate as a whole. We can pretty much estimate how many deaths occur on the nation’s highways each month, but there will be more variation in how many will occur on any particular day, or on a particular road, or by single-car accident vs. multi-car accident. Day-to-day variation in those counts doesn’t invalidate an overall guess at the total number for all causes for all locations.

http://scienceblogs.com/deltoid/2006/10/lancet_roundup.php

I notice that Mike H has responded to Robert in that thread and I think that’s where you are most likely to see an answer to his question. I will just comment on one of his remarks:

I thought mortality studies were as much about the “how” as they were the “how many.”

The more you want to know about precise cause, the larger your sample has to be.

12: Any chance of seeing some working for those figures? Perhaps a year by year breakdown, because after all, Lancet 1 covers a shorter time frame than Lancet 2. I’m not sure where you’re getting the ‘erases’ and ‘replaces’ from; they sound a bit rhetorical to me.

Kevin:

“The more you want to know about precise cause, the larger your sample has to be.”

And the largest mortality study done thus far is the UNDP effort. We know how its numbers stack up against Lancet 2 for violent (or “war-related” deaths), at least for year 1.

As I’ve said over at Deltoid, I can see allowing some latitude for discrepancy from study to study when it comes to the various subsets of non-violent and violent, but not the two main categories of death themselves. In order to be considered reasonably accurate, I think the studies have to be reasonably consistent in terms of violent and non-violent death.

Otherwise, this seems an awful lot like resurrecting tens of thousands of dead from study 1, only so you can kill them off again in study 2, but in a markedly different way, in order to balance their bottom lines

“It is surprising that beyond the elevation in infant mortality and the rate of violent death, mortality in Iraq seems otherwise to be similar to the period preceding the invasion. This similarity could be a reflection of the skill and function of the Iraqi health system or the capacity of the population to adapt to conditions of insecurity.”

Charlie, I meant to address this earlier, but forgot.

Point taken. I’d forgotten about that statement from the first study.

That doesn’t change the fact that the study authors and defenders of the study attributed the increase in non-violent deaths over the baseline extrapolation (whether by infant mortality, accidents, slight increases in heart disease deaths, etc) to the effects of invasion. I spent many hours arguing with learned defenders of the study, their position being that this increase was entirely expected, given there was a war going on and all.

Now, Lancet 2 tells us that Iraqis were actually less likely to die from natural causes and acidents for the first 2 years of the war.

It doesn’t make any sense.

Bob McManus, apparently unaware that the Hague conventions were signed before the airplane was invented, writes:

I thought aerial bombing of civilian areas by an occupying power was a war crime under the Hague Conventions.

Anyway, the closest that international law comes to banning these types of attacks is in article 51 of Geneva Protocol 1, which bans “indiscriminate attacks…not directed at a specific military objective” or “which may be expected to cause incidental loss of civilian life…which would be excessive in relation to the…military advantage anticipated.”

So I don’t think that these kind of aerial attacks are war crimes, since they’re not indiscriminate.

John Quiggin, on the basis of an article that discusses air attacks at a rate of 35 to 120 per month, concludes that there have been an average of 150 attacks per month. What am I missing?

I also doubt the 10 dead per attack figure. It wouldn’t surprise me if most air attacks killed no one at all. Most military attacks simply miss. The US has shot several billion bullets in Iraq, but they haven’t killed a billion people.

On the other hand, the explanation that most of the Lancet-reported air deaths come from helicopters or artillery makes a lot of sense to me. The US likes to fight at night, and I find it hard to believe that someone who was sleeping when an attack occured has any idea if their loved ones were killed by planes, helicopters, mortars, or what.

I got the impression that in most cases it’s local informers settling scores with their personal enemies by means of US-taxpayers-provided ‘smart bombs’. So it’s not entirely indiscriminate.

“Bob McManus, apparently unaware that the Hague conventions were signed before the airplane was invented, writes:”

Actually I was thinking of the Fourth Geneva Convention

“relative to the Protection of Civilian Persons in Time of War” (first adopted in 1949, based on parts of the 1907 Hague Convention IV)”

but I can’t find the relevant article, if it exists. Specifically, I had thought that an occupying power could not use aerial bombing in the pacification of an occupied territory. I am still looking.

“As regards Mike H’s comments, I think my response at #2 is perfectly adequate. This may be a bit hard to see if you don’t understand sampling theory, but the position is clear to anyone who does.”

I think it’s anything but adequate, John. While I respect your statistical kung fu, I’m having trouble taking it seriously when applying it to the real world, in the context of comparing the two Lancet studies.

Perhaps if we were dealing with a much larger population, and a much larger per annum baseline mortality number, the variances would be easier to dismiss as blips.

But we’re not dealing with a baseline mortality estimate of a million people, we’re dealing with one in the range of 120,000 – 132,000. Juxtapose the swings from study to study between violent and non-violent deaths and we’re looking at numbers that are huge, half the baseline number (although admittedly for an 18 month period).

John:

I’d also be interested in any thoughts you might have on my post to Kevin (17), in relation to the UNDP study, which was conducted using similar methodology to the Lancet studies, but with a sample size many times larger.

The difference between the two in terms of violence is dramatic.

I guesstimated roughly the same figures for air strike deaths—less than the Lancet number, but still significant.

The more I’ve thought about how you could carry out a study in Iraq under today’s violent conditions, the more skeptical I’ve become of how random the sampling was. It would have been foolhardly for researchers to pick neighborhoods out at random and start nosing around.

What I would have done if I was one of the Iraqi researchers assigned to do random surveys is call around and find neighborhoods where the Man-in-Charge (tribal leader, ayatollah, gang chieftan, whatever) wanted my presence. I’d tell the Americans we did it all randomly, but I’d actually only go where I was invited.

That might have a biasing effect on the results.

I’m going to have another go at stating the significance of the p-value because I’m not happy with my last effort. A p-value of 0.52 means that if you used the same methods to sample a (hypothetical) series of identical populations, there’s a 48% chance that you’d observe a smaller difference than your study showed and a 52% chance that you’d observe a larger difference than your study showed. So in the case of Lancet 2, the authors are saying we may as well assume that all years for which deaths were reported were identical in terms of the non-violent death rate.

Compare this with the Lancet 2 p-value for the change in the rate of violent deaths: 0.0001. This states that there would be a 99.99% chance of observing a smaller difference than the study showed if you used the same methods on a (hypothetical) series of identical populations.

Mike H, before commenting any further you need to read a text on sampling theory, particularly in relation to confidence intervals and hypothesis testing. Charlie gets it exactly right at #35.

There are many potential biases in the study but how can you tell which way they point?

If Steve is right, the surveyors might either have gone to where someone had a particularly large collection of bleeding stumps making the results a high estimate or they might have missed out the most chaotic and dangerous areas altogether making the results a low estimate.

On the violent/non-violent death split issue, as I understand it the confidence intervals of the two surveys overlap. Likely for some value in the middle neither estimate looks unlikely. Constructing a confidence interval along similar lines provides a lower bound of 400,000 deaths above Saddam. It seems odd to think that the former numbers give cause for concern but not to conclude that since the invasion there have likely been 10,000 excess deaths a month. The confidence intervals are there precisely to address issues like this.

Asking for the data on the exact period doesn’t seem likely to do much beyond the figures John Q quotes and it is, unfortunately, not standard practice to provide all possible breakouts of survey data.

If anyone was better positioned to do a better job of counting the dead it would be the US governent and they have signally failed to provide any meaningful statistics on the matter.

Ragout,
your estimate of deaths in the Israel Lebanon War may be low by a factor of two and the sorties are not of the same kind. The Israeli attacks were supposed to be based on excellent intelligence, aimed at destroying infrastructure mostly in evacuated areas and the count includes the massive clusterbombings of evacuated areas in the last days. By contrast the US strikes are not in evacuated areas, with the best will in the world not mostly based on excellent intelligence and mostly in a live combat support role mostly in urban areas.

Still, I will be sure to take arguments from gut feelings skeptically.

Ragout, there’s a factor of 10 error in your comparison. The Air Force reports around ten times as many sorties as air strikes.

Marc, Jack, and Chew2,

Yes, there are differences between Iraq and Lebanon, but I doubt that the differences were as big as you say. We’re talking orders of magnitude here. Even if US air strikes were 10 times more deadly than Israeli ones, Quiggin is still off by a factor of 10.

Let me use my imagination about how a typical air attack goes down. First, US forces and Iraqi insurgents get in a firefight. When the airstrike called in by US forces arrives, the Iraqis have all taken cover in holes they’ve dug in the ground. These “foxholes” prove a very effective defense, and no Iraqis are killed. However, Iraqi fire is suppressed, as the Americans intended.

In contrast to air strikes, here’s a passage about artillery from Rick Atkinson’s book about the conventional war in 2003 that’s stuck with me:

To avoid hitting American soldiers, an artillery tube deliberately fired a single round six hundred meters beyond the target. A spotter in the city radioed instructions to walk the artillery back in hundred-meter increments.

… [The] earlier suggestion of additional air strikes now seemed more appealing. “What we ought to do is put a precision munition on him instead of walking artillery all over town.”

.. and what Marc said.

Ragout, your example doesn’t have an urban feel to it. Why would you assume that’s typical?

John:

Any response to my question directed to you in post 31?

Mike H #52 was a response to #50.

On #51/#31, you’re wrong. The UNDP study was based on media reports like IBC.

“On #51/#31, you’re wrong. The UNDP study was based on media reports like IBC.”

Am I now? I could have sworn it was a cluster sample, many times larger than either Lancet study.

Here’s another way to scale Israel in Lebanon. 6 weeks of air assault killed approximately 1,000 people. The Iraq war contains about 28 6-week intervals, so on a pure scaled up basis that would be 28,000 dead, which is in the ballpark of John’s estimate.

Sorry Philosopher, my first question should read ” do you reject the assertion that the huge variance between violent and non-violent deaths has statistical significance?

In Lancet 1, 40% of the 98,000 ci mid-point deaths were attributed to non-violent causes. In Lancet 2, it’s 0%.

A p-value of 0.52 means that if you used the same methods to sample a (hypothetical) series of identical populations, there’s a 48% chance that you’d observe a smaller difference than your study showed and a 52% chance that you’d observe a larger difference than your study showed. So in the case of Lancet 2, the authors are saying we may as well assume that all years for which deaths were reported were identical in terms of the non-violent death rate.

Thanks, Charlie. Are there any conclusions we can draw about the probability that the number of non-violent deaths would fall by 40%? Does a p-value of around .5 not just mean that the figure has an equal chance of being higher or lower? In which case, the non-violent deaths could be even further into the negative than they already are (extrapolating from the pre-invasion baseline)?

At the very least – and even allowing for sampling theory explanations for the discrepancy – are the authors justified claiming mutual corroboration between the first and second studies if the percentage of deaths that can be attributed to non-violent causes has fallen more than 40% according to their own figures, however ‘safe’ they might be based on p-values?

JohnQ,

The UNDP ILCS report was a survey of over 21,000 households and not, as you say, based on media reports.

Mike H

I don’t know if this will help, but doing a quick browse to follow up on your points on UNHD survey, I found this http://iraqmortality.org/iraq-mortality

They discuss the differences in the surveys: the questions they asked, how they did the survey, etc (and yes, you are right it was a survey).

There are some big differences in the suveys about how things are classified, what was included, etc. But that link states that the results are consistent.

I apologise, I haven’t read through it all in detail but it might provide some more background.

Give me a break. You’re quibbling over factors of two (air attacks vs. sorties, or estimates of Lebanese deaths), when the figures suggest that Quiggin is off by a factor of 100. And I find your argument that there’s a vast difference between airstrikes targeting rockets in Lebanon and airstrikes targetting mortars in Iraq to be unconvincing.

In any event, Quiggin’s only justification for his estimate is that he “can’t imagine” dropping bombs that don’t kill anyone. As I’ve shown, that’s what happened 90% of the time in Lebanon.

By the way, this article puts the number of airstrikes in Iraq at about 300 a year in both 2004 and 2005. That’s a factor of 6 lower than Quiggin’s estimate.

ragout

“this article puts the number of airstrikes in Iraq at about 300 a year in both 2004 and 2005”

That article is USA Today, 3/15/06:

The increase in airstrikes is most dramatic in Afghanistan, where they nearly doubled to 157 last year, compared with 86 in 2004. In Iraq, strikes increased 7% from 285 to 306

But those figures are somewhat at odds with this (WaPo, 12/24/05):

U.S. Airstrikes Take Toll on Civilians … Eyewitnesses Cite Scores Killed in Marine Offensive in Western Iraq … U.S. Marine airstrikes targeting insurgents sheltering in Iraqi residential neighborhoods are killing civilians as well as guerrillas … The number of airstrikes carried out each month by U.S. aircraft rose almost fivefold this year, from roughly 25 in January to 120 in November, according to a tally provided by the military.

That article I think is very relevant to this discussion.

Similar figures corroborated here (UK Times, 1/1/06):

The number of airstrikes in 2005, running at a monthly average of 25 until August, surged to 120 in November and an expected 150 in December, according to official military figures.

“do you reject the assertion that the huge variance between violent and non-violent deaths has statistical significance?”

Yes, I am rejecting exactly that; without something confidence intervals, the point-values that you are focusing on cannot be meaningfully compared, no matter how large their difference. (since the confidence intervals might be bigger still).

Iraq Body Count has come out with its own analysis of the latest Lancet paper. They think it’s rubbish. They’re polite about it, though.
I thought they made a number of very good points, but won’t rehash them here.

I’d rather rehash the one I’ve been making at Deltoid—
The UNDP survey found that the number of war-related deaths (supposedly excluding criminal murders, from what I’ve read) was 19,000 to 28,000 in the first 13 months. The latest Lancet paper happens to have the violent mortality rate for exactly the same stretch of time in Table 3. The 95 percent confidence interval is 1.8 to 4.9 per thousand per year. Assuming a population of 25 million, that’s 45,000 deaths per year at the bottom of the confidence interval, or about 50,000 for 13 months.

To get the UNDP estimate to agree with the lowest end of the confidence interval, you have to tack on 20-30,000 criminal murders. To hit the midrange figure (3.2 per 1000 per year, or about 80,000 deaths per year or 90,000 in the 13 months), you’d have to tack on about 60-70,000 murders.

I don’t think the UNDP and the second Lancet report are in agreement, and it’s not a question of me taking a subset of the data and overinterpreting it—the Lancet authors did the confidence interval for that time period themselves.

Iraq Body Count makes other arguments. I hope people look at them calmly and think about them—there’s been a bit too much talk about how the only reason one could have for doubting this 400-900,000 estimate is rightwing buffoonery or sheer ignorance, and I don’t think that’s quite right.

BTW, John Emerson, there’s an enormous difference between the war as portrayed by Iraq Body Count statistics taken at face value and what one sees in the Lancet report. If you ever bother to read IBC’s analysis of their own data, in most months of the war the number of civilian deaths that the media attributes to American action is extraordinarily low. In most months it is a few dozen people. In the third year of the occupation IBC could clearly attribute 370 civilian deaths to occupation forces. If one takes those numbers at face value, the US troops are doing an extraordinary job avoiding civilian casualties, especially when you couple such (wildly implausible) numbers with the number of insurgents Michael O’Hanlon claims we are killing—he said in late 2005 it was 760-1000 per month.

OTOH, if you take the Lancet numbers as gospel, Iraq is a Vietnam-style free fire zone.

I think the truth is somewhere in-between, but I don’t know where exactly. I think the media is clearly underreporting killing by our side, but don’t know by how much. But it seems to me that American citizens should very much want to know the order of magnitude of the killing our own troops are committing. Tens of thousands of civilians killed per year by our troops in reckless free-fire zone style killing is qualitatively different from a few hundred per year tragically killed by accident.

BTW, so as not to mislead, IBC attributed about 10,000 civilian deaths to US forces in the first two years. But nearly 7000 of those died in the first two months (during the invasion) and roughly 2000 more died in the two separate attacks on Fallujah. What’s left over are months and months and months where it’s literally dozens of deaths per month caused by coalition forces, in comparison to many hundreds per month (in the first two years remember) caused by criminals or insurgents.

IBC also admits that in the third year there were thousands of deaths it couldn’t attribute to any one actor. But when you stick to what they can attribute, the overwhelming majority of violent deaths after April 2003 are caused by Iraqis (or the foreign jihadists).

Again, that’s a completely different war from the one depicted in the Lancet paper. It makes a difference if you’re an American citizen—it’s not just that the Iraq War was wrong, but it’s also a question of how many civilians our own forces are killing directly.

On the number of strikes, the report I cite, direct from USAF gives more than 300 strikes in the three months Sep-Nov 2005, so there seem to be some big contradictions in reports from the same source.

Looking over the CENTAF reports, they cite 50 or so “close air support” operations per day fairly consistently – this includes reconnaissance and similar, but not transport operations, and there are always several cases of combat support reported. The reported operations in Iraq seem to include a lot less “expenditure of munitions” than those for Afghanistan where the total number of operations is smaller.

So, it seems pretty clear that, if the reporting is correct, air strikes are being used very differently in Iraq than in Afghanistan and certainly than in Lebanon. The descriptions I’ve seen suggest a higher degree of lethality.

These numbers don’t appear to include missile stikes or helicopters.

David Kane, every time you raise your head in these forums it is to say that you have “asked for the data” and it wasn`t forthcoming, with the attendant insinuation of fraud or bloodymindedness. Anyone who has worked with these large survey samples knows that the organisations which collect them don`t just give up the data on a whim to anyone who comes along wanting to “replicate it”. They have serious ethics requirements which prevent them from doing that easily. You like to imply you are familiar with statistical analysis and data auditing, in which case you know this is true for almost any survey of this type, and the way you present their reluctance to share the data as evidence of suspicious dealing is quite disingenuous.

If, on the other hand, you have an actual criticism of the methods of the paper – and by “actual” I don`t mean “it disagrees with Dubya” or “that can`t be right!” or “it doesn`t concur with another study which I happen to believe is a fraudulent crock of shit” – then feel free to write a letter to the editor of the Lancet explaining your position. I`m sure they`ll be willing to publish whatever analysis you bring to bear on the substantive elements of the paper – i.e. the sample design, the sample weighting or the statistical comparisons. If you don`t have any argument with these elements of the paper then accept the figures presented and get over it.

Mike H, this issue of confidence intervals is not a smoke grenade or some other dodge, it`s an essential part of the consideration of the results. Given what you and Brownie have presented of the authors` statements in the conclusion one could assume that they have drawn too strong a conclusion about the comparability of the studies, but it is quite common for scientists to overstate the results of their own papers, and such an act in the discussion in no way invalidates the results of the paper. Choosing to disagree with their conclusion that the papers are comparable, and instead concluding that statistical error in the first paper prevents comparison, in no way invalidates the second paper (or the first).

Is it also not the case that in Lancet 1 they excluded the Fallujah cluster? Had they included it the confidence intervals for the violent deaths estimates would have been narrower, and the point estimate more like the point estimate from Lancet 2. In a way this whole discussion is driven by imprecision which the authors themselves introduced in order to make the paper less biassed. I think they should get some credit for that, regardless of whether you think their conclusions about comparability were too strong.

Given what you and Brownie have presented of the authors` statements in the conclusion one could assume that they have drawn too strong a conclusion about the comparability of the studies,
I don’t think it’s fair to assume that. As Robert pointed out more than once, you can get a significant agreement between studies about the total number of deaths without getting a significant agreement about the total number of non-violent deaths in particular, and the authors never claim the latter.

I would suggest that, given the statistical insignificance of the change in non-violent deaths, one of the authors shouldn’t have later listed the breakdown of excess deaths by cause the way he did after Lancet 1, since apparently the error bars on each of those numbers was so huge as to make them totally uninformative. However, the published study didn’t make that claim.

Is it also not the case that in Lancet 1 they excluded the Fallujah cluster? Had they included it the confidence intervals for the violent deaths estimates would have been narrower, and the point estimate more like the point estimate from Lancet 2.

No, actually, if it’s included the confidence intervals become much larger and the point estimate’s well above Lancet 2. Lancet 2’s between the two estimates you get from Lancet 1, including vs. excluding Falluja.

As I said on another thread, that makes sense to me—Falluja did happen, and really should raise the death count, but the actual Falluja cluster data they got in Lancet 1 is insanely high and (as the authors themselves point out) may not accurately represent the entire 3% of Iraq’s population that cluster was supposed to cover.

c.l.ball, your Apache link is broken

I’d encourage everybody to read the Iraq Body Count press release on the latest Lancet study:

http://www.iraqbodycount.org/press/pr14.php

It’s sensible, from a non-warhawk source, and does raise many reasons to doubt the scale of the Lancet 2 results. They take no position on what if anything might have biased the study sample.

One thing for sure at this point: the initial (Lancet 1) estimate of 100,000 excess civilian deaths is looking not just plausible but actually pretty low. Even the IBC’s passive count based completely on press reports of violent deaths alone is approaching 50,000. And passive counts are almost guaranteed to be underestimates. I suppose a few years down the line the Lancet people will come out with an estimate of 1 million and we’ll all be arguing vociferously over whether it’s only 400,000.

“Philosopher, I’m not sure what you mean in regard to ” without something confidence intervals…..” Did you miss a word?” Yeah, sorry, that was supposed to be “something like confidence intervals”, since there are other mathematical ways of getting at significance than the intervals per se. (There’s already been discussion on this thread about p-values, for example.) But nothing in this particular conversation is turning on that.

mq is right about how completely fundamental the distinction between the point-estimate and the confidence intervals are. I’m sure some defenders out there are sloppy about this, too—like I said, it’s a very common misunderstanding that people without any stats training make all the time, regardless of political persuasion. But all the people who seem to know anything about stats are putting a lot of emphasis on the CI, which is as it should be. The point-estimate is of course part of that—the CI is an interval around the point-estimate—and it is also of course easier to report, talk about, and throw around, but it is only part of the story, and a meaningless part without some indication of statistical significance.

Sometimes an extreme hypothetical case can help illustrate matters. Suppose someone did a truly awful survey of Iraqi mortality, in which they only check on 50 people; and with those 50 people, there had been 1 pre-invasion deaths and
4 post-invasion deaths. Even for the lower estimates, this is an entirely plausible kind of result for such a small sample—small samples are more hostage to the luck of the draw than large samples, which is why you want large samples when you do this right. Now, if we don’t check at all for significance, then we have something like a 20 per 1,000 pre-invasion death rate and a 80 per 1,000 post-invasion death rate. Multiply that difference by the population of Iraq, and you get something like
2.1 million excess deaths. Such a huge point estimate! But I hope that I have set this up so it is clear why the size of the point-estimate is simply no indicator whatsoever of whether you’ve actually found something statistically significant. In this case, the sample is so small, that the CI would be absurdly huge. Indeed, I suspect that if one quickly crunched the numbers, this wouldn’t even be a big enough difference, for samples so small, to indicate any statistically significant difference between pre- and post- at all, let alone one of such a magnitude. (It’d be pretty hard to guarantee representativeness in a sample that small, too, but that’s another matter.)

Moral of the story: the bigness of point-estimates or their differences is just no indicator whatsoever of any significance of the effect.

• * * * *

I’ve just looked back at the earlier Lancet paper, and if I didn’t miss anything (and, hey, it’s 3:30am where I’m at, so it might have slipped past me), then there’s something very telling in it, for the particular line of argument that Mike H. is running: the authors never make any claim to statistical significance for an increase in nonviolent deaths. On p. 1861, there is a specific claim about a “58-fold” increase in violent deaths, as well as the increases in the overall death rate—but I’m just not finding anything reported in there about specifically nonviolent deaths that has a CI attached to it. And it looks like Lancet II is the same in this regard. The closest they come to making any such claim is in the following from p. 5, but if you look closely you’ll note that no claims to significance are made there: “Excess mortality is attributed mainly to an increase in the violent death rate; however, an increase in the non-violent death rate was noted in the later part of the post-invasion period (2005–06). The post-invasion non-violent excess mortality rate was 0·7 per 1000 people per year (–1·2 to 3·0).” See? No CI reported, and on p.7 they explicitly say that this was not significant. (Did someone already say this part up-thread somewhere? My apologies for being redundant, if so.)

So, Lancet I makes no claims about nonviolent deaths increasing; and Lancet II likewise finds nothing of statistical interest to report about nonviolent deaths. So in this regard they are totally consonant with each other.

There’s all kinds of caveats to attach to this, but there’s as yet no historical basis to claim a civilian death rate due to bombing higher than 0.2 deaths per sortie.

In Hiroshima 100,000 people were killed by one airstrike. So, it’s between 0 and 100,000.

philosopher,

I appreciate your efforts to explain, but even for we non-statisticians the concept of a confidence interval is pretty well understood.

When Kaplan talked about a “dartboard” with respect to Lancet 1, he was ridiculed by supporters of the study who rushed to explain why 8,000 was not as likely a figure as 98,000 for projected excess mortality in Iraq. The net must be literally bursting with examples of comments and posts from sites like CT where the fundamentals were trotted out for the benefit of people like Kaplan. Similarly, it’s not difficult to find quotes from Horton or any of the Hopkins authors in which the 98,000 (more commonly referenced as 100,000) is mentioned as the best estimate we have for excess mortality.

Two years on and when some observers compare the figures in Lancet 1 with Lancet 2, we’re told that we ignore the confidence interval at our peril. Well, okay, let’s not ignore it, but let’s at least be consistent with the emphasis .

the authors never make any claim to statistical significance for an increase in nonviolent deaths.

It really isn’t the argument of those raising questions that the authors make this claim. The fact remains, whilst non-violent deaths account for 40% of the 98,000 point estimate for Lancet 1, they’ve dipped below zero (in that the rate has gone negative from the pre-invasion baseline) for Lancet 2’s 112,000 (or 130,000 extrapolated). Over 40,000 non-violent deaths have been replaced by an additional 60-odd thousand violent deaths.

Provide 99 statistics-based reasons how such a discrepancy may have come about, but then please go on to explain to me why the authors are still within their rights to say the studies are mutually corroborative?

The only sense in which the studies can be said to be mutually corroborative on the specific question of composition of excess deaths, is that the confidence interval’s in both overlap, which, as any fule no, is no corroboration at all.

This is hardly an unimportant point, given mutual corroboration is one of the strongest indicators that we can have faith in the results produced by similar studies. Which is probably why the authors both to claim corroboration at all.

Regarding air operations in Iraq: Close air support sorties are those flown in direct combat support of troops. It’s in the nature of CAS that the aircraft will only fire (or indeed do anything rather than circling) if they get a call from the forward air controller with the troops, unless somebody has a Canadians-in-Afghanistan speed-freak brain storm, so the ratio of strikes to sorties will be low. I’d also point out that more of this will be going on in Iraq than Lebanon, as there are five and a half coalition divisions spread out over a wide area, more than twice the maximum number of troops the Israelis ever sent into Lebanon.

Also, the Israelis launched a quasi-strategic air offensive, which implies they picked a list of targets off the map and set out to bomb them – so, once discounted for the inevitable aircraft that go technical, find the target weathered in, fail to meet the refuelling tanker etc, you’d expect the two numbers to be similar.

You can believe 655000 Iraqis have died.
But only if you believe it was paradise under Saddam.

The Lancet this week published a report claiming that 655,000 people have been killed in Iraq.
It is based on flawed research that would shame any publication, except one that had previously been responsible for giving Andrew Wakefields MMR “research” the oxygen of publicity.

The research is based on the pre war death rates under Saddam Husseins regime before the 2003 invasion, assessing those afterwards, and labelling the difference war related deaths.

Unfortunately the researchers have made two huge errors.
Their figures only stand up if adult Iraqis living under the Ba’athist regime hardly ever died. Either of natural or violent causes.

Secondly they included the data to undermine their conclusions in their own report.

The researchers claim the death rate in pre war Iraq was 5.5 per thousand and backed up this claim in a country of 25 million people with an analysis of 82 deaths. Thats a death rate, half that of the EU but perfectly possible in a “young” country with a high birth rate, low median age and good life expectancy.

Unbelievably the same research claims infant mortality was 10%.

5.5 deaths per thousand means that 137,500 Iraqis died each year.
10% infant mortality in a country of 34 births per thousand and 25 million means that 85,000 of these deaths were under one year old.

Leaving a death rate for Iraqis over one year old of 2.1 per thousand.

Less than half that of the lowest country in the world.

Paradise.

And on the subject of violent deaths pre invasion?

.

They drew the following conclusion “As there were few violent deaths in the survey population prior to the invasion all violent deaths can be considered “violent excess deaths”
82 deaths analysed.

Read that again. No violent deaths under Saddam.
Not even domestic violence. Prison works.

This may tell you something about the motivation and rigour of the analysis.

Or a desire for headlines.

The World Health Organisation gives a figure of 9.03 deaths per thousand in Iraq pre invasion.

A figure which ,put through the Lancet prism would have reduced its headline death toll by hundreds of thousands.

And the headlines by a few.

Tim

Secondly they included the data to undermine their conclusions in their own report.

That’s not an “error”. That’s called science.

They drew the following conclusion “As there were few violent deaths in the survey population prior to the invasion all violent deaths can be considered “violent excess deaths”
82 deaths analysed.

Read that again. No violent deaths under Saddam.
Not even domestic violence. Prison works.

Read that again. “Few violent deaths”. “Few” doesn’t mean “no”.

You, Sir, are an intellectually vacuous liar, or else functionally illiterate.

alex,
for the purposes of their research they assume none.therefore.
” all violent deaths can be considered “violent excess deaths”

But only if you believe it was paradise under Saddam.

Yes, paradise – or perhaps a well-run police state.

Don Johnson, 71: That point is a reasonable one, and some of the statistics and sampling questions are reasonable too, but all reasonable points on these questions are being swamped, and that was inevitable. And the statistical arguing makes swamping easier, not harder.

That the Lancet attempt at precision was, in the context of this particular debate, doomed. (If they cut a few corners to make “the best case”, it was even more doomed, of course).Any report they made whatever would have received enough flak that it wouldn’t be regarded as credible by the media. And while I fully support the political purpose of the study and think that their release date was perfectly timed, I stillthink that their effort was futile.

The population able to understand statistics and willing to have their mind changed by a careful statistical argument on a highly contentious issue is small, and there’s so much junk statistcs out on various issues that the good statistics are automatically devalued (just because so few people really understand stats, and also because so many don’t trust anyone at all).

My opposition to the war is mostly based on its fraudulent justifications, strategic stupidity, and failure, rather than on the brutality or not of the American troops. Based on what you’ve said, Id expect to see lots of war supporters say that we need to “take the gloves off”, since we’re killing too few Iraqis.

Charlie,

I think the problem is that you can’t meaningfully do arithmetic with point estimates.

Except, apparently, when you’re looking to use one point estimate in one study to corroborate another in another.

Are you saying that there is no statistical merit in the claims by the authors that the studies are mutually corroborative because the point estimates are in the same ballpark? If so, I’ll settle for that.

The second it bcomes legitimate to do the like for like comparison of point estimates and use this as corroboration, we surely are entitled to point out the glaring contradictions in the constitution of each?

Apple pies and steak pies are both pies, but I’d only put custard on the former.

103: Are you saying that there is no statistical merit in the claims by the authors that the studies are mutually corroborative because the point estimates are in the same ballpark? If so, I’ll settle for that.

I’m not competent to judge the study’s authors in that way, but it could be that their statement was incautious, yes.

It’s still the case that researchers have (twice now) been to Iraq and asked some of the people there how many in their households had been killed since the invasion. And they came back with some numbers. The problem we are all faced with is finding an explanation for those numbers. The choices here are, essentially:

– The death rate from violence is significantly higher than passive reporting suggests – The respondents were lying – The researchers committed a major error in their sampling methodology

And it’s possible that we could combine these explanations in some way.

There is no ‘jury out’ situation here. A criticism that doesn’t amount to more than a criticism of some of the rhetoric of the presentation does little to get you closer to understanding why the numbers are as they are; nor, crucially, does it make the numbers go away. In my view, without a plausible explanation for these numbers, concern is the appropriate response and hand washing should wait.

There is no ‘jury out’ situation here. A criticism that doesn’t amount to more than a criticism of some of the rhetoric of the presentation does little to get you closer to understanding why the numbers are as they are; nor, crucially, does it make the numbers go away. In my view, without a plausible explanation for these numbers, concern is the appropriate response and hand washing should wait.

It’s a little deeper than that, Charlie. One of the strongest indicators we could have that Hopkins has things about right is if Lancet 2 validates and corroborates Lancet 1. I suggest that’s why the authors have claimed this to be the case.

You say it could be that such claims are “incautious” yet don’t believe you are qualified to judge. Well, if the basis for claiming corroboration was a mystery beyond the comprehension of all but statistical sciences PHDs, then you might have point, but the basis for such a claim is a 98,000 point estimate in Lancet 1 and 112,000 in Lancet 2 for the same period. They’re in the same ballpark, aren’t they? The point is that the composition of each figure is entirely contradictory, invalidating any claim of mutual corroboration. Over 40% of deaths from one cause in Lancet 1 have been replaced by the only other possible cause in Lancet 2.

The importance of non-corroboration cannot be overstated, as it immediately invites the conclusion that both cannot be right, which in turn means one must be wrong.

You can fill in the rest.

Kevin,

The point of contradiction is not that Lancet 1 said the greater number of deaths was 30-40 year old shoemakers from Baghdad, whereas Lancet 2 claims it is 20-something plumbers from Mosul. The deaths have one of two causes: violence or non-violence. Even allowing for double weight of misattribution, to go from 40%+ non-violent deaths in Lancet 1 to sub-zero in Lancet 2 takes some doing if all bias has been avoided.

Moreover, that mortality has increased is not in dispute. If the authors used the findings of Lancet 2 and Lancet 1 to claim mutual corroboration for an increase in mortality and nothing more, you’ll get little argument from me or anyone else, I suspect. But they specifically went further than this. They highlighted a point estimate of 112,000 deaths for the first 18 months of the conflict and compared this with the 98,000 estimate from Lancet 1. They are explicitly claiming mutual corroboration not just for a rise in mortality, but in the estimation of that rise.

Given the finite number of causes of death is 2 (violent and non-violent) and not 22, the composition of both estimates is significant and where they are found to be contradictory – as they are – it is evidence against mutual corroboration on the specific question of point estimates.

If the authors want to row back on the corroboration claims about the point estimates for the first 18 months of the year, than I’ll accept that composition of those point estimates becomes less significant.

Hey!

With all due respect, I think something important is being ignored.

John: “The best source turns out to be the US Air Force Command itself.”

Yes, but I think you need to pay less attention to what they say about “airstrikes,” and more attention to what they say about tons of bombs dropped.

As far as I can tell, the above analysis is largely focused on number of aitstrikes/sorties, and relies on statistics released by the US. Trouble is, guns don’t kill people; bullets kill people. Airstrikes don’t kill people; bombs kill people. (I realize airstrikes deliver munitions aside from bombs, such as bullets, but I’m more concerned about the bombs.)

While the US does release certain vague figures regarding “airstrikes” (a term which I haven’t seen defined very clearly anywhere), it seems very, very reluctant to tell us the total tonnage of bombs dropped on Iraq.

Information can be found in daily “CENTAF airpower” summaries such as this, but I think the information is purposely vague, and never reveals tonnage, as far as I can tell (we see terminology such as “airstrikes,” “missions” and “sorties”). John cites Jamail’s article which takes a close look at those summaries. By the way, I’m puzzled by this statement of Jamail’s:

Carrier-based Navy and Marine aircraft flew over 21,000 hours of missions and dropped over 26 tons of ordnance in Fallujah alone during the November 2004 siege of that city.

I can’t figure out where he got that from (“26 tons”). I also think the idea that Fallujah received only 26 tons “during the November 2004 siege of that city” is dramatically understated.

Anyway, as far as I can tell, the US has released such information (tonnage dropped on Iraq) exactly once, here:

So far for OIF II, 3rd MAW has dropped more than 500,000 tons of ordnance … but this number is likely to be much higher by the end of operations

I believe this statement has been used as the basis for reporting by Hersh, here:

Since the beginning of the war, the press release said, the 3rd Marine Aircraft Wing alone had dropped more than five hundred thousand tons of ordnance

(John, you point to Solomon, who points to Hersh, who points to the US press release. However, Solomon and Hersh leave out URLs, which I’ve provided.)

As far as I can tell, the Marines press release has otherwise been overlooked, in general, even though I think it is credible and unintentionally revealing. When that statement is analyzed (please follow the li