Air war in Iraq

John,

What are your thoughts on the change in non-violent/violent deaths distribution between Lancet 1 and Lancet 2 (see Lambert and DD’s “Death Certifciates” post below for contributions from Mike H specifically)?

I’m no statistician (no, really I’m not) but Mike H’s observations have been out there now for well over 3 days and I’m not seeeing any credible rebuttals. Which, of course, doesn’t mean there are none.

Skeptics use the gut-check argument from incredulity a lot. The combination of foreign-language problems, the difficulties of collecting information in war time, and (perhaps most important) successful message control by the military (and self-censorship by the media) altogether mean that American intuitions about Iraq are pretty much worthless. (The idea that the gut is a usable source of information is a dubious one in any case, but especially here.)

A compounding factor is that many Bush loyalists and super-hawks would be happy enough if every single Iraqi were killed. They just don’t want the biased liberal media to report about thing like that.

Here are the non-violent deaths as proportions of the survey totals:

Lancet 1 count: including Fallujah (69/142 [49%]) or excluding Fallujah (68/89 [76%]). 988 households were visited.

Lancet 2 count: (247/547 [46%]). 1,849 households were visited.

If I’ve understood the complaint correctly, it’s this apparent drop in non-violent deaths from 76% of the count to 46% of the count that’s causing difficulty. The two studies used somewhat different methodologies; the sample size and period were different. Death certificates were asked for only in some of the Lancet 1 interviews. Given this, and given the specialism of the maths, I think it would take a statistician with access to population data to interpret these figures meaningfully (pace John Quiggin above; I was busy writing this in the meantime). And my gut feeling is that I don’t see any special reason to assume that non-violent deaths would scale with violent deaths. Perhaps infectious disease epidemics caused in part by degraded sanitation would do it: I don’t see any reason to expect an increase in heart disease, which was said to be the leading cause of death pre-war. And if news reports are to be believed, sectarian gun killings seem to be up; is there any reason for these to lead to a concomitant increase in non-violent deaths?

OK, enough of my speculation. Contrary to what some commenters are suggesting, Lancet 1 had this to say about the non-violent death rate:

It is surprising that beyond the elevation in infant mortality and the rate of violent death, mortality in Iraq seems otherwise to be similar to the period preceding the invasion. This similarity could be a reflection of the skill and function of the Iraqi health system or the capacity of the population to adapt to conditions of insecurity.

And Lancet 2 had this to say:

… families might have misclassified information about the circumstances of death. Deaths could have been over or under-attributed to coalition forces on a consistent basis. The numbers of non-violent deaths were low, thus, estimation of trends with confidence was difficult.

These studies have known limitations which the researchers freely admit to, but that doesn’t render them worthless. If you want to discern specific mortality trends at a higher level of accuracy, I think you need to lobby for a better survey. And if you don’t get that from your government, I think you should assume that the news from Iraq is bad.

What astonishes me is that with 800-900 attacks a week, and with two airstrikes a day why would anyone think that a lot of people were not dying?

Surely, if you were a fiscal conservative you ought to be livid with the administration if they were telling the truth: the huge amount of money being spent on ammunition, expensive air-strikes every day. And yet nobody is dead?

Doesn’t it just seem more logical to agree that lots of people are being killed? Precisely as intended when we drop bombs on them.

Mr. Quiggin, I think the concern that brownie has raised is that the previous study had a vastly different proportion of deaths due to violence. I think the argument is that in the first study a very large proportion of excess deaths were due to non-violent causes, while the new study estimates pre invasion deaths to be largely due to violent causes. In response to Brownie I think the explanation is very simple. The original study had a very wide confidence interval, and in the cause of death data (which divides the main data set into smaller groups of counts) the confidence intervals would be even wider. The confidence interval around the number of non-violent deaths in the previous study is probably so wide that it cannot be considered to be statistically significantly different to the figures observed in the new study – i.e. while the point estimates look radically different, the imprecision of the old study prevents any conclusions from being drawn about this.

(I have read the critique of this aspect of the study, which was presented somewhere in comments to Tim Lambert’s blog I think, and this critique was constructed based on the point estimates of deaths only. One needs to consider the errors as well when comparing the old and new studies, and the old study had very wide confidence intervals).

The ony way to get around this problem would be to do a really big study – say, on the scale of the sorts of national surveys which occur in Australia, I think about 5000 households – so that all subgroup analyses (women vs. men, type of death categories by year, etc.) can be performed with narrow confidence intervals. If the governments of the coalition of the willing really cared about Iraqi lives they would have commissioned such a survey using these very techniques several years ago…

I’ve posted this elsewhere, but it’s relevant here.

Over the centuries civil wars have caused hideous carnage. 600,000 Americans died in the US Civil war, more than died in World Wars 1 or 2.

Little Bosnia, with its population of 5 million, managed to slaughter 100,000 in its three year war.

England’s Civil War killed 10% of the populace at the time.

The Spanish Civil War, spookily like Iraq in many ways, killed maybe half a million.

Etc etc.

Given that all other major civil wars have indeed caused deaths in the hundreds of thousands, and killed up to 10% of the populace or more, why should Iraq be different?

Why should Iraq, unique amongs civil wars, have a death rate of only 0.2% of the population, 50,000 people? Rather than the normal 2-10%? Why should Iraq get away with only 50,000 dead, when, looking at history, you should expect 500,000? What is so specially peaceful about Iraq that makes it such a fortunate place?

Is it the lack of widespread violence? Is it because all sides are showing admirable restraint? Is it because the Americans are brilliantly precise in their bombing?

I repeat, we expect 2-10% of a population to die in a civil war, that’s what happens. Yet amazingly enough, according to the war mongers, Iraq has got away with an astonishing 0.2%!!

Truly Iraq is a blessed place. The Lucky Country.

Brownie,

Probably the main reason why the statisticians aren’t rushing to respond to Mike H is that this is really difficult to explain without getting into sampling theory. Robert made a good stab at it in comments at Tim Lambert’s:

The basic issue is that small pieces of larger aggregates are often less well-estimated than the aggregate as a whole. We can pretty much estimate how many deaths occur on the nation’s highways each month, but there will be more variation in how many will occur on any particular day, or on a particular road, or by single-car accident vs. multi-car accident. Day-to-day variation in those counts doesn’t invalidate an overall guess at the total number for all causes for all locations.

http://scienceblogs.com/deltoid/2006/10/lancet_roundup.php

I notice that Mike H has responded to Robert in that thread and I think that’s where you are most likely to see an answer to his question. I will just comment on one of his remarks:

I thought mortality studies were as much about the “how” as they were the “how many.”

The more you want to know about precise cause, the larger your sample has to be.

12: Any chance of seeing some working for those figures? Perhaps a year by year breakdown, because after all, Lancet 1 covers a shorter time frame than Lancet 2. I’m not sure where you’re getting the ‘erases’ and ‘replaces’ from; they sound a bit rhetorical to me.

Kevin:

“The more you want to know about precise cause, the larger your sample has to be.”

And the largest mortality study done thus far is the UNDP effort. We know how its numbers stack up against Lancet 2 for violent (or “war-related” deaths), at least for year 1.

As I’ve said over at Deltoid, I can see allowing some latitude for discrepancy from study to study when it comes to the various subsets of non-violent and violent, but not the two main categories of death themselves. In order to be considered reasonably accurate, I think the studies have to be reasonably consistent in terms of violent and non-violent death.

Otherwise, this seems an awful lot like resurrecting tens of thousands of dead from study 1, only so you can kill them off again in study 2, but in a markedly different way, in order to balance their bottom lines

“It is surprising that beyond the elevation in infant mortality and the rate of violent death, mortality in Iraq seems otherwise to be similar to the period preceding the invasion. This similarity could be a reflection of the skill and function of the Iraqi health system or the capacity of the population to adapt to conditions of insecurity.”

Charlie, I meant to address this earlier, but forgot.

Point taken. I’d forgotten about that statement from the first study.

That doesn’t change the fact that the study authors and defenders of the study attributed the increase in non-violent deaths over the baseline extrapolation (whether by infant mortality, accidents, slight increases in heart disease deaths, etc) to the effects of invasion. I spent many hours arguing with learned defenders of the study, their position being that this increase was entirely expected, given there was a war going on and all.

Now, Lancet 2 tells us that Iraqis were actually less likely to die from natural causes and acidents for the first 2 years of the war.

It doesn’t make any sense.

Bob McManus, apparently unaware that the Hague conventions were signed before the airplane was invented, writes:

I thought aerial bombing of civilian areas by an occupying power was a war crime under the Hague Conventions.

Anyway, the closest that international law comes to banning these types of attacks is in article 51 of Geneva Protocol 1, which bans “indiscriminate attacks…not directed at a specific military objective” or “which may be expected to cause incidental loss of civilian life…which would be excessive in relation to the…military advantage anticipated.”

So I don’t think that these kind of aerial attacks are war crimes, since they’re not indiscriminate.

John Quiggin, on the basis of an article that discusses air attacks at a rate of 35 to 120 per month, concludes that there have been an average of 150 attacks per month. What am I missing?

I also doubt the 10 dead per attack figure. It wouldn’t surprise me if most air attacks killed no one at all. Most military attacks simply miss. The US has shot several billion bullets in Iraq, but they haven’t killed a billion people.

On the other hand, the explanation that most of the Lancet-reported air deaths come from helicopters or artillery makes a lot of sense to me. The US likes to fight at night, and I find it hard to believe that someone who was sleeping when an attack occured has any idea if their loved ones were killed by planes, helicopters, mortars, or what.

I got the impression that in most cases it’s local informers settling scores with their personal enemies by means of US-taxpayers-provided ‘smart bombs’. So it’s not entirely indiscriminate.

“Bob McManus, apparently unaware that the Hague conventions were signed before the airplane was invented, writes:”

Actually I was thinking of the Fourth Geneva Convention

“relative to the Protection of Civilian Persons in Time of War” (first adopted in 1949, based on parts of the 1907 Hague Convention IV)”

but I can’t find the relevant article, if it exists. Specifically, I had thought that an occupying power could not use aerial bombing in the pacification of an occupied territory. I am still looking.

“As regards Mike H’s comments, I think my response at #2 is perfectly adequate. This may be a bit hard to see if you don’t understand sampling theory, but the position is clear to anyone who does.”

I think it’s anything but adequate, John. While I respect your statistical kung fu, I’m having trouble taking it seriously when applying it to the real world, in the context of comparing the two Lancet studies.

Perhaps if we were dealing with a much larger population, and a much larger per annum baseline mortality number, the variances would be easier to dismiss as blips.

But we’re not dealing with a baseline mortality estimate of a million people, we’re dealing with one in the range of 120,000 – 132,000. Juxtapose the swings from study to study between violent and non-violent deaths and we’re looking at numbers that are huge, half the baseline number (although admittedly for an 18 month period).

John:

I’d also be interested in any thoughts you might have on my post to Kevin (17), in relation to the UNDP study, which was conducted using similar methodology to the Lancet studies, but with a sample size many times larger.

The difference between the two in terms of violence is dramatic.

I guesstimated roughly the same figures for air strike deaths — less than the Lancet number, but still significant.

The more I’ve thought about how you could carry out a study in Iraq under today’s violent conditions, the more skeptical I’ve become of how random the sampling was. It would have been foolhardly for researchers to pick neighborhoods out at random and start nosing around.

What I would have done if I was one of the Iraqi researchers assigned to do random surveys is call around and find neighborhoods where the Man-in-Charge (tribal leader, ayatollah, gang chieftan, whatever) wanted my presence. I’d tell the Americans we did it all randomly, but I’d actually only go where I was invited.

That might have a biasing effect on the results.

I’m going to have another go at stating the significance of the p-value because I’m not happy with my last effort. A p-value of 0.52 means that if you used the same methods to sample a (hypothetical) series of identical populations, there’s a 48% chance that you’d observe a smaller difference than your study showed and a 52% chance that you’d observe a larger difference than your study showed. So in the case of Lancet 2, the authors are saying we may as well assume that all years for which deaths were reported were identical in terms of the non-violent death rate.

Compare this with the Lancet 2 p-value for the change in the rate of violent deaths: 0.0001. This states that there would be a 99.99% chance of observing a smaller difference than the study showed if you used the same methods on a (hypothetical) series of identical populations.

There are many potential biases in the study but how can you tell which way they point?

If Steve is right, the surveyors might either have gone to where someone had a particularly large collection of bleeding stumps making the results a high estimate or they might have missed out the most chaotic and dangerous areas altogether making the results a low estimate.

On the violent/non-violent death split issue, as I understand it the confidence intervals of the two surveys overlap. Likely for some value in the middle neither estimate looks unlikely. Constructing a confidence interval along similar lines provides a lower bound of 400,000 deaths above Saddam. It seems odd to think that the former numbers give cause for concern but not to conclude that since the invasion there have likely been 10,000 excess deaths a month. The confidence intervals are there precisely to address issues like this.

Asking for the data on the exact period doesn’t seem likely to do much beyond the figures John Q quotes and it is, unfortunately, not standard practice to provide all possible breakouts of survey data.

If anyone was better positioned to do a better job of counting the dead it would be the US governent and they have signally failed to provide any meaningful statistics on the matter.

Ragout,
your estimate of deaths in the Israel Lebanon War may be low by a factor of two and the sorties are not of the same kind. The Israeli attacks were supposed to be based on excellent intelligence, aimed at destroying infrastructure mostly in evacuated areas and the count includes the massive clusterbombings of evacuated areas in the last days. By contrast the US strikes are not in evacuated areas, with the best will in the world not mostly based on excellent intelligence and mostly in a live combat support role mostly in urban areas.

Still, I will be sure to take arguments from gut feelings skeptically.

Marc, Jack, and Chew2,

Yes, there are differences between Iraq and Lebanon, but I doubt that the differences were as big as you say. We’re talking orders of magnitude here. Even if US air strikes were 10 times more deadly than Israeli ones, Quiggin is still off by a factor of 10.

Let me use my imagination about how a typical air attack goes down. First, US forces and Iraqi insurgents get in a firefight. When the airstrike called in by US forces arrives, the Iraqis have all taken cover in holes they’ve dug in the ground. These “foxholes” prove a very effective defense, and no Iraqis are killed. However, Iraqi fire is suppressed, as the Americans intended.

In contrast to air strikes, here’s a passage about artillery from Rick Atkinson’s book about the conventional war in 2003 that’s stuck with me:

To avoid hitting American soldiers, an artillery tube deliberately fired a single round six hundred meters beyond the target. A spotter in the city radioed instructions to walk the artillery back in hundred-meter increments.

… [The] earlier suggestion of additional air strikes now seemed more appealing. “What we ought to do is put a precision munition on him instead of walking artillery all over town.”

.. and what Marc said.

Ragout, your example doesn’t have an urban feel to it. Why would you assume that’s typical?

John:

Any response to my question directed to you in post 31?

“On #51/#31, you’re wrong. The UNDP study was based on media reports like IBC.”

Am I now? I could have sworn it was a cluster sample, many times larger than either Lancet study.

Here’s another way to scale Israel in Lebanon. 6 weeks of air assault killed approximately 1,000 people. The Iraq war contains about 28 6-week intervals, so on a pure scaled up basis that would be 28,000 dead, which is in the ballpark of John’s estimate.

Sorry Philosopher, my first question should read ” do you reject the assertion that the huge variance between violent and non-violent deaths has statistical significance?

In Lancet 1, 40% of the 98,000 ci mid-point deaths were attributed to non-violent causes. In Lancet 2, it’s 0%.

A p-value of 0.52 means that if you used the same methods to sample a (hypothetical) series of identical populations, there’s a 48% chance that you’d observe a smaller difference than your study showed and a 52% chance that you’d observe a larger difference than your study showed. So in the case of Lancet 2, the authors are saying we may as well assume that all years for which deaths were reported were identical in terms of the non-violent death rate.

Thanks, Charlie. Are there any conclusions we can draw about the probability that the number of non-violent deaths would fall by 40%? Does a p-value of around .5 not just mean that the figure has an equal chance of being higher or lower? In which case, the non-violent deaths could be even further into the negative than they already are (extrapolating from the pre-invasion baseline)?

At the very least – and even allowing for sampling theory explanations for the discrepancy – are the authors justified claiming mutual corroboration between the first and second studies if the percentage of deaths that can be attributed to non-violent causes has fallen more than 40% according to their own figures, however ‘safe’ they might be based on p-values?

JohnQ,

The UNDP ILCS report was a survey of over 21,000 households and not, as you say, based on media reports.

Mike H

I don’t know if this will help, but doing a quick browse to follow up on your points on UNHD survey, I found this http://iraqmortality.org/iraq-mortality

They discuss the differences in the surveys: the questions they asked, how they did the survey, etc (and yes, you are right it was a survey).

There are some big differences in the suveys about how things are classified, what was included, etc. But that link states that the results are consistent.

I apologise, I haven’t read through it all in detail but it might provide some more background.

Give me a break. You’re quibbling over factors of two (air attacks vs. sorties, or estimates of Lebanese deaths), when the figures suggest that Quiggin is off by a factor of 100. And I find your argument that there’s a vast difference between airstrikes targeting rockets in Lebanon and airstrikes targetting mortars in Iraq to be unconvincing.

In any event, Quiggin’s *only* justification for his estimate is that he “can’t imagine” dropping bombs that don’t kill anyone. As I’ve shown, that’s what happened 90% of the time in Lebanon.

By the way, this article puts the number of airstrikes in Iraq at about 300 a year in both 2004 and 2005. That’s a factor of 6 lower than Quiggin’s estimate.

ragout

“this article puts the number of airstrikes in Iraq at about 300 a year in both 2004 and 2005”

That article is USA Today, 3/15/06:

The increase in airstrikes is most dramatic in Afghanistan, where they nearly doubled to 157 last year, compared with 86 in 2004. In Iraq, strikes increased 7% from 285 to 306

But those figures are somewhat at odds with this (WaPo, 12/24/05):

U.S. Airstrikes Take Toll on Civilians … Eyewitnesses Cite Scores Killed in Marine Offensive in Western Iraq … U.S. Marine airstrikes targeting insurgents sheltering in Iraqi residential neighborhoods are killing civilians as well as guerrillas … The number of airstrikes carried out each month by U.S. aircraft rose almost fivefold this year, from roughly 25 in January to 120 in November, according to a tally provided by the military.

That article I think is very relevant to this discussion.

Similar figures corroborated here (UK Times, 1/1/06):

The number of airstrikes in 2005, running at a monthly average of 25 until August, surged to 120 in November and an expected 150 in December, according to official military figures.

“do you reject the assertion that the huge variance between violent and non-violent deaths has statistical significance?”

Yes, I am rejecting exactly that; without something confidence intervals, the point-values that you are focusing on cannot be meaningfully compared, no matter how large their difference. (since the confidence intervals might be bigger still).

Iraq Body Count has come out with its own analysis of the latest Lancet paper. They think it’s rubbish. They’re polite about it, though.
I thought they made a number of very good points, but won’t rehash them here.

I’d rather rehash the one I’ve been making at Deltoid–

The UNDP survey found that the number of war-related deaths (supposedly excluding criminal murders, from what I’ve read) was 19,000 to 28,000 in the first 13 months. The latest Lancet paper happens to have the violent mortality rate for exactly the same stretch of time in Table 3. The 95 percent confidence interval is 1.8 to 4.9 per thousand per year. Assuming a population of 25 million, that’s 45,000 deaths per year at the bottom of the confidence interval, or about 50,000 for 13 months.

To get the UNDP estimate to agree with the lowest end of the confidence interval, you have to tack on 20-30,000 criminal murders. To hit the midrange figure (3.2 per 1000 per year, or about 80,000 deaths per year or 90,000 in the 13 months), you’d have to tack on about 60-70,000 murders.

I don’t think the UNDP and the second Lancet report are in agreement, and it’s not a question of me taking a subset of the data and overinterpreting it–the Lancet authors did the confidence interval for that time period themselves.

Iraq Body Count makes other arguments. I hope people look at them calmly and think about them–there’s been a bit too much talk about how the only reason one could have for doubting this 400-900,000 estimate is rightwing buffoonery or sheer ignorance, and I don’t think that’s quite right.

BTW, John Emerson, there’s an enormous difference between the war as portrayed by Iraq Body Count statistics taken at face value and what one sees in the Lancet report. If you ever bother to read IBC’s analysis of their own data, in most months of the war the number of civilian deaths that the media attributes to American action is extraordinarily low. In most months it is a few dozen people. In the third year of the occupation IBC could clearly attribute 370 civilian deaths to occupation forces. If one takes those numbers at face value, the US troops are doing an extraordinary job avoiding civilian casualties, especially when you couple such (wildly implausible) numbers with the number of insurgents Michael O’Hanlon claims we are killing–he said in late 2005 it was 760-1000 per month.

OTOH, if you take the Lancet numbers as gospel, Iraq is a Vietnam-style free fire zone.

I think the truth is somewhere in-between, but I don’t know where exactly. I think the media is clearly underreporting killing by our side, but don’t know by how much. But it seems to me that American citizens should very much want to know the order of magnitude of the killing our own troops are committing. Tens of thousands of civilians killed per year by our troops in reckless free-fire zone style killing is qualitatively different from a few hundred per year tragically killed by accident.

BTW, so as not to mislead, IBC attributed about 10,000 civilian deaths to US forces in the first two years. But nearly 7000 of those died in the first two months (during the invasion) and roughly 2000 more died in the two separate attacks on Fallujah. What’s left over are months and months and months where it’s literally dozens of deaths per month caused by coalition forces, in comparison to many hundreds per month (in the first two years remember) caused by criminals or insurgents.

IBC also admits that in the third year there were thousands of deaths it couldn’t attribute to any one actor. But when you stick to what they can attribute, the overwhelming majority of violent deaths after April 2003 are caused by Iraqis (or the foreign jihadists).

Again, that’s a completely different war from the one depicted in the Lancet paper. It makes a difference if you’re an American citizen–it’s not just that the Iraq War was wrong, but it’s also a question of how many civilians our own forces are killing directly.

David Kane, every time you raise your head in these forums it is to say that you have “asked for the data” and it wasn`t forthcoming, with the attendant insinuation of fraud or bloodymindedness. Anyone who has worked with these large survey samples knows that the organisations which collect them don`t just give up the data on a whim to anyone who comes along wanting to “replicate it”. They have serious ethics requirements which prevent them from doing that easily. You like to imply you are familiar with statistical analysis and data auditing, in which case you know this is true for almost any survey of this type, and the way you present their reluctance to share the data as evidence of suspicious dealing is quite disingenuous.

If, on the other hand, you have an actual criticism of the methods of the paper – and by “actual” I don`t mean “it disagrees with Dubya” or “that can`t be right!” or “it doesn`t concur with another study which I happen to believe is a fraudulent crock of shit” – then feel free to write a letter to the editor of the Lancet explaining your position. I`m sure they`ll be willing to publish whatever analysis you bring to bear on the substantive elements of the paper – i.e. the sample design, the sample weighting or the statistical comparisons. If you don`t have any argument with these elements of the paper then accept the figures presented and get over it.

Mike H, this issue of confidence intervals is not a smoke grenade or some other dodge, it`s an essential part of the consideration of the results. Given what you and Brownie have presented of the authors` statements in the conclusion one could assume that they have drawn too strong a conclusion about the comparability of the studies, but it is quite common for scientists to overstate the results of their own papers, and such an act in the discussion in no way invalidates the results of the paper. Choosing to disagree with their conclusion that the papers are comparable, and instead concluding that statistical error in the first paper prevents comparison, in no way invalidates the second paper (or the first).

Is it also not the case that in Lancet 1 they excluded the Fallujah cluster? Had they included it the confidence intervals for the violent deaths estimates would have been narrower, and the point estimate more like the point estimate from Lancet 2. In a way this whole discussion is driven by imprecision which the authors themselves introduced in order to make the paper less biassed. I think they should get some credit for that, regardless of whether you think their conclusions about comparability were too strong.

Given what you and Brownie have presented of the authors` statements in the conclusion one could assume that they have drawn too strong a conclusion about the comparability of the studies,
I don’t think it’s fair to assume that. As Robert pointed out more than once, you can get a significant agreement between studies about the total number of deaths without getting a significant agreement about the total number of non-violent deaths in particular, and the authors never claim the latter.

I would suggest that, given the statistical insignificance of the change in non-violent deaths, one of the authors shouldn’t have later listed the breakdown of excess deaths by cause the way he did after Lancet 1, since apparently the error bars on each of those numbers was so huge as to make them totally uninformative. However, the published study didn’t make that claim.

Is it also not the case that in Lancet 1 they excluded the Fallujah cluster? Had they included it the confidence intervals for the violent deaths estimates would have been narrower, and the point estimate more like the point estimate from Lancet 2.

No, actually, if it’s included the confidence intervals become much larger and the point estimate’s well above Lancet 2. Lancet 2’s between the two estimates you get from Lancet 1, including vs. excluding Falluja.

As I said on another thread, that makes sense to me–Falluja did happen, and really should raise the death count, but the actual Falluja cluster data they got in Lancet 1 is insanely high and (as the authors themselves point out) may not accurately represent the entire 3% of Iraq’s population that cluster was supposed to cover.

I’d encourage everybody to read the Iraq Body Count press release on the latest Lancet study:

http://www.iraqbodycount.org/press/pr14.php

It’s sensible, from a non-warhawk source, and does raise many reasons to doubt the scale of the Lancet 2 results. They take no position on what if anything might have biased the study sample.

One thing for sure at this point: the initial (Lancet 1) estimate of 100,000 excess civilian deaths is looking not just plausible but actually pretty low. Even the IBC’s passive count based completely on press reports of violent deaths alone is approaching 50,000. And passive counts are almost guaranteed to be underestimates. I suppose a few years down the line the Lancet people will come out with an estimate of 1 million and we’ll all be arguing vociferously over whether it’s only 400,000.

“Philosopher, I’m not sure what you mean in regard to ” without something confidence intervals…..” Did you miss a word?” Yeah, sorry, that was supposed to be “something like confidence intervals”, since there are other mathematical ways of getting at significance than the intervals per se. (There’s already been discussion on this thread about p-values, for example.) But nothing in this particular conversation is turning on that.

mq is right about how completely fundamental the distinction between the point-estimate and the confidence intervals are. I’m sure some defenders out there are sloppy about this, too — like I said, it’s a very common misunderstanding that people without any stats training make all the time, regardless of political persuasion. But all the people who seem to know anything about stats are putting a lot of emphasis on the CI, which is as it should be. The point-estimate is of course part of that — the CI is an interval around the point-estimate — and it is also of course easier to report, talk about, and throw around, but it is only part of the story, and a meaningless part without some indication of statistical significance.

Sometimes an extreme hypothetical case can help illustrate matters. Suppose someone did a truly awful survey of Iraqi mortality, in which they only check on 50 people; and with those 50 people, there had been 1 pre-invasion deaths and
4 post-invasion deaths. Even for the lower estimates, this is an entirely plausible kind of result for such a small sample — small samples are more hostage to the luck of the draw than large samples, which is why you want large samples when you do this right. Now, if we don’t check at all for significance, then we have something like a 20 per 1,000 pre-invasion death rate and a 80 per 1,000 post-invasion death rate. Multiply that difference by the population of Iraq, and you get something like
2.1 million excess deaths. Such a huge point estimate! But I hope that I have set this up so it is clear why the size of the point-estimate is simply no indicator whatsoever of whether you’ve actually found something statistically significant. In this case, the sample is so small, that the CI would be absurdly huge. Indeed, I suspect that if one quickly crunched the numbers, this wouldn’t even be a big enough difference, for samples so small, to indicate any statistically significant difference between pre- and post- at all, let alone one of such a magnitude. (It’d be pretty hard to guarantee representativeness in a sample that small, too, but that’s another matter.)

Moral of the story: the bigness of point-estimates or their differences is just no indicator whatsoever of any significance of the effect.

* * * * *

I’ve just looked back at the earlier _Lancet_ paper, and if I didn’t miss anything (and, hey, it’s 3:30am where I’m at, so it might have slipped past me), then there’s something very telling in it, for the particular line of argument that Mike H. is running: _the authors never make any claim to statistical significance for an increase in nonviolent deaths_. On p. 1861, there is a specific claim about a “58-fold” increase in _violent_ deaths, as well as the increases in the _overall_ death rate — but I’m just not finding anything reported in there about specifically nonviolent deaths that has a CI attached to it. And it looks like Lancet II is the same in this regard. The closest they come to making any such claim is in the following from p. 5, but if you look closely you’ll note that no claims to significance are made there: “Excess mortality is attributed mainly to an increase in the violent death rate; however, an increase in the non-violent death rate was noted in the later part of the post-invasion period (2005–06). The post-invasion non-violent excess mortality rate was 0·7 per 1000 people per year (–1·2 to 3·0).” See? No CI reported, and on p.7 they explicitly say that this was not significant. (Did someone already say this part up-thread somewhere? My apologies for being redundant, if so.)

So, Lancet I makes no claims about nonviolent deaths increasing; and Lancet II likewise finds nothing of statistical interest to report about nonviolent deaths. So in this regard they are totally consonant with each other.

There’s all kinds of caveats to attach to this, but there’s as yet no historical basis to claim a civilian death rate due to bombing higher than 0.2 deaths per sortie.

In Hiroshima 100,000 people were killed by one airstrike. So, it’s between 0 and 100,000.

philosopher,

I appreciate your efforts to explain, but even for we non-statisticians the concept of a confidence interval is pretty well understood.

When Kaplan talked about a “dartboard” with respect to Lancet 1, he was ridiculed by supporters of the study who rushed to explain why 8,000 was not as likely a figure as 98,000 for projected excess mortality in Iraq. The net must be literally bursting with examples of comments and posts from sites like CT where the fundamentals were trotted out for the benefit of people like Kaplan. Similarly, it’s not difficult to find quotes from Horton or any of the Hopkins authors in which the 98,000 (more commonly referenced as 100,000) is mentioned as the best estimate we have for excess mortality.

Two years on and when some observers compare the figures in Lancet 1 with Lancet 2, we’re told that we ignore the confidence interval at our peril. Well, okay, let’s not ignore it, but let’s at least be consistent with the emphasis .

the authors never make any claim to statistical significance for an increase in nonviolent deaths.

It really isn’t the argument of those raising questions that the authors make this claim. The fact remains, whilst non-violent deaths account for 40% of the 98,000 point estimate for Lancet 1, they’ve dipped below zero (in that the rate has gone negative from the pre-invasion baseline) for Lancet 2’s 112,000 (or 130,000 extrapolated). Over 40,000 non-violent deaths have been replaced by an additional 60-odd thousand violent deaths.

Provide 99 statistics-based reasons how such a discrepancy may have come about, but then please go on to explain to me why the authors are still within their rights to say the studies are mutually corroborative?

The only sense in which the studies can be said to be mutually corroborative on the specific question of composition of excess deaths, is that the confidence interval’s in both overlap, which, as any fule no, is no corroboration at all.

This is hardly an unimportant point, given mutual corroboration is one of the strongest indicators that we can have faith in the results produced by similar studies. Which is probably why the authors both to claim corroboration at all.

Regarding air operations in Iraq: Close air support sorties are those flown in direct combat support of troops. It’s in the nature of CAS that the aircraft will only fire (or indeed do anything rather than circling) if they get a call from the forward air controller with the troops, unless somebody has a Canadians-in-Afghanistan speed-freak brain storm, so the ratio of strikes to sorties will be low. I’d also point out that more of this will be going on in Iraq than Lebanon, as there are five and a half coalition divisions spread out over a wide area, more than twice the maximum number of troops the Israelis ever sent into Lebanon.

Also, the Israelis launched a quasi-strategic air offensive, which implies they picked a list of targets off the map and set out to bomb them – so, once discounted for the inevitable aircraft that go technical, find the target weathered in, fail to meet the refuelling tanker etc, you’d expect the two numbers to be similar.

You can believe 655000 Iraqis have died.
But only if you believe it was paradise under Saddam.

The Lancet this week published a report claiming that 655,000 people have been killed in Iraq.
It is based on flawed research that would shame any publication, except one that had previously been responsible for giving Andrew Wakefields MMR “research” the oxygen of publicity.

The research is based on the pre war death rates under Saddam Husseins regime before the 2003 invasion, assessing those afterwards, and labelling the difference war related deaths.

Unfortunately the researchers have made two huge errors.
Their figures only stand up if adult Iraqis living under the Ba’athist regime hardly ever died. Either of natural or violent causes.

Secondly they included the data to undermine their conclusions in their own report.

The researchers claim the death rate in pre war Iraq was 5.5 per thousand and backed up this claim in a country of 25 million people with an analysis of 82 deaths. Thats a death rate, half that of the EU but perfectly possible in a “young” country with a high birth rate, low median age and good life expectancy.

Unbelievably the same research claims infant mortality was 10%.

5.5 deaths per thousand means that 137,500 Iraqis died each year.
10% infant mortality in a country of 34 births per thousand and 25 million means that 85,000 of these deaths were under one year old.

Leaving a death rate for Iraqis over one year old of 2.1 per thousand.

Less than half that of the lowest country in the world.

Paradise.

And on the subject of violent deaths pre invasion?

.

They drew the following conclusion “As there were few violent deaths in the survey population prior to the invasion all violent deaths can be considered “violent excess deaths”
82 deaths analysed.

Read that again. No violent deaths under Saddam.
Not even domestic violence. Prison works.

This may tell you something about the motivation and rigour of the analysis.

Or a desire for headlines.

The World Health Organisation gives a figure of 9.03 deaths per thousand in Iraq pre invasion.

A figure which ,put through the Lancet prism would have reduced its headline death toll by hundreds of thousands.

And the headlines by a few.

Tim

Secondly they included the data to undermine their conclusions in their own report.

That’s not an “error”. That’s called science.

They drew the following conclusion “As there were few violent deaths in the survey population prior to the invasion all violent deaths can be considered “violent excess deaths”
82 deaths analysed.

Read that again. No violent deaths under Saddam.
Not even domestic violence. Prison works.

Read that again. “Few violent deaths”. “Few” doesn’t mean “no”.

You, Sir, are an intellectually vacuous liar, or else functionally illiterate.

alex,
for the purposes of their research they assume none.therefore.
” all violent deaths can be considered “violent excess deaths”

But only if you believe it was paradise under Saddam.

Yes, paradise – or perhaps a well-run police state.

Don Johnson, 71: That point is a reasonable one, and some of the statistics and sampling questions are reasonable too, but all reasonable points on these questions are being swamped, and that was inevitable. And the statistical arguing makes swamping easier, not harder.

That the Lancet attempt at precision was, in the context of this particular debate, doomed. (If they cut a few corners to make “the best case”, it was even more doomed, of course).Any report they made whatever would have received enough flak that it wouldn’t be regarded as credible by the media. And while I fully support the political purpose of the study and think that their release date was perfectly timed, I stillthink that their effort was futile.

The population able to understand statistics and willing to have their mind changed by a careful statistical argument on a highly contentious issue is small, and there’s so much junk statistcs out on various issues that the good statistics are automatically devalued (just because so few people really understand stats, and also because so many don’t trust anyone at all).

My opposition to the war is mostly based on its fraudulent justifications, strategic stupidity, and failure, rather than on the brutality or not of the American troops. Based on what you’ve said, Id expect to see lots of war supporters say that we need to “take the gloves off”, since we’re killing too few Iraqis.

Charlie,

I think the problem is that you can’t meaningfully do arithmetic with point estimates.

Except, apparently, when you’re looking to use one point estimate in one study to corroborate another in another.

Are you saying that there is no statistical merit in the claims by the authors that the studies are mutually corroborative because the point estimates are in the same ballpark? If so, I’ll settle for that.

The second it bcomes legitimate to do the like for like comparison of point estimates and use this as corroboration, we surely are entitled to point out the glaring contradictions in the constitution of each?

Apple pies and steak pies are both pies, but I’d only put custard on the former.

103: Are you saying that there is no statistical merit in the claims by the authors that the studies are mutually corroborative because the point estimates are in the same ballpark? If so, I’ll settle for that.

I’m not competent to judge the study’s authors in that way, but it could be that their statement was incautious, yes.

It’s still the case that researchers have (twice now) been to Iraq and asked some of the people there how many in their households had been killed since the invasion. And they came back with some numbers. The problem we are all faced with is finding an explanation for those numbers. The choices here are, essentially:

– The death rate from violence is significantly higher than passive reporting suggests
– The respondents were lying
– The researchers committed a major error in their sampling methodology

And it’s possible that we could combine these explanations in some way.

There is no ‘jury out’ situation here. A criticism that doesn’t amount to more than a criticism of some of the rhetoric of the presentation does little to get you closer to understanding why the numbers are as they are; nor, crucially, does it make the numbers go away. In my view, without a plausible explanation for these numbers, concern is the appropriate response and hand washing should wait.

There is no ‘jury out’ situation here. A criticism that doesn’t amount to more than a criticism of some of the rhetoric of the presentation does little to get you closer to understanding why the numbers are as they are; nor, crucially, does it make the numbers go away. In my view, without a plausible explanation for these numbers, concern is the appropriate response and hand washing should wait.

It’s a little deeper than that, Charlie. One of the strongest indicators we could have that Hopkins has things about right is if Lancet 2 validates and corroborates Lancet 1. I suggest that’s why the authors have claimed this to be the case.

You say it could be that such claims are “incautious” yet don’t believe you are qualified to judge. Well, if the basis for claiming corroboration was a mystery beyond the comprehension of all but statistical sciences PHDs, then you might have point, but the basis for such a claim is a 98,000 point estimate in Lancet 1 and 112,000 in Lancet 2 for the same period. They’re in the same ballpark, aren’t they? The point is that the composition of each figure is entirely contradictory, invalidating any claim of mutual corroboration. Over 40% of deaths from one cause in Lancet 1 have been replaced by the only other possible cause in Lancet 2.

The importance of non-corroboration cannot be overstated, as it immediately invites the conclusion that both cannot be right, which in turn means one must be wrong.

You can fill in the rest.

Kevin,

The point of contradiction is not that Lancet 1 said the greater number of deaths was 30-40 year old shoemakers from Baghdad, whereas Lancet 2 claims it is 20-something plumbers from Mosul. The deaths have one of two causes: violence or non-violence. Even allowing for double weight of misattribution, to go from 40%+ non-violent deaths in Lancet 1 to sub-zero in Lancet 2 takes some doing if all bias has been avoided.

Moreover, that mortality has increased is not in dispute. If the authors used the findings of Lancet 2 and Lancet 1 to claim mutual corroboration for an increase in mortality and nothing more, you’ll get little argument from me or anyone else, I suspect. But they specifically went further than this. They highlighted a point estimate of 112,000 deaths for the first 18 months of the conflict and compared this with the 98,000 estimate from Lancet 1. They are explicitly claiming mutual corroboration not just for a rise in mortality, but in the estimation of that rise.

Given the finite number of causes of death is 2 (violent and non-violent) and not 22, the composition of both estimates is significant and where they are found to be contradictory – as they are – it is evidence against mutual corroboration on the specific question of point estimates.

If the authors want to row back on the corroboration claims about the point estimates for the first 18 months of the year, than I’ll accept that composition of those point estimates becomes less significant.

Hey!

With all due respect, I think something important is being ignored.

John: “The best source turns out to be the US Air Force Command itself.”

Yes, but I think you need to pay less attention to what they say about “airstrikes,” and more attention to what they say about tons of bombs dropped.

As far as I can tell, the above analysis is largely focused on number of aitstrikes/sorties, and relies on statistics released by the US. Trouble is, guns don’t kill people; bullets kill people. Airstrikes don’t kill people; bombs kill people. (I realize airstrikes deliver munitions aside from bombs, such as bullets, but I’m more concerned about the bombs.)

While the US does release certain vague figures regarding “airstrikes” (a term which I haven’t seen defined very clearly anywhere), it seems very, very reluctant to tell us the total tonnage of bombs dropped on Iraq.

Information can be found in daily “CENTAF airpower” summaries such as this, but I think the information is purposely vague, and never reveals tonnage, as far as I can tell (we see terminology such as “airstrikes,” “missions” and “sorties”). John cites Jamail’s article which takes a close look at those summaries. By the way, I’m puzzled by this statement of Jamail’s:

Carrier-based Navy and Marine aircraft flew over 21,000 hours of missions and dropped over 26 tons of ordnance in Fallujah alone during the November 2004 siege of that city.

I can’t figure out where he got that from (“26 tons”). I also think the idea that Fallujah received only 26 tons “during the November 2004 siege of that city” is dramatically understated.

Anyway, as far as I can tell, the US has released such information (tonnage dropped on Iraq) exactly once, here:

So far for OIF II, 3rd MAW has dropped more than 500,000 tons of ordnance … but this number is likely to be much higher by the end of operations

I believe this statement has been used as the basis for reporting by Hersh, here:

Since the beginning of the war, the press release said, the 3rd Marine Aircraft Wing alone had dropped more than five hundred thousand tons of ordnance

(John, you point to Solomon, who points to Hersh, who points to the US press release. However, Solomon and Hersh leave out URLs, which I’ve provided.)

As far as I can tell, the Marines press release has otherwise been overlooked, in general, even though I think it is credible and unintentionally revealing. When that statement is analyzed (please follow the links cited above), I think it provides strong support for the hypothesis that any analysis based on number of “airstrikes” dramatically understates the situation.

The cited quote indicates that the 3rd MAW was dropping bombs at a rate of 2,000 tons per day.

If the 200 aircraft of the 3rd MAW fly to Fallujah one morning and drop, in aggregate, 2,000 tons of explosive, is that what the US calls one “airstrike?” I think the data shows that the answer is probably yes. For obvious reasons, this is a big problem, and I think it’s a problem that’s being completely overlooked.

Consider that this one hypothetical “airstrike” has delivered about 10 pounds of explosive for each person in the town. Therefore this one “airstrike,” if perfectly efficient, could kill 350,000 people.

By the way, it’s been reported that 20% of the buildings in Fallujah have been totally destroyed. The coverage area of the 3rd MAW includes Fallujah.

Detail supporting the above can be found by following the links cited in #64. This includes a reasonability check showing that the 3rd MAW is indeed capable of dropping 2,000 tons per day. I think when a lot of people see Hersh talking about 500,000 tons, they figure it must be a typo. I think there’s very good reason to understand that it’s not. By the way, a reasonability check based on cost per bomb (and comparing this to total expenses for the war) also checks out nicely.

“An average of 10 fatalities for each air strike seems plausible”

I think a plausible range of per-airstrike fatalities could be 0-300,000. I think the US is deliberately using a term that is utterly vague and therefore meaningless. That’s why I think we need to pay attention to tonnage, not “airstrikes.”

#76: “the problem is that we don’t have precise data on US strikes”

Exactly. That’s why we need to look beyond counting “airstrikes.”

By the way, Hersh makes sort of a factual error that has the effect of understating his case. He says this:

Since the beginning of the war, the press release said, the 3rd Marine Aircraft Wing alone had dropped more than five hundred thousand tons of ordnance

Strictly speaking, that is true. However, it implies that 3rd MAW started dropping bombs at “the beginning of the war.” They didn’t. They weren’t Iraq until 2/04.

In other words, someone reading Hersh could logically conclude that 3rd MAW dropped 500,000 tons in a period of 20 months. That’s incorrect. The actual period was 9 months.

Alex wrote
Let us get down to limiting cases. Baghdad makes up something over 20 per cent of the population, and 40-50 dead shot people are found daily. If the same pattern is national, that would be 160-200 sectarian war victims a day

1.the situation in Baghdad every day since the start?
2.Its not national.
Large areas of the Shia south and Kurdish north are relatively calm.

They are explicitly claiming mutual corroboration not just for a rise in mortality, but in the estimation of that rise.

I don’t see what you are getting at here. On my reading, they are saying that both studies show increased mortality and the magnitude of the increase is about the same. Putting it another way, if the national figures corresponded exactly to the point estimates in the 2004 study, it wouldn’t be at all surprising to get a sample like the 2006 sample. And vice versa.

To anyone who says we shouldn’t pay too much attention to point estimates, I nod in agreement. Daniel Davies made that point in his CIF post and he also said that about the 2004 study way back then. As he says, the right way to look at these things is to ask: if the mortality situation really wasn’t all that bad, what would be the chances of getting a sample like this?

I see Mike H complaining that defenders of the first study pushed the point estimates too aggressively. For my part I plead not guilty, but I concede that some people did. When Les Roberts got away from the Lancet referees he got a bit excited at times. That’s understandable but so is the fact that IBC are annoyed with him.

I said this: “When that statement is analyzed (please follow the links cited above).”

It would have been clearer if I had said this: “please follow the links cited in #64.”

On my reading, they are saying that both studies show increased mortality and the magnitude of the increase is about the same.

My issue is with the “magnitude” component. The magnitude is comprised one way in Lancet 1 (>40% 0f 98,000 deaths have a non-violent cause) and another in Lancet 2 (all 112,000 deaths from violence).

Where is the legitimacy of a claim that “the magnitude of the increase is about the same” if the respective compostions of the increases are contradictory? Again, where respondents were having to perm one from 50 when attributing cause, differneces in composition lose significance. Where the choice is “violent” or “non-violent”, nothing explains such a swing in attribution from study to study absent bias of some sort.

At the very least, such differences disallow claims of mutual corroboration, if they don’t threaten statistical validity directly.

The UN estimates that around 35,000 homes and businesses were destroyed in Lebanon during the war from all causes (air, artillery). Other estimates I’ve seen speak of 15,000 homes destroyed in South Lebanon. The Lebanese death toll is estimated at 1200.

This strongly suggests that very few of these destroyed buildings were occuppied at the time they were hit.

You know, I wish I’d been told at the time that the Israelis were being so discriminating and humane. It wasn’t the impression that I got from following anti-war websites and newspapers, but there you go.

Second, the figure of 40,000 claimed as the number of deaths recorded by the MoH in 2002 is false. No specific citation is offered by the Lancet authors for this figure other than a vague attribution to “informed sources in Iraq”. But official Iraqi figures for 2002, forwarded to IBC courtesy of the Los Angeles Times, show that the Ministry registered 84,025 deaths from all causes in that year. This excluded deaths in the Kurdish-administered regions, which contain 12% or more of the population.

Now that’s a pretty significant disagreement. I wonder if we can get more detail on either the Lancet authors’ “informed sources” or the IBC’s “official figures,” and whether or not both were based off counting death certificates?

“I appreciate your efforts to explain, but even for we non-statisticians the concept of a confidence interval is pretty well understood.” I’m afraid that this claim is patently falsified by your own discussion on this thread! Or, if you get the general concept of it, you seem unable to apply it to the particular case at hand.

“When Kaplan talked about a “dartboard” with respect to Lancet 1, he was ridiculed by supporters of the study who rushed to explain why 8,000 was not as likely a figure as 98,000 for projected excess mortality in Iraq.” But that is exactly something that _does_ follow from understanding the CI! In a study like this, not every value in the CI is equally likely, and indeed based on the statistics the point-estimate is the most likely value. 98,000 _is_ a more likely result than 8,000; an appropriate comparison to the 8,000 would have been the other extreme end of the interval, 194,000. They got statistically significant results in Lancet I, and those results put that 98,000 figure in the middle of the CI, and there’s nothing wrong with their reporting that. And they got statistically significant results in Lancet II, and for the corresponding period of time they had a CI of 69,000 – 155,000, with the peak at 112,000. One could do some more statistics on this to see just how likely either result is given the other, but that the latter study’s CI for this value is completely contained within the former’s is, indeed, rather corroborative.

“My issue is with the “magnitude” component. The magnitude is comprised one way in Lancet 1 (>40% 0f 98,000 deaths have a non-violent cause) and another in Lancet 2 (all 112,000 deaths from violence).” This kind of line that Brownie and Mike H. are running only makes sense if that claim about the magnitudes was itself something that the statistics showed. And, as I argued in my earlier comment, it wasn’t, and the authors never claimed that it was. Their data do not support putting any particular weight on _those_ point-estimates, because the confidence intervals around them are too large — they are like my hypothetical case with the 50 Iraqis. They are, given the data available, statistically unsupported claims. They have statistical significance for the overall mortality claims; and for the claim that there was an increase in the number of deaths attributable to violence; but not for the part of the data that you want to put so much weight upon. And this is the case with both studies, in that they found nothing statistically significant with regard to the nonviolent deaths.

Basically, everything that Lancet I found with significance, Lancet II did as well, and in a way that corroborated Lancet I; and in this particular area where Lancet I found nothing of statistical significance, so too did Lancet II not find anything of statistical significance. That’s pretty darn good.

Brownie and others: Lancet 2 does corroborate Lancet 1. This is based on estimates of the *total number of deaths* in periods where the two studies are both close to each other and precisely estimated. Of course you are going to be able to find a subsample which is imprecisely estimated where the point estimates from the two studies are far apart, but the variance is so high that the difference doesn’t mean anything. Again, confidence intervals *are not optional*, you need to *actually understand them* and not just pay lip service if you want to understand sampling results. I’m sorry that someone once criticized you for taking the lowest level of the Lancet 1 confidence interval as the most likely real-world result, but that has no bearing here.

Tim at 93: your comment is inaccurate enough that one suspects political motivation. There is no estimate of infant mortality in the Lancet studies, their sample is not large enough to make conclusions about infants alone. You cannot compare subtract infant mortality rates from some other study to the overall Lancet mortality rate because they are done using completely different methods. This is especially likely to be true for infant mortality, by the way. Also, as people have said over and over again here, the Lancet study methodology is based on comparing the difference in pre and post-invasion mortality from *answers on the very same survey*, which is the right way to do it. Errors from the study methodology should cancel each other out in the differencing. Fishing around in old UN stats to cherry-pick a number that disagrees with the Lancet figures is just motivated by a desire to smear the study. It is also true (confidence intervals again), the Lancet estimate of pre-invasion mortality is easily compatible with mortality rates above 6; indeed the top of the CI is over 7.

I wish someone unbiased would actually comment on the IBC press release, instead of having to wade through ignorant right-wing propaganda all the time. But here’s a quote from the blog “Healing Iraq” that I got from Steve Sailer’s web site. It certainly gives some sense of how bloody things actually are in the country.

“I have personally witnessed dozens of people killed in my neighbourhood over the last few months (15 people in the nearby vicinity of our house alone, over 4 months), and virtually none of them were mentioned in any media report while I was there. And that was in Baghdad where there is the highest density of journalists and media agencies. Don’t you think this is a common situation all over the country?”

It’s obvious enough that John Emerson is right: the study details may be distracting us from the simple fact that things are very bad. This is a humanitarian disaster, even if the current Lancet numbers are off excess deaths are probably well into the six figures.

114: On the specific question of composition of point estimates, the studies are indeed contradictory.

But I’m really not sure that the point estimates have a composition in the arithmetical sense; i.e. that you can take the percentages from the sample and apply those same percentages to the estimate. I think your argument would be better if it were framed around the sample data. Are there questions to be asked of Lancet 1? Maybe the Fallujah issue (in or out; either looks odd) or maybe the infant mortality issue (many of the reported non-violent deaths were neo-natal).

That these two surveys were carried out in different locations and two years apart from each other yet yielded results that were very similar to each other, is strong validation of both surveys.

Well, as I said, this may have been an incautious claim, but you really shouldn’t look to the likes of me for validation or otherwise of the statistical methods. My comment is on the language attached to the numbers; a somewhat different thing.

And you’re arguing with someone who accepts the general finding – that there have been significant excess deaths in Iraq since 2003, and that the rate may still be rising. Realistically, I don’t see what other situation would result in Iraqi households reporting deaths in those numbers.

‘Borders on the mendacious’ is pretty strong. Do you suspect the authors of the studies to be acting in bad faith? To the extent that they would fabricate the data?

What does “borders on the mendacious” mean, exactly? Hard to know, really. Is Brownie accusing the Lancet team of dishonesty? If so, why not just say so?

Since the same methodology was used to estimate deaths in Darfur and the Congo, I wonder if the Lancet critics want to go through them as well. The recent Lancet article on the DR Congo claimed that there were 1250 excess deaths per day. The method used was the same spatial sampling survey technique. The estimates of mortality in Darfur also rely on similar sampling techniques.

Now, I wonder why nobody is going through those Darfur and Congo reports with a fine toothed comb. The sudden interest in statistical method in Iraq somehow hasn’t been extended to the whole system of estimating mortality that has driven the “moral interventionists” in the first place. They are whacking away at the very source of the last bit of their own shaky moral legitimacy.

I hate having to play catch up in a busy thread, but sleep and everyday life make that inevitable.

To get back up to speed, I’ll have to be brief to some commenters who have directed posts to me. I’ll split it up between posts, in the interest of coherence.

Anyway, Brownie, I’d be remiss if I didn’t acknowledge you’re doing an excellent job pressing home my argument. Far more eloquently and concisely than I have, I might add.

Philosopher, MQ:

As Brownie notes in #89, us statistician neophytes have developed some understanding of the role and importance CIs play in these types of surveys. But you’re still missing my point. There is a tendency for supporters of the study (particularly the first Lancet) to freely interchange the downplaying/playing up of point estimates and CIs, whichever a particular discussion calls for at the time.

As Kevin has noted with his comment about Roberts, it isn’t non-stats types who have been guilty of this. Horton, the editor-in-chief of the Lancet, was also guilty of citing the 100,000 figure as gold (although Roberts, to his credit, later corrected him). I’ve personally encountered many knowledgable defenders of the study defending point estimates, not just on the total excess death toll, but on many subsets below the two main substes of violent and non-violent.

“It just means that your attempt to cherry-pick hopelessly imprecise numbers from subsamples and say that they are implausible is straight-up propaganda.”

MQ, I’m working exclusively with the two largest subsamples, violent and non-violent, and if these numbers are “hopelessly imprecise,” then what does that say about the total point estimate? But thanks for agreeing with me.

“So, Lancet I makes no claims about nonviolent deaths increasing; and Lancet II likewise finds nothing of statistical interest to report about nonviolent deaths. So in this regard they are totally consonant with each other.”

Let me see if I have this right, Philosopher. Lancet 1 ” makes no claims about non-violent deaths increasing,” but adds 40,000 excess non-violent deaths to its excess death point estimate, without which the study could not achieve its headline grabbing 100,000 figure. These 40,000 excess non-violent deaths are only statisically relevant because they can piggy-back onto the excess violent ones, even though the 40,000, by themselves, represent a 22% increase over the baseline mortality estimate for the 18 month period.

From a non-statistician’s perspective, that’s a pretty neat trick, even if it is ” statistically sound.”

The other point I’d be interested seeing someone tackle is the assumption that we can have several hundreds of thousands of Iraqis die violently in slightly more than 3 years, and yet see Statistically relevant rise in death from non-violent causes.

The Confidence Interval guys might want to sit this one out. The lower bound of the CI still comes in at 425,000 plus violent deaths.

Kevin:

“This is wrong. If, in sample A, mortality has increased and the main cause is traffic accidents, while in sample B mortality has increased and the main cause is heart disease, the conclusion that mortality has increased is indeed reinforced. As to the main cause in the population as a whole, “further research is required” – a conclusion which appears in all too many studies.”

That still leaves you with unreliable estimates for heart disease and traffic accidents, because the studies conflict. Would you be able to cite with any confidence that heart disease had truly gone up, and traffic accident deaths had declined?

In any event, violent death is unique. We don’t cluster sample it to obtain numbers in modern western countries, so far as I know. We count it. As I mentioned at Deltoid, I can see an Iraqi interviewee mistakenly telling a Lancet surveyor that a family member died from heart attack, when they really had died in the final stages of cancer. That has no effect on the violent/non-violent distibution of a study. Both are non-violent causes of death.

But I don’t see how an interviewee would mistakenly believe a family member had drowned, when they were found in the street with a bullet in their head.

In my view, the distinction between non-violent and violent deaths is too fundamental to allow statistical methodology to excuse huge variances from study to study. I’m running out of ways to express that, and I apologize for the repetitiveness of the premise.

137 should read

” …. and yet see no statistically relevant rise in death from non-violent causes.”

Good to see you back, MikeH. I’ll have to leave you to it as I’ve a stack of work to do for a meeting tomorrow.

Indientally, my wife just made me a sandwich. I asked for ham but she’s done blueberry jam. She claims the filling is statistically insignificant; I asked for a sandwich and I got a sandwich.

I blame philosopher et al…

Lopahkin: are you deliberately trying to miss my point? The stuff in Appendix E of the study is drawn from completely different sources and is not necessarily comparable with the survey results. You can’t add and subtract numbers collected from different sources that way.

Mike: Jesus, people keep pointing out basic stats to you and you keep replying with this weird rhetorical stuff about some subsample being important or fundamental or whatever. “Important” or “fundamental” are nothing but handwaving and speechifying from you. What is on the table here is actual, real, sample data. If you want to say something, say something about what conclusions can be drawn from the actual sample, or demonstrate why you think the sample is biased. What you are saying about the trend in non-violent deaths verges on being meaningless. The number of cases of non-violent death reported by respondents is small enough that the range of possible values for the difference is very large, especially over time small periods of a few months. The number of cases of total deaths is high enough that statistical theory allows us to bound the difference well away from zero. This is very elementary indeed. Yes, it is true because we are extrapolating from a small number of reported cases to the population of Iraq. But many good U.S. election surveys extrapolate from a smaller number of reported cases to a larger overall population. Either you believe in statistical methods or you don’t. I’m starting to get the feeling you really don’t, in the fashion that people don’t believe in what they can’t understand. The “confidence intervals” *are* the result of a random sample.

As for violent death: it is not unique, it’s like any other event in a population. The reason we don’t cluster sample it here is that we have good administrative mechanisms for tracking and reporting it. They don’t in Iraq. This is well covered in the study.

It looks to me like the UNDP results are compatible with estimates of around half of the Lancet ones. Which is still massive carnage. Still don’t understand the source of the difference. The Lancet people say it’s because their survey focused more on deaths, but I’m not sure why that alone should make such a difference. Deaths in the household aren’t hard to recall.

We’re now up to post 142, and I still can’t get any of the critics of the Lancet studies to address the fact that we’ve dropped at least 500,000 tons of explosives on Iraq, given that this alone is very likely to have created somewhere in the vicinity of 100,000 or more deaths.

I see plenty of ” takers ” for your offer, anon. Much of this thread involves a discussion of how much ordnance has been dropped, and what effect it has had. Both sides seem well represented to me.

I guess you haven’t been paying attention.

But that is exactly something that does follow from understanding the CI! In a study like this, not every value in the CI is equally likely, and indeed based on the statistics the point-estimate is the most likely value. 98,000 is a more likely result than 8,000;

This graph was used to correct me a few posts back:

http://anonymous.coward.free.fr/misc/roberts-iraq-bootstrap.png

Unless I’m missing something, looks like it can ne used to correct you too…

Anon in 142: I can’t attest to your 500KT figure — love to see the source — but some historical comparisons would be the First Gulf War, where 85KT were dropped, Kosovo (17) or Afghanistan (9). In fact, a meatball estimate based on my own knowledge of the equipment involved, sortie rates, etc. would put only about 20KT onto Iraq in the first six weeks of war, and something under 10KT since, even if I use John Quiggin’s estimate of 5 attack sorties per day, which I honestly think is a little inflated and unsupported by his citation. But I’m confident you’re off there by at least a factor of 10, regardless.

That’s still a lot of bombs, though: in Afghanistan the civilian fatality rate per KT was 140-150, which was something of a historic high. The same fatality rates in Iraq would put Iraqi civilian deaths due to airstrike alone in the vicinity of 5,000. Add in 20,000 or so insurgents also killed and it’s a significant number. It is not, however, anywhere near the number given in this Lancet study.

PS: Anyone who wants to challenge any of the above, please avoid citing the famous 3MAW press release during Fallujah, which was a pretty clear instance of mixing up tons and pounds. Cheers.

I think it’s the size of any given subsample relative to the population that’s key, not the number of other subsamples, nor its conceptual ‘level’. The smaller the sample, the harder it is to estimate with confidence, and below a certain size you can’t say anything meaningful. But analogy will only take you so far: at some point the maths kicks in, and I don’t have any experience with that.

I can see how the Lancet 1 sample could be troubling – conceptually – because, when Fallujah is excluded, the number of deaths attributed by respondents to violence is small: 21. But I think you’d need to work some maths to get insight into the significance of any estimate made on the basis of this number.

Okay, Mike, I get it now. I looked at your points upthread, as well as Brownie’s at 140. Your problem is apparently not with the statistically insignificant drop in non-violent deaths right after the invasion found in Lancet II (this criticism would make you pretty much completely innumerate, and is what I was reacting to). Instead, you have an issue with the difference in the fraction of first year post-invasion deaths that were non-violent in Lancet I vs. the fraction that were non-violent in Lancet II. Your beef seems to be that although the overall death rates in Lancet I vs. Lancet II are similar for the relevant period, the two component death rates (violent and non-violent) that make up the total have switched around somewhat. Is that it?

You shouldn’t be surprised that that criticism has opened you up to charges of fishing for something to discredit the study. Too take an analogy: I take two separate polls of 10 people about support for President Bush. In the first poll, I get 2 men and 1 woman saying they support him, with the rest not supporting him. In the second poll, I get 2 women and 1 man supporting him, with the rest not. Do the results from the two polls corroborate each other? Yes. Do the results from the two polls on the percent of his supporters who are men corroborate each other? No, one would need a larger survey to get more accuracy here. The results are inconclusive on the percentage of his supporters who are men. Likewise, the results here are inconclusive on the percent of first year post invasion deaths that were violent vs. non-violent.

Should the fact that a subsample result is significantly different between two polls on the same topic, while the overall result is the same, make us suspicious? This actually is a pretty complex question. The answer is: not necessarily. Just randomly, one would expect some of the subsample results to differ at a statistically significant level. But one wouldn’t expect all of them to differ. (At a 10% significance level, you’d expect one out of 10 measurements from different samples taken from the same population to differ). If every possible subsample that could be taken differed, but there was the same overall result then that would be suspicious. One needs a way of conceptualizing the “population” of subsamples that make up the whole to formalize this. Saying that some subsample you’ve found a problem with feels major or important really is just rhetoric. My expertise is more in regression analysis than sampling theory, but I suspect someone somewhere has done this. I gotta run so I’m not going to check out JASA at the moment, but I wonder if anyone else knows who it is.

(Obviously there is extra complexity because the subsample measurement is not independent of the overall measurement — but it is not perfectly correlated, and thus each subsample is to a certain degree an independent measure, but with a higher variance).

Um, maybe you can explain what you had in mind here, but it sure looks to me like 8,000 is a lot less likely than 98,000 on both versions…?.

On the second, the peak of the distribution is right next to the lower CI bound, not up in the middle as would be the case for an unadjusted normal distribution.

#150, charlie, “M&Ms”

Brilliant, thank you.

Charlie/mq/philosopher/roger/etc., regardless of where we’ve actually got to, I want to thank you for going the extra mile explaining some of the principles to me. I won’t pretend I haven’t learned anything: which is a convoluted way of saying I’ve benefited from all your contributions.

I still don’t understand why figures at the violent/non-violent level need to be treated with such caution given the binary nature of cause of death, but I do at least understand the principle of inherently less confidence when moving down the levels of classification.

Philosopher, you say:

It’s correct, in that it will be easier to get statistical significane for either of violent or non-violent than it would be to get for, say, a much more finely broken-down set of categories. But what it doesn’t mean is that you get significance for either category for free, just because you’ve got significance for all deaths.

Okay, so what determines whether the estimates for violent/non-violent deaths achieve statistical significance? More specifically, why are we precluded from drawing a conclusion about a jump in violent death attribution from 56% to 100% from Lancet 1 to Lancet 2? What is the statistical explanation which means we don’t need to worry about what appears to a layman to be a numerical anomaly in violent death attribution from study to study? This is where I’m currently stuck.

Finally, is it fair to think of Falluja as a rogue malteser?

Brownie, statistical significance of the subgroups is calculated the same way as for the main groups, i.e. the count of events in each subgroup is extrapolated to the whole population from the sample weights. Because the events are drawn from a poisson distribution, the count of events is the sole determinant of their variability in the sample (with additional effects from the sample weights when we extrapolate). Smaller numbers = wider confidence intervals.

David Kane, you`re doing the same thing again. Anyone who has tried to get access to survey sample data from major organisations doing these things knows that it is standard practice for them to refuse to let you have it until they have at least had time to deidentify it, and even then they may have ethical requirements which only allow them to share the data with collaborators. The latter situation is very common. You claim to audit data, you must know this and your continual implications that you should have received it straight away upon request are disingenuous. Also, while researchers may share data with one another all the time, this is not the case in public health research at all. It is dishonest to suggest so. Generally epidemiologists have to sign extremely rigorous ethics approvals which prevent them from disclosing their data to just anyone who pops up and asks for it. You may think this is “unusual” in your experience, in which case I assume your experience doesn`t include the analysis of survey samples of this type.

I presume you know how to do the cluster analysis they describe, but why should we assume that you can do a better job than they did? Why should we even assume you need to? If you don`t trust their statistical analysis why should we or anyone else trust yours? And if your concern is the data itself, you again are insinuating fraud (and how in any case could you tell if this data was entirely made up?)

Finally, have you, as someone else asked on this thread, attempted to audit the Sudan death rate estimates? Do you routinely audit US National Household Surveys? Did you audit the ICLS survey? Why not? Because you don`t like the number this one gives you? As I said, if you can`t fault the actual method, don`t pretend there is anything wrong with it just because the number seems a bit off to you. Especially when a lot of people have presented reasons for why the number might be right, in an environment (Iraq) where none of us has the ability to tell what is going on.

regarding the actual original topic of this post, I read in the Daily Yomiuri today (hard copy, don`t know if it is online) that the number of journalists embedded with US soldiers has dropped from 600 at the time of the war to under 2 dozen today. Is it any wonder that we don`t have a clear picture of what is hapepening in the air war?

This article also observed that all the major news agencies have significantly reduced their size in Iraq, and it is very difficult for new journalists to even get into Iraq (one doesn`t just buy a plane ticket…). This reduction occurs at a time when Iraq Body Count is receiving more casualty reports than ever before. If casualty reports are going up while journalist numbers are going down, this can only mean that the rate of killings in Iraq is escalating rapily, right? Unless we suppose that the journalists have greater freedom to move around Iraq now than 2 years ago – an unlikely supposition given they need armoured cars and an armed escort just to get to their hotel from the airport.

Yet more evidence that we don`t know what is happening in Iraq and our `gut` responses just don`t have any value.

Pfui! It is a long link and something funny keeps happening in the html. Let me try this

http://tinyurl.com/y3nuta

David Kane, you`re doing the same thing again. Anyone who has tried to get access to survey sample data from major organisations doing these things knows that it is standard practice for them to refuse to let you have it until they have at least had time to deidentify it, and even then they may have ethical requirements which only allow them to share the data with collaborators.

Yes. Of course. Roberts could have said to me (and others):

“Just give me a couple months so we can remove identifying information and you can see the data.”

or

“You need to sign this non-disclosure agreement (and have it signed by a Harvard attorney).”

or

“There is an IRB document that prevents us from sharing any data with anyone and here is a copy of that document.”

But he (and his co-authors) said none of those things. They hedged a couple times and then stopped replying to my e-mails. Others tried to get data from them and were similarly unsuccessful. Draw your own conclusions.

Also, while researchers may share data with one another all the time, this is not the case in public health research at all. It is dishonest to suggest so.

My claims are about academic in general and not public health in particular. If your description of the standards in public health research is correct (I have no reason to doubt it but I thought that places like the CDC were pretty good about this stuff), than that is another reason to take the results less seriously. I do not think you would get away with such amateurish behavior in, say, JASA.

Anon in #160, you really need to back up your 500KT with a cite. As I said, it’s certainly off by a factor of at least ten.

Aggregating the Kosovo and Afghanistan conflicts, the average CAS sortie dropped 1.3T of ordnance on its target. (26KT, 19,500 sorties). If you’re talking John’s 5 sorties average a day, that’s 6.5T dropped on Iraq per day during the insurgency period. Three years of conflict = 2.3KT. Add the 20KT dropped in the first six weeks of the war. I doubled it above to be generous and I still couldn’t reach 50KT.

As I also said above, the Afghan conflict was recognized at the time for having an abnormally high number of *civilian* casualties per KT dropped, at least compared to other recent wars: 140 to 150. Your estimate of 1,300 civilian fatalities per KT for WW2 and even higher numbers for the current war would not be supported by any serious scholar I’ve read. The generally accepted figures for that war are 3,000KT of ordnance dropped between USAAF and RAF, resulting in 600,000 civilian fatalities, or approximately 200 civilian deaths per KT (see Neillands, et al).

In my scratch analysis above, I was assuming 700-800 *aggregated* deaths per KT x 30-35 KT to get an upper bound of 25,000 *aggregated* fatalities from air strikes in Iraq to date. I believe the multiple-two error in your destructiveness estimate, multiplied by the order of magnitude error in kilotonnage, compounded by your math error in para 7 of your last post, is contributing to some unsustainable conclusions on your part.

#151, brucer: “I can’t attest to your 500KT figure—love to see the source”

Thanks for your detailed response.

I’ve already cited the source (twice, once directly and once indirectly). I realize it’s a source you think we should ignore. I’ll get back to that in a minute.

Here’s the source, for a third time: “Marine close air support hammers terrorists in Fallujah with precision JDAM” (11/10/04). That document contains the following text:

So far for OIF II, 3rd MAW has dropped more than 500,000 tons of ordnance … but this number is likely to be much higher by the end of operations

The same document can also be seen here and here.

I find it interesting that I cannot find this document on any official US site. According to google cache, it was once hosted at usmc.mil, and also at mcnews.info (both official sites).

It’s odd that this story can no longer be found at usmc.mil, since it’s not hard to find other stories there written by the same person, at around the same time (example, example, example).

Even though the USAF issues daily reports (example), as far as I can tell none of them ever mention tonnage. As far as I can tell, the press release I’ve cited is the only example of the US ever indicating this sort of revealing tonnage information (regarding Iraq). I suspect the release of this information might have been an accident. I think it’s also suspicious that the press release is now hosted only at non-official sites.

“please avoid citing the famous 3MAW press release during Fallujah, which was a pretty clear instance of mixing up tons and pounds”

Of course that’s exactly the press release I’m citing. But it would be great if you could help me with a couple of things. I wonder why you call it “famous,” since I have a really hard time finding anyone who noticed it besides Hersh. It would also be great if you could find an example of someone beside yourself making the claim that the passage “500,000 tons” was meant to say “500,000 pounds.” I can’t find an example of anyone making that claim.

Finally, please consider this:

In a two-week span, multinational force aircraft, comprised mostly of jets and helicopters from 3rd Marine Aircraft Wing, dropped or launched more than 500 precision-guided munitions against terrorist targets in the city.

This is a reference to Fallujah II (aka “Operation Al Fajr,” 11/8/04-11/22/04). “Precision-guided munitions” is probably a reference to JDAMs, which basically come in three sizes: 500, 1000 and 2000 pounds. For the moment let’s assume that an average “precision-guided munition” is 1000 pounds. That means that during Fallujah II, 3MAW dropped 250,000 pounds in a 2 week period.

If you are correct, then in the prior nine months 3MAW dropped only twice that, even though “the wing saw major combat action over Fallujah in April 2004, in An Najaf in August 2004.”

Here’s another way to look at it. If 3MAW dropped only 500,000 pounds in nine months, that works out to only 10 pounds per aircraft per day. (3MAW has 200 aircraft, supported by 13,000 people.) That sounds low to me (although I realize that not all 200 aircraft drop bombs, or are constantly deployed).

I realize that my analysis calls for 10 tons per aircraft per day. I realize that sounds high, but I think it might be plausible. An F/A-18 has a total payload of 8.5 tons.

“some historical comparisons would be the First Gulf War, where 85KT were dropped”

I’ve seen that figure, but I’ve also seen lower figures. A useful table is here, showing tonnage for WWII, Korea, Vietnam and GW1. If you calculate monthly rates, it’s interesting to notice that 3 of the 4 show remarkably similar rates, of 40-48 Ktons/month. Korea is an outlier, with only 12 Ktons/month. If the current war has the same relatively low rate as Korea, that would work out to 500,000 tons (12 Ktons per month times 42 months) for the entire war so far. Of course that’s the same figure in the press release I cited.

Your estimate (30KT for the whole war so far) works out to .7 Ktons/month, which is only about 6% of the rate achieved in Korea, which is itself by far the lowest rate of the wars listed in the table I cited. Therefore that seems low to me.

By the way, I realize that more accurate weapons means we have less need to drop a lot of them (as measured by weight). Maybe that’s what I’m missing.

Anyway, I realize there is good reason to suspect that 500,000 tons sounds too high, but I also think there’s good reason to suspect that 500,000 pounds sounds too low. So I’m interested in any further assistance with the puzzle.

PS: I now notice your #173. Thanks for this additional information, which I will think about.

“your math error in para 7 of your last post”

All I did in para 7 of #160 is claim that .75*500=375. I don’t see the math error.

#147, mike: “Much of this thread involves a discussion of how much ordnance has been dropped, and what effect it has had.”

No. Much of this thread involves a discussion using terms such as airstrikes, missions, attacks and sorties. These terms are quite vague and give us little or no handle on the fundamental question of “how much ordnance has been dropped.”

Aside from me, there’s exactly one person here (brucer) who has made an attempt to quantify “how much ordnance has been dropped,” either by units or by weight.

“I guess you haven’t been paying attention.”

Unless you can be more specific and point out something I missed, the attention deficit is all yours.

Mike H

You continually claim that the first Lancet study asserted 40,000 estimated non-violent excess deaths. I don’t see that assertion at all. Where in the first Lancet study is this stated?

Using the data in Table 2 of Lancet I and the discussion in the text I get:

Excluding Fallujah data:

45 non-violent deaths pre-invasion
68 non-violent deaths post-invasion

1 violent death pre
21 violent deaths post

Clearly a large increase in violent deaths, but only a minor increase in non-violent deaths.

So where do you get the 40,000 non-violent estimated deaths from?

Mike H, I understand your reasoning with regards to the media reports in Iraq, but my guess is that you aren`t there (i.e. in Iraq) to check them. Sure, you can see the ones that do get reported, and get a sense that there are x many coming in and there has been no change in the rate, and so on. But unless you can wander about in Iraq checking to see how many events get press coverage, you don`t know whether those 500 reports you see a day are because 500 journalists are fanning out over 500 events, or 10 journalists are standing around in baghdad watching 500 events, and missing the remaining 50,000 in the country. The article I mentioned hints at the possibility of the latter, with the withdrawal of journalists making it more likely that over time fewer areas have been getting coverage, but more events have been occurring in those areas. My point is that whatever the journalists are reporting, we aren`t there to check on what they aren`t reporting. So there could be an extra 60,000 attacks, sorties, bombardments, whatever you like to call them, and we would only have the military`s word for it. And if they said they used `precision` ordnance we cannot know if that ordnance was witnessed on the ground hitting a house, because we don`t know if a journalist could have been there to see it. The Daily Yomiuri article implies that the journalists aren`t there to see it, in increasing quantities of not-thereness. Which in itself hints at increasing amounts of violence. Perhaps we could track IBC`s daily or monthly death count with the number of western journalists in Iraq, and see how much one is increasing while the other is decreasing. That surely says something, doesn`t it?

“Unless you can be more specific and point out something I missed, the attention deficit is all yours.”

Ahhhhh, I see. So if someone wasn’t specifically talking in “KT” parlance, then the discussions involving numbers of civilians killed per sortie and numbers of sorties had more in common with a macrame chat room than what you wanted to talk about.

“Aside from me, there’s exactly one person here (brucer) who has made an attempt to quantify “how much ordnance has been dropped,” either by units or by weight.”

Well there you go. looks like you have a ” taker ” after all, don’t you?

Mike H

Garfield shows a chart purporting to breakdown excess deaths by category. He does not explain his calculation for apportioning the deaths between the categories:

chronic disease, 3600 deaths
infectious disease, 10,000 deaths
accidents, 24,000 deaths
violence. 57,600 deaths

He does not claim that the category breakdowns are statistically significant.
http://www.epic-usa.org/Default.aspx?tabid=440

The actual lancet I study language only makes a claim about the *overall* death rate increasing to 7.9 per thousand over the baseline of 5 per thousand. It makes no claim about any increase in non-violent or non-violent death rates. Why? Because the sample sizes were too small.

If you look at the other chart you will see apparently a large increase in death rate for violent death, and moderate increases for accidents and infectious diseases. In the aggregate these increased death rates permitted a statistical claim that the overall death rate had increased over the pre-invasion baseline in Lancet I. But the study could not claim that any of the individual death rates had increased.

Take a look at the Lancet I data:

accidents: 3 deaths pre-invasion, 13 deaths post
heart attack: 11 pre-, 18 post

Note the huge increase in accidental death in Lancet I. Lets say that 6 of those 13 post accident deaths were recategorized as violent deaths, (e.g. crash because of gunfire).

You could reduce your non-violent excess deaths to 0 in Lancet I which is what you claim is the case in Lancet II. Sampling is by clusters, and I believe each of those deaths is weighted by the cluster in which it is located. If those 6 recategorized deaths are located in high population clusters they should have a big weight.

So the death breakdowns by category in Lancet I are very sensitive to small changes in the underlying sample That is why they are not statistically significant. And that is why any difference between Lancet I and Lancet II with respect to violent vs. non-violent death categories probably is of minimal importance.

Mike H

I’ve shown you that the 40,000 non-violent claim could possibly be reduced to 0 excess deaths by recategorizing a very few of the deaths in the raw data.

Lancet I estimated approx 100,000 excess deaths with a very wide confidence interval. Whether the non-violent component of that 100,000 was 40,000 or 0 we can’t say from the data provided. Change a few deaths and you get 0 non-violent excess deaths.

Garfield’s claim was more than the data could support and was not a claim of statistical significance.

“I’ve shown you that the 40,000 non-violent claim could possibly be reduced to 0 excess deaths by recategorizing a very few of the deaths in the raw data.”

Oh, you’ve ” shown me ” allright. Let’s remind everyone how you pulled that off:

“Note the huge increase in accidental death in Lancet I. Lets say that 6 of those 13 post accident deaths were recategorized as violent deaths, (e.g. crash because of gunfire).”

Chewie, Are you for real?

I have another way of describing your “recategorizing ” of the deaths. It goes something like ” changing what people told the interviewers back in 2004.” Which is another way of saying ” changing the data itself, because Chewie doesn’t like it.”

This is an absolutely absurd argument, but if you get to play, then so do I. I think I’m gonna take, say 5 of the 9 coalition attributed violent deaths, and change them to heart attack deaths, just because I want to reinforce the statistical significance of that 40,000 non-violent excess death figure.

“Lancet I is claiming no significant increased non-violent death rate after the war. This is consistent with Lancet II.”

“Yawn.”

I’ve been over that with Philosopher, quite some time ago. Here was my final thought on the matter:

Let me see if I have this right, Philosopher. Lancet 1 ” makes no claims about non-violent deaths increasing,” but adds 40,000 excess non-violent deaths to its excess death point estimate, without which the study could not achieve its headline grabbing 100,000 figure. These 40,000 excess non-violent deaths are only statisically relevant because they can piggy-back onto the excess violent ones, even though the 40,000, by themselves, represent a 22% increase over the baseline mortality estimate for the 18 month period.

From a non-statistician’s perspective, that’s a pretty neat trick, even if it is ” statistically sound.”

“Another reason not rely on the Garfield slide.”

Whatever you say, Chewie. I mean, just who the hell is this Garfield fellow anyway? It isn’t like he’s one of the study authors, right? Oh wait….., hang on a bit……., I guess he is.

Carry on, Chewie.

chew2 –
Hasn’t Mike made a serious point here:

This is an absolutely absurd argument, but if you get to play, then so do I. I think I’m gonna take, say 5 of the 9 coalition attributed violent deaths, and change them to heart attack deaths, just because I want to reinforce the statistical significance of that 40,000 non-violent excess death figure.

If the confidence intervals for subgroups are so wide that you can’t make any serious statistical claims about them, then presumably that also goes for the subgroup of excess violent deaths in Lancet 1, and Lancet 2 for that matter. If you’re going around reclassifying, as you state above, then why not reclassify all the deaths recorded as being violent in Lancet 1 as in fact accidental, and thereby claim that the coalition killed noone? (Or, so as not to be silly, and factoring in the deaths that we *absolutely know* were violent, due to illness etc. – that 90,000 of them could have been accidental, and only 10,000 were killed by the coalition forces?)

I ask this question seriously, but I admit to not being a statistician. These ‘violent deaths’ subgroups are the largest subgroups, but the ‘excess violent deaths’ subgroup in Lancet 1 isn’t much bigger than the ‘excess non-violent deaths’ one, and my understanding is that you’re saying you can’t make any serious statistical claims about the latter.

JohnQ,

We’re approaching 200 comments on a topic that should have been closed by #2. If I’ve ever seen sample estimates that say “cannot reject the null hypothesis” more clearly, I can’t recall it. But we’re trying to explain this to people who don’t know what a null hypothesis is and refuse to find out.

Given there are about half-a-dozen contributors to this thread who claim to be statisticians, why did it take until comment #192 before someone mentioned the words “null hypothesis”?

In #176 James wrote:

I have a stats background, and I’ve made a living conducting market and social research surveys for more than 25 years. […] I’m also very worried about the fieldwork itself. I believe the reported refusal rate was 0.8% (I can’t find this in the report itself, so feel free to correct me). This is simply not believable. I have never conducted a survey with anything like a refusal rate that low, and before anyone talks about cultural differences, there are many non-cultural reasons for people to refuse to participate. If my survey was in a war-zone, I would expect refusal rates to be higher than normal.

I don’t have your 25 years of experience conducting surveys, but here are some surveys that have been conducted in various third world countries. If you pick a few of those surveys, you can get a sense of the response rates claimed; you can usually find them in Chapter 1 of the final reports. I think you’ll find that claimed response rates in the 98% and 99% range are not uncommon.

“Given the time Mike H and others are putting into this, a day or two with a good intro stats book would be a really good investment. It might not resolve all the questions, but it would make it possible to conduct a coherent debate, which can’t happen as long as one side is wilfully ignorant of the basics.”

That’s pretty rich John, coming from someone who believed the UNDP ILCS was a passive count rather than a massive cluster survey. You might want to give your friendly neighbourhood glass man a call.

While we’re on the subject of ” coherent debate,” it’s rather difficult to have one when people like you avoid addressing those points being made by the other side which make you uncomfortable.

For the third time, I’ll ask you for an explanation for the discrepancy between the ILCS’ 24,000 ” war related” deaths in the first 12 months of invasion, and Lancet 2’s 90,000 excess violent deaths for the first 14 months.

I’ll also ask you once again for an explanation as to how a country the size of Iraq (or any country for that matter) can undergo a minimum of 425,000 violent deaths in little more than 3 years time, and yet not have the non-violent mortality rate rise in a “statistically significant manner.”

Anon:

“One person on this thread raised that question (of trying to assess amount of ordnance dropped, rather than strictly focusing on number of sorties). That’s me. One person on this thread answered that question. That’s brucer. You, on the other hand, have been no help whatsoever.”

I didn’t realize it was mandatory that one fully dive into every tangetial discussion that crops up in this thread. If that’s the case, you’d better pull up your socks in relation to the discussions I’ve been involved in. You’ve been no help whatsoever.

#184: “I don’t believe any withholding of death reporting, if it is occurring, gets us anywhere near the Lancet’s 15,000 a month toll.”

“As someone recently said around these parts: sounds like one of those “gut feel” conclusions we should be studiously avoiding.”

As I recently said around these parts, a year from now, you and the rest of the Lancet true believers can tell us why just under a million dead by violence isn’t a ” gut feel ” conclusion. It’s “statistically sound.”

Robert, response rates of 98% or 99% are uncommon. The claimed Lancet non-refusal rate is 99.2%. This is in a situation where:

the teams faced suspicion… lengthy explanations were required – explaining that it would help the Iraqi people – were neccessary to allay fears (p.15)

And depite these lengthy explanations, the field team managed 40 interviews per day, at 99.2% participation.

The whole description of the procedure in the report seems odd. Candidates for interview were the head of the household or their spouse. Are we to conclude that there were no households where the head/spouse were not at home?

BTW, I can find no indication that the 2004 Iraq Living Conditions Survey “reported a response rate of 98.5%.” do you have a link?

It’s unbelievable that with so many airstrikes in Iraq we aren’t getting the big picture and all the information of what is really going on there. We get bits of information from the media and we’re always told when an American is killed, but we hardly ever get news of Iraqis who die. And with so many airstrikes, shouldn’t people logically believe that people are dying? Planes don’t just randomly drop bombs on empty fields. I just think the media should just focus on what’s going on and stick with the facts instead of always trying to make America look like the “good guy” because a lot of people already think this war is completely unnecessary.

JQ,

What are the confidence intervals around the estimates of violent deaths in the overlap period of lancet 1 and lancet 2? As I’ve said before, I wouldn’t be surprised to see that they overlapped, but if you want to rely on them, you should post them.

Your post #2 doesn’t provide them.

James insisted:

Robert, response rates of 98% or 99% are uncommon. The claimed Lancet non-refusal rate is 99.2%.

James, did you look at some of the studies I pointed to? I arbitrarily clicked on 5 of those reports. Here are the reported response rates (not non-refusal) from four of them:

Egypt 2005: 98.9% (of 21,972 HH’s)
Jordan 2002: 99.0% (of 7907 HH’s)
Burkina Faso 2003: 99.4% (of 9149 HH’s)
Turkmenistan 2000: 98.6% (of 6391 HH’s)

The fifth, an old report from Thailand, reported a response rate of 94%. So I stand by my claim that response rates of 98 and 99 percent are not uncommon.

James continued:

This is in a situation where:

the teams faced suspicion… lengthy explanations were required – explaining that it would help the Iraqi people – were neccessary to allay fears (p.15)

And depite these lengthy explanations, the field team managed 40 interviews per day, at 99.2% participation.

The whole description of the procedure in the report seems odd. Candidates for interview were the head of the household or their spouse. Are we to conclude that there were no households where the head/spouse were not at home?

So what you’re saying is, you are arguing from incredulity and not evidence? BTW, on p. 4 of the Burnham article, they write: “In 16 dwellings, residents were absent; 15 households refused to participate.” So while the refusal rate was about 0.8%, the total non-response was about 1.7%.

I can find no indication that the 2004 Iraq Living Conditions Survey “reported a response rate of 98.5%.” do you have a link?

Page 13 of volume 1 of the ILCS report says 21668 out of 22000 HH’s were interviewed. They don’t mention whether a non-interviewed HH was because of refusal or absence.

So, let’s review one more time: you have 25 years of experience in the field, and yet your intuition about response rates that are “simply not believable” does not appear to be supported by the evidence. Shouldn’t that be a cautionary tale? Doesn’t that suggest that “unbelievability” in and of itself isn’t a sufficient argument?

Just to answer a troll way back upthread – yes, the figures for urban Iraq are representative, because as stated the top 6 cities are a majority of the population. Yes, places like Muthanna Province are quiet, but nobody lives there.

Anyway, the “500,000 tonnes” thing has been thoroughly dealt with here. To be right, it would have required an impossibly high sortie rate, one that could not have been sustained long enough to get rid of the bombs.

JQ re #212

So you assert that that the CI overlap, but you can’t be bothered calculating them?

Brownie says that the estimates of violent death (for the overlap) are:

Lancet 1 – 57,000
Lancet 2 – 112,000

What are the confidence intervals?

James wrote:

Your quote from the Iraq Living Conditions Survey simply says that they had a target sample and an achieved sample.

Okay. But the DHS surveys indicate a number for the target, a number for the households that were found, and a number of completed interviews. That sure doesn’t sound like they were hunting for households until they got their quota. And 98% and 99% response rates in the DHS don’t appear to be uncommon.

So it sounds to me like your intuition that a response rate of 1849 out of 1880 HH’s as “simply not believable” based on 25 years of work in the field doesn’t transfer well to other countries. Doesn’t that suggest that “unbelievability” in and of itself isn’t a sufficient argument?

“The position you’re now taking is essentially this (paraphrase): “I/we didn’t answer your question (regarding how much ordnance has been dropped) because we had better things to do, and we had no interest in diving into a question we considered tangential.””

And you’re accusing me of a ” shift,” and “false statements?”

Get rid of the ” we.” I was referring to myself, not the collective. I feel no requirement to take part in every tangential sidebar issue that arises in this thread. Very few people do in a busy thread, and I don’t see a single other commenter here that has, including you. So where did you get the ” we ” from?

“That’s not the same thing as “how much ordnance” had been dropped, unless one is in a position to assert an ordnance/sortie ratio. Which is essentially what brucer helpfully did (#112, #151, #173, #178), in response to my question.”

Yes Bruce was helpful, wasn’t he? Now, about that post # 112 of his…… Tell me anon, do you recall what post you made your exasperated ” any takers” challenge to everyone? I’ll help you out, it was # 142.

I think you can figure out which number comes first.

“I pointed out that you were relying on a “gut feel” conclusion. Your defense was this (paraphrase): “it’s OK for me to rely on a ‘gut feel’ conclusion because I think you’re going to do that in the future.” Very impressive.”

I guess you missed the nuance, anon. I don’t believe any objective person would have a “gut feeling ” that nearly a million Iraqis had died violently by this time next year, assuming Sunnis and Shiites don’t start slaughtering, in Algeria-like fashion, entire villages and towns in the near future.

John Quiggan:

“So you assert that that the CI overlap, but you can’t be bothered calculating them?” – James

Talk about flypaper for innumerates! Hint:the point estimate is the central point of the CI.

OK, you’ve demonstrated that you don’t know what you’re talking about, I won’t ask again.

Kevin:

Thanks for the friendly warning. I’d rather be discouraged, I think.

Mike H, as a follow-up to that longwinded response, I thought I might add something regarding reporting air strikes. I know of a theory of war which posits three dimensions – moral, mental, physical – to war, and some argue that against insurgents winnning the moral campaign is more important than winning the physical. These people also argue that the insurgents know this and the US army doesn`t. But if the US army has even an inkling of this philosophy behind their “plan” (or lack thereof) in Iraq, it is in their interests to discredit reports of air-strikes. So when a precision bomb precisely kills 15 kids in woop-woop-stan, if that report gets to the media unconfirmed by US troops on the ground, or by “independent” journalists at the school, the US army will immediately declare it to be a lie. I think we have seen evidence of this where they blew up a wedding on the Syrian border and claimed it was an insurgent hideout, until al Jazeera produced the wedding video, or photos, or some such. This is common behaviour for a force which knows the moral war is important, but hasn`t figured out that it is the killing part, not the reporting part, which is the important bit.

So I would argue that we need journalists on the ground in order to accurately report casualties – it is insufficient for them not to be there and to rely on secondhand reports. Whatever one thinks of al Jazeera`s methods, their reporting from remote areas has exposed a lot of spin and falsehood in US reports – remember the govt denied soldiers had been kidnapped until al Jazeera reported it? they denied post-invasion casualties until al Jazeera showed some; they denied a helicopter had been shot down until al Jazeera showed it; and they lied about the wedding. Therefore I argue that journalists on the ground are important, and there is evidence that they are not there as much as they used to be.

Mike H

don`t waste your time reading a stats book. I don`t think anyone needs to know much about stats to legitimately question these or any figures, and if you end up looking obtuse because you don`t know the difference between a Borel set and a sigma-algebra, well, people don`t have to answer you.

SG:

Thanks very much for the replies. Yes, it’s one helluva long thread, but an excellent discussion all the same.

You make some solid arguments, although I don’t agree with everything.

I’m pressed for time this afternoon, I’ll have a response for you later today.

All the best…

Philosopher:

I understand your argument, and I don’t disagree that the 40,000 non-violent deaths is less statistically significant and less statistically robust than the overall point estimate for excess deaths, and the excess violent death point estimate.

However, there still remains an increase in reported deaths for 4 of the 6 non-violent subsets (chronic disorders and ” other,” stayed the same). And as you see from my post #227 to John, one of the most prominent and statistically knowledgable defenders of Lancet 1 not only accepts that the study recorded an increase in non-violent deaths, he makes arguments to bolster the plausibility of these increases.

The way I see it Philosopher, the non-violent excess death extrapolation is a bit of a conundrum for defenders of the first study. In a pure statistical sense, it may not constitute an effect. Yet at the same time, the increase has to be defended, because no non-statistician would accept the survey’s 100,000 excess death point estimate if they were told that nearly half of it was considered statistically unsignificant. That’s completely counter-intuitive to any non-statistician.

You may not agree with my wording of my paragraph that you cite to start your post, but I think it quite accurately reflects this dilemma for the study authors and their supporters.

Of course, there was a solution to this dilemma. The authors could simply have cited the increase in violent death. Had they done so, you and I wouldn’t be having this exchange. I think the reason why they chose not to do so is obvious.

“The point is that it’s just plain wrong to say that “nearly half of it was considered statistically insignificant”. The 100,000 number is significant on its own terms.”

Yes, the 100,000 number is significant on its own terms, Philosopher, but the composition of that figure is also significant, and the authors realize that, hence their efforts to apportion it into subsets.

Put yourself in my place, as a non-statistician. I dare say there are many more of me than there are of you. If you’re Les Roberts and associates, your goal is to change public perception and public opinion in relation to Iraq. Naturally, that means you have to appeal to the reasoning processes of non-statisticians, since we’re the overwhelming majority.

Well, you can’t convince us your study accurately reflects mortality in Iraq after regime change, if you’re not willing to break things down a bit beyond a gross, best estimate aggregate number of excess dead. This is, after all, a war-related study. If there wasn’t a war involved, Roberts and Co. probably wouldn’t have done a study. War generally means you have a significant number of dead from bullets and bombs. If a mortality study can’t reliably separate the ” bullets and bombs ” victims from those who died from peacetime causes, then non-statisticians like me tend to take a jaundiced view of the entire study.

“Note that if your subtractive reasoning made sense, then we could presumably break that 100,000 number into a very large number of statistically-unmeasurable sub-sub-subpopulations (“persons aged 47 who died of a heart attack between 3pm and 4pm on a Thursday”);……”

I think you’re getting a bit hyperbolic, Philosopher. In any event, myself and others criticized the subsets of death for their volatility and unreliability for extrapolation back in 2004. That didn’t stop defenders of the study from arguing that the data at the subset level made sense, probably was reasonably accurate, and was suitable for extrapolation, which was then debated ad nauseum.

“….there would be nothing statistically significant to be said about any of those sub-sub-subpopulations, but every excess death would have been put into one of them, and so it would follow (by your reasoning) that there is nothing significant to be found about the whole!”

I don’t follow that logic, Philosopher. If you can break numbers down into minute subsets, you can reconstitute them upward again. Nothing changes. You aren’t changing the data used to break things down further, and vice versa.

“As for Tim’s earlier defense of the 40,000 figure, I think you’re misconstruing what was going on there. His arguments were only meant to show that such a number was basically plausible—not that it was an iron-clad result of the study. (At least, that’s the case with the text you quoted).”

I’m not misconstruing, and your parenthesis clad qualification kicks in here. I have many other examples of Tim arguing that the non-violent subsets reflect an increase in non-violent mortality, post-invasion.

“And it seems like it would indeed have been an entirely plausible result—it was not a crazy possibility, after all. It just turns out not to have been the case.”

Then Tim has it wrong, although I don’t believe he has, at least not when considering the implication of the first Lancet study’s data. It’s much easier for the second study to assert an unequivocal statistical no effect for non-violent excess mortality. It extrapolated a similar number of non-violent excess deaths, for a period more than twice as long as that covered by Lancet 1.

“The use of the present tense there is perhaps misleading, since I don’t think that Tim now defends those numbers as the best estimates, in light of the greater evidence of Lancet II. (Certainly quoting text from 2005 does not show that he does so.)”

That’s cheating, Philosopher. We don’t get to hop in a time capsule to revise earlier opinions and arguments, especially when Tim didn’t know if there would be a ” Lancet 2″ when he voiced these opinions and arguments. I doubt Tim was a less qualified statistician then than he is now. If your argument is as statistically sound and statistically obvious as you assert, I find it incomprehensible that Tim would behave in such an apostate fashion.

“you can’t convince us your study accurately reflects mortality in Iraq after regime change, if you’re not willing to break things down” Well, then I’m afraid we’re back to the point that Quiggin and others have gotten to repeatedly on this thread: y’all just need to learn some statistics. There’s just _nothing_ wrong with defending a claim about a population without any breakdown into subpopulations. Nothing at all. If you can’t see this at this point, with all the explanations and analogies that various people have offered here, then I don’t really know what more to say. In any field, there are always going to be claims that look intuitive to a non-expert that are simply wrong — if there weren’t, then there’d be no need of experts! — and this idea that a claim about a population _must_ be re-interpretable into a claim about subpopulations is wrong in just this way.

“Then Tim has it wrong, although I don’t believe he has, at least not when considering the implication of the first Lancet study’s data.” No, he doesn’t — because he was only defending the plausibility of the claim, not its truth. He’s just saying that the number were _sensible_ at the time, not that they’re the way things had to be according to the study. Again, it’s just meant as a defense against a line of “No way!” type objections, to indicate that the numbers are not implausible on their face. One can make such an argument without being committed to the for-all-time truth of the numbers themselves.

And my objection about your citing a 2005 quote in the present tense isn’t not cheating at all — it’s directly relevant! The point is that he wasn’t defending those claims about the 40,000 as _true_, he was defending them as _plausible_; and it’s relevant to that point that he isn’t at all trying to bother with defending them now, when there is no dialectical need to.

SG:

Thanks for your patience. SG unintentionally jumped the queue.

“I think it is quite common for statisticians to get carried away with the validity of subgroups in a study, and to act as if the point estimates of the subgroups are just as accurate as the main conclusion, even though this is not really the case. It`s a common example of scientists taking an overly positive view of their own work.”

Thanks for that, SG. Naturally, I agree with you. I think you can understand then why I find it perplexing to be chided (albeit in a gentlemanly fashion) by Philosopher when I take on the subgroups myself.

“To have a statistically significant point estimate of 100,000 excess deaths you must have some positive numbers in the subgruops of violent and non-violent deaths – you just can`t say with any precision what they are.”

I agree again, on both counts. But there’s a difference between precision and rough numbers. If the study isn’t precise enough for even rough numbers for the two top subgroups, then the study should refrain from providing any figures for same. The study authors evidently decided Lancet 1 was capable of extapolating rough estimates for the two main subgroups. On the other count, I’m not questioning the fact that Lancet 1 was absolutely correct in measuring an increase in the relative risk.

“But the biggest fallacy critics of the methodology of the study (as opposed to its value in decision making) have made in this regard is to say it is a dodgy study because it is imprecise.”

I don’t think it’s ” dodgy ” either, SG, and I know many critics have said it is, or worse. I don’t believe for a minute there was any fraud involved in the data gathering process, and in the extremely remote chance that some of the field interviewers fabricated anything (again, I don’t see it), Roberts and the rest of his study leads would be unaware and blameless.

” And regardless of how convinced you are of the accuracy of the subgroups, does it matter whether they died from cholera or a bullet? Either way they`re dead because of our war, we just don`t know exactly how many we killed directly.”

No, it wouldn’t matter how they died, SG. “How many,” matters to me even more, but I believe a study needs to convince me they’ve gotten the “how” reasonably close, at the top of the excess death chain, above bullets and cholera, before I can be convinced of the ” how many.” This is a methodology that could easily contain 10,000 extrapolated excess deaths, derived solely from a single lightning strike into a farm field, killing 4 or 5 workers. As I’ve said many times before, going back to debates over Lancet 1, the methodology is statistically sound, my disagreement lies with the methodolgy’s ability to give us an accurate picture of both the ” how ” and the ” how many ” in a place like Iraq.

It isn’t enough for me to be told ” they’re right that death has increased, why won’t you accept the ” how many ” they provide?

I can agree with the former, without agreeing they’ve got the latter right.

SG:

Back to you earlier posts. Sorry for the delay in replying.

“I don`t pretend to claim that this post refers to any comments of yours or not (and I`m certainly not going to drag my heels through 226 posts looking for examples of who used “gut” reasoning!)”

No? Well why not? Suck it up, man! :>)

“I certainly agree with your suspicion that the US and Iraqi governments might be selectively releasing figures. After all, the US govt doesn`t do body counts. Also suppose that a casualty count of, say, 30 dead and 45 injured on one day gets in the news – if 5 days later 35 injured die of their burns, this doesn`t hit the news at all. The media only count the deaths that occur at the moment.”

Good point on the count of those who die later from their injuries, although one of the Iraqi goverment-released figures of deaths following each month are a combined morgue-hospital report count, if I’m not mistaken. I think some of the deaths which occur days after would be captured by the hospital count, although I’ve no way of knowing for certain.

I thing someone in the Pentagon let it slip very early in the conflict that the U.S. military was keeping a count of Iraqi deaths. Their numbers have never been made public, if in fact the Pentagon is counting.

“As for the possibility that journalists are not in remote areas but hear about all the deaths there because of Iraqi activism, this seems to me to be unlikely (please forgive me if I misunderstand your post). I admit that this is an argument from incredulity, but we all know that modern media operate by waiting for press releases, and I can`t imagine that people in war-torn regions of rural Iraq are very much able to get those press-releases to the media; and even if they could, they would be dismissed by western journalists as `not credible`.”

I think the method of reporting is a function of the perceived danger posed to reporters, particularly western ones, although I’m not sure that it matters why the media there relies on briefings. I don’t think much of the western media likes George Bush or agrees with regime change. I’d be very surprised if the media would withhold reports of large scale violence in rural backwaters of Iraq, if they were occurring. I think many major media entities have an agenda, and that agenda is to undermine support for a continuing American presence in Iraq. Minimizing violence is counterproductive to that agenda.

I think it’s also important to consider scale when assessing how much violence is being missed outside Baghdad. Supporters of Lancet 2’s extremely high violent death estimate point to the 2,500 – 3,000 a month death toll that has been the norm in Baghdad since early 2006 as corroboration. Most seem to accept that the 2,500 – 3000 is accurate, and helps press their case in relation to the overall huge Lancet 2 excess death estimate.

The final period covered by Lancet 2’s survey (June 05 – June 06) claims an average monthly toll from violence of about 22,000! Critics and supporters of the war alike seem to be in agreement that the worst of the violence is far and away in Baghdad, Anbar, and the Sunni triangle. You have to assume many thousands of dead every month in areas believed to be relatively quiet, to get anywhere near this 22,000 per month average.

“These people also argue that the insurgents know this and the US army doesn`t. But if the US army has even an inkling of this philosophy behind their “plan” (or lack thereof) in Iraq, it is in their interests to discredit reports of air-strikes. So when a precision bomb precisely kills 15 kids in woop-woop-stan, if that report gets to the media unconfirmed by US troops on the ground, or by “independent” journalists at the school, the US army will immediately declare it to be a lie.”

As I mentioned in my earlier post, some of these incidents are getting reported, and not only by the decedents’ relatives. I’ve seen Iraqi officials bring claims of non-combatant deaths (from air strikes and otherwise) forward to the media as well. I agree we don’t know what percentage are getting into the media, but that goes back to the entire debate over all casualties and the accuracy of government and U.S. military released figures.

“I think we have seen evidence of this where they blew up a wedding on the Syrian border and claimed it was an insurgent hideout, until al Jazeera produced the wedding video, or photos, or some such.”

I also recall a foreign insurgent being interviewed early on in the conflict. He talked about getting into the country via the Syrian border, and how U.S. air strikes had been successful in partially interdicting the flow of foreign jihadists into Iraq from Syria. He freely admitted that whenever such an attack resulted in dead fighters, the insurgency would propagandize the deaths as non-combatants to any media willing to listen. I may still have the article saved on my hard drive. I’ll have a look for it.

“I could say, after years of living in a country with violent police, the wife of your example considers that sort of behaviour to be normal, and considers the response to be normal, not war-related.”

Yes, but we have to consider probabilities. All scenarios aren’t equally plausible. We also don’t know how many homicide deaths the study authors might be willing to assign to the same time frame as the UNDP study. Some of the overall extrapolated homicide deaths will fall into the 6 months after the UNDP study stops, so even if the UNDP study captured no deaths from homicide, the fact that the Lancet did is somewhat diluted by this difference.

“I think that the Lancet is better for its purpose because a) it confirmed 80% of the deaths through certificates; b) it ruled out deaths of soldiers (as opposed to insurgents); c) the question it asked (did someone die) is directly related to the research question (how many people died).”

I’m not that hung up on the death certificates issue, SG. I don’t think many interviewees would fabricate a death. But not capturing the deaths of soldiers makes it less likely to claim corroboration with the UNDP war related death figure, if the UNDP did capture soldier fatalities.

“The ICLS question (how many people died in war) is maybe only broadly related to the study aim…”

You’re right about that.

“So again, I maintain that did someone in this house die a war-related death is only part of the question how has Iraqi death rate changed since the invasion. Therefore, we cannot expect comparable rates between the surveys.”

Without regurgitating my entire argument, I disagree to some extent, SG. Obviously, ” war-related ” death is only a part of the overall mortality picture, since we can safely assume ” war-related” will consist almost exclusively of violent deaths. So I agree with you there, that the death rate has other elements affecting it. Still, I think you can give the Lancet every possible concession (ie, compare it’s lowest possible violent death extrapolated figure with the highest possible UNDP war-related number, exclude all homicide and soldier deaths from the UNDP figure, etc), and there’s still a sizable gap between the bottom lines for both studies in terms of violent death.

Thanks for the advice concerning statistics studies. That’s two now, you and Kevin. I think you guys speak from experience, and I’d be wise to benefit from that.

Hope all of this makes sense, SG. If not, I blame some of it on the fact that it was a rush job. I’ve got to get myself off to work shortly.

Thanks again for the insightful comments.