Update: Unaccountably, I forgot to thank “Robotslave” for massive amounts of help provided in this research. Sorry and thanks!

This was meant to be a title for a post about the utterly fallacious but surprisingly common wishful-thinking belief on the part of wowsers, prigs and other good-government types that “seeing someone throwing a shoe (or custard pie or whatever) at a political figure makes most people think worse of the thrower and his political cause, and sympathetic toward the target”. This simply isn’t true[1]. The instinct to support the underdog isn’t there in nature and one has to arrive at a very high level of civilisation before it becomes a firmly established reaction. And at the sight of airborne humiliation flying toward a political foe, all those centures of evolution drip away, and one’s left just simply pointing and chortling. It’s perhaps not the noblest expression of the human mind, but frankly, any political movement which doesn’t regard this as funny, well, it’s the Emma Goldman moment as far as I’m concerned.

God, I’m never going to get tired of that. Anyway, there’s clearly a Laffer Curve here; on the one hand, if there were no pieings of politicians, the world would be a sad and sorry place, but I’m guessing that if people like Friedman were pied every time they went out, they’d never go out, and we’d still have to read their columns without seeing them occasionally pied. I suspect that we’re not actually on the rightward slope of the Laffer Curve[3] yet and that the world could stand having a few more of these incidents rather than less, but that’s a matter of personal taste.

On the other hand, I never really finished that post because of course, although seeing Bush get shoed was a laugh, it’s a nervous sort of laugh for anyone who is worried about the possibility of President Obama being assassinated, which I know that a lot of people are. And statistically, they’re right to be worried. Roughly 10% of all US Presidents ever elected have been assassinated (4 out of 43), which is roughly as high a death rate as street drug dealers[4]

Which brings me to the actual topic of the post; a link to a post on my other blog where, in comments, we have been working out job-related mortality rates for a variety of professions. To date, we have:

President of the USA: Five job related deaths since 1789 (assassinations of Lincoln, Garfield, McKinley, Kennedy, plus death of William Henry Harrison from job-related stress illness). That’s 0.022831 job related fatalities per year, in a job that can have only one person doing it at a time. Converting this to more normal units of occupational fatality for comparison with other jobs, that’s 2283 deaths per 100,000 President/years. This compares to 117 deaths per 100k worker/years for timber-cutters, the most dangerous standard occupational category.
Soldier, US Army, Iraq war: Up to March 2006, 392 fatalities per 100k soldier years spent in the Iraqi theatre. But of course, soldiers don’t spend all their time in battle zones, so based on reasonable estimates of fatalities outside combat zones, something around 80 would be more reasonable, more or less comparable to fishermen (anecdotally, the in-Iraq death rate would be comparable to Pacific Northwest crab fishermen, the single most dangerous occupation anyone could find).

Soldier, Iraqi Army, Iraq war: On the other hand, the Iraqi Army doesn’t get rotated out of Iraq, and thus by reasonably sourced estimates sees a job-related fatality rate of close to 800/100kwy.

Prime Minister, United Kingdom: It now gets rather controversial. Spencer Perceval was assassinated (definitely work-related) and Lord Palmerston died of a chill caught while on an official trip to Russia. Henry Pelham’s death in office was regarded at the time to have been related to the stress of his job, but how do we code the deaths of Pitt the Younger, and Lord Wilmington? How about George Canning, whose health was in rapid decline by his second appointment as PM? What about Thomas Cromwell, whose execution was clearly a job-related fatality, but who wasn’t head of government? On reflection, I would count three fatalities since 1688, for a job related fatality rate of 937.5/100k Pmy, but I can see how rates as high as twice that could be supported.

Pope: Even more controversial, here. By definition, all Popes die in office and it is very hard to get any data about Popes having died from job-related illnesses, so we had to scale back the project to estimate the rate of violent job related deaths of Popes. The consensus of historians has seven popes definitely having been murdered, which would give a job-related mortality rate of 476/100k Pope years since Gregory the Great in 540 AD. However …

There are a further ten Popes listed by Wikipedia as possibly having been murdered (this number includes John Paul I, so it apparently doesn’t take much in the way of real evidence to get on this list). Adding them in would boost the Papal fatality rate to 1158/100kPy. And furthermore …

The first 25 Popes were martyred. This is surely the very definition of a job related death, and I don’t really see much case for excluding them as an outlier[6] – in any long run of data, you’re bound to get a couple of periods under which Europe was dominated by a vehemently anti-Christian empire[7]. In order to take this additional data into account, we have to extend the window back to St Peter and 33AD[8], but you still get 2126 job related deaths/100kPy.

Monarch, England/United Kingdom. By the time we got onto these, I was getting a lot better at devising coding schemes, and so stipulated that for monarchs, I would count assassination (with a broad criterion for including rumoured murders), execution, death in battle, or death from illness or accident while travelling with an army in the field as job-related fatalities, and would require the record to be of a serving monarch in order to count (deaths post-abdication[9] not counting). On this basis, since 1066, there have been two deaths in battle (Harold Godwinson, Richard III), four murders (Edward II, Henry IV, Henry VI, Edward V), two executions (Jane Grey, Charles I) and one death during random pillaging on the way back from a Crusade (Richard I). That would make 955 job related deaths/100k king-years. But …

Starting the clock at 1066 seems very arbitrary. Although before Harold Godwinson there wasn’t really a king who controlled a territory recognisable as modern England, are we really going to say that Alfred the Great wasn’t King of England? A more logical place to start IMO would be Egbert of Wessex in 802, which means we have to add Edmund the Magnificent and Saint Edward the Martyr to the tally, but brings the overall death rate down to 912/100kky.

Monarch, France: By this time, very easy – having set up the coding for English monarchs, I knew exactly what to do with difficult cases like Napoleon I (out; would have been in on the basis of the inclusive criterion for rumoured murder, but abdicated and thus was not a monarch at the time of death). The only tricky bit was in adding up the denominator; since the first Carolingian monarch in 843, there have been 1010 years and 100 days of French monarchy, during which there were 9 work-related fatalities. That’s only 890 work related deaths/kky, which is doubly interesting as it is lower than the English figure, even though the French data doesn’t have the benefit of the massive improvements in monarchical health and safety which have been seen in the last couple of hundred years.

Active Party Member, Hezbollah. Data nearly impossible to get, but including suicide bombers[10] and dependent on an estimate of the size of Hezbollah, it’s as much as 1,000 to 2,000 per 100kHy. In any case, it’s very difficult to get any estimate under which being a Hezbollah member is as dangerous as being President.

So, recapitulating and tabulating, worst to safest:

President, USA - 2283
Pope – 2126
Hezbollah, active member – 1000-2000
Prime Minister, UK - 937
Monarch, England/UK – 912
Monarch, France – 890
Soldier, Iraqi Army – 800
Soldier, US Army (while in Iraq, March 2003-March 2006) – 392
Fisherman, Pacific Northwest crab fishing – c400
Timber-cutter – 117
Fisherman, overall – 85
Soldier, US Army – c80

Not necessarily what you’d have expected to see, is it? So anyway, where are we going with this?

First, a plug for The Politics of Large Numbers by Alain Desrosière, translated by Camille Naish. Although the translation is rather unfortunately accurate in rendering dry, pedantic officialese French into dry and pedantic English, it’s a great book (thanks to Laleh and Colin Danby in comments to this thread for encouraging me to persist with it). As you can see above, in order to get to my table of dangerous jobs, I had to make a lot of very arguable coding decisions which had a material effect on the relative positions (as far as I can see, no matter how I cut the numbers, I couldn’t have got the political and spiritual leaders down to the level of Pacific Northwest crab fishermen, but by making different judgement calls, I could certainly have made “Pope” look a lot safer). This is the nuts and bolts of the creation of official statistics and Desrosière’s book is very good on the history of their development and the various stages at which key decisions were made, almost always for political reasons (Germany, France and the UK had very different tracks for the development of their official statistics which reflected the realities of their administration in the 19th century).

Even something as seemingly straightforward as the space on hospital forms for “cause of death” is a minefield. Do you classify medical deaths by listing the organ which failed, or the cause of its failure (infection, trauma, etc)? The question always comes back to – what are you trying to achieve with these statistics? “The Politics of Large Numbers” is an excellent book for anyone who is ever tempted to think Bruno Latour’s work never had any really useful applications.

Second, though, there’s a philosophical issue here. More thoughtful readers will have looked through these mortality table calculations with a growing feeling of unease. The first instinct is to say that the sample sizes are too small for a lot of them, and that therefore the confidence intervals are too wide to make any meaningful comparisons. In my view, though, while that’s certainly evidence of good habits of thought, the actual numbers are large enough to be pretty sure that you can make at least some statements with decent confidence. In particular, it really does look to me as if the risk of assassination faced by a President of the USA is a lot higher than the risk of battlefield death faced by an American soldier. It’s true that the sample size is small nd the confidence interval is wide, but qualitatively, the facts are that modern military casualties are low compared to history[11].

I think the real source of philosophical unease at comparing death rates between heads of state and timber cutters as if it were an apples-to-apples basis[12] is not the small sample in and of itself, but the difference between a historical and a statistical mode of thinking. We’re tempted to think of the deaths of Popes, Presidents and kings as unique historical events, each with an individual set of causes. We know about them through their individual details, and in many cases we speculate and investigate to get a fuller picture of how they came about, whether we’re looking at the minutiae of the Texas Book Depository, or the grand sweep of the Hundred Years’ War.

The occupational deaths of fishermen and lumberjacks, on the other hand, for the most part appear to us mediated through lists of statistics[13], as risks which are part of the job. And in these occupations particularly, because the main sources of risk are such things as the pattern of storms and the direction of a falling tree, it’s even easier to fall into thinking of them in the whole, as stochastic processes, with the individual outcomes as the natural results of a stable distribution.

But on the other hand, of course, ordinary industrial fatalities (including those of timber cutters and fisher men) are unique historical events too, with their own individual causes. And these causes are both proximate and world-historical too; the families of many fishermen who died at sea are entirely aware of the way in which the European Single Market contributed to their bereavement. Similarly, there are statistical regularities even in the assassinations of kings and Popes.

The point is (and it’s a point that’s entirely relevant to a number of our present concerns)[14] that both these points of view need to be held at once[16]. As we head into a New Year which is likely to present uncommon challenges to all of us, bear in mind that nearly everything and everyone in the world is both unique unto themselves, and an example of any number of more general classes. Happy Christmas.

[1] Consider, for example, Bernard-Henri Levy. How many of us know anything about his philosophy in any real depth? How many of us, on the other hand, just know him as the repeated victim of a series of pieings from Noel Godin, and presume “hmmm, BH Levy must be a real pompous twat in order to provoke so many pieings”? In my case, I have read a bit of B-H L, and what I read pretty much confirmed my original opinion based on the pieings. Score one point for Malcom “Blink” Gladwell[2].

[2] Who surely must be well past his pie-by date?

[3] Obviously the equivalent to the tax rate here would be the strength and severity of the security response to pieings, in which context I suppose a few words on the vicious beating of Muntazer al-Zaidi are in order. Like any instance of police brutality, this is of course unacceptable. There is certainly a case to be made that given the high assassination risk faced by US Presidents, there is a public interest in dissuading pieings or shoeings of them, because having loads of “false positives” interferes with their security. But really; was there an epidemic of President-pieings beforehand? I would have thought that the normal legal risks and more or less certainty of getting caught would be deterrent enough in the general case.

[4] The death rate of the Black Gangster Disciple Nation gang studied by Sudhir Venkatesh was much higher – it went up to 25% for a four year period. However, this is generally recognised to have been anomalously high and probably reflects the peak of the crack epidemic rather than a “normal”[5] rate.

[5] What constitutes a “normal” death rate? What’s a “normal” environment for crack gangs, a population who are pretty much by definition abnormal? These and other highly politicised questions of statistics addressed further on down the post.

[6] I suspect that David Kane may take the fact that I have discussed the status of outliers here as being some sort of indication that he is not banned, or that I am interested in once more entering into discussion with him about the Lancet studies of Iraq and the correct treatment of Fallujah. Neither is the case.

[7] You might think that the experience of the Early Christians isn’t terribly relevant to someone considering a career as a Pope today but come on; this really is Mediocristan type of thinking! Lots of people didn’t think that there could be a sustained fall in nominal housing prices either!

[8] Using BCE notation seems something of an affectation in context.

[9] This stipulation also avoids having to consider what to do with Oliver Cromwell in the data for “British Head of State”; Cromwell was executed by beheading, but this wasn’t his cause of death.

[10] The death rate for suicide bombers per mission is obviously high, but less than 100%, as it is surprisingly common for Israeli police to successfully intercept them and take them alive.

[11] While modern medical advances will have helped Presidents to survive assassination attempts better than they have in the past (Garfield and McKinley both died at least partly because of the inability of their doctors to find the bullet), I suspect that they have helped troops in the field even more. Presidential security protection has also improved, and it’s true that in the 107 years since the Secret Service was given the job, the death rate has been only 934/kPy, but this is still twice as high as the estimate for in-theatre troops.

[12] Which is as good an opportunity as any to make my periodic plea about “comparing apples with oranges”. Apples and oranges are really very similar things indeed. If I’ve got three apples and you’ve got two oranges, who out of us has got the most fruit? The answer is “it’s me”, isn’t it, not “who can say, you can’t compare apples with oranges”. They’re roughly the same size, roughly the same calorific value and roughly the same price. You can compare apples to oranges, in as much as you can make comparisons at all. Also, the darkest hour is not just before dawn.

[13] Stalin underestimated statisticians on this one; as few as 454 deaths (the number of fishermen who died between 1976 and 1995 according to the files of the Registrar General for Shipping and Seamen) can be a statistic.

[14] I keep hoping to write a proper review of Taleb’s Fooled by Randomness and The Black Swan but realistically my record is not good on delivering on these promises, so I’ll just state the kernel of my disagreement with him here; Taleb rightly castigates people who simply rely on the statistical mode of thinking, but a) his criticism is more often of their use of particular kinds of statistical model rather than of the monoculture of statistical reasoning per se[15], and b) he regularly goes over the edge into claiming that one can or should throw away the whole statistical methodology, which is just as daft as relying on it entirely.

[15] And because I love to repeat a joke, I’ll repeat one of my favourite remarks on quantitative finance – the occasional necessity to remind quant modellers that the Great Depression did actually happen; it wasn’t just a particularly inaccurate observation of the underlying mean return on equities.

[16] Zizek, in The Parallax View makes a similar point and argues further than the nearest we have to an “objective” truth lies in the very act of switching between them, but everyone round here has conniptions when I mention Zizek so I’ll pursue that line no further for the sake of comity.

[17] And because I have more sense than to end on a Zizek footnote, I’ll plug my next big holiday read, “Beyond Human Error“, a book about safety science which seems to tackle the question of integrating the causal/historic and statistical modes of reasoning in a more systematic way.

Anyone headlining the 150,000 violent deaths figure as if it were the total.

No. That’s not what the study said. Anyone who is trying to claim “The Lancetissses said 650,000 but the true figure was only 150,000 OMGLOL!” or similar, is misrepresenting what the study says. This study found a big increase in the death rate in Iraq as a result of the invasion. They didn’t carry out an excess deaths calculation, but the change in the death rate is of the order of a doubling, which on the basis of their estimate of the pre-war death rate would indicate 400,000 excess deaths as an order of magnitude calculation.

There is a related mistake that people are also making at the moment, which is to try and talk down the Lancet total by claiming that “the apples to apples comparison” is between the Lancet’s estimate of excess deaths as a result of violence (of the order 600k) and the NEJM estimate of total deaths from violence (of the order 150k). This is a mistake, because it’s ignoring the fact that the NEJM study found a big increase in the non-violent death rate which isn’t in the Lancet study, and the question of the potential misclassification of deaths as violent or non-violent depending on who’s asking is a known issue – I seem to remember saying a few times in 2006 that the breakdowns by cause couldn’t bear the interpretative weight that people were putting on them. Lancet 1 got roughly the same balance between nonviolent and violent deaths as the NEJM/IFHS, but Lancet 2 didn’t and Lancet 2 is the only one that checked death certificates. It is hardly difficult to see why, given the atmosphere of fear of death squads in a lot of Iraq, people might want to pretend that a relative died of natural causes when he didn’t.

So, to make this comparison is to implicitly cherry-pick the lower increase in the violent death rate from the NEJM study, and the lower increase in the non-violent death rate from the Lancet study, giving a number that I would call “the Greengrocer’s 150k” (so called because it appears to be the result of trying to put apples to oranges, picking cherries and ending up with a lemon).

Anyone making bold assertions of the superiority of the NEJM study based on sample size

The NEJM does have a much bigger sample size than either Lancet study (9345 households in 1086 clusters versus 1849 households in 50 clusters). But the randomness of the selection was seriously compromised – 11% of the clusters were too dangerous to travel to, and they have their data filled in by extrapolation from the Iraq Body Count website.

This matters, of course, because if we’re thinking as Bayesians, the weighting we should put on the new study relative to the existing ones is inversely proportional to the uncertainty of the estimate. And the uncertainty of the current estimate is dependent on an assumption about how much inaccuracy this extrapolation introduced into the recipe. The authors write, fairly enough, that “Uncertainty in the missing cluster-adjustment factors was difficult to quantify, since we assumed that the excess risk of mortality in missing clusters in Baghdad and Anbar was normally distributed, with standard deviations of 0.2 and 0.1, respectively”, but if you plugged higher numbers into this guess, it would blow out the confidence intervals quite materially. Anbar is the real question mark here – it is a very big contributor to the death rate estimates in the Lancet studies, and it was one of the least well-covered areas in Iraq in terms of the media reports that IBC relies on.

Finally, this study and Lancet 2 cover the same time period (from the invasion to July 2006), but the fieldwork for the NEJM one was carried out later, and over a longer period of time. This means that more households disintegrated, more Iraqis left the country, and therefore the potential for undercount grew – and it grew quite sharply during this period as it was the absolute peak for violent death at the time. The survey notes that this lack of a stable population could quite seriously undermine the underreporting correction that they tried to make.

All these things are mentioned in the NEJM paper, by the way – I’m not Milloying about it here. Kieran is correct to say that this lower estimate shifts our central estimate of the excess deaths caused by the invasion. But there are plenty of people who seem to be suggesting that the correct response is to chuck Burnham et al away and latch onto this as the new definitive number, which is crap Bayesianism in the first place and doubly incorrect given the points made above.

Now onto the “apologies specifically not forthcoming” section …

Anyone who had a go at the first Lancet study and who is now pretending to be retrospectively vindicated by this one, despite the fact that the violent death rates measured in both are the same

Well I mean really. By the way, the link Tyler gives in his “mea culpa” post is to my very first post on the Lancet study, so he appears to be (perhaps accidentally) putting himself in this camp – either he hasn’t checked whether the NEJM study compared its estimate to Burnham et al (2006) or Roberts et al (2004), or he hasn’t read my blog post properly. I’d add that to have been sceptical of Lancet 1 (when it was the high number) but not to have a word of criticism for this study (now that it isn’t the high number) goes really badly for the old credibility.

Fred Kaplan and anyone else who reproduced the “dartboard” argument or “Kaplan’s Fallacy”

These two were idiotic at the time and since they were idiotic based on the maths, they were idiotic a priori and no empirical evidence at all could make them less idiotic. Even if Iraq had never been invaded, a Platonist would say that these arguments would still have existed in abstract space as hypothetical elements of the Form Of The Stupid.

Anyone who opined wisely on the existence of comprehensive central repositories of death certificates in Iraq

By the way, I don’t recall any fawning apologies coming my way on this subject; most of the people who made the asinine argument that L2 must have been flawed because of mythical Iraqi government statistics based on this mythical and wholly nonexistent centrally collated repository didn’t even post a correction. I suppose the idea is that any slightly ambiguous data point or small flaw immediately discredits a commentator on the Left, while no amount of disconfirming evidence ever merits so much as an apologetic harrumph from a commentator on the Right (cf, Moore, Michael; Kristol, William).

Anyone who made evidence-free accusations of fraud against Les Roberts, Gilbert Burnham, Riyadh Lafta et al

When we’re talking about “squelching heretics” or “castigating” people here, let’s just remember who was saying what about who, hey? Let’s get this straight. Falsifying interviews for a survey is serious scientific malpractice. Anyone guilty of it ought to be sacked from their job and publicly disgraced everywhere people respect science. If at any future date, Roberts, Burnham, Lafta or anyone else involved in either of the Lancet reports is shown to have falsified data, I will be right in the vanguard of those calling for the very severest of sanctions to be applied and I suspect Tim Lambert will be too (if you want to know how pissy I get when I feel like I’ve been played for a sucker, ask David Kane). Let’s be totally straight about this.

But if it’s a serious crime (and it is), then accusing people of it should only be done on the basis of evidence. If it’s not OK for me to say “Fred Bloggs is a bit of a hack because he’s reproducing this crap argument about the Lancet study which he plainly doesn’t understand”, then surely we can all agree that it’s much more not OK for Fred Bloggs to say “Well that devastating critique has just about wrapped it up for me, so I suppose that the Lancet team must have falsified their results in order to influence the US elections!”

Anyone trying to pretend that people who defended the Lancet studies against ill-informed criticism in some way “wanted” the death count to be higher and are “disappointed” by the IFHS survey

This is not so much a “no apology” as a heartfelt “Kiss My Arse”, with a side order of “Try Saying That To My Face, Sunshine”. This is and always was a pure, simple and disgusting insult. Anyone who ever did this, went straight on my shit-list and has been on a permanent 100% discount factor for their views on Iraq ever since. I’ve even lodged standing instructions with the Grice United Fund to make sure they don’t accidentally respect your opinions on my behalf. Which brings us on to the subject of …

In general, I just don’t agree with Tyler’s implicit view that there’s something illegitimate about making your case forcefully and not giving the kid gloves treatment to people who try to push weak, uninformed or fraudulent arguments against it (I’m glad to note that, revealed preference reveals, Alex Tabbarrok agrees with me on this one). Things which are definitely and provably wrong, do need to be squelched. We are not in relativist country here – there are some things which are true and some which aren’t, and “the government of Iraq has accurate mortality statistics, based on death certificates collected from around the country”, isn’t. Nobody benefits if people are allowed to pretend that this falsehood is true, and people who assert it without checking it don’t deserve very much in the way of condescending validation of their views.

Let’s not forget that, although we can all sit around and scratch our chins now saying “wwwelll, we all agree that thousands of people have died in Iraq, but …”, at the time when Lancet 1 came out, there were plenty of people (and not just right wing maniacs either) trying to claim that the Iraq Body Count figure was wildly inflated by antiwar political ideologues. This was the period of “all the good news that the MSM won’t report from Iraq”, remember?

A lot of the reason why the Lancet debates were so vicious was that so many people decided to join in them who clearly had no business getting involved (for lack of relevant knowledge), but who did so on the basis of the repetition of a small number of “seed” fallacies, like Kaplan’s Fallacy, the Cluster Sampling critique, the Mere Random Sample, the Mythical Death Certificate Repository, Kane’s Zombies and so on. Stamping on these, early and hard, before they take root, seems to me to be exactly the sort of thing that bloggers ought to be in the business of doing, if they care about the debate at all.

Background Estimates of the death toll in Iraq from the time of the U.S.-led invasion in March 2003 until June 2006 have ranged from 47,668 (from the Iraq Body Count) to 601,027 (from a national survey). Results from the Iraq Family Health Survey (IFHS), which was conducted in 2006 and 2007, provide new evidence on mortality in Iraq.

Methods The IFHS is a nationally representative survey of 9345 households that collected information on deaths in the household since June 2001. We used multiple methods for estimating the level of underreporting and compared reported rates of death with those from other sources.

Results Interviewers visited 89.4% of 1086 household clusters during the study period; the household response rate was 96.2%. From January 2002 through June 2006, there were 1325 reported deaths. After adjustment for missing clusters, the overall rate of death per 1000 person-years was 5.31 (95% confidence interval [CI], 4.89 to 5.77); the estimated rate of violence-related death was 1.09 (95% CI, 0.81 to 1.50). When underreporting was taken into account, the rate of violence-related death was estimated to be 1.67 (95% uncertainty range, 1.24 to 2.30). This rate translates into an estimated number of violent deaths of 151,000 (95% uncertainty range, 104,000 to 223,000) from March 2003 through June 2006.

Conclusions Violence is a leading cause of death for Iraqi adults and was the main cause of death in men between the ages of 15 and 59 years during the first 3 years after the 2003 invasion. Although the estimated range is substantially lower than a recent survey-based estimate, it nonetheless points to a massive death toll, only one of the many health and human consequences of an ongoing humanitarian crisis.

150,000 violent deaths in three years is a lot. You’ll recall that the Lancet study estimated about 655,000 excess deaths, which is a lot more. The two numbers aren’t directly comparable because excess deaths due to violence are only one component of all excess deaths (e.g., from preventable disease or other causes attributable to the war). Deaths due to violence rose from a very small 0.1 per 1000 person years in the pre-invasion period to about 1.1 per 1000py afterwards, or 1.67 adjusting for estimated underreporting. This is where the authors get their 151,000 number. The overall death rate rose from about 3.2 per 1000 person years to about 6, an increase of just over 2.8. Depending on whether you use the raw or adjusted estimated rate of violent death this would work out to an overall excess death total of just under 400,000 or just over 250,000. (But this is just a back-of-the-envelope calculation, as the overall death rate isn’t reported.)

The discussion section questions the Lancet result while emphasizing how difficult this kind of work is, given the appalling circumstances:

Recall of deaths in household surveys with very few exceptions suffer from underreporting of deaths. None of the methods to assess the level of underreporting provide a clear indication of the numbers of deaths missed in the IFHS. All methods presented here have shortcomings and can suggest only that as many as 50% of violent deaths may have gone unreported. Household migration affects not only the reporting of deaths but also the accuracy of sampling and computation of national rates of death.

The IFHS results for trends and distribution of deaths according to province are consistent with what has been reported from the scanning of press reports for civilian casualties through the Iraq Body Count project. The estimated number of deaths in the IFHS is about three times as high as that reported by the Iraq Body Count. Both sources indicate that the 2006 study by Burnham et al. considerably overestimated the number of violent deaths. For instance, to reach the 925 violent deaths per day reported by Burnham et al. for June 2005 through June 2006, as many as 87% of violent deaths would have been missed in the IFHS and more than 90% in the Iraq Body Count. This level of underreporting is highly improbable, given the internal and external consistency of the data and the much larger sample size and quality-control measures taken in the implementation of the IFHS.

At present, there are no better methods available to provide more accurate estimates of the death toll due to the humanitarian conflict in Iraq in the wake of the 2003 invasion. Rapid small-scale surveys of households are likely to yield unreliable estimates. Surveys of a large number of respondents with carefully prepared household interviews and multiple methods for collecting data on mortality still run into reporting problems because of the insecurity, instability, and migration associated with the conflict situation. The clustering of violent deaths may further affect uncertainty related to sampling, even though more than 1000 clusters were selected for the IFHS. It is unlikely that more accurate estimates of the death toll during the post-invasion period can be obtained by conducting more household surveys with recall questions on mortality. On the basis of press reports, the Iraq Body Count is also affected by considerable underreporting but is likely to be a valuable way to monitor trends over time. Further investment in such mechanisms is justified, especially if ways can be found to assess the level of underreporting and the consistency of the reporting mechanisms over time. Other methods, such as systematic reporting by mortuaries and hospitals and the strengthening of vital registrations with the use of sentinel sites, will also need to be explored.

Here at CT, Daniel was the one most involved in defending the Lancet study against its detractors, and he’s well able to speak for himself. But a word is probably in order to those on a mea culpa watch. A study like this gives us good reason to substantially revise our estimate of the total number of excess deaths downward. The Burhnam et al estimate of excess deaths looks like it was too high, assuming that the new survey is basically reliable. It’s good that the IBC effort and Burnham et al have been supplemented by new work. (Again, though, the 151,000 number is not an estimate of excess deaths.)

All of this is separate from the question of whether many or most of the reasons offered by earlier critics of the Lancet study were any good. Those who just said “this number just seems too high, I don’t believe it and want more data,” and left it at that, look a lot better than those who showed themselves ignorant of the methods used to calculate the estimates even as they tried to undermine them. The latter group should bear in mind that essentially the same cluster-sampling methods are used in the new study as the old, and the new survey was subject to many of the same constraints in accessing violent regions of the country. Those who simply floated accusations of fraud without any independent evidence at all look as bad as ever. Latching onto the new number just because it’s pleasingly lower doesn’t make any more sense than rejecting the old number because it was unpleasantly high for you.

More importantly, as the paper’s discussion makes clear, the main challenge facing those doing this sort of research is that there is a war going on, and wars kill a lot of people, bring about the dissolution of households, and compel very large numbers of people to flee the region. All of this makes the machinery of statistical science rather difficult to apply. None of the available numbers look any good, both on their own and given what they imply about what’s happening in Iraqi society. If you find yourself really delighted that a war of choice has resulted in the deaths of a population the size of Jersey City, or maybe Oakland, instead of one the size of Baltimore, you probably need to rethink your priorities.

Some might suggest that the truth or falsity of these stories doesn’t matter much in the light of this. or this or this or this, to list just a few of the disasters have taken place while the wingnutosphere has been defending the US Army’s commitment to animal welfare.

But that would miss the point. What matters, in the world of rightwing postmodernism, is not reality but the way the media reports it. One bogus memo is enough to turn George W. Bush from a scrimshank who used his family connections to line up a cushy billet to avoid war service, and then shirked even that, into a war hero.

So, lets stick to media criticism. Not long after Beauchamp’s piece ran in a single magazine of modest circulation, all the major MSM outlets ran a story by well known critics of the war, Michael O’Hanlon and Kenneth Pollack whose intrepid journey through recently pacified parts of Iraq had convinced them that the surge was working. Here, for example, is their piece in the NY Times.

Oddly enough, rightwing scepticism about the MSM was suspended for this piece. The fact that Pollack (author of The Threatening Storm: The Case for Invading Iraq) and O’Hanlon had consistently supported the war, the occupation and the surge was not seen as anything to worry about. And none of the armchair experts worried at all about the logistical and technical issues on which they are usually so keen to display their expertise. It was left to Glenn Greenwald to point out that Pollack and O’Hanlon went on a guided tour organised by the US military, spent every night in the Green Zone, and formed many of their most striking impressions on the basis of two-hour visits to places like Mosul. Meanwhile, a third report from Anthony Cordesman was much less optimistic.

Surely by now the wingnutosphere could have come up with evidence that two hours is more than enough time for a comprehensive helicopter tour of Mosul, avoiding the main street bias that they spent so much time on in relation to the Lancet study. And surely they could discredit the memos that appeared to show O’Hanlon and Pollack supporting the war when everyone knows they are not only consistent war critics but (shudder!) Democrats.

So where are the defences of O’Hanlon and Pollack? Technorati finds a few sites still trumpeting the initial report, and some pushing a similar one from Der Spiegel, but that’s about it. Apparently it’s more important to prove that an obscure private is telling tall tales than to offer a serious defence of the latest claims of imminent victory.

Update It really is a parallel universe. A string of rightwing blogs have commented on this and without exception they’re horrified by the suggestion that Scott Beauchamp’s veracity, and the credibility of The New Republic (!) as a news source are trivial issues compared to the question of whether there is any chance to salvage the current mess in Iraq. If anyone can follow Megan McArdle’s explanation of why Pollack and O’Hanlon’s account of the surge (which she does not attempt to defend) is in a category that renders it immune from refutation, please explain it to me. Moe Lane at RedState gets very upset at my failure to follow “blogging etiquette” and link to the nonsense on this topic, so here’s his link. Interestingly this guy offers, as his post title notes*, a “fake but accurate” defence of Pollack and O’Hanlon against Greenwald’s evidence that they could not have genuinely observed what they claimed, namely that “he [Greenwald] cannot account for Petraus, who has spent much longer in Iraq than O’Hanlon, seems to broadly agree with O’Hanlon’s optimism. That optimism appears to be largely shared by officers and senior NCOs with multiple tours in Iraq.” Well, not all of them, it seems. As for Petraeus …

• (A joke of course. He doesn’t realise he is adopting the very style of argument he claims to repudiate).

I started with the UK Census. I’ve always thought that there were maybe a few more, or possibly less, ethnic minorities in Camden than the census said, so I phoned them up and asked for the data. The woman on the end of the line pointed me toward their website and noted that there was quite a lot of county-level data there which might be helpful. I explained that no, I wanted the data, by which I meant the actual census forms. They won’t release the data! Really! I shouted that this was a fundamental building block of the scientific method, and that her sinister refusal to hand over the forms to any random person who asked was the equivalent of the Catholic Church burning Galileo[1]. While she was on the line, I asked for the last month’s death figures for Central London – after all, since she’s the central registering authority for births and deaths, she ought to have them at her fingertips as they must magically update every time a hospital morgue writes a certificate. I think she was in tears by the time she slammed the phone down, so Advantage: Blogosphere!

Next on to the Dow Jones Industrial Average people. Did you know that there are three entire missing days from their figures, which suspiciously enough[2] just “happen” to be September 12-15, 2001???????Q? I suppose we are meant to assume that this “missing cluster” was selected at RANDOM Some chance. Clearly the leftist MSMs of Dow Jones International censored these numbers, because they would have added so much to the variance of the DJIA that we could no longer be sure that it wasn’t 36,000! Perfidy! Wal-Mart are releasing their Q2 earnings numbers next week, or at least I should say “releasing” their “numbers”, because as I found out, when you go down to Bentonville demanding a look at the till rolls, you don’t even get let into the car park. Scientific method, my ASS!

Stay tuned for more science, readers, because until this case of Red Bull runs out, I am going to be a blogoscientific force of nature!

[1] Galileo was not actually burned, but I am now a right wing crazy person, so this kind of factul nitpicking no longer bothers me.

[2] The fonts are a lot more fun on this side of the political divide too.

For the wingers, this is a continuous pattern. Before this, there was a flap about a report that failures by contractors were resulting in troops in the field not getting adequate food. Before that, it was the Jamil Hussein case, a months-long brawl with AP arising from a report by a stringer about attacks on mosques. Before that, it was reports from Lebanon of ambulances being hit by Israeli fire. And so on.[fn1] There’s too much of this to try and give comprehensive coverage, and I’m not interested in debating the details, but a search on Instapundit will usually get you started.

The Beauchamp case fits the general pattern pretty well. First, the wingers claimed that the Diary was a fabrication and that “Scott Thomas” was the creation of a writer who’d never been near Iraq. Then, when it became evident he was a real person, they rolled out the slime machine to discredit him. Then they engaged in amateur forensics to discredit particular items in his account (acres of screen space have been devoted to the question of whether the driver of a Bradley fighting vehicle can run over a dog). Then they got to the central point – true or false, material like this is bad for the cause and shouldn’t be printed.

All of this, of course, is an attempt to replicate the one undoubted triumph of the blogospheric right, Rathergate. For those who somehow missed it, Dan Rather and CBS fooled by a bogus memo purportedly from Bush’s National Guard commander, and Rather eventually lost his job as a result.

As I said, I’m not interested in, and won’t debate, the details of these stories. The main question is: How anyone could imagine that this kind of exercise can have any value?

Suppose that everyone of the stories being discussed above was a deliberate fraud. It would not change the fact that the Iraq war has been a catastrophic failure, and that US media coverage, far from being overly pessimistic, failed to alert the US public to these disasters as they unravelled[fn2].

At one time of course, it was claimed that the media was failing to cover the “Good News from Iraq”. In that context, the idea that the bad news was bogus at least made some sort of sense. But the last “Good News” purveyor of any consequence, Winds of Change, quietly gave up this exercise at the beginning of this year. The news is nearly all bad, and what’s not reported (since reporters can’t travel much any more) is almost certainly worse. But still WoC and others persist in picking on individual, usually trivial, stories and giving them the Rathergate treatment.

The sheer volume of bad news makes piling on particular articles look really silly. In the time that’s been devoted to the treatment of Iraqi dogs, for example, Google News reports thousands of stories in which the only good news I can see is a win for the national soccer team (followed, inevitably, by this).

The fundamental problem here is that the argument-by-talking-point mode that characterizes the entire rightwing blogosphere (and in which some of the left sometimes gets involved also) works fine in the context of US political debate, where perception is all that matters. If you can sell George Bush as a hero and John Kerry as a coward, then that is, for electoral purposes, what they are. But when you start dealing with intractably factual problems like war, and making policy on the basis of wishful thinking and talking points, reality tends to bite back.

1. There’s also the endless quibbles about estimates of excess deaths (the ‘Lancet’ controversy) where the issues are a bit larger, but where the rhetorical approach and level of argument from the wingers is much the same.
2. As was the Israeli invasion of Lebanon, but the Israeli press did a much better job, and the failure of the invasion was quickly recognised there).

My advice is, David don’t hand this paper out. If not for the sake of your own reputation, think of the four (! On a tiny little paper like this!) research assistants you credit in it. The paper is a disaster. As the comments thread at Deltoid gradually teases out, it’s full of silly mistakes (the author constantly fails to make a distinction between an estimate and its confidence interval) and is based on a fundamental misreading of the paper (in that it assumes that the relative risk rate was estimated parametrically using a normal distribution when it wasn’t). But one doesn’t need to go into the maths of the thing to understand what’s wrong with it.

The mathematical guts of the paper is that under certain assumptions, the addition of the very violent cluster in Fallujah can add so much uncertainty to the estimate of the post-invasion death rate that it stretches the bottom end of the 95% confidence interval for the risk rate below 1. From this, David Kane concludes that the paper was wrong to reject the hypothesis that the Iraq War had not made things worse.

Let’s back up and look at that again. Under David Kane’s assumptions, the discovery of the Fallujah cluster was a reason to believe that things might have gone better in Iraq. This clearly means that these were the wrong assumptions.

The statistical problem here is basically that people can’t come back from the dead. The Fallujah datapoint increases the uncertainty of the estimate, but it doesn’t increase it in both directions, because there is no way that you could find an “anti-Fallujah” (a datapoint which brought the overall average down by as much as real Fallujah brought it up), because such a place would need to have a negative death rate.

And looking at the charts in David’s paper, it’s clear to see that the reason why the left edge of his estimate of the risk ratio has been dragged below 1 is that a substantial part of the distribution of his Bayesian estimate of the post-war death rate is below zero (and an even more substantial part is in regions of positive but wildly improbably death rates like one or two per 100K). That’s all there is to it, CT readers; the majority of the rest of the Deltoid thread consists of three or four people trying to explain that the Roberts et al. paper doesn’t make the same mistake.

As I note halfway down the thread, this is actually a nice example of some of the cases where the distinction between a frequentist confidence interval and a Bayesian credible interval makes an importance difference. In an infinitely repeated series of trials, you might very well get a small number of very unlucky or wild results that showed death rates of 1 or 2 per 100K. So if you’re thinking about the confidence interval as the limit of the empirical distribution of the estimator in repeated trials, it makes sense to have it where it is. But if you’re a Bayesian and you regard the confidence interval as your subjective probability distribution over a random variable, then it doesn’t make any sense at all to have any material weight on these low-end numbers (if you were a conscientious Bayesian, of course, you would never get into this position as you’d have used a sensible prior distribution which put zero probability on death rates below zero). In general, there is a distressing trend among statisticians to use the branding “Bayesian” (and the ubiquity of “diffuse” priors which don’t rule out silly cases like this one) as an excuse for talking crap about confidence intervals, and I think this is an example of the genre.

If any readers are attending the JSM, I’d be interested in any reports of how it went down. Now I’m off to spend the weekend playing Wingnut Whackamole I suppose …

(PS: actually, the cargo cult explanations of why the Lancet study has allegedly failed to enact someone’s ideas of the rituals of science are really quite interesting from a sociological point of view. Although I do wonder about the incuriousness here about the actual facts. I mean, if the “scientific method” regularly threw up conclusions like “knowing what we do now, it is quite likely that the Iraq War was actually a success”, do you think it would be as popular as it actually is?)

What on earth can be in this data? Presumably the UK and US authorities have reasoned that Dr Lafta is an ex Ba’ath Party member (as he would have had to have been to hold a position in the Iraqi Health Ministry), and thus the data he is carrying is not really about child cancer at all. Perhaps he is involved in some sort of “Boys from Brazil” type plot to clone an army of super-soldiers from Saddam Hussein’s DNA, and for this reason the UK cannot be exposed to this deadly information for even four hours in the Heathrow transit lounge.

The alternative – that Dr Lafta is being intentionally prevented from travelling in order to hush up his research on post-war deaths (research which even the Foreign Office have now more or less given up on trying to pretend isn’t broadly accurate), or to hush up the news about paediatric cancer for political convenience – is too horrible to contemplate. I’d note that there isn’t an election on in the USA at present, so the denialist crowd can shove that little slur up their backsides this time too.

(thanks to Tim Lambert as always)

In semi-related news, and with apologies to the person who gave me the tip for taking so long to post it, it appears that Professor Michael Spagat, the author of the “main street bias” critique, has a bit of previous form when it comes to making poorly substantiated and highly inflammatory statements about other people’s research. His involvement with the general issue came about because he’d been using some of the IBC data in support of a power law hypothesis[1] about the scaling of violent deaths. This carried on from previous work he’d done on Colombia, where he had also defended his own somewhat tendentious interpretation on the data by slagging off Human Rights Watch. I sense something of a pattern here; I noted in a previous post that although the “main street bias” critique appeared in the Lancet colloquium on the Burnham et al paper, Prof. Spagat himself did not, and I thought at the time it might be because of this habit.

[1] And one of Prof Spagat’s co-authors on the main street bias paper, and a few others in the power law of violence series was Neil Johnson of Oxford University, who was also a co-author of that paper about the Eurovision Song Contest that we had a go at a while ago, and so the circle of minor irritation is complete.

Several academics have tried to find out how the Lancet study was conducted; none regards their queries as having been addressed satisfactorily. Researchers contacted by The Times talk of unreturned e-mails or phone calls, or of being sent information that raises fresh doubts.

Yes indeed, out of the population of people with outstanding questions, none of them have had their questions resolved.

There are quite a lot of oddities in the article, covered by Tim Lambert here. I note that Professor Michael Spagat seems to be quite determined to talk about fellow researchers in an extremely abusive and (IMO) unprofessional manner to journalists; this doesn’t go very well for my assessment of his credibility[1]. It also seems rather weird to me that, while all the issues raised in the article are clearly drawn from the recent correspondence published in the Lancet on the subject, the article doesn’t even mention Glibert Burnham’s “Reply to Critics” published in the same issue and indeed strongly implies that it doesn’t exist. I suspect that Anjana Ahuja has got all her information from interviewees who should have told her about the correspondence but didn’t.

I’ve held off on writing about this for a couple of days, partly because of laziness, but partly because I wanted to test James Callaghan’s old adage “a lie will be half way round the world before the truth has got its boots on”. The thing is, that there is a “marker” in the Times article – as in, a statement that is not true and that is obviously not true to anyone who has read the article. It is in the following paragraph:

Dr Richard Garfield, an American academic who had collaborated with the authors on an earlier study, declined to join this one because he did not think that the risk to the interviewers was justifiable. Together with Professor Hans Rosling and Dr Johan Von Schreeb at the Karolinska Institute in Stockholm, Dr Garfield wrote to The Lancet to insist there must be a “substantial reporting error” because Burnham et al suggest that child deaths had dropped by two thirds since the invasion. The idea that war prevents children dying, Dr Garfield implies, points to something amiss.

This is not true. As table 2 of the study shows, infant mortality remained constant in the survey (when you adjust for the greater number of months in the post-war recall period) while child deaths increased substantially. They did not drop by two thirds, or indeed drop at all. Von Schreeb, Rosling and Garfield did not say they dropped either (presumably because they have read the survey). They said that the crude estimate of under-15 mortality was substantially lower than other estimates of under-5 mortality in Iraq, and that this implied that there may have been substantial under-reporting of child deaths. They then suggested that this reporting error might lead to additional uncertainty in the estimates of roughly the same size as the sampling error – +/- 30%. Note that, for bonus hack points, the “plus” sign in “+/- 30%” is not ornamental, and to treat Von Schreeb et al as providing evidence that the study was an overestimate is Kaplan’s Fallacy. This is my reason for believing that Anjana Ahuja didn’t read the research; it’s an error that could easily have been made in transcribing notes of a half-understood conversation but couldn’t have been made at all if you read the articles.

So anyway: As a Google search shows, the fact that this claim from the Times article is completely and obviously wrong has not stopped it from being reproduced all over the blogosphere. It is a pretty poor show. The real joke is that most of the people concerned are also screaming about “political bias” on the part of the Lancet team!

I am curious as to why anyone is bothering with this debate any more (in some of the discussion on Dr Madelyn Hsiao-Rei Hick’s comments, it has got parodic, as people discuss the minutiae of the “informed consent” requirements of the questionnaire). Does anyone think at this late date that they are going to come up with a result that proves that the whole war and occupation has been really good for the Iraqis? Have they not noticed that this debate (and the one on global warming too) is a bit like the Berlin Wall – people are only going from one side to the other in one direction?

Look, if you want a “resample”, here’s a small-sample survey of people involved in the original study. Of the eight Iraqi doctors who carried out the interviews in October 2006, one has been murdered.

[1] I think it is not a coincidence that Spagat did not get a letter published in the correspondence in the Lancet, although other people referencing his “Main street bias” theory did. He really seems to have gone off at the deep end on this one. He also appears to have resurrected the “three to one rule” (three wounded to one killed) which I have only ever otherwise seen in the most desperate parts of the blogosphere. If this rule exists at all (and I have never seen a citation to any actual literature), then it refers to soldiers fighting in battles. Soldiers are physically fit, wear tin hats and make use of cover. The majority of killings in Iraq are execution-style murders, carried out with close range shots or electric drills. I am tempted to call Prof. Spagat as a witness to exonerate Dr Harold Shipman – after all, there is no sign of the 750 old ladies who were merely wounded who we otherwise “ought” to see if the “three to one rule” was a universal rule.

Megan McArdle was wrong about Iraq, but those of us who were right on every important detail (we know who we are[1]) shouldn’t be acting all smug, and we certainly shouldn’t be making acid remarks about the credibility of the people who got it wrong, because hey, the important thing is to all learn from this experience so that we can make better decisions and predictions in future. No really, it was the antiwar side that had made up their minds before making any arguments, the antiwar side that kept changing their arguments with the prevailing winds, and the antiwar side that were the worst offenders in terms of nasty triumphalism afterwards. Me neither.

I’m not sure I recall this “let’s not argue about who killed who’s cat” stuff as being a particularly common reaction in the right wing blogosphere to massive failures in public sector projects, but there is more joy in heaven, etc, etc, etc. At the very least, presumably Megan won’t be quite as quick to start acting as a news aggregator for Lancet mortality study hacks and cranks, next time round.

So anyway, if we’re all learning from our mistakes and improving our decision making processes, then probably the best way of going forward is to take what we’ve learned about the advisability and consequences of the Iraq War as it was fought in 2003, and apply them to the new, very similar, but smaller and therefore easier problem of predicting what will be the result of the “surge” strategy of sending 20,000 more troops there now? Personally, I think it’s not going to work[2]. How about you, Jane Galt?

[1] Actually I screwed up pretty badly once, in 2005, on Iraq, because I predicted that the secularists would have much more success in the elections than they actually got, and that the base of support for the insurgents was much smaller than it actually was. Looking back at the original post, it appears that my main mistake was that I read the Economist and believed it. I have adjusted my weightings accordingly.

[2] Sorry, can’t be more specific than that. Off the top of my head I can come up with five scenarios:
a) Al-Sadr shifts his operations to another part of Iraq, leaving a load of troops doing nothing in Sadr City as open street war flares up in Basra
b) The US troops get bogged down in urban combat until their Iraqi allies turn on them and/or a massacre of civilians happens
c) Military coup or other collapse of the Maliki government
d) Al-Sadr demonstrates his political nous once more, and calms down his operations, carrying out only enough hit-and-run attacks on US troops to keep his popularity up. Then he forms a nationalist bloc with one or more of the Sunni parties. Political collapse of the Maliki government.
e) outright Rwanda-style massacre of Sunnis and Sadrists by the Badr Brigades.

the point is that there are about a million ways in which this could go wrong, and only one way it might go right, a point that John has made on a number of occasions.

Update: Kane’s accusations have been removed from the front page of the SSS blog. In a follow-up, Amy Perfors apologises for the error of judgment and says they removed the post because the “tone is unacceptable, the facts are shoddy, and the ideas are not endorsed by myself, the other authors on the sidebar, or the Harvard IQSS.” Good for them.

He doesn’t have any positive evidence. He just begins from the idea that the number is too big, and asks who would be responsible for faking it. His answer is, the survey team:

We know very little about these Iraqi teams. Besides monetary incentives to give the Lancet authors the answers they wanted, the Iraqis may have had political reasons as well. … Were any former members of the Baath Party? … How can anyone know that they are telling the truth? … The interviewers could, at their discretion, change the location of the sample. How many times did they do this?

Not having a bit of evidence on any of these points, he goes on to put a lot of weight on the paper’s extremely high (in fact, near perfect) response rate, and quotes a commenter from one of our own threads saying, correctly, that such rates would be seen as unbelievably high if reported by surveyors in the U.S. or Europe. Following the commenter, Kane concludes that the most likely possibility is that “the survey teams provided fraudulent data.”

The immediate problem with this charge is that, as it turns out, phenomenally high response rates are apparently very common in Iraq, and not just in this survey. UK Polling Report says the following:

The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that – incredibly – response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed.

If this is correct, then the only bit of circumstantial evidence that Kane proffers in support of his insinuation is in fact a misconception based on his own ignorance.

Kane says, “I can not find a single example of a survey with a 99%+ response rates in a large sample for any survey topic in any country ever.” I googled around a bit looking for information on previous Iraqi polls and their response rates. It took about two minutes. Here is the methodological statement for a poll conducted by Oxford Research International for ABC News (and others, including Time and the BBC) in November of 2005. The report says, “The survey had a contact rate of 98 percent and a cooperation rate of 84 percent for a total response rate of 82 percent.” Here is one from the International Republican Institute, done in July. The PowerPoint slides for that one say that “A total sample of 2,849 valid interviews were obtained from a total sample of 3,120 rendering a response rate of 91 percent.” And here is a report put out in 2003 by the former Coalition Provisional Authority, summarizing surveys conducted by the Office of Research and Gallup. In the former, “The overall response rate was 89 percent, ranging from 93% in Baghdad to 100% in Suleymania and Erbil.” In the latter, “Face-to-face interviews were conducted among 1,178 adults who resided in urban areas within the governorate of Baghdad … The response rate was 97 percent.” So much for Iraqi surveys with extraordinary response rates being hard to find.

Oddly, the comment that Kane picked up from our thread was quickly rebutted by other commenters, who made the same point as the one above. I guess Kane didn’t wait around to find out.

Accusations or insinuations of fraud are a serious matter, especially in a case like this. I have to say I am surprised—and dismayed—to see this balloon being floated at the Social Science Statistics Blog. I’m a fairly regular reader of theirs. It’s run under the auspices of the Institute for Quantitative Social Science, an interdisciplinary group at Harvard. Most of the posts are by Harvard grad students, but the sidebar also includes respected heavy-hitters like Jeff Gill and the Institute’s director, Gary King. The blogosphere being what it is, I expect posts with titles like “Harvard statistics blog says Iraq survey results may be fraudulent” to start popping up pretty soon. I wonder whether Prof. King is aware of Kane’s post, and whether he thinks it’s alright that his Institute is providing a platform to Kane to make his claims of fraud.

One number that is striking, but hasn’t attracted a lot of attention is the estimated death rate from air strikes, 13 per cent of the total or between 50 000 and 100 000 people. Around half the estimated deaths in the last year of the survey, from June 2005 to June 2006. That’s at least 25 000 deaths, or more than 70 per day.

Yet reports of such deaths are very rare. If you relied on media reports you could easily conclude that total deaths from air strikes would only be a few thousand for the entire war. The difference between the numbers of deaths implied by the Lancet study and the reports that shape the “gut perceptions” that the Lancet must have got it wrong are nowhere greater than here. So are the numbers plausible?

I recall seeing only a handful of mentions of air strikes in the mainstream press. In checking my perceptions on this, I found this piece by Norman Solomon (linked by by Dahr Jamil) who notes that a search for “air war” produces zero results for the NYT, Washington Post and Times. Solomon refers to the earlier New Yorker article by Seymour Hersh who makes the same point.

The best source turns out to be the US Air Force Command itself. For October and November 2005, the US Air Force recorded 120 or more air strikes, and this number was on an increasing trend. Most of the strikes appear to be in or near urban areas, and the recorded examples include Hellfire missiles fired by Predators, an F-16 firing a thousand 20mm cannon rounds and an F-15 reported to have fired three GBU-38s, the new satellite-guided 500-pound bomb designed for support of ground troops in close combat. Typical reports of air strikes involve the destruction of buildings in which suspected insurgents are seen taking shelter, or from which fire has been reported. Obviously there is no opportunity to check whether such buildings are occupied by civilians.

An average of 10 fatalities for each air strike seems plausible. If we assume the average number of US plane and missile strikes for the year as a whole was 150 per month, that’s 18000 fatalities for 2005-06. Taking into account strikes by British and other allied forces and by attack helicopters (which seem to be used a lot, but are also rarely reported) it seems likely that Coalition air strikes killed more than 20 000 people in 2005-06.

That’s below the Lancet range of estimates, but in the same ballpark. To explain the gap, I’d suggest that it’s likely that the cause of death has been reported wrongly (or at least, inconsistently with official US accounts) in some cases. I’ve seen quite a few cases where Iraqis have blamed US air strikes for deaths, while the US authorities have denied that there were any strikes in the area and have blamed the deaths on insurgent mortar attacks. That seems to suggest that deaths attributed to air strikes may actually have been caused by artillery on one side or the other.

Based on the survey, and allowing for some misclassification, it seems likely that Coalition air and ground forces have killed between 100 000 and 200 000 people since the war began. The majority of these are military age males, most of whom would have been targeted as suspected insurgents, although we have no real idea how many actually were insurgents and how many were just in the wrong place at the wrong time. Around 70 per cent of all violent deaths in the Lancet survey were of military age males, and presumably the proportion would be higher for the Coalition since they are at least trying to avoid civilian casualties. But even if 80 per cent of those killed were insurgents, that would leave somewhere between 20 000 and 40 000 innocent civilians killed by Coalition forces so far. And of course, the figure also implies that even after 80 000 to 160 000 suspected insurgents have been killed, the situation is going backwards.

For more reactions, see Tim Lambert.

1. Iraq is a young country. Therefore, it has a low “crude” death rate. “Crude” in this case means “not adjusted for demographic structure and therefore not meaningfully comparable across countries”. Therefore, it is not surprising that pre-war Iraq had a crude death rate similar to that of Denmark, any more than it is surprising that any other two completely non-comparable statistics might happen to be the same number.

2. When someone dies, you get a death certificate from the hospital, morgue or coroner, in your hand. This bit of the death infrastructure is still working in Iraq. Then the person who issued the death certificate is meant to send a copy to the central government records office where they collate them, tabulate them and collect the overall mortality statistics. This bit of the death infrastructure is not still working in Iraq. (It was never great before the war, broke down entirely during the year after the invasion when there was no government to send them to and has never really recovered; statistics agencies are often bottom of the queue after essential infrastructure, law and order and electricity). Therefore, there is no inconsistency between the fact that 92% of people with a dead relative could produce the certificate when asked, and the fact that Iraq has no remotely reliable mortality statistics and quite likely undercounts the rate of violent death by a factor of ten.

Go on and sin no more, or at least not on our Lancet comments threads.