Daniel wrote a piece for the Guardian’s blog saying that critics who wanted to reject the findings of Burnham et al.’s Lancet paper and believe the Iraq Body Count estimate (or similar-sized numbers) were going to have to come out and claim that the paper was fraudulent, “and presumably to accept the legal consequences of doing so.” Well, now David Kane has floated that balloon.
Update: Kane’s accusations have been removed from the front page of the SSS blog. In a follow-up, Amy Perfors apologises for the error of judgment and says they removed the post because the “tone is unacceptable, the facts are shoddy, and the ideas are not endorsed by myself, the other authors on the sidebar, or the Harvard IQSS.” Good for them.
He doesn’t have any positive evidence. He just begins from the idea that the number is too big, and asks who would be responsible for faking it. His answer is, the survey team:
We know very little about these Iraqi teams. Besides monetary incentives to give the Lancet authors the answers they wanted, the Iraqis may have had political reasons as well. … Were any former members of the Baath Party? … How can anyone know that they are telling the truth? … The interviewers could, at their discretion, change the location of the sample. How many times did they do this?
Not having a bit of evidence on any of these points, he goes on to put a lot of weight on the paper’s extremely high (in fact, near perfect) response rate, and quotes a commenter from one of our own threads saying, correctly, that such rates would be seen as unbelievably high if reported by surveyors in the U.S. or Europe. Following the commenter, Kane concludes that the most likely possibility is that “the survey teams provided fraudulent data.”
The immediate problem with this charge is that, as it turns out, phenomenally high response rates are apparently very common in Iraq, and not just in this survey. UK Polling Report says the following:
The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that – incredibly – response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed.
If this is correct, then the only bit of circumstantial evidence that Kane proffers in support of his insinuation is in fact a misconception based on his own ignorance.
Kane says, “I can not find a single example of a survey with a 99%+ response rates in a large sample for any survey topic in any country ever.” I googled around a bit looking for information on previous Iraqi polls and their response rates. It took about two minutes. Here is the methodological statement for a poll conducted by Oxford Research International for ABC News (and others, including Time and the BBC) in November of 2005. The report says, “The survey had a contact rate of 98 percent and a cooperation rate of 84 percent for a total response rate of 82 percent.” Here is one from the International Republican Institute, done in July. The PowerPoint slides for that one say that “A total sample of 2,849 valid interviews were obtained from a total sample of 3,120 rendering a response rate of 91 percent.” And here is a report put out in 2003 by the former Coalition Provisional Authority, summarizing surveys conducted by the Office of Research and Gallup. In the former, “The overall response rate was 89 percent, ranging from 93% in Baghdad to 100% in Suleymania and Erbil.” In the latter, “Face-to-face interviews were conducted among 1,178 adults who resided in urban areas within the governorate of Baghdad … The response rate was 97 percent.” So much for Iraqi surveys with extraordinary response rates being hard to find.
Accusations or insinuations of fraud are a serious matter, especially in a case like this. I have to say I am surprised—and dismayed—to see this balloon being floated at the Social Science Statistics Blog. I’m a fairly regular reader of theirs. It’s run under the auspices of the Institute for Quantitative Social Science, an interdisciplinary group at Harvard. Most of the posts are by Harvard grad students, but the sidebar also includes respected heavy-hitters like Jeff Gill and the Institute’s director, Gary King. The blogosphere being what it is, I expect posts with titles like “Harvard statistics blog says Iraq survey results may be fraudulent” to start popping up pretty soon. I wonder whether Prof. King is aware of Kane’s post, and whether he thinks it’s alright that his Institute is providing a platform to Kane to make his claims of fraud.