Alice in Wonderland and the Lancet study

by Daniel on July 27, 2007

(Initial bad temper warning: I am a little bit cross as I write this, because I think that the distribution of the paper on the Michelle Malkin website was both silly (because the paper has huge flaws that a mass audience can’t possibly be expected to understand) and rude (because at the time when he gave permission for it to be distributed, David was soliciting comments, seemingly in good faith, from the Deltoid community, aimed at improving it before distribution). The Malkin link has meant that this paper has metastatised and I will therefore presumably be dealing with cargo-cult versions of it by people who don’t understand what they’re talking about from now to the end of time. I see that Shannon Love of the Chicago Boyz website is claiming to have been “sweetly vindicated”, FFS. Ah well, the truth has now got its boots on, and big clumpy steel toe-capped boots they are too. C’mon boots, let’s get walking.)

At this late stage, does anyone believe that careful metaanalysis is going to reveal that the Lancet studies were totally wrong, and that the invasion and occupation of Iraq actually went really well? Apparently yes; David Kane of the Harvard Institute for Quantitative Social Science (who CT readers might remember from this rather embarrassing incident last year) does. He apparently intends to present this paper at the JSM in Salt Lake City on Monday, arguing that the 2004 Lancet study actually could not rule out the possibility that the death rate had fallen in Iraq.

My advice is, David don’t hand this paper out. If not for the sake of your own reputation, think of the four (! On a tiny little paper like this!) research assistants you credit in it. The paper is a disaster. As the comments thread at Deltoid gradually teases out, it’s full of silly mistakes (the author constantly fails to make a distinction between an estimate and its confidence interval) and is based on a fundamental misreading of the paper (in that it assumes that the relative risk rate was estimated parametrically using a normal distribution when it wasn’t). But one doesn’t need to go into the maths of the thing to understand what’s wrong with it.

The mathematical guts of the paper is that under certain assumptions, the addition of the very violent cluster in Fallujah can add so much uncertainty to the estimate of the post-invasion death rate that it stretches the bottom end of the 95% confidence interval for the risk rate below 1. From this, David Kane concludes that the paper was wrong to reject the hypothesis that the Iraq War had not made things worse.

Let’s back up and look at that again. Under David Kane’s assumptions, the discovery of the Fallujah cluster was a reason to believe that things might have gone better in Iraq. This clearly means that these were the wrong assumptions.

The statistical problem here is basically that people can’t come back from the dead. The Fallujah datapoint increases the uncertainty of the estimate, but it doesn’t increase it in both directions, because there is no way that you could find an “anti-Fallujah” (a datapoint which brought the overall average down by as much as real Fallujah brought it up), because such a place would need to have a negative death rate.

And looking at the charts in David’s paper, it’s clear to see that the reason why the left edge of his estimate of the risk ratio has been dragged below 1 is that a substantial part of the distribution of his Bayesian estimate of the post-war death rate is below zero (and an even more substantial part is in regions of positive but wildly improbably death rates like one or two per 100K). That’s all there is to it, CT readers; the majority of the rest of the Deltoid thread consists of three or four people trying to explain that the Roberts et al. paper doesn’t make the same mistake.

As I note halfway down the thread, this is actually a nice example of some of the cases where the distinction between a frequentist confidence interval and a Bayesian credible interval makes an importance difference. In an infinitely repeated series of trials, you might very well get a small number of very unlucky or wild results that showed death rates of 1 or 2 per 100K. So if you’re thinking about the confidence interval as the limit of the empirical distribution of the estimator in repeated trials, it makes sense to have it where it is. But if you’re a Bayesian and you regard the confidence interval as your subjective probability distribution over a random variable, then it doesn’t make any sense at all to have any material weight on these low-end numbers (if you were a conscientious Bayesian, of course, you would never get into this position as you’d have used a sensible prior distribution which put zero probability on death rates below zero). In general, there is a distressing trend among statisticians to use the branding “Bayesian” (and the ubiquity of “diffuse” priors which don’t rule out silly cases like this one) as an excuse for talking crap about confidence intervals, and I think this is an example of the genre.

If any readers are attending the JSM, I’d be interested in any reports of how it went down. Now I’m off to spend the weekend playing Wingnut Whackamole I suppose …

(PS: actually, the cargo cult explanations of why the Lancet study has allegedly failed to enact someone’s ideas of the rituals of science are really quite interesting from a sociological point of view. Although I do wonder about the incuriousness here about the actual facts. I mean, if the “scientific method” regularly threw up conclusions like “knowing what we do now, it is quite likely that the Iraq War was actually a success”, do you think it would be as popular as it actually is?)

{ 7 trackbacks }

Lancet debunking cargo cult [Deltoid] · New York Articles
07.27.07 at 2:04 pm
Lancet debunking cargo cult [Deltoid] · Kokorec
07.27.07 at 2:05 pm
Crooked Timber » » One endless Rathergate
07.29.07 at 6:52 am
Sadly, No! » Dawn Of The Davies
07.29.07 at 4:48 pm
Science Blog » Blog Archive » Lubos Motl vs the logarithm function
07.31.07 at 6:24 pm
Science Blog » Blog Archive » Sloppy reporting in the National Geographic on DDT
08.07.07 at 4:27 am
Science Blog » Blog Archive » The New Phone Books Are Here!!! The New Phone Books are Here!!!!
08.08.07 at 5:02 am

{ 153 comments }

1

Brett Bellmore 07.27.07 at 10:52 am

I never thought the problem with the Lancet study was in the math, rather than the notion of attributing killings by one side in a war to the other. With the idea that the more innocent people the folks you’re fighting murder, the greater YOUR guilt becomes.

Mistakes like that aren’t math errors.

2

dsquared 07.27.07 at 11:09 am

I never thought the problem with the Lancet study was in the math, rather than the notion of attributing killings by one side in a war to the other.

Neither of the Lancet studies did this, TYVM, HTH, NFO.

3

Leinad 07.27.07 at 11:59 am

…and NFI to boot.

4

Suvi 07.27.07 at 12:10 pm

The Lancet studies calculated the number of “excess deaths” that occurred after the invasion – in other words the number of people who wouldn’t have died if the invasion had not taken place.

No matter what inferences one draws from them, the studies themselves did not give any value judgements, and in particular didn’t attempt to say ‘who killed who’.

It’s all there in black and white, unless I’ve horribly misunderstood.

5

Barry 07.27.07 at 12:11 pm

Brett, certain basic ideas are that (a) an occupying force has responsibilities (‘you break it, you bought it’), and (b) from a viewpoint of what is supposed to constitute US political, military and moral victory, having Iraq become a h*llhole is a resounding defeat.

6

Rob 07.27.07 at 12:40 pm

As I skim the comments from Lambert’s sire, is Kane argument really just that by including Fallujah and making a terrible assumption about normally distributed estimates, you can no longer exclude zero from the CI? So basically we have an example here that would be an exam questioning asking a stats student to explain what the problem appears to be and how to solve with the answer being normality and use bootstrapping? And Kane seems to think nothings wrong?

7

dsquared 07.27.07 at 12:52 pm

As I skim the comments from Lambert’s sire, is Kane argument really just that by including Fallujah and making a terrible assumption about normally distributed estimates, you can no longer exclude zero from the CI?

Yes. He also seems to claim that a) the authors used normality assumptions for their CIs for the pre and post invasion crude mortality rates (which they might have done but the paper doesn’t say so) and b) if they did so, they are committed to doing the same thing for the CIs for the relative risk and excess deaths calculation (patently stupid). At times it looks like he is saying c) that because of b) it is actually likely that the truth is that the confidence interval doesn’t include zero, which means that every 25 comments or so, somebody has to remind him how ludicrous this claim is. I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.

8

Rob 07.27.07 at 1:22 pm

I mean that sort of thing is good enough for AEI work, but he’ll be ripped to shreads by others. And he won’t likely luck out and come to a presentation with no one having read the paper given the topic. That would have been bad but survivable. But thats unlikely to happen in this case.

9

Rich Puchalsky 07.27.07 at 1:32 pm

“David was soliciting comments, seemingly in good faith, from the Deltoid community”

The guy produces a hack paper that mangles numbers to show that maybe the Iraq War isn’t going so badly after all — I don’t think that “good faith” is really an important concept in any connection with this incident. Let’s assume, which I doubt, that he isn’t a knowing propagandist. In that case, terminal carelessness is just about as bad.

And who hands their paper to a Web site for review? Couldn’t he have shown it to a statistician first? I think it’s more likely that the distribution of this apparently unpublisheable paper to a mass audience was the point.

10

dsquared 07.27.07 at 1:51 pm

And who hands their paper to a Web site for review?

In fairness, David was and is part of the Deltoid community, so I thought that bit was quite reasonable. I was really quite hacked off at discovering that it had simultaneously been mass-distributed though – am I being weirdly prissy and prima-donnaish here, or am I right that if you ask someone for comments, there’s an implicit understanding that you’re going to take those comments seriously and not do something like this.

I think it’s more likely that the distribution of this apparently unpublisheable paper to a mass audience was the point

I have an uncomfortable feeling that you’re right. David is polite and usually thoughtful, but after the thing last year and now this, I have to say I am very uneasy about him.

11

abb1 07.27.07 at 2:01 pm

Nice graphics. Negative mortality rate on the red-blue-and-green one is indeed intriguing; but on the other hand I hear these kinda things do happen in the Middle East once in a while.

12

dsquared 07.27.07 at 2:16 pm

the thing is, I suppose, that two years ago I would have gone off on a chainsaw frenzy all over the blogosphere spitting insults over anyone who reproduced these hack critiques. But now, it’s kind of like, why bother? It is no longer the case that there is any material risk that anyone who matters will start believing that the war has gone well. I probably ought to make a token effort in order to make sure that Les Roberts doesn’t get unfairly slimed any more than he has been already, but realistically, we are down to the “Crazy 27%” bedrock now, nobody who actually cares about facts is on the other side.

13

P O'Neill 07.27.07 at 2:22 pm

Hitchens was way ahead on that argument that the invasion may have saved lives [in Slate ]

Make the assumption that some percentage of those killed by the coalition are the sort of people who have been blowing up mosques, beheading captives on video, detonating rush-hour car bombs, destroying pipelines, murdering aid workers, bombing the headquarters of the United Nations, and inciting ethnic and sectarian warfare. Make the allowance for the number of bystanders and innocents who lost their lives in the combat against these fanatics (one or two, alas, in the single case of the precision bombing of Abu Musab al-Zarqawi, just to take one instance). But who is to say how many people were saved from being murdered by the fact that the murderers were killed first?

14

Steve LaBonne 07.27.07 at 2:23 pm

Exactly, no need to let it upset you. Only the completely beyond-help true believers are listening at this point, the vast majority of the US (and presumably even moreso British) populations are well aware that the war is a complete clusterfuck. Only “elite” punditocracy opinion in the US has yet to quite catch up, but it’s visibly getting there.

15

Seth Finkelstein 07.27.07 at 2:23 pm

Further on the sociology aspect, I think it would make for some interesting academic examination to try to estimate if the ability of demagogues to widely trumpet propaganda on blogs which are like MSM syndicated columns, versus the the ability of honest researchers to post real science on blogs very few people read, makes for a better or worse situation compared to previous media. The standard evangelist line (my sarcastic phrasing), runs something like “So what if there’s a whole well-funded infrastructure to pump out nonsense, anyone could read the correct information, so go blogs“. We don’t lack for case studies :-(.

16

Ambitwistor 07.27.07 at 2:26 pm

Just to be absolutely sure I’m not missing something:

Did Kane really do a full Bayesian calculation (posterior distribution from a likelihood and prior), and fail to put zero prior probability on negative death rates?

Really?

If so, that should be incredibly embarrassing for him. I don’t know how you can call an analysis “Bayesian” if you use logically nonsensical priors.

17

Patrick 07.27.07 at 2:28 pm

Heh. I like it. So if I see that the average income in Some Suburb is $30,000 in 2007, and in 2008 I do a random survey that finds that the average reported income for 99 respondents is $31,000, but the 100th respondent turns out to be Bill Gates responding from his secret suburban summer home, I can actually conclude that the average income may have gone down, because the volatility he introduces widens my standard deviation to the point that it encompasses the possibility of an anti Gates earning negative millions per year?

Is that an accurate summary of the error? Or am I still not getting it exactly.

18

John Emerson 07.27.07 at 2:29 pm

As Labonne points out, the average voter on this issue is far ahead of the opinion-leaders, media, Democratic Party leaders, IR professionals, and othe members of the elite. Sometime someone should talk about the significance of this for the theory that populism is a bad thing and that populists are mostly murderous bigots.

19

Brian Weatherson 07.27.07 at 2:38 pm

I’ll probably just make a fool of myself here, but I think Patrick’s example doesn’t quite get at how bizarre Kane’s reasoning is. After all, it is possible to earn negative millions per year. In a certain kind of farming community, it might not even be that unlikely that someone does this. Negative deaths per year is somewhat less plausible.

To be sure, Patrick’s underlying point that an extreme outlier in one direction doesn’t thereby make it probable that there are extreme outliers in the other direction is, I think, basically right. But the analogy he raised isn’t I think as absurd as what he was trying to analogise.

20

Steve K 07.27.07 at 2:44 pm

I’m struck by Kane’s defense of giving permission to Malkin to print his paper; at Deltoid he writes, “To be clear, I did not seek Malkin out. She is a Deltoid reader (and who isn’t?) and contacted me. Should I have refused her permission to reprint? If someone contacts me from DailyKos, should I refuse him permission?” I am a college instructor, and on my syllabus I include a statement about how racist (classist, sexist, etc.) speech will not be tolerated in my course. I would assume that a scholar’s opposition to racism should carry itself over their work outside of the classroom. Even the most cursory examination of Malkin’s site would reveal that she routinely spouts juvenile racist trash (generally about Muslims). Why is a college instructor allowing his material to be reprinted at her site? The problem isn’t just that her audience doesn’t understand statistics; it’s also that much of her audience is also committed to at least two profoundly racist principles: 1. that the deaths of Muslims do not matter much (Kane’s work makes it that much easier for them to ignore the reality of Iraqi suffering) and 2. that the United States is a benevolent empire that will redeem the evil Muslims (the standard racist colonialist narrative, which in diminishing the humanitarian cost of the war, Kane’s article also can be used to support). I would think a more responsible scholar would not provide academic cover for Malkin.

21

dsquared 07.27.07 at 2:45 pm

Patrick – Bill Gates would raise the average of 100 people by a lot more than $1k but basically yes.

There are some situations in which it would be reasonable to make something like this assumption – if you find a few chocolate boxes that have been overfilled, then you might reasonably surmise that quality control is to hell and some others have been underfilled. But this isn’t one of them and I think that’s pretty obvious.

22

Patrick 07.27.07 at 2:45 pm

But am I getting the basic idea?

I have a strong technical proficiency in college math, but I’m not so knowledgeable on real world statistical practice. There’s an error in Kane’s reasoning, and I get the basic idea that he’s re-casting the data in such a way that he gets an extremely high standard deviation which allows his confidence interval to encompass a negative change in death rate as a possibility. Is that correct? Or is there an added step of bad math that I am missing.

23

SG 07.27.07 at 2:47 pm

I’ve been thinking about these examples too. Mine is:

you want to determine the average co2 emissions for air transport of the average american. You do a survey of 33 people asking “what is the furthest you have ever travelled in an airborne vehicle”, then multiply their given distance by the average co2 emisssions for a plane (obviously this paper is only getting published in a crappy psych journal, which David will be lucky to do). Your first 32 respondents represent the average american to a t. The 33rd is Buzz Aldrin.

As a consequence of getting Buzz Aldrin in your sample, variance is huge, and the confidence interval for co2 emissions includes a large chunk of negative emissions.

bingo! US air travel is actually absorbing co2

That’s the David Kane school of statistical analysis at work.

24

dsquared 07.27.07 at 2:50 pm

bingo! US air travel is actually absorbing co2

more precisely, bingo, you cannot reject the hypothesis that US air travel is absorbing co2 at the 95% confidence level, as Buzz Aldrin would obviously also massively push up the average, rather as Fallujah does.

25

SG 07.27.07 at 3:05 pm

Indeed, dsquared, but by the time it reaches Michelle Malkin it’ll be all “we absorb co2! Fly more planes! kill greenies!”

26

Rob 07.27.07 at 3:15 pm

sg-not quite. Travel distance can’t be negative, the relative death rate could be. Some argued for the war *cough* dsquared *cough* based upon the sanctions being so bad that not doing anything was horrendous. Now the problem with Fallujah is that its so high to assume normality puts positive probability weight to negative death regions.

27

J— 07.27.07 at 3:17 pm

Here’s Kane on the paper’s mass distribution:

I wish that left wing blogs would also highlight the paper. Anyone have a contact at DailyKos or Talking Points Memo? I sought to post it at Deltoid, where all the Lancet geeks hang out. See the excellent discussion at the last link which Lowell provides. Malkin picked it up from there, and then it spread.

From EphBlog.

28

ejh 07.27.07 at 3:42 pm

I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.

What’s the chances of that?

29

Ron F 07.27.07 at 3:48 pm

I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.

You overlook that David Kane served as an officer in the United States Marine Corps from 1988 to 1991. (1)

Those flabby statisticians wouldn’t stand a chance against a trained killer.

Strange coincidence, isn’t it, that two of the loudest Lancet critics – David Kane and Michael Spagat – have both had connections with the U.S. military, with the latter bankrolled by military contractor Radiance Technologies.

When you stats wizards have finished shredding Kane’s latest screed perhaps you could calculate the odds of that happening ;)

Incidentally, did Spagat and Co’s MSB paper ever get reviewed and published, or is it sitting gathering dust on a shelf marked “Job Done”?

(1) http://www.iq.harvard.edu/People/people.php?info=166&sub=6

30

Jim Harrison 07.27.07 at 5:11 pm

Some of the original statistical research that documented the relationship of smoking and cancer was published in Lancet and was met with ferocious criticism, most of it paid for by tobacco companies. The Lancet studies of excess mortality in Iraq have met a similar reception. In both instances, people who never gave a damn about the niceties of statistical inference were instantly sure that filling lungs with carcinogens or countries with bombs and bullets couldn’t possibly have a bad overall effect. Unfortunately, cigarettes cause cancer and wars cause excess deaths.

31

Kenny Easwaran 07.27.07 at 5:21 pm

I see someone else made something like this point, but isn’t there technically a possibility of a negative *relative* death rate, which is what the study was about? Just as Fallujah went from being a (perhaps?) relatively ordinary town to one with massive conflicts, it’s at least conceivable that there could have been towns with massive conflict or gas attacks or whatever beforehand but settled down to just the average level afterwards. Of course, I don’t think that’s likely at all, but at least the negative figures aren’t the equivalent of a negative death rate.

Or am I just horribly mistaken, because I haven’t read either paper?

32

Robert 07.27.07 at 5:41 pm

Kenny Easwaran and rob ask about “relative” death rates. Kane’s argument is about the crude (i.e., absolute) death rate.

33

Robert 07.27.07 at 5:52 pm

dsquared wrote:

My advice is, David don’t hand this paper out.

That was my first impression, too, but once it got out into wingnut-land, I would think the proper response now changes to, “David, please hand this paper out“.

34

roger 07.27.07 at 6:12 pm

That was an interesting thread at Deltoid, much of which was way over my head. However, re the heart of the matter, at one point David Kane seems to say that he believes there were 100,000 violent deaths in Iraq in the timeframe of the Lancet study, which as somebody else puts out, would surely push the excess death total up to at least 300,000, given other similar war situations – for instance, children dying for want of doctors who have been killed, etc.

That seems simple enough. Malkin should be crowing about the fact that she has found someone who thinks excess deaths in Iraq, excluding the last year, are 300,000 +. Which is 250,000 more than Bush admits.

The case for this war has fallen apart so completely that the only pro-war argument which is really convincing involves two steps, the first one of which is getting a lobotomy.

35

llewelly 07.27.07 at 6:25 pm

You ignorant liberals are casually and foolishly ignoring a dangerous possibility: Saddam Hussein’s Secret Zombie Technology may have been unleashed in Iraq!

36

Ben M 07.27.07 at 6:34 pm

sg-not quite. Travel distance can’t be negative, the relative death rate could be.

Rob, the paper isn’t positing negative relative death rates, nor negative rates of population change. It’s really positing negative absolute death rates, in the left tail of the blue graph in Fig. 3.

37

Barry 07.27.07 at 6:59 pm

“Those flabby statisticians wouldn’t stand a chance against a trained killer.”

Nah. (a) we fight dirty, (b) we’d outnumber him, and (c) some of us (not me) aren’t flabby.

38

Patrick 07.27.07 at 7:21 pm

They could beat him approximately to death.

Anyways, I’m genuinely surprised now that I’ve looked at the Malkin article post. It genuinely has a graph of death rate per 100k that shows a possibility of the death rate being negative. Not a graph of the rate of change of the death rate, a graph of the absolute death rate. It does not take an advanced degree in statistics to remember to review your projections to see if they’re still within the range of valid values for your particular population.

39

Patrick 07.27.07 at 7:23 pm

In fairness, he does note that you can’t have a negative result. He then dismisses the concern by claiming that the results would be similar if you used a truncated curve. He doesn’t explain how or why, or show any math, or explain what he means by truncated.

40

Sampling Soup - czrpb 07.27.07 at 7:38 pm

… sampling and polling and the like are similar to a cook taking a spoonful of their well-stirred soup to determine how it tastes …

41

abb1 07.27.07 at 7:50 pm

…results would be similar if you used a truncated curve…

I think it’s like the man of the thread said: if your math (e.g. your assuming a normal distribution) leads you to the conclusion that there’s a non-zero probability of a negative mortality rate, then it may already be too late to start truncating.

42

David Kane 07.27.07 at 8:40 pm

I have asked Daniel to correct this false claim about my paper.

“based on a fundamental misreading of the paper (in that it assumes that the relative risk rate was estimated parametrically using a normal distribution when it wasn’t).”

That’s not true. Nowhere does the paper claim this. Why would it, since I know that a bootstrap was used to estimate the relative risk. Since Daniel may not be available, perhaps another CT author could correct this. There may be other flaws in the paper, but this is not one of them.

The scientific discussion is continuing over at Deltoid. All are welcome.

43

walt 07.27.07 at 8:46 pm

If that’s the case, David, then I fail to see the point of your paper. They estimate the relative risk by a completely different method. How are any of the calculations in your paper relevant?

44

dsquared 07.27.07 at 9:24 pm

re #44 I have refused to “correct” this in email to David since his paper[1] does not mention the word “bootstrap” at all and its conclusion (that the null hypothesis RR=1.0 cannot be rejected) is utterly dependent on assuming something other than the bootstrap.

[1] Or at least, the version that I am commenting on – he might or might not have changed it since then, but this is one of the things I am complaining about.

45

Shannon Love 07.27.07 at 10:12 pm

Daniel,

I am afraid that you have gotten a couple of important things wrong.

Under David Kane’s assumptions, the discovery of the Fallujah cluster was a reason to believe that things might have gone better in Iraq. This clearly means that these were the wrong assumptions.

Kane never asserts anything like this. In fact, he asserts exactly the opposite. The study is so flawed that, even though death rates have obviously increased, the study cannot mathematically exclude the possibility that death rates have stayed the same or even decreased.

The Fallujah datapoint increases the uncertainty of the estimate, but it doesn’t increase it in both directions, because there is no way that you could find an “anti-Fallujah” (a datapoint which brought the overall average down by as much as real Fallujah brought it up), because such a place would need to have a negative death rate.

Are you sure you wish to claim this? This means the study is inherently biased to exaggerate deaths because a cluster can only increase but never decrease the death rate. In other words, the study by design could only discover whether the death rate stayed the same or had increased. It could never detect whether the liberation saved lives. This would make the study fraudulent on its face.

If, however, the study could detect an improvement in mortality then Kane’s assumption of symmetrical uncertainty is valid and thus confirm his math.

Which is it? Is the study design inherently flawed or is Kane right? I know the answer according the study itself. Can you guess what it is? (Hint:try page 3)

46

update? 07.27.07 at 10:25 pm

AT DELTOID:

dsquared asks: Did she confirm that the relative risk rate CIs were bootstrapped?
Yes. Note that it has always been clear (to me) that the RR estimate comes from a bootstrap. Nowhere in my paper do I assume that RR CIs are normal. You mistakenly assert that I do this in your CT post. A correction would be most welcome.

Posted by: David Kane | July 27, 2007 01:35 PM

47

nick s 07.27.07 at 10:27 pm

In other words,

Strawmen are good at scaring crows? Indeed.

You appear to be arguing that the failure to observe mass resurrection in Iraq invalidates the study.

48

abb1 07.27.07 at 10:29 pm

What a bunch of nonsense, Shannon. You think you’re defending the author but instead you’re embarrassing him. That’s painful to watch, man.

49

John Quiggin 07.27.07 at 10:40 pm

Shannon, I think you need to bone up a bit on the concepts “zero” and “positive” before discussing statistics again.

50

Barry 07.27.07 at 11:10 pm

BTW, who the f*ck is Shannon Love? I had sorta thought that the Chicago Boyz blog was composed of Chicago U amlumni, or at least Econ Ph.D.’s of that school of economics. But I can’t find much in a few minutes of searching.

51

Shannon Love 07.27.07 at 11:49 pm

nicks,

You appear to be arguing that the failure to observe mass resurrection in Iraq invalidates the study

No, I am arguing that study as outlined in the paper could have detected whether the mortality rate improved, stayed the same or worsened. If the study could not do this, then it could never detect whether mortality had improved.

If you think real hard, I am sure you can think of way that people come to be alive beyond return from the dead. Les Robert et al were in fact smart enough to structure their study to capture this somewhat less than mystical effect.

52

togolosh 07.28.07 at 12:10 am

Christ almighty you people are ignorant! This is obviously why the UK had to release killer badgers in Basra – to hunt down all the zombies. Duh!

53

Shannon Love 07.28.07 at 12:18 am

john quiggin,

I think you need to bone up a bit on the concepts “zero” and “positive” before discussing statistics again.

No, you need to understand the difference between a ratio, which cannot be negative, and the difference between to ratios, which can.

54

Shannon Love 07.28.07 at 12:20 am

Sorry, last post should read:

No, you need to understand the difference between a ratio, which cannot be negative, and the difference between two ratios, which can.

55

SG 07.28.07 at 1:47 am

Shannon, love

why? Which two ratios are we differencing?

56

Patrick 07.28.07 at 2:02 am

The fact that a high data point cannot make the results go lower is not a flaw in the study. The fact that a highly divergent data point cannot widen the range of the studied value past the leftmost bound of the physically possible range is also not a flaw in the study. If you cannot see this, then you have no business discussing statistics.

And his graph that includes a tail where the death rate is negative is NOT a graph of rate of change of death rate. It is explicitly a graph of the death rate. It says so. On the graph.

57

David Kane 07.28.07 at 3:44 am

dsquared writes:

re #44 I have refused to “correct” this in email to David since his paper[1] does not mention the word “bootstrap” at all and its conclusion (that the null hypothesis RR=1.0 cannot be rejected) is utterly dependent on assuming something other than the bootstrap.

[1] Or at least, the version that I am commenting on – he might or might not have changed it since then, but this is one of the things I am complaining about.

1) This footnote is ridiculous. We are all talking about the same paper. It is still posted at Deltoid.

2) The fact that I do not use the work “bootstrap” is evidence of nothing. You assert that I assume X. You need to quote me assuming X. You can’t. I don’t assume it so I don’t write it. I challenge you to provide a quote.

3) You claim that my conclusion is “utterly dependent” on this assumption which I do not make. How so? I provide a series of steps. I make many specific claims. I use all sorts of equations. Nowhere do I say or imply or assume the specific estimation method used by the authors to calculate the RR. The method is irrelevant to my argument.

I feel like you have just claimed that I have assumed in the paper that all unicorns are purple. Your evidence? I do not use the word “pink” and my conclusion is “utterly dependent” on unicorn-coloration. How should I refute such gibberish?

58

Shannon Love 07.28.07 at 4:10 am

sg,

Which two ratios are we differencing?

Crude Mortality Rate Post-invasion minus Crude Mortality Rate Pre-invasion.

or CMRpost-CMRpre=deltaCMR.

Its evident that deltaCMR can be greater than zero if mortality after the invasion increase, equal to zero if mortality stayed the same or less than zero if mortality decreased.

Daniel is confusing relative risk with death rate. Relative Risk(RR) is calculated by CMRpost/CMRpre. RR can assume any value greater than zero. If the value is greater than one then the death rate increased, if less than one it decreased.

So, RR doesn’t have to be below zero but only below 1.

Kane bases his argument not on the variance of the RR between clusters but on the variance of deltaCMR which in turn determines whether RR>1 for the entire study. (see figure 3)

If ∆ CMR is less than 0, then the RR is less than 1 because
CMRpost is less than CMRpre. Given the data, there is a10%chance that ∆CMR

Common sense suggest that if no measurement in another cluster could even theoretically offset the Falluja cluster then the studies design would be flawed from the get go. The idea that the death rate or relative risk can only go up is silly on its face.

59

SG 07.28.07 at 4:32 am

Shannon, love,

as I pointed out on deltoid, given the data there is also a near-zero probability that the point estimates of CMR pre and CMR post are the same. It depends on your choice of starting point for David Kane’s argument (he hasn’t responded to this yet). This is due to the poor behaviour of the confidence intervals.

Also shannon, love, CMR pre and CMR post are ratios in name only – they have the same denominator. So really you are talking about the difference of two numbers (the absolute death counts).

And finally shannon, love, it is also possible to have a contradiction between a Relative Risk confidence interval and the confidence intervals of its contributing counts. It’s very easy to see this if you take the time to construct some figures.

David Kane’s paper confuses or misunderstands all these points. And that’s on top of all the misunderstandings dsquared has described here.

60

Shannon Love 07.28.07 at 4:35 am

Patrick,

The fact that a highly divergent data point cannot widen the range of the studied value past the leftmost bound of the physically possible range is also not a flaw in the study.

What are you talking about Relative risk of deltaCMR? In neither case does Kane push the value past the leftmost bound. DeltaCMR has no leftmost bound because its left and right bounds are utterly symmetrical. RR has no left most bound because, as a ratio its value is zero or greater by definition.

And his graph that includes a tail where the death rate is negative is NOT a graph of rate of change of death rate. It is explicitly a graph of the death rate. It says so. On the graph.

Not sure what you mean. Fig 2 clearly has a caveat that the graph is not wholly accurate because CMR cannot be

61

John M 07.28.07 at 4:55 am

man i’m glad i don’t do social science.

62

Shannon Love 07.28.07 at 5:17 am

sg,

…given the data there is also a near-zero probability that the point estimates of CMR pre and CMR post are the same. …

The point estimates may differ but we know that CMR pre and CMR post are identical in the majority of clusters because the majority of clusters report zero excess deaths.

…CMR pre and CMR post are ratios in name only – they have the same denominator.

Umm, definitely wrong. See the 2004 Lancet paper p3 pg which define both CMRs as (number of deaths recorded/number of person-months lived in the interviewed households). That makes each CMR a true ratio. Beside rates are ratios by definition.

So really you are talking about the difference of two numbers (the absolute death counts).

No, the study document excess deaths attributable to chance in CMR pre and post. It calculates that by multiplying deltaCMR by the population.

it is also possible to have a contradiction between a Relative Risk confidence interval and the confidence intervals of its contributing counts

Perhaps, however, the confidence interval for deltaCMR is dependent wholly on the confidence interval of CMRpre and CMRpost. If the CI of deltaCMR dips below zero (allowed because it is not a ratio) then axiomatically that means that a chance exist that deltaCMR CMRpost. If that is true then a chance exist that RR

63

bad Jim 07.28.07 at 8:04 am

Common sense suggest that if no measurement in another cluster could even theoretically offset the Falluja cluster then the studies design would be flawed from the get go.

Indeed, it could be shown that no place on earth could ever have been struck by a meteorite or an asteroid because no event, or its absence, could theoretically offset it.

The dinosaurs live!

64

Kilo 07.28.07 at 8:38 am

No matter what inferences one draws from them, the studies themselves did not give any value judgements, and in particular didn’t attempt to say ‘who killed who’.

It’s all there in black and white, unless I’ve horribly misunderstood.
Posted by Suvi · July 27th, 2007 at 12:10 pm

Here’s an article where practically nothing but such attribution is stated… like 5 times in 6 paragraphs…
http://www.smh.com.au/articles/2004/10/29/1098992290312.html

65

SG 07.28.07 at 9:01 am

Shannon, love, I’ll grant you the ratio comment, but the substance of your “perhaps” is somewhat misleading. You can do the calculations with RR for simple cases to show that one can indeed have a negative deltaCMR, but a RR that contradicts this – or the opposite.

This is probably also the reason why David Kane’s proof in section 1.0.2 will give contradictory results depending on the starting point. In general, a proof which does that is not very binding.

66

dsquared 07.28.07 at 10:31 am

You assert that I assume X. You need to quote me assuming X.

No, I can also cite a paper you wrote that makes no sense whatsoever other than on the assumption of X.

67

dsquared 07.28.07 at 10:39 am

The study is so flawed that, even though death rates have obviously increased, the study cannot mathematically exclude the possibility that death rates have stayed the same or even decreased

Ey oop, it’s Shannon Love! Do you really think you had such a good time you tried to bluster statistics with me that you thought you’d do it again?

No. The study is not flawed in this regard, because it uses sensible, robust methods to determine the confidence interval. Kane’s methodology uses a silly, nonrobust method in which a positive outlier like Fallujah extends the lower end of the confidence interval symmetrically with the higher end. Kane then attributes this methodology to the paper in order to claim his “inconsistency”. He is now trying to backpedal and claim he never did this, but yes he did.

Are you sure you wish to claim this? This means the study is inherently biased to exaggerate deaths because a cluster can only increase but never decrease the death rate.

No, and I second John’s helpful advice that you think before engaging mouth in future. The addition of any cluster with a lower death rate than the previous average will lower the death rate. All clusters other than Fallujah bring the average down compared to Fallujah. However, no single cluster can lower the average by the same amount that Fallujah raises it because the death rate cannot be lower than zero.

If there were loads and loads of clusters with a postwar death rate of 1 or 2 per 100k, then it would be perfectly possible for this survey to report a fall in the death rate. There are not loads and loads of such clusters, so the study correctly reported a rise in the death rate.

68

David Kane 07.28.07 at 11:28 am

dsquared writes:

No, I can also cite a paper you wrote that makes no sense whatsoever other than on the assumption of X.

Perhaps we could invite John Quiggin or Kieran Healy to settle this dispute? Even though they are probably not fans of my work, I would trust their honesty on this.

You claim that I assume that the RR is not estimated with a bootstrap. Nowhere do I make this assumption. I only assume that the RR estimates and confidence interval reported by the paper are accurate. I use those numbers without regard to their origin. I challenge you to quote something from the paper to the contrary, but you can’t because there is nothing there. You now claim that the assumption must be there, albeit hidden from view behind some curtain, because the paper makes no sense without it. Fine. Demonstrate this claim! Show how the paper requires this assumption.

It doesn’t and so you can’t.

I apologize for calling you out on this, but, as your misunderstanding about the (lack of) use of the bootstrap in estimating the CMR demonstrates, you can make mistakes on this topic. Nothing wrong with mistakes, of course. But the honest thing to do is to correct them.

69

dsquared 07.28.07 at 11:41 am

I challenge you to quote something from the paper to the contrary, but you can’t because there is nothing there.

Your calculation on p6 and your table forming figure 2 do assume a normal distribution of the data including Fallujah. This is why you get different numbers for the excess deaths excluding Fallujah from those presented in the paper. Your charts all assume normal distributions – this is visible from simply looking at them. Your key conclusion is based on a chart which visibly has a chunk of its probability mass below zero. There is no correction to make here.

This is rather as if I had said “Kane believes that weasels are about to take over the world”, based on your having bought a weaselling gun, moved your home and possessions into a weasel-proof shelter and formed a committee called WEASELPAC to direct contributions to the Presidential candidate who promised to address the weasel problem, and then you challenged me to find a sentence where you had said “weasels are about to take over the world”. Your paper has no point to it at all without this assumption. You repeatedly claim that reported figures within the paper are “inconsistent” when they are only inconsistent on the basis of that assumption. You can ask as many times as you like but the answer is still going to be no.

By the way, on the Deltoid thread, halfway down, you have made the claim that Les Roberts carried out the estimation of excess deaths including Fallujah based on a normal distribution and then refused to print it because it didn’t reject the hypothesis. So this prissy attitude to your own precious paper is ill-placed.

70

Brett Bellmore 07.28.07 at 11:54 am

I’m back, and I have to get back to my initial point.

Yes, it’s true that the study itself does not make the mistake of moral attribution I’m talking about, but it IS made by just about everybody who seizes on the study as a club with which to attack the decision to invade Iraq. So it’s hardly irrelevant.

And I must insist that “You break it, you bought it.” can not apply to a situation where the damage is actually the result of decisions by other people, full blown moral agents in their own right, to go out and commit murders.

By this sort of reasoning, if a black couple moves into a white neighborhood, and the local KKK are so offended that they start blowing up daycare centers and shopping malls in an effort to bully the local populace into ejecting them, it’s the black couple who are at fault.

Nope, no way, this is invalid moral reasoning. And it’s the sort of moral reasoning that most people who care about the Lancet study are engaging in.

71

John Quiggin 07.28.07 at 12:17 pm

This is silly, Brett, and has been gone over many times before. Moral responsibility is not zero sum. If you start a war, with the foreseeable (and foreseen, and much derided and now, in this thread, denied) consequence that hundreds of thousands of people are killed, you’re responsible for those deaths. Equally, so are those who decide to fight you.

The practical consequence of your claim is that anyone who believes they have a just cause is entitled to go to war, and blameless for the consequence, since the resulting deaths will be the fault of the other side for unjustly resisting.

Leaving this aside, you have some responsibility to the truth. Instead of engaging in these apologetics won’t you at least have the decency to admit that the war has caused all these deaths and that those denying it, for whatever reason, are wrong to do so.

72

Steve K 07.28.07 at 1:28 pm

Brett, your comparison of an innocent black family to the illegal invasion of Iraq doesn’t work for a number of obvious reasons; I’m not going to waste my time with it. I will simply note that the particular “moral reasoning” you are advocating assumes an extreme form of free will that most intelligent people find ridiculous. You are really just wallowing around in a simplistic form of the compatibilist / noncompatibilist free will debate. In The View from Nowhere, Thomas Nagel shreds your argument to pieces, and even Christine Korsgaard who is a staunch defender of free will (see her Creating the Kingdom of Ends) steers clear of your particularly absurd form of it.

73

David Kane 07.28.07 at 1:28 pm

dsquared writes:

Your calculation on p6 and your table forming figure 2 do assume a normal distribution of the data including Fallujah.

False! (I assume you mean table 2 and not figure 2.) I am merely trying to replicate the (unreleased) formula by which the authors got 98,000 CI (8,000 — 194,000). I propose a formula. (If you know what the formula is, please tell us. The authors won’t tell me.) The formula has nothing to do with a normal distribution. It simply takes the CMR data given in the paper and plugs it in. Even if I did however, this formula and table do not have anything to do with RR. Nothing. RR does not enter in in any fashion. How can a formula and table which do not use RR demonstrate what assumption I am making about RR?

The authors used, as they clearly state, the CMR to calculate excess deaths, not the RR.

Your charts all assume normal distributions – this is visible from simply looking at them.

True. But this is correct! Consider figures 2 and 3. These are accurate descriptions of the CMR estimates. The authors have told us that they use the normal distributions for these. Those figures are spot on. Pick a specific one you object to and we can go throw it in detail. And, even if this were not true, figures 2 and 3 have nothing to do with RR. How are you able to deduce my assumptions about RR by looking at a figure about CMR, a figure which just describes graphically what the authors report?

Your weasel analogy is amusing, but if you can’t come up with a singe example that remotely illustrates your claim, then shouldn’t you retract it?

74

Brett Bellmore 07.28.07 at 1:39 pm

John, Steve, mathematics may represent the one realm where humans have access to ultimate truth, and can objectively demonstrate it. Ethics isn’t, it’s a realm where the very basics are fundamentally contested.

You expected, in the sense of prediction, that elements in Iraq would respond to our replacing Saddam’s tyranny with a democratically elected government by going on a killing spree. I expected, in a normative sense, that they would participate in elections. I maintain that, if you’re going to assess the moral aspects of an act on the basis of how people react to it, the normative sense of expectation is the relevant one.

And my analogy is spot on. We did good, and now the bad guys are trying to commit evil on a sufficiently vast scale to undo the good we did. This makes them monsters, not us, and the more they kill, the more monsterous they are.

75

Jennifer 07.28.07 at 1:45 pm

David Kane is, of course, a weasel and an especially immoral weasel at that, but carry on with the destructive idiocy David.

Brett Bellmore is simply a radical right trolling weasel, crazy as crazy can be.

What these weasels have in common is justifying our immoral war in Iraq and our more immoral occupation of Iraq.

76

Jennifer 07.28.07 at 1:47 pm

Brett Bellmore, as we can see in his insane last comment is simple a moral monster.

77

David Kane 07.28.07 at 1:48 pm

dsquared also wrote:

Your key conclusion is based on a chart which visibly has a chunk of its probability mass below zero.

This discussion would be more productive if you would specify which figure you are talking about. Figure 3? Again, there is nothing in this figure about RR. Nothing. I graph the two normal distributions that the paper provides and then the difference between them (specifying in the text that I am assuming zero covariance). Surely, you agree that the difference between two normal distributions is itself normal. Surely you agree that I have done the subtraction correctly.

While it is true that the resulting graph allows CMR to be negative, this is *exactly* what the authors allow in the original paper. They use the normal distribution. You may think that this is a critical flaw and that the paper may be ignored on that basis. I may think that this is OK and that other choices would produce similar answers.

But none of that is germane to the question of whether or not I assumed that the RR was not bootstrapped. I did not assume that. Please correct your claim to the contrary.

78

pete 07.28.07 at 1:49 pm

David,

Robert (who managed to replicate Roberts’ results) gave you a confidence interval for the RR including Fallujah in the Deltoid thread: (70 000, 975 000).

As you ought to expect, including Fallujah increases both the width and the lower bound of the CI.

I notice you are still referring to the CIs for the CMR estimates as simply “the CMR estimates”. This is one of the things d^2 complained of above.

The distinction is critical, because:

(a) the CMR estimates are relevant to the study’s results

(b) the CIs for the CMR estimates are wrong but are irrelevant to the study’s results.

I think you have successfully demonstrated (b); unfortunately that is not an important result.

79

John Emerson 07.28.07 at 2:12 pm

You’ve outdone yourself, Brett. I’m in awe. Events would have beaten some sense into a lesser man’s head by now.

“You will be expected to respond positively to our efforts to do you good. If you insist on acting like unruly schoolboys you will be soundly thrashed.”

What a sound basis for military planning!

80

soru 07.28.07 at 2:26 pm

Reading that thread on Deltoid, it does seem to be interesting the way that some people seem to have an absolute mental block on the idea that people could have died in Iraq before the official start date of the war. For them, the possibility of Saddam or his agents carrying out a massacre, shelling a village, or setting off a bomb, was zero, known to be zero, could not ever in any possible universe be non-zero.

In sarcasm, they even use the possibility of ‘resurrection’ as something that is inherently more likely than Saddam killing some Kurds.

Note: the discussion does get a bit complicated by fact that there is similar but unwrong argument that the post-war death rate cannot plausibly be a world record low, even if the detailed mathematics assigns a non-zero probability to a small clustered survey reporting that.

81

Steve K 07.28.07 at 2:27 pm

No Brett, we did not do “good”. We did a “war crime.” And if you’re naive enough to not understand nationalism I feel sorry for you. Anyway, here’s the death of your simplistic moral agent (sorry about the long quotes, but these ideas do kill one objection to the Lancet):

Thomas Nagel’s View from Nowhere (assume his “external view” corresponds to the viewpoint of the Lancet authors while “our natural human point of view” corresponds to our sense of ourselves as free moral agents): “In ordinary judgments of responsibility we do not go that far outside, but stay in our natural human point of view and project it into that of other, similar beings, stopping only where it will not fit. But judgments so based are vulnerable to the more external view, which can take in both the defendant and the judge. . . . The judge’s sense of the defendant’s alternatives is revealed as an illusion which derives from the judge’s projection of his own illusory—indeed unintelligible—sense of autonomy into the defendant.” (123). “[T]he external standpoint is always there as a possibility, and once having occupied it we can no longer regard our internal judgments of responsibility in the same way. From a point of view that is available to us, they can suddenly seem to depend on an illusion—a forgetting of the fact that we are just parts of the world and our lives just parts of its history” (124). This would suggest that an adequate moral theory must take into account that we are parts of the world and not merely actors within it.

In Creating the Kingdom of Ends, Korsgaard similarly stresses the difference between a holding another person responsible for their actions and that person in fact being responsible for their actions (which if Nagel is right, no one ultimately is). You hold people responsible when you are in a kind of personal relationship with them: “For instance, it may be perfectly reasonable for me to hold someone responsible for an attitude or an action, while at the same time acknowledging that it is just as reasonable for someone else not to hold the same person responsible for the very same attitude or action” (199). “In everyday personal interaction, we cannot get on without the concept of responsibility. And therefore we cannot rest with the view that agents take responsibility for their own actions but can refrain from judging others” (197). Brett, since I doubt that you have any “personal interaction” with Iraqis who are violently responding to the criminal destruction of their society around them, I wonder why you are trying to hold these Iraqis responsible for actions that stem (though not ultimately) from Bush’s illegal decision to invade Iraq.

82

Shannon Love 07.28.07 at 3:10 pm

kilo,

No matter what inferences one draws from them, the studies themselves did not give any value judgements, and in particular didn’t attempt to say ‘who killed who’.

From the 2004 Lancet paper:

Interpretation: Making conservative assumptions, we think that about 100000 excess deaths, or more have happened
since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths.

Looks like they are assigning blame to who killed who to me but as dsquared will tell you, I’m a little dense.

Those two sentences in combination are a lie by the way. No single data set supports that interpretation.

83

Cyrus Pinkerton 07.28.07 at 3:25 pm

Shannon Love writes:

Those two sentences in combination are a lie by the way. No single data set supports that interpretation.

Shannon, do you have evidence that proves the claim you cite to be incorrect? I’m assuming that since you assert that the claim is a “lie,” you have evidence to clearly establish that the claim is false. Will you reference this evidence, please?

84

dsquared 07.28.07 at 3:48 pm

The formula has nothing to do with a normal distribution. It simply takes the CMR data given in the paper and plugs it in.

This would only be a legitimate calculation if one was making a parametric assumption about the confidence intervals. You can’t just randomly “plug in” confidence intervals like this, unless you’re making a lot of assumptions.

85

Shannon Love 07.28.07 at 3:58 pm

dsquared,

The study is not flawed in this regard, because it uses sensible, robust methods to determine the confidence interval.

How can you tell? The authors never provide a confidence interval for the dataset including the Falluja cluster. They provide a CI for the data set excluding the Falluja cluster but not for the one including it. Kane is forced to infer the CI from the CI of rates of Relative Risk. If the authors had followed standard practice in the first place we wouldn’t be having this conversation.

However, no single cluster can lower the average by the same amount that Fallujah raises it because the death rate cannot be lower than zero..

No, because the study NEVER compares death rates between clusters. Instead it effectively compares changes in the death rate pre and post invasion. The delta can obviously be negative.

The entire notional point of the study was to determine the CHANGE in death rates. As such, any cluster that showed improvements in mortality post invasion could offset the Falluja cluster. In fact, if you look a fiq 1 (p 3) of the 2004 paper you will see that the province of Sulmamanhya shows a fairly pronounced improvement in mortality. It is easy to see visually that it could offset the increase of mortality in Falluja if it was larger.

If the study did actually average death RATEs between clusters then yours and Daniel’s argument that the distribution cannot be expected to be symmetrical would be true. However, since the study averages numbers, which can assume any value, we HAVE TO assume a normal distribution unless the authors say otherwise, which they do not. (see p3 pg4 – p4 pg1)

Don’t feel to bad. I made the same mistake the first time I read the paper. IIRC, it was you or Daniel that corrected me on it.

86

John Emerson 07.28.07 at 4:06 pm

Having won this battle, perhaps Shannon and David should shift their energies towards showing that electric production in Baghdad may be higher than before the war, rather than much lower as the lying media claim. I fear that they’re focussing their energies too narrowly on this particular question, while letting dozens of other media lies pass by unrefuted.

They might also take a look at the bogus claims that more than two million Iraqis have fled the country, and all the silly talk about dead bodies. It seems pretty likely that the dead bodies are just leftovers from the Saddam regime which are being trucked around to be discovered again and again and again.

87

Jim Harrison 07.28.07 at 4:39 pm

I note the absence of papers commenting on the possibility that the Lancet paper underestimates excess deaths in Iraq, just as in the tobacco/cancer controversy, the hired hands never complained about the conservatism of the assumptions made by statistical researchers who, like the authors of the Lancet study, had bent over backwards to hedge their results.

What we have here is another in the long series of asymmetrical arguments between people who believe that you ought to try to figure things out whatever conclusion you would prefer and those for whom the game is simply to fight fiercely for your ideology and/or employer regardless of the rules of evidence or, for that matter, ordinary morality. Critics of the Lancet study “knew” it was flawed long before they read it, just as tobacco publicists “knew” that the statistical arguments associating lung cancer and cigarettes were wrong and the Creationists and holocaust deniers—that is deniers of earlier holocausts–“knew” that the historians were wrong.

The point isn’t that people shouldn’t criticize the methodology of the Lancet study, but its most vocal critics aren’t fooling anybody. Which is why, even in this thread, some of them are playing the old game of “I didn’t steal the pig. It was my pig, and anyhow, it wasn’t very big.” Logically, of course, the conclusions of the Lancet study do not imply that the invasion of Iraq was a moral or political error. All those deaths may have been well worth it, at least from a sufficiently blooded minded point of view. And one can certainly imagine that a great many excess deaths may have resulted in the absence of an invasion by some unspecified mechanism. Unfortunately, these considerations are just ways of changing the subject.

88

Eli Rabett 07.28.07 at 4:39 pm

Oh Shannon, where are those spas in Iraq where the death rate has decreased so sharply? We can point to several places which were knocked flat like Fallujah where the excess mortality was extreme.

The other point is that while the survey was about EXCESS mortality, Kane’s idiocy comes up with a range of confidence intervals that includes NEGATIVE MORTALITY, probably the zombies rising from the dead and fighting for the central government and getting paid for it. Chicago Iraqi style.

89

dsquared 07.28.07 at 4:40 pm

The authors never provide a confidence interval for the dataset including the Falluja cluster

that is because it would not be sensible to do so, and they are sensible.

In fact, if you look a fiq 1 (p 3) of the 2004 paper you will see that the province of Sulmamanhya shows a fairly pronounced improvement in mortality. It is easy to see visually that it could offset the increase of mortality in Falluja if it was larger.

If you look at it closely, however, you would see that if the improvement in mortality was much larger, people would be coming back from the dead, which is kind of my point. Decreases in the death rate can only be so big, because they can’t be more than -100%. Increases in the death rate can be much more than +100%, and in Fallujah they were.

90

Stats 101 07.28.07 at 4:52 pm

Since normality is obviously a flawed assumption here can’t you just assume a log normal distribution?

91

John Emerson 07.28.07 at 5:17 pm

What gender is Shannon Love? There’s a comment I feel impelled to make, but I can’t formulate it properly without more information.

92

roger 07.28.07 at 5:29 pm

As I understand it – and the math is complicated – the point Love is making is based on an important distortion. The Lancet study assumed a death rate in pre-invasion Iraq based on, I believe, a UN estimate. Then they surveyed to see what the change in the death rate was. They discovered an upward slope. Love is claiming that they did not look for a downward slope. I guess this amounts to saying, are there more doctors in your area to which to take your kids? Now, there might be some plausibility in Love’s claim for the first couple months of the invasion, in 2003 – a period before the insurgency got going, when the sanction regime collapsed – but after that, his claim becomes increasingly absurd. The math is, after all, tied to real human causes. One might think, well, the violence fell in other areas of Iraq during the battle of Falluja because all of the insurgents streamed into that town, or something, but even a hasty perusal of the newspaper reporting in November and December 2004 would show the absurdity of that supposition. Plus, of course, the population expelled from Falluja – around 300,000 – had to find hasty quarters somewhere, since the U.S. didn’t provide any, and in that dispersal no doubt you could also find the excess deaths that come when, say, a seventy year old has to make a series of twenty mile marches to find food.

Love is rather distorting Kane’s work. As the thread on the Deltoid site shows, Kane himself is estimating about 100,000 violent deaths, and one can make a rule of thumb estimate that such deaths have to be multiplied to get an excess death total, for obvious reasons.

However, the reasons will never be obvious enough. While the pro-war people are pretty confident about the number killed by Saddam Hussein, and have never examined the numbers at all or how they were arrived at, they will never admit the bloody massacre unleashed by the invasion and the consequent decision of the CPA to liquidate security in the whole country, followed of course by an insurgency in which criminal gangs mixed with genuine political resistors, and punctuated by absurd punitive expeditions by the Americans, culminating in the war crime of Falluja.

I’m pretty confident, however, that if these same pro-war people lived in a community in which the mayor simply erased all security and turned a blind eye to crime, these pro-war people would BLAME THE MAYOR. While plainly lobotomized into perfect Bushbots when it comes to the suffering of other people, they are keenly aware of their own aches and pains.

93

David Kane 07.28.07 at 6:03 pm

dsquared claims

This would only be a legitimate calculation if one was making a parametric assumption about the confidence intervals. You can’t just randomly “plug in” confidence intervals like this, unless you’re making a lot of assumptions.

This is just false. Let x have probability distribution p(x). If I tell you *nothing* about p(x) except that the 2.5th percentile is 1.4, you can still manipulate that number linearly (which is all that I do). In other words, the 2.5th percentile of 2*p(x) is 2.8. The 2.5th percentile of 10*p(x) + 3 is 17. I need assume nothing about p(x) to derive linear transformations of specific percentiles.

I make no assumptions. And, even if I did, I am making no assumptions about the RR. Nothing in that calculation mentions RR. You claim that I assume that the RR was not estimated via bootstrap. Nowhere do I assume that. Care to try again? I recommend just correcting the original post. It’s OK, we all make mistakes.

94

Shannon Love 07.28.07 at 6:05 pm

John Emerson,

You appear to argue that because you know what is right, the actual accuracy, if not honesty of the study, does not matter? You only care that the study appears to confirm you preconceptions?

I don’t suppose it has occurred to you that wars come and go but that our scientific institutions form a major part of the core of our civilization. What happens when they become so corrupted by politics that they can no longer function? If people begin to widely expect that researcher’s political bias exert more influence on their conclusions then they will lose confidence in the institution. Then we will be back to settling disputes with theology.

95

Ragout 07.28.07 at 6:15 pm

Soru *In sarcasm, [commenters on Deltoid] even use the possibility of ‘resurrection’ as something that is inherently more likely than Saddam killing some Kurds.*

This isn’t true. The Lancet paper allows for the possibility death rates went down, and none of David’s critics deny that this is theoretically possible. When people are talking about ‘resurrection,’ they are saying that death rates can’t be negative, not that excess deaths can’t be negative. Believe it or not, the problem with David Kane’s calculations is that they allow death rates to be negative. Again, negative death rates aren’t just unlikely, they’re impossible.

96

dsquared 07.28.07 at 6:20 pm

If I tell you nothing about p(x) except that the 2.5th percentile is 1.4, you can still manipulate that number linearly (which is all that I do). In other words, the 2.5th percentile of 2*p(x) is 2.8. The 2.5th percentile of 10*p(x) + 3 is 17. I need assume nothing about p(x) to derive linear transformations of specific percentiles.

Multiplying, adding and subtracting other random variables is *not* a “linear transformation”. You think you are carrying out a linear transformation because your calculation on p6 only has the scalar value 5.0 in it. However, this value 5.0 is the estimate of the pre war mortality rate which has its own sampling distribution. Michael Spagat has also made this mistake when trying to scale the Roberts et al (2004) estimates to match the IBC data.

97

John Emerson 07.28.07 at 6:24 pm

Shannon, I’ll leave it to others to deal with your supposed contribution to the protection of our scientific institutions. The people competent to do so seem completely unimpressed with your work in that respect too, but I’m not able to participate in that argument.

My guess is that you have your own preconceptions, biases, motives, and theology and that your claim to scientific purity is bogus. To me your contribution seems pretty similar to all the other winger attempts to obscure the gross facts about Iraq behind a gnat swarm of quibbles and aspersions of various kinds — some sophisticated, some crude, some slanderous, some trivial, and many simply wrong. By so doing, and by stubbornly repeating discredited arguments, wingers often succeed in casting doubt on valid information. There’s a repeated pattern.

If your criticisms are bogus, as most here claim that they are, their sophistication makes them the more culpable, since very few non-professionals are able to clearly understand either the criticisms or the defenses of the study, making it tempting for laymen to believe that both sides have valid points and no one can really be sure who is right. The effect, possibly the intended effect, will be to neutralize the effect of a carefully-done scientific study.

“Chicago Boyz” is a strongly ideological site and no one coming from there can be taken seriously if they try to claim the high scientific ground.

98

John Emerson 07.28.07 at 6:31 pm

As I have said several times in the past, the error of the “Lancet” authors was to think that carefully-done scientific work would survive a trip into the ideological swamp of American politics. It was inevitable that someone with bit of statistical sophistication would get to work on discrediting the study, and each of the little tweaks that the Lancet authors put into their study to make it more robust and unassailable provided an opportunity for a denialist to make a sophisticated-seeming criticism.

99

Shannon Love 07.28.07 at 6:33 pm

dsquared,

If you look at it closely, however, you would see that if the improvement in mortality was much larger, people would be coming back from the dead, which is kind of my point. Decreases in the death rate can only be so big, because they can’t be more than -100%. Increases in the death rate can be much more than +100%, and in Fallujah they were.

For the last time, we are not talking about death RATES but rather about the difference between death rates in two separate time frames. Each cluster produces two death rates one for before the invasion and one for after.
These two death rates are never averaged in with the rates from any other cluster! Instead, the differences are used to calculate whether post invasion, the mortality, improved, stayed the same, or increased.

Example: Cluster A. death rate prior to invasion=10/1000. Death rate after invasion=100/1000. Difference = 100/1000-10/1000=90. CHANGE in death rate=+90/1000.

Cluster B. death rate prior to invasion=100/1000. Death rate after invasion=100/1000. Difference = 10/1000-100/1000=(-)90. CHANGE in death rate=(-)90/1000.

In cluster A, 90 people MOREper 1000 die that would not have prior to the invasion. In cluster B, 90 peopleFEWER die than would have after the invasion.

Now using some very sophisticated math (90/1000+(-)90/1000) we can show that every excess death in cluster A is offset by an extra saved live in cluster B. In a population composed of cluster A and cluster B, the combined change in the death rate would be zero.

You keep saying that the paper compares death rates between clusters but it actually compares changes in death rates. The changes can be negative, zero or positive and thus CAN cancel each other out In a random sample, clusters are mathematically as likely to offset another cluster as to not. Therefore, there is no reason to suspect that Lancet calculation require an asymmetrical uncertainty distribution.

Fallujah doesn’t wreck the confidence interval because it can never be offset. It wrecks the confidence level because it is a massive outlier created by poor study design.

100

Ragout 07.28.07 at 6:38 pm

Eli Rabbet asks, where are those spas in Iraq where the death rate has decreased so sharply?

They’re in Kurdistan, Syria, Iran, and Jordan. The people Saddam was targeting before the war were Kurds, Chaldeans, Turkomen, and other minorities. It was called the “Arabization Campaign.” Before the war, about 1 million of them were internal refugees, and another million or so were refugees in neighboring countries.

In general, the Lancet studies exclude or undercover somewhere between 2 to 6 million Iraqis who were most oppressed by Saddam. The Lancet studies excluded refugees in neighboring countries, probably undercovered refugee camps in Kurdistan and the rest of Iraq, and used only Arabic-speaking interviewers but not speakers or any of the languages spoken by Saddam’s favorite victims.

Since the invasion, it’s Iraq’s Shiite and Sunni majority that’s suffering, so no doubt the death toll is higher than in the days when Saddam was killing and expropriating minority groups. But don’t tell me that Saddam’s body count has been carefully tallied, because it hasn’t.

101

John Emerson 07.28.07 at 6:43 pm

To me it’s boggling to have someone from “Chicago Boyz” tell me that he’s primarily concerned with our scientific instutitions, whereas epidemiologists from The Lancet and Johns Hopkins are political hacks. This does not pass the smell test.

102

John Emerson 07.28.07 at 6:45 pm

Kurdistan was protected from Saddam well before the invasion, and the Kurds are the largest single minority. Driving even more Iraqis into exile than Saddam did doesn’t sound like a positive accomplishment, even if they’re safe there.

103

dsquared 07.28.07 at 6:59 pm

Shannon, you’re wretchedly confused here. The study does not publish 33 different excess death rates. It publishes one, for the country of Iraq, which aggregates all the clusters together.

In cluster B, 90 peopleFEWER die than would have after the invasion

Since the average death rate in these clusters is about 5, no such cluster could exist. This is the whole point.

Btw, if you are going to continue to blah on about “our precious scientific institutions” and “intellectual dishonesty”, then I may become irritated enough to dig up some of your own greatest hits, and this will not reflect well on you or your website.

104

Shannon Love 07.28.07 at 7:00 pm

john emerson,

“Chicago Boyz” is a strongly ideological site and no one coming from there can be taken seriously if they try to claim the high scientific ground.

Its a group blog so there is a lot of diversity. However, I do not insist that I have no bias. I have a lot. However, I have never asserted that anyone should accept what I say because I am so damn smart and good looking.

The neat thing about science though is that individual biases don’t matter. The entire social function of science is that people from all different perspectives can arrive at the same conclusion. Its not easy and its not perfect but over the long run it works.

Science depends on honesty and disclosure so that other can evaluate and repeats the original work. By iterating this process we can create progressively more accurate models of the world around us.

The Lancet authors broke the rules. They hid and obscured data. Most damningly, they cherry picked data to create just the result they wanted. By there own admission, they rushed the study and its publication in order to influence the 2004 US Presidential election. They are trying to use their status as trusted scientist to accomplish narrow political ends.

Don’t take my word for it. Get a PDF of the study yourself and go through it thoroughly. Pay close attention to which assertions in the study such a excess deaths, cause of death, age and sex killed etc come from which data. Once you do this the cherry picking becomes obvious.

Then ask yourself whether you would like a drug company to cherry pick the data they use to prove their products safety and effectiveness.

A loss of scientific integrity might benefit you today but down the road it will come back to bite you.

105

Cyrus Pinkerton 07.28.07 at 7:01 pm

Shannon Love writes:

I don’t suppose it has occurred to you that wars come and go but that our scientific institutions form a major part of the core of our civilization. What happens when they become so corrupted by politics that they can no longer function? If people begin to widely expect that researcher’s political bias exert more influence on their conclusions then they will lose confidence in the institution. Then we will be back to settling disputes with theology.

Shannon Love, assuming you believe what you’ve written here, will you please address the point I raised in comment 85? You’ve accused Roberts et al of lying. Since your statement above assures us that you make this accusation based on clear scientific evidence rather than political considerations, will you please cite this evidence now?

106

dsquared 07.28.07 at 7:11 pm

The Lancet authors broke the rules. They hid and obscured data. Most damningly, they cherry picked data to create just the result they wanted. By there own admission, they rushed the study and its publication in order to influence the 2004 US Presidential election. They are trying to use their status as trusted scientist to accomplish narrow political ends.

As I say, back in 2004 I was disgusted with this sort of behaviour but now I’m just amused. Back in 2004 it seemed to matter what people like Shannon Love said, but who listens to the crazy 26% these days?

107

John Emerson 07.28.07 at 7:21 pm

Since I don’t know statistics and don’t work in statistics, I’m willing to let the questions about scientific method come out in the wash over the next few years.

The reason someone like me is hearing about statistical problems in the Lancet study isn’t because people in science are concerned about its methodological deficiencies. It’s because the topic of the study is highly controversial and the study’s conclusions are objectionable to the winger hack battalion.

If the study is bad, there could be two bad outcomes: people might come to substantially false conclusions about the Iraq War, and a bad professional precedent might be set which would be followed by others. The latter is not something I can involve myself in, but it seems highly, highly unlikely.

The former I understand better, and to me it seems impossible. I have other sources about the Iraq War, and the conclusions I come to about the war without the Lancet study are about the same as the ones I would come to with it. The small adjustments that may or may not be mandated by the criticisms of the study seem unlikely to change anything substantial, though I admit I would simply reject out of hand any tweak of the study showing that the invasion had improved Iraq’s mortality rate.

Your own accusations against the study’s authors make it impossible for me to come away from the controversy without thinking that someone is in bad faith. It seems most likely to me that the one in bad faith is Shannon Love.

108

Cyrus Pinkerton 07.28.07 at 7:35 pm

Shannon Love writes:

By there own admission, they rushed the study and its publication in order to influence the 2004 US Presidential election.

Actually the authors have stated the opposite of this. If you have a quote in which they admit they tried to influence the 2004 US Presidential election, will you provide it please? Otherwise I will be left to conclude that once again, you’ve made an assertion that has no basis in fact.

109

Shannon Love 07.28.07 at 7:37 pm

dsquared,

The study does not publish 33 different excess death rates. It publishes one, for the country of Iraq, which aggregates all the clusters together.

The study actually publishes the chance of increased mortality but that is neither here nor there. Perhaps you could enlighten me on how you think they did calculate it?

Since the average death rate in these clusters is about 5, no such cluster could exist. This is the whole point.

Sigh. So if the day before the invasion Saddam launched an attack on a Shia town and inflicted exactly the same number of death in the same size population as in Fallujah then those deaths would not have offset the deaths occurred in Fallujah post-invasion?

If Saddam had gassed Halabja in the study period prior to the invasion wouldn’t the 5,000 people who died there offset Fallujah by just a bit?

Try another way, suppose we do abandon Iraq and freed from cruel American tyranny the Iraqi immediately stop fighting, eliminate crime and improve their health care system. Suppose Les Roberts goes back to duplicate his study. Now I would naively expect him to find that a significant drop in mortality post-abandonment would offset the pre-abandonment mortality. You however, seem to think that the study could never show that the study saved lives because their is no way a post decrease in mortality could offset the pre levels because the post rate could never be less than zero.

I am boggled that you cannot grasp this. If one cluster has a preCMR exactly equivalent to postCMR of another cluster and they have equivalent populations, then they will exactly offset each other in the calculation of national change in mortality rate.

110

J— 07.28.07 at 7:39 pm

Re: #105

I’ll kick off the hit parade, if you don’t mind:

If you want to examine how politics distorts science, look at the Left first. Virtually, all the bogus science panics of that 40 years have been pushed by the Left. I think that environmental and technological threats serve the same purpose for the political Left that wars once did. They are used to justify increasing the power of the state. AGW [anthropogenic global warming] is the perfect “permanent emergency” that can be used to justify state control of the economy for decades. This creates a powerful incentive to distort the publics perception of the actual science. So, I will restate my original claim. Given the real-world track record of the last 40 years, political interference in the science of global warming is more likely to produce an exaggeration of the threat than an underestimation.

Shannon Love, 2/27/2006

111

soru 07.28.07 at 8:04 pm

1: This isn’t true. The Lancet paper allows for the possibility death rates went down, and none of David’s critics deny that this is theoretically possible. When people are talking about ‘resurrection,’ they are saying that death rates can’t be negative, not that excess deaths can’t be negative. Believe it or not, the problem with David Kane’s calculations is that they allow death rates to be negative. Again, negative death rates aren’t just unlikely, they’re impossible.

As I said in the original post, _some_ critics of Kane, to the extent that I follow the maths, seem to have a valid point.

Others, however:

2: Since the average death rate in these clusters is about 5, no such cluster could exist. This is the whole point

An unsampled cluster with a pre-war death rate of greater than 5 does not seem to be precluded by any law of mathematics I know of, so unless I am missing something, that statement by dsquared seems to be either a mathematical error, or a political judgement about the relative benevolence of the last days of saddam’s regime in comparison to the early days of the occupation.

He’s hardly alone in making that kind of comment.

112

dsquared 07.28.07 at 8:15 pm

Sigh. So if the day before the invasion Saddam launched an attack on a Shia town and inflicted exactly the same number of death in the same size population as in Fallujah then those deaths would not have offset the deaths occurred in Fallujah post-invasion?

Sigh. If my Aunt Sally had a penis and testicles, then indeed I would have four uncles.

113

Shannon Love 07.28.07 at 8:22 pm

john emerson,

Your own accusations against the study’s authors make it impossible for me to come away from the controversy without thinking that someone is in bad faith. It seems most likely to me that the one in bad faith is Shannon Love.

So I must be wrong because authority figures you trust haven’t come to the same conclusion as me? Have you considered the possibility that they are not paying attention, are afraid of controversy or, worse, have been co-opted. After the idea that a little creative exaggeration in the service of a greater good is very popular in academic circles now.

In the end, the only person you can rely on is yourself.

Let me help you out: Suppose a drug company did a study of its drug using 33 different hospitals as their clusters. Suppose the main finding of the study said something like:

70% of patients showed improvements. Significant side effects occurred in only 10% of patients.

Would you be comfortable taking that drug if you knew that the rate of improvement and the rate of side effects came from two different sets of data? Would you be upset if no SINGLE data set contained both figures? Wouldn’t you want to know what the improvement and side effect rates were for each data set? Would you be upset if you discovered that the drug company cherry picked the results it wanted from each data set just to make their drug look good?

That is precisely what the authors of the Lancet paper did.

The main finding of the paper, the one that states their most accurate findings and the one that got turned into a sound bite everywhere said:

Making conservative assumptions, we think that about 100000 excess deaths, or more have happened
since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces
accounted for most violent deaths.

Now, would it disturb you to know that those two sentences are a flat out lie. Each is supported by one of two data sets in the study but there is no data set in which both sentences are true. Worse, there is no technical reason why one set of data was used in the first sentence and another in the second. The only explanation is that the author’s cherry picked the data in order to create just the story they wanted.

I believe the authors cherry picked the excess deaths from the first data set because the second data sets excess deaths (300,000) was implausibly high and they didn’t want anyone to know that it was based on a single neighborhood. They cherry picked the cause of death in the next sentence from the second data because only that data set (based almost entirely on one neighborhood) showed the majority of deaths resulted from Coalition action.

Do you understand now why I am so upset? The actual underlying study isn’t that bad. Once you toss out the Fallujah neighborhood the study’s result compare with those produced by other methods. The authors, however, intentionally mis-reported there results in order to create a propaganda tool.

Like I said, check for yourself. You don’t need to understand statistics.

114

roger 07.28.07 at 8:35 pm

Soru, the cluster is assumed by Kane’s study, or at least he has never said a word against it on the Deltoid thread. Kane’s study is the basis of Love’s claim. And, as I remember the Lancet study, the mortality basis, pre-war, was based on UN estimates that had been, until that time, claimed to be too high by conservatives supporting the sanction regime.

Daniel nails the heart of Love’s incoherence there. Yes, you could theoretically sink a 5 per thousand to zero for a while, but you couldn’t sink it below that. No blustering will get you across that mortal threshhold.

Blustering, of course, there will be. But I think Daniel nails that too. This argument has been won. Kane’s paper concedes a much higher excess death toll than is conceded by, say, Bush. To add on a dispute about the numbers of the cluster is to pretend that Kane’s paper does something it doesn’t pretend to do, but this is just what one has come to expect from the legions of the warmongers. The pro-war side has fallen apart because, on the one hand, Iraq has fallen apart, four years after the American occupation that was supposed to go up and up, like, uh, Japan and Germany, and on the other hand, the warmongers have consistently lied, exaggerated, slandered, slimed, contradicted themselves, and in every way shown that they are moved less by the plight of the struggling Iraqis towards liberty, fraternity, and oil contracts with Exxon, and more by simple dittoheadism. Refusing to admit today’s reality, they busily work on creating an illusion out of yesterday. That will convince the convinced. Nobody else.

115

Cyrus Pinkerton 07.28.07 at 8:44 pm

Shannon Love writes:

The authors, however, intentionally mis-reported there results in order to create a propaganda tool.

Shannon Love, once again you’ve made an irresponsible accusation for which you have no evidence. It’s tedious to watch you try to smear these scientists while claiming to have respect for the scientific process. If you want to be a hypocritical partisan hack, fine, but can you find a different forum, please?

Also, assuming that you think David Kane’s work is respectable, may I suggest you withdraw from the discussion of it? You are certainly doing him no favors with your contributions here.

BTW, at this point I’ve given up any hope that you will provide evidence to support any of your wild accusations. Nor do I anticipate that you will withdraw them. Rather, I suspect you will repeat the same irresponsible charges ad nauseam in the hope that someone will accept them.

116

John Quiggin 07.28.07 at 8:59 pm

As Daniel says, there’s no point in trying to convince the 26-per-centers. The only thing needful at this point is to keep good records against the day when anyone suggests believing anything these guys say in the future.

117

Drm 07.28.07 at 9:03 pm

Sturtevant once quipped that elegant science was a paper that reached conclusions he liked using methods he didn’t understand.

Hence, as someone who has vehemently opposed the Iraq war from the outset, I (and many other CT readers apparently) find a degree of elegance in the Lancet study. However, reading the paper also leaves an impression that while the statistical analysis may be technically valid, the estimate of the change in death rate may not be very robust. I.e. are there other equally valid approaches to the same dataset that would give a substantially different estimate? I can’t say, but at best they had to work very hard to get an estimate of borderline significance from very noisey data.

The one conclusion that everyone in the present discussion seems to agree on is that the variance of death rates increased dramatically post-invasion with or without Faluja. Common sense would indicate that is not due to positive changes on the ground. In that light, the whole CI question seems like a red herring that the Lancet authors might have avoided. I.e. they might have said that our data indicate that its bad, by one estimate at least its very very bad, we need more data to determine just how bad it really is. I don’t suppose that would have gotten the same press.

118

Shannon Love 07.28.07 at 9:33 pm

Cyrus Pinkerton,

Shannon Love, once again you’ve made an irresponsible accusation for which you have no evidence.

Here

Start at the bottom and work upward. You can probably get through them all in an afternoon if you concentrate.

or

you could download the PDF of the study and examine it yourself. The only thing you need to pay attention to is which statements, especially in the Interpretation are only true based the set of 33 clusters including Fallajuh and which statements are only true based on the set of 32 clusters excluding Falujah.

If you need help, try my Frisking Falluja post.

I may be wrong but at least I spent the time. Honestly don’t any of you people check anything for your selves?

119

Jacob (2) 07.28.07 at 10:15 pm

I don’t suppose that would have gotten the same press.

Yes, that’s pity.

Having been against the Iraq invasion from the start, is it possible, at once, to believe that the war was misguided, devious and has had disastrous consequences and that the 2004 Lancet study was incomplete, a work in which definition and disinterested science seem, in part, to have taken a back seat to political expediency? If so, count me in.

120

Shannon Love 07.28.07 at 10:17 pm

roger,

Daniel nails the heart of Love’s incoherence there. Yes, you could theoretically sink a 5 per thousand to zero for a while, but you couldn’t sink it below that. No blustering will get you across that mortal threshhold.

Oh good grief. I am seriously worried about the basic numeracy of you people.

I think the problem is that people look at the pretty pictures in fig 2 and fig 3, see that curves for CMRpre and CMRpost go past zero without ever looking at the text where Kane explains that plots are drawn inaccurately because, of course, CMRpre and CMRpost cannot be less than zero.

Kane’s argument is based entirely around the idea that deltaCMR can be negative and I as demonstrated with real world thought experiments above, it is realistically possible. The actual study itself shows a cluster with negative deltaCMR.

If the CMRpre anywhere in the country could never, ever, even theoretically off set any CMRpost then the study would have a profound built-in bias to exaggerate deaths. If the offset was only practically impossible it would mean the study’s design was fatally flawed.

Kane might be wrong. He has to fudge a lot numbers because so much of the original study remains secret.

121

Cyrus Pinkerton 07.28.07 at 10:28 pm

Shannon Love,

Apparently you don’t understand the nature of the error you’ve made since you continue to repeat it in defense of your earlier errors. Let me explain it to you so that you won’t continue down this road.

You have made assertions for which you apparently have NO credible evidence. I’ve asked you repeatedly for evidence and now, finally, in response you provide this:

a) a link to archive articles at “Chicago Boyz,” and
b) an old commentary of yours which repeats some of the accusations you’ve made here, again without evidence to support them.

This is not an acceptable response; it’s the equivalent of saying “find the evidence yourself.” You don’t seem to understand, though, that I’m calling your bluff. It’s your responsibility, not mine, to cite the evidence on which you base your charges of scientific misconduct.
Let me use as an example one of your irresponsible accusations in order to make it clear to you what your burden of proof requires.

In one instance you claim that the authors admitted to trying to influence the 2004 US Presidential election with the publication of their research. In order to support this claim, you need to provide a source for this admission. This should be straightforward. Simply cite the source by which you learned of this admission. What could be simpler?

Unfortunately, the fact that you are ignoring/evading/filibustering regarding this charge leads me to believe you are not commenting in good faith here. If you’ve made an error in your accusation, acknowledge your mistake and withdraw the charge. If you stand by your claim, support it with evidence. I will accept either as an honest response. On the other hand, continuing as you have been suggests nothing but dishonesty.

Do you finally understand that the burden of proof is on you to support your accusations of scientific misconduct? Do you understand that making such accusations without credible evidence is irresponsible?

I’d like to see some of the scientific integrity that you claim to admire in your response to this. As things now stand, you’ve discredited yourself with your performance.

122

roger 07.28.07 at 10:33 pm

Shannon, your real world thought experiment is irrelevant. There is a limit to mortality in a population and that’s all there is to it. However, on the deltoid thread, there is an excellent explanation of what is going on that is not as fanciful, as counterintuitive,and as bizarre as your parabl by Robert. Here’s the quote :

“Here’s what I hope is a simple explanation. Suppose you are counting the average occupancy of cars on a freeway. You look at many cars as they pass and count the occupants and divide by the number of cars. You happen to do this on a weekday during rush hour. You can get the average number of occupants and the variance around that average, from which you can make an estimate of the average occupancy of all cars. No problemo. You repeat that experiment on a weekend day, and there are more families in cars so the average occupancy happens to go up. However, just by chance, a bus comes by filled with weekend tourists.

The bus is Falluja. The Roberts team made two estimates, one including the bus, and one excluding the bus, and concentrated on the one excluding the bus; then concluded that even if you exclude the bus the average occupancy on weekends went up. David Kane argues that, according to a model that treats the bus as if it were a car you find two things: 1) the average occupancy on weekends goes up, but 2) the variance goes up so fast that you can no longer exclude the possibility that the average occupancy during weekends went down even though all of your observations went up. In fact, David Kane’s model is so weird that it does not exclude the possibility that the average occupancy of all weekend cars is negative.”

123

Cyrus Pinkerton 07.28.07 at 10:38 pm

The author who writes this (in reaction to the David Kane paper):

Ah, vindication is sweet

also writes this:

Kane might be wrong. He has to fudge a lot numbers because so much of the original study remains secret.

Does Shannon Love really expect to be taken seriously?

124

Jennifer 07.28.07 at 10:39 pm

Shannon Love is another horrid liar, but lie on she will. A line of 2 of the wretch and you can tell who Shannon is. Just remember and move on.

David Kane, you are a shameless disgrace to Harvard.

125

Jennifer 07.28.07 at 10:41 pm

Shannon and David Kand are not interested in errors, they are crazed frauds and no more. Sigh. Duh.

126

Rich Puchalsky 07.28.07 at 10:47 pm

“The one conclusion that everyone in the present discussion seems to agree on […]”

drm, you’re really going to have to change that way of thinking. You have to get it through your head that there are people (e.g. shannon love) who participate in these discussions only so that there will not be perceived agreement, trusting that most people will be unwilling or unable to track back their errors.

If there was an Internet discussion about whether the Earth was flat, you wouldn’t write “Well, whether it’s flat or roughly spherical, at least the one conclusion that everyone in the present discussion seems to agree on is that it’s in some way round.”

jacob (2): “the 2004 Lancet study was incomplete, a work in which definition and disinterested science seem, in part, to have taken a back seat to political expediency”

Evidence? Any beyond the fact that there are wingnuts doing their best to fog a straightforward paper?

127

Quo Vadis 07.29.07 at 12:03 am

I don’t think that one can blame David Kane for his criticisms of the Lancet study. I think the problem is that the authors of the Lancet study broke some rules that in some areas of research would not be broken. Essentially they discarded some of their data in order to create a sample about which they could make statistically significant claims. Whether or not this invalidates the study would depend upon what the goals of the study were and how those goals guided their original sample design. In my experience with research, studies are designed as inputs to specific decision making processes which provide a solid foundation about which to evaluate goals and sample design. It is not clear to me that the Lancet study had such a specific purpose, but then one wonders what motivated the authors to attempt statistical analysis of the data and draw conclusions from it.

In my opinion, the real problem with the Lancet study is that after having discarded some of the data to make the statistical claims, they re-introduce the data elsewhere in their conclusions. This is guaranteed to produce exactly the sort of criticisms that persons like David Kane are making.

128

Drm 07.29.07 at 12:22 am

rich puchalsky,
I’m sure you are right that a more conservative approach in the Lancet paper would not have satisfied the polemicists. All I can say is that I would have been.

129

John Emerson 07.29.07 at 12:48 am

So I must be wrong because authority figures you trust haven’t come to the same conclusion as me?

What I said is that someone must be dishonest, based on what you said. Based on what I know, most likely you are the dishonest one. Trusting authority figures is the method I use in areas I don’t understand well, such as quantum physics, relativity, and global warming. It’s really the only method most people can use in such cases, and it’s a valid method.

Have you considered the possibility that they are not paying attention, are afraid of controversy or, worse, have been co-opted? After all, the idea that a little creative exaggeration in the service of a greater good is very popular in academic circles now.

In the end, the only person you can rely on is yourself.

The authority of myself tells me that you are the liar, and not the CT people or the Lancet authors. People on your side of the fence are mor often and more egregiously guilty of the stuff you talk about. Nice try, though.

130

Walt 07.29.07 at 1:52 am

quo vadis: That’s simply wrong. The Fallujah data is an obvious and gigantic outlier. Any rule to identify outliers would identify Fallujah. Any research looking at that data would do what they did: report results with and without Fallujah.

131

Quo Vadis 07.29.07 at 2:18 am

@123

The first thing anyone who had designed a research program that included the collection of primary data would do upon finding an “obvious and gigantic outlier” in their primary data is to review their assumptions, design and process and find out why they failed to anticipate the outlier.

If they had anticipated the outlier and had intended to exclude it from the statistical analysis from the start, then they intensionally collected two distinctly different sets of data for two distinctly different purposes. If they then conduct statistical analysis of one set and apply conclusions from that analysis to the other set, they expose themselves to exactly the type of criticism that David Kane if making.

132

Quo Vadis 07.29.07 at 2:42 am

@133

Oops, my previous comment was in reference to 132 not 123.

133

John Emerson 07.29.07 at 3:16 am

No explanation is needed for any failure to anticipate an outlier of the Fallujah type. Outliers happen all the time.

If there had been attempt to come up with a predictive general theory of Iraqi history, outliers would be a massive problem. If you’re just trying to describe and analyze a specific span of time, you just recognize that there’s a discontinuity in the historical data, which is a common case that doesn’t need explanation unless you’re trying to prove uniformity in the data.

134

Walt 07.29.07 at 3:34 am

Quo vadis, what the Lancet people did is completely normal. I’m not sure what sort of background you have, so maybe you are generalizing from your own experience, but in most data work there are outliers (frequently unexpected) that you have to deal with. There are a host of techniques to deal with them, which go under the generic name of “robust methods”. Here, there’s just one outlier, and it’s a gigantic one, so they report their stats both with and without the outlier.

135

Walt 07.29.07 at 3:50 am

On the off change that there’s still anyone reading this thread:

There is nothing unusual about the statistical analyses in the Lancet paper. Nothing. It is a completely ordinary example of the use of statistics to analyze data. They discover — surprise, surprise — that the death rate in Fallujah is much higher than anywhere else in their sample. They compute the average death rates pre- and post-invasion, both with and without Fallujah. If they were really trying to fool everyone, they could have thrown out the Fallujah data, and never reported it. Then (and this is their main finding), they use a more sophisticated technique to verify that the ratio of the post- and pre-invasion death rates is bigger than one.

This is how it’s done. You analyze your data in different ways, you report the results of different analyses, and then you draw a conclusion. Other people who read the paper are free to interpret their results differently, and draw a different conclusion. Science goes on.

136

Robert 07.29.07 at 5:49 am

In #119, drm asks:

However, reading the paper also leaves an impression that while the statistical analysis may be technically valid, the estimate of the change in death rate may not be very robust. I.e. are there other equally valid approaches to the same dataset that would give a substantially different estimate?

Haven’t found one. However, equally valid approaches to the same dataset using none of the assumptions used by David Kane give substantially the same estimate.

137

Quo Vadis 07.29.07 at 5:55 am

@136, 137

Walt,

I’m not sure we disagree entirely on my main point, which is that the Lancet study is a legitimate subject for criticism or at least that it is not the last word on the data it covered.

To the matter of research process: I come at this from the marketing domain, so perhaps there are different standards when there’s no money on the line, but one doesn’t start and end by collecting a lot of data and playing with it in the hope that something significant will emerge. You start with the questions you are trying to answer and express them in such a way that they can be answered using specific analysis techniques on specific sets of data. You then design a data collection process that will give you exactly the data you need to provide a statistically significant result. If after analysis, you can’t come to a statistically significant conclusion you can only conclude that there is no statistically significant relationship between the data collected and the question you were trying to answer.

At this point you can go back over your process to see if you have made any mistakes, incorrect assumptions or left anything out. Outliers in the data are good places to look for clues about where you have gone wrong. A legitimate outlier (not attributable to your process) that is large enough to invalidate your results is an indication that there is something very important about the phenomenon you are studying that you do not understand and/or have not accounted for in your research and conclusions. If you simply handwave such things, the quality of your research and veracity of your conclusions may be called into question. In some domains, that something you handwave could be the difference between an iPod and an MP3 player.

138

Robert 07.29.07 at 7:18 am

In 139, quo vadis wrote:

You start with the questions you are trying to answer and express them in such a way that they can be answered using specific analysis techniques on specific sets of data. You then design a data collection process that will give you exactly the data you need to provide a statistically significant result. If after analysis, you can’t come to a statistically significant conclusion you can only conclude that there is no statistically significant relationship between the data collected and the question you were trying to answer.

I believe this is what they did.

At this point you can go back over your process to see if you have made any mistakes, incorrect assumptions or left anything out. Outliers in the data are good places to look for clues about where you have gone wrong. A legitimate outlier (not attributable to your process) that is large enough to invalidate your results

Perhaps so, but your argument is moot. The Falluja cluster does not invalidate the Roberts results. David Kane and Shannon Love appear to want it to, but it does not.

139

nick s 07.29.07 at 8:19 am

A hypothetical:

Suppose that Roberts et al. (2004) includes the stat-munging that David Kane has now served up after three years, after a circuitous route involving borderline libellous accusations towards Roberts, his co-authors and his survey team? How do Kane and Love et al. react to the use of that model? Do they say that such dodgy modelling renders the study invalid and shows the authors to be hacks, charlatans and silly people? I have the distinct feeling they do.

140

hardindr 07.29.07 at 12:24 pm

I may have missed the answer to this in this thread’s comments or somewhere else, but I have just one question for David Kane and his supporters: when will they submit their critique of the Johns Hopkins study published in the Lancet to a reputable peer-reviewed journal for publication, the bare minimum standard for doing scientific research? If they aren’t willing to do that, and I’m not implying that they necessarily aren’t, then this seems like a one huge waste of time.

141

Mark G 07.29.07 at 2:21 pm

Well, to be fair, hardindr, it is quite normal for scientific papers to be discussed in conferences (or even, as this one was, online) before being submitted to a peer-reviewed journal. I would assume that David Kane’s eventual goal is to publish this paper somewhere reputable–although if dsquared et al are right it might not actually be accepted anywhere.

I am with you (and dsquared’s original post), though, that it’s not appropriate to release the paper to Michelle Malkin and the like before it’s been through any review processes.

Unfortunately, that’s the flip side of the wonderful changes the web has brought to information dissemination. I think similar issues have cropped up in physics, with Arxiv preprint making big splashes in the news that are then quietly withdrawn when all the mistakes are pointed out. (Can’t find a link to the story I’m thinking of, sorry.)

142

Barry 07.29.07 at 3:28 pm

quo vadis, just in case you didn’t read the paper, the authors were in fact aware that what they were trying to measure (death rates in a war zone) would be expected to be ‘clumpy’, with outliers. They dealt with it.

143

Barry 07.29.07 at 3:33 pm

In #188, John Quiggin wrote: “As Daniel says, there’s no point in trying to convince the 26-per-centers. The only thing needful at this point is to keep good records against the day when anyone suggests believing anything these guys say in the future.”

If one follows the link posted by ‘j’ in 112, one finds a thread wherein Shannon Love denounces scientific consensus, and uses the old Club of Rome study from the 1970’s as an example. Of consensus within and over a field of sicence. Which, of course, it’s not.

That makes Shannon either a fool or a liar (I’d include the posssibility of a ‘brain fart’, something I’ve been guilty of, but Shannon is remarkably consistent).

144

Bruce Sharp 07.29.07 at 4:15 pm

Re #143 and #144 above: Over in the Deltoid thread, David Kane mentions that his paper is currently under review at the Lancet.

I think it’s perfectly reasonable to have posted it at Deltoid to solicit comments before its publication. However, I don’t think it was appropriate to have allowed it to be posted on Andy Kaufman’s website. (Oops, I mean Michelle Malkin’s website.)

When someone in the Deltoid thread criticized him for giving Malkin permission to post the paper, he replied: “To be clear, I did not seek Malkin out. She is a Deltoid reader (and who isn’t?) and contacted me. Should I have refused her permission to reprint? If someone contacts me from DailyKos, should I refuse him permission? If, say, Andrew Gelman asked for permission to reprint should I give it to him? That seems positively unscientific to me. Once you decide that a paper, even in draft form, is ready for wider distribution (not necessarily ready for publication, but ready for comment and criticism from smart people), it seems silly to m to restrict its distribution.”

But this is where I think David’s actions put him on thin ice. In my opinion, he should have simply told her that she was welcome to link to the Deltoid thread, but not to post the paper. By allowing her to post his work on her site, he has effectively become a contributor; and by virtue of being a contributor, he has a responsibility to ensure that his work is not misrepresented. And in fact, Malkin did misrepresent Kane’s conclusions: She quoted Kane’s statement that “The Lancet authors cannot reject the null hypothesis that mortality in Iraq is unchanged,” and claimed that this is “Kane’s bottom line.”

First, I would argue that Kane should have said “mortality was unchanged,” rather than “mortality is unchanged,” since we’re talking about a study from 2004. But in any case, does Kane really think that mortality was unchanged? I don’t think that he does: elsewhere in the Deltoid thread, he suggested that he thought excess violent mortality was on the order of 100,000, apparently referring to the period covered by the Lancet’s subsequent 2006 study. And remember, that is only excess violent mortality: overall increases from ancillary causes would be far, far higher.

If Kane does indeed think that the most important idea to be gleaned from his paper is that there may not have been any excess mortality during the period in question, then fine: he is under no obligation to make further comment. If, however, it was not his intention to make this claim, a commitment to honest discourse would dictate that he should make that clear to Malkin’s readers.

Regards,
Bruce
http://www.mekong.net/cambodia

145

hardindr 07.29.07 at 8:11 pm

Just to be clear, David Kane has indeed claimed that his Johns Hopkins critique is “under review” at (surprisingly enough) the Lancet.

http://scienceblogs.com/deltoid/2007/07/david_kane_on_lancet_confidenc.php#comment-513812

I can’t imagine that it will be published there, though. Is this some type of elaborate joke?

146

hardindr 07.29.07 at 8:25 pm

Additionally, I hope that someone can attend Mr. Kane’s presentation tomorrow and let us know what the response is from the audience of his peers.

147

agum 07.29.07 at 9:58 pm

I hope that someone can attend Mr. Kane’s presentation tomorrow and let us know what the response is from the audience of his peers.

Show up for the 8:35 AM slot? Right…

Sounds like his peers weren’t very impressed by the draft he submitted.

148

lemuel pitkin 07.29.07 at 10:42 pm

Ragout writes that the populations where mortality has improved post-invasion

are in Kurdistan, Syria, Iran, and Jordan. The people Saddam was targeting before the war were Kurds, Chaldeans, Turkomen, and other minorities. It was called the “Arabization Campaign.” Before the war, about 1 million of them were internal refugees, and another million or so were refugees in neighboring countries.

But these people were already refugees before the invasion, no? So if they have not returned since, why would we think that their lives have been improved by the invasion? And in any case, why should a study of changes in mortality in Iraq since the invasion include people who haven’t lived there during that period?

Or are you talking about people who have been displaced since the invasion? This is the majority of the refugees in Syria, Jordan, etc. It seems like you’re suggesting that the refugee population is disproportionately composed of people whose lives have gotten relatively better since the invasion. But why would you assume that? Wouldn’t it make more sense that the groups most likely to flee the country are those who are most at risk of violence?

149

Ragout 07.30.07 at 1:53 am

Lemuel,

The pre-war refugees probably have lower death rates after the invasion, not because of the invasion, but because time has passed. They’re not being persecuted by Saddam anymore and they’ve had time to get back on their feet. They should be counted because they represent the counterfactual: if Saddam were still in power, he’d still be denying food ration cards to Kurds in Kirkuk, launching military attacks against the Marsh Arabs (or some other group), and creating new refugees.

The experience of refugees matters because they’re some of the people who’ve probably suffered the most. What if you studied mortality in Darfur, and didn’t survey the refugees in Chad? Even if we say that we’re just studying Iraq, many of the refugees were living in Iraq during the period covered by the Lancet study.

But I agree with your point about post-war refugees. There probably weren’t many post-war refugees at the time of the the first Lancet study, but there were a lot by the second. So that’s one way that Lancet 2 probably undercounted the death toll due to the invasion, which is a depressing thought.

150

J Thomas 07.30.07 at 4:43 am

The lancet studies suffered the bias that they chose households and then asked each household about deaths in that household.

So it completely failed to sample households that had no survivors.

For this reason it undercounted deaths both before and after the invasion, by an unknown amount.

151

Bruce Sharp 07.30.07 at 7:47 am

Hi j –

You’ve correctly identified a potential problem, but it wouldn’t be an insurmountable one in a well-designed survey… and, in fact, the Lancet survey did attempt to account for this problem.

The problem, incidentally, extends beyond households with no survivors: depending on the method of selection, the results of the survey may also be affected by the difficulty of finding households with few survivors, and not just households with no survivors.

I wrote an analysis of the death toll of the Khmer Rouge regime a couple years ago, and discussed this problem. A friend also designed a (very crude) javascript program to simulate the likely effects of clustering of mortality on a purely random survey. If anyone is curious, the simulation (and an explanation of the problem) is at http://www.mekong.net/cambodia/survey.htm.

It would in theory be possible to adjust the results of the survey to offset errors introduced by this type clustering. Someone with a good understanding of statistics (like Daniel, here on CT, or Robert on the Deltoid thread) could probably explain how that could be done.

In the 2004 Lancet study, the authors tried to account for this “survivor bias” thusly:

“At the end of interviewing every 30 household clusters, one or two households were asked if in the area of the cluster there were any entire families that had died or most of a family had died and surivors were now living elsewhere. We did this to explore the likelihood that families with many deaths were now unlikely to be found and interviewed, creating a survivor bias among those interviewed.”

Is that an adequate method of overcoming the survivor bias? Honestly, I have no idea. It’s as good a solution as anything that I can think of.

I’m convinced that an accurate toll for a severe mortality crisis can best be derived by correlating the results from different methods of calculation. I’d love to see someone attempt this for Iraq. If anyone feels like undertaking that, feel free to contact me via email. I’ll happily cheer you on while people on both sides of the debate screech at you for being a shill for the other, evil side.

Regards,
Bruce
cambodia@aol.com

152

engels 07.30.07 at 5:22 pm

I’d be interested to know who Brett blames for the deaths caused by lack of clean water, food, health care, electricity or sanitation. Iraqi plumbers, farmers, doctors and engineers, presumably?

153

Jon H 07.31.07 at 1:53 am

““Those flabby statisticians wouldn’t stand a chance against a trained killer.””

Ah, but you fail to consider the Bayesian Death Touch.

Comments on this entry are closed.