Alice in Wonderland and the Lancet study

I never thought the problem with the Lancet study was in the math, rather than the notion of attributing killings by one side in a war to the other. With the idea that the more innocent people the folks you’re fighting murder, the greater YOUR guilt becomes.

Mistakes like that aren’t math errors.

…and NFI to boot.

Brett, certain basic ideas are that (a) an occupying force has responsibilities (‘you break it, you bought it’), and (b) from a viewpoint of what is supposed to constitute US political, military and moral victory, having Iraq become a h*llhole is a resounding defeat.

As I skim the comments from Lambert’s sire, is Kane argument really just that by including Fallujah and making a terrible assumption about normally distributed estimates, you can no longer exclude zero from the CI?

Yes. He also seems to claim that a) the authors used normality assumptions for their CIs for the pre and post invasion crude mortality rates (which they might have done but the paper doesn’t say so) and b) if they did so, they are committed to doing the same thing for the CIs for the relative risk and excess deaths calculation (patently stupid). At times it looks like he is saying c) that because of b) it is actually likely that the truth is that the confidence interval doesn’t include zero, which means that every 25 comments or so, somebody has to remind him how ludicrous this claim is. I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.

“David was soliciting comments, seemingly in good faith, from the Deltoid community”

The guy produces a hack paper that mangles numbers to show that maybe the Iraq War isn’t going so badly after all — I don’t think that “good faith” is really an important concept in any connection with this incident. Let’s assume, which I doubt, that he isn’t a knowing propagandist. In that case, terminal carelessness is just about as bad.

And who hands their paper to a Web site for review? Couldn’t he have shown it to a statistician first? I think it’s more likely that the distribution of this apparently unpublisheable paper to a mass audience was the point.

Nice graphics. Negative mortality rate on the red-blue-and-green one is indeed intriguing; but on the other hand I hear these kinda things do happen in the Middle East once in a while.

Hitchens was way ahead on that argument that the invasion may have saved lives [in Slate ]

Make the assumption that some percentage of those killed by the coalition are the sort of people who have been blowing up mosques, beheading captives on video, detonating rush-hour car bombs, destroying pipelines, murdering aid workers, bombing the headquarters of the United Nations, and inciting ethnic and sectarian warfare. Make the allowance for the number of bystanders and innocents who lost their lives in the combat against these fanatics (one or two, alas, in the single case of the precision bombing of Abu Musab al-Zarqawi, just to take one instance). But who is to say how many people were saved from being murdered by the fact that the murderers were killed first?

Further on the sociology aspect, I think it would make for some interesting academic examination to try to estimate if the ability of demagogues to widely trumpet propaganda on blogs which are like MSM syndicated columns, versus the the ability of honest researchers to post real science on blogs very few people read, makes for a better or worse situation compared to previous media. The standard evangelist line (my sarcastic phrasing), runs something like “So what if there’s a whole well-funded infrastructure to pump out nonsense, anyone could read the correct information, so go blogs“. We don’t lack for case studies :-(.

Heh. I like it. So if I see that the average income in Some Suburb is $30,000 in 2007, and in 2008 I do a random survey that finds that the average reported income for 99 respondents is $31,000, but the 100th respondent turns out to be Bill Gates responding from his secret suburban summer home, I can actually conclude that the average income may have gone down, because the volatility he introduces widens my standard deviation to the point that it encompasses the possibility of an anti Gates earning negative millions per year?

Is that an accurate summary of the error? Or am I still not getting it exactly.

I’ll probably just make a fool of myself here, but I think Patrick’s example doesn’t quite get at how bizarre Kane’s reasoning is. After all, it is possible to earn negative millions per year. In a certain kind of farming community, it might not even be that unlikely that someone does this. Negative deaths per year is somewhat less plausible.

To be sure, Patrick’s underlying point that an extreme outlier in one direction doesn’t thereby make it probable that there are extreme outliers in the other direction is, I think, basically right. But the analogy he raised isn’t I think as absurd as what he was trying to analogise.

Patrick – Bill Gates would raise the average of 100 people by a lot more than $1k but basically yes.

There are some situations in which it would be reasonable to make something like this assumption – if you find a few chocolate boxes that have been overfilled, then you might reasonably surmise that quality control is to hell and some others have been underfilled. But this isn’t one of them and I think that’s pretty obvious.

I’ve been thinking about these examples too. Mine is:

you want to determine the average co2 emissions for air transport of the average american. You do a survey of 33 people asking “what is the furthest you have ever travelled in an airborne vehicle”, then multiply their given distance by the average co2 emisssions for a plane (obviously this paper is only getting published in a crappy psych journal, which David will be lucky to do). Your first 32 respondents represent the average american to a t. The 33rd is Buzz Aldrin.

As a consequence of getting Buzz Aldrin in your sample, variance is huge, and the confidence interval for co2 emissions includes a large chunk of negative emissions.

bingo! US air travel is actually absorbing co2

That’s the David Kane school of statistical analysis at work.

Indeed, dsquared, but by the time it reaches Michelle Malkin it’ll be all “we absorb co2! Fly more planes! kill greenies!”

Here’s Kane on the paper’s mass distribution:

I wish that left wing blogs would also highlight the paper. Anyone have a contact at DailyKos or Talking Points Memo? I sought to post it at Deltoid, where all the Lancet geeks hang out. See the excellent discussion at the last link which Lowell provides. Malkin picked it up from there, and then it spread.

From EphBlog.

I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.

You overlook that David Kane served as an officer in the United States Marine Corps from 1988 to 1991. (1)

Those flabby statisticians wouldn’t stand a chance against a trained killer.

Strange coincidence, isn’t it, that two of the loudest Lancet critics – David Kane and Michael Spagat – have both had connections with the U.S. military, with the latter bankrolled by military contractor Radiance Technologies.

When you stats wizards have finished shredding Kane’s latest screed perhaps you could calculate the odds of that happening ;)

Incidentally, did Spagat and Co’s MSB paper ever get reviewed and published, or is it sitting gathering dust on a shelf marked “Job Done”?

(1) http://www.iq.harvard.edu/People/people.php?info=166&sub=6

I see someone else made something like this point, but isn’t there technically a possibility of a negative *relative* death rate, which is what the study was about? Just as Fallujah went from being a (perhaps?) relatively ordinary town to one with massive conflicts, it’s at least conceivable that there could have been towns with massive conflict or gas attacks or whatever beforehand but settled down to just the average level afterwards. Of course, I don’t think that’s likely at all, but at least the negative figures aren’t the equivalent of a negative death rate.

Or am I just horribly mistaken, because I haven’t read either paper?

dsquared wrote:

My advice is, David don’t hand this paper out.

That was my first impression, too, but once it got out into wingnut-land, I would think the proper response now changes to, “David, please hand this paper out“.

You ignorant liberals are casually and foolishly ignoring a dangerous possibility: Saddam Hussein’s Secret Zombie Technology may have been unleashed in Iraq!

“Those flabby statisticians wouldn’t stand a chance against a trained killer.”

Nah. (a) we fight dirty, (b) we’d outnumber him, and (c) some of us (not me) aren’t flabby.

In fairness, he does note that you can’t have a negative result. He then dismisses the concern by claiming that the results would be similar if you used a truncated curve. He doesn’t explain how or why, or show any math, or explain what he means by truncated.

…results would be similar if you used a truncated curve…

I think it’s like the man of the thread said: if your math (e.g. your assuming a normal distribution) leads you to the conclusion that there’s a non-zero probability of a negative mortality rate, then it may already be too late to start truncating.

If that’s the case, David, then I fail to see the point of your paper. They estimate the relative risk by a completely different method. How are any of the calculations in your paper relevant?

Daniel,

I am afraid that you have gotten a couple of important things wrong.

Under David Kane’s assumptions, the discovery of the Fallujah cluster was a reason to believe that things might have gone better in Iraq. This clearly means that these were the wrong assumptions.

Kane never asserts anything like this. In fact, he asserts exactly the opposite. The study is so flawed that, even though death rates have obviously increased, the study cannot mathematically exclude the possibility that death rates have stayed the same or even decreased.

The Fallujah datapoint increases the uncertainty of the estimate, but it doesn’t increase it in both directions, because there is no way that you could find an “anti-Fallujah” (a datapoint which brought the overall average down by as much as real Fallujah brought it up), because such a place would need to have a negative death rate.

Are you sure you wish to claim this? This means the study is inherently biased to exaggerate deaths because a cluster can only increase but never decrease the death rate. In other words, the study by design could only discover whether the death rate stayed the same or had increased. It could never detect whether the liberation saved lives. This would make the study fraudulent on its face.

If, however, the study could detect an improvement in mortality then Kane’s assumption of symmetrical uncertainty is valid and thus confirm his math.

Which is it? Is the study design inherently flawed or is Kane right? I know the answer according the study itself. Can you guess what it is? (Hint:try page 3)

In other words,

Strawmen are good at scaring crows? Indeed.

You appear to be arguing that the failure to observe mass resurrection in Iraq invalidates the study.

Shannon, I think you need to bone up a bit on the concepts “zero” and “positive” before discussing statistics again.

nicks,

You appear to be arguing that the failure to observe mass resurrection in Iraq invalidates the study

No, I am arguing that study as outlined in the paper could have detected whether the mortality rate improved, stayed the same or worsened. If the study could not do this, then it could never detect whether mortality had improved.

If you think real hard, I am sure you can think of way that people come to be alive beyond return from the dead. Les Robert et al were in fact smart enough to structure their study to capture this somewhat less than mystical effect.

john quiggin,

I think you need to bone up a bit on the concepts “zero” and “positive” before discussing statistics again.

No, you need to understand the difference between a ratio, which cannot be negative, and the difference between to ratios, which can.

Shannon, love

why? Which two ratios are we differencing?

dsquared writes:

re #44 I have refused to “correct” this in email to David since his paper[1] does not mention the word “bootstrap” at all and its conclusion (that the null hypothesis RR=1.0 cannot be rejected) is utterly dependent on assuming something other than the bootstrap.

[1] Or at least, the version that I am commenting on – he might or might not have changed it since then, but this is one of the things I am complaining about.

1) This footnote is ridiculous. We are all talking about the same paper. It is still posted at Deltoid.

2) The fact that I do not use the work “bootstrap” is evidence of nothing. You assert that I assume X. You need to quote me assuming X. You can’t. I don’t assume it so I don’t write it. I challenge you to provide a quote.

3) You claim that my conclusion is “utterly dependent” on this assumption which I do not make. How so? I provide a series of steps. I make many specific claims. I use all sorts of equations. Nowhere do I say or imply or assume the specific estimation method used by the authors to calculate the RR. The method is irrelevant to my argument.

I feel like you have just claimed that I have assumed in the paper that all unicorns are purple. Your evidence? I do not use the word “pink” and my conclusion is “utterly dependent” on unicorn-coloration. How should I refute such gibberish?

Shannon, love,

as I pointed out on deltoid, given the data there is also a near-zero probability that the point estimates of CMR pre and CMR post are the same. It depends on your choice of starting point for David Kane’s argument (he hasn’t responded to this yet). This is due to the poor behaviour of the confidence intervals.

Also shannon, love, CMR pre and CMR post are ratios in name only – they have the same denominator. So really you are talking about the difference of two numbers (the absolute death counts).

And finally shannon, love, it is also possible to have a contradiction between a Relative Risk confidence interval and the confidence intervals of its contributing counts. It’s very easy to see this if you take the time to construct some figures.

David Kane’s paper confuses or misunderstands all these points. And that’s on top of all the misunderstandings dsquared has described here.

man i’m glad i don’t do social science.

Common sense suggest that if no measurement in another cluster could even theoretically offset the Falluja cluster then the studies design would be flawed from the get go.

Indeed, it could be shown that no place on earth could ever have been struck by a meteorite or an asteroid because no event, or its absence, could theoretically offset it.

The dinosaurs live!

Shannon, love, I’ll grant you the ratio comment, but the substance of your “perhaps” is somewhat misleading. You can do the calculations with RR for simple cases to show that one can indeed have a negative deltaCMR, but a RR that contradicts this – or the opposite.

This is probably also the reason why David Kane’s proof in section 1.0.2 will give contradictory results depending on the starting point. In general, a proof which does that is not very binding.

The study is so flawed that, even though death rates have obviously increased, the study cannot mathematically exclude the possibility that death rates have stayed the same or even decreased

Ey oop, it’s Shannon Love! Do you really think you had such a good time you tried to bluster statistics with me that you thought you’d do it again?

No. The study is not flawed in this regard, because it uses sensible, robust methods to determine the confidence interval. Kane’s methodology uses a silly, nonrobust method in which a positive outlier like Fallujah extends the lower end of the confidence interval symmetrically with the higher end. Kane then attributes this methodology to the paper in order to claim his “inconsistency”. He is now trying to backpedal and claim he never did this, but yes he did.

Are you sure you wish to claim this? This means the study is inherently biased to exaggerate deaths because a cluster can only increase but never decrease the death rate.

No, and I second John’s helpful advice that you think before engaging mouth in future. The addition of any cluster with a lower death rate than the previous average will lower the death rate. All clusters other than Fallujah bring the average down compared to Fallujah. However, no single cluster can lower the average by the same amount that Fallujah raises it because the death rate cannot be lower than zero.

If there were loads and loads of clusters with a postwar death rate of 1 or 2 per 100k, then it would be perfectly possible for this survey to report a fall in the death rate. There are not loads and loads of such clusters, so the study correctly reported a rise in the death rate.

I challenge you to quote something from the paper to the contrary, but you can’t because there is nothing there.

Your calculation on p6 and your table forming figure 2 do assume a normal distribution of the data including Fallujah. This is why you get different numbers for the excess deaths excluding Fallujah from those presented in the paper. Your charts all assume normal distributions – this is visible from simply looking at them. Your key conclusion is based on a chart which visibly has a chunk of its probability mass below zero. There is no correction to make here.

This is rather as if I had said “Kane believes that weasels are about to take over the world”, based on your having bought a weaselling gun, moved your home and possessions into a weasel-proof shelter and formed a committee called WEASELPAC to direct contributions to the Presidential candidate who promised to address the weasel problem, and then you challenged me to find a sentence where you had said “weasels are about to take over the world”. Your paper has no point to it at all without this assumption. You repeatedly claim that reported figures within the paper are “inconsistent” when they are only inconsistent on the basis of that assumption. You can ask as many times as you like but the answer is still going to be no.

By the way, on the Deltoid thread, halfway down, you have made the claim that Les Roberts carried out the estimation of excess deaths including Fallujah based on a normal distribution and then refused to print it because it didn’t reject the hypothesis. So this prissy attitude to your own precious paper is ill-placed.

This is silly, Brett, and has been gone over many times before. Moral responsibility is not zero sum. If you start a war, with the foreseeable (and foreseen, and much derided and now, in this thread, denied) consequence that hundreds of thousands of people are killed, you’re responsible for those deaths. Equally, so are those who decide to fight you.

The practical consequence of your claim is that anyone who believes they have a just cause is entitled to go to war, and blameless for the consequence, since the resulting deaths will be the fault of the other side for unjustly resisting.

Leaving this aside, you have some responsibility to the truth. Instead of engaging in these apologetics won’t you at least have the decency to admit that the war has caused all these deaths and that those denying it, for whatever reason, are wrong to do so.

dsquared writes:

Your calculation on p6 and your table forming figure 2 do assume a normal distribution of the data including Fallujah.

False! (I assume you mean table 2 and not figure 2.) I am merely trying to replicate the (unreleased) formula by which the authors got 98,000 CI (8,000 — 194,000). I propose a formula. (If you know what the formula is, please tell us. The authors won’t tell me.) The formula has nothing to do with a normal distribution. It simply takes the CMR data given in the paper and plugs it in. Even if I did however, this formula and table do not have anything to do with RR. Nothing. RR does not enter in in any fashion. How can a formula and table which do not use RR demonstrate what assumption I am making about RR?

The authors used, as they clearly state, the CMR to calculate excess deaths, not the RR.

Your charts all assume normal distributions – this is visible from simply looking at them.

True. But this is correct! Consider figures 2 and 3. These are accurate descriptions of the CMR estimates. The authors have told us that they use the normal distributions for these. Those figures are spot on. Pick a specific one you object to and we can go throw it in detail. And, even if this were not true, figures 2 and 3 have nothing to do with RR. How are you able to deduce my assumptions about RR by looking at a figure about CMR, a figure which just describes graphically what the authors report?

Your weasel analogy is amusing, but if you can’t come up with a singe example that remotely illustrates your claim, then shouldn’t you retract it?

David Kane is, of course, a weasel and an especially immoral weasel at that, but carry on with the destructive idiocy David.

Brett Bellmore is simply a radical right trolling weasel, crazy as crazy can be.

What these weasels have in common is justifying our immoral war in Iraq and our more immoral occupation of Iraq.

dsquared also wrote:

Your key conclusion is based on a chart which visibly has a chunk of its probability mass below zero.

This discussion would be more productive if you would specify which figure you are talking about. Figure 3? Again, there is nothing in this figure about RR. Nothing. I graph the two normal distributions that the paper provides and then the difference between them (specifying in the text that I am assuming zero covariance). Surely, you agree that the difference between two normal distributions is itself normal. Surely you agree that I have done the subtraction correctly.

While it is true that the resulting graph allows CMR to be negative, this is *exactly* what the authors allow in the original paper. They use the normal distribution. You may think that this is a critical flaw and that the paper may be ignored on that basis. I may think that this is OK and that other choices would produce similar answers.

But none of that is germane to the question of whether or not I assumed that the RR was not bootstrapped. I did not assume that. Please correct your claim to the contrary.

You’ve outdone yourself, Brett. I’m in awe. Events would have beaten some sense into a lesser man’s head by now.

“You will be expected to respond positively to our efforts to do you good. If you insist on acting like unruly schoolboys you will be soundly thrashed.”

What a sound basis for military planning!

No Brett, we did not do “good”. We did a “war crime.” And if you’re naive enough to not understand nationalism I feel sorry for you. Anyway, here’s the death of your simplistic moral agent (sorry about the long quotes, but these ideas do kill one objection to the Lancet):

Thomas Nagel’s View from Nowhere (assume his “external view” corresponds to the viewpoint of the Lancet authors while “our natural human point of view” corresponds to our sense of ourselves as free moral agents): “In ordinary judgments of responsibility we do not go that far outside, but stay in our natural human point of view and project it into that of other, similar beings, stopping only where it will not fit. But judgments so based are vulnerable to the more external view, which can take in both the defendant and the judge. . . . The judge’s sense of the defendant’s alternatives is revealed as an illusion which derives from the judge’s projection of his own illusory—indeed unintelligible—sense of autonomy into the defendant.” (123). “[T]he external standpoint is always there as a possibility, and once having occupied it we can no longer regard our internal judgments of responsibility in the same way. From a point of view that is available to us, they can suddenly seem to depend on an illusion—a forgetting of the fact that we are just parts of the world and our lives just parts of its history” (124). This would suggest that an adequate moral theory must take into account that we are parts of the world and not merely actors within it.

In Creating the Kingdom of Ends, Korsgaard similarly stresses the difference between a holding another person responsible for their actions and that person in fact being responsible for their actions (which if Nagel is right, no one ultimately is). You hold people responsible when you are in a kind of personal relationship with them: “For instance, it may be perfectly reasonable for me to hold someone responsible for an attitude or an action, while at the same time acknowledging that it is just as reasonable for someone else not to hold the same person responsible for the very same attitude or action” (199). “In everyday personal interaction, we cannot get on without the concept of responsibility. And therefore we cannot rest with the view that agents take responsibility for their own actions but can refrain from judging others” (197). Brett, since I doubt that you have any “personal interaction” with Iraqis who are violently responding to the criminal destruction of their society around them, I wonder why you are trying to hold these Iraqis responsible for actions that stem (though not ultimately) from Bush’s illegal decision to invade Iraq.

Shannon Love writes:

Those two sentences in combination are a lie by the way. No single data set supports that interpretation.

Shannon, do you have evidence that proves the claim you cite to be incorrect? I’m assuming that since you assert that the claim is a “lie,” you have evidence to clearly establish that the claim is false. Will you reference this evidence, please?

dsquared,

The study is not flawed in this regard, because it uses sensible, robust methods to determine the confidence interval.

How can you tell? The authors never provide a confidence interval for the dataset including the Falluja cluster. They provide a CI for the data set excluding the Falluja cluster but not for the one including it. Kane is forced to infer the CI from the CI of rates of Relative Risk. If the authors had followed standard practice in the first place we wouldn’t be having this conversation.

However, no single cluster can lower the average by the same amount that Fallujah raises it because the death rate cannot be lower than zero..

No, because the study NEVER compares death rates between clusters. Instead it effectively compares changes in the death rate pre and post invasion. The delta can obviously be negative.

The entire notional point of the study was to determine the CHANGE in death rates. As such, any cluster that showed improvements in mortality post invasion could offset the Falluja cluster. In fact, if you look a fiq 1 (p 3) of the 2004 paper you will see that the province of Sulmamanhya shows a fairly pronounced improvement in mortality. It is easy to see visually that it could offset the increase of mortality in Falluja if it was larger.

If the study did actually average death RATEs between clusters then yours and Daniel’s argument that the distribution cannot be expected to be symmetrical would be true. However, since the study averages numbers, which can assume any value, we HAVE TO assume a normal distribution unless the authors say otherwise, which they do not. (see p3 pg4 – p4 pg1)

Don’t feel to bad. I made the same mistake the first time I read the paper. IIRC, it was you or Daniel that corrected me on it.

I note the absence of papers commenting on the possibility that the Lancet paper underestimates excess deaths in Iraq, just as in the tobacco/cancer controversy, the hired hands never complained about the conservatism of the assumptions made by statistical researchers who, like the authors of the Lancet study, had bent over backwards to hedge their results.

What we have here is another in the long series of asymmetrical arguments between people who believe that you ought to try to figure things out whatever conclusion you would prefer and those for whom the game is simply to fight fiercely for your ideology and/or employer regardless of the rules of evidence or, for that matter, ordinary morality. Critics of the Lancet study “knew” it was flawed long before they read it, just as tobacco publicists “knew” that the statistical arguments associating lung cancer and cigarettes were wrong and the Creationists and holocaust deniers—that is deniers of earlier holocausts–“knew” that the historians were wrong.

The point isn’t that people shouldn’t criticize the methodology of the Lancet study, but its most vocal critics aren’t fooling anybody. Which is why, even in this thread, some of them are playing the old game of “I didn’t steal the pig. It was my pig, and anyhow, it wasn’t very big.” Logically, of course, the conclusions of the Lancet study do not imply that the invasion of Iraq was a moral or political error. All those deaths may have been well worth it, at least from a sufficiently blooded minded point of view. And one can certainly imagine that a great many excess deaths may have resulted in the absence of an invasion by some unspecified mechanism. Unfortunately, these considerations are just ways of changing the subject.

The authors never provide a confidence interval for the dataset including the Falluja cluster

that is because it would not be sensible to do so, and they are sensible.

In fact, if you look a fiq 1 (p 3) of the 2004 paper you will see that the province of Sulmamanhya shows a fairly pronounced improvement in mortality. It is easy to see visually that it could offset the increase of mortality in Falluja if it was larger.

If you look at it closely, however, you would see that if the improvement in mortality was much larger, people would be coming back from the dead, which is kind of my point. Decreases in the death rate can only be so big, because they can’t be more than -100%. Increases in the death rate can be much more than +100%, and in Fallujah they were.

What gender is Shannon Love? There’s a comment I feel impelled to make, but I can’t formulate it properly without more information.

dsquared claims

This would only be a legitimate calculation if one was making a parametric assumption about the confidence intervals. You can’t just randomly “plug in” confidence intervals like this, unless you’re making a lot of assumptions.

This is just false. Let x have probability distribution p(x). If I tell you *nothing* about p(x) except that the 2.5th percentile is 1.4, you can still manipulate that number linearly (which is all that I do). In other words, the 2.5th percentile of 2*p(x) is 2.8. The 2.5th percentile of 10*p(x) + 3 is 17. I need assume nothing about p(x) to derive linear transformations of specific percentiles.

I make no assumptions. And, even if I did, I am making no assumptions about the RR. Nothing in that calculation mentions RR. You claim that I assume that the RR was not estimated via bootstrap. Nowhere do I assume that. Care to try again? I recommend just correcting the original post. It’s OK, we all make mistakes.

Soru *In sarcasm, [commenters on Deltoid] even use the possibility of ‘resurrection’ as something that is inherently more likely than Saddam killing some Kurds.*

This isn’t true. The Lancet paper allows for the possibility death rates went down, and none of David’s critics deny that this is theoretically possible. When people are talking about ‘resurrection,’ they are saying that death rates can’t be negative, not that excess deaths can’t be negative. Believe it or not, the problem with David Kane’s calculations is that they allow death rates to be negative. Again, negative death rates aren’t just unlikely, they’re impossible.

Shannon, I’ll leave it to others to deal with your supposed contribution to the protection of our scientific institutions. The people competent to do so seem completely unimpressed with your work in that respect too, but I’m not able to participate in that argument.

My guess is that you have your own preconceptions, biases, motives, and theology and that your claim to scientific purity is bogus. To me your contribution seems pretty similar to all the other winger attempts to obscure the gross facts about Iraq behind a gnat swarm of quibbles and aspersions of various kinds — some sophisticated, some crude, some slanderous, some trivial, and many simply wrong. By so doing, and by stubbornly repeating discredited arguments, wingers often succeed in casting doubt on valid information. There’s a repeated pattern.

If your criticisms are bogus, as most here claim that they are, their sophistication makes them the more culpable, since very few non-professionals are able to clearly understand either the criticisms or the defenses of the study, making it tempting for laymen to believe that both sides have valid points and no one can really be sure who is right. The effect, possibly the intended effect, will be to neutralize the effect of a carefully-done scientific study.

“Chicago Boyz” is a strongly ideological site and no one coming from there can be taken seriously if they try to claim the high scientific ground.

dsquared,

If you look at it closely, however, you would see that if the improvement in mortality was much larger, people would be coming back from the dead, which is kind of my point. Decreases in the death rate can only be so big, because they can’t be more than -100%. Increases in the death rate can be much more than +100%, and in Fallujah they were.

For the last time, we are not talking about death RATES but rather about the difference between death rates in two separate time frames. Each cluster produces two death rates one for before the invasion and one for after.
These two death rates are never averaged in with the rates from any other cluster! Instead, the differences are used to calculate whether post invasion, the mortality, improved, stayed the same, or increased.

Example: Cluster A. death rate prior to invasion=10/1000. Death rate after invasion=100/1000. Difference = 100/1000-10/1000=90. CHANGE in death rate=+90/1000.

Cluster B. death rate prior to invasion=100/1000. Death rate after invasion=100/1000. Difference = 10/1000-100/1000=(-)90. CHANGE in death rate=(-)90/1000.

In cluster A, 90 people MOREper 1000 die that would not have prior to the invasion. In cluster B, 90 peopleFEWER die than would have after the invasion.

Now using some very sophisticated math (90/1000+(-)90/1000) we can show that every excess death in cluster A is offset by an extra saved live in cluster B. In a population composed of cluster A and cluster B, the combined change in the death rate would be zero.

You keep saying that the paper compares death rates between clusters but it actually compares changes in death rates. The changes can be negative, zero or positive and thus CAN cancel each other out In a random sample, clusters are mathematically as likely to offset another cluster as to not. Therefore, there is no reason to suspect that Lancet calculation require an asymmetrical uncertainty distribution.

Fallujah doesn’t wreck the confidence interval because it can never be offset. It wrecks the confidence level because it is a massive outlier created by poor study design.

To me it’s boggling to have someone from “Chicago Boyz” tell me that he’s primarily concerned with our scientific instutitions, whereas epidemiologists from The Lancet and Johns Hopkins are political hacks. This does not pass the smell test.

Shannon, you’re wretchedly confused here. The study does not publish 33 different excess death rates. It publishes one, for the country of Iraq, which aggregates all the clusters together.

In cluster B, 90 peopleFEWER die than would have after the invasion

Since the average death rate in these clusters is about 5, no such cluster could exist. This is the whole point.

Btw, if you are going to continue to blah on about “our precious scientific institutions” and “intellectual dishonesty”, then I may become irritated enough to dig up some of your own greatest hits, and this will not reflect well on you or your website.

Shannon Love writes:

I don’t suppose it has occurred to you that wars come and go but that our scientific institutions form a major part of the core of our civilization. What happens when they become so corrupted by politics that they can no longer function? If people begin to widely expect that researcher’s political bias exert more influence on their conclusions then they will lose confidence in the institution. Then we will be back to settling disputes with theology.

Shannon Love, assuming you believe what you’ve written here, will you please address the point I raised in comment 85? You’ve accused Roberts et al of lying. Since your statement above assures us that you make this accusation based on clear scientific evidence rather than political considerations, will you please cite this evidence now?

Since I don’t know statistics and don’t work in statistics, I’m willing to let the questions about scientific method come out in the wash over the next few years.

The reason someone like me is hearing about statistical problems in the Lancet study isn’t because people in science are concerned about its methodological deficiencies. It’s because the topic of the study is highly controversial and the study’s conclusions are objectionable to the winger hack battalion.

If the study is bad, there could be two bad outcomes: people might come to substantially false conclusions about the Iraq War, and a bad professional precedent might be set which would be followed by others. The latter is not something I can involve myself in, but it seems highly, highly unlikely.

The former I understand better, and to me it seems impossible. I have other sources about the Iraq War, and the conclusions I come to about the war without the Lancet study are about the same as the ones I would come to with it. The small adjustments that may or may not be mandated by the criticisms of the study seem unlikely to change anything substantial, though I admit I would simply reject out of hand any tweak of the study showing that the invasion had improved Iraq’s mortality rate.

Your own accusations against the study’s authors make it impossible for me to come away from the controversy without thinking that someone is in bad faith. It seems most likely to me that the one in bad faith is Shannon Love.

dsquared,

The study does not publish 33 different excess death rates. It publishes one, for the country of Iraq, which aggregates all the clusters together.

The study actually publishes the chance of increased mortality but that is neither here nor there. Perhaps you could enlighten me on how you think they did calculate it?

Since the average death rate in these clusters is about 5, no such cluster could exist. This is the whole point.

Sigh. So if the day before the invasion Saddam launched an attack on a Shia town and inflicted exactly the same number of death in the same size population as in Fallujah then those deaths would not have offset the deaths occurred in Fallujah post-invasion?

If Saddam had gassed Halabja in the study period prior to the invasion wouldn’t the 5,000 people who died there offset Fallujah by just a bit?

Try another way, suppose we do abandon Iraq and freed from cruel American tyranny the Iraqi immediately stop fighting, eliminate crime and improve their health care system. Suppose Les Roberts goes back to duplicate his study. Now I would naively expect him to find that a significant drop in mortality post-abandonment would offset the pre-abandonment mortality. You however, seem to think that the study could never show that the study saved lives because their is no way a post decrease in mortality could offset the pre levels because the post rate could never be less than zero.

I am boggled that you cannot grasp this. If one cluster has a preCMR exactly equivalent to postCMR of another cluster and they have equivalent populations, then they will exactly offset each other in the calculation of national change in mortality rate.

1: This isn’t true. The Lancet paper allows for the possibility death rates went down, and none of David’s critics deny that this is theoretically possible. When people are talking about ‘resurrection,’ they are saying that death rates can’t be negative, not that excess deaths can’t be negative. Believe it or not, the problem with David Kane’s calculations is that they allow death rates to be negative. Again, negative death rates aren’t just unlikely, they’re impossible.

As I said in the original post, _some_ critics of Kane, to the extent that I follow the maths, seem to have a valid point.

Others, however:

2: Since the average death rate in these clusters is about 5, no such cluster could exist. This is the whole point

An unsampled cluster with a pre-war death rate of greater than 5 does not seem to be precluded by any law of mathematics I know of, so unless I am missing something, that statement by dsquared seems to be either a mathematical error, or a political judgement about the relative benevolence of the last days of saddam’s regime in comparison to the early days of the occupation.

He’s hardly alone in making that kind of comment.

john emerson,

Your own accusations against the study’s authors make it impossible for me to come away from the controversy without thinking that someone is in bad faith. It seems most likely to me that the one in bad faith is Shannon Love.

So I must be wrong because authority figures you trust haven’t come to the same conclusion as me? Have you considered the possibility that they are not paying attention, are afraid of controversy or, worse, have been co-opted. After the idea that a little creative exaggeration in the service of a greater good is very popular in academic circles now.

In the end, the only person you can rely on is yourself.

Let me help you out: Suppose a drug company did a study of its drug using 33 different hospitals as their clusters. Suppose the main finding of the study said something like:

70% of patients showed improvements. Significant side effects occurred in only 10% of patients.

Would you be comfortable taking that drug if you knew that the rate of improvement and the rate of side effects came from two different sets of data? Would you be upset if no SINGLE data set contained both figures? Wouldn’t you want to know what the improvement and side effect rates were for each data set? Would you be upset if you discovered that the drug company cherry picked the results it wanted from each data set just to make their drug look good?

That is precisely what the authors of the Lancet paper did.

The main finding of the paper, the one that states their most accurate findings and the one that got turned into a sound bite everywhere said:

Making conservative assumptions, we think that about 100000 excess deaths, or more have happened
since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces
accounted for most violent deaths.

Now, would it disturb you to know that those two sentences are a flat out lie. Each is supported by one of two data sets in the study but there is no data set in which both sentences are true. Worse, there is no technical reason why one set of data was used in the first sentence and another in the second. The only explanation is that the author’s cherry picked the data in order to create just the story they wanted.

I believe the authors cherry picked the excess deaths from the first data set because the second data sets excess deaths (300,000) was implausibly high and they didn’t want anyone to know that it was based on a single neighborhood. They cherry picked the cause of death in the next sentence from the second data because only that data set (based almost entirely on one neighborhood) showed the majority of deaths resulted from Coalition action.

Do you understand now why I am so upset? The actual underlying study isn’t that bad. Once you toss out the Fallujah neighborhood the study’s result compare with those produced by other methods. The authors, however, intentionally mis-reported there results in order to create a propaganda tool.

Like I said, check for yourself. You don’t need to understand statistics.

Shannon Love writes:

The authors, however, intentionally mis-reported there results in order to create a propaganda tool.

Shannon Love, once again you’ve made an irresponsible accusation for which you have no evidence. It’s tedious to watch you try to smear these scientists while claiming to have respect for the scientific process. If you want to be a hypocritical partisan hack, fine, but can you find a different forum, please?

Also, assuming that you think David Kane’s work is respectable, may I suggest you withdraw from the discussion of it? You are certainly doing him no favors with your contributions here.

BTW, at this point I’ve given up any hope that you will provide evidence to support any of your wild accusations. Nor do I anticipate that you will withdraw them. Rather, I suspect you will repeat the same irresponsible charges ad nauseam in the hope that someone will accept them.

Sturtevant once quipped that elegant science was a paper that reached conclusions he liked using methods he didn’t understand.

Hence, as someone who has vehemently opposed the Iraq war from the outset, I (and many other CT readers apparently) find a degree of elegance in the Lancet study. However, reading the paper also leaves an impression that while the statistical analysis may be technically valid, the estimate of the change in death rate may not be very robust. I.e. are there other equally valid approaches to the same dataset that would give a substantially different estimate? I can’t say, but at best they had to work very hard to get an estimate of borderline significance from very noisey data.

The one conclusion that everyone in the present discussion seems to agree on is that the variance of death rates increased dramatically post-invasion with or without Faluja. Common sense would indicate that is not due to positive changes on the ground. In that light, the whole CI question seems like a red herring that the Lancet authors might have avoided. I.e. they might have said that our data indicate that its bad, by one estimate at least its very very bad, we need more data to determine just how bad it really is. I don’t suppose that would have gotten the same press.

I don’t suppose that would have gotten the same press.

Yes, that’s pity.

Having been against the Iraq invasion from the start, is it possible, at once, to believe that the war was misguided, devious and has had disastrous consequences and that the 2004 Lancet study was incomplete, a work in which definition and disinterested science seem, in part, to have taken a back seat to political expediency? If so, count me in.

Shannon Love,

Apparently you don’t understand the nature of the error you’ve made since you continue to repeat it in defense of your earlier errors. Let me explain it to you so that you won’t continue down this road.

You have made assertions for which you apparently have NO credible evidence. I’ve asked you repeatedly for evidence and now, finally, in response you provide this:

a) a link to archive articles at “Chicago Boyz,” and
b) an old commentary of yours which repeats some of the accusations you’ve made here, again without evidence to support them.

This is not an acceptable response; it’s the equivalent of saying “find the evidence yourself.” You don’t seem to understand, though, that I’m calling your bluff. It’s your responsibility, not mine, to cite the evidence on which you base your charges of scientific misconduct.
Let me use as an example one of your irresponsible accusations in order to make it clear to you what your burden of proof requires.

In one instance you claim that the authors admitted to trying to influence the 2004 US Presidential election with the publication of their research. In order to support this claim, you need to provide a source for this admission. This should be straightforward. Simply cite the source by which you learned of this admission. What could be simpler?

Unfortunately, the fact that you are ignoring/evading/filibustering regarding this charge leads me to believe you are not commenting in good faith here. If you’ve made an error in your accusation, acknowledge your mistake and withdraw the charge. If you stand by your claim, support it with evidence. I will accept either as an honest response. On the other hand, continuing as you have been suggests nothing but dishonesty.

Do you finally understand that the burden of proof is on you to support your accusations of scientific misconduct? Do you understand that making such accusations without credible evidence is irresponsible?

I’d like to see some of the scientific integrity that you claim to admire in your response to this. As things now stand, you’ve discredited yourself with your performance.

The author who writes this (in reaction to the David Kane paper):

Ah, vindication is sweet

also writes this:

Kane might be wrong. He has to fudge a lot numbers because so much of the original study remains secret.

Does Shannon Love really expect to be taken seriously?

Shannon and David Kand are not interested in errors, they are crazed frauds and no more. Sigh. Duh.

I don’t think that one can blame David Kane for his criticisms of the Lancet study. I think the problem is that the authors of the Lancet study broke some rules that in some areas of research would not be broken. Essentially they discarded some of their data in order to create a sample about which they could make statistically significant claims. Whether or not this invalidates the study would depend upon what the goals of the study were and how those goals guided their original sample design. In my experience with research, studies are designed as inputs to specific decision making processes which provide a solid foundation about which to evaluate goals and sample design. It is not clear to me that the Lancet study had such a specific purpose, but then one wonders what motivated the authors to attempt statistical analysis of the data and draw conclusions from it.

In my opinion, the real problem with the Lancet study is that after having discarded some of the data to make the statistical claims, they re-introduce the data elsewhere in their conclusions. This is guaranteed to produce exactly the sort of criticisms that persons like David Kane are making.

So I must be wrong because authority figures you trust haven’t come to the same conclusion as me?

What I said is that someone must be dishonest, based on what you said. Based on what I know, most likely you are the dishonest one. Trusting authority figures is the method I use in areas I don’t understand well, such as quantum physics, relativity, and global warming. It’s really the only method most people can use in such cases, and it’s a valid method.

Have you considered the possibility that they are not paying attention, are afraid of controversy or, worse, have been co-opted? After all, the idea that a little creative exaggeration in the service of a greater good is very popular in academic circles now.

In the end, the only person you can rely on is yourself.

The authority of myself tells me that you are the liar, and not the CT people or the Lancet authors. People on your side of the fence are mor often and more egregiously guilty of the stuff you talk about. Nice try, though.

@123

The first thing anyone who had designed a research program that included the collection of primary data would do upon finding an “obvious and gigantic outlier” in their primary data is to review their assumptions, design and process and find out why they failed to anticipate the outlier.

If they had anticipated the outlier and had intended to exclude it from the statistical analysis from the start, then they intensionally collected two distinctly different sets of data for two distinctly different purposes. If they then conduct statistical analysis of one set and apply conclusions from that analysis to the other set, they expose themselves to exactly the type of criticism that David Kane if making.

No explanation is needed for any failure to anticipate an outlier of the Fallujah type. Outliers happen all the time.

If there had been attempt to come up with a predictive general theory of Iraqi history, outliers would be a massive problem. If you’re just trying to describe and analyze a specific span of time, you just recognize that there’s a discontinuity in the historical data, which is a common case that doesn’t need explanation unless you’re trying to prove uniformity in the data.

On the off change that there’s still anyone reading this thread:

There is nothing unusual about the statistical analyses in the Lancet paper. Nothing. It is a completely ordinary example of the use of statistics to analyze data. They discover — surprise, surprise — that the death rate in Fallujah is much higher than anywhere else in their sample. They compute the average death rates pre- and post-invasion, both with and without Fallujah. If they were really trying to fool everyone, they could have thrown out the Fallujah data, and never reported it. Then (and this is their main finding), they use a more sophisticated technique to verify that the ratio of the post- and pre-invasion death rates is bigger than one.

This is how it’s done. You analyze your data in different ways, you report the results of different analyses, and then you draw a conclusion. Other people who read the paper are free to interpret their results differently, and draw a different conclusion. Science goes on.

@136, 137

Walt,

I’m not sure we disagree entirely on my main point, which is that the Lancet study is a legitimate subject for criticism or at least that it is not the last word on the data it covered.

To the matter of research process: I come at this from the marketing domain, so perhaps there are different standards when there’s no money on the line, but one doesn’t start and end by collecting a lot of data and playing with it in the hope that something significant will emerge. You start with the questions you are trying to answer and express them in such a way that they can be answered using specific analysis techniques on specific sets of data. You then design a data collection process that will give you exactly the data you need to provide a statistically significant result. If after analysis, you can’t come to a statistically significant conclusion you can only conclude that there is no statistically significant relationship between the data collected and the question you were trying to answer.

At this point you can go back over your process to see if you have made any mistakes, incorrect assumptions or left anything out. Outliers in the data are good places to look for clues about where you have gone wrong. A legitimate outlier (not attributable to your process) that is large enough to invalidate your results is an indication that there is something very important about the phenomenon you are studying that you do not understand and/or have not accounted for in your research and conclusions. If you simply handwave such things, the quality of your research and veracity of your conclusions may be called into question. In some domains, that something you handwave could be the difference between an iPod and an MP3 player.

A hypothetical:

Suppose that Roberts et al. (2004) includes the stat-munging that David Kane has now served up after three years, after a circuitous route involving borderline libellous accusations towards Roberts, his co-authors and his survey team? How do Kane and Love et al. react to the use of that model? Do they say that such dodgy modelling renders the study invalid and shows the authors to be hacks, charlatans and silly people? I have the distinct feeling they do.

Well, to be fair, hardindr, it is quite normal for scientific papers to be discussed in conferences (or even, as this one was, online) before being submitted to a peer-reviewed journal. I would assume that David Kane’s eventual goal is to publish this paper somewhere reputable–although if dsquared et al are right it might not actually be accepted anywhere.

I am with you (and dsquared’s original post), though, that it’s not appropriate to release the paper to Michelle Malkin and the like before it’s been through any review processes.

Unfortunately, that’s the flip side of the wonderful changes the web has brought to information dissemination. I think similar issues have cropped up in physics, with Arxiv preprint making big splashes in the news that are then quietly withdrawn when all the mistakes are pointed out. (Can’t find a link to the story I’m thinking of, sorry.)

In #188, John Quiggin wrote: “As Daniel says, there’s no point in trying to convince the 26-per-centers. The only thing needful at this point is to keep good records against the day when anyone suggests believing anything these guys say in the future.”

If one follows the link posted by ‘j’ in 112, one finds a thread wherein Shannon Love denounces scientific consensus, and uses the old Club of Rome study from the 1970’s as an example. Of consensus within and over a field of sicence. Which, of course, it’s not.

That makes Shannon either a fool or a liar (I’d include the posssibility of a ‘brain fart’, something I’ve been guilty of, but Shannon is remarkably consistent).

Just to be clear, David Kane has indeed claimed that his Johns Hopkins critique is “under review” at (surprisingly enough) the Lancet.

http://scienceblogs.com/deltoid/2007/07/david_kane_on_lancet_confidenc.php#comment-513812

I can’t imagine that it will be published there, though. Is this some type of elaborate joke?

I hope that someone can attend Mr. Kane’s presentation tomorrow and let us know what the response is from the audience of his peers.

Show up for the 8:35 AM slot? Right…

Sounds like his peers weren’t very impressed by the draft he submitted.

Lemuel,

The pre-war refugees probably have lower death rates after the invasion, not because of the invasion, but because time has passed. They’re not being persecuted by Saddam anymore and they’ve had time to get back on their feet. They should be counted because they represent the counterfactual: if Saddam were still in power, he’d still be denying food ration cards to Kurds in Kirkuk, launching military attacks against the Marsh Arabs (or some other group), and creating new refugees.

The experience of refugees matters because they’re some of the people who’ve probably suffered the most. What if you studied mortality in Darfur, and didn’t survey the refugees in Chad? Even if we say that we’re just studying Iraq, many of the refugees were living in Iraq during the period covered by the Lancet study.

But I agree with your point about post-war refugees. There probably weren’t many post-war refugees at the time of the the first Lancet study, but there were a lot by the second. So that’s one way that Lancet 2 probably undercounted the death toll due to the invasion, which is a depressing thought.

Hi j –

You’ve correctly identified a potential problem, but it wouldn’t be an insurmountable one in a well-designed survey… and, in fact, the Lancet survey did attempt to account for this problem.

The problem, incidentally, extends beyond households with no survivors: depending on the method of selection, the results of the survey may also be affected by the difficulty of finding households with few survivors, and not just households with no survivors.

I wrote an analysis of the death toll of the Khmer Rouge regime a couple years ago, and discussed this problem. A friend also designed a (very crude) javascript program to simulate the likely effects of clustering of mortality on a purely random survey. If anyone is curious, the simulation (and an explanation of the problem) is at http://www.mekong.net/cambodia/survey.htm.

It would in theory be possible to adjust the results of the survey to offset errors introduced by this type clustering. Someone with a good understanding of statistics (like Daniel, here on CT, or Robert on the Deltoid thread) could probably explain how that could be done.

In the 2004 Lancet study, the authors tried to account for this “survivor bias” thusly:

“At the end of interviewing every 30 household clusters, one or two households were asked if in the area of the cluster there were any entire families that had died or most of a family had died and surivors were now living elsewhere. We did this to explore the likelihood that families with many deaths were now unlikely to be found and interviewed, creating a survivor bias among those interviewed.”

Is that an adequate method of overcoming the survivor bias? Honestly, I have no idea. It’s as good a solution as anything that I can think of.

I’m convinced that an accurate toll for a severe mortality crisis can best be derived by correlating the results from different methods of calculation. I’d love to see someone attempt this for Iraq. If anyone feels like undertaking that, feel free to contact me via email. I’ll happily cheer you on while people on both sides of the debate screech at you for being a shill for the other, evil side.

Regards,
Bruce
cambodia@aol.com

““Those flabby statisticians wouldn’t stand a chance against a trained killer.””

Ah, but you fail to consider the Bayesian Death Touch.