(Initial bad temper warning: I am a little bit cross as I write this, because I think that the distribution of the paper on the Michelle Malkin website was both silly (because the paper has huge flaws that a mass audience can’t possibly be expected to understand) and rude (because at the time when he gave permission for it to be distributed, David was soliciting comments, seemingly in good faith, from the Deltoid community, aimed at improving it before distribution). The Malkin link has meant that this paper has metastatised and I will therefore presumably be dealing with cargo-cult versions of it by people who don’t understand what they’re talking about from now to the end of time. I see that Shannon Love of the Chicago Boyz website is claiming to have been “sweetly vindicated”, FFS. Ah well, the truth has now got its boots on, and big clumpy steel toe-capped boots they are too. C’mon boots, let’s get walking.)
At this late stage, does anyone believe that careful metaanalysis is going to reveal that the Lancet studies were totally wrong, and that the invasion and occupation of Iraq actually went really well? Apparently yes; David Kane of the Harvard Institute for Quantitative Social Science (who CT readers might remember from this rather embarrassing incident last year) does. He apparently intends to present this paper at the JSM in Salt Lake City on Monday, arguing that the 2004 Lancet study actually could not rule out the possibility that the death rate had fallen in Iraq.
My advice is, David don’t hand this paper out. If not for the sake of your own reputation, think of the four (! On a tiny little paper like this!) research assistants you credit in it. The paper is a disaster. As the comments thread at Deltoid gradually teases out, it’s full of silly mistakes (the author constantly fails to make a distinction between an estimate and its confidence interval) and is based on a fundamental misreading of the paper (in that it assumes that the relative risk rate was estimated parametrically using a normal distribution when it wasn’t). But one doesn’t need to go into the maths of the thing to understand what’s wrong with it.
The mathematical guts of the paper is that under certain assumptions, the addition of the very violent cluster in Fallujah can add so much uncertainty to the estimate of the post-invasion death rate that it stretches the bottom end of the 95% confidence interval for the risk rate below 1. From this, David Kane concludes that the paper was wrong to reject the hypothesis that the Iraq War had not made things worse.
Let’s back up and look at that again. Under David Kane’s assumptions, the discovery of the Fallujah cluster was a reason to believe that things might have gone better in Iraq. This clearly means that these were the wrong assumptions.
The statistical problem here is basically that people can’t come back from the dead. The Fallujah datapoint increases the uncertainty of the estimate, but it doesn’t increase it in both directions, because there is no way that you could find an “anti-Fallujah” (a datapoint which brought the overall average down by as much as real Fallujah brought it up), because such a place would need to have a negative death rate.
And looking at the charts in David’s paper, it’s clear to see that the reason why the left edge of his estimate of the risk ratio has been dragged below 1 is that a substantial part of the distribution of his Bayesian estimate of the post-war death rate is below zero (and an even more substantial part is in regions of positive but wildly improbably death rates like one or two per 100K). That’s all there is to it, CT readers; the majority of the rest of the Deltoid thread consists of three or four people trying to explain that the Roberts et al. paper doesn’t make the same mistake.
As I note halfway down the thread, this is actually a nice example of some of the cases where the distinction between a frequentist confidence interval and a Bayesian credible interval makes an importance difference. In an infinitely repeated series of trials, you might very well get a small number of very unlucky or wild results that showed death rates of 1 or 2 per 100K. So if you’re thinking about the confidence interval as the limit of the empirical distribution of the estimator in repeated trials, it makes sense to have it where it is. But if you’re a Bayesian and you regard the confidence interval as your subjective probability distribution over a random variable, then it doesn’t make any sense at all to have any material weight on these low-end numbers (if you were a conscientious Bayesian, of course, you would never get into this position as you’d have used a sensible prior distribution which put zero probability on death rates below zero). In general, there is a distressing trend among statisticians to use the branding “Bayesian” (and the ubiquity of “diffuse” priors which don’t rule out silly cases like this one) as an excuse for talking crap about confidence intervals, and I think this is an example of the genre.
If any readers are attending the JSM, I’d be interested in any reports of how it went down. Now I’m off to spend the weekend playing Wingnut Whackamole I suppose …
(PS: actually, the cargo cult explanations of why the Lancet study has allegedly failed to enact someone’s ideas of the rituals of science are really quite interesting from a sociological point of view. Although I do wonder about the incuriousness here about the actual facts. I mean, if the “scientific method” regularly threw up conclusions like “knowing what we do now, it is quite likely that the Iraq War was actually a success”, do you think it would be as popular as it actually is?)
{ 7 trackbacks }
{ 153 comments }
I never thought the problem with the Lancet study was in the math, rather than the notion of attributing killings by one side in a war to the other. With the idea that the more innocent people the folks you’re fighting murder, the greater YOUR guilt becomes.
Mistakes like that aren’t math errors.
I never thought the problem with the Lancet study was in the math, rather than the notion of attributing killings by one side in a war to the other.
Neither of the Lancet studies did this, TYVM, HTH, NFO.
…and NFI to boot.
The Lancet studies calculated the number of “excess deaths” that occurred after the invasion – in other words the number of people who wouldn’t have died if the invasion had not taken place.
No matter what inferences one draws from them, the studies themselves did not give any value judgements, and in particular didn’t attempt to say ‘who killed who’.
It’s all there in black and white, unless I’ve horribly misunderstood.
Brett, certain basic ideas are that (a) an occupying force has responsibilities (‘you break it, you bought it’), and (b) from a viewpoint of what is supposed to constitute US political, military and moral victory, having Iraq become a h*llhole is a resounding defeat.
As I skim the comments from Lambert’s sire, is Kane argument really just that by including Fallujah and making a terrible assumption about normally distributed estimates, you can no longer exclude zero from the CI? So basically we have an example here that would be an exam questioning asking a stats student to explain what the problem appears to be and how to solve with the answer being normality and use bootstrapping? And Kane seems to think nothings wrong?
As I skim the comments from Lambert’s sire, is Kane argument really just that by including Fallujah and making a terrible assumption about normally distributed estimates, you can no longer exclude zero from the CI?
Yes. He also seems to claim that a) the authors used normality assumptions for their CIs for the pre and post invasion crude mortality rates (which they might have done but the paper doesn’t say so) and b) if they did so, they are committed to doing the same thing for the CIs for the relative risk and excess deaths calculation (patently stupid). At times it looks like he is saying c) that because of b) it is actually likely that the truth is that the confidence interval doesn’t include zero, which means that every 25 comments or so, somebody has to remind him how ludicrous this claim is. I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.
I mean that sort of thing is good enough for AEI work, but he’ll be ripped to shreads by others. And he won’t likely luck out and come to a presentation with no one having read the paper given the topic. That would have been bad but survivable. But thats unlikely to happen in this case.
“David was soliciting comments, seemingly in good faith, from the Deltoid community”
The guy produces a hack paper that mangles numbers to show that maybe the Iraq War isn’t going so badly after all—I don’t think that “good faith” is really an important concept in any connection with this incident. Let’s assume, which I doubt, that he isn’t a knowing propagandist. In that case, terminal carelessness is just about as bad.
And who hands their paper to a Web site for review? Couldn’t he have shown it to a statistician first? I think it’s more likely that the distribution of this apparently unpublisheable paper to a mass audience was the point.
And who hands their paper to a Web site for review?
In fairness, David was and is part of the Deltoid community, so I thought that bit was quite reasonable. I was really quite hacked off at discovering that it had simultaneously been mass-distributed though – am I being weirdly prissy and prima-donnaish here, or am I right that if you ask someone for comments, there’s an implicit understanding that you’re going to take those comments seriously and not do something like this.
I think it’s more likely that the distribution of this apparently unpublisheable paper to a mass audience was the point
I have an uncomfortable feeling that you’re right. David is polite and usually thoughtful, but after the thing last year and now this, I have to say I am very uneasy about him.
Nice graphics. Negative mortality rate on the red-blue-and-green one is indeed intriguing; but on the other hand I hear these kinda things do happen in the Middle East once in a while.
the thing is, I suppose, that two years ago I would have gone off on a chainsaw frenzy all over the blogosphere spitting insults over anyone who reproduced these hack critiques. But now, it’s kind of like, why bother? It is no longer the case that there is any material risk that anyone who matters will start believing that the war has gone well. I probably ought to make a token effort in order to make sure that Les Roberts doesn’t get unfairly slimed any more than he has been already, but realistically, we are down to the “Crazy 27%” bedrock now, nobody who actually cares about facts is on the other side.
Hitchens was way ahead on that argument that the invasion may have saved lives [in Slate ]
Make the assumption that some percentage of those killed by the coalition are the sort of people who have been blowing up mosques, beheading captives on video, detonating rush-hour car bombs, destroying pipelines, murdering aid workers, bombing the headquarters of the United Nations, and inciting ethnic and sectarian warfare. Make the allowance for the number of bystanders and innocents who lost their lives in the combat against these fanatics (one or two, alas, in the single case of the precision bombing of Abu Musab al-Zarqawi, just to take one instance). But who is to say how many people were saved from being murdered by the fact that the murderers were killed first?
Exactly, no need to let it upset you. Only the completely beyond-help true believers are listening at this point, the vast majority of the US (and presumably even moreso British) populations are well aware that the war is a complete clusterfuck. Only “elite” punditocracy opinion in the US has yet to quite catch up, but it’s visibly getting there.
Further on the sociology aspect, I think it would make for some interesting academic examination to try to estimate if the ability of demagogues to widely trumpet propaganda on blogs which are like MSM syndicated columns, versus the the ability of honest researchers to post real science on blogs very few people read, makes for a better or worse situation compared to previous media. The standard evangelist line (my sarcastic phrasing), runs something like “So what if there’s a whole well-funded infrastructure to pump out nonsense, anyone could read the correct information, so go blogs“. We don’t lack for case studies :-(.
Just to be absolutely sure I’m not missing something:
Did Kane really do a full Bayesian calculation (posterior distribution from a likelihood and prior), and fail to put zero prior probability on negative death rates?
Really?
If so, that should be incredibly embarrassing for him. I don’t know how you can call an analysis “Bayesian” if you use logically nonsensical priors.
Heh. I like it. So if I see that the average income in Some Suburb is $30,000 in 2007, and in 2008 I do a random survey that finds that the average reported income for 99 respondents is $31,000, but the 100th respondent turns out to be Bill Gates responding from his secret suburban summer home, I can actually conclude that the average income may have gone down, because the volatility he introduces widens my standard deviation to the point that it encompasses the possibility of an anti Gates earning negative millions per year?
Is that an accurate summary of the error? Or am I still not getting it exactly.
As Labonne points out, the average voter on this issue is far ahead of the opinion-leaders, media, Democratic Party leaders, IR professionals, and othe members of the elite. Sometime someone should talk about the significance of this for the theory that populism is a bad thing and that populists are mostly murderous bigots.
I’ll probably just make a fool of myself here, but I think Patrick’s example doesn’t quite get at how bizarre Kane’s reasoning is. After all, it is possible to earn negative millions per year. In a certain kind of farming community, it might not even be that unlikely that someone does this. Negative deaths per year is somewhat less plausible.
To be sure, Patrick’s underlying point that an extreme outlier in one direction doesn’t thereby make it probable that there are extreme outliers in the other direction is, I think, basically right. But the analogy he raised isn’t I think as absurd as what he was trying to analogise.
I’m struck by Kane’s defense of giving permission to Malkin to print his paper; at Deltoid he writes, “To be clear, I did not seek Malkin out. She is a Deltoid reader (and who isn’t?) and contacted me. Should I have refused her permission to reprint? If someone contacts me from DailyKos, should I refuse him permission?” I am a college instructor, and on my syllabus I include a statement about how racist (classist, sexist, etc.) speech will not be tolerated in my course. I would assume that a scholar’s opposition to racism should carry itself over their work outside of the classroom. Even the most cursory examination of Malkin’s site would reveal that she routinely spouts juvenile racist trash (generally about Muslims). Why is a college instructor allowing his material to be reprinted at her site? The problem isn’t just that her audience doesn’t understand statistics; it’s also that much of her audience is also committed to at least two profoundly racist principles: 1. that the deaths of Muslims do not matter much (Kane’s work makes it that much easier for them to ignore the reality of Iraqi suffering) and 2. that the United States is a benevolent empire that will redeem the evil Muslims (the standard racist colonialist narrative, which in diminishing the humanitarian cost of the war, Kane’s article also can be used to support). I would think a more responsible scholar would not provide academic cover for Malkin.
Patrick – Bill Gates would raise the average of 100 people by a lot more than $1k but basically yes.
There are some situations in which it would be reasonable to make something like this assumption – if you find a few chocolate boxes that have been overfilled, then you might reasonably surmise that quality control is to hell and some others have been underfilled. But this isn’t one of them and I think that’s pretty obvious.
But am I getting the basic idea?
I have a strong technical proficiency in college math, but I’m not so knowledgeable on real world statistical practice. There’s an error in Kane’s reasoning, and I get the basic idea that he’s re-casting the data in such a way that he gets an extremely high standard deviation which allows his confidence interval to encompass a negative change in death rate as a possibility. Is that correct? Or is there an added step of bad math that I am missing.
I’ve been thinking about these examples too. Mine is:
you want to determine the average co2 emissions for air transport of the average american. You do a survey of 33 people asking “what is the furthest you have ever travelled in an airborne vehicle”, then multiply their given distance by the average co2 emisssions for a plane (obviously this paper is only getting published in a crappy psych journal, which David will be lucky to do). Your first 32 respondents represent the average american to a t. The 33rd is Buzz Aldrin.
As a consequence of getting Buzz Aldrin in your sample, variance is huge, and the confidence interval for co2 emissions includes a large chunk of negative emissions.
bingo! US air travel is actually absorbing co2
That’s the David Kane school of statistical analysis at work.
bingo! US air travel is actually absorbing co2
more precisely, bingo, you cannot reject the hypothesis that US air travel is absorbing co2 at the 95% confidence level, as Buzz Aldrin would obviously also massively push up the average, rather as Fallujah does.
Indeed, dsquared, but by the time it reaches Michelle Malkin it’ll be all “we absorb co2! Fly more planes! kill greenies!”
sg-not quite. Travel distance can’t be negative, the relative death rate could be. Some argued for the war cough dsquared cough based upon the sanctions being so bad that not doing anything was horrendous. Now the problem with Fallujah is that its so high to assume normality puts positive probability weight to negative death regions.
Here’s Kane on the paper’s mass distribution:
From EphBlog.
I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.
What’s the chances of that?
I honestly believe that anyone who tries to present this to a room of statisticians will be lucky to escape with his life.
You overlook that David Kane served as an officer in the United States Marine Corps from 1988 to 1991. (1)
Those flabby statisticians wouldn’t stand a chance against a trained killer.
Strange coincidence, isn’t it, that two of the loudest Lancet critics – David Kane and Michael Spagat – have both had connections with the U.S. military, with the latter bankrolled by military contractor Radiance Technologies.
When you stats wizards have finished shredding Kane’s latest screed perhaps you could calculate the odds of that happening ;)
Incidentally, did Spagat and Co’s MSB paper ever get reviewed and published, or is it sitting gathering dust on a shelf marked “Job Done”?
(1) http://www.iq.harvard.edu/People/people.php?info=166&sub=6
Some of the original statistical research that documented the relationship of smoking and cancer was published in Lancet and was met with ferocious criticism, most of it paid for by tobacco companies. The Lancet studies of excess mortality in Iraq have met a similar reception. In both instances, people who never gave a damn about the niceties of statistical inference were instantly sure that filling lungs with carcinogens or countries with bombs and bullets couldn’t possibly have a bad overall effect. Unfortunately, cigarettes cause cancer and wars cause excess deaths.
I see someone else made something like this point, but isn’t there technically a possibility of a negative relative death rate, which is what the study was about? Just as Fallujah went from being a (perhaps?) relatively ordinary town to one with massive conflicts, it’s at least conceivable that there could have been towns with massive conflict or gas attacks or whatever beforehand but settled down to just the average level afterwards. Of course, I don’t think that’s likely at all, but at least the negative figures aren’t the equivalent of a negative death rate.
Or am I just horribly mistaken, because I haven’t read either paper?
Kenny Easwaran and rob ask about “relative” death rates. Kane’s argument is about the crude (i.e., absolute) death rate.
dsquared wrote:
That was my first impression, too, but once it got out into wingnut-land, I would think the proper response now changes to, “David, please hand this paper out“.
That was an interesting thread at Deltoid, much of which was way over my head. However, re the heart of the matter, at one point David Kane seems to say that he believes there were 100,000 violent deaths in Iraq in the timeframe of the Lancet study, which as somebody else puts out, would surely push the excess death total up to at least 300,000, given other similar war situations – for instance, children dying for want of doctors who have been killed, etc.
That seems simple enough. Malkin should be crowing about the fact that she has found someone who thinks excess deaths in Iraq, excluding the last year, are 300,000 +. Which is 250,000 more than Bush admits.
The case for this war has fallen apart so completely that the only pro-war argument which is really convincing involves two steps, the first one of which is getting a lobotomy.
You ignorant liberals are casually and foolishly ignoring a dangerous possibility: Saddam Hussein’s Secret Zombie Technology may have been unleashed in Iraq!
Rob, the paper isn’t positing negative relative death rates, nor negative rates of population change. It’s really positing negative absolute death rates, in the left tail of the blue graph in Fig. 3.
“Those flabby statisticians wouldn’t stand a chance against a trained killer.”
Nah. (a) we fight dirty, (b) we’d outnumber him, and© some of us (not me) aren’t flabby.
They could beat him approximately to death.
Anyways, I’m genuinely surprised now that I’ve looked at the Malkin article post. It genuinely has a graph of death rate per 100k that shows a possibility of the death rate being negative. Not a graph of the rate of change of the death rate, a graph of the absolute death rate. It does not take an advanced degree in statistics to remember to review your projections to see if they’re still within the range of valid values for your particular population.
In fairness, he does note that you can’t have a negative result. He then dismisses the concern by claiming that the results would be similar if you used a truncated curve. He doesn’t explain how or why, or show any math, or explain what he means by truncated.
… sampling and polling and the like are similar to a cook taking a spoonful of their well-stirred soup to determine how it tastes …
…results would be similar if you used a truncated curve…
I think it’s like the man of the thread said: if your math (e.g. your assuming a normal distribution) leads you to the conclusion that there’s a non-zero probability of a negative mortality rate, then it may already be too late to start truncating.
I have asked Daniel to correct this false claim about my paper.
“based on a fundamental misreading of the paper (in that it assumes that the relative risk rate was estimated parametrically using a normal distribution when it wasn’t).”
That’s not true. Nowhere does the paper claim this. Why would it, since I know that a bootstrap was used to estimate the relative risk. Since Daniel may not be available, perhaps another CT author could correct this. There may be other flaws in the paper, but this is not one of them.
The scientific discussion is continuing over at Deltoid. All are welcome.
If that’s the case, David, then I fail to see the point of your paper. They estimate the relative risk by a completely different method. How are any of the calculations in your paper relevant?
re #44 I have refused to “correct” this in email to David since his paper[1] does not mention the word “bootstrap” at all and its conclusion (that the null hypothesis RR=1.0 cannot be rejected) is utterly dependent on assuming something other than the bootstrap.
[1] Or at least, the version that I am commenting on – he might or might not have changed it since then, but this is one of the things I am complaining about.
Daniel,
I am afraid that you have gotten a couple of important things wrong.
Under David Kane’s assumptions, the discovery of the Fallujah cluster was a reason to believe that things might have gone better in Iraq. This clearly means that these were the wrong assumptions.
Kane never asserts anything like this. In fact, he asserts exactly the opposite. The study is so flawed that, even though death rates have obviously increased, the study cannot mathematically exclude the possibility that death rates have stayed the same or even decreased.
The Fallujah datapoint increases the uncertainty of the estimate, but it doesn’t increase it in both directions, because there is no way that you could find an “anti-Fallujah” (a datapoint which brought the overall average down by as much as real Fallujah brought it up), because such a place would need to have a negative death rate.
Are you sure you wish to claim this? This means the study is inherently biased to exaggerate deaths because a cluster can only increase but never decrease the death rate. In other words, the study by design could only discover whether the death rate stayed the same or had increased. It could never detect whether the liberation saved lives. This would make the study fraudulent on its face.
If, however, the study could detect an improvement in mortality then Kane’s assumption of symmetrical uncertainty is valid and thus confirm his math.
Which is it? Is the study design inherently flawed or is Kane right? I know the answer according the study itself. Can you guess what it is? (Hint:try page 3)
AT DELTOID:
Posted by: David Kane | July 27, 2007 01:35 PM
In other words,
Strawmen are good at scaring crows? Indeed.
You appear to be arguing that the failure to observe mass resurrection in Iraq invalidates the study.
What a bunch of nonsense, Shannon. You think you’re defending the author but instead you’re embarrassing him. That’s painful to watch, man.
Shannon, I think you need to bone up a bit on the concepts “zero” and “positive” before discussing statistics again.
BTW, who the f*ck is Shannon Love? I had sorta thought that the Chicago Boyz blog was composed of Chicago U amlumni, or at least Econ Ph.D.’s of that school of economics. But I can’t find much in a few minutes of searching.
nicks,
You appear to be arguing that the failure to observe mass resurrection in Iraq invalidates the study
No, I am arguing that study as outlined in the paper could have detected whether the mortality rate improved, stayed the same or worsened. If the study could not do this, then it could never detect whether mortality had improved.
If you think real hard, I am sure you can think of way that people come to be alive beyond return from the dead. Les Robert et al were in fact smart enough to structure their study to capture this somewhat less than mystical effect.
Christ almighty you people are ignorant! This is obviously why the UK had to release killer badgers in Basra – to hunt down all the zombies. Duh!
john quiggin,
I think you need to bone up a bit on the concepts “zero” and “positive” before discussing statistics again.
No, you need to understand the difference between a ratio, which cannot be negative, and the difference between to ratios, which can.
Sorry, last post should read:
No, you need to understand the difference between a ratio, which cannot be negative, and the difference between two ratios, which can.
Shannon, love
why? Which two ratios are we differencing?
The fact that a high data point cannot make the results go lower is not a flaw in the study. The fact that a highly divergent data point cannot widen the range of the studied value past the leftmost bound of the physically possible range is also not a flaw in the study. If you cannot see this, then you have no business discussing statistics.
And his graph that includes a tail where the death rate is negative is NOT a graph of rate of change of death rate. It is explicitly a graph of the death rate. It says so. On the graph.
dsquared writes:
[1] Or at least, the version that I am commenting on – he might or might not have changed it since then, but this is one of the things I am complaining about.
1) This footnote is ridiculous. We are all talking about the same paper. It is still posted at Deltoid.
2) The fact that I do not use the work “bootstrap” is evidence of nothing. You assert that I assume X. You need to quote me assuming X. You can’t. I don’t assume it so I don’t write it. I challenge you to provide a quote.
3) You claim that my conclusion is “utterly dependent” on this assumption which I do not make. How so? I provide a series of steps. I make many specific claims. I use all sorts of equations. Nowhere do I say or imply or assume the specific estimation method used by the authors to calculate the RR. The method is irrelevant to my argument.
I feel like you have just claimed that I have assumed in the paper that all unicorns are purple. Your evidence? I do not use the word “pink” and my conclusion is “utterly dependent” on unicorn-coloration. How should I refute such gibberish?
sg,
Which two ratios are we differencing?
Crude Mortality Rate Post-invasion minus Crude Mortality Rate Pre-invasion.
or CMRpost-CMRpre=deltaCMR.
Its evident that deltaCMR can be greater than zero if mortality after the invasion increase, equal to zero if mortality stayed the same or less than zero if mortality decreased.
Daniel is confusing relative risk with death rate. Relative Risk(RR) is calculated by CMRpost/CMRpre. RR can assume any value greater than zero. If the value is greater than one then the death rate increased, if less than one it decreased.
So, RR doesn’t have to be below zero but only below 1.
Kane bases his argument not on the variance of the RR between clusters but on the variance of deltaCMR which in turn determines whether RR>1 for the entire study. (see figure 3)
If ∆ CMR is less than 0, then the RR is less than 1 because
CMRpost is less than CMRpre. Given the data, there is a10%chance that ∆CMR
Common sense suggest that if no measurement in another cluster could even theoretically offset the Falluja cluster then the studies design would be flawed from the get go. The idea that the death rate or relative risk can only go up is silly on its face.
Shannon, love,
as I pointed out on deltoid, given the data there is also a near-zero probability that the point estimates of CMR pre and CMR post are the same. It depends on your choice of starting point for David Kane’s argument (he hasn’t responded to this yet). This is due to the poor behaviour of the confidence intervals.
Also shannon, love, CMR pre and CMR post are ratios in name only – they have the same denominator. So really you are talking about the difference of two numbers (the absolute death counts).
And finally shannon, love, it is also possible to have a contradiction between a Relative Risk confidence interval and the confidence intervals of its contributing counts. It’s very easy to see this if you take the time to construct some figures.
David Kane’s paper confuses or misunderstands all these points. And that’s on top of all the misunderstandings dsquared has described here.
Patrick,
The fact that a highly divergent data point cannot widen the range of the studied value past the leftmost bound of the physically possible range is also not a flaw in the study.
What are you talking about Relative risk of deltaCMR? In neither case does Kane push the value past the leftmost bound. DeltaCMR has no leftmost bound because its left and right bounds are utterly symmetrical. RR has no left most bound because, as a ratio its value is zero or greater by definition.
And his graph that includes a tail where the death rate is negative is NOT a graph of rate of change of death rate. It is explicitly a graph of the death rate. It says so. On the graph.
Not sure what you mean. Fig 2 clearly has a caveat that the graph is not wholly accurate because CMR cannot be
man i’m glad i don’t do social science.
sg,
…given the data there is also a near-zero probability that the point estimates of CMR pre and CMR post are the same. …
The point estimates may differ but we know that CMR pre and CMR post are identical in the majority of clusters because the majority of clusters report zero excess deaths.
…CMR pre and CMR post are ratios in name only – they have the same denominator.
Umm, definitely wrong. See the 2004 Lancet paper p3 pg which define both CMRs as (number of deaths recorded/number of person-months lived in the interviewed households). That makes each CMR a true ratio. Beside rates are ratios by definition.
So really you are talking about the difference of two numbers (the absolute death counts).
No, the study document excess deaths attributable to chance in CMR pre and post. It calculates that by multiplying deltaCMR by the population.
it is also possible to have a contradiction between a Relative Risk confidence interval and the confidence intervals of its contributing counts
Perhaps, however, the confidence interval for deltaCMR is dependent wholly on the confidence interval of CMRpre and CMRpost. If the CI of deltaCMR dips below zero (allowed because it is not a ratio) then axiomatically that means that a chance exist that deltaCMR CMRpost. If that is true then a chance exist that RR
Indeed, it could be shown that no place on earth could ever have been struck by a meteorite or an asteroid because no event, or its absence, could theoretically offset it.
The dinosaurs live!
No matter what inferences one draws from them, the studies themselves did not give any value judgements, and in particular didn’t attempt to say ‘who killed who’.
It’s all there in black and white, unless I’ve horribly misunderstood.
Posted by Suvi · July 27th, 2007 at 12:10 pm
Here’s an article where practically nothing but such attribution is stated… like 5 times in 6 paragraphs…
http://www.smh.com.au/articles/2004/10/29/1098992290312.html
Shannon, love, I’ll grant you the ratio comment, but the substance of your “perhaps” is somewhat misleading. You can do the calculations with RR for simple cases to show that one can indeed have a negative deltaCMR, but a RR that contradicts this – or the opposite.
This is probably also the reason why David Kane’s proof in section 1.0.2 will give contradictory results depending on the starting point. In general, a proof which does that is not very binding.
You assert that I assume X. You need to quote me assuming X.
No, I can also cite a paper you wrote that makes no sense whatsoever other than on the assumption of X.
The study is so flawed that, even though death rates have obviously increased, the study cannot mathematically exclude the possibility that death rates have stayed the same or even decreased
Ey oop, it’s Shannon Love! Do you really think you had such a good time you tried to bluster statistics with me that you thought you’d do it again?
No. The study is not flawed in this regard, because it uses sensible, robust methods to determine the confidence interval. Kane’s methodology uses a silly, nonrobust method in which a positive outlier like Fallujah extends the lower end of the confidence interval symmetrically with the higher end. Kane then attributes this methodology to the paper in order to claim his “inconsistency”. He is now trying to backpedal and claim he never did this, but yes he did.
Are you sure you wish to claim this? This means the study is inherently biased to exaggerate deaths because a cluster can only increase but never decrease the death rate.
No, and I second John’s helpful advice that you think before engaging mouth in future. The addition of any cluster with a lower death rate than the previous average will lower the death rate. All clusters other than Fallujah bring the average down compared to Fallujah. However, no single cluster can lower the average by the same amount that Fallujah raises it because the death rate cannot be lower than zero.
If there were loads and loads of clusters with a postwar death rate of 1 or 2 per 100k, then it would be perfectly possible for this survey to report a fall in the death rate. There are not loads and loads of such clusters, so the study correctly reported a rise in the death rate.
dsquared writes:
Perhaps we could invite John Quiggin or Kieran Healy to settle this dispute? Even though they are probably not fans of my work, I would trust their honesty on this.
You claim that I assume that the RR is not estimated with a bootstrap. Nowhere do I make this assumption. I only assume that the RR estimates and confidence interval reported by the paper are accurate. I use those numbers without regard to their origin. I challenge you to quote something from the paper to the contrary, but you can’t because there is nothing there. You now claim that the assumption must be there, albeit hidden from view behind some curtain, because the paper makes no sense without it. Fine. Demonstrate this claim! Show how the paper requires this assumption.
It doesn’t and so you can’t.
I apologize for calling you out on this, but, as your misunderstanding about the (lack of) use of the bootstrap in estimating the CMR demonstrates, you can make mistakes on this topic. Nothing wrong with mistakes, of course. But the honest thing to do is to correct them.
I challenge you to quote something from the paper to the contrary, but you can’t because there is nothing there.
Your calculation on p6 and your table forming figure 2 do assume a normal distribution of the data including Fallujah. This is why you get different numbers for the excess deaths excluding Fallujah from those presented in the paper. Your charts all assume normal distributions – this is visible from simply looking at them. Your key conclusion is based on a chart which visibly has a chunk of its probability mass below zero. There is no correction to make here.
This is rather as if I had said “Kane believes that weasels are about to take over the world”, based on your having bought a weaselling gun, moved your home and possessions into a weasel-proof shelter and formed a committee called WEASELPAC to direct contributions to the Presidential candidate who promised to address the weasel problem, and then you challenged me to find a sentence where you had said “weasels are about to take over the world”. Your paper has no point to it at all without this assumption. You repeatedly claim that reported figures within the paper are “inconsistent” when they are only inconsistent on the basis of that assumption. You can ask as many times as you like but the answer is still going to be no.
By the way, on the Deltoid thread, halfway down, you have made the claim that Les Roberts carried out the estimation of excess deaths including Fallujah based on a normal distribution and then refused to print it because it didn’t reject the hypothesis. So this prissy attitude to your own precious paper is ill-placed.
I’m back, and I have to get back to my initial point.
Yes, it’s true that the study itself does not make the mistake of moral attribution I’m talking about, but it IS made by just about everybody who seizes on the study as a club with which to attack the decision to invade Iraq. So it’s hardly irrelevant.
And I must insist that “You break it, you bought it.” can not apply to a situation where the damage is actually the result of decisions by other people, full blown moral agents in their own right, to go out and commit murders.
By this sort of reasoning, if a black couple moves into a white neighborhood, and the local KKK are so offended that they start blowing up daycare centers and shopping malls in an effort to bully the local populace into ejecting them, it’s the black couple who are at fault.
Nope, no way, this is invalid moral reasoning. And it’s the sort of moral reasoning that most people who care about the Lancet study are engaging in.
This is silly, Brett, and has been gone over many times before. Moral responsibility is not zero sum. If you start a war, with the foreseeable (and foreseen, and much derided and now, in this thread, denied) consequence that hundreds of thousands of people are killed, you’re responsible for those deaths. Equally, so are those who decide to fight you.
The practical consequence of your claim is that anyone who believes they have a just cause is entitled to go to war, and blameless for the consequence, since the resulting deaths will be the fault of the other side for unjustly resisting.
Leaving this aside, you have some responsibility to the truth. Instead of engaging in these apologetics won’t you at least have the decency to admit that the war has caused all these deaths and that those denying it, for whatever reason, are wrong to do so.
Brett, your comparison of an innocent black family to the illegal invasion of Iraq doesn’t work for a number of obvious reasons; I’m not going to waste my time with it. I will simply note that the particular “moral reasoning” you are advocating assumes an extreme form of free will that most intelligent people find ridiculous. You are really just wallowing around in a simplistic form of the compatibilist / noncompatibilist free will debate. In The View from Nowhere, Thomas Nagel shreds your argument to pieces, and even Christine Korsgaard who is a staunch defender of free will (see her Creating the Kingdom of Ends) steers clear of your particularly absurd form of it.
dsquared writes:
False! (I assume you mean table 2 and not figure 2.) I am merely trying to replicate the (unreleased) formula by which the authors got 98,000 CI (8,000—194,000). I propose a formula. (If you know what the formula is, please tell us. The authors won’t tell me.) The formula has nothing to do with a normal distribution. It simply takes the CMR data given in the paper and plugs it in. Even if I did however, this formula and table do not have anything to do with RR. Nothing. RR does not enter in in any fashion. How can a formula and table which do not use RR demonstrate what assumption I am making about RR?
The authors used, as they clearly state, the CMR to calculate excess deaths, not the RR.
True. But this is correct! Consider figures 2 and 3. These are accurate descriptions of the CMR estimates. The authors have told us that they use the normal distributions for these. Those figures are spot on. Pick a specific one you object to and we can go throw it in detail. And, even if this were not true, figures 2 and 3 have nothing to do with RR. How are you able to deduce my assumptions about RR by looking at a figure about CMR, a figure which just describes graphically what the authors report?
Your weasel analogy is amusing, but if you can’t come up with a singe example that remotely illustrates your claim, then shouldn’t you retract it?
John, Steve, mathematics may represent the one realm where humans have access to ultimate truth, and can objectively demonstrate it. Ethics isn’t, it’s a realm where the very basics are fundamentally contested.
You expected, in the sense of prediction, that elements in Iraq would respond to our replacing Saddam’s tyranny with a democratically elected government by going on a killing spree. I expected, in a normative sense, that they would participate in elections. I maintain that, if you’re going to assess the moral aspects of an act on the basis of how people react to it, the normative sense of expectation is the relevant one.
And my analogy is spot on. We did good, and now the bad guys are trying to commit evil on a sufficiently vast scale to undo the good we did. This makes them monsters, not us, and the more they kill, the more monsterous they are.
David Kane is, of course, a weasel and an especially immoral weasel at that, but carry on with the destructive idiocy David.
Brett Bellmore is simply a radical right trolling weasel, crazy as crazy can be.
What these weasels have in common is justifying our immoral war in Iraq and our more immoral occupation of Iraq.
Brett Bellmore, as we can see in his insane last comment is simple a moral monster.
dsquared also wrote:
This discussion would be more productive if you would specify which figure you are talking about. Figure 3? Again, there is nothing in this figure about RR. Nothing. I graph the two normal distributions that the paper provides and then the difference between them (specifying in the text that I am assuming zero covariance). Surely, you agree that the difference between two normal distributions is itself normal. Surely you agree that I have done the subtraction correctly.
While it is true that the resulting graph allows CMR to be negative, this is exactly what the authors allow in the original paper. They use the normal distribution. You may think that this is a critical flaw and that the paper may be ignored on that basis. I may think that this is OK and that other choices would produce similar answers.
But none of that is germane to the question of whether or not I assumed that the RR was not bootstrapped. I did not assume that. Please correct your claim to the contrary.
David,
Robert (who managed to replicate Roberts’ results) gave you a confidence interval for the RR including Fallujah in the Deltoid thread: (70 000, 975 000).
As you ought to expect, including Fallujah increases both the width and the lower bound of the CI.
I notice you are still referring to the CIs for the CMR estimates as simply “the CMR estimates”. This is one of the things d^2 complained of above.
The distinction is critical, because:
(a) the CMR estimates are relevant to the study’s results
(b) the CIs for the CMR estimates are wrong but are irrelevant to the study’s results.
I think you have successfully demonstrated (b); unfortunately that is not an important result.
You’ve outdone yourself, Brett. I’m in awe. Events would have beaten some sense into a lesser man’s head by now.
“You will be expected to respond positively to our efforts to do you good. If you insist on acting like unruly schoolboys you will be soundly thrashed.”
What a sound basis for military planning!
Reading that thread on Deltoid, it does seem to be interesting the way that some people seem to have an absolute mental block on the idea that people could have died in Iraq before the official start date of the war. For them, the possibility of Saddam or his agents carrying out a massacre, shelling a village, or setting off a bomb, was zero, known to be zero, could not ever in any possible universe be non-zero.
In sarcasm, they even use the possibility of ‘resurrection’ as something that is inherently more likely than Saddam killing some Kurds.
Note: the discussion does get a bit complicated by fact that there is similar but unwrong argument that the post-war death rate cannot plausibly be a world record low, even if the detailed mathematics assigns a non-zero probability to a small clustered survey reporting that.
No Brett, we did not do “good”. We did a “war crime.” And if you’re naive enough to not understand nationalism I feel sorry for you. Anyway, here’s the death of your simplistic moral agent (sorry about the long quotes, but these ideas do kill one objection to the Lancet):
Thomas Nagel’s View from Nowhere (assume his “external view” corresponds to the viewpoint of the Lancet authors while “our natural human point of view” corresponds to our sense of ourselves as free moral agents): “In ordinary judgments of responsibility we do not go that far outside, but stay in our natural human point of view and project it into that of other, similar beings, stopping only where it will not fit. But judgments so based are vulnerable to the more external view, which can take in both the defendant and the judge. . . . The judge’s sense of the defendant’s alternatives is revealed as an illusion which derives from the judge’s projection of his own illusory—indeed unintelligible—sense of autonomy into the defendant.” (123). “[T]he external standpoint is always there as a possibility, and once having occupied it we can no longer regard our internal judgments of responsibility in the same way. From a point of view that is available to us, they can suddenly seem to depend on an illusion—a forgetting of the fact that we are just parts of the world and our lives just parts of its history” (124). This would suggest that an adequate moral theory must take into account that we are parts of the world and not merely actors within it.
In Creating the Kingdom of Ends, Korsgaard similarly stresses the difference between a holding another person responsible for their actions and that person in fact being responsible for their actions (which if Nagel is right, no one ultimately is). You hold people responsible when you are in a kind of personal relationship with them: “For instance, it may be perfectly reasonable for me to hold someone responsible for an attitude or an action, while at the same time acknowledging that it is just as reasonable for someone else not to hold the same person responsible for the very same attitude or action” (199). “In everyday personal interaction, we cannot get on without the concept of responsibility. And therefore we cannot rest with the view that agents take responsibility for their own actions but can refrain from judging others” (197). Brett, since I doubt that you have any “personal interaction” with Iraqis who are violently responding to the criminal destruction of their society around them, I wonder why you are trying to hold these Iraqis responsible for actions that stem (though not ultimately) from Bush’s illegal decision to invade Iraq.
kilo,
No matter what inferences one draws from them, the studies themselves did not give any value judgements, and in particular didn’t attempt to say ‘who killed who’.
From the 2004 Lancet paper:
Interpretation: Making conservative assumptions, we think that about 100000 excess deaths, or more have happened
since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths.
Looks like they are assigning blame to who killed who to me but as dsquared will tell you, I’m a little dense.
Those two sentences in combination are a lie by the way. No single data set supports that interpretation.
Shannon Love writes:
Those two sentences in combination are a lie by the way. No single data set supports that interpretation.
Shannon, do you have evidence that proves the claim you cite to be incorrect? I’m assuming that since you assert that the claim is a “lie,” you have evidence to clearly establish that the claim is false. Will you reference this evidence, please?
The formula has nothing to do with a normal distribution. It simply takes the CMR data given in the paper and plugs it in.
This would only be a legitimate calculation if one was making a parametric assumption about the confidence intervals. You can’t just randomly “plug in” confidence intervals like this, unless you’re making a lot of assumptions.
dsquared,
The study is not flawed in this regard, because it uses sensible, robust methods to determine the confidence interval.
How can you tell? The authors never provide a confidence interval for the dataset including the Falluja cluster. They provide a CI for the data set excluding the Falluja cluster but not for the one including it. Kane is forced to infer the CI from the CI of rates of Relative Risk. If the authors had followed standard practice in the first place we wouldn’t be having this conversation.
However, no single cluster can lower the average by the same amount that Fallujah raises it because the death rate cannot be lower than zero..
No, because the study NEVER compares death rates between clusters. Instead it effectively compares changes in the death rate pre and post invasion. The delta can obviously be negative.
The entire notional point of the study was to determine the CHANGE in death rates. As such, any cluster that showed improvements in mortality post invasion could offset the Falluja cluster. In fact, if you look a fiq 1 (p 3) of the 2004 paper you will see that the province of Sulmamanhya shows a fairly pronounced improvement in mortality. It is easy to see visually that it could offset the increase of mortality in Falluja if it was larger.
If the study did actually average death RATEs between clusters then yours and Daniel’s argument that the distribution cannot be expected to be symmetrical would be true. However, since the study averages numbers, which can assume any value, we HAVE TO assume a normal distribution unless the authors say otherwise, which they do not. (see p3 pg4 – p4 pg1)
Don’t feel to bad. I made the same mistake the first time I read the paper. IIRC, it was you or Daniel that corrected me on it.
Having won this battle, perhaps Shannon and David should shift their energies towards showing that electric production in Baghdad may be higher than before the war, rather than much lower as the lying media claim. I fear that they’re focussing their energies too narrowly on this particular question, while letting dozens of other media lies pass by unrefuted.
They might also take a look at the bogus claims that more than two million Iraqis have fled the country, and all the silly talk about dead bodies. It seems pretty likely that the dead bodies are just leftovers from the Saddam regime which are being trucked around to be discovered again and again and again.
I note the absence of papers commenting on the possibility that the Lancet paper underestimates excess deaths in Iraq, just as in the tobacco/cancer controversy, the hired hands never complained about the conservatism of the assumptions made by statistical researchers who, like the authors of the Lancet study, had bent over backwards to hedge their results.
What we have here is another in the long series of asymmetrical arguments between people who believe that you ought to try to figure things out whatever conclusion you would prefer and those for whom the game is simply to fight fiercely for your ideology and/or employer regardless of the rules of evidence or, for that matter, ordinary morality. Critics of the Lancet study “knew” it was flawed long before they read it, just as tobacco publicists “knew” that the statistical arguments associating lung cancer and cigarettes were wrong and the Creationists and holocaust deniers—that is deniers of earlier holocausts—”knew” that the historians were wrong.
The point isn’t that people shouldn’t criticize the methodology of the Lancet study, but its most vocal critics aren’t fooling anybody. Which is why, even in this thread, some of them are playing the old game of “I didn’t steal the pig. It was my pig, and anyhow, it wasn’t very big.” Logically, of course, the conclusions of the Lancet study do not imply that the invasion of Iraq was a moral or political error. All those deaths may have been well worth it, at least from a sufficiently blooded minded point of view. And one can certainly imagine that a great many excess deaths may have resulted in the absence of an invasion by some unspecified mechanism. Unfortunately, these considerations are just ways of changing the subject.
Oh Shannon, where are those spas in Iraq where the death rate has decreased so sharply? We can point to several places which were knocked flat like Fallujah where the excess mortality was extreme.
The other point is that while the survey was about EXCESS mortality, Kane’s idiocy comes up with a range of confidence intervals that includes NEGATIVE MORTALITY, probably the zombies rising from the dead and fighting for the central government and getting paid for it. Chicago Iraqi style.
The authors never provide a confidence interval for the dataset including the Falluja cluster
that is because it would not be sensible to do so, and they are sensible.
In fact, if you look a fiq 1 (p 3) of the 2004 paper you will see that the province of Sulmamanhya shows a fairly pronounced improvement in mortality. It is easy to see visually that it could offset the increase of mortality in Falluja if it was larger.
If you look at it closely, however, you would see that if the improvement in mortality was much larger, people would be coming back from the dead, which is kind of my point. Decreases in the death rate can only be so big, because they can’t be more than -100%. Increases in the death rate can be much more than +100%, and in Fallujah they were.
Since normality is obviously a flawed assumption here can’t you just assume a log normal distribution?
What gender is Shannon Love? There’s a comment I feel impelled to make, but I can’t formulate it properly without more information.
As I understand it – and the math is complicated – the point Love is making is based on an important distortion. The Lancet study assumed a death rate in pre-invasion Iraq based on, I believe, a UN estimate. Then they surveyed to see what the change in the death rate was. They discovered an upward slope. Love is claiming that they did not look for a downward slope. I guess this amounts to saying, are there more doctors in your area to which to take your kids? Now, there might be some plausibility in Love’s claim for the first couple months of the invasion, in 2003 – a period before the insurgency got going, when the sanction regime collapsed – but after that, his claim becomes increasingly absurd. The math is, after all, tied to real human causes. One might think, well, the violence fell in other areas of Iraq during the battle of Falluja because all of the insurgents streamed into that town, or something, but even a hasty perusal of the newspaper reporting in November and December 2004 would show the absurdity of that supposition. Plus, of course, the population expelled from Falluja – around 300,000 – had to find hasty quarters somewhere, since the U.S. didn’t provide any, and in that dispersal no doubt you could also find the excess deaths that come when, say, a seventy year old has to make a series of twenty mile marches to find food.
Love is rather distorting Kane’s work. As the thread on the Deltoid site shows, Kane himself is estimating about 100,000 violent deaths, and one can make a rule of thumb estimate that such deaths have to be multiplied to get an excess death total, for obvious reasons.
However, the reasons will never be obvious enough. While the pro-war people are pretty confident about the number killed by Saddam Hussein, and have never examined the numbers at all or how they were arrived at, they will never admit the bloody massacre unleashed by the invasion and the consequent decision of the CPA to liquidate security in the whole country, followed of course by an insurgency in which criminal gangs mixed with genuine political resistors, and punctuated by absurd punitive expeditions by the Americans, culminating in the war crime of Falluja.
I’m pretty confident, however, that if these same pro-war people lived in a community in which the mayor simply erased all security and turned a blind eye to crime, these pro-war people would BLAME THE MAYOR. While plainly lobotomized into perfect Bushbots when it comes to the suffering of other people, they are keenly aware of their own aches and pains.
dsquared claims
This is just false. Let x have probability distribution p(x). If I tell you nothing about p(x) except that the 2.5th percentile is 1.4, you can still manipulate that number linearly (which is all that I do). In other words, the 2.5th percentile of 2*p(x) is 2.8. The 2.5th percentile of 10*p(x) + 3 is 17. I need assume nothing about p(x) to derive linear transformations of specific percentiles.
I make no assumptions. And, even if I did, I am making no assumptions about the RR. Nothing in that calculation mentions RR. You claim that I assume that the RR was not estimated via bootstrap. Nowhere do I assume that. Care to try again? I recommend just correcting the original post. It’s OK, we all make mistakes.
John Emerson,
You appear to argue that because you know what is right, the actual accuracy, if not honesty of the study, does not matter? You only care that the study appears to confirm you preconceptions?
I don’t suppose it has occurred to you that wars come and go but that our scientific institutions form a major part of the core of our civilization. What happens when they become so corrupted by politics that they can no longer function? If people begin to widely expect that researcher’s political bias exert more influence on their conclusions then they will lose confidence in the institution. Then we will be back to settling disputes with theology.
Soru In sarcasm, [commenters on Deltoid] even use the possibility of ‘resurrection’ as something that is inherently more likely than Saddam killing some Kurds.
This isn’t true. The Lancet paper allows for the possibility death rates went down, and none of David’s critics deny that this is theoretically possible. When people are talking about ‘resurrection,’ they are saying that death rates can’t be negative, not that excess deaths can’t be negative. Believe it or not, the problem with David Kane’s calculations is that they allow death rates to be negative. Again, negative death rates aren’t just unlikely, they’re impossible.
If I tell you nothing about p(x) except that the 2.5th percentile is 1.4, you can still manipulate that number linearly (which is all that I do). In other words, the 2.5th percentile of 2*p(x) is 2.8. The 2.5th percentile of 10*p(x) + 3 is 17. I need assume nothing about p(x) to derive linear transformations of specific percentiles.
Multiplying, adding and subtracting other random variables is not a “linear transformation”. You think you are carrying out a linear transformation because your calculation on p6 only has the scalar value 5.0 in it. However, this value 5.0 is the estimate of the pre war mortality rate which has its own sampling distribution. Michael Spagat has also made this mistake when trying to scale the Roberts et al (2004) estimates to match the IBC data.
Shannon, I’ll leave it to others to deal with your supposed contribution to the protection of our scientific institutions. The people competent to do so seem completely unimpressed with your work in that respect too, but I’m not able to participate in that argument.
My guess is that you have your own preconceptions, biases, motives, and theology and that your claim to scientific purity is bogus. To me your contribution seems pretty similar to all the other winger attempts to obscure the gross facts about Iraq behind a gnat swarm of quibbles and aspersions of various kinds—some sophisticated, some crude, some slanderous, some trivial, and many simply wrong. By so doing, and by stubbornly repeating discredited arguments, wingers often succeed in casting doubt on valid information. There’s a repeated pattern.
If your criticisms are bogus, as most here claim that they are, their sophistication makes them the more culpable, since very few non-professionals are able to clearly understand either the criticisms or the defenses of the study, making it tempting for laymen to believe that both sides have valid points and no one can really be sure who is right. The effect, possibly the intended effect, will be to neutralize the effect of a carefully-done scientific study.
“Chicago Boyz” is a strongly ideological site and no one coming from there can be taken seriously if they try to claim the high scientific ground.
As I have said several times in the past, the error of the “Lancet” authors was to think that carefully-done scientific work would survive a trip into the ideological swamp of American politics. It was inevitable that someone with bit of statistical sophistication would get to work on discrediting the study, and each of the little tweaks that the Lancet authors put into their study to make it more robust and unassailable provided an opportunity for a denialist to make a sophisticated-seeming criticism.
dsquared,
If you look at it closely, however, you would see that if the improvement in mortality was much larger, people would be coming back from the dead, which is kind of my point. Decreases in the death rate can only be so big, because they can’t be more than -100%. Increases in the death rate can be much more than +100%, and in Fallujah they were.
For the last time, we are not talking about death RATES but rather about the difference between death rates in two separate time frames. Each cluster produces two death rates one for before the invasion and one for after.
These two death rates are never averaged in with the rates from any other cluster! Instead, the differences are used to calculate whether post invasion, the mortality, improved, stayed the same, or increased.
Example: Cluster A. death rate prior to invasion=10/1000. Death rate after invasion=100/1000. Difference = 100/1000-10/1000=90. CHANGE in death rate=+90/1000.
Cluster B. death rate prior to invasion=100/1000. Death rate after invasion=100/1000. Difference = 10/1000-100/1000=(-)90. CHANGE in death rate=(-)90/1000.
In cluster A, 90 people MOREper 1000 die that would not have prior to the invasion. In cluster B, 90 peopleFEWER die than would have after the invasion.
Now using some very sophisticated math (90/1000+(-)90/1000) we can show that every excess death in cluster A is offset by an extra saved live in cluster B. In a population composed of cluster A and cluster B, the combined change in the death rate would be zero.
You keep saying that the paper compares death rates between clusters but it actually compares changes in death rates. The changes can be negative, zero or positive and thus CAN cancel each other out In a random sample, clusters are mathematically as likely to offset another cluster as to not. Therefore, there is no reason to suspect that Lancet calculation require an asymmetrical uncertainty distribution.
Fallujah doesn’t wreck the confidence interval because it can never be offset. It wrecks the confidence level because it is a massive outlier created by poor study design.
Eli Rabbet asks, where are those spas in Iraq where the death rate has decreased so sharply?
They’re in Kurdistan, Syria, Iran, and Jordan. The people Saddam was targeting before the war were Kurds, Chaldeans, Turkomen, and other minorities. It was called the “Arabization Campaign.” Before the war, about 1 million of them were internal refugees, and another million or so were refugees in neighboring countries.
In general, the Lancet studies exclude or undercover somewhere between 2 to 6 million Iraqis who were most oppressed by Saddam. The Lancet studies excluded refugees in neighboring countries, probably undercovered refugee camps in Kurdistan and the rest of Iraq, and used only Arabic-speaking interviewers but not speakers or any of the languages spoken by Saddam’s favorite victims.
Since the invasion, it’s Iraq’s Shiite and Sunni majority that’s suffering, so no doubt the death toll is higher than in the days when Saddam was killing and expropriating minority groups. But don’t tell me that Saddam’s body count has been carefully tallied, because it hasn’t.
To me it’s boggling to have someone from “Chicago Boyz” tell me that he’s primarily concerned with our scientific instutitions, whereas epidemiologists from The Lancet and Johns Hopkins are political hacks. This does not pass the smell test.
Kurdistan was protected from Saddam well before the invasion, and the Kurds are the largest single minority. Driving even more Iraqis into exile than Saddam did doesn’t sound like a positive accomplishment, even if they’re safe there.
Shannon, you’re wretchedly confused here. The study does not publish 33 different excess death rates. It publishes one, for the country of Iraq, which aggregates all the clusters together.
In cluster B, 90 peopleFEWER die than would have after the invasion
Since the average death rate in these clusters is about 5, no such cluster could exist. This is the whole point.
Btw, if you are going to continue to blah on about “our precious scientific institutions” and “intellectual dishonesty”, then I may become irritated enough to dig up some of your own greatest hits, and this will not reflect well on you or your website.
john emerson,
“Chicago Boyz” is a strongly ideological site and no one coming from there can be taken seriously if they try to claim the high scientific ground.
Its a group blog so there is a lot of diversity. However, I do not insist that I have no bias. I have a lot. However, I have never asserted that anyone should accept what I say because I am so damn smart and good looking.
The neat thing about science though is that individual biases don’t matter. The entire social function of science is that people from all different perspectives can arrive at the same conclusion. Its not easy and its not perfect but over the long run it works.
Science depends on honesty and disclosure so that other can evaluate and repeats the original work. By iterating this process we can create progressively more accurate models of the world around us.
The Lancet authors broke the rules. They hid and obscured data. Most damningly, they cherry picked data to create just the result they wanted. By there own admission, they rushed the study and its publication in order to influence the 2004 US Presidential election. They are trying to use their status as trusted scientist to accomplish narrow political ends.
Don’t take my word for it. Get a PDF of the study yourself and go through it thoroughly. Pay close attention to which assertions in the study such a excess deaths, cause of death, age and sex killed etc come from which data. Once you do this the cherry picking becomes obvious.
Then ask yourself whether you would like a drug company to cherry pick the data they use to prove their products safety and effectiveness.
A loss of scientific integrity might benefit you today but down the road it will come back to bite you.
Shannon Love writes:
I don’t suppose it has occurred to you that wars come and go but that our scientific institutions form a major part of the core of our civilization. What happens when they become so corrupted by politics that they can no longer function? If people begin to widely expect that researcher’s political bias exert more influence on their conclusions then they will lose confidence in the institution. Then we will be back to settling disputes with theology.
Shannon Love, assuming you believe what you’ve written here, will you please address the point I raised in comment 85? You’ve accused Roberts et al of lying. Since your statement above assures us that you make this accusation based on clear scientific evidence rather than political considerations, will you please cite this evidence now?
The Lancet authors broke the rules. They hid and obscured data. Most damningly, they cherry picked data to create just the result they wanted. By there own admission, they rushed the study and its publication in order to influence the 2004 US Presidential election. They are trying to use their status as trusted scientist to accomplish narrow political ends.
As I say, back in 2004 I was disgusted with this sort of behaviour but now I’m just amused. Back in 2004 it seemed to matter what people like Shannon Love said, but who listens to the crazy 26% these days?
Since I don’t know statistics and don’t work in statistics, I’m willing to let the questions about scientific method come out in the wash over the next few years.
The reason someone like me is hearing about statistical problems in the Lancet study isn’t because people in science are concerned about its methodological deficiencies. It’s because the topic of the study is highly controversial and the study’s conclusions are objectionable to the winger hack battalion.
If the study is bad, there could be two bad outcomes: people might come to substantially false conclusions about the Iraq War, and a bad professional precedent might be set which would be followed by others. The latter is not something I can involve myself in, but it seems highly, highly unlikely.
The former I understand better, and to me it seems impossible. I have other sources about the Iraq War, and the conclusions I come to about the war without the Lancet study are about the same as the ones I would come to with it. The small adjustments that may or may not be mandated by the criticisms of the study seem unlikely to change anything substantial, though I admit I would simply reject out of hand any tweak of the study showing that the invasion had improved Iraq’s mortality rate.
Your own accusations against the study’s authors make it impossible for me to come away from the controversy without thinking that someone is in bad faith. It seems most likely to me that the one in bad faith is Shannon Love.
Shannon Love writes:
By there own admission, they rushed the study and its publication in order to influence the 2004 US Presidential election.
Actually the authors have stated the opposite of this. If you have a quote in which they admit they tried to influence the 2004 US Presidential election, will you provide it please? Otherwise I will be left to conclude that once again, you’ve made an assertion that has no basis in fact.
dsquared,
The study does not publish 33 different excess death rates. It publishes one, for the country of Iraq, which aggregates all the clusters together.
The study actually publishes the chance of increased mortality but that is neither h