This probably doesn’t mount to all that much, but it’s been irritating me slightly for the last couple of days …
We all know that the second most dispiriting phrase in the English language is “Steve Milloy has a devastating critique …” (the first most dispiriting phrase is “My new column is up at Tech Central Station”.) The original reason why the Volokh post linked above irritated me was that it came the day after a post on Tim Lambert’s marvellous spot on the radians/degrees error in that global warming error. It rather irked me that Tim Lambert should get referenced with caveats (“Of course, that’s the claim; if there’s a rebuttal somewhere, please point me to it”) while Steven Milloy got three paragraphs of direct quotation with no caveats at all. Anyone wh knows even a little bit about the two chaps knows that Tim has always been tirelessly and scrupulously accurate, while Steven Milloy, proprietor of “junkscience.com”, is a bit of a hack, who got his start with a bit part towards the end of the single largest and most impressive work of intellectual dishonesty of the previous century, the effort to discredit the scientific work on the link between tobacco and lung cancer.
So I decided to take a look at the “devastating critique” to see whether it was really all that.
Surprise surprise, it wasn’t. In the linked article, Milloy is commenting on an article in the Journal of the American Medical Association. It’s subscription only, but I managed to get a look at a copy and a few things stuck out at me.
First, it’s a paper written using a very large sample; a panel study of about fifty thousand nurses. In particular, it’s looking at what happened to that portion of the sample of nurses who significantly increased their consumption of sugary (ie, non-diet) soft drinks. The finding is that, apparently, they tended to get diabetes at a significantly higher rate.
Milloy doesn’t seem to understand this; possibly by choice. He keeps on talking about the differences (which the study also reports) between the populations of nurses who had consistently low consumption of sweet drinks over the period and the population who had consistently high consumption. He also attributes claims to the report (such as “the simple-minded notion” that the entire weight gain reported in the switching group was due to the sugar in their drinks) which purely and simply aren’t there. This is irritating, but I’m prepared to read through it as it is most likely the result of genuine confusion on either my part or Milloy’s.
The point which really irked me, though, is the hallmark of a true Milloy hack-piece (as in, if it isn’t there, it’s not the real Milloy) – the discussion of things mentioned in the study phrased so as to suggest that only Milloy has noticed that they completely invalidate the study. In this piece, it’s a vintage example; Todd Zywicki at Volokh correctly excerpts it as the most important paragraph, and what a pity that he didn’t think about what he was pasting.
When the researchers statistically adjusted their results for bodyweight (a risk factor for diabetes) and for caloric intake (a proxy measure for consumption of sweetened foods other than soda), the 83 percent increase dropped to an even more statistically dubious (and soft-pedaled) 32 percent increase. That result is of the same magnitude as the study’s reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day
Think about this for a second. According to Milloy, the correct (even the ethical) thing to have done in presenting the results of this study would have been to have headlined the “32 per cent increase” (1.32 relative risk) that one gets in a model which controls for body mass and caloric intake. This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?
And then, assuming that we were to accept this stupid model, is Milloy reporting the results correctly? Here’s a clue; the phrase “the same order as magnitude” is a phrase with a precise meaning which is not here being used precisely. Nine is “of the same order of magnitude” as one, but if my missus was to find out I’d had nine sexual partners last month rather than one, I’d imagine she’d call it significant.
The phrase “order of magnitude” is presumably meant to convey that the sugary soft drink risk ratio is not significantly different from the diet soft drink ratio, which is a curious way to summarise the paper, because the paper explicitlysays that the sugary drinks risk ratio is significant while the diet drinks one isn’t. Anyone who thinks that a paraphresis of this sort isn’t actually “fibbing” is welcome to their view; personally, I think it’s pretty bad.
Now, I have no real brief for the medical profession on this one; it seems to me pretty intuitive that since a can of Pepsi has the equivalent of thirteen sugar lumps in it, then drinking a lot of the stuff might not be the best thing on earth for the old pancreas, but on the other hand, since everyone drinks soft drinks and there are a lot of people walking around, I’m guessing that soft drinks aren’t poisonous. But it can’t help the debate to have the likes of Milloy misrepresenting research like this, and the less it gets promoted by reputable sources like the Volokhs, the better. And when Milloy reports it as a “in my opinion … flagrant and inexcusable omission” that the researchers didn’t include a separate, tangentially related study that one of them carried out earlier, the correct response should not be endorsement, but rather a horse-laugh and a note that Milloy has a certain amount of previous form when it comes to inventing ethical codes from whole cloth.
So anyway, I think my only conclusion for this is shape up, Volokhs; if you are suspicious of a piece of scientific research, it is always better to spend the extra few minutes and find a critique that isn’t by Milloy.
If one was of a mind to defend Milloy, then one might say that the main work of junkscience.com was in fighting the battle against the link between second-hand smoke and cancer rather than the Big Evil of smoking and cancer. But as the linked .pdf above shows, he’s not been above making flip remarks about the smoking-cancer link.
Two coefficients of this kind can be as close together as you like, but if they have different standard errors, then one might be very significant and the other not at all.
You’re going to have to trust me on this one unless you want to shell out the cash for a JAMA subscription, but Milloy’s case is very weak here. The previous study carried out by one of the junior coauthors dealt with steady consumption of relatively small amounts of sugar in the context of an overall diet. This one was about a sudden increase in one’s consumption of drinks containing lots of sugar. In any case, there is certainly no quasi-ethical presumption in the econometrics behaviour that any new piece of work needs to have a comprehensive literature review and I don’t believe that there is in the medical literature either. Milloy’s statement that “Her new study only presented data concerning a potential association between increasing soft drink consumption and weight gain. It presented no data on increasing soft drink consumption and diabetes” appears to me to be outright false as well.