Soft drinks and hard evidence

by Daniel on September 1, 2004

This probably doesn’t mount to all that much, but it’s been irritating me slightly for the last couple of days …

We all know that the second most dispiriting phrase in the English language is “Steve Milloy has a devastating critique …” (the first most dispiriting phrase is “My new column is up at Tech Central Station”.) The original reason why the Volokh post linked above irritated me was that it came the day after a post on Tim Lambert’s marvellous spot on the radians/degrees error in that global warming error. It rather irked me that Tim Lambert should get referenced with caveats (“Of course, that’s the claim; if there’s a rebuttal somewhere, please point me to it”) while Steven Milloy got three paragraphs of direct quotation with no caveats at all. Anyone wh knows even a little bit about the two chaps knows that Tim has always been tirelessly and scrupulously accurate, while Steven Milloy, proprietor of “junkscience.com”, is a bit of a hack, who got his start with a bit part towards the end of the single largest and most impressive work of intellectual dishonesty of the previous century[1], the effort to discredit the scientific work on the link between tobacco and lung cancer.

So I decided to take a look at the “devastating critique” to see whether it was really all that.

Surprise surprise, it wasn’t. In the linked article, Milloy is commenting on an article in the Journal of the American Medical Association. It’s subscription only, but I managed to get a look at a copy and a few things stuck out at me.

First, it’s a paper written using a very large sample; a panel study of about fifty thousand nurses. In particular, it’s looking at what happened to that portion of the sample of nurses who significantly increased their consumption of sugary (ie, non-diet) soft drinks. The finding is that, apparently, they tended to get diabetes at a significantly higher rate.

Milloy doesn’t seem to understand this; possibly by choice. He keeps on talking about the differences (which the study also reports) between the populations of nurses who had consistently low consumption of sweet drinks over the period and the population who had consistently high consumption. He also attributes claims to the report (such as “the simple-minded notion” that the entire weight gain reported in the switching group was due to the sugar in their drinks) which purely and simply aren’t there. This is irritating, but I’m prepared to read through it as it is most likely the result of genuine confusion on either my part or Milloy’s.

The point which really irked me, though, is the hallmark of a true Milloy hack-piece (as in, if it isn’t there, it’s not the real Milloy) – the discussion of things mentioned in the study phrased so as to suggest that only Milloy has noticed that they completely invalidate the study. In this piece, it’s a vintage example; Todd Zywicki at Volokh correctly excerpts it as the most important paragraph, and what a pity that he didn’t think about what he was pasting.

When the researchers statistically adjusted their results for bodyweight (a risk factor for diabetes) and for caloric intake (a proxy measure for consumption of sweetened foods other than soda), the 83 percent increase dropped to an even more statistically dubious (and soft-pedaled) 32 percent increase. That result is of the same magnitude as the study’s reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day

Think about this for a second. According to Milloy, the correct (even the ethical) thing to have done in presenting the results of this study would have been to have headlined the “32 per cent increase” (1.32 relative risk) that one gets in a model which controls for body mass and caloric intake. This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?

And then, assuming that we were to accept this stupid model, is Milloy reporting the results correctly? Here’s a clue; the phrase “the same order as magnitude” is a phrase with a precise meaning which is not here being used precisely. Nine is “of the same order of magnitude” as one, but if my missus was to find out I’d had nine sexual partners last month rather than one, I’d imagine she’d call it significant.

The phrase “order of magnitude” is presumably meant to convey that the sugary soft drink risk ratio is not significantly different from the diet soft drink ratio, which is a curious way to summarise the paper, because the paper explicitlysays that the sugary drinks risk ratio is significant while the diet drinks one isn’t[2]. Anyone who thinks that a paraphresis of this sort isn’t actually “fibbing” is welcome to their view; personally, I think it’s pretty bad.

Now, I have no real brief for the medical profession on this one; it seems to me pretty intuitive that since a can of Pepsi has the equivalent of thirteen sugar lumps in it, then drinking a lot of the stuff might not be the best thing on earth for the old pancreas, but on the other hand, since everyone drinks soft drinks and there are a lot of people walking around, I’m guessing that soft drinks aren’t poisonous. But it can’t help the debate to have the likes of Milloy misrepresenting research like this, and the less it gets promoted by reputable sources like the Volokhs, the better. And when Milloy reports it as a “in my opinion … flagrant and inexcusable omission” that the researchers didn’t include a separate, tangentially related[3] study that one of them carried out earlier, the correct response should not be endorsement, but rather a horse-laugh and a note that Milloy has a certain amount of previous form when it comes to inventing ethical codes from whole cloth.

So anyway, I think my only conclusion for this is shape up, Volokhs; if you are suspicious of a piece of scientific research, it is always better to spend the extra few minutes and find a critique that isn’t by Milloy.

Footnotes:
[1]If one was of a mind to defend Milloy, then one might say that the main work of junkscience.com was in fighting the battle against the link between second-hand smoke and cancer rather than the Big Evil of smoking and cancer. But as the linked .pdf above shows, he’s not been above making flip remarks about the smoking-cancer link.
[2]Two coefficients of this kind can be as close together as you like, but if they have different standard errors, then one might be very significant and the other not at all.
[3]You’re going to have to trust me on this one unless you want to shell out the cash for a JAMA subscription, but Milloy’s case is very weak here. The previous study carried out by one of the junior coauthors dealt with steady consumption of relatively small amounts of sugar in the context of an overall diet. This one was about a sudden increase in one’s consumption of drinks containing lots of sugar. In any case, there is certainly no quasi-ethical presumption in the econometrics behaviour that any new piece of work needs to have a comprehensive literature review and I don’t believe that there is in the medical literature either. Milloy’s statement that “Her new study only presented data concerning a potential association between increasing soft drink consumption and weight gain. It presented no data on increasing soft drink consumption and diabetes” appears to me to be outright false as well.

{ 50 comments }

1

nick 09.01.04 at 2:00 am

ty0p: s/order as magnitude/order of magnitude/g

2

dsquared 09.01.04 at 2:08 am

sorted, thanks

3

John Quiggin 09.01.04 at 2:35 am

Thanks for this. I was thinking of chasing it up, but you’ve done a great job.

4

Barry 09.01.04 at 3:14 am

A comment – a literature review should always be done, for a number of reasons (not reinventing the wheel, previous contradictory findings will be a source of counterargument, there might be related issues and methodology, etc.). However, if somebody is reviewing a piece of work, other irrelevent works are still irrelevant.

5

eudoxis 09.01.04 at 4:44 am

That “critique” raised my hackles as well, starting with the glaring absence of a cite. Fortunately, the study is receiving wider exposure than the lawyered review. Note that the study has an comparison for different weight gain cohorts as well as a comparison with association between other sweet drinks like fruit juice. The results are significant enough to induce life style changes for those (women) who drink more than 1 soft drink a day and experience weight gain.

6

Kieran Healy 09.01.04 at 6:07 am

I was half way (well, ten percent of the way, but it’s the same order of magnitude) through a post about this, one beginning with a similar observation on the phrase “X has a devastating critique of Y” for all Ys falling into the category “Research Findings I Do Not Wish To Believe Under Any Circumstances.” Thanks for getting there first; I will now have to find another fish to fry and hopefully do it half as well. Perhaps Iain Murray’s recent fantasies about impeaching Tony Blair. On second thoughts, perhaps not.

7

Kieran Healy 09.01.04 at 6:10 am

By the way, Eudoxis, there’s a good post somewhere in that phrase “lawyered review” I think. Something combining this sort of thing with, eg, the recent Harvard Law treatment of the intelligent design book.

8

eudoxis 09.01.04 at 6:55 am

ID, yes, that junk science. Here’s an explanation of how this review ended up in HLR. There’s a (lazy) tendency to think that the critiques of decent science are so ludicrous, a good defense isn’t necessary, but with lawyer point men like Phil Johnson always looking for “wedges”, nothing is sacred anymore.

9

Glen 09.01.04 at 7:59 am

“This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?”

I haven’t read the study, so I’m not sure what the point was… but I *thought* the point was to show that soft drinks were worse than other sources of the same calories. Otherwise, they could have ignored soft drinks in the study and just looked at calories. If the point is to show that soft drinks are worse, then it makes perfect sense to control for calories.

10

bad Jim 09.01.04 at 8:52 am

At the end of Super Size Me, Morgan Spurlock and girlfriend line up all the elements of his dietary demons: one big container of fat and a dozen containers of sugar.

Don’t the latest U.S. government dietary recommendations omit the customary caution against soft drinks? (Can’t google a quick cite, sorry.) Soft drink consumption has doubled in the last two decades, and probably accounts for at least a third of our expanding waistlines.

My car has cupholders, but they aren’t big enough for the bottles or cups that I’m served. Perhaps I shouldn’t be driving a Japanese car.

11

bad Jim 09.01.04 at 9:01 am

People drink soft drinks like water or like coffee, for rehydration and stimulation (and because they taste good). How many limit themselves to one a day?

12

coward 09.01.04 at 9:33 am

volokh is becoming more hackish as the election approaches.

13

Martin 09.01.04 at 10:35 am

People drink coffee for rehydration?

14

bad Jim 09.01.04 at 10:49 am

Yeah. Double espressos when available. Got a problem with that?

15

dsquared 09.01.04 at 10:56 am

I haven’t read the study, so I’m not sure what the point was… but I thought the point was to show that soft drinks were worse than other sources of the same calories. Otherwise, they could have ignored soft drinks in the study and just looked at calories.

In actual fact, they find that roughly half the effect on diabetes risk from increased soft drink consumption is particular, over and above the calories. I don’t know about this; I’m personally quite sceptical about “good” and “bad” calories (though possibly less so in the case of diabetes where carbohydrate content is obviously relevant). But I still think it would be crazy to headline a number that ignored the fact that sugar has calories.

16

Timothy Burke 09.01.04 at 1:20 pm

There’s another story lurking underneath this one, and it has to do with the public-sphere-in-a-bottle nature of blog writing. Bloggers are rapidly developing standards and discussions which are hermetically confined to blogs, a by-product of the need to find content and link to it and to build relations through complimentary or condemnatory linkages. So the phrase that rankles here is “devastating critique”, which is cued by the original piece’s wild overselling of its claims.

If the whole discussion had begun with Molloy making a more typically academic criticism–reserved, collegial, heavily qualified, but pointed–and the link had been more of the usual citational norm in academic life (“This critique makes some interesting points, though I wonder…”) not only would any subsequent conversation shake out differently, but the entire thing might actually matter to the authors of this study.

It’s not that strong rhetoric or harsh words are wrong or inappropriate in the blogosphere. They’re very welcome for the right kind of claims, the right kind of writing, the right kind of discussion, the right kind of aspirations. What makes me a little queasy is the mixing of standards and aspirations. It seems to me that if Molloy is going to take the time to write that critique, he’d like to be taken seriously by the people who did that study, and by the people who might use that study. If he’s going to do that, he’s got some standards to live up to. If Zywicki is going to pass it on, he’s also got some standards to live up to.

I’m not known for praising or reproducing academic norms as they conventionally stand, but they’re not wholly valueless. In certain conversations, at certain moments, they are important. At those moments, blog writers still tend to operate in the bush leagues, and the aggressive disinterest that most academics show in blogs–even when blogs purport to be critiquing the work of academics–is probably warranted.

17

Matt Weiner 09.01.04 at 3:15 pm

OT, but bad jim, isn’t coffee a diuretic? And doesn’t that make drinking coffee for rehydration counterproductive except as a short-term measure?

18

JP 09.01.04 at 3:35 pm

Coffee coffee everywhere, and not a drop to drink…

I thought Bad Jim meant to say water for rehydration and coffee for stimulation, but I guess not.

19

Jim Henley 09.01.04 at 4:05 pm

Once again, I am much less impressed by a Lambert/CT debunking of Milloy than John Lott. This one comes down to battling opinions: Milloy thinks it shoddy not to control for calories and weight when trying to isolate the effect of the consumption of soft drinks specifically. Daniel’s “debunking” of Milloy on this point strikes me as a lot of hand-waving. Yes, IMHO, since the point of “analysis” is to “break into parts,” the 32% figure IS the relevant number.

I say this as someone who swore off sugared beverages almost two years ago because of fear of diabetes, btw, and as a gun rights proponent who has had no problem denouncing John Lott without reseration.

20

David 09.01.04 at 5:32 pm

I think the whole thing was a setup for Daniel to use the phrase “real Malloy”. The rest is just filler. ;)

21

DJW 09.01.04 at 6:56 pm

Jim, congrats on your willingness to consider the possibility that empirical evidence that doesn’t conveniently support your views might not be wholly inaccurate.

Now that that’s cleared up, can you explain what is wrong with DD’s reasoning for why the 32% number is problematic, instead of just calling it “hand-waving” and moving on? Your mysterious use of scare quotes obfuscates whatever claim you’re trying to make.

It seems commonsensical to me that adjusting for some of the bodyweight difference might well make good sense, but an effort should be made to figure out an estimate for how much of it can be reasonably attributed to the extra empty calories. Maybe that’s a little and maybe it’s a lot, but it should hardly be ignored.

22

dsquared 09.01.04 at 7:16 pm

But Jim, come on:

1) the study did control for calories and reported their results. They just didn’t headline the figure when they were saying “this is the risk from soft drinks”.

2) They are right not to headline that figure for that purpose. As I explained above (and I promise that I wasn’t waving my hands while I did it), the 32% figure refers to the risk associated with soft drinks over and above the calories they contain. To claim that the only risk factor for soft drinks is the 32% over and above the caloric impact is ignoring the fact that sugar contains calories. You only have to look at Milloy’s implicit hypothesis – that people who decide to increase their soft drink consumption cut back on other foods to maintain constant caloric intake – to see it’s daft.

3. Finally, it can’t be emphasised enough, Milloy shouldn’t be given the benefit of the doubt when it comes to assessing whether his opinions on what’s “shoddy” or not are genuine. After the “sound science” fiasco and the Good Epidemiological Practice fiasco, his credibility is gone, particularly when writing pieces which appear to be politically and/or financially convenient for companies with big PR budgets.

The stopped clock theorem suggests that from time to time, Milloy will have a point. But in cases of genuinely bad science, it’s usually possible to find someone other than Milloy who thinks so. Since one tends to judge a chap by the company he keeps, it’s always better to do so.

23

Silent E 09.01.04 at 8:37 pm

“good” calories and “bad” – it should be obvious that increasing total caloric intake would increase risk of diabetes and weight gain.

But I think Milloy is onto something with his bizarre-o soda-substitution model. If I normally drink a large glass of fatty milk (2% or whole) and switch to sugared sodas containing the same number of calories, the study says that I’m still increasing my risk of diabetes!

Why? Glycemic index – the fave of Atkins dieters everywhere. Those sugary sodas are metabolized very fast – much faster than the fats and protiens in the milk. The sugar rush blitzes the pancreas and raises the risks of diabetes, as well as increasing the liklihood that I’ll be hungry soon when the blood sugar crash hits.

So, Milloy’s substitution model is useful because it highlights the additional dangers from consuming sugary sodas as opposed to any other beverage (e.g., fatty milk, or protein-powder fortified shakes and smoothies), even when no additional calories are consumed.

Of course, Milloy thinks this is a devastating critique, but what does he know?

24

John Quiggin 09.01.04 at 9:16 pm

Jim, there’s a difference between Lott and Milloy in that Lott actually does research, whereas Milloy surveys it and criticises the stuff he claims is “junk”.

It would be quite possible to do Milloy’s kind of work in a thoroughly biased and misleading fashion, without ever committing an error of the kind that’s noted here.

All that would be necessary is to stick to valid criticisms, but to make them if and only if the papers in question produce results you don’t like (this is the equivalent of the kind of cherrypicking that’s evident in Lott’s selection of regressions).

Milloy attempts to do this (I don’t think I’ve ever seen him criticise a paper that produces results convenient for his employers, or even one that’s neutral). However, he doesn’t manage it, as has been shown here.

25

Thomas 09.01.04 at 9:27 pm

Correct me if I’m not remembering these VC posts right, but the two posts were made by two different people. The “devastating critique” post wasn’t made by Eugene, while the post on Tim Lambert was. Eugene is more scrupulous than most bloggers on referencing his limited knowledge. The different treatment at VC is the result of the fact that there are different bloggers posting there. I wouldn’t over-interpret it as some sort of judgment on the relative merits of those discussed.

26

Jim Henley 09.01.04 at 9:31 pm

Daniel: Like I said before, having read “debunkings” of Milloy here and at least once on Lambert’s site, I’ve found said debunkings much weaker than the kind the same folks have meted out to Lott. The quotient of provable factual error to innuendo about funding sources and unsound opinions is just much much lower.

Now, couple of things: Milloy makes a factual claim that you, with access to the article, are able to check and I am not. I’d be very interested if he is telling the truth. Milloy says that “the study reports that women who consistently drank one or more regular soft drinks per day during those four years actually gained slightly less weight than women who consistently drank less than one soda per week during that same period.” Is that true? Or is Milloy misspeaking here?

Next, you make much of the fact that the study’s main interest is the link between women who increase their consumption over the four years and diabetes, not steady high-consumers. Okay. We know there are such things as “pre-diabetes” and “Syndrome X.” Is it not possible that these conditions cause increased soda intake (via unstable insulin levels) rather than a simple “drink more soda, get more diabetes” mechanism? Then you are dealing with a feedback loop.

Note that Milloy doesn’t bring this up. I bring it up because the reporting on this study seems to elide the problems of distinguishing correlation and causation and the quotes by the authors certainly don’t seem to be stressing the limits of their research.

Next, Willett states “The message is: Anyone who cares about their health or the health of their family would not consume these beverages. Parents who care about their children’s health should not keep them at home.” Does this strike you as a responsible conclusion for Willett to draw from the study, which, as you say, focused specifically on a cohort that increases its consumption substantially as opposed to low-consumption or steady-state high-consumption users?

Lastly, I have to call you on a bit of rhetorical distortion yourself. You take Milloy to task for misusing “order of magnitude” in his minimizing the difference between a an increase from 1.21 to 1.32. Your response is the bit about how sleeping with 9 partners is the same “order of magnitude” as sleeping with 1, but that wouldn’t please your wife. Do you think that deprecating the difference between a statistical increase from 1.21 to 1.32 is equivalent to minimizing the difference between 1 and 9?

djw: You seem bitter. So bitter you can’t tell the difference between scare quotes and the regular kind. I hope your day improves.

27

Jim Henley 09.01.04 at 9:53 pm

Daniel: Like I said before, having read “debunkings” of Milloy here and at least once on Lambert’s site, I’ve found said debunkings much weaker than the kind the same folks have meted out to Lott. The quotient of provable factual error to innuendo about funding sources and unsound opinions is just much much lower.

Now, couple of things: Milloy makes a factual claim that you, with access to the article, are able to check and I am not. I’d be very interested if he is telling the truth. Milloy says that “the study reports that women who consistently drank one or more regular soft drinks per day during those four years actually gained slightly less weight than women who consistently drank less than one soda per week during that same period.” Is that true? Or is Milloy misspeaking here?

Next, you make much of the fact that the study’s main interest is the link between women who increase their consumption over the four years and diabetes, not steady high-consumers. Okay. We know there are such things as “pre-diabetes” and “Syndrome X.” Is it not possible that these conditions cause increased soda intake (via unstable insulin levels) rather than a simple “drink more soda, get more diabetes” mechanism? Then you are dealing with a feedback loop.

Note that Milloy doesn’t bring this up. I bring it up because the reporting on this study seems to elide the problems of distinguishing correlation and causation and the quotes by the authors certainly don’t seem to be stressing the limits of their research.

Next, Willett states “The message is: Anyone who cares about their health or the health of their family would not consume these beverages. Parents who care about their children’s health should not keep them at home.” Does this strike you as a responsible conclusion for Willett to draw from the study, which, as you say, focused specifically on a cohort that increases its consumption substantially as opposed to low-consumption or steady-state high-consumption users?

Lastly, I have to call you on a bit of rhetorical distortion yourself. You take Milloy to task for misusing “order of magnitude” in his minimizing the difference between a an increase from 1.21 to 1.32. Your response is the bit about how sleeping with 9 partners is the same “order of magnitude” as sleeping with 1, but that wouldn’t please your wife. Do you think that deprecating the difference between a statistical increase from 1.21 to 1.32 is equivalent to minimizing the difference between 1 and 9?

djw: You seem bitter. So bitter you can’t tell the difference between scare quotes and the regular kind. I hope your day improves.

28

Jim Henley 09.01.04 at 10:13 pm

Doh! The dreaded double post! Sorry, CTers.

silent e: “Why? Glycemic index – the fave of Atkins dieters everywhere.”

This is a candidate, yeah. But it would be a stronger explanatory factor if the study also controlled for caffeine and carbonation. I’ve seen reports that caffeine can muck up blood sugar levels in at least some people. Carbonation? Beats me if there’s a connection, but you’d like to eliminate it. What about people who increase their soda consumption versus people who start drinking more sweet tea (non-carbonated)? What about people who drink more colas (caffeinated)versus people who drink more root bear or lemon-lime sodas (uncaffeinated)?

Since I started dieting and fitness-blogging not quite two years ago I’ve been thunderously impressed with just how poor the quality of most nutrition/diet research is. And the reporting on those studies – [shudder!].

29

dsquared 09.01.04 at 10:17 pm

Is that true?

It’s true that the report says it, but it omits to mention that the report notes that the difference in weights between the two categories is not significant and it has to be seen in the context of the statement that the study is about weight gain rather than diabetes, which isn’t true. So I’d compare it to Lott saying “Some states which passed CCW laws saw falling crime levels”.

We know there are such things as “pre-diabetes” and “Syndrome X.” Is it not possible that these conditions cause increased soda intake (via unstable insulin levels) rather than a simple “drink more soda, get more diabetes” mechanism?

Dunno. It’s not logically inconsistent, so in that sense it’s possible. Doesn’t seem all that likely to me, though. Fruit punch is also associated with higher diabetes risk, but with a lower risk factor. That would be consistent with the causal link, but not with your theory, I think. But since I’m not a doctor and Milloy didn’t make this argument, I’m not inclined to pursue the matter.

Does this strike you as a responsible conclusion for Willett to draw from the study

I don’t know what other work Willett has done. Since as far as I’m aware, Willett hasn’t done paid work trying to cover up the link between smoking and lung cancer, he hasn’t aroused my ire yet.

Do you think that deprecating the difference between a statistical increase from 1.21 to 1.32 is equivalent to minimizing the difference between 1 and 9?

Jim, this isn’t a statistical “Increase” from 1.21 to 1.32. It’s not a movement in one risk ratio. It’s the difference between two risk ratios, each of which has a different standard error. There’s a cut and dried fact of the matter about whether these ratios passed tests of significance. The 1.32 one did and the 1.21 one didn’t. If you try to pretend that they were “basically the same”, then in my book you’ve started fibbing and the rest is just haggling over terms. Finally, the context for my remark is a cheap joke; I’m not distorting a single word that Milloy wrote.

It’s true that debunkings of Milloy are weaker than those of Lott, because Milloy isn’t such an out-and-out, every-single-day hack as Lott. But he’s still pretty bad. This isn’t “innuendoes about funding”. This is a man who has, in the past, provably from documents, taken money from Philip Morris explicitly for the purpose of lobbying the epidemiological profession to take a meaningless and incorrect position on the subject of risk ratios, because it would be convenient for tobacco companies in future litigation. In other words, someone who needs watching like a hawk.

30

Steve Reuland 09.01.04 at 11:18 pm

OT, but bad jim, isn’t coffee a diuretic? And doesn’t that make drinking coffee for rehydration counterproductive except as a short-term measure?

Caffeine is indeed a diuretic, but the amount you find in a cup of coffee isn’t enough to off-set the water. In other words, you recieve net hydration from drinking a cup of coffee, even after the diruetic effects of caffeine are taken into account. So I suppose if you were trapped on a desert island with nothing to drink but coffee, you’d survive. Beer on the other hand…

31

eudoxis 09.01.04 at 11:57 pm

Milloy says that “the study reports that women who consistently drank one or more regular soft drinks per day during those four years actually gained slightly less weight than women who consistently drank less than one soda per week during that same period.” Is that true? Or is Milloy misspeaking here?

That is false. The study reports stable weight during the four year stable consumption period. Is it not possible that these [other] conditions cause increased soda intake (via unstable insulin levels) rather than a simple “drink more soda, get more diabetes” mechanism?

Yes, that is why factors like family history and obesity are taken into account. It is from these analyses that Milloy uses the 1.35 RR for obese women and implies that this means a 1.35 RR for all women when obesity is taken into account. This is misleading because the analyses used to adjust for obesity (or family history, etc.) is used only on those who are obese (or, etc.). The RR for non-obese is 1.78. We know from the weight gain portion of the study that soft drink intake is positively associated with weight gain. In other words, soft drinks are directly or indirectly associated with a significant increase in risk of adult onset diabetes. That increase is more than 35% for those without confounding factors. The greatest of those factors, btw, is family history, (RR 1.3). Neither of these diminish the RR associated with soft drinks, which remains at 1.83.

32

eudoxis 09.02.04 at 12:17 am

Milloy: Moreover, the study reports that women who consistently drank one or more regular soft drinks per day during those four years actually gained slightly less weight than women who consistently drank less than one soda per week during that same period.

The numbers in study:
2.24 kg for >1 drink/day
2.04 kg for

33

eudoxis 09.02.04 at 12:19 am

2.04 kg for <1 drink/week (Milloy lists them in a different order from the paper.)

34

dsquared 09.02.04 at 12:21 am

Eudoxis; this one confused the hell out of me too, but I don’t think Milloy made the exact mistake you attribute to him here. The 1.32 figure is IIRC in Table 3; it’s the RR from a multivariate model including BMI, calorific intake and about a dozen other factors. The figures for obese women are in table 4.

(John’s point is completely apropos, by the way; it would be entirely possible for a more skilled operator than Milloy, and one who wasn’t so hung up on weird ideas about risk ratios, to do his flack job without ever making a statement you could pin him down on.)

35

eudoxis 09.02.04 at 1:22 am

Hm, confusing because it was left out of the tables and placed in the text. It should have been in the table because it’s significant from a medical etiology standpoint. Thanks, Daniel.

36

ed 09.02.04 at 4:03 am

You say: “Two coefficients of this kind can be as close together as you like, but if they have different standard errors, then one might be very significant and the other not at all.”

You are confusing statistical significance with substantive significance. What you say is true for statistical significance, but what we really care about is how important the effect is.

To quote Diedre McClosky:

“No finding of fit or statistical significance testifies in itself to the scientific importance of an effect. Fit and importance are not the same thing. Nor is fit something that you “first” determine, and “then” move to substance. The substance of an effect is, to use a technical term, its oomph. Oomph ordinarily has nothing whatever to do with whether the coefficient is statistically significant at the .01 or .05 or .10 level.”

37

dsquared 09.02.04 at 1:44 pm

Ed; I’m aware of the distinction. In context, I was talking about statistical significance, which still matters to some of us crusty old souls.

38

ed 09.02.04 at 5:10 pm

Dsquared, if you’re aware of the disctinction you should be more careful about making it in your rhetoric. In fact, you are specifically attacking Milloy for saying that the point estimates are similar in magnitude, which is certainly relevant.

(BTW, I’m not defending Milloy in general, just on this point.)

39

ed 09.02.04 at 5:21 pm

Furthermore, you say:
“The phrase “order of magnitude” is presumably meant to convey that the sugary soft drink risk ratio is not significantly different from the diet soft drink ratio, which is a curious way to summarise the paper, because the paper explicitlysays that the sugary drinks risk ratio is significant while the diet drinks one isn’t.”

This is misleading: the fact that one parameter is statistically significant and the other isn’t does NOT imply that we can reject the hypothesis that the two coefficients are equal. You seem to be implying that we can reject this hypothesis. The fact that the point estimates are close together, as Milloy points out, makes it likely that the difference is not statistically significant (but we don’t know this for sure without more information).

40

Alex Fradera 09.02.04 at 5:23 pm

It seemed pretty clear that D2 was talking about statistical significance – I and any other scientist I know would not use significance to mean ‘substansive significance’, but prefer to talk about magnitudes of effects – either effect sizes, causal efficacy or even confidence limits. We wouldn’t use significance, as it already has a well-worn connotation when discussion results – that of statistical significance.

41

Alex Fradera 09.02.04 at 5:27 pm

It seemed pretty clear that D2 was talking about statistical significance – I and any other scientist I know would not use significance to mean ‘substansive significance’, but prefer to talk about magnitudes of effects – either effect sizes, causal efficacy or even confidence limits. We wouldn’t use significance, as it already has a well-worn connotation when discussion results – that of statistical significance.

42

ed 09.02.04 at 6:10 pm

Well, Milloy did use the term “magnitude,” and D2 attacked him by switching the topic to statistical significance and assuming that that was what Milloy had been talking about.

Plus, I believe this blog is written for a general audience, and therefore terms like “significance” that have common everyday meanings should be used with care.

43

Another Damned Medievalist 09.02.04 at 6:46 pm

And he still managed not to mention Bill Maher’s favorite bugaboo, high fructose corn syrup! Seriously, though. Nothing in this study should be surprising. When I were a lad … sodas came in 7 ounce bottles and 12? ounce bottles and cans. They were deposit bottles. You could make chains out of the pop-tops. You could make the world sing in perfect harmony. No one I knew was allowed all soda, all the time. Most of us had to ask, and only the really lucky kids (the ones who never ate PBJ sandwiches and got chips and ho-hos {or the equivalent} in their lunch boxes {remember those? food prepared at and brought from home?}) got them as a matter of course. Heaven forbid they were offered at school.
Besides the fact that one can now buy junk food with food stamps, how the hell did we get here? With all the weight-consciousness propaganda flooding the airwaves, how is it that anybody regularly chooses to drink more than the occasional sugary soda? The diet stuff is questionable too — sweeteners rushed through FDA testing to market that turn out to have not-so-healthy side effects? Carbonation that can lead to osteoporosis? Perhaps we need even clearer labeling — like, “This 64 oz. soda has 800, THAT’S 800, you bonehead! calories — this means that it is 2/3 the calories needed PER DAY by a fairly active woman of average height and weight”

44

dsquared 09.02.04 at 9:18 pm

Ed: Milloy’s article was plainly meant to create the impression that the risk ratio for sugary soft drinks was not of practical significance, because it was “of the same magnitude” as that for diet drinks. If this means anything, it is suggesting that it makes no difference in terms of diabetes risk whether you drink regular or diet soda.

What the paper actually said was that sugary drinks were (statistically significantly) associated with a higher risk of diabetes, while the risk associated with diet drinks was not (statistically significantly) different from the baseline.

I think what I wrote is a fair enough summary of the above.

45

tim 09.03.04 at 6:19 pm

Milloy: When the researchers statistically adjusted their results for bodyweight…and for caloric intake…, the 83 percent increase dropped to an even more statistically dubious … 32 percent increase. That result is of the same magnitude as the study’s reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day

Daniel: Think about this for a second. According to Milloy, the correct (even the ethical) thing to have done in presenting the results of this study would have been to have headlined the ’32 per cent increase’ (1.32 relative risk) that one gets in a model which controls for body mass and caloric intake. This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?
——————————–

I’m not an economist, so I obviously don’t know that much about science, and I haven’t read the original article or Milloy, but I thought one of the keys to doing science is the control population.

If the point of your study is to show that a high calorie diet is worse than a low calorie diet, then you should compare individuals with as many of their non-diet characteristics as possible — for example, exercise and exposure to environmental toxins — the same.

If the point of your study is to show that a high sugar diet is bad for you, you should compare a population with a diet high in sugar to a population with a diet comparable in other aspects — for example, calories — but different sugar intake.

If the point of your study is to show that drinking non-diet soda is bad, you need to ensure that the effect is coming from soda per-se, and not just sugar or calories (or perhaps other environmental effects: one population living in Elizabeth, NJ, and the other in Eugene, OR), or your study is not a warning about soda consumption, but a warning about calories or air quality.

The correct way to account for the health risks due to soda and not due to calories in general *is* to compare populations with the same calories but different levels of soda intake.

Or, alternatively, I suppose, you could show a correlation between non-diet soda and calories — but that wouldn’t get you grant money, journal publications, or an appearance on NPR.

If your point is to give Milloy (whom I don’t know from Adam) a beating, though, I suppose any old straw man will do in the end.

46

dsquared 09.03.04 at 6:46 pm

Tim, the question that is being adressed (and if you had access to the paper you’d know this) is “should doctors advise their patients to reduce their consumption of non-diet soda to less than one per day?”.

To answer this question, you have to take into account the interactive effects as well as the marginal effects; here’s an example of what I mean:

If I was carrying out a study into whether injecting heroin was good or bad for your health, then it would probably in principle be possible for me to control out the risks of AIDS, hepatitis, contaminants etc and get a fully controlled estimate of the increase in risk of premature death associated with injecting clinically pure diamorphine through new needles. While this would be interesting from a scientific point of view, would you really say that best practice would be for me to make this the headline number in reporting my study?

Or look at this another way; does your “control population” make any sense? Why are you trying to control for calories in this way, when soda contains calories? If person A is taking in 5000 calories per day from food, while person B is taking in 2500 calories from food and 2500 from soda, then person B actually has a much healthier lifestyle, apart from the soda she drinks.

Would you recommend that studies of whether smoking tobacco is bad for you should use dope-smokers as their control group?

47

ed 09.03.04 at 10:10 pm

dsquared: How about this question: should doctors advise their patients to reduce their consumption of Mr. Pibb?

You’d probably say: no, she should advise her patients to reduce consumption of all soda.

Milloy would say: no, she should advise her patients to reduce calories overall.

It is not obvious that Milloy is wrong about this.

If the real problem is calories, and the newspaper headlines are screaming about soda, it is natural that soda lovers and soda producers will be upset, with good reason. But if soda is the main cause of high calories, then your advice might be better. I don’t know if the study addressed that point.

48

tim 09.03.04 at 10:56 pm

“the question that is being adressed…is ‘should doctors advise their patients to reduce their consumption of non-diet soda to less than one per day?’.”

If the study doesn’t control for caloric intake, and if you control for caloric intake and the non-diet soda has no statistically significant impact, then the answer is “No, doctors should advise their patients to reduce their consumption of calories (or sugar, or whatever), but should otherwise make no special dire predictions about the health effects of soda.”

Why? Because suppose I simple-mindedly took the advice, gave up non-diet sodas, but supplemented my diet soda with a couple of candy bars (to ease that sugar craving that used to send me to the soda machine in the first place).

Has the advice to give up sodas done any good? Not if the problem is the sugar. In fact, this is just the simple inversion of your own objection! The patient might well change his or her diet (adding other forms of sugar) if forced to give up sodas because they were led to believe that soda, not sugar was the significant dietary risk.

Now for your examples:

“If I was carrying out a study into whether injecting heroin was good or bad for your health, then it would probably in principle be possible for me to control out the risks of AIDS, hepatitis, contaminants etc and get a fully controlled estimate of the increase in risk of premature death associated with injecting clinically pure diamorphine through new needles. While this would be interesting from a scientific point of view, would you really say that best practice would be for me to make this the headline number in reporting my study?”

How slippery this argument has become. The study Milloy was critiquing was published in a medical journal: it was presented to an audience of doctors as of scientific interest. The scientific interest was precisely the point and the basis of the criticism, I thought.

But, to address the objection on what independent merit it may have: you should make the headline read that *dirty needles* are the problem – because injection of anything through dirty needles carries that same risk. That’s why hospitals don’t re-use needles – see the value of getting the study right! You wouldn’t want hospitals re-using needles from the emergency room just because they are being used for things other than heroin. (Heroin has its own problems, but those are apparently not coming out in your study.)

“Or look at this another way; does your ‘control population’ make any sense? Why are you trying to control for calories in this way, when soda contains calories? If person A is taking in 5000 calories per day from food, while person B is taking in 2500 calories from food and 2500 from soda, then person B actually has a much healthier lifestyle, apart from the soda she drinks.”

Of course it makes sense. Does person B *actually* have a healthier lifestyle overall? I mean, how would you know? Oh, wait, I know. You would know by controlling for soda intake independently of controlling for caloric intake (and other factors, such as exercise). Then you would know whether trading food calories for soda calories carried an increased or decreased health risk, and then you could headline whether or not *soda* was a health risk.

“Would you recommend that studies of whether smoking tobacco is bad for you should use dope-smokers as their control group?”

If you want to assert that smoking tobacco in particular is a health risk that smoking dope (or oregano, or corn silk) is not, you *have* to.

49

tim 09.03.04 at 10:57 pm

“the question that is being adressed…is ‘should doctors advise their patients to reduce their consumption of non-diet soda to less than one per day?’.”

If the study doesn’t control for caloric intake, and if you control for caloric intake and the non-diet soda has no statistically significant impact, then the answer is “No, doctors should advise their patients to reduce their consumption of calories (or sugar, or whatever), but should otherwise make no special dire predictions about the health effects of soda.”

Why? Because suppose I simple-mindedly took the advice, gave up non-diet sodas, but supplemented my diet soda with a couple of candy bars (to ease that sugar craving that used to send me to the soda machine in the first place).

Has the advice to give up sodas done any good? Not if the problem is the sugar. In fact, this is just the simple inversion of your own objection! The patient might well change his or her diet (adding other forms of sugar) if forced to give up sodas because they were led to believe that soda, not sugar was the significant dietary risk.

Now for your examples:

“If I was carrying out a study into whether injecting heroin was good or bad for your health, then it would probably in principle be possible for me to control out the risks of AIDS, hepatitis, contaminants etc and get a fully controlled estimate of the increase in risk of premature death associated with injecting clinically pure diamorphine through new needles. While this would be interesting from a scientific point of view, would you really say that best practice would be for me to make this the headline number in reporting my study?”

How slippery this argument has become. The study Milloy was critiquing was published in a medical journal: it was presented to an audience of doctors as of scientific interest. The scientific interest was precisely the point and the basis of the criticism, I thought.

But, to address the objection on what independent merit it may have: you should make the headline read that *dirty needles* are the problem – because injection of anything through dirty needles carries that same risk. That’s why hospitals don’t re-use needles – see the value of getting the study right! You wouldn’t want hospitals re-using needles from the emergency room just because they are being used for things other than heroin. (Heroin has its own problems, but those are apparently not coming out in your study.)

“Or look at this another way; does your ‘control population’ make any sense? Why are you trying to control for calories in this way, when soda contains calories? If person A is taking in 5000 calories per day from food, while person B is taking in 2500 calories from food and 2500 from soda, then person B actually has a much healthier lifestyle, apart from the soda she drinks.”

Of course it makes sense. Does person B *actually* have a healthier lifestyle overall? I mean, how would you know? Oh, wait, I know. You would know by controlling for soda intake independently of controlling for caloric intake (and other factors, such as exercise). Then you would know whether trading food calories for soda calories carried an increased or decreased health risk, and then you could headline whether or not *soda* was a health risk.

“Would you recommend that studies of whether smoking tobacco is bad for you should use dope-smokers as their control group?”

If you want to assert that smoking tobacco in particular is a health risk that smoking dope (or oregano, or corn silk) is not, you *have* to.

50

baa 09.04.04 at 4:18 pm

Tim and ed have done a nice job here. But I recommend any readers of this thread also check out the Volkh link in the trackback. I carry no brief for Milloy or for “Big Soda,” but DD’s comments seem consistantly off here. For example:

“There’s a cut and dried fact of the matter about whether these ratios passed tests of significance. The 1.32 one did and the 1.21 one didn’t.”

This a very poor way to think about what significance means, and DD should know this. Significance isn’t an on/off switch for most tests.

Comments on this entry are closed.