Comments on: Soft drinks and hard evidence

By: baa

baa — Sat, 04 Sep 2004 16:18:34 +0000

Tim and ed have done a nice job here. But I recommend any readers of this thread also check out the Volkh link in the trackback. I carry no brief for Milloy or for “Big Soda,” but DD’s comments seem consistantly off here. For example: “There’s a cut and dried fact of the matter about whether these ratios passed tests of significance. The 1.32 one did and the 1.21 one didn’t.” This a very poor way to think about what significance means, and DD should know this. Significance isn’t an on/off switch for most tests.

By: tim

tim — Fri, 03 Sep 2004 22:57:40 +0000

“the question that is being adressed…is ‘should doctors advise their patients to reduce their consumption of non-diet soda to less than one per day?’.” If the study doesn’t control for caloric intake, and if you control for caloric intake and the non-diet soda has no statistically significant impact, then the answer is “No, doctors should advise their patients to reduce their consumption of calories (or sugar, or whatever), but should otherwise make no special dire predictions about the health effects of soda.” Why? Because suppose I simple-mindedly took the advice, gave up non-diet sodas, but supplemented my diet soda with a couple of candy bars (to ease that sugar craving that used to send me to the soda machine in the first place). Has the advice to give up sodas done any good? Not if the problem is the sugar. In fact, this is just the simple inversion of your own objection! The patient might well change his or her diet (adding other forms of sugar) if forced to give up sodas because they were led to believe that soda, not sugar was the significant dietary risk. Now for your examples: “If I was carrying out a study into whether injecting heroin was good or bad for your health, then it would probably in principle be possible for me to control out the risks of AIDS, hepatitis, contaminants etc and get a fully controlled estimate of the increase in risk of premature death associated with injecting clinically pure diamorphine through new needles. While this would be interesting from a scientific point of view, would you really say that best practice would be for me to make this the headline number in reporting my study?” How slippery this argument has become. The study Milloy was critiquing was published in a medical journal: it was presented to an audience of doctors as of scientific interest. The scientific interest was precisely the point and the basis of the criticism, I thought. But, to address the objection on what independent merit it may have: you should make the headline read that dirty needles are the problem – because injection of anything through dirty needles carries that same risk. That’s why hospitals don’t re-use needles – see the value of getting the study right! You wouldn’t want hospitals re-using needles from the emergency room just because they are being used for things other than heroin. (Heroin has its own problems, but those are apparently not coming out in your study.) “Or look at this another way; does your ‘control population’ make any sense? Why are you trying to control for calories in this way, when soda contains calories? If person A is taking in 5000 calories per day from food, while person B is taking in 2500 calories from food and 2500 from soda, then person B actually has a much healthier lifestyle, apart from the soda she drinks.” Of course it makes sense. Does person B actually have a healthier lifestyle overall? I mean, how would you know? Oh, wait, I know. You would know by controlling for soda intake independently of controlling for caloric intake (and other factors, such as exercise). Then you would know whether trading food calories for soda calories carried an increased or decreased health risk, and then you could headline whether or not soda was a health risk. “Would you recommend that studies of whether smoking tobacco is bad for you should use dope-smokers as their control group?” If you want to assert that smoking tobacco in particular is a health risk that smoking dope (or oregano, or corn silk) is not, you have to.

By: tim

tim — Fri, 03 Sep 2004 22:56:16 +0000

By: ed

ed — Fri, 03 Sep 2004 22:10:42 +0000

dsquared: How about this question: should doctors advise their patients to reduce their consumption of Mr. Pibb? You’d probably say: no, she should advise her patients to reduce consumption of all soda. Milloy would say: no, she should advise her patients to reduce calories overall. It is not obvious that Milloy is wrong about this. If the real problem is calories, and the newspaper headlines are screaming about soda, it is natural that soda lovers and soda producers will be upset, with good reason. But if soda is the main cause of high calories, then your advice might be better. I don’t know if the study addressed that point.

By: dsquared

dsquared — Fri, 03 Sep 2004 18:46:01 +0000

Tim, the question that is being adressed (and if you had access to the paper you’d know this) is “should doctors advise their patients to reduce their consumption of non-diet soda to less than one per day?”. To answer this question, you have to take into account the interactive effects as well as the marginal effects; here’s an example of what I mean: If I was carrying out a study into whether injecting heroin was good or bad for your health, then it would probably in principle be possible for me to control out the risks of AIDS, hepatitis, contaminants etc and get a fully controlled estimate of the increase in risk of premature death associated with injecting clinically pure diamorphine through new needles. While this would be interesting from a scientific point of view, would you really say that best practice would be for me to make this the headline number in reporting my study? Or look at this another way; does your “control population” make any sense? Why are you trying to control for calories in this way, when soda contains calories? If person A is taking in 5000 calories per day from food, while person B is taking in 2500 calories from food and 2500 from soda, then person B actually has a much healthier lifestyle, apart from the soda she drinks. Would you recommend that studies of whether smoking tobacco is bad for you should use dope-smokers as their control group?

By: tim

tim — Fri, 03 Sep 2004 18:19:27 +0000

Milloy: When the researchers statistically adjusted their results for bodyweight…and for caloric intake…, the 83 percent increase dropped to an even more statistically dubious … 32 percent increase. That result is of the same magnitude as the study’s reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day Daniel: Think about this for a second. According to Milloy, the correct (even the ethical) thing to have done in presenting the results of this study would have been to have headlined the ‘32 per cent increase’ (1.32 relative risk) that one gets in a model which controls for body mass and caloric intake. This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?———————————————— I’m not an economist, so I obviously don’t know that much about science, and I haven’t read the original article or Milloy, but I thought one of the keys to doing science is the control population. If the point of your study is to show that a high calorie diet is worse than a low calorie diet, then you should compare individuals with as many of their non-diet characteristics as possible—for example, exercise and exposure to environmental toxins—the same. If the point of your study is to show that a high sugar diet is bad for you, you should compare a population with a diet high in sugar to a population with a diet comparable in other aspects—for example, calories—but different sugar intake. If the point of your study is to show that drinking non-diet soda is bad, you need to ensure that the effect is coming from soda per-se, and not just sugar or calories (or perhaps other environmental effects: one population living in Elizabeth, NJ, and the other in Eugene, OR), or your study is not a warning about soda consumption, but a warning about calories or air quality. The correct way to account for the health risks due to soda and not due to calories in general is to compare populations with the same calories but different levels of soda intake. Or, alternatively, I suppose, you could show a correlation between non-diet soda and calories—but that wouldn’t get you grant money, journal publications, or an appearance on NPR. If your point is to give Milloy (whom I don’t know from Adam) a beating, though, I suppose any old straw man will do in the end.

By: dsquared

dsquared — Thu, 02 Sep 2004 21:18:29 +0000

Ed: Milloy’s article was plainly meant to create the impression that the risk ratio for sugary soft drinks was not of practical significance, because it was “of the same magnitude” as that for diet drinks. If this means anything, it is suggesting that it makes no difference in terms of diabetes risk whether you drink regular or diet soda. What the paper actually said was that sugary drinks were (statistically significantly) associated with a higher risk of diabetes, while the risk associated with diet drinks was not (statistically significantly) different from the baseline. I think what I wrote is a fair enough summary of the above.

By: Another Damned Medievalist

Another Damned Medievalist — Thu, 02 Sep 2004 18:46:08 +0000

And he still managed not to mention Bill Maher’s favorite bugaboo, high fructose corn syrup! Seriously, though. Nothing in this study should be surprising. When I were a lad … sodas came in 7 ounce bottles and 12? ounce bottles and cans. They were deposit bottles. You could make chains out of the pop-tops. You could make the world sing in perfect harmony. No one I knew was allowed all soda, all the time. Most of us had to ask, and only the really lucky kids (the ones who never ate PBJ sandwiches and got chips and ho-hos {or the equivalent} in their lunch boxes {remember those? food prepared at and brought from home?}) got them as a matter of course. Heaven forbid they were offered at school. Besides the fact that one can now buy junk food with food stamps, how the hell did we get here? With all the weight-consciousness propaganda flooding the airwaves, how is it that anybody regularly chooses to drink more than the occasional sugary soda? The diet stuff is questionable too—sweeteners rushed through FDA testing to market that turn out to have not-so-healthy side effects? Carbonation that can lead to osteoporosis? Perhaps we need even clearer labeling—like, “This 64 oz. soda has 800, THAT’S 800, you bonehead! calories—this means that it is 2/3 the calories needed PER DAY by a fairly active woman of average height and weight”

By: ed

ed — Thu, 02 Sep 2004 18:10:43 +0000

Well, Milloy did use the term “magnitude,” and D2 attacked him by switching the topic to statistical significance and assuming that that was what Milloy had been talking about. Plus, I believe this blog is written for a general audience, and therefore terms like “significance” that have common everyday meanings should be used with care.

By: Alex Fradera

Alex Fradera — Thu, 02 Sep 2004 17:27:41 +0000

It seemed pretty clear that D2 was talking about statistical significance – I and any other scientist I know would not use significance to mean ‘substansive significance’, but prefer to talk about magnitudes of effects – either effect sizes, causal efficacy or even confidence limits. We wouldn’t use significance, as it already has a well-worn connotation when discussion results – that of statistical significance.

By: Alex Fradera

Alex Fradera — Thu, 02 Sep 2004 17:23:32 +0000

By: ed

ed — Thu, 02 Sep 2004 17:21:46 +0000

Furthermore, you say: “The phrase “order of magnitude” is presumably meant to convey that the sugary soft drink risk ratio is not significantly different from the diet soft drink ratio, which is a curious way to summarise the paper, because the paper explicitlysays that the sugary drinks risk ratio is significant while the diet drinks one isn’t.” This is misleading: the fact that one parameter is statistically significant and the other isn’t does NOT imply that we can reject the hypothesis that the two coefficients are equal. You seem to be implying that we can reject this hypothesis. The fact that the point estimates are close together, as Milloy points out, makes it likely that the difference is not statistically significant (but we don’t know this for sure without more information).

By: ed

ed — Thu, 02 Sep 2004 17:10:32 +0000

Dsquared, if you’re aware of the disctinction you should be more careful about making it in your rhetoric. In fact, you are specifically attacking Milloy for saying that the point estimates are similar in magnitude, which is certainly relevant. (BTW, I’m not defending Milloy in general, just on this point.)

By: dsquared

dsquared — Thu, 02 Sep 2004 13:44:19 +0000

Ed; I’m aware of the distinction. In context, I was talking about statistical significance, which still matters to some of us crusty old souls.

By: ed

ed — Thu, 02 Sep 2004 04:03:51 +0000

You say: “Two coefficients of this kind can be as close together as you like, but if they have different standard errors, then one might be very significant and the other not at all.” You are confusing statistical significance with substantive significance. What you say is true for statistical significance, but what we really care about is how important the effect is. To quote Diedre McClosky: “No finding of fit or statistical significance testifies in itself to the scientific importance of an effect. Fit and importance are not the same thing. Nor is fit something that you “first” determine, and “then” move to substance. The substance of an effect is, to use a technical term, its oomph. Oomph ordinarily has nothing whatever to do with whether the coefficient is statistically significant at the .01 or .05 or .10 level.”