Soft drinks and hard evidence

ty0p: s/order as magnitude/order of magnitude/g

Thanks for this. I was thinking of chasing it up, but you’ve done a great job.

That “critique” raised my hackles as well, starting with the glaring absence of a cite. Fortunately, the study is receiving wider exposure than the lawyered review. Note that the study has an comparison for different weight gain cohorts as well as a comparison with association between other sweet drinks like fruit juice. The results are significant enough to induce life style changes for those (women) who drink more than 1 soft drink a day and experience weight gain.

By the way, Eudoxis, there’s a good post somewhere in that phrase “lawyered review” I think. Something combining this sort of thing with, eg, the recent Harvard Law treatment of the intelligent design book.

“This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?”

I haven’t read the study, so I’m not sure what the point was… but I *thought* the point was to show that soft drinks were worse than other sources of the same calories. Otherwise, they could have ignored soft drinks in the study and just looked at calories. If the point is to show that soft drinks are worse, then it makes perfect sense to control for calories.

People drink soft drinks like water or like coffee, for rehydration and stimulation (and because they taste good). How many limit themselves to one a day?

People drink coffee for rehydration?

I haven’t read the study, so I’m not sure what the point was… but I thought the point was to show that soft drinks were worse than other sources of the same calories. Otherwise, they could have ignored soft drinks in the study and just looked at calories.

In actual fact, they find that roughly half the effect on diabetes risk from increased soft drink consumption is particular, over and above the calories. I don’t know about this; I’m personally quite sceptical about “good” and “bad” calories (though possibly less so in the case of diabetes where carbohydrate content is obviously relevant). But I still think it would be crazy to headline a number that ignored the fact that sugar has calories.

OT, but bad jim, isn’t coffee a diuretic? And doesn’t that make drinking coffee for rehydration counterproductive except as a short-term measure?

Once again, I am much less impressed by a Lambert/CT debunking of Milloy than John Lott. This one comes down to battling opinions: Milloy thinks it shoddy not to control for calories and weight when trying to isolate the effect of the consumption of soft drinks specifically. Daniel’s “debunking” of Milloy on this point strikes me as a lot of hand-waving. Yes, IMHO, since the point of “analysis” is to “break into parts,” the 32% figure IS the relevant number.

I say this as someone who swore off sugared beverages almost two years ago because of fear of diabetes, btw, and as a gun rights proponent who has had no problem denouncing John Lott without reseration.

Jim, congrats on your willingness to consider the possibility that empirical evidence that doesn’t conveniently support your views might not be wholly inaccurate.

Now that that’s cleared up, can you explain what is wrong with DD’s reasoning for why the 32% number is problematic, instead of just calling it “hand-waving” and moving on? Your mysterious use of scare quotes obfuscates whatever claim you’re trying to make.

It seems commonsensical to me that adjusting for some of the bodyweight difference might well make good sense, but an effort should be made to figure out an estimate for how much of it can be reasonably attributed to the extra empty calories. Maybe that’s a little and maybe it’s a lot, but it should hardly be ignored.

“good” calories and “bad” – it should be obvious that increasing total caloric intake would increase risk of diabetes and weight gain.

But I think Milloy is onto something with his bizarre-o soda-substitution model. If I normally drink a large glass of fatty milk (2% or whole) and switch to sugared sodas containing the same number of calories, the study says that I’m still increasing my risk of diabetes!

Why? Glycemic index – the fave of Atkins dieters everywhere. Those sugary sodas are metabolized very fast – much faster than the fats and protiens in the milk. The sugar rush blitzes the pancreas and raises the risks of diabetes, as well as increasing the liklihood that I’ll be hungry soon when the blood sugar crash hits.

So, Milloy’s substitution model is useful because it highlights the additional dangers from consuming sugary sodas as opposed to any other beverage (e.g., fatty milk, or protein-powder fortified shakes and smoothies), even when no additional calories are consumed.

Of course, Milloy thinks this is a devastating critique, but what does he know?

Correct me if I’m not remembering these VC posts right, but the two posts were made by two different people. The “devastating critique” post wasn’t made by Eugene, while the post on Tim Lambert was. Eugene is more scrupulous than most bloggers on referencing his limited knowledge. The different treatment at VC is the result of the fact that there are different bloggers posting there. I wouldn’t over-interpret it as some sort of judgment on the relative merits of those discussed.

Daniel: Like I said before, having read “debunkings” of Milloy here and at least once on Lambert’s site, I’ve found said debunkings much weaker than the kind the same folks have meted out to Lott. The quotient of provable factual error to innuendo about funding sources and unsound opinions is just much much lower.

Now, couple of things: Milloy makes a factual claim that you, with access to the article, are able to check and I am not. I’d be very interested if he is telling the truth. Milloy says that “the study reports that women who consistently drank one or more regular soft drinks per day during those four years actually gained slightly less weight than women who consistently drank less than one soda per week during that same period.” Is that true? Or is Milloy misspeaking here?

Next, you make much of the fact that the study’s main interest is the link between women who increase their consumption over the four years and diabetes, not steady high-consumers. Okay. We know there are such things as “pre-diabetes” and “Syndrome X.” Is it not possible that these conditions cause increased soda intake (via unstable insulin levels) rather than a simple “drink more soda, get more diabetes” mechanism? Then you are dealing with a feedback loop.

Note that Milloy doesn’t bring this up. I bring it up because the reporting on this study seems to elide the problems of distinguishing correlation and causation and the quotes by the authors certainly don’t seem to be stressing the limits of their research.

Next, Willett states “The message is: Anyone who cares about their health or the health of their family would not consume these beverages. Parents who care about their children’s health should not keep them at home.” Does this strike you as a responsible conclusion for Willett to draw from the study, which, as you say, focused specifically on a cohort that increases its consumption substantially as opposed to low-consumption or steady-state high-consumption users?

Lastly, I have to call you on a bit of rhetorical distortion yourself. You take Milloy to task for misusing “order of magnitude” in his minimizing the difference between a an increase from 1.21 to 1.32. Your response is the bit about how sleeping with 9 partners is the same “order of magnitude” as sleeping with 1, but that wouldn’t please your wife. Do you think that deprecating the difference between a statistical increase from 1.21 to 1.32 is equivalent to minimizing the difference between 1 and 9?

djw: You seem bitter. So bitter you can’t tell the difference between scare quotes and the regular kind. I hope your day improves.

Is that true?

It’s true that the report says it, but it omits to mention that the report notes that the difference in weights between the two categories is not significant and it has to be seen in the context of the statement that the study is about weight gain rather than diabetes, which isn’t true. So I’d compare it to Lott saying “Some states which passed CCW laws saw falling crime levels”.

We know there are such things as “pre-diabetes” and “Syndrome X.” Is it not possible that these conditions cause increased soda intake (via unstable insulin levels) rather than a simple “drink more soda, get more diabetes” mechanism?

Dunno. It’s not logically inconsistent, so in that sense it’s possible. Doesn’t seem all that likely to me, though. Fruit punch is also associated with higher diabetes risk, but with a lower risk factor. That would be consistent with the causal link, but not with your theory, I think. But since I’m not a doctor and Milloy didn’t make this argument, I’m not inclined to pursue the matter.

Does this strike you as a responsible conclusion for Willett to draw from the study

I don’t know what other work Willett has done. Since as far as I’m aware, Willett hasn’t done paid work trying to cover up the link between smoking and lung cancer, he hasn’t aroused my ire yet.

Do you think that deprecating the difference between a statistical increase from 1.21 to 1.32 is equivalent to minimizing the difference between 1 and 9?

Jim, this isn’t a statistical “Increase” from 1.21 to 1.32. It’s not a movement in one risk ratio. It’s the difference between two risk ratios, each of which has a different standard error. There’s a cut and dried fact of the matter about whether these ratios passed tests of significance. The 1.32 one did and the 1.21 one didn’t. If you try to pretend that they were “basically the same”, then in my book you’ve started fibbing and the rest is just haggling over terms. Finally, the context for my remark is a cheap joke; I’m not distorting a single word that Milloy wrote.

It’s true that debunkings of Milloy are weaker than those of Lott, because Milloy isn’t such an out-and-out, every-single-day hack as Lott. But he’s still pretty bad. This isn’t “innuendoes about funding”. This is a man who has, in the past, provably from documents, taken money from Philip Morris explicitly for the purpose of lobbying the epidemiological profession to take a meaningless and incorrect position on the subject of risk ratios, because it would be convenient for tobacco companies in future litigation. In other words, someone who needs watching like a hawk.

Milloy says that “the study reports that women who consistently drank one or more regular soft drinks per day during those four years actually gained slightly less weight than women who consistently drank less than one soda per week during that same period.” Is that true? Or is Milloy misspeaking here?

That is false. The study reports stable weight during the four year stable consumption period. Is it not possible that these [other] conditions cause increased soda intake (via unstable insulin levels) rather than a simple “drink more soda, get more diabetes” mechanism?

Yes, that is why factors like family history and obesity are taken into account. It is from these analyses that Milloy uses the 1.35 RR for obese women and implies that this means a 1.35 RR for all women when obesity is taken into account. This is misleading because the analyses used to adjust for obesity (or family history, etc.) is used only on those who are obese (or, etc.). The RR for non-obese is 1.78. We know from the weight gain portion of the study that soft drink intake is positively associated with weight gain. In other words, soft drinks are directly or indirectly associated with a significant increase in risk of adult onset diabetes. That increase is more than 35% for those without confounding factors. The greatest of those factors, btw, is family history, (RR 1.3). Neither of these diminish the RR associated with soft drinks, which remains at 1.83.

2.04 kg for <1 drink/week

(Milloy lists them in a different order from the paper.)

Hm, confusing because it was left out of the tables and placed in the text. It should have been in the table because it’s significant from a medical etiology standpoint. Thanks, Daniel.

Ed; I’m aware of the distinction. In context, I was talking about statistical significance, which still matters to some of us crusty old souls.

Furthermore, you say:
“The phrase “order of magnitude” is presumably meant to convey that the sugary soft drink risk ratio is not significantly different from the diet soft drink ratio, which is a curious way to summarise the paper, because the paper explicitlysays that the sugary drinks risk ratio is significant while the diet drinks one isn’t.”

This is misleading: the fact that one parameter is statistically significant and the other isn’t does NOT imply that we can reject the hypothesis that the two coefficients are equal. You seem to be implying that we can reject this hypothesis. The fact that the point estimates are close together, as Milloy points out, makes it likely that the difference is not statistically significant (but we don’t know this for sure without more information).

It seemed pretty clear that D2 was talking about statistical significance – I and any other scientist I know would not use significance to mean ‘substansive significance’, but prefer to talk about magnitudes of effects – either effect sizes, causal efficacy or even confidence limits. We wouldn’t use significance, as it already has a well-worn connotation when discussion results – that of statistical significance.

And he still managed not to mention Bill Maher’s favorite bugaboo, high fructose corn syrup! Seriously, though. Nothing in this study should be surprising. When I were a lad … sodas came in 7 ounce bottles and 12? ounce bottles and cans. They were deposit bottles. You could make chains out of the pop-tops. You could make the world sing in perfect harmony. No one I knew was allowed all soda, all the time. Most of us had to ask, and only the really lucky kids (the ones who never ate PBJ sandwiches and got chips and ho-hos {or the equivalent} in their lunch boxes {remember those? food prepared at and brought from home?}) got them as a matter of course. Heaven forbid they were offered at school.
Besides the fact that one can now buy junk food with food stamps, how the hell did we get here? With all the weight-consciousness propaganda flooding the airwaves, how is it that anybody regularly chooses to drink more than the occasional sugary soda? The diet stuff is questionable too — sweeteners rushed through FDA testing to market that turn out to have not-so-healthy side effects? Carbonation that can lead to osteoporosis? Perhaps we need even clearer labeling — like, “This 64 oz. soda has 800, THAT’S 800, you bonehead! calories — this means that it is 2/3 the calories needed PER DAY by a fairly active woman of average height and weight”

Milloy: When the researchers statistically adjusted their results for bodyweight…and for caloric intake…, the 83 percent increase dropped to an even more statistically dubious … 32 percent increase. That result is of the same magnitude as the study’s reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day

Daniel: Think about this for a second. According to Milloy, the correct (even the ethical) thing to have done in presenting the results of this study would have been to have headlined the ’32 per cent increase’ (1.32 relative risk) that one gets in a model which controls for body mass and caloric intake. This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?
——————————–

I’m not an economist, so I obviously don’t know that much about science, and I haven’t read the original article or Milloy, but I thought one of the keys to doing science is the control population.

If the point of your study is to show that a high calorie diet is worse than a low calorie diet, then you should compare individuals with as many of their non-diet characteristics as possible — for example, exercise and exposure to environmental toxins — the same.

If the point of your study is to show that a high sugar diet is bad for you, you should compare a population with a diet high in sugar to a population with a diet comparable in other aspects — for example, calories — but different sugar intake.

If the point of your study is to show that drinking non-diet soda is bad, you need to ensure that the effect is coming from soda per-se, and not just sugar or calories (or perhaps other environmental effects: one population living in Elizabeth, NJ, and the other in Eugene, OR), or your study is not a warning about soda consumption, but a warning about calories or air quality.

The correct way to account for the health risks due to soda and not due to calories in general *is* to compare populations with the same calories but different levels of soda intake.

Or, alternatively, I suppose, you could show a correlation between non-diet soda and calories — but that wouldn’t get you grant money, journal publications, or an appearance on NPR.

If your point is to give Milloy (whom I don’t know from Adam) a beating, though, I suppose any old straw man will do in the end.

dsquared: How about this question: should doctors advise their patients to reduce their consumption of Mr. Pibb?

You’d probably say: no, she should advise her patients to reduce consumption of all soda.

Milloy would say: no, she should advise her patients to reduce calories overall.

It is not obvious that Milloy is wrong about this.

If the real problem is calories, and the newspaper headlines are screaming about soda, it is natural that soda lovers and soda producers will be upset, with good reason. But if soda is the main cause of high calories, then your advice might be better. I don’t know if the study addressed that point.

“the question that is being adressed…is ‘should doctors advise their patients to reduce their consumption of non-diet soda to less than one per day?’.”

If the study doesn’t control for caloric intake, and if you control for caloric intake and the non-diet soda has no statistically significant impact, then the answer is “No, doctors should advise their patients to reduce their consumption of calories (or sugar, or whatever), but should otherwise make no special dire predictions about the health effects of soda.”

Why? Because suppose I simple-mindedly took the advice, gave up non-diet sodas, but supplemented my diet soda with a couple of candy bars (to ease that sugar craving that used to send me to the soda machine in the first place).

Has the advice to give up sodas done any good? Not if the problem is the sugar. In fact, this is just the simple inversion of your own objection! The patient might well change his or her diet (adding other forms of sugar) if forced to give up sodas because they were led to believe that soda, not sugar was the significant dietary risk.

Now for your examples:

“If I was carrying out a study into whether injecting heroin was good or bad for your health, then it would probably in principle be possible for me to control out the risks of AIDS, hepatitis, contaminants etc and get a fully controlled estimate of the increase in risk of premature death associated with injecting clinically pure diamorphine through new needles. While this would be interesting from a scientific point of view, would you really say that best practice would be for me to make this the headline number in reporting my study?”

How slippery this argument has become. The study Milloy was critiquing was published in a medical journal: it was presented to an audience of doctors as of scientific interest. The scientific interest was precisely the point and the basis of the criticism, I thought.

But, to address the objection on what independent merit it may have: you should make the headline read that *dirty needles* are the problem – because injection of anything through dirty needles carries that same risk. That’s why hospitals don’t re-use needles – see the value of getting the study right! You wouldn’t want hospitals re-using needles from the emergency room just because they are being used for things other than heroin. (Heroin has its own problems, but those are apparently not coming out in your study.)

“Or look at this another way; does your ‘control population’ make any sense? Why are you trying to control for calories in this way, when soda contains calories? If person A is taking in 5000 calories per day from food, while person B is taking in 2500 calories from food and 2500 from soda, then person B actually has a much healthier lifestyle, apart from the soda she drinks.”

Of course it makes sense. Does person B *actually* have a healthier lifestyle overall? I mean, how would you know? Oh, wait, I know. You would know by controlling for soda intake independently of controlling for caloric intake (and other factors, such as exercise). Then you would know whether trading food calories for soda calories carried an increased or decreased health risk, and then you could headline whether or not *soda* was a health risk.

“Would you recommend that studies of whether smoking tobacco is bad for you should use dope-smokers as their control group?”

If you want to assert that smoking tobacco in particular is a health risk that smoking dope (or oregano, or corn silk) is not, you *have* to.