Comments on: False positives

By: Chris

Chris — Tue, 30 Sep 2003 17:54:14 +0000

D-squared and Brian: the testing Gigerenzer discusses is routine screening on the lines of breast cancer screening. So those tested aren’t pre-selected because of some symptomatic indication that they might be at risk. Assuming that the prevalence is given for all those in, say, the age group selected for testing (rather than the general population), and that all this was explained to the doctors beforehand, it looks implausible that we can let them off the hook in the way you suggest and that they should, indeed go for the “orthodox” answer.

By: Brian Weatherson

Brian Weatherson — Tue, 30 Sep 2003 17:21:03 +0000

I agree with dsquared entirely on this one. On the question of priors, the ‘orthodox’ answer to these questions relies on treating the person taking the test to be a randomly selected member of the population. That’s, to say the least, unsupportable in theory and unreasonable in practice. On the question of probability, I’m enough of a Bayesian to think that we can talk about a probability here even if it is a single case, although it just means something like reasonable degree of belief. (I don’t think dsquared means to disagree that Bayesians can talk that way.) If you’re not a Bayesian, it requires an odd interpretation of the terms involved as (something like) generics rather than (something like) referring expressions in order for those terms to denote a class for the terms to be defined over. Having said all that, I’m still worried the doctors were off by so much. Bayesian/Hayekian/Keynesian considerations can account for some movement from 5% (in particular upwards movement) but not I’d have thought all that we see. It would be interesting to know the proportion of people who actually take the test and test positive who have the disease. If that’s closer to 5% than 50%, the doctors have some explaining to do.

By: dsquared

dsquared — Tue, 30 Sep 2003 06:52:01 +0000

The question is phrased so as to refer to the probability of a single event, surely, and I don’t think that phrasing it in the counterfactual changes the reference sufficiently (I will defer to Mr Weatherson on this one if he disagrees). In the context of a Monty Haul page, the reply would not be so much that hypotheticals aren’t single events, but that the structure of the question (“Would you switch?”) invites a reply phrased in terms of probability-as-degree-of-belief rather than classical probability. The question “Do you do better by switching?” is explicitly a question about the expectation of a process rather than the probability of an individual event.

By: Keith M Ellis

Keith M Ellis — Mon, 29 Sep 2003 20:50:07 +0000

Daniel, “…is ‘1 if they have colorectal cancer and zero if they don’t’, because probability isn’t defined over single events in conventional probability theory.” Yes, but isn’t it the case that since this is a hypothetical, we’re not talking about a single event? This objection comes up occasionaly from people that write to me about my Monty Hall Problem page.

By: Martin

Martin — Mon, 29 Sep 2003 16:53:45 +0000

I strongly suspect that communication between my wife and I and doctors was negatively affected in an important circumstance by the tendency of many doctors not to think in statistical terms. In the 1980s, my pregnant wife had an amnocentesis and the relevant doctor initially reported that one of the fetus’s chromosomes appeared short. After further examination, and comparison with our chromosomes, the doctors told us that they had concluded that things were OK. Upon our child’s birth, it turned out that the relevant chromosome was, in fact, missing a portion of genetic material, with very serious developmental consequences. Lookingback on things, I think that, in the genetic counselling sessions, my wife and I implicitly were expecting information in some (necessarily rough) statistical form—“You have about an x% change of a problem.”—and were prepared to make a decision about having an abortion based on this sort of information. The doctors’ instinct, however, seems to have been to reduce the available information to a binary OK/not-OK form. (My wife thinks that the doctors were tilted toward the binary way of presenting the data because, consciously or unconsciously, they did not want to encourage an abortion unless they were sure there was a problem. If so, this had the effect of placing a lot of the relevant moral, social, etc. decision process in the doctors’ hands instead of our’s.) At the time, I developed a hypothesis, based on no real evidence, that doctors (often) are bad at statistics because in many situations the practical decision they make will be the same for a wide range of statistical results. For example, suppose a test shows a probability of some medical condition of fifteen percent. if the condition is significant, the doctor will have to take action (more tests, precautionary therapy, etc.) while taking into account the possibility that the condition does not exist (e.g., keep an eye out for signs that the patient’s symtoms are caused by something else). However, if the true probability of the condition were 85%, it seems to me that the doctor’s practical decisions would be much the same—take action to deal with the condition while also allowing for the possibility that the condition does not exist. I hypothesize that frequent exposure to such situations trains doctors to ignore the fine points of statistics. On the other hand, our pediatrician frequently refers to statistical results from the literature when advising us on medical treatment and tests for our son. However, he was recommended to us by a sociology professor, so there’s some selection bias right there.

By: dsquared

dsquared — Mon, 29 Sep 2003 14:42:59 +0000

Call me a master of the bleedin’ obvious, but the question here wasn’t asked about mammograms. It was asked in a particular way, with particular framing effects (“imagine giving someone the Haemoccult test”). Notoriously, trained statisticians get statistics problems wrong if you set the problem up with the right frame. And secondly: Also – if a doctor is offering an estimate based on some intuitive judgement about ‘priors’ surely to do so they need to have some statistical data about the population, if they believe the sample isn’t drawn form a truly random population? Not at all, IMO. This is a point of controversy in the Bayesian/Classical statistics debate, and it’s one that interests me greatly. A prior can be informative without being based on statistical data about the population (indeed, a Bayesian purist would argue that if it’s based on statistical data, it isn’t a prior). The relationship of informative Bayesian priors to Hayekian tacit knowledge is a fertile ground for research …

By: Ian

Ian — Mon, 29 Sep 2003 10:58:16 +0000

So how does this argument affect the probabilities when applied to screening tests such as mammograms? Also – if a doctor is offering an estimate based on some intuitive judgement about ‘priors’ surely to do so they need to have some statistical data about the population, if they believe the sample isn’t drawn form a truly random population?

By: Matt McIrvin

Matt McIrvin — Mon, 29 Sep 2003 05:26:24 +0000

When I described this question to my wife, she said the same thing dsquared did: if the doctor is testing the person for the cancer in the first place, then the doctor probably saw some additional indication that the patient might have cancer. So a good prior really ought to take that evidence into account as well; it would therefore be higher than the prevalence in the general population. It’s possible that the doctors were thinking about that when they answered the question, instead of making the assumption that the test is being given to a random person with no independent indications of cancer.

By: dsquared

dsquared — Sun, 28 Sep 2003 20:48:05 +0000

I think I’ll stand up for the doctors here, and also make the same Hayekian comment I make every time one of these stories come round which puports to prove that professionals don’t know anything about their field of expertise. The maths of the matter was stated succinctly above: So, of the people who take the test, 3% + (0.3% * 50%), or 3.15%, will get positive results. Of that 3.15%, 0.15/3.15, or 4.8%, will actually have cancer. But note that this is only true if you’re going to take “the people who take the test” as referring to “the population as defined by frequentist probability theory”[1]. In the mind of a doctor, this phrase is much more likely to have the referent “the kind of people who you test for colorectal cancer”. This means that the assumption of non-informative priors made above isn’t valid. I’d say that the doctors are using a rule of thumb (or working from tacit knowledge of colorectal cancer), and that you’d need to do a lot more testing on whether it was a valid rule of thumb before you started calling them ignorant. On the other hand, back in the days when I was taking an interest in the debate over the harmfulness of the MMR vaccine, I heard enough outright ludicrous statistical arguments coming from members of the medical profession that the underlying accusation is certainly not without grounds. [1] Of course, a mathematical purist looking at this example would excoriate the lot of us for our ignorance and point out that the only correct answer to the question “what is the probability that someone who tests positive actually has colorectal cancer?” is “1 if they have colorectal cancer and zero if they don’t”,because probability isn’t defined over single events in conventional probability theory.

By: pathos

pathos — Sun, 28 Sep 2003 20:01:03 +0000

The true philosophical issue though, is what if the person has the disease, the test is not sensitive enough to detect it, but nonetheless gives a “false positive” for unrelated reasons that correctly diagnoses the cancer. Was the test accurate or not?

By: Bill

Bill — Sun, 28 Sep 2003 18:12:12 +0000

You can also use an “odds” method, where the odds of having the disease equals the probability of having the disease divided by the probability of not having the disease. Then, the odds of having the disease after seeing the positive test result equals (odds before test)*(true positive rate/false positive rate) This isn’t particularly obvious, but you can prove it using the formulas given above, and it quickly gives an answer.

By: Harry

Harry — Sun, 28 Sep 2003 17:06:55 +0000

Sorry to have written so unguardedly Keith—I was being ironic myself. It was what I came to think after these extremely frustrating experiences. The point is that neither medical schools nor the HMOs that employ doctors (or the NHS for that matter) think of doctors as needing to understand these technicalities. I’m not defening them, just observing.

By: Keith M Ellis

Keith M Ellis — Sun, 28 Sep 2003 15:10:21 +0000

“Why expect people who have no training in a highly technical area to know anything about it?” Well, the, uh, irony, is that so much of medicine is all about statitistics and probabilities. Really, physicians should have a very strong grasp of these concepts as well as a strong technical facility.

By: Harry

Harry — Sun, 28 Sep 2003 14:44:27 +0000

This is unteresting, but not at all surprising, at least if you’ve tried to get useful information about probabilities from doctors. My two personal experiences—discussing the side-effects of a life-long every-day medication; and trying to get advice about whether to schedule a second Caesarian (for my spouse, obviously, who is small, and cursed with having unnaturally large babies)—started out with me believeing they were evasive, but left with me believing that they didn’t understand the questions I was asking. Why expect people who have no training in a highly technical area to know anything about it?

By: Anarch

Anarch — Sun, 28 Sep 2003 06:54:48 +0000

Does it say how long the doctors were given to answer the question? I’ve been under the weather today, but even as a math grad student I wasn’t able to answer that without some kind of back-of-an-envelope calculation. Mind you, those doctors should be able to estimate that figure off the top of their heads, but the results would be even more damning if they were given a significant amount of time to think about it.