Colliding with Mystery – or – It is difficult to get a man to intuit p-values when his h-index depends upon his not intuiting them

I’d say overall, you did quite well! The most obvious next step is to just admit that we should all be Bayesian in our treatment of credences, and that one’s prior probabilities actually matter quite a bit for theory choice. (Which is almost what you said, without the jargon — if your prior for a coin being unfair is 1 in ten million, the fact that the likelihood of five heads in a row is greater under that hypothesis than under the fair-coin hypothesis won’t budge your credences all that much.)

Particle physicists have the luxury of waiting for five sigma since their data is very clean and they know how to collect more and more of it. But being honest Bayesians is becoming more popular even there.

I had a go at this a while back. My conclusion

what has been the dominant response in practice is to disregard the “95 per cent” number associated with classical hypothesis testing theory and to treat research findings as a kind of Bayesian update on our beliefs about the issue in question. If we have no prior beliefs one way or the other, a rough estimate is that a finding reported with “95 per cent” confidence is about 50 per cent likely to be right.

…Daniel Drezner, when he decried “piss-poor monocausal social science.”

He’s exactly right, though. This is really the source of all of our problems today.

I don’t know what it means to “intuit p-values” but I would suggest that “It is difficult for a scientist to report p-values greater than .05 when their H-index depends upon rejecting the null hypothesis.”

It should be mentioned that the appropriate choice of p-value depends on how many hypothesis tests are (or can be) conducted. Observing a 5-head run is unremarkable if hundreds of teams are conducting coin flips experiments. One reason for a very high statistical threshold in physics is that they do have virtually unlimited data. You would always expect to find some irregularities due to random effects. It may be appropriate to have a lower threshold in other sciences but a p-value is never in itself proof of a theory.

TM says:”One reason for a very high statistical threshold in physics is that they do have virtually unlimited data. You would always expect to find some irregularities due to random effects.”

I don’t think this is correct. The statistics of particle physics treat the entire LHC as 1 experiment, or at best as a few campaigns. As such, the effect the large dataset is already taken into account in the sigma-values.

Instead of flipping coins, what about drawing different-colored balls from an urn? (Urns are nicely Grecian.) You draw one ball and it’s black. What confidence do you have that all the balls are white?

The math of how you know a given number of observations gives you a particular p is yet another thing, however.

Argh–flip one of those colors to the other one. The math may well be different for coin-flipping and urn-drawing. I’m used to seeing urn-drawing in this kind of situation and feel like it’s more intuitive.

11: Let’s try again. The *power* of a significance test (to detect an effect) depends on the number of observations. There is always a trade-off between power and significance or Type I and Type II error. However, if you have an immense number of observations, you can afford to set the significance level very low and still conduct a powerful test. Most experimenters aren’t in that position, they have to choose a relatively high significance level, otherwise they would never detect an effect even if it is real. That means of course that they can’t just be content to report a barely significant result: more studies are needed to confirm the result.

The coin flip is a good example. It is easy to devise a very powerful test and choose a very low significance level – just throw the coin a thousand times instead of just five. That strategy just isn’t usually available to real world experimenters.

kiwanda @16
it is called money and time: LIGO spend a lot of money on a lot of smart people over a long time
you do enough of that , you can measure really really reaaaaaalllly small things

I think the coin examples are a bit confusing, as we are rarely interested in the behavior of coins, fair or not. (If we want to test whether a coin is fair, we’d likely use a chi-square test).

Here’s a better example: You poll 100 Republican voters in Montana, 75 say they will vote for Trump, 25 for Cruz. Let’s assume just for the sake of argument that everybody votes, so we don’t have to worry about likely voters.

The null hypothesis would be: This was just a statistical quirk, Trump is no more likely to win Montana than Cruz.

The alternate hypothesis would be: It is not a quirk, Trump is favored in Montana.

Perhaps the willingness to engage in reductionism contributes to a difference that would weaken the analogy? We’re inferring invIsabel causes from observations of something else. The medical researcher, on the contrary, doesn’t have to infer that the efficacy of a test is reducible to lower-order biochemistry and so on. We dislike reductionism in social science. And for good reason.

Oops, I screwed up the link. Should be: Why Most Published Research Findings Are False

People do tend to favor their own survival. If productivity is measured in p-values, then p-values will be produced. The alternatives which have been suggested – that scientists, say, should all be given endless amounts of money to produce nothing in particular – should be silly prima facie, but have been given surprising amounts of attention, even supported by arguments that _not_ doing so amounts to censorship and anti-scientific behavior.

The real problem is that curiosity-driven science wasn’t putting food in peoples’ mouths much of the time. All scientists were suspicious of applied science publications for good reason; but pure science was published by people who made their money as teachers, physicians, or something else. They could afford, if you will, to have high standards.

Mostly, just like Sean said @3.

One point that hasn’t been emphasized here is that the dopey p-value convention not only produces a lot of false positives but also in important cases produces a lot of false negatives. If, for example, the manufacturer of some chlorinated hydrocarbon that looks just like a known carcinogen, but with an added methyl group somewhere, is allowed to pick a “null hypothesis” that the new substance is safe, an unreasonable amount of evidence will be required before concluding that it isn’t. Bayesian methods have limits, but they at least don’t hide all the subjectivity in an occult process by which null hypotheses are chosen. Sander Greenland has written extensively on this issue.

erk @19: I’ve heard “proof by money and authority” before; it doesn’t answer the question. There are a lot of really really reaaaaaalllly small things around, including the ones I mentioned; how do they know that the one they detected must have been a gravitational wave? I’m sure all those smart people (well, smart for physicists) had ways to eliminate all other explanations. What were they?

I see the gravity wave deniers are here (Kiwanda @16) …

with no specific knowledge of this, firstly the wave was detected in Washington state and Louisiana, so there is a record of a wave passing through being detected in two places, and if it was a truck passing by or a dog barking then events such as this will happen quite a lot so should be being able to be detected, and again two trucks have to pass or two dogs bark by the two detectors at the right time separation and be not like other trucks passing or dogs barking at other times …

https://www.ligo.caltech.edu/news/ligo20160211

It’s not just bad stats.

If read social science papers, you’ll start to notice certain tropes. Like this:

“Here are our results. We studied twenty things. Some had statistically significant correlations with our variable and some didn’t. Some replicated prior studies and some didn’t and some prior studies were themselves inconsistent with each other. Here are ten paragraphs interpreting those results via cultivation theory. Here are conclusively statements about how our results support cultivation theory, except for the ones that weren’t statistically significant. Here are explanations minimizing the importance of the statistically insignificant parts and the parts that don’t replicate prior studies. Here are three sentences conceding that there are other explanations and cultivation theory might not be real, but please don’t read this far.”

Swap out cultivation theory for whatever pet interpretive framework is at issue.

@33 Sorry Dipper, posted before yours came up on my page.

Look forward to reading this. Jordan Ellenberg has a delightful discussion of p-values in How Not to Be Wrong.

Frisbee beats urn beats coin. Sure thing.

38

Come on. Do you really think they haven’t thought about this? The LIGO people put tremendous effort into imagining as many possible alternate explanations as they could. What they know is that

a) it’s not any of the (very large) number of possible alternate explanations they considered
b) it is exactly consistent with the predictions of gravity waves.

Could it be some other random unexplained thing? Sure. But if you have an observation that matches one explanation and you have ruled out as many other possible explanations as possible, Occam’s razor says you go with the explanation that matches until you have evidence that contradicts it.

Gelman’s good (and I follow), but Cosma Shalizi’s my go-to for this sort of thing: f’rinstance,
Any P-Value Distinguishable from Zero is Insufficiently Informative
(or search bactra.org for other such) (also via, see Jordan Ellenberg, How Not to Be Wrong: The Power of Mathematical Thinking)

A-and, Pynchon’s Proverbs for Paranoids #3, ” If they can get you asking the wrong questions, they don’t have to worry about answers,” is invertible: “If you worry about answers, be sure to ask your questions wrong.”

1) If this coin is fair, odds are less than 1 in 20 that you could match or beat that 5-heads run I just got!

2) Probably this is a trick coin!

I am probably just confused, but “if this coin is fair” (and the “1/32 < 5%" thing) seems relevant only if your null hypothesis is p = .5, i.e. that the coin is fair. "This is a trick coin" only seems relevant if your null hypothesis is p = 0, i.e. that "someone has slipped you a trick coin that always comes up heads."

If your null is p = 0, you obviously can't reject it the coin always comes up heads. You can only reject it if it comes up tails at least once.

If your null is p = 0.5, then if your alternative is p not= 0.5, then actually I think "1/32 < 5%" isn't quite right, you have a 2/32 chance of getting a result that "extreme," as they say, because you could also get 5 straight tails. If your alternative is p < .5 then the "1/32 < 5%" I think works, so it seems to me that implicitly that is what is really being tested for, that it is a trick coin that comes up tails less than 50% of the time, rather than a trick coin that always comes up heads. And of course if all of the coins in the world are fair, 1/32 of the time you'll prove the coin is not fair, and 1/32 of the time you'll be making a type-1 error.

Like I said, I could be very confused. I await correction….

People gloss it wrongly: ‘the p-value tells you the likelihood that your result happened just by chance’ (and variations on that thought.)

This seems not wrong, just incomplete. How about: the p-value tells you the likelihood that, if the null hypothesis is true,, your result (i.e. sample), or a result (sample) even more favorable for the alternative hypothesis,, happened just by chance.

It is not surprising this stuff is difficult and counter-intuitive. We’ve had an understanding of the syntax of probability statements and their mathematical manipulation since the 1660s, but we still have no agreement on what they mean (their semantics), nor even on whether probability is the best formalism for representing uncertainty.

Naomi Orskes and Erik Conway imagining Chinese scientists three hundred or so years from now reflecting on inaction on climate change:

“Yet overwhelming evidence suggests that 20th c scientists believed that a claim [linking e.g. intensifying hurricanes and warmer sea surface temperatures]could be accepted only if, by the standards of Fisherian statistics, the possibility that an observed event could have happened by chance was less than 1 in 20. Many phenomena whose causal mechanisms were physically, chemically or biologically linked were dismissed as ‘unproven’ because they did not adhere to this standard of demonstration. Historians have long argued about why this standard was accepted, given that it had neither epistemological nor substantive mathematical basis. We have to come understand the 95% confidence limit as a social convention rooted in scientists’ desire to demonstrate their disciplinary severity.”

@58 So say you flip a coin five times on 100 different occasions. On how many of those occasions would you have to get a run of five heads to reject the null hypothesis that the coin is fair?

Loving this, but must dash away. Will return later, but without Babbie!

Kiwanda, here is the NASA press conference annoucning the dsicovery of gravitational waves:

https://www.youtube.com/watch?v=_582rU6neLc

I found the evidence presented there very convincing, and I expect more confirming detections of similar events will be found with greater precision as more LIGO installations go on-line. I realize however that no amount of evidence need convince someone who does not want to be convinced.

@JimV #69 — Quick note: the press conference announcing the observation of gravitational waves was at the NSF, not NASA. This may be OT and minor to some, but important to others; the NSF funded LIGO consistently for decades and deserves the credit for the scientific success.

JimV, thanks, but I doubt if the press conference had more detail than the PhysRevLett and arXiv papers I linked. I apologize for being curious about the question of eliminating false positives, which was not covered in the popular press articles I saw.

I have read all the comments and I don’t understand the coin flipping example either. You want your null hypothesis, which you are trying to reject, to be “this coin is fair” or something like that. But that misses the essence of what statistics is about. Given a particular coin (“this coin”), we can simply test whether it is fair or not. The point is that statistics is about populations where you can’t test every member. That’s why reasoning about them is hard.

Reviewing my comment @68, I want to clarify my last paragraph there. There are always only two possible outcomes of any coin flip, H or T, regardless of whether or not the coin is fair or biased. Therefore, we always use 2 as our factor of possible outcomes of a coin flip. We never want to “adjust” that factor unless we are working with a population of biased coins whose distribution of H and T flips we know, and unless we want to study outcomes of flips among the population of those coins.

In practical use p values deal with two fundamentally different things. One is to deal with rejecting the null hypothesis given a mass of data (and therefore potential hypotheses) which may be small or large. For an enormous mass of observations, you need a correspondingly low p value for a null hypothesis to be soundly rejected. This explains the astonishingly low p values required in physics and genetic SNP experiments, for example. The requirements here can be established in a fairly straightforward fashion given the other thing…

The other thing is to deal with the null hypothesis given a particular number of experimenters or long running experiments or publications. This is really where the p < 0.05 convention plays. It may be tradition (and little else) to accept that 1/20 experiments may be rejecting the null hypothesis falsely. Back when there were far fewer experimenters doing only long running experiments for which papers were rarely published this made a lot of sense; I doubt this is still true.

I fear if you look carefully at the traditional requirements in specific fields, they boil down to p < 0.05 adjusted appropriately to typical numbers of potential hypotheses in experiments in those fields. But they do little to negate the problem of 1/20 experimenters/experiments/papers rejecting the null hypothesis falsely.

Because of the above, p value alone can't be taken as proof of anything. p < 0.05 is really only a starting point, to weed out the garbage and give us a somewhat smaller set of hypotheses to consider. Proof isn't in individual p values–it's in endless replication, true forward prediction, and intuitive explanation–which helps to drive the pursuit of endless replication.

Then you have vast areas of human knowledge where controlled experiments are not typically done or would be impossible. That is how things are in MOST areas of human knowledge or whatever you want to call it.

61 and others, the key here is understanding what a binomial experiment is. When you throw a fair coin say 1000 times, you can calculate precisely the probability of observing 600 heads. That probability is tiny (it does in fact constitute a 6-sigma discrepancy from the expectation value, since sigma=sqrt(npq) for a binomial experiment). So if you do observe 600 heads, you can conclude with a very high confidence that the coin is not fair.

It really shows that the basics of statistics are poorly taught. Too much emphasis is on advanced techniques (even in introductory courses) before students have had time to grasp the basics.

79 corrections: Should be assuming N<<n, and p=k/n. Regret the mistakes.

On the gravity wave question above: I have no connection with gravity wave experiments, but use stats a lot in my day job. A question you might want to ask: given that I have two detectors, each of which produces an output with a fair amount of random noise, how likely is it that both detectors will see random noise that (a) looks approximately the same at both detectors, delayed by speed of light between them and (b) looks approximately like the kind of chirp you’re looking for. “approximately” because the detector output is noisy, and outputs are not expected to match exactly even when you see a genuine gravitational wave. I presume the people working on that experiment have done the statistics. But, re. the discussion above, in experiments of that form it isn’t always obvious that the false positive rate will be small, and you”ld better actually work it out in case it’s embarrassingly large. (Looking for an event of duration about 1 second with a detector running for months/years is a lot of parallel testing).

This is in principle the same as John’s coin flip example, just more complicated to work out the details.

You can only test a specific hypothesis. In statistics, a null hypothesis is always expressed as a mathematical equality and the alternative as an inequality. The hypothesis “the coin is fair” is p=.5 (where p is the probability of throwing a head). You can also test the hypothesis p=.6 or whatever, if there’s a good reason for doing so. It is NOT possible however to test the hypothesis “this coin is rigged”, because a coin can be rigged in an infinite number of ways and you cannot test each of them simultaneously.

That is also, I believe, the real reason why it’s important to insist that rejecting the hull hypothesis is NOT the same as accepting the alternative hypothesis. This is another complication that students have a hard time understanding. Rejecting p equals .5 seems to be logically equivalent with accepting p unequal .5 but the latter isn’t in itself a testable hypothesis. It’s an infinity of distinct hypotheses and you don’t know which of them is true, even if you have strong evidence that one of them is probably true.

Gelman is cited as writing “Ultimately the problem is not with p-values but with null-hypothesis significance testing, that parody of falsificationism in which straw-man null hypothesis A is rejected and this is taken as evidence in favor of preferred alternative B.”

The thing is, it is not at all clear in what sense null-hypothesis testing is a parody of falsificationism. (In fact, it is not at all clear to me that “the” null-hypothesis is always the same thing, or at least, that it should always be. Is Ronald Fisher the seal of the prophets?) In the case of evolutionary psychology, it really is difficult to tell how, for instance, statistical significance testing of the hypothesis that men evolved to prefer specific ratios in female body measurements doesn’t falsify the hypothesis when p-values indicate no statistically significant difference in outcomes of an experiment. Then, it is difficult to tell why, if the p-values do demonstrate the existence of a statistically significant difference, the hypothesis should not merit further testing.

If, like Gelman (and the OP too?) I use my intuition drawn from seeing variations in body type from cursory familiarity with other cultures, plus visible changes in old movies, inexplicable as an evolutionary change, to conclude that something’s wrong with this notion? That’s not falsification. As I understand it, that sort of thing is verification, something falsificationism explicitly rejects. (By the way, as I understand it philosophers of science among themselves will claim Popper, the great proponent of falsification, is somewhat passe. But they more or less keep this news to themselves, and Popperianism tends to dominate everywhere but philosophy journals.) Falsificationism may be a broken hammer, but people will keep using a broken hammer when good new hammers are not on sale.

I’m not sure that Bayesian reasoning isn’t a good deal like Moliere’s prose. I suspect there is frequentist Bayesian reasoning, though I may be misled by a persistent tendency to think of Bayes’ theorem when anyone says “Bayesian.” It seems to me that Bayes’ theorem is easily understood in frequentist terms. On the other hand, it’s not at all clear to me that propositions/hypotheses have probabilities in any natural sense.

It’s not clear how Bayesian reasoning creates a non-straw man falsificationism. It is especially not clear how Gelman’s preferred improvement of measurements will do that. Skipping around I see Gelman talks about “noisy data,” which certainly seems to have a natural meaning, but it’s not clear what that means in falsification. He also complains about “open ended theories,” but Googling just finds a contrast between open ended and stage theories of human development. Stanford Encylopedia of Philosophy doesn’t seem to know that either. I assume it’s not really that obscure but somehow I’ve missed this.

At any rate it’s certainly depressing to find that only the most sophisticated statistics will lead us to true falsificationism. I guess that means most scientists are not really scientific? And of course most fields of study are not scientific either?

TM’s comments are helpful here, including this @82: “Of course you CAN posit a hypothesis.” But I was responding to what may very well be my misunderstanding that John was setting a condition, for the sake of his illustration about his 1) and 2), that the coin is fair. I thought he was positing that the coin is fair for the sake of his argument, not as a hypothesis to be tested. It’s as if he were saying, “I’m flipping what I know to be a fair coin. It lands five heads. The odds of this outcome are 1/32. If I didn’t know this were a fair coin, a p-value<.05 would lead me to guess this is a trick coin." My point was that it makes no sense to test a known fair coin for fairness.

John@86: "Equally obviously, you suspect 2 is not the right number." This is not obvious to me. That was the gist of my comment @77. Setting aside the slim possibility that a coin will land on its edge, there are only two possible outcomes of a coin flip. That goes for fair and trick coins. But on reflection I see where you're heading. (I've had my coffee and am now capable of crafting puns.) I was aiming that direction in @77, too. If we were working with coins biased to land heads 1/3 of the time, we would calculate the odds of five heads as the reciprocal of 3x3x3x3x3 or 1/243. Then we would flip random coins five times. If we encounter five heads we would strongly suspect we had flipped a coin biased differently or not at all. In any event, the world we are in remains one in which fair coins will land heads 50% of the time and our special known biased coins will land heads 33% of the time.

There is a 1 in 32 likelihood that this happened just by (longshot) chance. That is, given 5-heads, there is a 1-in-32 chance that you happen to have picked a fair coin (as likely as the alternative); then (flukily) you flipped 5 heads with it. On the other hand, there is a 31 out of 32 likelihood that this didn’t happen (just) by chance. Rather, you picked a trick coin….

Now that I am beginning to see the point (maybe), I thought I’d point out that you meant “1 in 33” and “32 in 33” here. (Pick up 64 coins, 32 Mysterioso, 32 yours, on average 33 times you should get 5 heads in 5 flips….).

If 31/63 of the coins are Mysterioso’s and 32/63 are fair then the p-value and the actual probability will coincide.

So if you want to explain to someone why their ‘likelihood that this thing happened just by chance’ intuition about p-values is wrong, flip it and tell them what they are thinking could be right, but only if they just collided with Mysterioso, as it were.

So I think what’s really being said is that in Mysterioso world, where P(true null) = .5, the p-value (1/32) is very close to the actual probability (1/33) of the event “happening just by chance;” whereas in the real world, where P(true null) = 1 or else P(true null) = 1, the actual probability of the event “happening just by chance” is either 100% (if P(true null) = 1) or some probability not likely to equal the p-value (if P(true null = 0).

The way I would think of this is that if you pick a coin out of the fair coin jar, the probability of 5 heads occurring “just by chance” is 100%. If you pick it out of the Mysterioso always-Heads jar, it’s 0%. If you pick it out of a jar with 31 Mysterioso always-Heads coins and 32 fair coins, it’s 1/32.

94: “p-value claims are not about the population. They are claims about our sample from the population. We are willing to tolerate a 5% chance that the sample is not a fair representation of the population and that our remarkable result is due to an unrepresentative sample.”

No, this is backwards. We are testing a hypothesis about the population, not about the sample. And we do assume – we have to assume – that the sample is “fair”, i.e. it is a true random sample. A 5-head run isn’t an “unfair representation” of flipping a fair coin, it’s just a somewhat rare outcome. Such an outcome was nevertheless to be expected, with probability 1/32; it is NOT evidence of anything going wrong (gust of wind, coin not flipped properly). Of course it could well be that something went wrong with the experiment, but that is an issue outside of statistics. If the experiment wasn’t conducted properly, statistics can’t fix it.

TM @97 & 98: Taken together, I agree with these statements, both in deference to your authority and because it’s what I intended. A significance test is about the sample, and a hypothesis is about the population. We generalize from our observation of the sample to the population when and if the significance test gives us the confidence to do so. Statistics CAN give us a clue as to whether or not the experiment was properly conducted. Thus, a five-head result could be evidence that something is amiss, but not evidence of anything in particular being amiss. Yet if we review our tally, say, and realize we mistakenly coded two flips as heads when we remember observing two tails, then we credit the significance test for alerting us to the sampling error. Granted, a coding error isn’t precisely a sampling error in the sense that we failed to assure a random sample, but the illustration with which we’re working–a single coin flipped five times–renders the question of sampling error moot. The population is the single coin, hence it isn’t really a sample at all.

I wrote something once about coin tosses and p-values.

hang on, i asked this at 14

Sorry, mea culpa, perhaps i didnt

I like 107. This null hypothesis stuff always struck me as being a bit dodgy.

I’m wrecking their brains enough by making them read Plato. They oughta be able to relax with something easier.

The Nicomachean Ethics is the obvious answer here.

That joke was irresistible (probably only to me!) But seriously, I don’t get p values at all and this post (and thread!) were totally helpful. Thanks all!

Thanks for the reply, JH. (And I also found parts of the post/thread helpful.)

Sorry. I did actually read it, but that joke makes no sense.

100: “And 95% of the time we will make the right decision—i.e., the right inference from our data–if we follow this procedure.”

Important quibble: the chance of a false positive is 5% or less. The chance of a false negative however is usually not known and could be quite high.

That is essentially what Oreskes has criticized: scientific conservatism dictates to minimize false positives but maybe we should be more concerned about false negatives? The example of toxicity testing was brought up by mbw @29. Under the precautionary principle, one should balance the risk of not banning a toxic substance against the risk of banning the substance when in fact it is not toxic. Or the risk of not acting against global warming against the risk of acting prematurely, or some such. In the case of Climate Change, I think it’s a red herring. Political inaction can’t be blamed on p-values. Scientific evidence is strong, and remaining uncertainties don’t justify inaction. Deniers have exploited the fact that warming between 2000 an 2012 wasn’t statistically significant, but that was always a transparent abuse of statistical methods. Climate Change denialism can’t be construed as a failure of scientific method, it’s squarely a political failure. And of course, changing the scientific methodology will do nothing to change the politics.

John Holbo, I think this is exactly wrong:

Rather than saying ‘given the evidence, what’s the likelihood of the hypothesis’ they are saying ‘given the hypothesis, what’s the likelihood of this evidence’.

As an example, here is Rutherford:

It were as likely as shooting a cannon ball at a piece of tissue and having the cannon ball bounce back.

This null hypothesis vs. alternative hypothesis approach is the main way that we do experimental reasoning.

To use my example above, how do we decide the globe is warming? We check to see if the trend in the temperature is greater than 0. It doesn’t matter what your specific mathematical process for doing that check is, the reasoning is always the same: set up a null hypothesis (no warming) and test to see if what you observe fits it (a slope in the temperature series).

There is no other way. You can use Bayesian stats or p values or whatever but ultimately you’ll be doing a Rutherford: positing a null hypothesis under an existing theory and breaking it.

If it was good enough for Michelson and Morley, it should be good enough for you!

I should say, your objection is not the size of the p value, it’s the logical process.

You have failed to guard against this reading, John. You said p-values are “unintuitive” because they get the logical process backwards, and identified exactly where. This is obviously a criticism of the structure of the reasoning, not the fact that you think they only work in rare cases. And why should they? We do comparisons that depend on not so great differences all the time. For example, differences between the sexes, or between occupational groups – these may not be so great as to drive huge p values. In the case of Rutherford and Michelson Morley they were comparing one theory with another theory, so we can expect them to fail huge. But that is not what stats is for – stats is for comparing differences between groups that share a common theory.

To use the example I put up at Gelman’s, suppose you’re trying to identify whether there are different rates of female genital mutilation (FGM) between muslims and non-muslims in Nigeria. Many different religious and ethnic groups practice this to some extent, and you’re not comparing to radically different physical theories – you’re simply trying to identify if there’s a difference. If that comparison annoys you, think of a difference in malaria risk between wealth quintiles, which could be due to differences in uptake of insecticide treated nets or could be due to environmental/structural differences (more sources of stagnant water, etc). This is a case of small differences that arise from realistic application of a theory of social determinants of health, not two conflicting theories and a carefully designed experiment that can distinguish between them. But it’s an important difference – who to target and how. Your experiment is going to come down to relatively small differences.

The advocates of alternatives to p-values typically don’t think of how to handle these problems. But I can bet you that whatever approach they take will end up using the same logical reasoning (null vs. alternative). The only difference will be the particular mathematics they use to compare their two hypotheses. Do you have a suggestion about some radical alternative that would change the structure of the reasoning? Or is your argument purely that we shouldn’t boil it down to a single number that is formally equivalent to one of the major alternatives?

“Do you have a suggestion about some radical alternative that would change the structure of the reasoning?”

First, I agree with John, there is nothing wrong with the reasoning per se. But it can be misunderstood, due to its counter-intuitive nature, and it can also be abused—perform enough experiments and inevitably 5% of them will turn up something “significant.”

But there are also a couple of things that can be done to address the problems inherent in p-values—one frequentist, one Bayesian.

Within the frequentist school, you can introduce the concept of the “power” of the test, as a companion to “significance.” As various commenters have noted, the p-value is good if you primarily want to avoid false positives. (TM’s “important quibble” @123 above is quite correct.) The null hypothesis is like the accused in Anglo-Saxon criminal justice—innocent until proven guilty. And the bar for “guilty” is set very high. Users of p-values, whether they realise it or not, would prefer to see hundreds of “guilty” null hypotheses go free in order to prevent even one “innocent” null hypothesis from being convicted.

The concept of “power” introduces trade-offs between avoiding false positives and avoiding false negatives—it says that avoiding false positives comes at a price that we need to recognize. Specifically the “power” of a test is the probability that the test will lead to acceptance of the alternative to the null hypothesis when the alternative hypothesis is in fact correct. It is equal to 1 minus the false negative rate. It asks the following question: “Conditional on, or given that, (i) the alternative hypothesis is true, AND (ii) the experimenter will only reject the null based on the p-value being less than 5% (i.e., conditional on the null only being rejected if, conditional on the null being true, it would only have a 5% chance of producing the evidence that we saw), what is the probability that the alternative hypothesis will be accepted?” Put in different terms, it asks what is the probability that, if the alternative hypothesis is true, it would produce evidence (e.g., a sample mean) that lies in the zone where we would accept the null IF IT WERE TRUE. If we minimise the latter probability, then we maximise the power. But, all other things equal, greater power only comes at the expense of less significance (i.e., a higher false positive rate). (This is much easier to illustrate with a diagram of overlapping normal distributions–one centered on the null, one centered on the alternative–but cumbersome, I see, to express in words.)

Note that, as was the case with the frequentist concept of significance, the concept of power relies on ASSUMING the truth of a hypothesis, in this case the alternative to the null, and then asking what is the probability of the alternative producing evidence in a certain range of values that would lead to the alternative being accepted. In other words it is the same “backwards” idea of looking at evidence conditional on a hypothesis being true, instead of the more intuitive working from the evidence to the probable truth of an hypothesis. Specifying the alternative hypothesis correctly is also a problem for this approach.

For a description of the Bayesian approach, see the last paragraph to my comment 100 above. As discussed there, this approach allows us to “invert” the logic and make a statement about the probability of the hypothesis GIVEN the evidence.

Some dogmatic frequentists DO believe this to be a radical, and even illegitimate, alternative to the frequentist significance/power paradigm. The charge of illegitimacy gets into some very philosophical, and to me very interesting questions, regarding probability as “degree of belief” versus probability as “relative frequency over time.” The frequentists object to the idea of attaching a probability to a hypothesis. In their view an hypothesis is either true or it is not, and it is only the evidence—the sample mean—that takes on values according to chance. But we speak of probable guilt in criminal cases all the time. And the Bayesian approach gives us, I think, a legitimate way to talk about the probability of hypotheses. Where both Bayesians and frequentists agree, however, is that it is totally wrong to use a p-value to make an inference about the probability of a hypothesis.

I’d like to seize on this as an example of why I think the criticism is wrong, from Bob above:

In other words it is the same “backwards” idea of looking at evidence conditional on a hypothesis being true, instead of the more intuitive working from the evidence to the probable truth of an hypothesis.

We can’t do the latter with observational data; we have to do the former as a surrogate for the latter.

I’m a gamer, I spend a lot of time rolling dice and I hang out with a lot of non-scientists who roll dice. The issue of loaded dice comes up a lot (every die is loaded one way or another so we always seek the die or set of dice that loads in our favour). The mechanism for identifying this loadedness is very simple: assume a hypothesis, roll the dice a bunch, and make a judgment about whether the numbers fit the hypothesis. This is exactly the method of hypothesis testing.

What you can’t do in this instance is design an experiment to work back to your theory. You can do that with, say, the Michelson Morley experiment, because you’re comparing two fundamentally different theories, so in one theory something will happen and in the other it won’t. But often even then, identifying the thing that happened often relies on hypothesis testing (e.g. in Millikan’s oil drop experiment you still need an average charge and its 95% CI).

This idea of careful experimentation to get to a hypothesis is largely a fallacy. What happens in practice is you infer your hypothesis from the data you can construct – which is usually done through a hypothesis test.

Also, isn’t it the case that if you’re doing a test of the dice that I described above, or that has been mentioned above with the coin toss, it can also be seen as a Bayesian experiment with a Dirichlet prior on the probability of the event, p. This has to be the case surely, since in the large sample limit any Bayesian credible interval will converge to the frequentist one. Given the large sample similarity of the two frameworks, there has to surely be some relationship between the testing philosophies in the small sample situation.

Time to read our Wesley Salmon. When we’re acknowledging that 100% is “by chance,” we are using language to miss a point.

Whyever did you stop John? They just get better with age!

Karl 136: A null hypothesis should always be specific (i.e. an equality) because that is what allows us to calculate the sampling distribution for that *specific* hypothesis and determine the likelihood of the observed sample within that distribution.

It is important to understand the concept of a statisticallay testable hypothesis and not mix it up with intuitive notions (which I know is hard). Null hypothesis and alternative hypothesis are not interchangeable. In my experience, formulating correct null hypotheses is among students’ biggest difficulty. Again, in my view statistics instruction is a failure if students don’t really understand these fundamental concepts, even if they somehow manage to do the calculations.

First of all, a wonderful thread.

Second, my 2 cents.
(If I understood things correctly) we are in the following situation: you are, in your life, asked many questions; to each of them you can answer “yes”, “no”, or “I don’t know”.
An answer “I don’t know” is *always* correct; however, you want to answer as many questions as possible with a “yes” or “no”, subject to the condition that you get only 5% of your answers wrong. If you follow the rule “give an answer only when p < 0.05" you will be wrong less than 5% of the times in your life. For instance, if you live in a universe where ALL coins are fair, if you answer "yes, the coin is rigged" to Holbo's problem, you will be wrong: but that will happen only 5% of the times in your life, even in this extreme situation where all coins are fair (or, equivalently, when you KNOW that the coin is fair).

In other words, I think the "p < 0.05" should not be read as a likelihood of being wrong in a particular given case ("is this coins is rigged?", even if we know a priori that the coin is fair!), but as a move in a general strategy on how to minimize errors and still give useful answers.

A different analogy.

Suppose you are playing the following game against the Devil.
The game is in many (ideally, infinitely many) rounds.
In each round, first you choose a strategy for how to answer questions, and tell this strategy to the Devil.
Then the Adversary gives you an urn with a large number of balls, some of them black and some white. You pick 5 balls, look at them, and then guess whether or not at least half the balls in the urns are black; you can either answer “yes” or “I don’t know”.

At the end of the game
1) you must be wrong at most 1/32 of the times (that is, answered “yes” when the correct answer was “no”) ,
and 2) try to guess correctly as many urns as possible (that is, answer “I don’t know” as few many times as possible).
Notice that in this example the Bayesan analysis breaks down: you cannot give any a priori probability to how many black balls are in each urn.

But it still make sense to use the “p <= 1/32" strategy (that is, to answer "yes" when all the 5 balls are black, and "I don't know" in all the other cases).
(The "p <= 1/32" is the only deterministic strategy that satisfies the above conditions: if in a given round e.g. you choose to answer "yes" also when you get 4 out of 5 black balls, the Devil would give you an urn with equal amount of black and whites, and you would guess wrongly too many times; on the other hand, if you decide that in a given round you answer "I don't know" even when you pick 5 black balls, the devil will give you an urn with only black balls, and you will waste a round).

On the other hand, I don't see the justification for the "p < 0.05" as a criterion on which papers to publish and which to reject (that is, the strategy above will ensure that you will answer wrongly at most 1/32 of ALL questions, not of all questions when you say "yes": in fact, the Devil could give you all urns with equal quantities of blacks and white, and 100% of the paper you publish would be wrong).

I hope I didn't make mistakes in thee above analysis…

I’d like to add to the coin toss example a little, though I guess this thread is dead now. If you were to try a Bayesian approach to the coin toss example you would end up using exactly the same logic as a frequentist approach, just with different numbers.

Under a Bayesian approach with some prior you would get an estimate of the posterior distribution of the probability of a head, say p=0.5 with 95% credible interval 0.42 to 0.58. You would then have to make a judgement about whether this is fair. 0.5 seems fair but what about that interval? Is 0.44 fair? What about 0.42? How can you tell? Perhaps you present this interval to a committee of referees before the next world cup and they want to be sure that their coin tosses will be fair?

Fortunately you can check: you calculate hte posterior probability distributions of observing 1 head in a row, two heads in a row, etc. You present this to the committee of refs and they say: Look, the 95% credible interval for the probability of getting 5 heads in a row is below our preferred threshold for an unfair coin. This coin is fair!

And what would the threshold be? Around about 0.05, I bet …

Running out the door, and I am wondering whether there is any connection between Popper’s falsificationism and Ellenberg’s idea that the data have to be inconsistent with the null hypothesis. At any rate, I have always stumbled on Popper’s falsificationism. Can it be related in any way to the discussion we are having? No idea–just asking.

@ RNB Yes, I believe that the p-value approach has been seen as a sort of implementation of Popperian falsificationism. Perhaps Bayesian approaches are a better fit with more realistic versions of that philosophy, e.g. Lakatos.