The great replication crisis

From the article above:

‘Bayesian statistics, alone among these first eight, ought to be able to help with this problem. After all, a good Bayesian should be able to say “Well, I got some impressive results, but my prior for psi is very low, so this raises my belief in psi slightly, but raises my belief that the experiments were confounded a lot.”’

See also: http://www.andrews.edu/~rbailey/Chapter%20two/7217331.pdf

Oh and finally: as a piece de resistance: https://en.wikipedia.org/wiki/Quantum_Bayesianism

From what I can tell, all, or almost all of the problems in the social sciences (and, as the Wikipedia article above implies, the hard sciences) stem from their adoption of ‘frequentist’ approaches. The problem is frequentism: the solution is Bayesianism.

Thoughts?

(PS: see also https://en.wikipedia.org/wiki/Instrumentalism)

(Please note that both of these ideass are either implicit or explicit in the OP so I hope I’m not dragging the thread OT).

E.O Wilson got slammed a few years ago for arguing that there is too much focus on mathematical skills in the sciences (especially life and social science), but I think his point was misunderstood. There seem to be a lot of researchers out there who know enough statistics and maths to carry out a competent study, but without the deep specialist knowledge to identify systematic flaws in the standard techniques.

I think there is a strong argument to be made across the sciences for specialist statisticians to be used both as co-authors and peer reviewers in a whole range of subjects. The old saying ‘a little knowledge is a dangerous thing’ might seem a facile cliche, but sometimes there is truth to it – it might be better for a lot of researchers to simply acknowledge that the use of statistics is a highly specialised subject, and they should defer to the experts, rather than using it as a standard tool in research.

The standard deviation you’re looking for in footnote 3 is 0.35.

However, if you take into account that the result could be statistically significant in the wrong direction, you get a bimodal distribution with sd 2.3

Quigging, your explanation would apply to all sciences, correct?

If you are right, a similar replication project in, say, particle physics would also fail to replicated 2/3 of the experiments published in the top journals.

Manta@8: no, particle physics requires p-values of 0,0000003 to claim discovery, not p=0,05

Ah, thank you Akshay
(however, your links send me to a 404 page. I think the right link is http://blogs.scientificamerican.com/observations/five-sigmawhats-that/)

Let me amend my question then: those scientific disciplines using p=0,05 convention should have a similar replication crisis? (which are those, btw?)

‘No – with Bayesianism one would still have to rigorously nail down ….a plausible range of prior probabilities.’

Well not really. I’m a subjective not an objective Bayesian, so your priors are just your priors. It’s just a guess based on what we all sorta think might be the case. So in the example linked to (@1) we already know, a priori, that telepathy is extremely unlikely, as our prior is set low. The precise numeric value doesn’t really matter so much as long as it reflects that, and we gather enough data.

‘ If your question is “Does this drug kill people” or “Does this drug lengthen survival times” the classical null/alternative approach works just fine.’

No the ‘classical’ approach absolutely does not work fine, especially not in ambiguous scenarios like the second. The reason that frequentist approach is so popular is not because it ‘works’ but because, philosophically, it fits in with the Anglo-Saxon ‘model’ of how science ‘ought to’ work and Bayesianism (especially ‘subjective’ Bayesianism) doesn’t. Same with the Copenhagen Interpretation/Quantum Bayesianism: physicists don’t like it because they don’t like what it implies, not because of what it says, per se.

With regards to p-values, also see

Calibration of p Values for Testing Precise Null Hypotheses.
T Sellke, MJ Bayarri, and JO Berger.
The American Statistician v.55:62-71, 2001.

where a p=0.04 value (say) is shown to correspond to a 26% chance the null hypothesis is correct (in several different senses).

‘Bayesian updating doesn’t really solve the problem for you if your priors are wrong, as pointed out in @3. ‘
???

The concept of a ‘wrong’ prior is incoherent. As I pointed out, from a subjective Bayesian viewpoint, your prior is just your prior. Mine is different from yours, yours is different from your Aunt Sally’s and so it goes. Probability doesn’t ‘exist’ in any objective sense (i.e before we start to ‘test’ it) so, to repeat, the concept of a ‘wrong’ subjective prior makes no sense. Likewise converging on a ‘correct’ probability assessment. Likewise, as an instrumentalist, discussing whether or not the effect is ‘real’ makes no sense (i.e. to me….although this is a slightly different issue. Bayes’ theory clearly relates more clearly to instrumentalism than does frequentism though, equally, it is not quite the same idea).

Incidentally here’s a blast from the past for all you frequentists out there:

http://ist-socrates.berkeley.edu/~maccoun/PP279_Cohen1.pdf

Hidari, trivially, if your prior assigns no probability to the correct hypothesis, no amount of updating can converge on anything like a correct answer. And a prior like “assume all is equally likely” will lead to you asserting that Obama might be a Kenyan-born Muslim. Taking pure Bayesian subjectivism at its word requires you to provide extra logic to justify using probability to inform decisions at all. Better to lose the pure subjectivism and acknowledge that probability calculations take place in a framework of prior knowledge and a context of construction that can just plain be wrong and requires some other justification.

@Manta, @Akshay,

There are more fundamental reason why particle physics shouldn’t have a similar replication problem, or at least not to the same degree.

Reason 1: we publish negative results. For an example, take a look at the public results page for the CMS LHC experiment. Every title there that starts with the words “Search for” is a negative result: we looked for something, we didn’t find it, we set an upper limit on the rate.

Reason 2: we don’t p-hack. Standard best practice in particle physics is to work out the entire analysis procedure before looking at the real data, and to “blind” the data analysis so that the final result isn’t known until after the analysis procedure is fixed. We’re also generally relatively rigorous about or statistical analyses, particularly wrt multiple comparisons

Reason 3: we routinely replicate. Generally, any major result needs to be seen by at least two independent experiments before being accepted (e.g., for the Higgs Boson observation, both the ATLAS and CMS experiments had to see it).

We do have the luxury of being able to make very strong assumptions about statistical independence, and often have very large statistics to work with. Fields that work with biological (or, far worse, human) subjects generally have it a lot harder wrt sample size and statistical independence. Nevertheless, registration of experiments, a less dogmatic approach to p-values, and a generally more sophisticated approach to statistics would help those fields a lot, IMO.

Hidari, I don’t know much about this philosophy you’re presenting, but it seems unlikely to be able to get a drug through a regulatory agency. Is it put to use in such circumstances?

Dan Riley, replication is an impossibility in many social science and medical settings. Consider for example the work of Lee Robins on addiction. Replicating it would be almost impossible given the scale of the project and the need for a war to replicate it. When you can’t control experimental assignments, drop out or even the measurement process, in an urgent and important problem, you just need to work with what you’ve got. Fortunately though the questions you need to answer in such cases are usually simple, and elegant statistics are less important than elegant design to manage the biggest sources of bias and confounding.

Which is why I think the problem John is describing is not one of statistical method.

[My day job involves designing psychology experiments, but take this posting with a pinch of salt anyway…]

I thought replication studies are supposed to use a sample size that is adequate to control the type II errors, and so may often need a larger sample than the original experiment.

That is, in the original experiment you can take the risk of missing a small but real effect because your sample size is too small. [MAJOR CAVEAT: This assumes the only goal of the research is to get a publication out of it. If you actually want to know if a drug has a side-effect of killing people, you might want to be more careful. (“We gave the drug to Bob (N=1) and he seems OK” may not cut it).]

But if you’re going to publish a paper saying. “we tried to replicate this result but we didn’t find it”, you need to guard against the possibility (type II error) that the effect is real, and the original authors were right. So larger sample may be needed.

The probability of Type I error in psychology is zero. The brain is a complex system; everything has an effect on everything else.

@Hidari “Even Plato didn’t think that. (In fact, especially Plato).”

I see that we do agree on something.

@Bruce Wilder: Your infographic misses the best quote of all:

Small wonder that students have trouble [with statistical hypothesis testing]. They may be trying to think.

— W. Edwards Deming

@24 https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing

This is the difficulty I have with most of these discussions. Most people wilfully decide not to know anything about the problems with NHST (and it is wilful…as the Cohen paper I linked to demonstrated, we have known the problems with frequentism since at least the 1930s) and then I get asked bizarre questions with phrases like ‘this philosophy you are representing’ as if I am Thomas Bayes or if I personally invented the concept of instrumentalism and both theories are used solely by me.

If anyone cares, the Bayesian approach to probability (subjective) is sketched out in Nate Silver’s The Signal and the Noise.

http://www.amazon.co.uk/The-Signal-Noise-Science-Prediction/dp/0141975652

The issue is larger than just shortcomings of statistical analysis and replicability. Fortunately, accountants are on the case! Here is a Call for Papers for the 2017 Journal of Accounting Research Conference soliciting proposals for registered reports. Deadline for proposals is November 1; a wide range of methods and topics are encouraged, not just experiments (as with most Registered Reports) and not just accounting.

From the details (pdf) here is information about the Registration-based Editorial Process (REP) they are using:

REP operates as follows. Editors determine whether the initial proposal submission is promising enough to be sent to one or more referees for review, and if so, use the review(s) to determine whether the proposal should be rejected, returned to the authors for revision, or approved. Approval letters spell out the conditions under which they will accept the second stage report for publication. These conditions will always require that authors fulfill their commitments to gather and analyze data as proposed, and never require that the results support any particular conclusion (such as the stated hypotheses). However, editors may also include other conditions to address specifically identifiable concerns about the informativeness of the data or the thoroughness of the additional analyses. To the extent possible, conditions will be crafted to allow authors to guarantee publication simply by living up to commitments under their control. Authors can withdraw an approved proposal at any time, but cannot submit their resulting manuscript to another journal until they do so.

By allowing editors and authors to agree on conditions for publication early on, REP encourages authors to make larger investments in research studies, enhances the reliability of the resulting report, and accelerates the input authors receive so they have it when they need it most (before they gather data):

•REP Encourages Investment. Without the commitments embodied in REP, authors are effectively creating research “on spec”, speculating that the end result will be attractive to an editor. Authors are wise to limit their investment in such circumstances, since they cannot easily predict either editors’ tastes or the end result of the study. The acceptance decision may also be far in the future. REP encourages authors to propose studies that are more ambitious (e.g., in the scope of their data gathering, by deviating from conventional tastes) because they can defer the bulk of their investment until after an editor has committed to publish the end result.
•REP Enhances Reliability. In the traditional editorial process, authors offer their completed research results to editors for evaluation. This leaves authors with both the incentives and the ability to overstate their results by choosing statistical tests that indicate strong support for their predictions, and revising those predictions to make their theory seem more powerful. REP reduces the incentives and ability to pursue such strategies, by making publication of an accepted proposal contingent on whether the author lived up to their commitments to gather data and analyze it appropriately, rather than on the outcome of those results.
•REP Accelerates Input. An editor’s goal is to publish good papers. Editors accomplish this goal partly by making wise decisions about which papers to publish and which to reject, and partly by providing input that helps authors improve their initial submissions. Such input is often very painful in the traditional editorial process, because it comes after the author has made crucial and sometimes irrevocable decisions on how to gather data. REP provides authors with input before they gather data, improving the likelihood that the editors (and authors) can publish good papers.

JQ’s analysis is fundamentally sound but he doesn’t spell out the real lesson we should take from it: to really scientifically confirm a phenomenon, one replication is still not enough (and zero replication is just bizarre – a one time observation can never justify a scientific theory). In the hard sciences, experiments are replicated many times. If replication were a routine procedure in psychology, we wouldn’t be guessing whether the errors are type I, type II are more fundamental experimental blunders. We would have a fairly good idea which results have stood the test of science and which haven’t.

TM how do you propose to replicate an experiment into e.g. the Australian gun buyback scheme? It’s of considerable public policy interest internationally, statistical analysis will require 20 or more years of follow up from the original intervention, and it has happened precisely once. To replicate it you need a counter-factual Australia or a real second intervention in a similar country and another 20 years. There’s a mass shooting every day in America – waiting multiples of 20 years for definitive proof that a given intervention will work isn’t a luxury anyone has.

Sure you could speed up the process of answering this question by a Bayesian analysis with a carefully selected prior (e.g. one that assumes an effect of the intervention). Do you think that an analysis with a pre-selected prior favouring a controversial intervention is going to improve its acceptability to wavering policy-makers in other countries?

40: my understanding is that the psychology replication study concerns lab experiments. There is no reason why there couldn’t have been more much more aggressive attempts at replication. Field studies do of course pose a whole different set of problems.

23: I’m not aware of anybody who professes to believe that “‘probability’ actually exists (in the same way that tables and chairs exists)”.

At a rough guess, half of the observed failures were probably Type I errors in the original study, and half were Type II errors in the replication.

Huh? The original studies were almost certainly affected by lots of publication bias and “p-hacking”, while the replications were more powerful statistically (larger sample sizes) and not subject to publication bias (all were preregistered and published). I don’t see how half of the replication failures could be due to Type II errors.

5 isn’t an objective fact, it’s a measure of reality. If I drop a coin from my hand , the P=1. The P is real, just as a yardstick with numbers is real.

44: good point.

Hidari@34: I read Nate Silver’s book and was disappointed in his description of his “Bayesian” methods. All he seems to be doing is using standard conditional probabilities, (and so does the Wikipedia excerpt on drug screening.) I didn’t find anything remotely controversial to a “frequentist” in either reference, just standard probability theory. What am I missing here?

My (very very poor) understanding is that current state-of-the-art Bayesianism is all about model building rather than Null Hypothesis Testing. Am I wrong on this?

And with respect to the original topic, I think the problem with the psychological studies (and likely a lot of medical studies) is a confusion between investigatory statistics and what one might call confirmatory statistics. Call it the “garden of forking paths” or “data mining” or “p-hacking” or whatever—this is why physicists value replications and why other sciences should figure out how to value them as well.

TM, for the reasons I specified above (sample selection, instrument validity, etc) it’s unlikely that even laboratory studies can be replicated in the same way as for physics.

e.g. “German group replicates Australian group’s findings on sub-atomic particle Y” is very different in implication to “German group replicates Australian group’s findings on the relationship between high school bullying and [insert pointless lab result].”

The kinds of findings in psychology that can be obtained from a lab experiment that is universally repeatable are, I would guess, typically not very interesting.

journal editors were right to pragmatically not waste space on null results.

They have become “null”? Interesting slip.

Something has gone wrong in our psychology that overwhelms our deliberate understanding of what we are doing.

Z, in maths you are supposed NOT to trust the author’s word and check the proof yourself. Of course, in practice many times you cannot do that, and rely on other people’s checking….

There are (important) authors notorious for publishing wrong “proofs” (or “theorems”, for that matter): these you know you have to hande with extra care.

The practice is often not to relegate the technical details to a subsequent publication, but not to publish them at all (as rule-of-thumb, the words “it’s obvious that” or “it’s trivial that” or “by X,”, where is a word for some set of standard tecniques, most of the time mean “I’ve never bothered to actually check the following statement”).

And finally, the standard of proof and what mathematical rigor is has changed a lot with time. (For instance, infinitesimals were allright in Newton’s formulation of calculus, were banished as non rigorous with Cauchy, Dedekind, etc, and have been brought back as a legitimate tools with Robinson…).

In real experiments, I usually more worried about the reproducibility of the sampling, rather than type I/type II errors. Increasing N is usually easier than dealing with sampling bias.

For example, in a typical experiment investigating something about autism, you might recruit 30 children with a diagnosis of autism from a special school, and 30 children without an autism diagnosis from a mainstream school. Carefully matched for age, gender, handedness,… of course: you wouldn’t want your experiment to be confounded by your criterion and control groups being at different developmental ages, or the control group being 50% female while the criterion group has more boys than girls.

But suppose someone wants to replicate your result. They’re going to recruit their sample of participants from different schools, at a different time and place. Lots of things could be different about the replication sample that swamp the original effect. e.g. was being able to get into the special school correlated with family socioeconomic status, in a way that differs between the original and replication sample? Even the diagnosis of “autism” isn’t as stable an entity as one might like for replication purposes. It’s a spectrum condition: do the original and replication samples have it to the same level of severity? Was the level of severity of symptoms needed to get a place in the special schools different between the original and replication samples (due to changes in say, local education authority policy, or health insurance policy, or….) etc.

fn 54, I am aware that people aren’t as replicable as physical particles. But what is the point of doing lab experiments in psychology if you assume from the start that you can’t isolate the phenomenon you are studying? Psychologists are doing lab studies because they want to be taken seriously as scientists. If these supposedly scientific studies aren’t actually replicabe, they need to give up the pretense. The NYT published a truly pathetic apologia for psychology:

“But the failure to replicate is not a cause for alarm; in fact, it is a normal part of how science works.” (http://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html)

The way science works is that if you can’t replicate, you have to assume that your result was a fluke. It is certainly possible that you can come up with a more sphisticated theory or hypothesis that explains both the initial result and the replication failure. But then you have to do a new study to test your new hypothesis. What you can’t do is say that replication doesn’t matter. Or you can, but then you can’t be expected to be taken seriously.

TM @64: “But then you have to do a new study to test your new hypothesis. ” Indeed, and so careers are built. Isn’t that really the motor of the whole machinery?

Great OP. On null hypotheses:

Which of these is a good null hypothesis:
1) Our search for resonances in a particular energy range will just find the usual background events?
or
2) Our results will require throwing out the framework of modern particle physics.

Obviously, (1) is a good null. (2) is an extreme alternate. Right?
Except if the experiment was the recent Higgs search, (1) and (2) were in fact the same hypothesis, logically equivalent statement.

To be continued.

Faustus asks:

“I’m also not convinced by your position that p-values don’t provide “sensible” answers. If your question is “Does this drug kill people” or “Does this drug lengthen survival times” the classical null/alternative approach works just fine, and I think there are very good reasons why alternative methods (such as Bayesian stats) aren’t used in pharmaceutical trials as much as might be expected under this philosophical framework. ”

The classical approach is terrible for questions like that. The answer to “does this drug kill people” is just plan yes. E.g. aspirin via hemorrhagic strokes, even pure sugar pills via diabetes, etc. The question is more like “How many people does it kill when vs. how many it saves when?” Giving special privileges to the hypotheses “it doesn’t kill anybody” or even (in most cases that get to big trials) “it doesn’t save anybody” makes no sense. Doing that nonsensical procedure costs lives.

@manta: So for the 65+ flu vaccine, despite the known efficacy of the vaccine in under 65 groups, the null hypothesis is equivalence to placebo? Or “no better than established medicines” (of which, at the time of the most important studies, there were none?)

Seriously?

By the way, (spoiler alert) the correct answer is:
about 50% effective. Right in between the two implausible nulls.

Further mbw, your example is a good one to show why students are so easily confused by HT. The fact is that the choice of HT always depends on the context and prior knowledge.
There are no clear and specific rules and many text book examples appear contrived.

@TM- I think we’re in agreement. HT is sometimes a useful first screen to decide whether to pay attention to something. As the endpoint of an analysis, it’s nonsense. As the organizing principle for stats education, it’s a disaster.

76: Yep.

@73 Well this (https://www.sciencenews.org/article/odds-are-its-wrong) is an interesting article, in that it clearly states what some of the other posters in this thread are under the impression are the ramblings of a lunatic (at least when I say it).

An extract: ‘Such sad statistical situations suggest that the marriage of science and math may be desperately in need of counseling. Perhaps it could be provided by the Rev. Thomas Bayes.

Most critics of standard statistics advocate the Bayesian approach to statistical reasoning, a methodology that derives from a theorem credited to Bayes, an 18th century English clergyman. His approach uses similar math, but requires the added twist of a “prior probability” — in essence, an informed guess about the expected probability of something in advance of the study. Often this prior probability is more than a mere guess — it could be based, for instance, on previous studies.

Bayesian math seems baffling at first, even to many scientists, but it basically just reflects the need to include previous knowledge when drawing conclusions from new observations. To infer the odds that a barking dog is hungry, for instance, it is not enough to know how often the dog barks when well-fed. You also need to know how often it eats — in order to calculate the prior probability of being hungry. Bayesian math combines a prior probability with observed data to produce an estimate of the likelihood of the hunger hypothesis. “A scientific hypothesis cannot be properly assessed solely by reference to the observational data,” but only by viewing the data in light of prior belief in the hypothesis, wrote George Diamond and Sanjay Kaul of UCLA’s School of Medicine in 2004 in the Journal of the American College of Cardiology. “Bayes’ theorem is … a logically consistent, mathematically valid, and intuitive way to draw inferences about the hypothesis.” (See Box 4)

With the increasing availability of computer power to perform its complex calculations, the Bayesian approach has become more widely applied in medicine and other fields in recent years. In many real-life contexts, Bayesian methods do produce the best answers to important questions. In medical diagnoses, for instance, the likelihood that a test for a disease is correct depends on the prevalence of the disease in the population, a factor that Bayesian math would take into account.

But Bayesian methods introduce a confusion into the actual meaning of the mathematical concept of “probability” in the real world. Standard or “frequentist” statistics treat probabilities as objective realities; Bayesians treat probabilities as “degrees of belief” based in part on a personal assessment or subjective decision about what to include in the calculation. That’s a tough placebo to swallow for scientists wedded to the “objective” ideal of standard statistics. “Subjective prior beliefs are anathema to the frequentist, who relies instead on a series of ad hoc algorithms that maintain the facade of scientific objectivity,” Diamond and Kaul wrote..” ‘

@Hidari- As you can see, my approaches to most problems are Bayesian. It’s not wise to get too dogmatic about that approach, however. E.g. what do you make of the rigorous argument by Wasserman and Robins that for an important class of real-world high-dimensional problems, no Bayesian estimator converges with useful rapidity, while there is a simple frequentist estimator that works quite well.

@Dogen. OK, let’s do the standard example:
old HIV test, 95% accurate either way, population (say all male residents of Toledo between 20 and 30 yr old) with 1% prevalence. etc. The usual answer that you know, about 1/6 chance that a positive screening result on someone from that group means actual HIV.

Now your screening accidentally picks up a guy 31 yr old. What would a positive result on him mean? (formally, not enough info to say anything?)

Or you notice needle marks on the arm of one guy. What do you think the his positive result means? (too much info?)

Or You start testing in Akron but don’t have prior population results there.
etc.

This is the real world of decision-making.

Nice data. It strongly indicates that the drug makes a big reduction in HIV transmission- maybe by more than a factor of 10.
It’s easy to deal with sloppy inputs like this from a Bayesian viewpoint. I don’t know what a rigid frequentist would do with it, but perhaps one will write in.

@ Dogen #87. Right. Straight conditional probabilities (as in the standard intro screening exercise) don’t require Bayes. Real-world use usually does.

In my opinion there are a few misconceptions here. To address one or two …

mbw at 70, aspirin kills lots of people so classical stats failed: we know that aspirin kills through hemorhage precisely because of classical tests that showed higher rates of incidence. Nothing secret there: p-values proved your point.

mbw at 69, flu shots and hypothesis tests vs. CIs: there’s no reason not to do a hypothesis test and report a CI as well (in fact most journals require it). In the case of flu, there’s good reason to think the vaccination may be less effective than in other age groups but because the risk of flu is higher in the elderly we need to adjust our vaccination policy to account for effectiveness. Test against the 2/3 null, reject if necessary, then report the actual effectiveness with CI that can then be used in assessing policy.

Bruce Wilder re pre-exposure prophylaxis: I don’t see the big deal here. Prep has been well understood for years, the question is whether it is effective after adjusting for risk compensation and whether it is cost-effective on top of test-and-treat strategies. Risk compensation can’t be replicated easily in labs and cost-effectiveness requires good data. If Kaiser have added to this information that’s great but the primary issue here isn’t Bayesian vs. frequentist false dichotomies. As I keep saying it’s about experimental design.

Lots of people re: 95% CIs: CIs and t-tests are formally equivalent, and using one instead of the other is not philosophically different. In the case of a Bayesian credible interval, yes you have incorporated a prior but you are still doing philosophically the same thing as you do when you do a standard CI comparison. In fact if you look at how Bayesian stats are generally used to answer practical problems, what they do is report a credible interval and say “since these values are outside the interval, we conclude that …” They’re doing the same thing with a different numerical result.

This is easily seen in some of the basic examples. If you use a simple non-informative prior for a simple estimate of a mean and its credible interval, for example, you’ll usually get a formula for the 95% CI that is very similar to the classical formula but with an extra bit, n-weighted. This bit declines to zero with large n, indicating that the two CIs converge in the large sample limit (as they must). This convergence raises problems for the classical contrast between “frequentist” and “Bayesian” CIs. Gelman elegantly explains the contrast and why Bayesian CIs are more intuitive, but to the best of my knowledge he doesn’t have a good explanation for the implication from the large sample limit that the two types of CI are essentially the same. This might also be why everyone in practice treats “frequentist” 95% CIs as if they were Bayesian (i.e. interpreting values near the middle of the CI as more likely in some sense than those near the fringe).

The problem I see with Bayesian stats as it is used in practice is that it is usually presented in almost exactly the same way as Frequentist stats – i.e. a credible interval is used to draw conclusions about what it means to be outside the interval. I think this is because scientists don’t have many other ways of coming to conclusions but to accept/reject (hence my dispute with John’s presentation of classical hypothesis testing). It’s a natural way of thinking: “since we gave people this drug their strokes stopped happening, the drop is way too big to be just luck, something’s working.”

So two requests for the “Bayesians” commenting here:

1. please stop using “frequentist” as an insult and making false dichotomies (e.g. mbw’s throwaway “strict frequentist” comment): this isn’t an R message board, there’s no need to be an arsehole;

2. show how you would approach the question of whether a drug is effective in a way that is fundamentally different to a “frequentist” approach. i.e. don’t tell me that your CI would be slightly different, but tell me how you would use that CI in a way that is fundamentally different to a classical drug trial.

I think you can’t!

Does it really need to be spelt out that Bayes theorem is a standard result of “classical” (sensu 88 for want of a better term) probability theory and there is nothing the least bit inconsistent between Bayes theorem and frequentism? Formally you can easily express it in frequentist terms. In fact, in the examples about HIV prevalence for example, what you are talking about ARE frequencies (there is a definite number of HIV positive people within a certain population – that is a frequency; and so on). The suggestion that frequentism “overlooks” conditional probabilities is pure nonsense as Dogen and christian are correct to point out. The real issue as far as I can see is in statements such as the one quoted in last paragraph of 80, which most empirical scientists won’t swallow.

Ans since Nate Silver has been mentioned several times, remember this much debunked example of “Bayesian analysis” in action from fivethirtyeight:
http://fivethirtyeight.com/features/a-formula-for-decoding-health-news/
(It seems that the comments were removed but see https://ksj.mit.edu/tracker/2014/03/nate-silvers-new-fivethirtyeight-dishes/ or http://www.cjr.org/the_observatory/fivethirtyeights_disappointing.php).

@96
John
you are of course absolutely correct. However I think the problem is not that faustnotes, TM etc know too little. It’s probably because they know too much. They (presumably) are experts in statistics/probability theory etc. and they assume that everyone else is too.* But the OP was about statistical testing and the replicability problem in psychology . I have a great deal of knowledge of what psychologists are taught in their ‘statistics course’ (at undergraduate level) and I can absolutely assure everyone right now that Bayes theory is not taught at any point (to the best of my knowledge, this is true of essentially every ‘Western’ university, although of course I could be wrong about this).

If anyone cares I have been taught a course in statistical theory at post-graduate level at a Russell Group university and Bayes theory was not mentioned once, nor was it alluded to.

*In other words, when they talk about ‘statistics courses’ they are talking about someone studying applied statistics as a major, at undegraduate level, where what they are saying, I am sure, is true. They don’t know what is taught on a ‘statistics course’ as a module or whatever on a psychology or sociology undergraduate degree.

@faustus #92, adam.smith #95. Yes of course every frequentist uses Bayes thm. Instead of rapidly typing “use Bayes” I should have spelled out “use Bayesian priors”. BTW, if you’ll check #82 you’ll see that I’m not exactly a hardcore Bayes-only type.

Now to the specifics re #92:

My point was not that with very large N you can’t exclude false hypotheses, e.g. that aspirin is 100% safe, by any standard statistical technique. It’s that the standard prior (zero effect) is generally known to be false ahead of time. So going through the exercise of rejecting it (with huge N) or failing to reject it (with smaller N) simply doesn’t correspond to the actual problem at hand. This is not exactly a new point, it’s what Deming and many others went on about.

You say test against the 2/3 effective null. Ok, but in all the actual papers the 0% effective null was used. Until there was an extremely clever natural-experiment paper with N of about 3,000,000, nobody was in position to reject any nulls. So you would have gotten the opposite answer to the standard literature because you picked the opposite null.
Look, to pick an old cliche, my best friend is (literally) a frequentist. But when I write of rigid frequentists I mean those who feel compelled to pick null, do HT, give p-values, even when that’s totally inappropriate for the problem at hand.

You want an example of how things would be done differently, not just at the edges. Ok, real world example. Say that a company has a vax for (IIRC) meningitis. In a big phase 3 trial on kids. Working well, as it did in small trials. Do you interrupt the trial early to start giving the vax widely and save dozens of lives. Or do you wait til the trial is over, as (no surprise) a rival company urges the FDA. Your BIL is called in as an expert to help decide. He tells you about subtleties of the statistics of trial interruption. You ask “But what if we took some reasonable prior distribution on the effectiveness, rather than pretending we thought it was probably zero?” He does a double take, says “Oh, you mean does it really work? Yes, we know it works. But the FDA uses p-values.” And no, I’m not a fiction writer.

100: “when I write of rigid frequentists I mean those who feel compelled to pick null, do HT, give p-values, even when that’s totally inappropriate for the problem at hand.”

If your definition of “frequentist” is “somebody who does inappropriate statistics”, I guess you have proven your point. Or seriously, let’s examine the logic here:
– Frequentism is the prevalent statistical paradigm.
– There are many mistakes made in statistical analyses.
– Therefore, frequentism is to blame. QED

@TM Good point about my inappropriate use of the phrase “rigid frequentists” to describe those who try to apply an HT algorithm where it doesn’t belong. My point is not that intelligent frequentists would do that, just that current stats education leads many people to do that.
The vax point is that if you’ve got one that worked in some animal models, worked on one age group, and seemed to work (big uncertainty) in small trials on the group in question, does it really make sense to pretend that the point of the big trial (I may not remember names precisely) is to test the hypothesis that it doesn’t work at all? Setting something like a CI employing both the new data and the modestly informative prior data is what you’re trying to do. And remember, behind those dry words is the question of many months of real-world use or non-use, with dozens of deaths and brain damages at stake. I do understand that those may be costs that need to be paid to maintain confidence that only reliable treatments are being used. And of course in many cases reasonable Bayesian priors would lead to requiring stronger rather than weaker new evidence (compared to HT) before accepting a new treatment.

The questions about how to teach and how to convey information to the public etc. are very important ones. I’m not confident about the answers. The current system seems ineffective, with some very reproducible failure patterns. Maybe that’s partly because it’s sort of jerry-rigged together out of tools that were especially useful 100 years ago. Whether a straight Bayesian approach would work, I don’t know, but I believe that Downey at Olin College may be trying something along those lines. It would be interesting to hear how it’s going.

mbw, your examples all seem to boil down to this contrast.

– frequentist would assume a likelihood, calculate a confidence interval, compare to a threshold, and reject/retain
– bayesian would assume a likelihood, assume a prior, calculate a credible interval, compare to a threshold, and reject/retain

Either method can tweak CI through likelihood; either can choose the wrong threshold. But they have fundamentally the same decision making process. I asked you to explain to me how they are different decision making processes, not to quibble about how to calculate the CI.

Your final example seems to indicate disappointment that the FDA doesn’t allow drug companies to choose a prior. Can you see any reasons for why it might be a bad idea to let a drug company, with millions riding on marketing a product that will cost the health service millions, choose a prior that assumes the drug is effective? Is it possible that the FDA prefers p-values because they want to force all drug companies to work from a null position?

Note also that Bayesian stats doesn’t solve the power problem at the heart of John’s post. You can calculate a probability of an event or outcome on the basis of a bayesian CI, but we know that if you increase n then you get a smaller probability of certain events (usually the rejection events; see above). So you haven’t solved the problem of small n leading to failed tests. Sure your prior might change the n threshold, but when you don’t know the truth the n threshold is unknown anyway – how much have you gained?

And none of this gets out of the fact that I have repeatedly pointed out here: replicability and poor results in psychology are much more determined by study design, measurement method, sample selection and experimental process than the final statistical method. Something John didn’t talk about in the OP, and something that destroys your stats no matter what dichotomy you choose to throw at them.

@106 No, I don’t want drug companies to pick the priors. Duh. In the example I cited, the two drug companies would have picked wildly different priors. Now that you raise the question, I’d want the FDA panel to pick the prior.
On the flip side of the same question, currently polluters are allowed to pick the prior (zero harm) under the HT system. So if you’ve got another chlorinated hydrocarbon, just like the last strong carcinogen only with a methyl group tacked on somewhere, until somebody has p<0.05 (the sacred number) on your particular compound, then it hasn't been "shown scientifically to be harmful". Etc. Cuts both ways.

@106 On decision making process. For flu vax for elderly, there were two different standard nulls. One was the one used in the literature, the other was assumed obvious in some of the comments above. Frequentists following the procedure you outline would make opposite decisions depending on the choice of null. Bayesians with a smooth prior wouldn’t have to make that arbitrary choice and could have come to the correct decision with less tsuris.

Gelman actually just posted on this, particularly the role of priors: http://andrewgelman.com/2015/09/04/p-values-and-statistical-practice-2/
good stuff. I think his recurring theme that testing against a zero-effect null in the social sciences is misleading and that a Bayesian approach is more likely to produce relevant/interesting results is quite good. It also shows, though, that incompetent/schematic Bayesian approaches would just as easily lead to bad statistics.

Apart from the “bad design” point that faustusnote makes, I’m not convinced that Bayesian approaches are really more robust to application by people who don’t really understand them. It’s just that that’s currently less common, because people with only basic stats training all do frequentist stuff.

mbw @ 110: how would frequentists come to a decision? Against what criterion? How is it different. You consistently elide the actual decision making process in Bayesian stats. You suggest they are calculating a different number, fine, but how are they using that different number.

Also, lots of statistical packages either have only recently introduced Bayesian stats (Stata), need languages that are not at all common in the standard practical fields (python in SPSS) or are inaccessible (e.g. JAGS in R) to all but the most advanced users. Computational complexity remains a problem. I’m running problems in BUGS that take weeks (don’t even get me started on BUGS!) Anyone who thinks computational complexity is not a problem is ignoring the basic needs of basic users, and taking an elitist position.

110: The point of your flu vax example somewhat eludes me, probably because I am missing context. What decision was actually at stake and how is it that the choice of null would cause opposite decisions? If they tested the zero null, they would conclude that the vaccine is effective in elderly people. If they tested the 66% null, they would conclude that the vaccine is less effective in elderly people than in the general population. Both of these conclusions are correct and there is no contradiction. What am I missing?

An egregious example of misused hypothesis testing is the media claim “There has been no global warming since 1995” because the warming trend wasn’t statistically significant. The overall trend has been robust and whether or not an arbitrarily selected time period was too short to produce a significant result was irrelevant.

This is a case where the choice of the null hypothesis does indeed matter and has been used to manipulate the media. The abuse would be obvious to anybody who understands basic statistics and is best explained by showing the confidence intervals.

See https://tamino.wordpress.com/2014/12/04/a-pause-or-not-a-pause-that-is-the-question/

@faustus 113. Sorry, I’m being too compressed. The Bayesian method I describe is absolutely not what you took it to be. The integral is of utility (either positive or negative) times probability density. That’s after the step of just multiplying priors by likelihood, not another version of applying a cutoff. The subsequent “cutoff” is then trivial: is net expected utility positive or negative?

So these really are very different approaches, so much so that one of the Bayesian steps is unfamiliar to you.

This is a great thread. I don’t know enough to make a meaningful contribution, but I know _just about enough_ to have learned a lot. My favorite kind of CT thread, pretty much. Thanks!

mbw @121: I think we’re talking at cross purposes from different fields. In health, the presentation of a difference in outcomes between two groups, and the conclusion it was not by chance, is the information that the policy-maker is looking for (since we don’t use utility calculations). so the two policy questions here (to flu vax or not flu vax; to increase investment in indigenous health or not) are both being answered in the same way.

in social epidemiology (such as is being deployed in the Indigenous example) the “treatment” is going to be very challenging to evaluate and will largely be a political decision. If we do evaluate it, this job will be undermined by the experimental design and sampling process long before we get to the stats.

I think my example has been undermined by the different policy processes at work in our fields, maybe.

@ 124 I’m surprised you say ” In health, the presentation of a difference in outcomes between two groups, and the conclusion it was not by chance, is the information that the policy-maker is looking for (since we don’t use utility calculations)” . Think of the voluminous recent arguments about mammography and PSA screening. It’s partly about expectation values of treatment effects and largely about assigning utility to various outcomes (death, loss of a breast, incontinence,…). This side of the question appears in almost every discussion. I’m very close to people in the medical stats field and they describe utility evaluations (including simple $ cost) as central to many decisions.

BTW, you made some comment on my “field”, but I don’t know what that was presumed to be. I’m a retired physicist, whose only claim to special credibility in this discussion is that I’ve never had a stats course.

@mbw –

@adam.smith #112. Unfortunately, in that particular line of argument Gelman has invented something he calls a “p-value” that doesn’t have the basic definitional property of p-values.

well, that’s the whole point of the article (by, in case that wasn’t clear, one of the world’s leading applied Bayesians): the definitional properties of p values make them useless to how people think. When is (or isn’t) interpreting p values as something different (i.e. one-side p values as approximations of directional posterior probabilities as Greenberg&Poole suggest) useful. But it does also have the implication that running Bayesian analysis mindlessly with non-informative priors

If you don’t know the population is normal, you’ve gotta use some other distribution.

no, that’s too strong. The t-statistic refers to the sampling distribution of the test statistic, not the population distribution, and because of CLT, T-tests are perfectly fine for random samples from any number of distributions.

“How is this process in any meaningful philosophical way different to the frequentist framework that John is rejecting? In health we don’t use utilities, we just compare directly, but so what?”

Well it’s basically just a massively disingenuous cheat isn’t it? With a Frequentist result you can say “most published research results are wrong“, *if and only if* you’re a Frequentist, because then you have a concept of Type I and Type II errors and a concept of coverage. And you actually think things can be wrong.

I don’t know how a Bayesian can possibly appropriate that criticism if they believe you only think in terms of degrees of belief. The equivalent honest Bayesian objection is basically just the completely uncontroversial and pointless “different people have different beliefs, but you can’t be wrong, after all it’s all subjective anyway”. Well, yeah, thanks for that. If they can show the CI can’t be generated from some prior they can add, “your beliefs are irrational”, but Frequentists will still have actual coverage.

@BW 133 Agreed.

@AF 132 Say that those psychology papers had published posterior pdf’s rather than p-values. A Bayesian can look at the same collection of pdf’s, transform them to variables in which they’re uniform. Now look at how the replication results fit on that distribution. Whoops, they’re way clustered over toward the left. You’ve shown there’s systematic bias of some sort. Up to a point, that could be mainly publication bias (especially for results just passing the arbitrary significance cutoff). The reported tendency of surprising results to reproduce less well is just what any Bayesian would expect. When the replication result is really way off in the left-hand side, you say in that individual case there was probably a serious goof. Either the original study overestimated the effect or the replication underestimated it to a surprising degree.

The authors of the Science paper were careful not to get bent out of shape by results crossing from p=0.048 to p=0.052, but sloppy type I/type II thinking does lead to just that sort of confusion.

@135 Yes, that’s what he writes. I hope you’ll forgive for saying it reminds me of:

“I say orgies, not because it’s the common term, because it ain’t—obsequies bein’ the common term—but because orgies is the right term.
Obsequies ain’t used in England no more now—it’s gone out. We say orgies now in England. Orgies is better, because it means the thing you’re after more exact. It’s a word that’s made up out’n the Greek orgo, outside, open, abroad; and the Hebrew jeesum, to plant, cover up; hence inter. So, you see, funeral orgies is an open er public funeral.”

Also I think Abbe Faria at 132 makes a good point. Defenders of the value of Bayesian stats to policy-makers and clinicians need to give better explanations of how to justify and defend priors.

@137 I’m out of words. Maybe best to swipe some from Brecht:

The aim of science is not to open the door to infinite wisdom, but to set a limit to infinite error.

John 126: The natural question for Bayesians is “What choice should decisionmaker Y make in this situation”.

Question: how does that apply to your original topic, the psychological experiments? I thought these psychologists were trying to increase our scientific understanding of human behavior, not making or proposing decisions.

Whatever the purpose of statistical analysis, cost benefit analysis and policy decision making are outside of its domain (this also in response to mbw). Decision making is based on many inputs, including but certainly not restricted to scientific data and statistical analysis. I strongly disagree with conflating these domains.

I’m also very uncomfortable with incorporating “subjective beliefs and preferences” into statistical procedures. That is precisely the problem with Jeff Leek in 94 and Nic Lewis in 125: statistical formalism is just camouflage for a priori subjective beliefs.

Also, a quibble but an important one, the outcome of a classical hypothesis test is never never never that “hypothesis X is accepted as true”. It’s always rejection or failure to reject H0. This is an inexcusable (though pervasive) mistake and it really matters.

@TM 143 You’ve put your finger on the key question. The standard procedure puts those value judgements into setting the p-value thresholds, all mixed in with the statistical analysis. My point is that because Bayesian analysis doesn’t insist on compressing the result prematurely but outputs a full pdf, one can keep the values and decision making process entirely out of the statistical argument. Then you can come along afterward with a separate alleged utility function, and argue over that without then having to go back and change cutoffs etc. in the statistical part of the argument.
So there are two separate Bayesian pluses.
1. Priors come in explicitly and in a form more representative of our actual knowledge, rather than in a canned special-null form suited only to a few problems. (And there’s no dopey sacred number custom, although in principle frequentists can’t be blamed for that.)
2. Values come in explicitly and entirely separate from statistical analysis when you use the statistical analysis to make decisions. And you use the whole analysis, including relatively unlikely major events.
Thus the relatively objective likelihood analysis stands cleanly in the middle of the process, uncontaminated by values and by the more arguable aspects of the priors.

Some are better guessers, and you begrudge them this gift? Oh, bob . . . why so resentful? ;-)

“While the marks have been refining their “systems” for prediction, the reality-based community have always known that the two-headed coin, marked cards, stacked deck and especially owning and running the “house” have always constituted the bestest priors.” mcmanus @145

I don’t know, seems to me “the reality-based community” has always and only been playing the most comprehensive game of make-believe. They just have the best tools (or is it toys?) for the game.

@Ram #147. First, a minor point: it’s true that if the exact null is true then 95% of the time it will not be rejected in a study with those power conventions. However, the exact null forms a set of measure zero in the parameter space, and in most cases there’s no special reason to think it’s exactly true. So then for effects that are too small to be important you don’t actually know what percent will produce “positives”.

More importantly, what you say about replicating positives is just plain mathematically false. For effect sizes that would give (in a measurement that happens to be right on the button) p just less than 0.05, the probability that a replication will again give a positive result is just barely over 50%. When you say that the study has 80% power to detect a positive effect of some size, that size is picked based on some utility criterion. That means that an effect meeting your criterion must by ~3 sigma away from the null to have 80% chance of coming out positive with those cutoffs. So if you actually see a result at say p=0.04 what’s the probability that a replication will also have p<0.05?
There's no frequentist answer to that. A Bayesian can give an answer, but of dubious reliability since it depends on the assumed prior distribution. Another Bayesian might give a different answer.

In my opinion confusion about the meaning of p-values is the norm, not the exception.

That’s an unreasonable quibble, John, since the analysis was conducted fraudulently. An interesting point though, since in order to bend the frequentist method to match their priors and show something that everyone knew was not in the data, they had to commit fraud. Where’s the problem with the frequentist framework there?

Whereas a Bayesian could just prove it was true by selecting a prior.

When you reject H0 in favour of H1 you are not accepting H1 as true.

mbw @ 150,

You’ve misinterpreted my comment. Assuming H0, if the original study failed to reject H0, then the study should replicate 95% of the time. Assuming H1, and the effect size the study was powered to detect, if the original study rejected H0, then the study should replicate 80% of the time.

On the other hand, assuming H0, if the study rejected H0, then it will replicate 5% of the time. Assuming H1, and the effect size the study was powered to detect, if the study failed to reject H0, then it will replicate 20% of the time.

What, then, is the overall replication rate? Assuming H0, it is .95 * .95 + .05 * .05= .905, and assuming H1, and the effect size the study was powered to detect, it is .80 * .80 + .20 * .20 = 0.68. If the actual effect size is bigger than what the study was powered to detect, the replication rate is even higher. If it is a lot smaller, we can end up with a low replication rate. Which is just to say that underpowered studies do not replicate particularly well. Which was the point of John’s post. My point is simply that this is why, if you decide to do a hypothesis test, it is important to control both error rates, since these are the principal determinants of replicability. That many people do underpowered studies, or studies with (implicit) multiple comparisons problems, does not invalidate the testing procedures we teach, though it may point to some failures in how we teach it.

I’m an applied statistician by the way, and I use Bayesian methods to solve all sorts of problems. But people tend to conflate classical statistics with frequentist statistics, and ordinary practice with frequentist recommendations. Frequentism doesn’t have any methods, it just provides a way of evaluating the performance of data-dependent procedures under replication. In statistics we basically always want to use methods with good frequentist properties, but those methods can be anything at all. We might arrive at them using a Bayesian approach, a classical approach, some machine learning method, or whatever. But the justification for using THAT procedure is that it, e.g., controls type I/II error, provides adequate coverage, is admissible, minimizes (say) MSE, etc.

mbw 144: I think many of us are skeptical of allowing subjective priors. You say that priors are not related to value judgments, I’m not so sure. How can you not be tempted to choose a prior that will give you the preferred result?

“My point is that because Bayesian analysis doesn’t insist on compressing the result prematurely” – we have already agreed that compressing a study to a one-bit HT is poor practice. Nothing prevents us from handing the decision-maker a CI. They can then apply their cost-benefit parameters to the CI and use that to make the decision. We can even calculate several CIs – 90%, 95%, 99%. The decision-maker must understand statistics well enough to be able to interpret these results.

One concern that I would have about the Bayesian alternative: how hard is it to explain your results to a decision-maker?

@Ram #154. The first part of this sounds correct. But it’s very different from what you wrote the previous time. Yes, if the specific H1 is correct then with these cutoffs it will give “positive” results 80% of the time. But before you wrote that p<0.05 results will replicate a positives 80% of the time, which isn't even close to being the same thing.
BTW, how likely is H1 to be right? Generally, as a set of measure zero picked not by any prior belief about probability but by some utility criterion of importance, the probability that H1 is true is zero. This detracts somewhat from the importance of calculating the rate of nominal positives if it were true.
Then to calculate the total replication rate on the assumption that either H0 or H1 must be true (an event of probability zero) is odd.

mbw @ 158,

I apologize if my initial comment was imprecise. This is probably not the ideal forum to reevaluate the foundations of hypothesis testing theory. Suffice it to say, the frequentist interpretation of probability, frequentist decision theory, and its applications to point estimation, interval estimation, hypothesis testing, and numerous other inference problems, are philosophically coherent and mathematically exact. Online criticism of p-values and the like usually exhibit no awareness of this, suggesting that the entire framework is fundamentally flawed. It is not. Any data-dependent procedure that has good frequentist properties, which are well-defined properties that can (in principle) be empirically demonstrated through repeated experiment or simulation, is a frequentist procedure. Which frequentist properties we want a procedure to have depends on the problem we are trying to solve. Translating every study into a hypothesis test in order to generate a p-value is a problem with current publication practices, but it is not a problem with hypothesis testing. Hypothesis testing. Hypothesis testing, when performed correctly in appropriate applications, delivers exactly what is promised: a conclusion which will be correct the vast majority of the time. As do other procedures.

Any method can be used to generate procedures with good frequentist properties. It turns out that Bayesian methods, suitably applied, do this particularly often, which makes them useful for solving many different kind of problems. Still, if one produces a 95% posterior interval that has 2% coverage, I think one has produced something that is pretty useless. It is not hard to generate examples like this. Good statistical procedures have good statistical properties, and good statistical properties are good frequentist properties. Good Bayesian properties often but not always generate good frequentist properties, which makes them useful, but good Bayesian properties alone are not enough.

mbw @ 161

I took AP statistics in high school, which was taught from a classical point of view. It made no sense, appearing to be a hodge podge of unrelated procedures for tackling unrelated problems, leading me to pursue other subjects. It was only when I learned of the Bayesian perspective (in a philosophy of science class!) that I became interested in statistics once more. The logic of Bayes is remarkably elegant: give me a probability model, and a loss function, and I will give you a decision optimally informed by the data. The problem is that Bayes itself gives no guidance about where the probability model should come from. This is often stated in terms of a problem with the prior, but the likelihood has the same problem–a Bayesian likelihood does not mean the same thing as a frequentist likelihood, even if they look the same and follow the same mathematical rules. P(y | theta) cannot be checked against the data, but P(y; theta) can be.

Frequentism’s vice is that, while it gives us a conceptual framework for evaluating data-dependent procedures, it does not tell us how to construct good procedures. This is why classical statistics seems like a hodge podge. In my view, frequentism and Bayesianism are two great tastes that taste great together. Give me a problem, and I can formulate a reasonable probability model capturing it. Using Bayes, I can update it with the data and arrive at a decision. And using frequentism, I can run a simulation to verify that my decision has the right sorts of properties. Bayesian means, frequentist ends.

161: “I’m impressed, however, with how consistently the standard presentation of stats leads to deep confusion on the part of most ordinary practitioners.”

Let me offer a different perspective. The biggest problem for students of statistics is that most yearn for certainty and clear and easy answers. They just want cookbook recipes that will give them some answer. Maybe they have grown up with a kind of techno-optimism that doesn’t leave much space for ambivalence and uncertainty. They haven’t learned to appreciate science as an error-prone human endeavor. Maybe they are confused because reality is confusing.

But maybe Bayesians are less confused than frequentists. Somebody should conduct a hypothesis test ;-)

I took AP statistics in high school, which was taught from a classical point of view. It made no sense, appearing to be a hodge podge of unrelated procedures for tackling unrelated problems.

Barring one exceptional class with a great teacher that I had in the sixth grade, this description eerily resembles all the mathematics instruction I had from first to twelfth grades, including a couple of AP classes.

I was expected to learn procedures for problems, apply those procedures and only those procedures to arrive at an answer, and show my work to prove that I was arriving at the right answer the right way. As opposed to getting the wrong answer, or the right answer the wrong way. By high school I had finally learned not to arrive at correct answers my own way, no matter how much easier it was to remember, because I would always score lower. They had taught me mathematical pattern-matching instead of thinking.

This was in the USA. Is it better in other countries? Was this an unusually bad series of school experiences for the USA?

166: That is a depressing picture. My Math experience (outside the US) is totally different. Math was the subject where you never had to do any rote learning because everything could be deduced by reason (really now!).

167, I remain unconvinced. Really, hodgepodge of unrelated procedures? A typical intro statistics class might cover the following hypothesis tests:
– Small sample HT for a population mean
– Large sample HT for population mean
– HT for population proportion
– HT for sample mean for paired observations
– Comparing two population means
– Comparing two population proportions

OMG, already a hodge-podge of six “unrelated procedures”! But of course they aren’t unrelated at all, they are all variations on a single idea. Successful statistics instruction would stress the remarkably simple, coherent theory behind these procedures. Also, as mentioned somewhere above, CI and HT should be taught as fundamentally related concepts rather than in isolation. Students who don’t understand this will probably never understand and never be able to correctly interpret hypothesis testing.

Unfortunately, many students will never understand the unifying principles behind all these procedures, whether due to poor statistics instruction (rushing students to practical applications before they have understood the theory), or because students have been spoiled by years of failed math instructions as described by Matt, or frankly because students couldn’t care less. To me that is a tragedy. But I find it hard to believe that teaching from a Bayesian perspective will “make statistics appear more integrated and compelling”. One reason for doubting this is the experience (already mentioned a few times) that students find Bayes (and even simple conditional probability) damn hard. I can see how the best, most motivated students may benefit from more immersion in Bayesianism but I don’t see it for the mass of students who can’t or don’t want to go beyond the cookbook recipe approach to statistics.

@ all the recent comments: (this is a preliminary response while a fuller one with links is hung up in moderation, as is another one that mentioned an apparently dangerous word))

Frequentist statistics can be well-taught to rather ordinary college students. My wife took a Freedman-based course and built it up to the point where it’s chosen by over 3000 students per year. They mostly get over the addiction to formulas and get hooked on reason. It’s now taught very successfully by a new teacher, so it’s not just a personal anomaly.

People can look over some materials with a bit of googling.
It doesn’t look fancy. Baby stuff, right? Students coming out of this course typically can spot the big confounder in an observational study claiming that school drug testing has no effect on drug use, etc. Compare with typical formula-driven students.

Theft of the materials is encouraged.

Ok, that little note got through. Here’s some tips from what I’ve seen work.
Use box models.
Discuss confounders and casuation.
Don’t allow formula sheets.
Use lots of data from anonymous surveys of the class.
Make a lot of the survey questions about sex.
Don’t dwell on picky points in the first semester.
Get great undergrads to do videos on the hard points.
Make randomized automated homework so students can’t pass answers.
Most importantly, fill the exams with questions that can’t be answered without getting the real concepts.
Institute intense anti-cheating techniques on exams.
etc.

But that last note is not the one that has useful url’s in it, because anything like that seems to be hung up.