Annals of Interesting Peer Review Decisions

I once had an article on a sex work-related topic sent off for review to a known anti-sex work campaigner. Their review recommended rejection because we didn’t write the article from an anti-sex work perspective. Data, analysis, public health theory, all irrelevant. We used the term “sex worker” instead of “prostituted woman” so …

On the other hand, my partner once was tasked with setting up a peer reviewed journal in a very broad field, and she had to find peer reviewers from scratch for a new journal with a very wide remit. Inevitably, if she didn’t get very clear recommendations from the editorial board (most of whom were asleep) or the authors themselves, there was a risk she would select exactly the wrong reviewer. She was clever enough to avoid this mistake, but I think in journals with broad remits and unhelpful boards, this will happen. Maybe that’s what happened here…? Or maybe psychology is just full of crackpots.

Which RPG was it in which you could have an “Atheist” skill that worked to counter the various cleric or magical skills? Simply having a high-level atheist in the same room as someone who could normally reliably heal someone with deity-granted powers could cause their powers to fail. So there is precedent for this idea in geek culture. Maybe the reviewer figured that psychology is more like an RPG than a science anyways.

A referee for a political philosophy journal recommended rejection of a paper of mine for its “conservative political implications” (these, in the referee’s imagination imho). To be fair, the paper was correctly rejected for other and better reasons, contained in another report. But I was mildly shocked that an anonymous reviewer was prepared explicitly to list “conservatism” as a reason for rejection.

Because there’s no theory about what exactly ESP is and how it works, there’s no experimental result that can’t be explained in some ad-hoc manner

A good objection but one with such general application to all manner of concepts in psychology that it’s hard for them to reject papers on that ground alone.

Online journals could just have a little button with a replication thumbs-up icon.

dsquared beat me to it. So, what he said (@10).

in public health I think replication is challenged by ethics. If a well designed study finds a treatment works then it’s really hard to justify replicating the experiment ethically, because you’re expected to give the proven treatment to the control group as well.

If parapsychology researchers want to force their random results into the mainstream they should try public health. Keep plugging away until one of your p values proves that your psychic powers are a boon to public health, and no one will ever be able to refute you – on ethical grounds.

sg states: “… replication is challenged by ethics. If a well designed study finds a treatment works then it’s really hard to justify replicating the experiment ethically, because you’re expected to give the proven treatment to the control group as well.”

Unfortunately, the results of what appear to be well-designed medical studies often cannot be replicated. Initial studies generate enthusiasm, which wanes as further studies contradict these optimistic results. So, it is best to be skeptical until more supporting data are available.

The problem is that the results of medical research are influenced by the study protocol, patient population, hidden biases, etc., etc.

“This latter review didn’t find any problems in our methodology or writeup itself, but suggested that, since the three of us (Richard Wiseman, Chris French and I) are all skeptical of ESP, we might have unconsciously influenced the results using our own psychic powers. … Anyway, the BJP editor agreed with the second reviewer, and said that he’d only accept our paper if we ran a fourth experiment where we got a believer to run all the participants, to control for these experimenter effects.”

…and when that fourth experiment is added the reviewer will of course complain that the authors unconscious psychic powers (UPP) influenced that too: in selecting the believer the UPP had a spillover effect on the believer which in turn distorted the results. Clearly, a believer is needed to select the believer. (Begin regress!).

To avoid that the reviewer and the journal editor has a justificatory task of providing reason to believe that UPP can only work in one interpersonal step, not two. Or three, or…

Oh, wiki tells me Student was William Sealy Gosset. So he was a he.

Replication studies would seem to be an obvious place for experimentation with new approaches to peer review and online publication outside the established, largely commercialized structure. This work is essential yet carries no commercial value and very little academic prestige; on the other hand, a near-guarantee of some form of publication may have a certain attraction. Scholars write book reviews, after all.

It’s not just that failed replications get rejected; in some cases they are suppressed or discarded by the authors or funders. What is required is a registry of empirical research with which all experiments are to be registered, with a complete spec of the methods to be used, hypotheses to be tested, calculations to be run (etc), before the experiment is run. As results are obtained or calculations performed (etc), results must be lodged with the registry.

The registry could then keep records of all replications, too. In an electronic age, it ought to be possible to keep copies of all data and documentation along with the experiment. Administratively speaking, while not trivial the project would certainly be relatively simple.

Experiments not so registered, or for which not all of the pre-specified information would be regarded as methodologically compromised. Likewise researchers who fail to supply data for experiments registered. Last minute changes would be visible and their consequences for the integrity of the experiment assessed.

This would impose no great burden on researchers and could improve the integrity of scientific work no end, since it would reduce the scope for bias conscious or unconscious, in particular such practices as cherrypicking statistical tests and various other Texan practices. I imagine a single repository for scientific data would be very useful in all sorts of other ways too.

Replicating previous experiments might be useful as a training exercise for third year undergrads, MSc students or something – presumably in most cases exactly re-running an experimemt will be much more straightforward than designing, developing and performing one for the first time.

(pa3 should have ‘for which not all of the pre-specified information has been supplied, or not all that supplied pre-specified‘

All right, I suppose I should come out and confess it: *I* used my psychic powers to make the BJP editor present his demand to the paper’s authors.

I’ve never met the editor, don’t even know his name — but my powers are just that impressive.

1) why on earth would anyone care about ESP enough to work on it ? You could also waste your time on phlogiston, angels on the head of a pin, aether, etc…but shouldn’t you have better things to do ?
2) this argument is about quality, not philosophy of science: if you are doing quality science, your work is solid (although I’m an experimental molecular biologists, so I can replicate things before I send the MS off )
[I assume most readers here know the statistics on citation rates; allowing a large heaping of salt for Garfield’s ego, still, the avg citation rate is, what <1 ?
what does that tell you ?
3) Journals don't publish nonreplication studies cause it would bring up point 2 above, and suggest that the journal's papers are poor

why on earth would anyone care about ESP enough to work on it ?

(1) I knew that would be your reaction.

(2) Those spoons don’t bend themselves.

Timothy Scriven @24: Then you may be interested to see that the current ( 2 December 2011) Science is a special issue on data replication and reproducibility. And they have a little poll: http://www.sciencemag.org/content/334/6060/1203.2

I predict I will be shown erotic pictures in the future.

There is no spoon.

The interesting implication of the BJP comments is that “psychic powers” or the belief of the experimenter could influence the results. Given the experimenters followed standard social psychology procedure (e.g. the computer doing most of the work), and they clearly are agreeing with the original Bem conclusions, doesn’t it follow that most of Social Psychology could therefore be up for critique on the grounds that most experimenters do have a hypothesis in mind before running a series of experiments. I mean, isn’t that why it’s called hypothesis testing?
weight loss los angeles

” I’d be interested too, but don’t have access.”

Here’s the poll question at http://scim.ag/rGWIxA, and the current distribution of votes:

READERS’ POLL

How should we promote publication of data that can be replicated and/or reproduced? The articles in the special section on Data Replication & Reproducibility propose a number of possibilities that target funders, journals, and the research culture itself. What do you think?

Ideally, scientists would fully disclose their own raw data and methods and also spend time replicating others’ work. What would best ensure this good behavior?

Recognition and rewards from institutions 7.45%

Funding earmarked for replication studies 20.42%

More publication by journals of data that confirm or refute previous work 43.52%

Rewards from funders on subsequent grant applications for depositing sufficient details for replication (or penalties for noncompliance) 24.14%

Other (add your ideas in the comments section) 4.47%

Additionally, now that I have read over some of the other comments, is it that surprising that psych journal have a habit of not publishing null results given the possibility of Type II errors in the field? Alpha is always set at .05 or lower, but beta is always left free to vary. It’s hard to take null results seriously when most studies are severely under powered.

I haven’t read your paper, but I wonder whether you did a power analysis to make a strong argument for “proving” the null hypothesis. Did you try looking at the effect size of your results and arguing that if would take X thousand more participants to make it significant? This wouldn’t get around the “experimenter effects” argument, but would make your null effects more believable.

As an aside, I have found it very interesting to look at the recent precognition debate in psychology through the lens of how the field of physics has changed its view on the possibility of time travel in recent years. Not so long ago physicists who though time travel was possible were considered crack pots, but now is seems to be a generally accepted idea that it may be possible (says the non-physicist). I think it provides an interesting take on the recent precog findings in psychology). Perhaps precog research ought to still be relegated to the fringes of psychology, but maybe it shouldn’t be written off completely just yet.

Attention: comment spam @39. Comment copied from the parallel post at the Monkey Cage; spammy link added.