Here is a very old joke. A soldier is captured during a long-running war and thrown into the most stereotypical prison cell imaginable. Inside the cell is another solider. He has an enormous, disgusting-smelling beard and has clearly been there a long time. The young solider immediately sets about trying to escape. He is resourceful and possessed of great willpower. He bribes a guard with his emergency supply of cash. The guard gets him into a supply truck and he makes it to the prison garage, but is found during a routine vehicle search while exiting the compound. He is returned to his cell. His mangy companion says nothing about his departure or return. Undeterred, the young soldier works on the bars of the cell for weeks, filing them down with a shim made from a toothbrush. The whole time the old soldier looks on, silently. The young soldier finally breaks the bars, slips out the window and makes it to the outer wall, where he is spotted and recaptured. He is thrown back in the cell. He glowers at his grizzled companion, who still remains silent. Calming himself and mastering his despair, he tries yet again, this time digging a tunnel with the narrow end of a broken plastic coffee spoon. After about two years of work he succeeds in escaping under the wall and making it to the nearest town, only to be captured again at the train station. He is delivered, once again, back to his cell and its taciturn occupant. At the end of his wits, the young soldier finally confronts the old soldier, shouting, “Couldn’t you at least offer to help me with this?! I mean, I’ve come up with all these great plans—you could have joined me in executing them! What’s wrong with you?” The old soldier looks at him and says, “Oh I tried all these methods years ago—bribery, the bars, a tunnel, and a few others besides—none of them work.” The young soldier looks at him, incredulous, and screams “Well if you knew they didn’t work, WHY THE FUCK DIDN’T YOU TELL ME BEFORE I TRIED THEM, YOU BASTARD?!” The old soldier scratches his filthy beard and says, “Hey, who publishes negative results?“
{ 20 comments }
Martin Langeland 04.23.11 at 8:06 pm
Null studies are the most significant perishable of science.
And they provide great joy the marketers who delight in context-less quotes.
–ml
mcd 04.23.11 at 11:00 pm
Should’ve been in a Sunday “Wizard of Id” comic.
But then the warden should’ve published them since they are positive results from his POV.
JP Stormcrow 04.24.11 at 12:35 am
The old soldier finally dies and is replaced by an economist who refuses to help because, “If it is possible to escape you would have already.”
Neil 04.24.11 at 1:51 am
There is a real issue here, but Goldacre’s implicit suggestion that JP&SP or Science (!) should have published the failures to replicate is silly. Failures to replicate are boring. They should be published, but what social science needs is a repository of such studies, on line, rather than to have these things appear in good journals.
Eszter Hargittai 04.24.11 at 1:51 am
This reminds me of the media hype a couple of years ago about the link between Facebook use and grades. The press release claiming a link – based on extremely problematic evidence – got a ton of coverage. Our paper following up a few weeks later challenging those findings got some coverage, but considerably less.
I’ll take this opportunity to mention that sociologist Jeremy Freese did a very nice job writing up how he got null findings published in a top sociology journal. See the piece [pdf] he contributed to my book Research Confidential.
Trevor 04.24.11 at 2:05 am
For the several readers of this blog who haven’t already seen it, xkcd’s recent treatment of the selective reporting of positive findings was very nice. http://xkcd.com/882/
Hidari 04.24.11 at 11:09 am
What would be interesting (although something tells me that Goldacre will not want to have this discussion) would be if we used this anecdote and joke to illustrate why NHST is (and must be) a lot of crap.
Hidari 04.24.11 at 11:11 am
http://www.andrews.edu/~rbailey/Chapter%20two/7217331.pdf
ejh 04.24.11 at 11:23 am
The old soldier finally dies and is replaced by an economist who refuses to help because, “If it is possible to escape you would have already.â€
Unfortunately I know of no example of any economist being imprisoned for their crimes.
sg 04.24.11 at 12:58 pm
Is the enormous disgusting-smelling beard in this story a hint at something?
jim 04.24.11 at 4:06 pm
@9:
Merely corroborative detail, intended to give artistic verisimilitude to an otherwise bald and unconvincing narrative.
Anand Manikutty 04.24.11 at 5:50 pm
To continue Stormcrow’s story – And the first cellmate told the economist : “That is a lie! Of course, there is the possibility that someone could escape from this prison. Did you say that because you are an economist?” Whereupon the economist, Mihir Desai, replied with some indignation, “I, sir, am an economist *and* a liar.” (Some names may have been changed to help preserve anonymity).
garymar 04.24.11 at 10:59 pm
Yes, but he’s in a different field. What entomologist reads Physics Letters B?
BenSix 04.25.11 at 1:31 pm
I sympathise with Wiseman and his comrades on this one but, as he knows, it can swing the other way. Doubtless, science reporting – and publication – has its problems but the very fact that Wiseman, Randi, Shermer and the rest are such public figures suggests that it’s not simply weighted against the sceptics.
chris 04.25.11 at 2:02 pm
If the new inmate had thought about it, he would have realized that the old inmate’s very presence was evidence about the difficulty of escape (although not necessarily a comment on any particular method). Did he think that the old inmate had never thought of trying to escape, or hadn’t thought of the same methods he himself was thinking of? Or did he just not think that much about the implications of his cellmate’s presence and what his experiences might be?
Clearly he didn’t bother to *ask*…
Michael 04.25.11 at 3:16 pm
I disagree with Neil. When researchers cannot replicate a study of enormous interest, one that has been re-reported thousands of times in the popular press, even a prestigious high-circulation journal should be obligated to publish a short letter (of less than a page) on it. In fact, Nature has a separate track for this sort of letter called Communications Arising. Science has one called Technical Comments. Why not JPSP?
Neil 04.25.11 at 10:59 pm
Michael, that’s not a bad idea. But you don’t disagree with me, since you do not advocate that the journal publish a full paper.
Dr. Hilarius 04.26.11 at 7:25 pm
Neil @4: so a claim for cold fusion merits publication but failures to replicate it don’t? Replication of existing studies obtaining the same results may be boring but studies calling into question extraordinary claims? Hardly.
SusanC 04.27.11 at 9:07 am
There’s a difference between negative results–“We looked for a correlation between X and Y and didn’t find one” and failure to replicate “Dr. X’s paper in the previous issue of this journal reported a correlation between X and Y, but when we repeated the experiment we didn’t find one.”
For most of the journals I review for, a “failure to replicate” of an experiment published in the same journal is almost guaranteed to be accepted (unless it’s really badly written, or they’ve made some new error of methodology that wasn’t present in the original). The general view is that if the original result was interesting enough to be worth publishing, failure to replicate must also be interesting enough to publish. (Which leaves bad writing or flawed methods as the only likely reasons for rejection.) By “failure to replicate” I’d also include re-analysis of the mathematics or statistics, as well as repeating the experiments. (e.g. if the original result only appeared to be statistically significant because of some egregious blunder in the calculation).
Negative results (e.g. observing that X is not correlated with Y, when no-one has ever seriously suggested that it was) are another matter, and don’t tend to go down to well. There’s an argument for publishing negative results as well. A couple of weeks ago someone was telling me the story of how several different universities had each had a PhD student waste a considerable amount of time on the same fruitless approach to a problem; if they’d known the approach had already been tried and didn’t pan out (for technical reasons that only became apparent after months of work), they wouldn’t have bothered.
However, the problem with negative results is that there are so many of them. (But there can’t be more “failure to replicate” results than there were original positive results to try and replicate).
Jay Livingston 04.27.11 at 4:47 pm
“the Gifted will Foresee the Punchline” The gifted. . . and maybe those who saw it on your blog in 2006 and maybe stole it to repost on their blogs.
Comments on this entry are closed.