Thoughts on Peer Review and Mongooses

by John Holbo on June 4, 2011

First, let me jot some thoughts about Ingrid’s peer-review post that also relate to Henry’s. Then, mongooses. [click to continue…]

The Blank #Slatepitch

by Henry Farrell on June 4, 2011

Via the “ICCI blog”:http://www.cognitionandculture.net/ICCI-blog/the-dark-side-of-evolutionary-psychology.html, some sixty-odd evolutionary psychologists have published a collective letter, disassociating themselves from Satoshi Kanazawa.

bq. We have previously pursued the usual scientific channels open to us to counteract what in our view is Kanazawa’s poor quality science by reviewing and rejecting his papers from scientific journals, and by publishing critiques of his papers in the scientific literature. This has not stopped him from continuing to produce poor quality science and promoting it directly to the public. We have therefore taken the unusual step of making this statement to counteract the damage we believe he is doing to the perception of our discipline in the media and among the public. … Many of these critiques completely undermine the work: the statistician Andrew Gelman, for example, has re-analysed the data Kanazawa used in 2007 to suggest that “Beautiful people have more daughters” and has demonstrated that Kanazawa’s conclusions are simply not supported by the data. Despite this, Kanazawa has not withdrawn the critiqued paper nor published a correction. … The peer review process is not perfect and appears to have failed when dealing with Kanazawa’s poor quality work.Those of us who have reviewed his papers have had experiences where we have rejected papers of his for certain journals on scientific grounds, only to see the papers appear virtually unaltered in print in other journals, despite the detailed critiques of the papers given to Kanazawa by the reviewers and editors of the journals that rejected his papers.

I’ve no doubt that Kanazawa’s work is bad by the commonly accepted standards of evolutionary psychology. But as the ICCI blog politely suggests, there is a broader problem with the field that the collective letter doesn’t address as directly as it should. Evolutionary psychology has benefited from media attention, but also been distorted by it – there are significant incentives to produce ‘shocking’ and ‘contrarian’ findings. I saw this first hand a few years ago when I got involved in an email discussion with the co-editor of an evolutionary psychology journal which had published one of Kanazawa’s more egregious stinkers. When I pushed the person in question on how obviously bad the piece was, the response was that:

bq. I happen to think it is a great thought provoking document, and one of the few in the last ten years that have actually gotten people to talk about issues. … I would rather have an article that causes people to think and talk and yes, argue and criticize than to publish an article that is one more facet of the same old thing.

There’s something to be said for stirring it up every once in a while. But there’s also something to be said for trying to get things right. Typically, academic journals are supposed to emphasize the latter rather than the former. It’s beyond dispute that Kanazawa can produce “thought provoking”1 articles that get people to “argue and criticize.”2 But the peer review process is supposed to do a bit more than to verify that your ideas are daring and controversial. That at least one journal editor (and surely more than one, given Kanazawa’s publication record) in the field don’t seem to understand this suggests that there is a problem.

1 If “thoughts” can be taken to encompass internal queries-to-self along the lines of ‘how the fuck did this ever get published?’

2 The joint letter mentions some 24 critiques of his papers by 59 social and natural scientists. If Satoshi Kanazawa did not exist, theorists and methodologists would have to invent him as a cautionary example.

Update: I should probably link manually to Cosma’s webpost responding to this since, you know, that whole Trackback thing doesn’t work so well anymore. I should also say that while I am no very great fan of Stephen Pinker’s work, the title of this post should not be read in any way as implying that it’s on a level with Kanazawa’s ( I just don’t think of halfway decent puns often enough that I can easily junk them).