Playing nice

by Henry on October 4, 2007

“Anmik,” commenting on Kieran’s post below writes:

But then, what about the very real, very reasonable impulse to softpedal book reviews, to write overly generous tenure evaluations, and on and on? I only ask because the whole idea of peer review, and the review process more broadly, has begun to feel a bit, how to say this, corrupt of late. I noted yesterday some very minor flap in which Chris Matthews, the political “journalist,” admitted that it’s hard to cover the people he sees at cocktail parties. The same is true for scholars, no? And blogs only compound the problem, I suppose.

This is a real problem, albeit not one of recent vintage; mutual backscratching is especially endemic in academic book reviews. But it isn’t one that is very easy to fix. Some months ago, I wrote a quite negative review of a book that seemed lazy to me; I thought it only fair to signal this to potential buyers. But precisely because there’s a kind of pooling equilibrium going on here I worried a bit about whether or not I was sending out too negative a signal to readers and colleagues about the book’s quality. If everyone says nice things about everyone else’s books, then a negative review will likely be overinterpreted as saying that this book really stinks to high heaven, its author should be denied tenure etc. In this case, it wasn’t a major issue, because the book was written by a seniorish figure, who presumably wouldn’t be damaged too much by an asst. prof’s evaluation of his work. However, I would worry if I were writing a review of a more junior person’s work, to try to be sure that whatever signal I was sending out wasn’t misinterpreted.

I suspect that this is even more of an issue in tenure review letters, where, I understand, negative comments will receive far more attention than positive ones on the assumption that their informational content is much higher. Given this, someone who wants to give an honest assessment that is critical but basically favorable is going to have to choose her words very carefully indeed so that she says only what she wants to say and isn’t interpreted as saying more. So the problem isn’t simply one of corruption – it’s also one of signalling conventions that are quite hard to overturn once they become established. Anonymous peer review is different in my experience – negative signals by and large don’t receive undue weight, because there aren’t any obvious pressures to converge on an equilibrium where we say nice things about each other even though we don’t mean them.

{ 6 comments }

1

Michael D 10.04.07 at 7:12 pm

I wonder how peer review inflation differs meaningfully from grade inflation.

One, I suppose the grader does not normally get to choose words carefully, and must depend on the highly coded B+ to register criticism. Two, instead of having review committees (and, to a lesser extent, collegiality) to blame for the practice, grade inflation seems (in the popular(?) anti-education-establishment tradition) set at the feet of lazy students and the educators unwilling to stand up to them.

It is interesting to hear this dynamic described without the twin presumptions of worthlessness and fecklessness.

2

Paul J. Reber 10.04.07 at 7:23 pm

I don’t disagree about the academic benefit of mutual back-scratching in peer review, but feel obliged to point out some counter-pressure.

First, I think it’s pretty well understood to be a good way to get famous as a young scientist to pick a public fight. Ideally with a senior person (preferred due to increased visibility) or another junior person if there’s a suitable ideological divide. This could be a feature of my field (psychology, neuroscience) in which either there isn’t really a dominant old guard or maybe that the old guard leaves a lot of room for other people to fight over. I don’t think it’s unique to my field, though.

Second, tenure decisions at my university are heavily based on external letters (perhaps more than some others), but a negative letter is not a killer. The culture of this process is that not only is it ok to get a “bad” letter, but that it’s a sign of impact. If somebody writes that “so-and-so is wrong about everything and this is hugely annoying!” then that’s a sign that so-and-so is a player. The killers are letters that damn by faint praise or worse, “never heard of so-and-so.”

I’m not recommending running out and picking fights or making colleagues mad (via blogs or any other way) as your field or department may be different. I’m just saying that there’s at least some small incentive towards being negative, perhaps enough to get honest appraisals now and then.

3

terence 10.04.07 at 7:56 pm

I suspect that this is even more of an issue in tenure review letters, where, I understand, negative comments will receive far more attention than positive ones on the assumption that their informational content is much higher.

But isn’t this also the case in life in general. For example, if someone asks me what I think about Person X, if I like Person X I will usually just limit my comments to that point alone. This because the moment I add caveats it is the caveats that stick in the listener’s head – not the actual overall assessment.

4

slolernr 10.04.07 at 7:58 pm

Anonymous peer review is different in my experience – negative signals by and large don’t receive undue weight

Indeed, perhaps the opposite. After all, anonymous peer refereeing creates incentives for a different set of abuses, one of which is the ill-founded but strongly phrased report that boils down to “you didn’t cite my article/book!” or, if you did, “you didn’t fawningly parrot my argument!” IME editors know this, and treat such reports accordingly.

5

Kenny Easwaran 10.05.07 at 3:42 am

I would think the problem with negative reviews is that readers just judge the author of the review more than the author of the book.

6

Witt 10.05.07 at 4:32 pm

The overwhelmingly important thing to me in a book review or personnel review is how specific the comments are. Generic praise – bah, nearly meaningless. Generic criticism (even if strongly worded) – okay, so the writer feels strongly, but unless I know them quite well I have very little context for gauging how seriously to take their strong feelings.

When I write reviews I try to be extremely specific. Praise particular activities, describe projects, etc. Likewise with criticism.

This has the happy side effect of making it very hard to write a dishonest review.

Comments on this entry are closed.