The “NYT”:http://www.nytimes.com/2004/02/14/technology/14AMAZ.html?ex=1392094800&en=183dc1d16a0c7b4c&ei=5007&partner=USERLAND has an article on anonymous reviews on Amazon, and how they’re manipulated in different ways by authors, authors’ friends, and authors’ most bitter enemies. It’s a real problem with a system that allows uncontrolled anonymity or pseudonymity – the information content of the average review quickly drops to zero, unless (like “Tyler Cowen”:http://www.marginalrevolution.com/marginalrevolution/2004/01/how_do_consumer.html you’re interested in the degree of controversy that surrounds the book, rather than the ratio of positive to negative reviews). For an academic, the obvious point of comparison is peer review. Most halfway decent scholarly journals[1] get anonymous scholars to review any articles that are submitted to them so as to assess publishability. Although the editor of the journal usually has the final say, the anonymous reviewers’ findings count for a lot. There’s a lot of bitching and griping about this in the particular, especially because it’s sometimes not too difficult for the paper’s author to guess the identity of the ‘anonymous’ reviewer who did a hatchet-job on their cherished piece. The identity of particularly venomous reviewers is the subject of (frequently lurid) speculation and gossip.
Still, the system works reasonably well in the general, for three reasons. First, even if the reviewers are anonymous from the point of view of the article’s author, the journal’s editor knows who they are. This encourages at least some degree of responsibility on the part of the reviewer; even those with malice in their hearts may prefer not to run the risk of becoming known as a partisan hack by a journal editor, who may be receiving their own pieces in the future. Second, most journals will solicit at least two, and very likely three or four reviews, which ideally will be written by people from a variety of backgrounds, so that neither the author’s friends nor foes determine the article’s fate. This doesn’t always work as well as it should – but most journals at least make good-faith efforts to ensure that a piece receives a fair hearing. Finally, anonymity does provide some protection for fair criticism. Even in contexts where the disgruntled author of a rejected article can make a fair guess at who the reviewers were, they can’t be entirely sure; thus, it’s hard for them to retaliate, even when they’re powerful figures in the field. Anonymous peer review isn’t perfect – but by and large the articles that get published in the better known journals in the social sciences are reasonably good, interesting pieces (I don’t know other disciplines well enough to comment properly on their journals).
fn1. Legal journals are the most obvious exception.
{ 12 comments }
tcb 02.16.04 at 6:54 pm
Well, isn’t it a bit naive to rely on reviews posted by a bookseller, whether online or not? It’s like reading the jacket review. Use Amazon to buy if you like, but never to judge.
Ophelia Benson 02.16.04 at 7:07 pm
“the articles that get published in the better known journals in the social sciences are reasonably good, interesting pieces”
Always excepting, of course, Social Text.
Sorry, sorry, I just couldn’t resist!
[Disclaimer: And it’s not really better known, and not a journal, not peer-reviewed, etc. That was their defense, anyway.]
eszter 02.16.04 at 7:56 pm
by and large the articles that get published in the better known journals in the social sciences are reasonably good, interesting pieces
Hmm.. I don’t know if I’m quite that enthusiastic about all that gets published. Moreover, we don’t know what the alternatives would have been as we only see the pieces that were published, not those that were rejected. (Of course, ideally pieces that get rejected from one journal are revised and sent to another so if they are reasonably good and interesting they get published at some point, but I’m not convinced that this is always the case.)
Regarding the double-blind review process, I agree with you that for the most part it works well. I do think, however, that there should be something built into the system that discourages bad reviews. When I say “bad reviews” I don’t mean critical reviews but really short generic ones. Sometimes reviewers will write no more than a tiny paragraph and it is fairly clear they did not read the piece (or not carefully in any case). It is unfortunate that when reviewers get acknowledged for their contributions to journals (once a year or so there’s a list of reviewers acknowledged by the editor/s) everyone is lumped into one group as though all reviewers were equally helpful.. which is definitely not the case. Given that entire careers can depend on such reviews, it would be nice to see people take them more seriously.
aardvark 02.16.04 at 9:31 pm
Some journals try to guard against “bad” (i.e. lazy or sloppy or ad hominem) reviews by sending copies of all the reviews, along with the editor’s letter to the author explaining his/her decision, to all the reviewers. That creates a certain amount of peer pressure – if the would-be villain know that two or three of his peers are going to see how he reviewed the same manuscript that they did, it may keep him honest.
Ted 02.16.04 at 9:38 pm
Maybe this is the place to mention the reviewer who’s been tirelessly working over Bil Keane’s Family Circus collections on Amazon. It’s been going on for years. Check it out: http://www.amazon.com/exec/obidos/tg/detail/-/0449125238/qid=1076967445/sr=1-7/ref=sr_1_7/103-4241131-2560652?v=glance&s=books
harry 02.16.04 at 10:41 pm
bq. Some journals try to guard against “bad†(i.e. lazy or sloppy or ad hominem) reviews by sending copies of all the reviews
As a reviewer I love this practice — I really enjoy seeing what other people have said about the same paper. I’m not sure it affects my reviewing much — but I’m a pretty boring reviewer. I *don’t* like knowing an author’s identity (which is soemtimes unavoidable in a small field, esp. given that some are not very careful about rendering their paper anonymous.
Eszter says
bq. I don’t know if I’m quite that enthusiastic about all that gets published. Moreover, we don’t know what the alternatives would have been as we only see the pieces that were published, not those that were rejected
In philosophy and the bits of education I know well I’m pretty sure that the following is the case — very good stuff almost always gets into pretty good journals, but some not-good stuff gets past reviewers and editors occasionally (much more in education than philosophy in my opinion). But, really, really good cutting edge stuff can be very hard to publish in journals — and there are some terrific philosophers who basically rely on invites to get their best stuff out (I have in mind one senior philosopher in particular whom I won’t name but among whose absolutely top-notch papers only two are in journals — and I know it is because several of his most important pieces were repeatedly rejected by journals — and I can see why they were, even though they are better than 90% of what goes in those journals).
Tom T. 02.16.04 at 10:52 pm
As for reader reviewers on Amazon, I must point out the incomparably amusing Henry Raddick. His reviews have turned that medium into an unexpected comic art form.
http://www.amazon.com/exec/obidos/tg/cm/member-reviews/-/AA9IP6AYACFK5/1/ref=cm_cr_auth/002-8909259-5048036
harry 02.16.04 at 11:55 pm
I can’t stand it tom t. You shouldn’t be allowed to send people to the Henry Raddick page without some sort of warning. I’m in tears and now have to compose myself for my daughter’s swimming lesson.
John Quiggin 02.17.04 at 2:38 am
I’ve never been very happy about the way the system works in economics. The core of the problem is that rejection rates are so high (90 per cent) that a single negative or even tepid review is enough to sink a paper in many cases. The result, I find, is that original stuff is much harder to publish than PhD variation on established themes (I’ve done my share of both).
The system works fairly well in its negative role of keeping out junk, but, IMO does poorly in selecting the best papers for publication.
eszter 02.17.04 at 3:36 am
Aardvark – I’m not sure how you can guard against “bad” reviews by sending copies of all reviewers’ evaluations to all others. My experience with that has been that it’s all anonymous so I don’t know who wrote the bad reviews.
Harry and John – I agree that the process probably keeps out most really bad stuff, but it is hard for very innovative material to get through (as you mention). My father wrote a book about Nobel Prize winners (sciences) and he mentions this there as well. Really innovative work (in this case the type that may have eventually led to a Nobel Prize) is sometimes extremely difficult to publish and gets rejected multiple times.
Tom T – Thanks for sharing!:)
Mrs Tilton 02.17.04 at 1:25 pm
Ophelia,
much as I love the Sokal/Social Text fiasco, I’d point out that even Nature must retract the occasional paper. Peer review is no panacea.
That said, that ST did not bother to check (or was incapable of checking) whether ‘Transgressing the Boundaries’ even passed the laugh test is, well, laughable.
TomD 02.17.04 at 2:04 pm
Unfortunately in my field (theoretical physics) the referee’s word is God to most editors, who see their roles as little more than postal workers forwarding manuscripts back and forth between author and reviewer. If the referee is ignorant, unfair or pigheaded it is very rare for the editor to notice it. If the referee approves the paper but has nothing to say about it the editor doesn’t mind either.
If the editors have any authority in the subject (which they certainly ought to) they don’t usually bother to use it.
Comments on this entry are closed.