The NYT has an article on anonymous reviews on Amazon, and how they’re manipulated in different ways by authors, authors’ friends, and authors’ most bitter enemies. It’s a real problem with a system that allows uncontrolled anonymity or pseudonymity - the information content of the average review quickly drops to zero, unless (like Tyler Cowen you’re interested in the degree of controversy that surrounds the book, rather than the ratio of positive to negative reviews). For an academic, the obvious point of comparison is peer review. Most halfway decent scholarly journals1 get anonymous scholars to review any articles that are submitted to them so as to assess publishability. Although the editor of the journal usually has the final say, the anonymous reviewers’ findings count for a lot. There’s a lot of bitching and griping about this in the particular, especially because it’s sometimes not too difficult for the paper’s author to guess the identity of the ‘anonymous’ reviewer who did a hatchet-job on their cherished piece. The identity of particularly venomous reviewers is the subject of (frequently lurid) speculation and gossip.
Still, the system works reasonably well in the general, for three reasons. First, even if the reviewers are anonymous from the point of view of the article’s author, the journal’s editor knows who they are. This encourages at least some degree of responsibility on the part of the reviewer; even those with malice in their hearts may prefer not to run the risk of becoming known as a partisan hack by a journal editor, who may be receiving their own pieces in the future. Second, most journals will solicit at least two, and very likely three or four reviews, which ideally will be written by people from a variety of backgrounds, so that neither the author’s friends nor foes determine the article’s fate. This doesn’t always work as well as it should - but most journals at least make good-faith efforts to ensure that a piece receives a fair hearing. Finally, anonymity does provide some protection for fair criticism. Even in contexts where the disgruntled author of a rejected article can make a fair guess at who the reviewers were, they can’t be entirely sure; thus, it’s hard for them to retaliate, even when they’re powerful figures in the field. Anonymous peer review isn’t perfect - but by and large the articles that get published in the better known journals in the social sciences are reasonably good, interesting pieces (I don’t know other disciplines well enough to comment properly on their journals).
1 Legal journals are the most obvious exception.
Well, isn’t it a bit naive to rely on reviews posted by a bookseller, whether online or not? It’s like reading the jacket review. Use Amazon to buy if you like, but never to judge.
“the articles that get published in the better known journals in the social sciences are reasonably good, interesting pieces”
Always excepting, of course, Social Text.
Sorry, sorry, I just couldn’t resist!
[Disclaimer: And it’s not really better known, and not a journal, not peer-reviewed, etc. That was their defense, anyway.]
by and large the articles that get published in the better known journals in the social sciences are reasonably good, interesting pieces
Hmm.. I don’t know if I’m quite that enthusiastic about all that gets published. Moreover, we don’t know what the alternatives would have been as we only see the pieces that were published, not those that were rejected. (Of course, ideally pieces that get rejected from one journal are revised and sent to another so if they are reasonably good and interesting they get published at some point, but I’m not convinced that this is always the case.)
Regarding the double-blind review process, I agree with you that for the most part it works well. I do think, however, that there should be something built into the system that discourages bad reviews. When I say “bad reviews” I don’t mean critical reviews but really short generic ones. Sometimes reviewers will write no more than a tiny paragraph and it is fairly clear they did not read the piece (or not carefully in any case). It is unfortunate that when reviewers get acknowledged for their contributions to journals (once a year or so there’s a list of reviewers acknowledged by the editor/s) everyone is lumped into one group as though all reviewers were equally helpful.. which is definitely not the case. Given that entire careers can depend on such reviews, it would be nice to see people take them more seriously.
Some journals try to guard against “bad” (i.e. lazy or sloppy or ad hominem) reviews by sending copies of all the reviews, along with the editor’s letter to the author explaining his/her decision, to all the reviewers. That creates a certain amount of peer pressure - if the would-be villain know that two or three of his peers are going to see how he reviewed the same manuscript that they did, it may keep him honest.
Maybe this is the place to mention the reviewer who’s been tirelessly working over Bil Keane’s Family Circus collections on Amazon. It’s been going on for years. Check it out: http://www.amazon.com/exec/obidos/tg/detail/-/0449125238/qid=1076967445/sr=1-7/ref=sr_1_7/103-4241131-2560652?v=glance&s=books
Some journals try to guard against “bad” (i.e. lazy or sloppy or ad hominem) reviews by sending copies of all the reviews
As a reviewer I love this practice — I really enjoy seeing what other people have said about the same paper. I’m not sure it affects my reviewing much — but I’m a pretty boring reviewer. I don’t like knowing an author’s identity (which is soemtimes unavoidable in a small field, esp. given that some are not very careful about rendering their paper anonymous.
Eszter says
I don’t know if I’m quite that enthusiastic about all that gets published. Moreover, we don’t know what the alternatives would have been as we only see the pieces that were published, not those that were rejected
In philosophy and the bits of education I know well I’m pretty sure that the following is the case — very good stuff almost always gets into pretty good journals, but some not-good stuff gets past reviewers and editors occasionally (much more in education than philosophy in my opinion). But, really, really good cutting edge stuff can be very hard to publish in journals — and there are some terrific philosophers who basically rely on invites to get their best stuff out (I have in mind one senior philosopher in particular whom I won’t name but among whose absolutely top-notch papers only two are in journals — and I know it is because several of his most important pieces were repeatedly rejected by journals — and I can see why they were, even though they are better than 90% of what goes in those journals).
As for reader reviewers on Amazon, I must point out the incomparably amusing Henry Raddick. His reviews have turned that medium into an unexpected comic art form.
http://www.amazon.com/exec/obidos/tg/cm/member-reviews/-/AA9IP6AYACFK5/1/ref=cm_cr_auth/002-8909259-5048036
I can’t stand it tom t. You shouldn’t be allowed to send people to the Henry Raddick page without some sort of warning. I’m in tears and now have to compose myself for my daughter’s swimming lesson.
I’ve never been very happy about the way the system works in economics. The core of the problem is that rejection rates are so high (90 per cent) that a single negative or even tepid review is enough to sink a paper in many cases. The result, I find, is that original stuff is much harder to publish than PhD variation on established themes (I’ve done my share of both).
The system works fairly well in its negative role of keeping out junk, but, IMO does poorly in selecting the best papers for publication.
Aardvark - I’m not sure how you can guard against “bad” reviews by sending copies of all reviewers’ evaluations to all others. My experience with that has been that it’s all anonymous so I don’t know who wrote the bad reviews.
Harry and John - I agree that the process probably keeps out most really bad stuff, but it is hard for very innovative material to get through (as you mention). My father wrote a book about Nobel Prize winners (sciences) and he mentions this there as well. Really innovative work (in this case the type that may have eventually led to a Nobel Prize) is sometimes extremely difficult to publish and gets rejected multiple times.
Tom T - Thanks for sharing!:)
Ophelia,
much as I love the Sokal/Social Text fiasco, I’d point out that even Nature must retract the occasional paper. Peer review is no panacea.
That said, that ST did not bother to check (or was incapable of checking) whether ‘Transgressing the Boundaries’ even passed the laugh test is, well, laughable.
Unfortunately in my field (theoretical physics) the referee’s word is God to most editors, who see their roles as little more than postal workers forwarding manuscripts back and forth between author and reviewer. If the referee is ignorant, unfair or pigheaded it is very rare for the editor to notice it. If the referee approves the paper but has nothing to say about it the editor doesn’t mind either.
If the editors have any authority in the subject (which they certainly ought to) they don’t usually bother to use it.
À Gauche
Jeremy Alder
Amaravati
Anggarrgoon
Audhumlan Conspiracy
H.E. Baber
Philip Blosser
Paul Broderick
Matt Brown
Diana Buccafurni
Brandon Butler
Keith Burgess-Jackson
Certain Doubts
David Chalmers
Noam Chomsky
The Conservative Philosopher
Desert Landscapes
Denis Dutton
David Efird
Karl Elliott
David Estlund
Experimental Philosophy
Fake Barn County
Kai von Fintel
Russell Arben Fox
Garden of Forking Paths
Roger Gathman
Michael Green
Scott Hagaman
Helen Habermann
David Hildebrand
John Holbo
Christopher Grau
Jonathan Ichikawa
Tom Irish
Michelle Jenkins
Adam Kotsko
Barry Lam
Language Hat
Language Log
Christian Lee
Brian Leiter
Stephen Lenhart
Clayton Littlejohn
Roderick T. Long
Joshua Macy
Mad Grad
Jonathan Martin
Matthew McGrattan
Marc Moffett
Geoffrey Nunberg
Orange Philosophy
Philosophy Carnival
Philosophy, et cetera
Philosophy of Art
Douglas Portmore
Philosophy from the 617 (moribund)
Jeremy Pierce
Punishment Theory
Geoff Pynn
Timothy Quigley (moribund?)
Conor Roddy
Sappho's Breathing
Anders Schoubye
Wolfgang Schwartz
Scribo
Michael Sevel
Tom Stoneham (moribund)
Adam Swenson
Peter Suber
Eddie Thomas
Joe Ulatowski
Bruce Umbaugh
What is the name ...
Matt Weiner
Will Wilkinson
Jessica Wilson
Young Hegelian
Richard Zach
Psychology
Donyell Coleman
Deborah Frisch
Milt Rosenberg
Tom Stafford
Law
Ann Althouse
Stephen Bainbridge
Jack Balkin
Douglass A. Berman
Francesca Bignami
BlunkettWatch
Jack Bogdanski
Paul L. Caron
Conglomerate
Jeff Cooper
Disability Law
Displacement of Concepts
Wayne Eastman
Eric Fink
Victor Fleischer (on hiatus)
Peter Friedman
Michael Froomkin
Bernard Hibbitts
Walter Hutchens
InstaPundit
Andis Kaulins
Lawmeme
Edward Lee
Karl-Friedrich Lenz
Larry Lessig
Mirror of Justice
Eric Muller
Nathan Oman
Opinio Juris
John Palfrey
Ken Parish
Punishment Theory
Larry Ribstein
The Right Coast
D. Gordon Smith
Lawrence Solum
Peter Tillers
Transatlantic Assembly
Lawrence Velvel
David Wagner
Kim Weatherall
Yale Constitution Society
Tun Yin
History
Blogenspiel
Timothy Burke
Rebunk
Naomi Chana
Chapati Mystery
Cliopatria
Juan Cole
Cranky Professor
Greg Daly
James Davila
Sherman Dorn
Michael Drout
Frog in a Well
Frogs and Ravens
Early Modern Notes
Evan Garcia
George Mason History bloggers
Ghost in the Machine
Rebecca Goetz
Invisible Adjunct (inactive)
Jason Kuznicki
Konrad Mitchell Lawson
Danny Loss
Liberty and Power
Danny Loss
Ether MacAllum Stewart
Pam Mack
Heather Mathews
James Meadway
Medieval Studies
H.D. Miller
Caleb McDaniel
Marc Mulholland
Received Ideas
Renaissance Weblog
Nathaniel Robinson
Jacob Remes (moribund?)
Christopher Sheil
Red Ted
Time Travelling Is Easy
Brian Ulrich
Shana Worthen
Computers/media/communication
Lauren Andreacchi (moribund)
Eric Behrens
Joseph Bosco
Danah Boyd
David Brake
Collin Brooke
Maximilian Dornseif (moribund)
Jeff Erickson
Ed Felten
Lance Fortnow
Louise Ferguson
Anne Galloway
Jason Gallo
Josh Greenberg
Alex Halavais
Sariel Har-Peled
Tracy Kennedy
Tim Lambert
Liz Lawley
Michael O'Foghlu
Jose Luis Orihuela (moribund)
Alex Pang
Sebastian Paquet
Fernando Pereira
Pink Bunny of Battle
Ranting Professors
Jay Rosen
Ken Rufo
Douglas Rushkoff
Vika Safrin
Rob Schaap (Blogorrhoea)
Frank Schaap
Robert A. Stewart
Suresh Venkatasubramanian
Ray Trygstad
Jill Walker
Phil Windley
Siva Vaidahyanathan
Anthropology
Kerim Friedman
Alex Golub
Martijn de Koning
Nicholas Packwood
Geography
Stentor Danielson
Benjamin Heumann
Scott Whitlock
Education
Edward Bilodeau
Jenny D.
Richard Kahn
Progressive Teachers
Kelvin Thompson (defunct?)
Mark Byron
Business administration
Michael Watkins (moribund)
Literature, language, culture
Mike Arnzen
Brandon Barr
Michael Berube
The Blogora
Colin Brayton
John Bruce
Miriam Burstein
Chris Cagle
Jean Chu
Hans Coppens
Tyler Curtain
Cultural Revolution
Terry Dean
Joseph Duemer
Flaschenpost
Kathleen Fitzpatrick
Jonathan Goodwin
Rachael Groner
Alison Hale
Household Opera
Dennis Jerz
Jason Jones
Miriam Jones
Matthew Kirschenbaum
Steven Krause
Lilliputian Lilith
Catherine Liu
John Lovas
Gerald Lucas
Making Contact
Barry Mauer
Erin O'Connor
Print Culture
Clancy Ratcliff
Matthias Rip
A.G. Rud
Amardeep Singh
Steve Shaviro
Thanks ... Zombie
Vera Tobin
Chuck Tryon
University Diaries
Classics
Michael Hendry
David Meadows
Religion
AKM Adam
Ryan Overbey
Telford Work (moribund)
Library Science
Norma Bruce
Music
Kyle Gann
ionarts
Tim Rutherford-Johnson
Greg Sandow
Scott Spiegelberg
Biology/Medicine
Pradeep Atluri
Bloviator
Anthony Cox
Susan Ferrari (moribund)
Amy Greenwood
La Di Da
John M. Lynch
Charles Murtaugh (moribund)
Paul Z. Myers
Respectful of Otters
Josh Rosenau
Universal Acid
Amity Wilczek (moribund)
Theodore Wong (moribund)
Physics/Applied Physics
Trish Amuntrud
Sean Carroll
Jacques Distler
Stephen Hsu
Irascible Professor
Andrew Jaffe
Michael Nielsen
Chad Orzel
String Coffee Table
Math/Statistics
Dead Parrots
Andrew Gelman
Christopher Genovese
Moment, Linger on
Jason Rosenhouse
Vlorbik
Peter Woit
Complex Systems
Petter Holme
Luis Rocha
Cosma Shalizi
Bill Tozier
Chemistry
"Keneth Miles"
Engineering
Zack Amjal
Chris Hall
University Administration
Frank Admissions (moribund?)
Architecture/Urban development
City Comforts (urban planning)
Unfolio
Panchromatica
Earth Sciences
Our Take
Who Knows?
Bitch Ph.D.
Just Tenured
Playing School
Professor Goose
This Academic Life
Other sources of information
Arts and Letters Daily
Boston Review
Imprints
Political Theory Daily Review
Science and Technology Daily Review