Comments on: How much should we referee?

By: Joe Hoover

Joe Hoover — Fri, 15 Feb 2008 13:38:45 +0000

As a graduate student working as an editor for a student run IR journal, meaning with limited experience, I find it is fairly easy to get reviewers. I did think it was worth mentioning that one of the best things that a reviewer can do is to say no quickly and if possible suggest others, this is especially true for those who are well known experts in the field and may have information on lesser known but excellent scholars. Waiting for a response is much worse than having a quick no.

Also, I would think a commitment to fewer reviews of a higher quality is probably better. Though, I must express my gratitude to all those who do offer substantive and timely reviews.

By: dr ngo

dr ngo — Thu, 14 Feb 2008 06:20:48 +0000

Not directly on the question of numbers, but perhaps a germane general consideration:

I’ve always taken it as a general proposition that I ought not to agree to referee an article or a book manuscript, or, for that matter, to review a book, if I don’t have a reasonable hope that it is worthwhile. (And, of course, that I am competent to referee/review it.)

Sometimes I am disappointed, and say so – with as much force as seems warranted – but I never start out with the assumption that the work in question is likely to be unacceptable. That’s unfair. Besides, life is too short.

By: Sortition

Sortition — Thu, 14 Feb 2008 02:26:13 +0000

I consider it a duty of any scholar who is sending manuscripts to journals.

Must do your part to keep all those sub-standard papers from contaminating the stream of officially approved high-quality research.

By: greensmile

greensmile — Thu, 14 Feb 2008 01:26:59 +0000

I should add in light of that Open Lab, that the decent thing to do is to excuse yourself if you are either out of your league in the topic…don’t set up situations where you are the only likely beneficiary. or Likewise, the crappy feeling that you have wound up doing a disservice to the authors and the profession by committing to something you were unable to give a proper effort can be avoided by a little honesty, even if you are a name brand maven in the subject at hand. You always know somebody who would gain by the exposure and is not half bad or double booked…you always do.

By: greensmile

greensmile — Thu, 14 Feb 2008 01:19:42 +0000

a little to the left of topic but not really OT:
I was one of the “judges” for the recently edited Open Lab 2007. That experience comes in on the bleeding popular edge of refereeing papers. About 20 on line aquaintences of the editors were invited to help rate the papers. It was a daunting assignement with over 400 submissions needing to be thinned to more like 50. Some of us read nearly all, all of us read some. I can’t say I agree with the consensus [a generous term in this case] opinions that finally settled the winners but it WAS a broadening task and rewarding very much in proportion to how much effort one expended reading papers.

By: greensmile

greensmile — Thu, 14 Feb 2008 01:07:47 +0000

its like grading papers for a class except for that, you usually get paid. It takes an optimist or a masochist to look forward to a stack of other peoples untested writings with hope of consistently gaining something.

By: aa

aa — Wed, 13 Feb 2008 03:00:40 +0000

When you’re young you should referee whatever you can, when you’re old you should referee only what you don’t trust anyone else to referee properly, and as little as possible. You’ve probably got better ways to serve the discipline. If something spectacular comes your way you should probably referee it. But don’t do anything ever for, say, Gordon and Breach, and other malefactors of great wealth. And if the stuff you could have been writing is distinctly better than the stuff you could have been reading, then consider sticking to writing. If not, not.

I’ve been saying no a lot. But I tell them where to go. I mean, I tell them where they should go.

By: cdas

cdas — Wed, 13 Feb 2008 02:04:54 +0000

At the journal I edit the referee “acceptance rate” stands at about seven out of ten. When I explain the peer review process (for instance, to new colleagues), I generally say that it’s easier to find good reviewers than one might think, rather than that it’s harder than one might like.

It would be difficult to judge any long term trends, since even if it were the case that potential referees were saying no more often than was once the case, the longer an editor stays in the post, the wider their own network and knowledge of the field becomes.

One should bear in mind that some potential referees who refuse to take on certain papers do so for reasons other than that they are overstretched or feel under-obligated, such as that the paper in question has already found its way to them several times before.

Equally, some referees who accept their invitations ultimately produce mere opinions (or, for that matter, nothing at all) rather than full reviews. Indeed, in these cases, and excepting those rarest of occasions where the paper in question is simply so strong that nothing needed to be said, all concerned—referee, editor, and author—would have been better off had the referee simply refused the invitation.

Among senior people, our referee refusal rate is probably higher than three out of ten, and among junior people it falls to virtually zero.

(That said, it might be interesting to divide up the records by discipline, as I have a sense that senior philosophers say yes more often than senior political scientists, for instance.)

(Another aside: We have in the past approached more than one of the contributors to this thread for a review, and all did indeed say yes!)

By: Jamie Dreier

Jamie Dreier — Tue, 12 Feb 2008 23:51:30 +0000

Chris Hallquist,

It takes me roughly five hours to referee a paper.

(Of course, it varies a lot.) I’m not sure what you mean about checking to make sure the literature is cited. I don’t do an independent literature review—I’m supposed to know the literature, since I’m supposed to be an expert on the topic of the paper. Of course, I could be wrong! But I don’t think it’s a standard expectation on referees that they review the literature on the topic themselves to make sure the paper has cited it.

I think I typically write about 600 words in my reports. For a very good paper the report might be much shorter.

Here’s something that’s been bothering me recently. I think lots of good papers deserve a “Revise and Resubmit” verdict. But when I give one, I’m buying myself another refereeing job. This is a significant disincentive, which I try to ignore, but I’m pretty sure I fail. What can be done about this?

By: Mark van Roojen

Mark van Roojen — Tue, 12 Feb 2008 23:06:38 +0000

I’m in the ballpark with most of the philosophers above, about 1 paper per month, plus a book or two per year and sometimes a tenure case. (I’m not submitting anywhere close to this amount.) The rules of thumb have to be adjusted up for those of us who are more senior. Many of the people whose papers we referee are not yet at the career stage where they themselves are refereeing as much as they are writing. That slack needs to be taken up somewhere.

I’m not so worried about B-list referees. I’m one myself. I’ve heard various journal editors comment that the not famous but respectable folks often do the most conscientious jobs. That wouldn’t be all that surprising since we still need to submit papers for review to get published and thus we still get referee’s reports both well and badly done. If the difference is salient and you have a little bit of pride you’ll try to make the ones you do like the good ones you get.

By: Laleh

Laleh — Tue, 12 Feb 2008 21:17:16 +0000

I review about 4 times as many articles as I have in a review process (plus a couple of book manuscripts at any given time) and like Thom above, I try to turn around the article in less than a week (for books, I take a couple of weeks).The only time I turn down a refereeing request is when I don’t have expertise/competence in the particular area an article addresses.

As member of an editorial collective of a feminist journal, I find that about half the referees we approach actually agree to review an article for us.

By: Ingrid Robeyns

Ingrid Robeyns — Tue, 12 Feb 2008 18:48:19 +0000

I also agree with Tom Hurka’s suggestion that those of us who write better/more detailed reports should not feel under the same obligation to do as many. In fact, I’ve noticed that when I have several papers to read, that I am more likely to try to write reports quickly, and probably write worse reports. I tend to think of the quality of my own reports as rather variable—I know I’ve written some very useful and detailed reports, but also some that were rather poor – and that happens either when the paper is really poor or quickly written and I therefore feel under less of an obligation to do my best (the referee system is not for drafty drafts), or when I have several papers on my desk to review. If this is true and it applies to more of us, then there is another reason not to take on too many review requests, since it is likely to negatively affect the quality of the reports.

By: How much should we review? « An Ergodic Walk

How much should we review? « An Ergodic Walk — Tue, 12 Feb 2008 18:10:17 +0000

[...] 12, 2008 How much should we review? Posted by asarwate under Academia There’s an discussion going on over at Crooked Timber on how many papers one should agree to review. Most of the [...]

By: bob

bob — Tue, 12 Feb 2008 18:09:46 +0000

As a librarian, I would suggest that an additional critereon to consider would be the cost effectiveness of the journal – is the publisher’s subscription pricing to libraries set as a moderate return on costs, or is it driven by what the market will bear? Thus, is your time as a referee contributing more to the advancement of scholarship or the the bottom line of the publisher?

Some tools for assessing journal cost-effectiveness include http://www.journalprices.com/
http://www.eigenfactor.org/
http://www.wendt.wisc.edu/projects/jvp/welcome.do

By: Tom Hurka

Tom Hurka — Tue, 12 Feb 2008 17:23:28 +0000

When I edited a philosophy journal I too was pleasantly surprised by how many people agreed to referee. But I think it’s a mistake to tie refereeing responsibilities to referee reports received: many senior academics publish mostly in invited venues, where they don’t get reports at all. Yet they shouldn’t be off the hook.

And shouldn’t how many reports you ought to write depend on how thoroughly you do each one? I can’t stop myself from writing extensive comments on each paper I referee. I hope that means I can do a few fewer. (It seems to be around ten a year.)