As many readers of this blog, I frequently receive requests from academic journals to referee papers. Sometimes refereeing a paper creates benefits for the referee (like reading an interesting argument or getting inspiration for a new project), but on balance I find referee work a burden. Still, I do a lot of it (I think), since I consider it a duty of any scholar who is sending manuscripts to journals.
How much should we referee? If I were to accept all referee requests that I get, I would hardly be able to do any research myself. So I want to find out how many papers I should referee before I have fulfilled my professional duty. In the last months, I talked to some (international) colleagues about how much they referee and how they decide whether to accept or reject a referee request, and I’ve discovered that some of them don’t find it difficult at all to refuse to referee virtually all the requests they get. Not me: I feel bad every time I turn down an editor (but I’m getting better at it!). Surely there is some sort of collective action problem here, since the system can only be sustained if enough people do referee; so I feel anyone who wants to be part of this system (that is, who submits papers to refereed journals), should feel a professional duty to referee. I think one should referee at least the same number of papers as the number of reports one receives; so if in the last 12 months you’ve received 10 reports, you should referee at least 10 papers in the same period (if asked and if you feel competent to referee them, of course). I’ve been told that this rule was once suggested at a meeting of editors at the APSA meetings – and it makes perfect sense to me. Perhaps we should add 10% or 20% as a margin, since there will be people who submit papers but are not yet being asked to referee, as they are not known by journal editors as potential referees.
Since we have several journal editors among our readers, I’d like to ask: how difficult is it these days to find (good) referees? And if you’ve been in the business for some time: is it getting easier or harder to find good referees? And to anyone who feels like commenting: what do you think of the above rule to decide when we’ve done our fair share of refereeing — any better proposals?
{ 1 trackback }
{ 26 comments }
Jacob T. Levy 02.12.08 at 1:31 pm
I say no whenever I have three already on my plate and not done. Since I don’t tend to let article manuscripts sit very long, that doesn’t happen too often just with them. But I find it hard to treat article refereeing as wholly distinct from other kinds (book manuscripts, promotion files, grants and fellowships), and will sometimes say no to an article if I’ve got a pile of the other sort. It averages out above the benchmark Ingrid describes, but still lets me not get overwhelmed with reports.
Some journal editors deliberately ask the authors of just-accepted articles to referee, to drive home the reciprocity point (or take advantage of the lingering feeling of gratitude). It’s pretty effective. While the timing’s never worked to force the question for me, I’m pretty sure I’d waive my no-more-than-three-undone rule under those circumstances.
That said, one report written per report received is a lot, and junior people should certainly consider themselves to have moral permission not to do that many. We can aim for lifetime balance without aiming for balance across a 12-month timespan. And I’m not sure an author’s karmic debt is increased because an editor chose to go out to 5 referees instead of 2, or by a multiple-rounds R&R.
Anyone who, over the medium term, writes twice as many reports as he or she submits articles is probably fulfilling the basic duties of reciprocity.
Thom Brooks 02.12.08 at 1:42 pm
As a referee, I accept all requests (about 8-10 per year) unless I lack competence in an area. In addition, I always try to referee in one week or less (as I can’t stand having papers pile up around me). This in addition to reading all submissions to the Journal of Moral Philosophy that I edit.
As a journal editor, I almost never have trouble finding great referees. This is because philosophers are generally so supportive and helpful. More than 90% of everyone I ask accepts. Needless to say, their help —plus the help of the editorial board— makes all the difference in the world.
Aeon J. Skoble 02.12.08 at 1:53 pm
As a journal editor, my experiences trying to get referees are much like those of Thom Brooks upthread, pretty much everyone says yes. As a referee myself, there’s no formula or rule I hold myself to, I say yes unless I’m too busy to get it done within 2 weeks or so. As a general principle, I do agree that scholars have some (vague) obligation to participate in the peer review process, although Jacob is right that junior faculty might want to prioritize their own work.
Eszter 02.12.08 at 2:01 pm
I was going to post about this exact question myself! As you said, I could not do any of my own research if I said yes to all requests. I think it’s especially tricky for those of us who do interdisciplinary work and are known in several areas. I get requests from a wide range of journals. In 2007, it was around 30-40 (a quick count suggests 35, I’m pretty good at organizing my email and I just looked so this is not an exaggerated number).
Like Jacob, if I have three on my desk already, I say no. Last year I was also editing a special issue (those articles not included in the above figure) so that helped me say no easier.
Some editors send you a request while one of your own papers is being considered by the journal. I don’t think I’ve ever said no in such a case. If a journal is not very close to my area then I’m more likely to say no easily.
If I’m on the editorial board of a journal then I’ll take all requests that I consider a good match. Otherwise, I won’t take more than one review from a journal per calendar year and explicitly say that my rule is to do no more than one for journals where I’m not on the board.
I think the idea of doing as many reviews as the number of reports you’ll be getting sounds like a reasonable approach. I should follow that rule.
wissen 02.12.08 at 2:05 pm
If Ingrid’s international colleagues are correct then one could infer that many articles are being refereed instead by the ‘B-list’ of academics, i.e. not the first-choice expert in the field. Which would surely be worrying in terms of the standard of peer-reviewed research. From my own viewpoint I wouldn’t be surprised if time constraints were impacting negatively on the peer review process, and thus on the standard of published knowledge.
Jamie Dreier 02.12.08 at 2:11 pm
I referee about as much as Thom Brooks, and probably a lot less than Jacob. I do 8+ per year, and I’m also an editor of JESP (jesp.org, an on-line journal). We haven’t found it to be terribly difficult to get good referees, but we haven’t had quite such an easy time as Thom or Aeon — I’d say we get a little over half of the refs we ask.
Jacob’s right that it makes sense to count up ‘advising to the profession’ service as a whole, I think, so that if you’re doing four tenure reviews this year you should feel free to excuse yourself from unwanted journal refereeing. I’d add, though, that if you’re lucky enough to have a cushy tenured job, it seems fair to do a whole lot more than the input = output ‘fair share’ by Ingrid’s formula.
franck 02.12.08 at 2:11 pm
It’s simple for me. I don’t referee for journals I don’t respect. I had a devil of a time publishing a paper in an Elsevier journal once, and consequently they keep asking me to referee for them, and I keep refusing them, because they don’t even use email – they keep sending things to me by airmail!
Ingrid Robeyns 02.12.08 at 2:16 pm
I’m glad to read that it’s not as bad as I had feared – perhaps these first three reactions for editors are not representative but I had feared that it was much harder to find referees.
I agree with Jacob’s modifications – book reviewing or tenure or grant applications are in the same basket of ‘professional service’ where some sort of reciprocity rules shoudl apply. And I also agree that ideally we would have to apply the ‘formula’ (if we apply it at all, that is) to our professional lifetimes, but I for one wouldn’t trust my memory over such a timespan.
olderwoman 02.12.08 at 4:08 pm
I have been getting an average of 3-4 requests to review per month. I’ve been saying yes to 1/3 to 1/2 of them and refusing to promise fast turnaround on any of them. This is becoming overwhelming. I have the feeling that I am on everyone’s hit list. There must be people out there not doing their share. But maybe they won’t out themselves?
Dan Nexon 02.12.08 at 4:51 pm
My limited experience in assisting journals find reviewers is that the status of the journal matters a great deal. Scholars seem much more willing to review for the “top” journals than for new or B-list journals.
But my concern pivots less on finding qualified reviewers than on the number of quality reviews. Too many scholars don’t seem willing to treat manuscripts with care and respect, e.g., to make sure that they get the argument right, to explain why the omission of particular citations actually matters for the article, to recognize that a paragraph-long review is often an insufficient engagement with a 10K+ word article, to actually supply references for literatures the reviewer considers important for the author to engage with, to recognize the difference between “disagreement” and “this article doesn’t deserve to be published, and so forth.
It should shock us that, in disciplines that use peer-reviewed output as a proxy for quality, a significant percentage of reviewers don’t do “due diligence,” but it doesn’t. In my own discipline, international relations, I sometimes wonder if we’re not engaged in a kind of “organized hypocrisy” in which everyone will say how stochastic the review process is, but we don’t really adapt to that fact in our behavior.
I write all this as someone who has done reasonably well in this process. Most of what I write that, in retrospect, was publication-worthy has eventually found a home. Indeed, some of my most depressed moments come when I read other reviews of work that I’ve peer reviewed.
Arthur Stinchcombe argued, quite some time ago, that as the importance of peer-reviewed journal articles increased the quality of peer review would decrease. IIRC, the number of manuscripts will outstrip the ability of qualified and quality peer reviewers to handle the volume, leading to decreasing quality of decisions. So, at the risk of setting off academic rants, I’d like to ask commentators whether they feel that, in their disciplines, peer review functions effectively.
Most of the studies I’ve read in sociology and the natural sciences produce dispiriting conclusions: high rates of reviewers missing elementary errors, no-better-than-random outcomes from peer review, and so on.
Guru 02.12.08 at 4:53 pm
Recently, in a letter to Science, William F Perrin argued that for every accepted paper, you should review at least four:
Thus, the number one should referee depends on the number of papers you publish, the acceptance rate at the journals in which you publish, and the number of reviews needed for a paper to be published in those journals.
Chris Hallquist 02.12.08 at 5:19 pm
I’m an undergrad philosophy student looking at becoming a professional philosopher, and I’m curious to know how much work goes into refereeing a paper. Particularly, how much time is spent making sure they’re use of cited literature (especially stuff they’re criticizing) is reasonable, and how much time is spent fact-checking scientific claims. As I begin to study the professional literature more and more, these are two things that make me look at a lot of papers and wonder if they should have really have been published. The second one is especially big–I’m a double-major in a science, and a lot of philosophers just don’t seem to put enough work into getting the science right.
Tom Hurka 02.12.08 at 5:23 pm
When I edited a philosophy journal I too was pleasantly surprised by how many people agreed to referee. But I think it’s a mistake to tie refereeing responsibilities to referee reports received: many senior academics publish mostly in invited venues, where they don’t get reports at all. Yet they shouldn’t be off the hook.
And shouldn’t how many reports you ought to write depend on how thoroughly you do each one? I can’t stop myself from writing extensive comments on each paper I referee. I hope that means I can do a few fewer. (It seems to be around ten a year.)
bob 02.12.08 at 6:09 pm
As a librarian, I would suggest that an additional critereon to consider would be the cost effectiveness of the journal – is the publisher’s subscription pricing to libraries set as a moderate return on costs, or is it driven by what the market will bear? Thus, is your time as a referee contributing more to the advancement of scholarship or the the bottom line of the publisher?
Some tools for assessing journal cost-effectiveness include http://www.journalprices.com/
http://www.eigenfactor.org/
http://www.wendt.wisc.edu/projects/jvp/welcome.do
Ingrid Robeyns 02.12.08 at 6:48 pm
I also agree with Tom Hurka’s suggestion that those of us who write better/more detailed reports should not feel under the same obligation to do as many. In fact, I’ve noticed that when I have several papers to read, that I am more likely to try to write reports quickly, and probably write worse reports. I tend to think of the quality of my own reports as rather variable — I know I’ve written some very useful and detailed reports, but also some that were rather poor – and that happens either when the paper is really poor or quickly written and I therefore feel under less of an obligation to do my best (the referee system is not for drafty drafts), or when I have several papers on my desk to review. If this is true and it applies to more of us, then there is another reason not to take on too many review requests, since it is likely to negatively affect the quality of the reports.
Laleh 02.12.08 at 9:17 pm
I review about 4 times as many articles as I have in a review process (plus a couple of book manuscripts at any given time) and like Thom above, I try to turn around the article in less than a week (for books, I take a couple of weeks).The only time I turn down a refereeing request is when I don’t have expertise/competence in the particular area an article addresses.
As member of an editorial collective of a feminist journal, I find that about half the referees we approach actually agree to review an article for us.
Mark van Roojen 02.12.08 at 11:06 pm
I’m in the ballpark with most of the philosophers above, about 1 paper per month, plus a book or two per year and sometimes a tenure case. (I’m not submitting anywhere close to this amount.) The rules of thumb have to be adjusted up for those of us who are more senior. Many of the people whose papers we referee are not yet at the career stage where they themselves are refereeing as much as they are writing. That slack needs to be taken up somewhere.
I’m not so worried about B-list referees. I’m one myself. I’ve heard various journal editors comment that the not famous but respectable folks often do the most conscientious jobs. That wouldn’t be all that surprising since we still need to submit papers for review to get published and thus we still get referee’s reports both well and badly done. If the difference is salient and you have a little bit of pride you’ll try to make the ones you do like the good ones you get.
Jamie Dreier 02.12.08 at 11:51 pm
Chris Hallquist,
It takes me roughly five hours to referee a paper.
(Of course, it varies a lot.) I’m not sure what you mean about checking to make sure the literature is cited. I don’t do an independent literature review — I’m supposed to know the literature, since I’m supposed to be an expert on the topic of the paper. Of course, I could be wrong! But I don’t think it’s a standard expectation on referees that they review the literature on the topic themselves to make sure the paper has cited it.
I think I typically write about 600 words in my reports. For a very good paper the report might be much shorter.
Here’s something that’s been bothering me recently. I think lots of good papers deserve a “Revise and Resubmit” verdict. But when I give one, I’m buying myself another refereeing job. This is a significant disincentive, which I try to ignore, but I’m pretty sure I fail. What can be done about this?
cdas 02.13.08 at 2:04 am
At the journal I edit the referee “acceptance rate” stands at about seven out of ten. When I explain the peer review process (for instance, to new colleagues), I generally say that it’s easier to find good reviewers than one might think, rather than that it’s harder than one might like.
It would be difficult to judge any long term trends, since even if it were the case that potential referees were saying no more often than was once the case, the longer an editor stays in the post, the wider their own network and knowledge of the field becomes.
One should bear in mind that some potential referees who refuse to take on certain papers do so for reasons other than that they are overstretched or feel under-obligated, such as that the paper in question has already found its way to them several times before.
Equally, some referees who accept their invitations ultimately produce mere opinions (or, for that matter, nothing at all) rather than full reviews. Indeed, in these cases, and excepting those rarest of occasions where the paper in question is simply so strong that nothing needed to be said, all concerned — referee, editor, and author — would have been better off had the referee simply refused the invitation.
Among senior people, our referee refusal rate is probably higher than three out of ten, and among junior people it falls to virtually zero.
(That said, it might be interesting to divide up the records by discipline, as I have a sense that senior philosophers say yes more often than senior political scientists, for instance.)
(Another aside: We have in the past approached more than one of the contributors to this thread for a review, and all did indeed say yes!)
aa 02.13.08 at 3:00 am
When you’re young you should referee whatever you can, when you’re old you should referee only what you don’t trust anyone else to referee properly, and as little as possible. You’ve probably got better ways to serve the discipline. If something spectacular comes your way you should probably referee it. But don’t do anything ever for, say, Gordon and Breach, and other malefactors of great wealth. And if the stuff you could have been writing is distinctly better than the stuff you could have been reading, then consider sticking to writing. If not, not.
I’ve been saying no a lot. But I tell them where to go. I mean, I tell them where they should go.
greensmile 02.14.08 at 1:07 am
its like grading papers for a class except for that, you usually get paid. It takes an optimist or a masochist to look forward to a stack of other peoples untested writings with hope of consistently gaining something.
greensmile 02.14.08 at 1:19 am
a little to the left of topic but not really OT:
I was one of the “judges” for the recently edited Open Lab 2007. That experience comes in on the bleeding popular edge of refereeing papers. About 20 on line aquaintences of the editors were invited to help rate the papers. It was a daunting assignement with over 400 submissions needing to be thinned to more like 50. Some of us read nearly all, all of us read some. I can’t say I agree with the consensus [a generous term in this case] opinions that finally settled the winners but it WAS a broadening task and rewarding very much in proportion to how much effort one expended reading papers.
greensmile 02.14.08 at 1:26 am
I should add in light of that Open Lab, that the decent thing to do is to excuse yourself if you are either out of your league in the topic…don’t set up situations where you are the only likely beneficiary. -or- Likewise, the crappy feeling that you have wound up doing a disservice to the authors and the profession by committing to something you were unable to give a proper effort can be avoided by a little honesty, even if you are a name brand maven in the subject at hand. You always know somebody who would gain by the exposure and is not half bad or double booked…you always do.
Sortition 02.14.08 at 2:26 am
Must do your part to keep all those sub-standard papers from contaminating the stream of officially approved high-quality research.
dr ngo 02.14.08 at 6:20 am
Not directly on the question of numbers, but perhaps a germane general consideration:
I’ve always taken it as a general proposition that I ought not to agree to referee an article or a book manuscript, or, for that matter, to review a book, if I don’t have a reasonable hope that it is worthwhile. (And, of course, that I am competent to referee/review it.)
Sometimes I am disappointed, and say so – with as much force as seems warranted – but I never start out with the assumption that the work in question is likely to be unacceptable. That’s unfair. Besides, life is too short.
Joe Hoover 02.15.08 at 1:38 pm
As a graduate student working as an editor for a student run IR journal, meaning with limited experience, I find it is fairly easy to get reviewers. I did think it was worth mentioning that one of the best things that a reviewer can do is to say no quickly and if possible suggest others, this is especially true for those who are well known experts in the field and may have information on lesser known but excellent scholars. Waiting for a response is much worse than having a quick no.
Also, I would think a commitment to fewer reviews of a higher quality is probably better. Though, I must express my gratitude to all those who do offer substantive and timely reviews.
Comments on this entry are closed.