Speeding up the academic refereeing process?

If I were asked to referee a paper within a week from receiving it, I would feel no hesitation to refuse (unless the abstract looked really, really interesting). Whereas I could not really claim to be too busy for sixty days after receiving the paper … So my guess is that editors want to have as many referees as possible. I my view it would also really help if they were not ask you to re-evaluate the paper after an R&R but leave the re-evaluation to the editor

Seconding Margo’s point about re-evaluating R&Rs. I’ve been asked to read the same paper three times (even after I already supported publication). The editors should ultimately make the decision about what to publish, and my guess is that this is partly an artifact of the big publishing org’s newish online submission systems that take discretion out of the hands of editors, and are probably creating more work rather than less.

In the biomedical sciences we are sometimes asked to return a review in a week; I am a day late this morning on one of those due to problems related to the pandemic. 2-4 weeks is normal and expected. When serving on a grant review panel we generally have 4-5 weeks to prepare for the panel meeting (now on Zoom or equivalent, which saves a lot of money), during which a typical reviewer may have 5-7 grants for which s/he is first or second reviewer (full review) and several as “reader” (impressions if there is something to add to the panel discussion). The best way to speed up the process is to pay reviewers an honorarium large enough to cover their Starbucks tab for a week…cue howls of indignation in 4,3,2,1…

Computer Science is weird in that we tend to deal in conferences rather than journals as the premier publications. All of the conferences I’ve reviewed for have given between 2 weeks and a month for reviews, and I think every period has been extended because not enough of us got our reviews back on time. I could probably return a review within a week if I only got one at a time. The latest conference gave me 4. I ended up doing one of them right away and the other three close to the deadline.

Overall, I’d say it would be great to speed up the review process. That said, it doesn’t even make the first page of my list of complaints. My top priority would be to add a mechanism for reviewers (or meta-reviewers after discussion among the reviewers) to get factual questions answered by the authors. This has been a publish-or-reject level issue for something like half of the submissions I’ve reviewed this year.

I’m a computer scientist, where we have a considerably different publishing culture. Most papers are published in conferences, where there’s a fixed timeline — a deadline for submission, and a deadline for all the decisions, usually about 4-5 months apart, with maybe 1 subsequent month for changes before the final version appears. Of course, this has its own drawbacks — with no real “revise and resubmit” option, good work can get rejected again and again, with different reviewers. A friend shared her experience of having the same paper rejected 6 times before finally being accepted.

That said, journal reviewing in CS takes longer, partially because CS journal papers are themselves longer. But even holding paper length constant, I think I would basically never accept a review request if it had a 1-week deadline.

I generally enjoy reviewing papers, and I don’t think I’ve ever actually turned down a request from a journal; it’s the sort of bounded task that can be fitted into my commute, and offers a break from my own research to think about something else briefly (so, yes, taking on reviews is also a sort of procrastination). Generally I get them done in a week or so – but if that was the specified deadline, I would be turning down lots of requests if they weren’t directly related to my interests; I can feel sure that I will find commuting time that doesn’t have to be spent on emails within a 3-4 week period, but within a week or two this can’t be guaranteed, and reviews (unless directly related) aren’t something I can prioritise for non-commuting work time within such a period.

And as an AE for a journal in a fairly niche field, I have no doubt it would make it harder to find reviewers; we have a standard six-week turnaround, and still get people asking for more time, or declining because they’re too busy. There is a particular problem, at least in this niche field, with hyper-specialisation, so that someone might rule themselves out from reviewing a paper on e.g. C18 adaptations of Greek tragedy in Austria on the grounds that their expertise is in adaptations of Greek comedy – which leaves you with a referee pool in single digits, all of whom know each other, and so have real power to dictate the terms under which they’ll take on a review. As AE I’d be quite happy with a reviewer who knows about C18 Austrian drama in general and a reviewer who knows about the reception of Greek tragedy, and so I devote a fair amount of effort to crafting invitation emails to say, yes, we DO want you to review this paper, and this is why we think you’re fully qualified to do so. And still get lots of refusals on the grounds of expertise.

I should first caveat that I can only speak from a very limited perspective, and am purely doing so with respect to my own field (materials science and energy storage), as I have no wish to try to generalise this to other areas or subjects. With that said, I will now throw my own handful of grit into the works :-)

So again, purely form personal experience, there can be considerable variation in review times – one Nature paper, for example, took over 1 year before the reviewers finished their deliberations, while conversely one in Energy and Environmental Science took about 3 weeks. I would say the latter was not an unreasonable period, but the former probably was. Many journals typically aim for 1 month, but 6 – 8 weeks is not uncommon either.

On the other hand, the perspective of reviewers should also be considered – and while I would by no means claim great expertise, I do typically review about 12 – 15 papers a year. Generally, for the journals I review for, I am expected to turn around my review to the editor in 2 weeks (which is not an exceptionally long time to read and thoroughly review the sorts of papers published in this field). I will admit I do go over the deadline from time to time, though not regularly (and I think the longest I delayed by was an additional week, mainly because I had a confluence of unexpected deadlines resulting in working rather long hours which was not conducive to making a thorough review). This is not a point of pride, but rather a function of fatigue. And though I will refuse to review if I think I cannot make a good stab at it (e.g. if I believe I have insufficient time), I will invariably receive another review request within a day or two afterwards – which suggests to me there is something of a demand for reviewers.

While certainly reviewers can be slow (and indeed, can often be so for bad reasons), I also think it a little unfair to lay the blame solely at their feet or to characterise this as always some sort of point of contrarian pride. I suspect there may be a number of reasons (some of which have no doubt been raised here before), which include the usual points of contention (again specific to my area, though perhaps some universality may exist):

1) Reviewing a paper represents quite a commitment.

I am often called upon to review papers which are 30+ pages (not including SI, Figures, ToC, etc.) very carefully and to an exacting standard, then to make (hopefully) sensible comments and suggestions. It is by no means as simple as merely providing feedback on whether or not the paper is “suitable for publication”, as I am expected to provide discussion regarding the quality of the work and its presentation, how well it fits with the journal and the field, what impact is predicted, spot any errors (from the trivial typographical to significant issues with methodology or conclusions, to actual fraud), and to provide long and detailed thoughts regarding how this work may be improved (sometimes this is easy, such as editing for clarity, and sometimes it can be more-or-less writing a brief dissertation on the topic and putting together a short research project including techniques and data analysis which should be carried out).

This is, I would suggest, hardly a negligible amount of effort.

2) All my reviewing is considered “not work”.

Any reviewing I do has to be in my free time – which, given the expected dedication of work (as previously noted), does in fact represent quite a burden. I often experience quite high levels of exhaustion, and it is not unusual for me to spend most of my evenings and weekends undertaking this endeavour (often for months at a time). And while I am prepared to do this out of a sense of comity and responsibility, I would find it hard to blame anyone who didn’t particularly feel like devoting a good 10-20 hr of unpaid labour per week purely out of a sense of duty (particularly given that, for some of m’ colleagues in academia, there are often many other things you are supposed to be doing in your “free time” – such as writing grant proposals, grading, preparing lesson plans, balancing budgets, planning activities, managing your postdocs/students, etc. etc.).

3) The ever-expanding workload.

While I have no evidence for this, I do have a suspicion that editors tend to rely on reviewers they have good experiences with (it would, I dare say, make sense to send work to someone who has a track record of fast turn around times rather than keeping searching for new people you have no data for). Whether or not this is true, it is certainly the case that I am reviewing more papers now than I used to – and one of my older colleagues has the same impression, stating that they have seen something close to a quadrupling of the number of review requests they receive). I doubt this is so much due to any increase in my personal standing within the community, and venture it is simply a function of there being many more papers to review.

There have already been long debates regarding the impact of publish-or-perish, and I won’t propose to rehash them here, but I think it worth pointing out that if researchers are measured by how many papers are produced it is not unreasonable to expect that to be a targeted outcome (you incentivise the behaviours you wish to see increase, after all). To put this simplistically, if the performance target is for every researcher to publish 5 papers, then one would expect that at a minimum (and assuming everyone shares the burden equally) every researcher would need to review 10 papers (2 reviewers per paper is generally the minimum). The more papers set as a target, the more papers to review…

TL;DR while I would by no means say that reviewers are by any means perfect (far from it!), I am unconvinced it is more due to a matter of personal failings than it is a system which demands a dedication of labour which is unrewarded and (if one considers the opportunity costs) may well be actively harmful if pursued to too great an extent. I would suggest that the system, therefore, also plays something of a part in all this.

Specific thoughts:

As a point of pedantry, I think it is not universally true that scientific journals are not purchased (IIRC Elsevier, for example, make up about 34% of the revenue for RELX, who’s total revenue was ca. $9.4 billion). In part this seems (at least to me) to have been driven by that litigious old publishing tycoon, Maxwell. Now this is not to say that paying people to review is necessarily desirable or the most productive route forward – but I am unsure that it would not be possible (however, as this risks diverting back into the weeds again about the usual hoary old topics, such as access and such, I will try to refrain from commenting further on this specific point). Instead, fundamentally speaking I am a bit skeptical that monetary rewards would help. Not only because I doubt many publishers would be happy to see their profit margins decrease (at least for those journals who are “for profit”), but also because I suspect many would not find it much of an incentive unless the sums involved were a bit more significant than “beer money” (if one is writing an EU grant, I doubt a 10 euro voucher for a journal subscription would exactly lead to a fundamental shift in priorities).

I am also rather unconvinced by the proposal that people should only be allowed to publish if they review for the journal – unless all publishers share a central database, surely the obvious outcome will be that the incentive is to review only for those journals you plan to publish in? One suspects this might well lead to rather unequal distributions of volunteers, which may well exacerbate the situation…

My own (poorly considered) modest proposal

I am unsure that nudges and incentives will lead to much progress here – I think a more drastic change to the culture is needed.

Personally, what I think might help (again, purely speaking from my personal perspective on my field) would be if the demands for meeting targets for numbers of papers were relaxed, and more of an emphasis (at the institutional and funding body levels) were placed on things like reviewing. For instance, if (just as an example, not necessarily as an actual proposal) universities were expected to set aside a certain number of work hours for their research and teaching staff to undertake reviewing, I think this would be far more effective than using sticks and/or carrots at the individual level. In short, I would say a systematic problem requires a systematic solution – all nudges are likely to do is place more burdens on people who (not infrequently) already dedicate a disproportionate amount of their lives to their work.

Indeed, I would like to see systematically more value being placed on these sorts of “community services” researchers are expected to carry out – such as reviewing, replications studies, on-going peer review (i.e. post-publication literature study), etc.

And then, perhaps, subsequently I may collect my magical flying pony and head off to the moon for a scoop of delicious cheese!

The time commitment for reviewing a paper must vary quite a lot across fields, and across papers within fields. A twenty-page paper in mathematics might involve substantial work, sentence by sentence, in verifying each step of a proof, checking that theorems from other papers imply what is claimed they do, and so on. A comparable paper in, say, machine learning or biology, might be reviewable in a quarter the time, or less. Less rigorous, more impressionistic fields, like physics, I imagine take much less reviewing time.

Fields for which the time commitment is greater likely have correspondingly greater difficulty in finding people willing to make that commitment. Finding reviewers can itself be a long process:
– struggle to find two people both knowledgeable in the relevant sub-sub-field, and without a conflict of interest;
– ask them; wait awhile for them to respond;
– ask them again;
– wait awhile for them to respond;
– give up;
– repeat the above until two people agree to review;
– wait awhile;
– ask each reviewer for the report;
– wait awhile;
– ask again;
– they respond “within two weeks”
– forget about it for a month;
– ask them again;
– no response;
– wait a week or two
– ask them again;
– they respond “it’s a very busy time; just after Halloween?”
– wait until after Thanksgiving;
– ask them again;
– no response, or the response is “based on a quick skim, I think….”;
– give up on them, start process all over again.

It’s true that there’s ways to speed this up, for example: ask four people, hoping for two. This takes more editor time, less elapsed time. Or: give up earlier on people who agree to review, but then don’t. But some people can be very persuasive: they’ll do it soon! Really they will! And hope springs eternal.

Like David Steinsaltz, I’m a mathematician, and I had difficulty parsing the indignation in the first paragraph of this post: I have only on one or two occasions been asked to review a paper in less than 60 days. Some journals ask 3 months, some 4, one journal I am quite fond of asks referees to write back within 6 months. I recently received a first report on a 35-page (so, medium-length, but fairly technical) paper just over 2 years after it was initially submitted; a companion paper (longer, equally technical) was submitted in March 2019 and I have not received referee reports. The AMS keeps records of publishing times in mathematics; e.g., here’s the report from 2019: https://www.ams.org/journals/notices/201910/rnoti-p1713.pdf

My spouse is in the biological sciences, where turnaround on the timescale of a few weeks is normal — but often referees request additional months of labor.

My sense is that nearly all of the typical delay in mathematics is in the “finding referees” and “getting reports from referees” stages (i.e., the second and fourth points above). Perhaps, in other fields, editors do more editing than they do in mathematics!

I think a good rule of thumb is: if a typical paper in your field receives n referee reports, then you should do at least n referee reports for every paper you submit to a journal. But this is a question of quantity, not speed. Ingrid, I think your goal of doing reports within a week is noble, and perhaps Neil is right that there is something to be gained by journals asking for much shorter windows (then giving, say, two extra weeks before moving on to new referees). I know of one journal in mathematics (IMRN) that prioritizes quicker reviews, but that means that one may receive from them a desk rejection for the reason that the paper probably cannot be reviewed quickly.

The above argument assumes it’s always the same 90% of authors who are getting rejected, which isn’t fully realistic.

@yes, if your paper took two submissions to accepted, you would need to referee 2n papers for things to balance out.

But the effect I was alluding to is: if you submit to a “top tier” journal, get rejected, then resubmit to a lower ranked journal and get accepted, the top tier may not consider you to be competent as a reviewer. So for things to balance out for the top tier journal, the people whose papers it did accept (and who it considers to be competent reviewers) have to do correspondingly more reviewing.