Editors of academic journals have been reporting that they find it increasingly hard to secure referees for papers that have been submitted to their journals. When I’ve been discussing this issue over the years with colleagues, I’ve heard a few remarks that made me wonder what our considerations are to decide whether or not to accept a review request. Clearly, there must be a content-wise fit: if one thinks the paper is outside one’s area of expertise, one should not accept the referee request. But then I have heard considerations such as “I decline because I have already refereed for this journal before”, or “I referee as many papers as I receive reports”, or “I referee 5 papers a year”. Are these valid reasons to decline?
Clearly, the answer cannot be that how much we choose to referee is purely a private affair. All academics would benefit if there would not be a shortage of referees, hence it cannot be a purely private affair. Yet the referee shortage takes the structure of a collective action problem. And we know that there are two principle ways to address collective action problems – either by having a collective decision maker (such as the government), which is not a solution available for this problem; or else by way of establishing a social norm.
Solving the referee crisis in academic peer review will require multiple measures, but when it comes to securing that enough people are willing to referee, I propose to discuss the number we should treat as the lower boundary of how much we should referee. Let’s call the number of reports a person writes for journals divided by the number of reports that person receives in response to their own paper submissions a person’s referee-ratio. I want to defend that the referee ratio should be at least 1.2. In other words, for every 4 reports we receive, we should write at least 5 (adjusted for the number of authors of a paper).
Why is that number not 1, as some seem to think? The first reason is that there are a number of reports being written by authors who will not be able to [fully] reciprocate. Think of PhD-students who submit their work, but are too junior to review themselves (they may become more sufficiently experienced towards the end of their PhD-trajectory, but I think it’s a reasonable assumption that some people submit to journals who do not have the skills and expertise (yet) to serve as referees). Some of them will submit for a few years and then stop doing academic research, and will at that point no longer be part of the system of peer-review.
The second reason is that there might be authors who temporarily should not be expected to reciprocate. I am thinking in particular of editors of journals, and associate editors with significant work loads, who are doing crucial work in making the system run in the first place. But if we agree on this, then it might well be the case that the minimal referee ratio should be 1.5 rather than 1.2 – I am not sure.
The third reason is that the system needs a bit of buffer in order to function. We need some oil to make the machine run smoothly. If everyone were to agree to adopt as a social norm that we should seek to have a referee ratio of minimally 1.2, then there would be more scholars who receive a referee request who would accept because the social norm tells them they should accept.
It is actually pretty easy to calculate our own referee ratio. Many people keep track of the papers they have refereed, often as part of their annual assessment conversations with their line managers. And it’s also quite easy to keep track of (or reconstruct for the past) the number of reports we have received. It might take us a few hours, but the potential objection that this is too burdensome isn’t very strong, I’d think.
My hope is that agreeing on a minimal referee ratio would help address the referee shortage problem. But it will also address the issue that some people are, qua character, much more prone to feel guilty if they decline a referee request. I am sadly in that camp (I generally blame it on having been raised in a Catholic culture). It has led to much agonizing, and contributed to an excessive work load, which has negatively affected my health. Once, when I was close to burn-out, I had a coach who told me to protect myself by quantifying upper limits to my professional commitments, because otherwise they would crush me. In short, people who are insufficiently able to say ‘no’ might be helped if they make the calculation and see their referee ratio is not around 1 but rather way over 2.
There are two alternatives that I can think of. One is to pay referees. But there are at least three reasons against this. First, it would increase the bureaucracy and paperwork involved in refereeing. Second, it is quite unfair against a background of huge inequalities in financial resources, especially on a global scale, but even within continents and countries. Third, it would commodify another aspect of academia – is this something we should want?
There are other strategies that journals can use that are complementary. Some journals now state that one can only submit if one is also willing to review for that journal; to me that seems absolutely reasonable. But if that were our only expression of reciprocity-duties, it would be too strict; for example, given my theoretical/conceptual expertise on the capability approach, I’ve reviewed papers that used that framework for a number of journals from other disciplines (such as Social Sciences and Medicine). So I think we should, to some extent, also be willing to review for journals that we will most likely never submit to, because it helps colleagues in other disciplines.
It might be that the number of 1.2 is not the right one. Perhaps it should be 1.5, or even 2. But the prior question is whether we agree there should be such a number that functions as a professional social norm. Or is there a better way to solve the referee crisis?
{ 36 comments }
Alex SL 12.05.24 at 1:17 pm
Agreed, but 1.2 is nowhere near enough. A rule of thumb among my immediate colleagues is to do at least 2x as many reports as one publishes manuscripts, but even that would only work under perfect circumstances. Manuscripts generally need two referees, but some need three or, in a recent case for me, four, and some get rejected or withdrawn.
IMO, legitimate reasons to reject review requests are: lack of expertise; manuscript is so obviously bad that it should have been desk-rejected; conflict of interest; predatory garbage journal that cannot be trusted to actually make good use of the report; or acute lack of time, with expression of willingness to review for the same journal again when one has time.
I am actually kind of puzzled why it has become so difficult to find reviewers. Do colleagues not understand the problem, getting annoyed at their manuscripts not finding reviewers while themselves rejecting all requests to review? Is it plain laziness or laziness masquerading as a principled stance against for-profit journals, which is only legitimate if one never tries to publish in them oneself?
Payment has two other problem in addition to the ones you mention – where is the money coming from? Either from the readers (not open access) or from the authors (problem for people from poor institutions and/or countries, whose careers are then harmed by being unable to publish). Ideally, it would all be publicly funded, but ideally, I’d be a billionaire, so… The second problem, as I keep pointing out, is that we get a salary already, and it would be absurd in the eyes of the broader public to be paid extra for something that is expected to part of our jobs. That is not a thing that happens elsewhere, e.g., a nurse isn’t paid an extra fee every time they give an injection, it is just part of the job even if a pharma company profits from the activity.
Ingrid Robeyns 12.05.24 at 3:40 pm
Alex – the referee ratio = the number of reports you write/the number of reports you receive. So if you submit one single-authored paper and receive 3 reports, you owe the system 3 times 1,2 = 4 (rounded up) reports. My sense is that, to the extent that people impose such a number on them right now, it tends to be 1. And my argument is that 1 is too low. But for me it’s fine to have a social norm that the referee ration should be 1.5 or even 2. If we were to all then stick to it, it should solve the problem (at least, in theory).
On your list of legitimate reasons to decline a request: I am not sure what the difference is between “acute lack of time” and “lack of time”. In my opinion the problem is that refereeing very often belong to the category of things that get done in overtime (evenings, weekends). Since academics have very different rules in what they are willing to do outside office hours, I think “lack of time” shouldn’t be included since on one interpretation it applies to almost everyone, and on another to no-one; instead, I would argue that “having minimally done one’s fair share” is a legitimate reason to turn down requests.
Sashas 12.05.24 at 6:02 pm
The computer science education journal TOCE is experimenting with mandating a ratio. I don’t remember where they got the idea from, but they didn’t originate it. So enforcement “from above” is at least in the short term possible. I see no reason to believe it won’t be sustainable.
John Q 12.05.24 at 6:51 pm
There was one economics journal where I was never successful, despite submitting papers on disparate topics which eventually got accepted elsewhere, at reasonably good places. The long-standing editor would regularly send me articles for review, noting how ideally qualified I was as a referee. I mentally drafted, but didn’t send the obvious response, and finally stopped both submitting and refereeing there.
That’s an extreme case, but the problem of creating and deploying a norm of reciprocity in fields like economics with >90% rejection rates is going to be difficult, I think. If, as seems reasonable, editors ask for reports from authors they’ve previously published, the implied ratio has to be way above 1.
SusanC 12.05.24 at 7:04 pm
I’d consider a reasonable ratio to be at least 10.
“Conflict of interest” or “not my area” sound like good reasons to decline to review a paper. Also, if you think the journal is too low quality to submit to yourself, seems reasonable to also not review. (Though I can be persuaded by arguments from the editor along the lines of “we’d really like to start a conference in Slovenia (or wherever) and need internationally respected experts to review the submissions from the local academics”)
Alex SL 12.05.24 at 7:47 pm
Ingrid Robeyns,
Sorry, I completely misread what you meant with the ratio. Too late in the evening when I replied, perhaps.
What I meant with acute lack of time is to differentiate it from a general lack of responsibility. Meaning, I can understand rejection if a request comes in when they are already reviewing two others, or if somebody gets asked two weeks before an important grant deadline or during exam grading, but not so much if it is a general attitude of not finding reviewing as important as other things one could do instead with one’s, yes, always limited time.
John Q,
I have not had the experience at the same level, but I have had journals desk-reject me and then immediately ask me to review. I created an account with their editorial management system, after all, so now I come up on keyword searches when the editor tries to find referees! Yes, to me that seems like a potentially legitimate reason to say no, and all the more so if rejections are repeated and seemingly unthinking. Conversely, if a journal published one of my papers, I am more than happy to review at least twice for them in the coming months. I actually feel that I get fewer review requests in my core area of expertise than I should (but too many where I have to reject for not being qualified).
Matt 12.05.24 at 9:01 pm
I referee a alot more than I submit. Over the last 10+ years, I’ve refereed a minimum of 10 papers and more often about 15 per year. (I have a couple I need to crank out as soon as I’m done marking exams.) I know some people who do significantly more than that. It’s pretty rare that I submit more than two papers to peer-reviewed journals per year. But what interests me is that there are quite a few good people who report being asked rarely or never. The Daily Nous blog tried to set up a spreed sheet for people who rarely or never get asked to volunteer, but, perhaps ironically, Google seemes to have taken it down. That was here: https://dailynous.com/2024/11/12/philosophers-available-to-referee/
(Lately, the cases where I’ve said “no” involve ones where I recommended rejecting an article, but it’s sent back for an R&R – in one case more than once – where it’s clear that the editors and other referees liked the article a lot more than I did. In those cases, I’ve felt like it was reasonable to let someone else have a fresh look at the pieces, even if it slows things down a bit.)
Maxhgns 12.06.24 at 12:14 am
It seems to me that a big part of the problem is that we all have to wait to be contacted by a journal before we can referee. Which helps explain why the same people are tapped regularly, while others are never asked; you have to be a person who comes to the editor’s mind. It also means that we can’t control when requests come in, meaning we may well have to decline to balance our own workloads.
I’d like to see a journal that allows prospective referees to volunteer for particular papers at particular times. Kind of like Dialectica’s fish pond, but for the referee pool.
Ingrid Robeyns 12.06.24 at 8:27 am
Matt and Maxhgns – thanks! – you point at something I implicitly took for granted (and wrongly so!), which is that I assumed that everyone is asked to do more than they can chew. It’s only under those circumstances that it becomes a collective action problem. But you might be right that a large group of people might perhaps simply never or hardly ever asked (I’ve never been in that situation, perhaps that explains why I didn’t think of this group).
Perhaps then there are two things to discuss:
(1) establishing a social norm on how much to referee, if one is asked;
(2) a strategy for making sure more people are asked to referee.
As to (2) – one thing that might work is for each person who is asked too often, to actively construct and maintain a list of other scholars, of whom one might suspect they do not get asked so often (but how does one know??), to pass on as names when one declines. The risk of simply giving two or three names when the journal prompts us to do so, is that we will come up with names of people that are ‘dominant’ in our heads, which might again be people who are overasked.
Ingrid Robeyns 12.06.24 at 8:36 am
I just read the Daily Nous post, which is great (and I hope Justin gets the google doc up soon). I also agree with the commenter who mentioned that this is really something where PhilPapers could play a useful role.
I still think though that we need both issues – since I also think those of us who get asked 20+ times a year should know when we can, justifiably, decline.
SusanC @5 proposes a referee ration of 10. So if in a given year you submit one paper, that gets 2 reports and a rejection from journal A, and then 3 reports and an R&R from journal B, then you need to do 5 times 10 = 50 reports in that year? That seems crazy to me.
mengel 12.06.24 at 9:42 am
I generally do not review for for-profit publishers. I am simply not willing to donate my time and experience so that publishers can sustain their absurd profits. I am happy if they have problems finding reviewers. Frankly, let them all go bankrupt.
Contrary to what Alex SL suggests, this is not “plain laziness or laziness masquerading as a principled stance against for-profit journals”: in my field, most reasonable venues have left for-profit publishers by now (I have contributed to this move by pushing one venue that I am involved in), so I have only had a single paper with a for-profit publisher since 2021. In that case, my co-authors wanted to publish there, and I promptly “paid back” by reviewing a paper for that journal. I have published 6 papers in 2024 (a lot for me) with roughly 3 reviews each, so the overall number of reviews is 18. Normalized by the number of authors, I am at ~5.43 reviews that others wrote for me. On the other hand, I have written 26 reviews this year (pretty normal year), so my ratio is ~1.44 unnormalized and ~4.79 normalized. So I think that I am not lazy.
I think the expected ratio should also take into account academic age. For young people this was already discussed. But on the other end of the spectrum, there are often senior community members who have other very heavy duties (editors, deans, work for funding agencies, academic politics, …). So I guess most of the work has to be done by the “middle-aged” crowd that should in my opinion aim for a higher ration than 2.
Alex SL 12.06.24 at 1:41 pm
mengel,
Just to be clear, my preferred system would be one where all journals are run as diamond open access public utilities.
That being said, when somebody submits a manuscript to a journal published by a for-profit publishers, I do not review the manuscript “so that publishers can sustain their absurd profits”. I review it so that there is quality control in the scientific literature, and out of gratitude for others reviewing mine. The question of under what economic model the paper is type-set and printed is completely and utterly orthogonal to that. Reviewing for each other is part of the expectation when we draw a salary as a researcher, regardless of the nature of the publishers, and it is not as if this is arms dealing or the manufacture of land mines.
As I implied in an earlier comment, just imagine for a moment the bizarreness of taking the same approach you outline in any other job where people receive a salary in compensation for fulfilling their duties:
A police officer refuses to drive the police car because petrol companies are a quasi-cartel who coordinate price increases.
A nurse refuses to hand out medication to a patient because a pharma company profits from that.
A teacher absolutely refuses to mark their students’ exams unless the paper and the red pen were sourced from a non-profit manufacturer owned by its staff as a cooperative.
An electrician refuses to fix your air circulation because running it would mean increasing the revenue of a privatised electricity supplier.
A customer refuses to read your emails unless you can plausibly promise that none of them are ever sent from a computer running iOS or Windows; only Linux will do. And, of course, the customer expects you not to earn anything from your interactions either, because no profit!
When considering the suggestion of paid peer review, the previous sentences can have added to their ends “and/or unless paid an extra fee for each action of that kind on top of their salary“. This attitude of not wanting to do a central part of one’s job because somebody else involved in the process is a for-profit entity would be an utter absurdity to anybody outside of academia. For once they would actually have good reason in that case to speak of out of touch liberal elites, if not to use terms like ‘lost their minds’ and ‘what are they smoking?’.
Yes, by all means try to convert all economic activity into public ownership without a profit motive. I will help. That would be my preferred arrangement! I consider the profit motive toxic, and organising everything into markets and competition to be corrosive to our welfare and future! But I will still do my duty to the scientific community and to the job that the tax payers can reasonably expect me to do even while we have not organised ourselves that way. That’s what being a grown-up means.
Michael Kates 12.06.24 at 3:42 pm
Like Matt, I also review far more papers than I submit; indeed, I’m working on two book projects at the moment and so have not submitted a paper to a journal in quite some time (despite the fact that I regularly review 10-15 papers per year).
However, one reason why I decline referee requests, which has not yet been discussed here, is that I have already (negatively) reviewed the paper at a different journal. This has now happened to me about 3-4 different times in the past couple of years. I suspect the problem is that there are not that many people working in my particular area, and so editors keep asking the same people to review those types of papers. It’s hard for me to see what the solution to this problem is.
Ellen Simms 12.06.24 at 5:56 pm
Michael,
Re-reviewing a manuscript should be considered part of the job. When I receive the same paper submitted to another journal, I briefly compare the first and recent versions. If they haven’t changed, I simply say that in my review, which usually elicits a prompt reject from the editor. If they have changed, then I will evaluate the extent to and quality with which they responded to suggestions from me and other reviewers and include that information in my review.
Nuance also arises, of course, due to differing expectations of the two journals. I might find a manuscript unacceptable for a large general field journal such as Ecology or Journal of Ecology, but find it perfectly fitted to a more specialized journal such as Western Society of Naturalists or Madrono.
Jeremy Fox 12.06.24 at 7:22 pm
Years ago, a colleague and I compiled some data on the referee ratios of authors submitting to several of the leading journals in our field (ecology). The take-home was that there was a lot of variation. Many people were well above the minimum referee ratio (for any plausible choice of an acceptable minimum), and many others were well below. We also found that individuals who submit more papers to these journals also tend to do more reviews for these journals.
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0092896
notGoodenough 12.06.24 at 8:04 pm
I’d be fascinated to hear more about how refereeing is part of your job and the assessment protocols in place – e.g. how much does it contribute to your performance assessment, what targets there are in place for it, what consequences for meeting/not meeting those targets are, etc.?
Personally, I’ve yet to encounter any university or research institute which does consider refereeing part of one’s job (let alone a central one), though to be clear I absolutely think it should be. Yet despite this general feeling (or should be), everywhere I’ve looked has not actually counted it as a work activity in any formal sense (and it is generally expected to be something done “in your own time”).
I would be more supportive of a minimum referee quota if it were instituted at the workplace level (which would, perhaps, also help reduce the demand to increase publishing all the time). Otherwise, it seems to me to just be placing the burden on the individual…
Nathan Lillie 12.06.24 at 9:22 pm
I review papers if I am interested in the topic and want to know what people are up to in that area. Pretty often, the answer is not much, so I give a few hints of what they should do and recommend a rejection. But there is a lot value in reviewing on the cutting edge and seeing where the field is headed, even if you have to kiss a few frogs enroute. I might not be able to guess who it is, but if it is good stuff, I keep and eye out and track that person’s work in the future. This usually leads to me accepting more articles than I probably should for review, in terms of time, but actually I think its worth it overall, in the end.
Beyond that, I never review for predatory journals, or any journal that gets close to being predatory. Also, I stick to what I know. I review if I owe the editor a favour, or if I’ve published in the journal recently.
Most of the journals in my area seems to have come under the shadow of for-profit publishers and in an ideal world we would all just refuse to review for them for free – they are making a profit, so should we. But given that everybody else in my academic community is also caught in this ridiculous system and there is no prospect of extricating ourselves, it seems mean spirited to refuse to help out.
Matt 12.06.24 at 9:58 pm
I’d be fascinated to hear more about how refereeing is part of your job and the assessment protocols in place – e.g. how much does it contribute to your performance assessment, what targets there are in place for it, what consequences for meeting/not meeting those targets are, etc.?
My job is officially 40% teaching, 40% research, and 20% service. The “service” part is both internal (being on some committees and the like) and “service to the profession”. Refereeing falls under “service to the profession”. At our tedious end of the year review, we are supposed to report everything we do. (Because this is Australia, and Australia loves bean-counting, we get “points” for different things, and different levels of career advancement are suppose to have increasing number of “points”. This is supposed to make things “objective.” We get “points” for having done peer reviews of journal articles and book proposals. That I’ve done more than average is taken as a good thing on its own (I got “points”!) but also because it (supposedly) shows some regard in the field.
Could I get away with doing fewer referee reports? Yes, I’m sure I could. But then, I could get away with doing a poor job, or freeloading, on many parts of my job. Ideally, we should not need, and not want, to have a stick ready to beat us to make us do our fair share of our work.
Michael Kates 12.06.24 at 11:23 pm
Thanks for the response, Ellen.
I totally understand where you’re coming from, but I’m very weary of re-reviewing a paper because I don’t think I have a monopoly on good judgment of work in my area and I’m reluctant to exercise so much influence on someone else’s publishing record (especially if they might be a junior scholar). I’ve also been on the receiving end of this situation, and I found it very frustrating that the same person kept tanking my paper.
Alex SL 12.06.24 at 11:32 pm
notGoodenough,
This isn’t about quantitative KPIs in my performance assessment, because at any rate I have very little of that. Our performance assessments are more strategic and broad strokes than that. Think “lead project XYZ, publish its results” instead of “publish at least three papers this year”. So, the logic is as follows:
It is an integral part of my performance expectations to publish my research in the form of peer-reviewed papers.
If nobody does peer review, I cannot achieve that expectation, and nor can others in the same role as I am.
Therefore, to be able to achieve our performance expectations, we all have to contribute to peer review for each other’s sake.
This is one of those things that are so obvious that they don’t have to be written down in the performance agreement, like being respectful to somebody who sends a public query to you or not messing up somebody else’s bench in the lab; it is a social contract and basic to being able to function at all. Of course, some can hide behind the “not for Elsevier” card, as long as they don’t then publish with Elsevier, and everybody gets to decide when they have time to review and which ones they should accept and which ones they can decline.
But if somebody free-loads, their colleagues will eventually take note.
notGoodenough 12.07.24 at 11:31 am
Alex SL and Matt
I infer you are assuming I am taking a position which, in fact, I do not take. For the purposes of clarifying, let me try (once again) to simplify:
There are many things which should be “part of the job” (e.g. teaching, marking, writing papers, refereeing, engaging in community outreach, and so on). These are things that we should generally do – not only because it is our work, but also because it is generally beneficial to society. On this I think we are broadly in agreement.
My point is that there are people (and I was one) for whom things which should be part of the job are not in fact seen as part of the job, and instead are counted as “things you should do on your own time”. While I am delighted that others have the experience that such things are indeed counted as work, again, for some people it definitely isn’t (when I was a postdoc, I was “caught” refereeing a paper during working hours at a university, and was informed in no uncertain terms I should only do so in my personal time – and that if I was unhappy with this arrangement, I could find a new position at my earliest convenience). Personally I think social contracts only really work if you are not actively punishing people for upholding them – but it seems you disagree.
As I noted in my first comment, I think that it would be good to have minimum refereeing quotas at an institutional level, and to require institutions to count it as work activities (which it seems for some people it does). I am merely suggesting it might benefit the conversation if you consider that not everyone has your experience.
If you think I am being unreasonable, then I would ask that you simulate the working conditions I have experienced – which is to say, carry out your job as though only paper writing, teaching in the class, and lab work count as actual work activities (i.e. all teaching preparation, refereeing, writing grant proposals, etc. must be done in non-working hours on pain of demotion and/or losing your job). My personal experience of this was that it was unsustainable (and one of the reasons I no longer work in academia), but I’d be fascinated for your feedback after a reasonable period of time (let’s say a month). Since apparently you believe this should be trivially easy, I am sure you will have no issues with these conditions at all.
Plarry 12.07.24 at 6:39 pm
I don’t think that a number like this can function as a professional social norm in many academic disciplines. The reason is that many types of reviewing occur, and they all take different amounts of time and effort. For example, there are reviews of grants, which in my discipline is typically more intensive than reviewing a manuscript, and there are reviews of people, i.e., tenure and promotion letters, which are the most intensive of all. Then there are the service loads of associate editor, editor-in-chief, program chair of a conference, and so on, which are the glue that holds peer review together.
Alex SL 12.07.24 at 9:16 pm
notGoodenough,
Yes, seems like we are indeed talking past each other. Because, admittedly, I did not consider things specifically from the perspective of a postdoc under an exploitative and abusive employer but only from a more senior career stage like my own, and conversely, in contrast to what you took from the conversation, nobody here has claimed that finding the time to peer review is “trivially easy”.
Matt 12.08.24 at 3:05 am
notGoodenough – it sounds like you’re in a positon that’s very much different from that discussed in the post, or that I’ve encountered in working in philosophy, law, or a business school (or have heard from from people I know in many other disciplines in the humanities and social sciences.) If that’s so, then it’s likely that different standards might apply, but also it would be the case that your unuusal situation wouldn’t throw much light on the situation discussed in post, either.
John Q 12.08.24 at 5:37 am
I generally decline to do a second review of a paper for which I’ve already recommended rejection. I can’t exactly say why, but in a field with 90 per cent rejection rate, the risk of a Type II error isn’t great.
notGoodenough 12.08.24 at 8:57 am
Alex SL
Thank you for the gracious reply – I apologise for getting a little testy in my previous reply, and appreciate your giving consideration to broader implications.
Matt
While I’m quite sure you are correct in what you are describing, I fear that the situation I have described is not that unusual (I know of several people at different universities which had far worse experiences than myself). This isn’t, of course, to say that this is a universal situation – and in many places what has been discussed here makes sense. I would merely suggest that some institutions can be quite exploitative, and that it is worth considering the broader implications of proposals. However, as you find this somewhat derailing, then I will leave it there.
M Caswell 12.08.24 at 1:52 pm
My ratio has been infinite for years.
KT2 12.14.24 at 11:46 pm
IR; “Are these valid reasons to decline?”
Dorothy Bishop’s are, but it was published anyway. How do you put a ratio on this?
“The most interesting part of the story is that the publisher went through all these steps of reviewing and revising. If they just want to make money by publishing crap, why bother engaging outside reviewers at all?”
Posted on December 14, 2024 9:17 AM by Andrew [Gelman]
“Dorothy Bishop shares this hilarious/horrible story from neurosurgery researcher René Aquarius:
…
“And, the unsurprising conclusion:
(Dorothy Bishop:) “Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it.”
“The most interesting part of the story is that the publisher went through all these steps of reviewing and revising.
“If they just want to make money by publishing crap, why bother engaging outside reviewers at all? I’m not really sure.
…
https://statmodeling.stat.columbia.edu/2024/12/14/the-most-interesting-part-of-the-story-is-that-the-publisher-went-through-all-these-steps-of-reviewing-and-revising-if-they-just-want-to-make-money-by-publishing-crap-why-bother-engaging-outside-re/
John Q 12.15.24 at 1:51 am
Journal editors must be paying attention to CT, I think. My ratio was below one for most of this year, but I just got three requests inside a week.
Ingrid Robeyns 12.15.24 at 8:45 am
Yes, John, at least some of them do, since one of them wrote to me telling a story of how famous scholars who tough they no longer had to referee, was given an answer that was roughly along the lines of telling them they should have a referee ratio of above 1. But also saying that editors should do many more desk-rejections, so that there would be less need to rely on referees, which is indeed another way to address the referee crisis (but makes the job of editor heavier, since they do not just have to skim, but instead carefully read the paper).
SusanC 12.15.24 at 3:24 pm
It’s an interesting question whether you should refuse to re-review a resubmitted revised version of something you”ve refereed before
Obviously, if the first time you rejected with “needs to fix X, Y and Z” and the resubmission is to the same venue, its ok for the prevision reviewer to read the neww version and check and X, Y and Z have indeed been fixed.
slightly murkier if the resubmission is to a different venue, and you just got sent it again by chance co-incidence [for which read, the only expert in the world on the specific topic of the paper is you, so of course every editor is going to ask you to review it] Should you refuse, on the grounds of fairness; that is, if each resubmit is reviewed by different people, the paper gets seen by a more representative sample of reviewers, which lessens the chance of an unfair rejection due to the biases of a particular reviewer [like, you’re the main author the paper cites, and the main point of the paper is that you’re wrong]
SusanC 12.15.24 at 3:30 pm
I think my personal record for re-viewing the same paper submitted to different places is four times. Recommended accept on the fourth time, due to some combination of the paper was getting noticeable better each time (like, they were addressing the comments made by the reviewers) and they’d chosen to submit it to a second-tier venue where it was facing less competition
SusanC 12.15.24 at 6:55 pm
I thought the idea behind desk rejection was that a large proportion of submissions are obviously rubbish, and can be rejected quickly without needing to read them closely.
There’s a question over what the social convention is for how much explanation the editor needs to give for the rejection,
Have to admit, I have once been sent a paper to review that I privately suggested to the editor that we give a sudden death desk rejection, but editor insisted on a minimal referees report that at least laid out (a) why this idea is not original (cite a source where this has been described before (b) why this does not work (cite the source that explained why the previous publication of this idea doesn’t work. Now, of course, had the authors done a minimal literature search they would have discovered all this, and saved everyone some time.
Alex SL 12.15.24 at 8:41 pm
Ingrid Robeyns,
Well, one would bloody well hope so. Being an editor of a prestigious journal is good to have on one’s CV (it is a criterion for promotion to senior levels at my institution, for starters), so editors who want to have that on their CV should also be willing to do the work that comes with it.
I have got the feeling that editors have, on average, become lazier since I started my career. I used to get more thoughtful instructions about what exactly I need to do to make the manuscript acceptable. These days, most editors I interact with could be replaced with two scripts:
One to pick reviewers from a database entirely on the basis of keyword matches but without any thought for actual expertise. (As in, I frequently receive invitations to review where I would have no understanding of the methodology. Once, I was invited to waste my and the authors’ time by reviewing a paper that was shown to be out of scope for the journal by skimming the abstract. Also, my current boss was once invited to review her own manuscript.)
A second to forward the reviewer reports to the author with some kind of simplistic “if both are major revision or worse, reject, if at least one of them is minor revision, request revision” logic but entirely without any comment on which of two completely contradictory reports should be followed, and also entirely without filtering of abusive or incompetent reports.
IMO, if they can be replaced by those two scripts (which even I could hack together in one day), then they shouldn’t be allowed to claim being editors on their CVs.
John Q 12.19.24 at 7:43 am
Elsevier’s editorial manager screwed up the login process for two of the three articles I agreed to review. I emailed the editors directly with my comments.
From now on, I’m going to short-circuit this. If I get a review request from an Elsevier journal, I’ll decline, and note in the “reasons” section that the editor is welcome to send me the paper directly, rather than using this appalling system, which just makes referees’ lives harder. I guess it saves work for editors, and for Elsevier, but I don’t feel any obligation to either.
John Q 12.19.24 at 8:57 am
I’ll correct myself on the claim that Editorial Manager saves work for editors. One of the editors who received my direct report, responded saying
“Sorry about Editorial Manager – I have to deal with this pig of a program every day. “
Comments on this entry are closed.