When to bury an academic paper?

I am useless at this. I hate rejection (but then, who doesn’t?) and tend to greet even the most constructive revise-and-resubmit letter as a Mene Tekel on my entire career. I got a rather bracingly written, “shape up, soldier, you’ll have to do better than this!” kind of R+R a bit back, and after reading it I couldn’t look at the email, or the paper, for three days.

I find it less troubling when Reviewer 2 clearly just wants you to have written a different paper – although even then the devil on my shoulder is saying “that’s because they want you to have written a professional paper…“. Two or three different Reviewer 2s wanting two or three different different papers… that’s a tough one. If people are citing it, though, there’s obviously something useful there.

I once contributed to a seminar series at the university where I worked, then had a paper based on my contribution rejected by the journal that was publishing a special issue based on the seminars, then had the same paper rejected for the anthology that colleagues were putting together from the same seminars, and then lost my job. Which would probably have happened if the paper had been published, to be fair – it was an everyday story of precarity – but it was a bad conjunction of events (and may explain why I take rejection letters badly!). That paper is well and truly buried.

Since then, though, my thing has been burying half-papers; on several occasions I’ve written anywhere from 50% to 150% too many words and had to say goodbye to great slabs of text. I always tell myself I’ll go back and make something of them, but never do.

I don’t have a sophisticated decision procedure. My habits can be discerned after the fact by looking at notes:

I have a paper forthcoming that was rejected by six journals before finding a home. Some of those were desk rejections on account of fit, but for incompatible reasons. Another paper from a few years ago was rejected four times and went through three rounds of revising and resubmitting at the fifth journal before acceptance. That’s approaching my limit, I guess, because I buried a paper after rejection from eight journals.

That’s a tough one, and you have my deepest sympathies. Congratulations on powering through the revisions and resubmitting.

The advice I always got was “You should publish a lot, and some of it should be good.” Not very uplifting, but probably ruthlessly realistic. So my first inclination would be to never bury it. Just do whatever you have to do to get it published, including rewriting, splitting, recombining, whatever.

My postdoc advisor was incredibly unsentimental about papers and viewed our group as a paper factory that was supposed to pump them out. Figure out what the paper is about, generate 4 or 5 figures, then write the text around it. If it wasn’t accepted, rewrite it and submit it somewhere else. He always had a bunch of papers in various stages of production and had a deep Zen lack of attachment to any one. He was also really good at putting out papers and I learned a lot from him. A different field of course, with I’m sure very different standards on paper length and number of papers per year, but I felt a useful emotional attachment none the less.

Given the astonishing CV arms race going on, I think burying papers, or even letting them sit for an extended period of time, is a luxury reserved for those with job security.

I usually read the comments carefully, sit with them for a week, and then revise for a week without looking again at them. I resubmit until someone into my kind of thing referees it. Every essay I have published has been rejected multiple times. Usually it is around 2-5, but in one case it took 10 or 11. Rejections/acceptances do not in my experience track journal reputation or, really, anything other than whether the referee is antecedently keen on my approach.

One thing I’ve learned is that if I get rejected after an R&R, I need to check whether the changes I made to appease the original referee are worth keeping. Often, the objections I get on subsequent submissions are to additions included to answer a previous referee’s questions.

Another important thing I’ve learned is to use footnotes/citations to try to steer an essay towards the right referees.

Luckily I think I never had to try more than three journals so far.

In my eyes when things like these happen it is often a failure of editors to do their job. It is one thing if several referees agree on the same critical flaw in the manuscript; in that case the authors should seriously consider that the manuscript is indeed badly flawed. And that is also the only case where I would give up.

But if, as here, the referees are all contradicting each other, or one is very positive and one is very negative, or they recommend expanding the manuscript by discussion that would require another 2,000 words when you are 17 words under the journal’s limit, or they clearly have no idea what they are even talking about*, I would say it is the editor’s job to critically assess those reports and give clear, actionable recommendations to the authors.

“I have carefully read the reports and will consider acceptance if the following five points can be addressed: … Please note I disagree with referees 2 and 4, and you can ignore their suggestions.” But these days quite a few editors at least in my field seem to find it sufficient to dump mutually incompatible recommendations in the authors’ laps, accompanied merely by a generic R&R or “requires revision” statement. Have fun figuring my criteria for acceptance out for yourself, I’m off! If that is all the editor is willing to do they can easily be replaced with an algorithm, and they should reconsider listing a prestigious editorial position on their CV. /rant. At any rate, this does not indicate the manuscript deserves to be buried.

*) Anecdote: I was once asked to turn a grid map of Australia with some cells marked red and some cells marked blue into a Venn diagram, and I still have absolutely no idea what the referee thought that would look like. They were also clearly confused about the difference between Bayesian and maximum likelihood analyses, but the editor reflexively rejected the manuscript because the report was so negative. The next, equally selective, journal I tried accepted the paper after minor changes to the text, largely, I think, because the editor chose referees who understood the research question and the methodology.

I should note that, in economics and (I think) philosophy, what counts in the eyes of many potential employers is neither the quantity nor the quality of work, but the average ranking of the journal. If you have a moderate number of articles in the Top 5 econ journals (there is near-complete consensus on which these are), adding more articles in top field journals, or very good (but not Top 5) general journals will harm rather than help your career prospects. So, you could argue for burying articles as soon you have exhausted the outlets that would raise your average.
BTW, a quick look at my CV or Google Scholar will confirm that I don’t follow this advice, and I’ve done fine in terms of my career. But I’ve stayed in Australia, and even here I’m getting some pushback.

I think that simply throwing publications against the wall is counterproductive in the ways you indicate, but I, at least, cannot write more than one or maybe two essays a year which I am willing to submit. I’m in the middle of a post-doc, and I applied selectively this year. One of the faculty members at my PhD institute questioned whether I should use ESSAY IN TOP GENERAL JOURNAL as a writing sample since it was published ‘a while ago’, which, in fact, was May 2018. (He later agreed that concern was unreasonable and he had thought the publication date was earier, but it will probably be reasonable next year.) So I at least feel great pressure to have things coming out consistently.

Philosophy is also I think a bit different from economics in at least four ways.

(1) For many sub-disciplines, a publication in the top field journal will help as much as, if not more than, a publication in a top general journal. This is obviously so with philosophy of science and philosophy of the special sciences journals, but it is generally agreed, at least among specialists, as concerns ethics, aesthetics, political philosophy, and so on, from my experience.

(2) I think just about everyone at this point recognizes that with an acceptance rate between 3-5% at top journals, there are way more essays of the appropriate quality than there are available slots. This is hammered home by two of the consensus top 5 generalist journals being closed for submissions for a large part of the year because of the volume of submissions.

(3) Philosophy is a very fractured field. People with hiring power often have very different ideas of what constitutes significance or importance than those with journal power. This is especially true for those eminent enough to have stopped submitting to journals long ago.

(3) Less likely to be admitted out loud, but I think more people are recognizing that philosophy often functions like a gift economy and journal publications are no exception. Given that no editor is an expert in most topics and will often choose referees from their social circle or from the works cited in the essay in question, referee cartels form around certain topics and journals by happenstance and by design. What topics get published in a journal, and what types of approaches to those topics, is then often a function of those limited referee pools.

“What’s wrong with the employers?”

They have learned that anyone can publish anything somewhere, with enough shamelessness, as amply illustrated by the comments above. Since mere publication reflects lack of self-restraint rather than quality of scholarship, the employers need to use other metrics than mere publication.

” Is there any chance it can be fixed?”

It will be a long process to undo the mess, but there’s some hope. Incentives will need to change all around in order for publication to more nearly reflect merit. One thing that might help is for every article to be published with a complete list of the journals from which it was previously rejected. This would be less punitive than charging economic damages to authors who flagrantly abuse the time and energies of dozens of referees.

How can people cite a paper that has not been published? Submissions are confidential; referees and editors are bound to secrecy with other peoples work. I work in physics, not philosophy, but didn’t expect such differences in the manner in which submissions are treated.

Christopher, presumably they’re citing an earlier instantiation of it (like a conference presentation), but Ingrid please clarify.

Harry, great story, thanks for sharing!

Sent me scurrying off to my records. My personal record seems to be 7 (that is, accepted on the 7th try). I was surprised to see how many papers I have buried well before that. I recently got a paper rejected for the third time, and decided to bury it. When papers are either very specialized or (like this one) quirky, they are right for only a few venues.

As always in discussions like these I am astounded by how different the approaches in different fields of academica are.

“adding more articles … will harm rather than help your career prospects” – What J-D said. Okay, I have heard of colleagues who think that publishing in non-academic venues can hurt one’s standing (e.g. an ichtyologist writing an outreach paper in a fishkeepers’ society newsletter), but how can publishing more papers in top journals be a negative? That incentivises lack of productivity.

“those eminent enough to have stopped submitting to journals long ago” – What do they receive a salary for if they do not publish new work? Or does that merely mean they write books instead of journal papers?

(Other differences between fields that can be very confusing are authorship attributions and authorship order.)

I don’t think the norms of the economics profession are good ones, and have ignored them myself, but it’s hard to advise junior colleagues to follow my example.

Since mere publication reflects lack of self-restraint rather than quality of scholarship, the employers need to use other metrics than mere publication.

Sure, I get that. That makes sense to me. For example, an employer might ask a potential employee ‘Which are your best papers?’ and judge on the quality of those, not on total quantity of publications. I’m sure there are other alternative metrics as well, and probably the best plan would be to use several. But if I understood correctly the situation John Quiggin was describing was one in which employers are using quantity of publications as a metric, only with reversed sign–that is, if a candidate added more publications, they assessed that candidate less favourably. Maybe I misunderstood the description–I am not an academic, I have no experience from any angle with the academic labour market except for hearing stories about it (and not much even of that), I am only judging by what I am reading, and I could easily have misinterpreted that.

What I’m saying is that if you have a situation where there are two candidates for an academic job, one with a small number of extremely good publications and nothing else, and another with a similar number of publications of similar quality and also additional publications of little merit, then it does not make sense to assess the second candidate as inferior to the first on the basis of the additional low-quality publications. An employer might not want to treat the second candidate as superior on the basis of that additional bulk of low-quality publications, but I can’t figure how it makes sense to treate the second candidate as inferior on that basis, and I thought (perhaps mistakenly) that was the situation John Quiggin was describing.

“adding more articles … will harm rather than help your career prospects” – What J-D said. Okay, I have heard of colleagues who think that publishing in non-academic venues can hurt one’s standing (e.g. an ichtyologist writing an outreach paper in a fishkeepers’ society newsletter), but how can publishing more papers in top journals be a negative? That incentivises lack of productivity.

But why should publishing in a non-academic venue be considered a negative? I can understand how it might not be considered a positive, but how can it be a negative? If you’ve got two candidates for academic jobs in, for example, a department of literature, with comparable academic records, but with one of them (and only one of them) also being a published poet, I don’t know enough to say that the poet must be the better choice, but how could being a published poet be a reason for rejecting the candidate? Taking a more topical example, if somebody has had published their advice for the general public about pandemic precautions, how could that possibly make them a less suitable candidate for an academic job in a department of public health?

@alfredlordbleep

I’m not aware of any philosophy journal that doesn’t require anonymous submissions. I review in both cognitive science and in philosophy. In cog sci, submissions are usually single anonymous (I know who they are, but not vice versa). I really dislike it. In philosophy, it is rare that I know who the author is (and I decline the invitation if I do).

JDF, oldster,

I see, thanks. That happens in my area too, but not to the degree that a top researcher would or could ever stop submitting non-invited papers. The difference may be, I guess, that the topics of research papers are more narrow and specific, so being invited to contribute to a special issue on speciation is great but no good for the results you have in your drawer from a grant-funded study on the correlates of shifts in mutation rates. And you need to publish those to show to the funding agency that you can deliver results for their money, otherwise you won’t get the next grant.

I am curious about Bradley’s comment, “It would be unethical for the same referee to review the same paper submitted to a different journal”.

I’ve been in that situation twice— one I sent back saying “the thing that made me recommend rejection last time still hasn’t been fixed”. The other, for an article that didn’t have anything wrong with it, but was just uninteresting to me, I declined to review. (Though it WAS tempting to recycle my previous review.)

What does everyone else do?

Emeritus now, still asked to review occasionally, but not so much that I get the same thing twice from different journals.

But back in the day it used to happen pretty often, and my strict policy was to decline the second time (and the third, éventuellement). No young scholar should suffer twice from my judgement, which is highly fallible.

And isn’t epistemic humility the theme of this thread? There are some guys — and it’s mostly guys — who are so sure of the importance of their opinions, so sure of their infallibility, that they insist on their right to be heard. For them, “no” does not mean no, it simply means that they will be more insistent, more demanding, no matter how clear the message has been.

Conversely, there are some folks — and it’s often women — who are more easily discouraged and daunted, more easily shouted down. For them, the encouragement to try again seems salutary.

It’s a pity that there’s no good way to buck up the bashful without appearing to condone the shameless.

oldster,

That is certainly true when it is a matter of opinion, or perhaps of different schools of thought colliding. But there are also cases of critical flaws in experimental design or analysis, where the authors are quite simply wrong to move from journal to journal until they are lucky enough to find lazy reviewers and editors who overlook the problem.

I have one paper, from early in my career (more than twenty years ago now) that I gave up on after the fifth. The referees’ comments had started to become directly contradictory – not from a single journal, where I could have tried to argue my way through by setting one against the other, but from one journal to the next (and come to think of it, I don’t recall getting more than one report from any journal; different times…). By the end, I had changed the paper so much – trying to respond to criticisms that I didn’t really accept – that it seemed to lose a lot of what it was supposed to be about. I always thought that, if I ever attained the sort of eminence where you get to published a Collected/Selected Papers volume, I would sneak it in there – but I fear I have now lost the earlier versions that I actually liked.

The really annoying thing is that the first journal I submitted it to had been willing to publish, if I would add just one paragraph. But it was a paragraph that completely missed the point of the whole paper; in retrospect, I don’t think this would have weakened it to any significant degree, but at the time it felt like an unacceptable compromise, and I took the piece elsewhere – without success…