In response to my post about whether journal article reviewers’ identities should remain confidential (most commenters seem to hold the view that they should), Jeremy Fox mentioned a phenomenon of which I was not aware: people posting on their Web sites – once the paper has been published – reviews their papers had received during the refereeing process. Here is an example. Just to be clear, these tend to concern anonymous reviews so the author is not breaching confidentiality (unless the reviewer offers feedback in a way that outs the person’s identity – see more on that below). If the paper had been rejected from a journal, the file could include the reviews from those other journals as well (see examples here).
This is an interesting idea. As Terry McGlynn notes, there are numerous potential benefits to such practice. It would certainly make the publishing process more transparent (imagine the value to graduate students, but also others). It might lead to a more civil tone for reviewer comments (for more on the unnecessarily harsh and dismissive tone of some referee reports, see this recent piece in the Chronicle). That is, while reviews would continue to be anonymous, it may encourage editors to keep a closer eye on how reviewers are communicating their feedback when it comes to style. After all, would you want to be known as the editor of the journal whose reviewers tend to be jerks? (I guess some may actually take pride in that, but I would like to think people recognize the difference between giving constructive criticism and simply being a prick.) It might also discourage reviewers from using the referee report as a PR machine for their own work and agenda (i.e., you haven’t cited my work from 1999, my work from 2001, my work from 2010, 2011 and 2012 nor have you realized that my work is the only relevant angle on this topic).
I also wonder if such practice would encourage more thought before people submit papers for review. I realize that the last thing some people need is more hurdles, but others treat the process too nonchalantly. Some people openly admit to using reviews as just one step in the revision process often having no expectation of a paper being accepted to a particular journal, but submitting it nonetheless for the feedback they hope to receive. This is a huge drain on scarce reviewer resources. If reviews were available after publication then perhaps people would think twice about sending out half-baked ideas and unpolished drafts.
I realize this idea may be riskier at the junior level than for more senior academics. Most graduate students and junior faculty are already sufficiently anxious about the promotion process that adding anything else that may contribute even a tint of stress is probably unwise. On the other hand, the transparency could have some utility relevant especially in their case. If a paper has several low-quality reviews attached to it from journals that rejected the piece then readers (hiring committees, promotion letter writers) could see that it was not the scholar’s fault that a particular piece took quite some time to publish.
Of course, there would be little way to monitor if anyone’s actually posting all reviews a paper had received. One could easily hold back information about submissions to journals from which the paper had been rejected. (Then again, if someone who had reviewed the paper for such an omitted journal came across the shared reports, that person would know.) But even reviews from the journal where the paper was finally published could be helpful.
Arguments against? More arguments for? More examples of people doing this? Anyone in the social sciences?
{ 16 comments }
christian_h 07.30.13 at 3:03 am
I don’t know about this. I’m not a jerk when reviewing, but I also do not spend too much time on polished formulations or spell checking, for example; I do not write for public consumption. I fear publishing reviews would make them more time-consuming, making review times – which are already excessive in my field (mathematics) – even longer.
js. 07.30.13 at 3:52 am
I don’t know about this either. I mean, if someone wants to put up my anonymous report on their website, props to them I guess, but I would really not like this to become accepted practice. This is already something I’m doing pretty much for the sake of the discipline when I already have a thousand other things to do (not a famous senior academic, me!). I’m never mean (and I write obsessively good copy) but I write my reports with the understanding that they’re not for public consumption.
Also, I don’t at all see why it would make submissions more polished. After all, the author could simply use the report for feedback and not publish it. You somewhat address this, but there’s just no pressure in that direction, for the author I mean.
BT 07.30.13 at 6:15 am
I think the only viable alternatives are the two extreme positions: either enforced complete disclosure of everything, with no anonymity for any party at any stage whatsoever; or the process should be completely secretive. Any compromise position between these two extremes opens up the possibility for various abuses, as where highlighted in the comments of the previous post.
It would be curious to see which position other commentators would prefer. Personally, I like full disclosure given the chance, so would take the former, but am perfectly satisfied with the latter position as well.
praisegod barebones 07.30.13 at 6:29 am
This is an interesting issue – I found the examples of posted reports from Ecology and Biology journals illuminating (I’m a philosopher.) One reason why I think the practice would be a good one is that it would make publicly visible any differences between reviewing standards/conventions between different disciplines (or parts of the same discipline).
One prominent philosophy journal – Mind – has an explicit policy that authors should not discuss the contents of referee’s reports with any third party. I’ve never seen a good reason for this (and it seems to be something which makes abusive forms of refereeing easier to get away with.)
I think that the practice suggested in the Main Post might lead to some improvements in the standard of refereeing (especially as it’s currently one of the things we do as academics that we never get feedback on). But I wonder how many journals would respond by instituting a policy like that at Mind.
David 07.30.13 at 6:48 am
I’m not convinced that it’s reasonable to expect reviewers to provide pages of detailed feedback to authors. The primary purpose is supposed to be (or at any rate it used to be) to give advice to editors as to whether a piece is publishable, not to give free advice to authors, which they should be getting from colleagues. Over the years, my experience is that editors have put pressure on reviewers to provide more detailed reviews as a marketing tool; it helps attract submissions. If journals want me to act as a consultant to authors, they should be paying me for my time.
Neville Morley 07.30.13 at 8:58 am
I did once think of doing something like this, in the case of an article that had been rejected about ten times or so for more or less contradictory reasons – that is, the revisions I made in response to the reviews from one journal seemed to be precisely the things that annoyed the reviewers of the next journal. I was considering developing the piece into an analysis of these different responses as a reflection of the nature of the subject (classical conceptions of cannibalism, a topic that might well provoke extreme reactions), but then realised that I hadn’t kept copies of any of the early reviews. Never did publish the thing, partly because I ran out of likely journals and partly because I came to the conclusion, at about version 11, that version 3 had actually been the best but I no longer had a copy…
Generally I write reviews that *could* be published, simply because I tend to address them to the author at least as much as to the editors. Recently had interesting experience with Global Discourse, where practice is that reviewers, once they’ve okayed the publication of a piece that’s been revised in the light of their comments, then also publish a response to it that can either be a version of their initial review or something completely new. Did alter the dynamic, couldn’t quite define how.
Carmen de Macedo 07.30.13 at 9:14 am
https://dl.dropboxusercontent.com/u/50501346/Reviewing-the-reviewers.pdf
Phil 07.30.13 at 9:15 am
I think BT’s right – when the starting-point is that the whole process is sub rosa, it’s very tricky to make bits of it public.
I think the real problem with the idea is that it’s partly justified in terms of getting stuff out in the open on general principles, and partly in terms of the way it would modify various people’s behaviour. I’m worried about a scheme to incentivise certain people in certain ways going off at half-cock, but being hailed as a success anyway because information wants to be free. (Not that anything like this has ever happened before.)
I’ve had two anonymous reviews where the reviewer left their name on – would they be publishable too, on the grounds that the reviewer should have known the risks? Both, incidentally, were from authorities in their fields, and on one level I’d love to post up the reviews and name them, for different reasons. (One was a rave review: great stuff, nobody else is doing this, maybe tweak it here and here, otherwise publish immediately. I’ve never met the person, but I’d like to shake their hand. The other was a grumpy four-line dismissal (“who does he think he is, obviously hasn’t done the most basic research, don’t waste my time”)… which, as I say, I’d like to publicise for different reasons. Possibly quite petty reasons, thinking about it now.)
I’m not going to name those reviewers, though, because… it’s not what you do. I think on balance I’d rather it stayed not what you do.
RSA 07.30.13 at 12:57 pm
With public anonymous reviews, I can predict what would happen in my field: An enterprising researcher would run information retrieval/unsupervised learning algorithms to match up reviews with people working in the area. It’s hard to know whether the results would be accurate or could even be verified as being accurate, but that just makes it an interesting challenge. It might turn out as an unintended consequence of the policy that any author could know who his or her nominally anonymous reviewers were, with high probability.
praisegod barebones 07.30.13 at 2:10 pm
David @ 5: ‘I’m not convinced that it’s reasonable to expect reviewers to provide pages of detailed feedback to authors.’
I’m not sure who, in this conversation, you think is expecting this. But in case it’s me, I should probably say that when I talk of ‘improving the quality’ of referee’s reports, I’m less interested in seeing them get longer, and more interestedin eliminating, say, the one paragraph report that says ‘This paper is unpublishable because it doesn’t cite X 2002′ while ignoring a 3-page discussion of X 2002 and 3 responses to it. (And yes, this does happen, and editors in my academic discipline tend to claim that they are too pressed for time to do the kind of quality control of referees’ reports that would eliminate this).
Paul J. Reber 07.30.13 at 6:19 pm
I review in Psychology (specifically Cognitive Neuroscience) and always have in mind that the review will be read by the authors and the editor — and therefore that it is essentially a public document. I would not mind any review I wrote being posted afterward. If I knew in advance, I would have slightly more worry about making mistakes in review (which is already a worry), but I think my reviewing style is ok for this. I think I’m reasonably fair but generally late (you can’t have everything).
Of note, though, I never ever sign my reviews. I have worked in areas where there were pitched ideological differences to the point there was fear that the other side would potentially do damaging things to one’s career. In those areas, junior people prefer not signing reviews and a curious consequence is that if the senior people do, it implicitly risks exposing the junior people. While many of my colleagues sign all their reviews, I don’t sign mine in order to provide some minor indirect cover for other people who need to not sign for some reason.
This second point is related to the original question of contacting the authors. There’s a minor cost in outing yourself as a reviewer and not just to you — it affects other reviewers who would prefer not to.
fgw 07.30.13 at 7:33 pm
I guess if I were Einstein being reviewed by Bohr, someone might be interested. As it is, these days you are lucky if people read more than the abstract of the published version, let alone all the previous versions that have been rejected along the way. But I guess there is enough room on the internets to store any vanity project.
More constructively: in my field EMBO Journal has adopted a transparent review process, I guess the intent is to exert quality control via peer pressure, even if anonymity for the reviewer is maintained. One benefit might be for younger scientists (especially) who can learn how the process actually plays out. I can partially out myself and say I haven’t reviewed for them during this period and I wonder how it would affect my style. Anyone have experience?
A different Marc 07.30.13 at 7:41 pm
The EMBO Journal (one of the more important journals in molecular biology/genetics/biochemistry) decided several years ago to make all review-related correspondence for papers they publish publicly available, at the discretion of the paper’s authors. Examples can be seen in the most recent issue – click on “Review Process File” for any of the original research papers.
People I know who have published there (of whom I am not one) say this seemed to make the review process possibly less contentious – no one wanted their bad behavior exposed, even anonymously.
Courtney 07.30.13 at 7:56 pm
Some of the Marketing journals have started to do this for new authors. The Journal of Consumer Research for example, an A level journal in Marketing, has started posting on their website reviews and revisions for some previously published articles. As a junior faculty member thinking about submitting some of my research to this journal, I find it very informative to read the reviews and to see how they were addressed.
Moshe Vardi 07.31.13 at 12:43 am
Authors who post reviews without permission are violating the copyright of the reviewers, who own the copyright for their reviews.
NR 08.01.13 at 1:38 am
I have an argument against.
On more than one occasion I have reviewed papers and documented what I saw as serious flaws in the author’s argument, which the author did not address in subsequent revisions and yet their paper was published anyway. (Yes, I realize this does not necessarily mean the author is dishonest or lazy. It could also be that what I saw as basic conceptual flaws really weren’t that big a deal in the considered judgement of the editor. Or it could mean that the editor is dishonest/lazy.)
Now, what happens if the author in cases like this makes public other reviewer reports on their paper, but prudently decides not to publicize a report that exposes deep conceptual flaws in their argument? Or what if they publicize an edited version of my report? How would anyone know that they were holding information back? Conceivably I could find some anonymous way of “busting” them. But this assumes I am bothering to follow up on the afterlife of my reviewer reports, and I really don’t think this should be added to anonymous reviewers’ responsibilities.
So why is this such a problem. After all, isn’t some information better than no information at all? I don’t think so. Not if there is no way for anyone to know that some information is being selectively withheld. It creates a false sense of the whole processes and of the overall tenor of anonymous peer reviewers’ reactions. Sometimes an absence of data is better—and more true—than cherry-picked data.
Is there an easy solution? Perhaps the journal itself could be in charge of organizing a system of full disclosure. But given that some fields are very small (I’m an anthropologist) and some subtopics have relatively few people working on them, I can imagine the knowledge that one’s reviewer report will be made public might have a constraining effect on what one writes.
Comments on this entry are closed.