First, let me jot some thoughts about Ingrid’s peer-review post that also relate to Henry’s. Then, mongooses.
The main virtue of the peer-review system is that it’s basically uncorrupt. It’s main vice: it’s weakly accountable and, as a result, intermittently neglectful. Reviewers can’t illegitimately profit from the review process because they can’t profit from it, period. Except in the abstract sense that it’s nice to be part of a healthy field, and to feel the feeling of doing one’s duty. It seems to me there is a simple fix. Publish referee reports with the journal articles themselves. (If that takes too much paper, make the referee reports a standard e-option, at least.) Give referees the option of de-anonymizing themselves, to stand behind their words. If they prefer, they can preserve their anonymity, although their critical verdicts become public. You still hide the author’s name from the refs during the review process. (Obviously referees should also have sensible options for editing reports in light of the final product. You wouldn’t want to publish a blisteringly negative referee report, demanding fixes for problems that were, indeed, fixed in response to that very blistering report.)
Disadvantages: none. No, really. None. No increase in corruption. (Extra paper costs. So make it e-only.)
Advantages: many.
Reviewer who unmasks gets a minor publication, small line on the ol’ CV. This would need to become an acceptable, minor publication form before you could count it, of course, but no time like the present for establishing a healthy new norm. (A solid referee’s report ought to be worth the same as a ‘reply to’ or short critical note.) Being a ‘power referee’ could become a viable professional option. With academics pushed to publish, as things stand, this could balance things out. Get the pipes unclogged.
Reviewer who doesn’t unmask, perhaps because she is worried she is savaging a potentially powerful person in the field, at least gets the satisfaction of having her critique on the record. (If you feel that there are powerful people/influential views in need of getting taken down a peg, you will be in favor of this being done publicly. It would make the process of writing the referee report more satisfactory, if you expected to go this route.)
Reviewer gets first crack. If you labor to lay out a strong objection to a published piece, it’s sort of annoying if, then, someone else is first to publish the objection you already worked out in your referee report.
Reviewer will take the time to write clearly, cogently and considerately, rather than grumpily backhanding a thing she doesn’t like. Because her good name is attached. (I think referees would up their writing game even if they intended to stay anonymous, just knowing their words would be published.)
Transparency of social network is a benefit. It’s interesting to be able to see who agrees and disagrees with whom, about what. Blind peer review mostly aims at preventing self-dealing, cocooning and cronyism. But the enemy relation is just as potentially pernicious as the friend relation, and blind review gives wide scope for unaccountable expressions of intellectual hostility. In a more positive sense, the highest quality signals are given off by friends disagreeing with friends and enemies agreeing with enemies. It would be interesting, then, to know when you face such likely high-quality cases.
Brief referee reports would function like abstracts, which are nice to have. Brief, negative reports would be contrarian abstracts – a genre no one writes about their own work, so it’s a damn shame so many instances are not part of the public record. In general, it can only be helpful to have the option of availing oneself of some ready-made critical frames.
You could publish a wider variety of types of academic ‘results’. This brings us to Henry’s post about Kanazawa. Let me re-quote the editorial apologetics Henry quotes: “I happen to think it is a great thought provoking document, and one of the few in the last ten years that have actually gotten people to talk about issues. … I would rather have an article that causes people to think and talk and yes, argue and criticize than to publish an article that is one more facet of the same old thing.” This is not a deplorable sentiment, in the general scheme of things. But there is a serious problem with publishing work that aspires to qualify under the ‘get people to talk’ standard in the self-same sober-sided manner that you publish stuff that isn’t wild, speculative, bold, blue-sky provocation; that purports to establish results. If you published a piece along with a referee report that says ‘this is not just wrong but irresponsible and, frankly, plain nuts’ and a referee report that says ‘this is wild stuff, but worth talking about’ then you have more truth in advertising. The author of the piece in question will have a slightly harder time pointing to a publication refereed in such a manner as evidence of a ‘solid result’. If the issue really is worth talking about, then the talk can proceed. (There is an ambiguity in our attitudes toward academic ‘results’ hereabouts. On the one hand, we think you should only publish if it’s really solid. On the other hand, we think anything apparently solid could turn out not to be, and probably will, to some degree, at some point. Even solid results are conversation starters, rather than stoppers. But that’s not to say you should consciously set out to publish conversation starters – although you could. That could be a style of journal. Contributors could muse, publicly, in what they take to be a bold and provocative manner, about issues in the field, without pretending to nail anything down. Reviewers could be asked to judge contributions according to this rather vague metric.)
This isn’t a fix for the Kanazawa case. I have no personal opinion about that, above and beyond the distinct, second-hand impression I am coming away with that he is a very seriously bad actor. What I am suggesting wouldn’t make such a case any worse. And it might actually make it somewhat less bad. He seems to have submitted, been rejected, and then just re-submitted elsewhere until he eventually got let in through neglect or editorially non-explicit application of some ‘hey, it’ll stir the pot’ principle. Referees’ serious concerns about plain incompetence were not part of the publication record, and certainly weren’t permanently afixed to the pieces in question. If it’s going to be published eventually anyway, it might as well be tagged as being the kind of thing it probably is. (Obviously saying so is consistent with saying it shouldn’t be published at all.)
Now, mongooses. No lesser a personage than Nakul himself showed up in comments to recommend a mongoose at least as interesting as any jewel-spewing one. Namely, a half-golden one. Click link for truly first rate oh-SNAP! trash-talking from the mongoose in question: “‘Ye kings, this great sacrifice is not equal to a prastha of powdered barley given away by a liberal Brahmana of Kurukshetra who was observing the Unccha vow.” If the mongoose were to execute the right sort of uh-uh-UH! side-to-side head movement, while throwing down in this exemplary fashion, truly the virtue of the ritual occasion would be unexceeded. (Also, it just goes to show. Damn liberals.)
Also, more than ever, we need a Kirby-cracklin splash page to honor the mongoose in question. A Hulk-style origin scene. “huge … explosion … of … virtue! Radiation of … goodness … battering my very being! I am becoming … HALF-GOLD!’ (Meanwhile, priests monitoring the scene can see it all. ‘My Vishnu! There was some sort of creature within the blast radius!’) Then the half-gold mongoose goes around, fighting crime, always looking for a ‘cure’. That is, something to turn the rest of him gold. Fighting criminals and demons that want to melt him or eat him.
There is an interesting philosophy of charity implicit in the tale in question. It make the valid point that a lot of ‘charitable giving’ is, basically, a consumption good, for what it’s worth. The mongoose’s philosophy is not consequentialist enough, not by half. But I think Matthew Yglesias would probably be willing to concede that, if some liberal were to give his very last prastha of powdered barley to the Harvard alumni fund, conditional on Harvard changing the name of its sports teams to the Half-Golden Mongooses – ye and verily, that they might, in every subsequent athletic match, serve as an object lesson in the nature of virtue – that this would be not just morally permissible but even an act of charity aimed at ‘social justice’. Even a consequentialist might concede as much.
(post tweaked for clarity, or perhaps just out of a shallow sense of my own cleverness)
{ 51 comments }
Daniel Butt 06.04.11 at 8:50 am
“Disadvantages: none. No, really. None. No increase in corruption”
Well – I don’t know. It seems to me that there’s a possible networking benefit to be had in writing a positive review for someone who will then see your name attached to the review. That’s potentially true in cases where people genuinely have no idea whom they’re reviewing, but it’s particularly true in the not infrequent case where the reviewer does know full well who wrote the article – either because it’s obvious, or because they’ve tried to find out. So I think you introduce a new reason to write a favourable review that’s unrelated to the quality of the paper, and that’s at least *an* argument to go in the “con” column.
John Holbo 06.04.11 at 8:56 am
“It seems to me that there’s a possible networking benefit to be had in writing a positive review for someone who will then see your name attached to the review. ”
That’s probably true. So, ok. One minor addition to the negative column. Of course, if someone really wanted to curry favor in this way they could always just write an email to the author in question, on the side, saying ‘I was the writer of the favorable, anonymous referee report. I was happy to see your very good paper published.’ That could be a perfectly acceptable, friendly thing to do, I suppose (although I’ve never done it myself, as a referee.) Or it could be an attempt to logroll.
John Holbo 06.04.11 at 8:59 am
Adding to that thought, it’s probably easier to suck-up, in a winning manner, in a private email, then in a public, critical notice. If such is your goal. If your referee report says ‘this is the most awesome piece published in philosophy of language in the last ten year’, you will look like a fawning fooling. If you wrote that in an email, you might make a friend.
Daniel Butt 06.04.11 at 9:10 am
Although writing a positive review doesn’t just perform the function of sucking up – it also is potentially doing someone a very substantial favour.
John Holbo 06.04.11 at 9:43 am
“Although writing a positive review doesn’t just perform the function of sucking up – it also is potentially doing someone a very substantial favour.”
True, but the real value is in publication itself. For which purpose a private favorable report is as good. Also, we take it for granted that book reviewing is a perfectly legitimate scholarly activity, fraught with all the networking risks you mention. But point taken.
Bill Benzon 06.04.11 at 9:47 am
I don’t see how public reviewers’ comments does much in the case of an article that’s rejected for reasons that don’t make any sense. Since the article’s not published, the bone-headed rejection isn’t published either.
John Holbo 06.04.11 at 9:54 am
“Since the article’s not published, the bone-headed rejection isn’t published either.”
But the bone-headed referee doesn’t know it won’t be published at the time of writing the bone-headed review. If you are worried your bone-headedness might be published, you may feel obliged to try to do a bit better. Obviously this only addresses people who write referee reports that make no sense out of negligence, not those who write referee reports that make no sense because they are incapable of making sense, no matter how hard they try to fulfill their professional duties.
dominic 06.04.11 at 10:29 am
Your reform would indeed lead to more painstaking, thorough, reports. That sounds like more work. So fewer people will agree to review, and reviewers are in short supply already. I don’t think the incentives suffice to offset the burden of the extra work. After all, how much CV credit does a short discussion piece get you?
Tom Hurka 06.04.11 at 10:45 am
What dominic said. This proposal would cause many people to refuse to referee, seriously worsening the problem journals have in finding referees and significantly engthening the already overlong time it takes to get papers refereed.
And wouldn’t the editors also have to publish reports? After all, it’s the editor who makes the accept/reject decision, often on grounds different from those the referees gave. For example, an editor make decide to accept a paper that has a negative referee’s report because he thinks the referee is wrong. Why then publish the referee’s report that didn’t affect the selection but not the editor’s opinion that did? Or the editor may reject a paper for reasons not mentioned by any referee.
John Holbo 06.04.11 at 10:49 am
“fewer people will agree to review, and reviewers are in short supply already.”
This is a good point. I guess I think we ought to be able to get to the point where the CV compensation is adequate to motivate people.
Cosma Shalizi 06.04.11 at 1:24 pm
What about the people who write good reports on papers that are not published at that journal? No mini-publication or CV line for them.
John Holbo 06.04.11 at 1:37 pm
“What about the people who write good reports on papers that are not published at that journal? No mini-publication or CV line for them.”
True. But then: sometimes you write a perfectly good paper and can’t get it published either. Hopefully it’s not such a total effort sink. After all, most referee reports will be op-ed sized, not article sized.
Adam Kotsko 06.04.11 at 1:39 pm
I think someone really needs to found The Journal of Bold Conjecture — maybe the same people who do The Journal of Spurious Correlations?
Jonathan Mayhew 06.04.11 at 2:20 pm
It’s an intriguing suggestion. I think I’d be tempted to be even more over-bearing than I already am in my reports, to show off and push my own agenda rather than selflessly helping the author improve her own work. Also, in the case of a revise and resubmit, I can catch embarrassing errors or minor details pre-publication that have no business being aired in public. I would essentially make two reports, one for private consumption to point out things too trivial, distracting, or embarrassing for the readers of the journal to read. The other, more of a formal response worthy of publication.
Fr. 06.04.11 at 3:16 pm
Wouldn’t that system lead to a lot of emulation between reviewers? Could be a good or a bad thing, I lack experience and do not know.
Matt McIrvin 06.04.11 at 3:58 pm
The Journal of Bold Conjecture idea reminds me of edge.org’s occasional exercises in asking various public intellectuals “what do you believe but cannot prove?” and suchlike.
Scott 06.04.11 at 4:01 pm
The journal _Neurosurgery_ sometimes does this.
Scott 06.04.11 at 4:10 pm
(Sorry, I mean _Neurosurgery_ sometimes publishes the peer reviewers comments along with the article, if it’s accepted for publication.)
praisegod barebones 06.04.11 at 4:19 pm
I think it would be an interesting experiment for some journals to do this. Then we could find out whether Tom Hurka’s conjecture that people would be less likely to referee was correct.
I’d also like to mention – just in case any journal editors are out there reading this: I don’t think that I’ve ever been informed of a journal’s final decision on a paper I’ver refereed. Perhaps I’m an unusual case, but I find that a touch demotivating. (I’d imagine that this is the kind of thing that could be fixed with a small tweak to the editorial managemant software that a lot of journals are using these days. )
(Better still, I think would be a system where, in a case where there are multiple referees, each referee got sent a copy of the other referees reports. Generally, feedback helps us to get better at doing things: writing referee’s reports strikes me as one of the relatively unusual cases in academia where no-one gets any. Here’s an – as far as I can see – almost cost-free way of remedying that.)
praisegod barebones 06.04.11 at 5:09 pm
Incidentally – I’d have thought the obvious solution to the Kanazawa problem would be for journals to maintain lists of papers (with title and abstract) that they have rejected on their websites. (Possibly there’s a case for also putting the referee’s reports there – or for referees to have the option to do so – though Tom Hurka’s concerns might rule this out.)
Colin Danby 06.04.11 at 5:27 pm
If you exclude referee comments that were acted upon (and unless you also publish all successive versions of the paper, you really have to) won’t the bulk of reviewing labor remain invisible? Consider what’s often the most useful and onerous review task, the grad student paper that contains a good core, but a lot of dross. Reviewing is a matter of helping the author tighten the core argument, jettison excess baggage, etc. If it works and a good paper results, your final-round comment may be little more than “congratulations.”
What John proposes seems well-adapted to a situation in which a referee continues to have a substantive objection to a paper even after revision, and after the point that the editor decides to publish anyway.
I also wonder whether being named (and even if you opt for anonymity, you’re more likely to be sussed out if the thing is public) might make reviewers shy away from more controversial stuff — might a favorable review be misconstrued?
Tim O'Keefe 06.04.11 at 6:00 pm
I think that the proposal here won’t won’t, basically for dominic’s/Tom Hurka’s reason. Also, insofar as a ‘published paper review’ is counted for more than merely reviewing as such, it gives people bad incentives to recommend publication.
Tim O'Keefe 06.04.11 at 6:08 pm
Oops. “won’t work,” that is.
But on a more positive note: I like praisegod barebones’ suggestion of letting reviewers know of the journal’s decision. In fact, once a paper is accepted for publication, I think it would be a good idea to send the reviewers a copy of the paper as revised for publication, ahead of the journal coming out. Why not? The reviewer’s already thought about the paper, and if they still did have something to say about the argument, it would give them a head start in composing a discussion note (if the journal does such things), or something more substantial. So that would be a potential reward for agreeing to review a paper, and it also would help get discussion about the paper started more quickly.
Sebastian (2) 06.04.11 at 6:27 pm
What Colin says – the ref reports I’ve put most work on have on several occasions led to an R&R which then led to very solid revisions, which then led to a quick “I think this still needs fixing but overall great job” 2nd review – publishing the first review would seem silly, but it’s those types of reviews which are both the most work and probably the largest service to the community.
@praisegod – I’ve only received feedback about acceptance/rejection once (incidentally from a pretty low caliber journal)
Z 06.04.11 at 6:30 pm
I say, write the review and let it go. Give up control of the piece at that point — don’t insist that what amounts to a gloss by you be included when it is published. Also, you’re reviewing specialized work for specialized journals, with specialized readers who will know how to contextualize things. Let people draw their own conclusions, don’t breathe down their necks. You’ve reviewed the piece already; now perhaps it would be nice to let someone else react first.
Barry Cotter 06.04.11 at 6:46 pm
How common is actual double blind reviewing? I’ve recently seen someone refer to it as a farce, as if one is competent to review a paper one is intimately familiar with the literature. In that case you could narrow down the contenders considerably just by whose work they build on, and what they work on. If the reviewer actually makes an effort to figure out the author, like comparing style or looking up working papers in those fields where blind review is a vestige it becomes even more obvious.
Adam Kotsko 06.04.11 at 10:16 pm
It seems to me that this proposal is conceptually related to JH’s efforts to use blogs to promote better academic conversation — the idea is to provide journal articles with one or two (presumably good or informed) comments instead of a whole open-ended thread, but still.
onymous 06.04.11 at 11:27 pm
After complaining about peer review here yesterday, I just spent my entire Saturday afternoon painstakingly crafting a review of a paper which was fundamentally correct but full of minor errors, confusions, and misleading statements. I suggested it be accepted with revisions. I’ll check back in a month or two and see if it turned up in another journal completely unchanged….
John Holbo 06.05.11 at 1:35 am
“The journal Neurosurgery sometimes does this.”
Before Scott clarified his comment, I was genuinely curious whether the journal Neurosurgery ran an occasional ‘mad science’ issue about crazy brain operations that just might work – if it weren’t for these damn Ethical Guidelines! In a Wild Conjecture sort of way.
John Holbo 06.05.11 at 3:19 am
A lot of the concerns expressed here are reasonable but some could be addressed like so: the style in which one writes referee reports would have to change a bit, and obviously referee reports will always potentially contain substantial bits that aren’t really appropriate for publication – mostly the picky stuff. But I think referee reports I’ve written, of the revise and resubmit sort, could be easily rewritten for publication. ‘The author has responded to many point made in my original report but not all. Let me now restate them as critical concerns about the final version … [cut and paste lightly edited bits from your original report.]’ (This would be the lazy way to do it. Of course, you could always do it more painstakingly.) Knowing your referee report would be published would encourage making it more suitable for publication. This might encourage a certain amount of personal grandstanding, i.e. just rattling on about whatever interests the referee, but that’s already a danger as things stand.
Adam K is right that this proposal of mine is continuous with ideas I’ve expressed before about how every book ought to get a book event, and every journal article really ought to get an associated blog post or posts, with commentary and criticism. Anything worth publishing is worth reacting to. If you can’t get a decent reaction, you really don’t have a need to publish anyway. Journals really ought to recalibrate themselves as platforms for debate, with journal articles representing sound occasions for debate.
“I say, write the review and let it go. Give up control of the piece at that point—don’t insist that what amounts to a gloss by you be included when it is published.”
This doesn’t really address the concerns of Ingrid’s post. Namely, it’s hard to get enough peer reviewers. Telling them to ‘just do it’, in effect, hardly seems likely to have sufficient effect.
Jonathan Mayhew 06.05.11 at 4:15 am
I can see another disincentive because not everyone wants to get into a debate / exchange about every single possible issue. I would probably pass on peer-reviewing for a journal that obliged me to publish my report as a response, because I pick and choose my interventions. So instead of making it easier to get peer reviewers, it might be even more difficult. A lot of people in the position of reviewing, senior people like myself, don’t need extra lines on our cvs either. We wouldn’t necessarily want to criticize a junior person in public if it’s not relevant to our own research agenda, or, what’s worse, have to praise insincerely.
That being said, I do believe every book deserves a book event.
John Holbo 06.05.11 at 5:15 am
“I would probably pass on peer-reviewing for a journal that obliged me to publish my report as a response, because I pick and choose my interventions.”
Admittedly, a cultural concern. But perhaps we could get to the point where publishing a report is no more a high-commitment intervention that making criticisms after someone gives a department talk or delivers a conference paper. Admittedly, those are often pure shoot-from-the-hip affairs. But we might get something in between that and something like a true personal commitment to involve yourself in some debate, long-term.
onymous 06.05.11 at 5:49 am
But perhaps we could get to the point where publishing a report is no more a high-commitment intervention that making criticisms after someone gives a department talk or delivers a conference paper.
You humanists are so weird.
John Quiggin 06.05.11 at 8:22 am
@19 Most of the journals I review for advise me of the result. I must admit, I don’t pay close attention, but it probably has some subconscious influence the next time they ask me for a review. Certainly, it’s an elementary courtesy to someone who has done you a favor.
John Holbo 06.05.11 at 9:05 am
“You humanists are so weird.”
I don’t see what’s so weird about what I said. Maybe it was misunderstood. I don’t think that commenting on someone’s paper on topic X needs to mark a personal commitment to become a worker in the area of X in the future. If you started ‘publishing in the area of X’ that might seem to signify intent in this regard, so my suggestion that referee reports become a form of publication may have made Jonathan balk at what seems to him overcommittment to make something ‘your area’. I think it ought to be possible to be a referee in an area that isn’t your area of research, even though you should obviously be competent.
herr doktor bimler 06.05.11 at 10:33 am
Your reform would indeed lead to more painstaking, thorough, reports. That sounds like more work. So fewer people will agree to review, and reviewers are in short supply already.
Is Dominic implying that the majority of reviews within the current paradigm are less than thorough and painstaking? Slipshod bits of fluff based on a partial, half-understood reading? Inconceivable.
Bill Benzon 06.05.11 at 12:53 pm
OTOH, why worry? Isn’t there a body of research saying that most journal articles don’t get read? If so, the perhaps we ought to encourage more people to slack off on reviewing and thereby cut down on the publication of unread articles.
Bill Gardner 06.05.11 at 2:09 pm
Behavioral and Brain Sciences publishes theory articles followed by lots of brief invited peer commentaries, and I think it works well. Similarly, the Journal of the Royal Statistical Society publishes commentaries by invited discussants.
However, these commentaries are themselves edited. This would be a major cost that would deter most journals. Yes, they could publish unedited reviewer reports, but the best journals would want to ensure that commentaries meet their standards for writing and brevity. Moreover, why should journals publish unedited commentaries? There are many alternative venues for such communications (such as Crooked Timber).
Bill Gardner 06.05.11 at 2:14 pm
Just to clarify, the last sentence of my comment @38 was not a sneer at CT. Unedited commentary is valuable, and academic blogs supply plenty of it.
onymous 06.05.11 at 2:51 pm
Sorry, the “weird” comment was about “making criticisms after someone gives a department talk or delivers a conference paper,” which I was assuming refers to the practice I’ve encountered once or twice where someone is designated to stand up after the talk and give a short critical response to the talk they’ve just heard. We don’t do that sort of thing in the sciences (in my corner of them, at least), and it’s one of those cultural things that seems alien.
sg 06.05.11 at 3:10 pm
Wouldn’t an alternative to this be: every journal has a list of articles submitted in the past X months, with links to the reviews. So every journal is required to publish an electronic list of abstracts and associated reviews (maybe minus the text of the original article). You could even reduce this to just the results of the reviews. Then we all know what everyone is working on, we all know what everyone is saying, and negative as well as positive reviews get published.
I think newspapers should do a version of this, in which they publish a list of all the letters to the editor that they rejected, and the topic fo those letters (journals should do this, too). So you can see how many letters are being written about what, and form a view of whether or not the newspaper is biased.
I don’t think reviewers should get credit as a “minor publication.” It’s something we should all willingly take on as much as we can for the benefit of the field.
John Holbo 06.05.11 at 3:26 pm
““making criticisms after someone gives a department talk or delivers a conference paper,†which I was assuming refers to the practice I’ve encountered once or twice where someone is designated to stand up after the talk and give a short critical response to the talk they’ve just heard.”
Oh, we don’t usually do that either. I was just thinking more generically about how it’s useful to have a question and answer period. But it’s not so strange to have a designated respondent either. That can be quite a good format, I think. And, indeed, more like what I have in mind for referees.
John Holbo 06.05.11 at 3:27 pm
“I don’t think reviewers should get credit as a “minor publication.†It’s something we should all willingly take on as much as we can for the benefit of the field.”
Yes, but that really ought to go for publications, too. Given that the system is sort of screwed up, for more or less institutional reasons, we have to come up with something better than ‘we should all just try to do what’s best for the field’. We should try to think of ways to shift professional, institutional norms.
Ahistoricality 06.05.11 at 3:56 pm
Jonathan Mayhew’s comment (14) pretty well captures my first immediate reaction, which is similar to the “cameras in the courtroom” problem: by making the end result public, the process will change. It’s not necessarily a bad thing to engage more people, and wider publics, but there’s also some value in conversations that happen quietly among the people that concern them rather than for public consumption.
Ingrid 06.05.11 at 6:12 pm
This is a fascinating proposal, though my gut feeling is also that it won’t solve the problem of relative undersupply of reviewers while keeping quality up (I think, in fact, quality would go up with John’ proposal, but as others have already suggested supply would go down). But proposals such as these will trigger our imagination to come up with other ideas.
sg @ 41: “It’s something we should all willingly take on as much as we can for the benefit of the field.”
Yes, I agree with this, but this will not do: the problem is that (at least from where I’m situated) senior scholars are either all overburdened or severely overburdened, or else they stopped doing serious research a while ago, which may disqualify them as reviewers (they may be excellent teachers or have turned into valuable public figures, but that’s another story).
One thing we should investigate is whether there isn’t a way to signal which junior scholars (advanced excellent PhD students or postdocs) would be great reviewers, and ask them. They have an incentive since their CV is less full, and they can learn more by reading others’ unpublished research, I’d assume. Perhaps all editorial board members should annually draw up a list of such people and send them to the editors of their journals (who, if there is a collective of AEs, may pass them on). But somehow I think these would still be marginal improvements, not a ‘fix’ of the system.
On getting responses from the journals one reviews for: In the last year, I’ve reviewed for Ethics and for the APSR who both not only copied me on the decision letter, but also shared the AE report and the other reviewers’ reports. That makes reviewing interesting, since I learnt from those other reviews. But I can’t recall that this has happened for other journals for which I’ve reviewed.
Sebastian (2) 06.05.11 at 10:04 pm
(advanced excellent PhD students or postdocs)
@Ingrid – that’s done. I’m a final-stages of dissertation PhD student and have gotten a bunch of papers to review, two of them from top ten journals in polisci. One of them was – I assume – by referral from one of my committee members, the other one after I submitted a paper to the journal that the editor apparently liked – so there are ways to do this and it’s very much done.
John Holbo 06.06.11 at 1:31 am
One thing that’s fairly uncontroversial, I think, is that it would be a good idea for journals to try out different systems. If a few journals were to do what I suggest, that would be fine. I think it might be a nice way for lesser-known journals to distinguish themselves, potentially. Call your journal: Philosophical Discussion. And make it explicitly a journal in which every article comes complete with substantive responses from the refs. If the model works, if you can find refs, and people like the results, then the model can spread. I think the need for publication is so great that offering young folks the opportunity to get a small cv leg up might be enough to entice a lot of them into participating. Writing smart referee reports could be a way of getting your name out there (not by bomb-throwing and grandstanding, I’m not suggesting that. Just by being a perceptive reader.)
So, re: Ingrid’s top post right now. I would say a journal that feels a change might be better should give it a try, so long as there is no clearly bloody awful problem with whatever is proposed. If it isn’t going to be a total disaster, if you can always go back after a few issues if it doesn’t work out, take a flutter on it. We ought to be pluralistic and open to innovation. Of course, mucking about constantly with the standards may produce the impression that there aren’t really any standards, only moving goalposts. But if that would make people more cynical, I think that’s probably a sign that they should have been more cynical to begin with. The existing system works ok, but it’s not like it produces some distinctively superlative value – peer-reviewed! – with which we can shrink-wrap some products and send them forth into the world, intellectually sanitized and inspected for your protection! Since what we’ve got is hit-and-miss, we ought to be willing to try for a better sort of hit-and-miss system, at the very least.
Ingrid Robeyns 06.06.11 at 11:30 am
Sebastian (2): I agree it’s done and I do it myself too, but I was wondered whether it could be done more, to such an extent that it could make a significant contribution to solving the problem of the shortage of reviewers. If your experience is the general one, then my proposal will not work, since there is no ‘hidden/unused pool of excellent PhD students and postdocs’ to draw from.
piglet 06.06.11 at 9:20 pm
I recently stumbled on a different question. I was looking at an online profile of a professor in the social sciences. He lists a long list of “peer-reviewed publications” and several “peer-reviewed working papers”. Most of the “peer-reviewed publications” however were book chapters and think tank publications. I personally have never seen book chapters let alone working papers listed as “peer-reviewed”. Think tank publications of course may claim to conduct peer review but everybody knows this isn’t what we mean when we refer to “peer review”. Are there rules in your disciplines how to present publications? Would it be seen as dishonest in your discipline to inflate your publication record by listing papers that were not published in refereed scholarly journals as “peer reviewed”?
piglet 06.06.11 at 9:22 pm
And who sets the standards for what counts as a “refereed scholarly journal”?
Bill Benzon 06.07.11 at 10:11 pm
FWIW, I believe the Nobel Prize nominations are limited to listing a dozen or so publications for nominees. Which makes sense.
Everyone knows that the way to ‘stellar’ publication record is to publish the same ideal three to five times a year for a half dozen years and then, maybe, to work up another idea and peddle it for another half-decade or more. The Nobel committee doesn’t want to see this junk. If a thinker can’t establish a first-class roster of ideas in a dozen papers, then they’re not in the game.
What matter is idea count. But no one knows how to count them. So publication count is used as a third-rate proxy.
Comments on this entry are closed.