Incentives for reviewing

by Henry Farrell on March 13, 2008

“Tyler Cowen”: responds to the discussion on open publishing.

I don’t envision the free access system as the status quo but free. Papers would be ranked directly in terms of status and popularity rather than ranked through the journals they are published in. Ultimately there wouldn’t be journals and this would make a big difference as journals are the current carrier of selective incentives and status rewards. It would be easy to refuse to referee, since you wouldn’t fear being shut out of publication of that journal; I suspect refereeing might die. And if status were attached to the individual paper rather than the journal, who would bother to become an editor? It would be a very different world and in some ways more like (academic) blogging than its proponents may wish to think. In other words, the partial monopolization of for-fee journals makes it possible to produce status returns to motivate both editors and referees. Returning to the free setting, refereeing will survive insofar as writing detailed referee comments on other people’s work helps with your own research; it is interesting to ponder in which fields this might hold.

The interesting bit for me here is Tyler’s suggestion about the implicit incentives for reviewing; that people referee papers for fear of not being able to get published in the journal in question. My personal take on it (as is the take of a number of other people, if “this discussion”: is anything to go by), is a little different. I review not so much because I feel that if I don’t review a paper for journal _x_ that the editors of that journal will look unkindly on me in future, but because of a broad sense that I send papers out that others ought to review, and hence there’s a diffuse obligation on me to review other people’s papers in turn. In other words, I think that the motivating factor is general reciprocity rather than specific reciprocity. Not only that: when I have been on search committees where we are considering people who have been in the field for a few years, I usually check their resumes to see whether they have been reviewers for a few journals. This isn’t so much to figure out what the editors think of them (very often, editors are happy with whoever they can get as a reviewer), as because it seems to me to be the best publicly available proxy for whether the candidate is the kind of person who is likely to take on their share of the unofficial responsibilities that any school or department has.

This isn’t to say that Tyler may not be right when he suggests that an open publication world might not support the kinds of detailed and thoughtful review that we hope for, and sometimes get, in the current system. But I suspect (perhaps wrongly) that the mechanism that would undermine reviewing would primarily be a sociological one rather than an economic one. That is, it would have more to do with the disappearance of the social role of reviewer, and the set of perceived general responsibilities that go with it, than with the opportunities for specific quid-for-quo interactions between reviewer and editor that the current review system lends it to.

{ 5 trackbacks }

liberating authors (and reviewers)? «
03.13.08 at 7:31 pm
Can blogging be a kind of peer review? « Ad Nauseam
03.13.08 at 9:49 pm
Some thoughts on peer review model « Entertaining Research
03.14.08 at 6:00 am
03.14.08 at 8:03 am
The importance of the journal (peer review) system « The Wobbling Mind
03.14.08 at 12:13 pm



Steve LaBonne 03.13.08 at 5:32 pm

I think you’re being polite; Cowen’s remark about refereeing is a classic example of the sort of thing you have to be an economist to believe. When I was in academia (life science) I never knew anyone who did it out of any motive other than a sense of reciprocal obligation to the community. ANY healthy field of endeavor relies to some on altruistic behavior deriving from a genuine commitment to said field. In my work nowadays as a forensic scientist, I do various sorts of uncompensated services (and such are done for me from time to time) for colleagues in other labs, when requested. A significantly time-consuming one for me just now is serving as external peer-reviewer (all DNA cases must be peer-reviewed before being reported out) and technical leader for an acquaintance whose lab is even smaller than the one where I work. I do it because it needs to be done and it serves the goals we both share.


Henry 03.13.08 at 5:43 pm

Steve – I think that is fair up to a point – but there is some circumstantial evidence that people are motivated by the kinds of things that Tyler points to as well – there are a couple of comments in Ingrid’s thread that could certainly be interpreted to support Tyler’s story. I still think that the sociological stuff swamps the specific incentives stuff – but I don’t think that the latter is entirely absent. And I’m sure that you have had colleagues (as have had we all – I am thinking of one in particular in an institution I worked for previously) whose attitude is explicitly that they only do something if there is a direct benefit for them.


Steve LaBonne 03.13.08 at 5:51 pm

Sure, there are- um, not so nice people- in any field. But I also never heard of a biology journal that had a preference for publishing people who referee for them (as opposed to the notorious general propensity to treat papers from bigwigs’ labs more gently than those of small fry.)

I’m willing to suppose though that in at least some humanities and social science fields things are a bit more “personalized” than they are in the more “industrialized” world of well-funded laboratory sciences, in a way that would make Cowen’s story make some kind of sense.


Josh Jones 03.13.08 at 6:08 pm

As I mentioned on Tyler’s page, if blogs are at least a small indication of academics’ willingness to comment on other academics’ work in an “open access forum,” then the problem may be an overabundance of reviews rather than a shortage there of (even if people do it for different reasons).

Granted, Henry’s point about “detailed and thoughtful” reviewing is absolutely valid. It seems, though, that circumstantial evidence in the blogging world (or on, even) that one’s legitimacy as a reviewer and not just as the author can be assessed and “judged” by other readers fairly easily.


Steve LaBonne 03.13.08 at 6:14 pm

There are plenty of undetailed and thoughtless reviews under the current system (goes for grant proposals as well as papers). I’ve been on the receiving end of some, back in the day- who hasn’t?


Eric Nilsson 03.13.08 at 6:30 pm

Perhaps different disciplines are, well, different.

In economics (more so than in, say, some fields in biology), you have insiders and outsiders. Social factors, such as fitting in, citing the right people, and doing reviews for journals run by the powerful, play a larger role in economics than in many other fields. Also reaching the “correct” ideological conclusion is very important in economics. These things are, I believe, less important in the “real sciences.”

Of course, all fields have a mix of “objective” criteria for publishing and “social” criteria for publishing. But the mix differs by discipline.

Cowen reveals more about the social factors in economic publishing that perhaps he intended. Or, perhaps he suffers from incomplete–indeed false–information. ;>


Ingrid Robeyns 03.13.08 at 6:33 pm

I usually check their resumes to see whether they have been reviewers for a few journals. … because it seems to me to be the best publicly available proxy for whether the candidate is the kind of person who is likely to take on their share of the unofficial responsibilities that any school or department has.

That’s interesting — now thinking about it, I’ve always regarded it as a proxy of how ‘internationally connected’ a scholar is — believe it or not, there are plenty of (full) professors in this country (and other continental European countries) who hardly ever review for internationally refereed journals, simply because they are not playing the academic game at the international level. And if my own experience on being asked to referee is at all representative, then there is a significant positive correlation between having met an (associate) editor at a conference, seminar or meeting, and subsequently being asked to review a paper for their journal.


HH 03.13.08 at 6:37 pm

The recursion here is hilarious. The sociology and political science professors need to solve the Internet trust synthesis problem in order to modernize their mechanism for refereed publishing. They will then be able to bestow this gift to the world at large, then write about it as a profound new social paradigm.

It would have been equally amusing if Gutenberg had used his press primarily to print labels for wine bottles.


c.l. ball 03.13.08 at 6:54 pm

Dan Nexon’s comment in the thread Henry links to above is worth reading. I was the de facto managing editor at ISQ for a year and half — I had no executive editorial role; I chaired the editors’ meetings but did not vote on or recommend for or against publication ever. I did recommend potential reviewers to the editors handling an article if their initial set did not work out, especially when I had to point out that I couldn’t disinter someone just to ask him to review.

The decline to review rate was high: a third declined. Some would never even reply to the query, repeated queries too. That rate might not sound bad, but if you are trying to get three reviewers — our target — there was only a 30% chance this would happen on the initial requests. It was not unusual to make five requests to get a single reviewer.

I was shocked, however, at the poor quality of some reviews — often from senior people. A junior scholar was more likely to submit a detailed reply. Unless the ms. was really bad, I don’t think less than a full-page response to a serious research effort is appropriate. That said, the majority of the reviews were good, but a lousy review when one reviewer has gone AWOL is the kiss of death to a manuscript. Even if the third reviewer’s report is positive, the lousy review undermines confidence that an R&R is a good idea. AWOL reviewers are evil as well — I won’t assign one senior academics books now given his excremental evasions on completing a review. When you get 2 opposing reviews and the 3rd is AWOL we would have to find a 3rd reviewer late in the game, or else give a really weak R&R, often on the condition of finding a new reviewer on the revised ms. Why? Because the 1st round positive reviewer was a shoo-in for acceptance, and the 1st round negative reviewer was unlikely to change his or her mind. Indeed, some reviewers who said “reject” the first time refused to review revised ms.

The worst decline I ever received from a potential reviewer was in March 2003: “sorry. Too busy war-blogging.” The best decline came in late April 2003: “I’m sorry, but I just landed in Kuwait City and head up to Baghdad tomorrow.”

Re 3, in the international/comparative wing of political science, there are some journals — not association based ones — that exercise editorial judgments over what to send out for review (e.g., International Security, World Politics). They might decline to send a piece out for review on the grounds that they published on the topic recently or that they find the argument poor, but they might send out pieces of friends or big-wigs without doing so. Association journals tend to send out any piece that meets formatting and subject guidelines. This annoyed some ed board members at ISQ, who felt that the editors should have decided whether a piece was worthy of review.


Sortition 03.13.08 at 7:34 pm

The problem, it seems, is that reviewing is currently a position with significant power but no recognition for quality work. Acting as an anonymous censor is not a situation which brings out the best in people. In such situations petty politics, minor utility considerations and arbitrariness dominate.

Compare this situation with that of a person who has been asked to write a comment for a discussion of a paper which has already been accepted and is about to be published, as is sometimes being done. In that case, the writer does not act as a censor – he cannot prevent the publication of the original paper, only to respond to it. Secondly, the quality of work by the commenter is viewable to all and impacts the reputation of the commenter.

I would guess that both the positive response rate and the quality of the work are much higher for comments than for reviews.


Kieran Healy 03.13.08 at 7:37 pm

The problem, it seems, is that reviewing is currently a position with significant power but no recognition for quality work.

Some fields do have recognition — e.g., prizes — for good reviewers. The Academy of Management do this, I think.


Tyler Cowen 03.13.08 at 7:56 pm

I don’t think Henry and I are so far apart. I, too, feel the “general” obligation to be important and not just the specific obligation to a particular journal. But as other people starting disregarding their “general obligations,” I expect the whole norm would change over time, if only for the next generation.


ben wolfson 03.13.08 at 8:17 pm

Why would individual journals die out? After all, they perform important roles—conferring status is one, but dividing things by topic or style is another—I wouldn’t look for a short piece on contemporary metaphysics in Ethics, say.

If all the philosophy journals went the Philosopher’s Imprint way, that’d be just dandy, and I don’t see why it would lead to their being merged into one in the Overmind.

(This probably came up in the comments to one of Henry’s previous posts, but however much it costs to host arXiv, I would bet that it’s less than the combined costs of all libraries that want to keep up with even a modest range of Springer Verlag journals, and that those institutions would be only too happy to front the bucks for a different solution.)


Lee Sigelman 03.13.08 at 9:06 pm

When I was editing the APSR, (1) I didn’t mind if people reviewed because they were afraid not to, but I must say that (2) never in six years did it even enter my mind to base a decision to accept, reject, or invite a revision, even a little bit, on the author’s history as a reviewer; in a few cases, I sort of wish it had, because there are a few authors out there who are heavy submitters but apparently feel neither fear of not reviewing nor obligation to review.


SusanC 03.13.08 at 10:06 pm

But I also never heard of a biology journal that had a preference for publishing people who referee for them

I don’t know of any conference or journal who actually does this. But some people working in Computer Science have seriously proposed schemes where authors have to do some number of reviews for each paper they submit.

One of the incentives to do reviews is that if the journal/conference folds due to lack of reviewers, your paper won’t be accepted either.


Scott Hughes 03.13.08 at 10:57 pm

What would be so bad about making it less of a business? What would be so bad about papers being published on their merits and not as a return for working as a referee?


Martin Bento 03.13.08 at 11:16 pm

It sounds like the basic problem here is that – to borrow a distinction that I think comes from Clay Shirky – academic publishing, like traditional commercial publishing, follows a model of “filter, then publish”, whereas the Internet lends itself to “publish, then filter”. Perhaps the way to bring the advantages of open access to academic papers is to find a way to move them to the second approach. I’m not of that world, so what I’m proposing may be naive, but let’s think it out. Papers are simply posted online. Various people review them after the fact, anonymously or otherwise. Metareview becomes possible, as people can see whether the review is fair and competent. If the main value of the journals is the reviewing, “journals” can actually become brands for groups of individually anonymous reviewers who will come to have a group reputation as a brand. If one wants further to have anonymous or identified reviewing, one can. If it is important to keep out the “cranks”, one can block reviewers who are anonymous to the hosting site or who lack proper credentials, while still keeping identities unknown to the public, if desired. This, of course, leaves out the economics of the whole thing, which is the usual weakness of open source endeavors, but some form of subsidy would probably be much cheaper than the current system anyway, no? Any obvious problems with this?


SG 03.14.08 at 3:45 am

If universities en masse agreed to shift to an open source system, and instead of paying exorbitant fees (through libraries) to view articles, spent the money on paying inhouse librarians or computer people to manage open source publishing efforts, I suspect the cost of the system would not be too great. If as emma said on an earlier thread, it is easy for an academic to run a single open source program themselves, surely a few web-savvy librarians could run a whole bunch for a particular university. The universities could then compete for the best quality articles and reviewers to establish the “brand” of which you speak. Surely the cost of licensing for even the second-rate journals at most universities would more than cover the cost of a few librarians devoted to open access journals.

I imagine this would make a new type of librarian position at most universities, too, and broaden the range of tasks and career opportunities for existing librarians.


David Estlund 03.14.08 at 3:56 am

Martin beat me to it, so here’s roughly the same idea: What if everyone just posted whatever they want (with special places for final drafts), then journal-like entities could gather what they liked and gain a reputation for good judgment. The imprimatur function survives, the (non-) journal can gain prestige, the work is all available for free, and the papers are publicly available whether or not any such publication picks them up.

It has promise, although it could get complicated. No such non-journal has any exclusive rights to pieces, so there would be a lot of overlap. Also, one worries about how the journals would choose which articles to read and consider. There might be a risk of reputational double-counting that traditional journals can largely avoid by reading what’s sent to them.

Here is a variant, the IPT Beacon A free online journal, with a traditional (and distinguished) editorial board, that gathers pieces published by traditional journals, links to them in a journal-like way, offering them for free online. These pieces were first vetted by traditional journals, however, with the advantages and disadvantages that brings. (They also accept original submissions, but none have been published yet.) This journal deserves attention and kudos, so go check it out. (Full disclosure: a piece of mine was chosen for Issue 4.)

A smaller step is simply to have free online journals. All the advantages of traditional journals plus world-wide free availability. Why would anyone do it if there’s no money in it? God only knows. But people are doing it. (I’m one of four editors of JESP , one of only a few examples in philosophy…so far.)


ben wolfson 03.14.08 at 7:10 am

The idea that the internet is inherently publish-then-filter is wrongheaded. The internet makes it very easy to publish, so much is published, so you have to filter, and it “lends itself” to this model only insofar as what’s published hasn’t already been filtered. But there’s no reason to think that you couldn’t have individual sites that filter then publish, as, for example, existing web-based peer-reviewed journals do.


Martin Bento 03.14.08 at 7:48 am

Ben, the Internet doesn’t *have* to be publish then filter, but it does, as I said, lend itself to this, because it makes it practical to do things this way. Where there is non-trivial cost to publication, publish then filter makes no sense. Conversely, where filter, then publish is not necessary, one must ask what advantages it has to outweigh the disadvantage of restricting access to information, including metainformation (information about how information is filtered). There may be some in some situations, but it is not obvious that there are general such advantages; that needs to be argued.


ben wolfson 03.14.08 at 3:50 pm

Conversely, where filter, then publish is not necessary, one must ask what advantages it has to outweigh the disadvantage of restricting access to information, including metainformation (information about how information is filtered).

Nothing about filtering then publishing means keeping the filtering process secret, or restricting access to information; it just means that some information isn’t going to be distributing in this venue. Suppose I send my paper around to all the big filter-then-publish websites, and none of them wants it. Then I put it up on my own personal site.

Of course I’d only do that if I couldn’t get it in elsewhere, which is a de facto sort of filtering. But I’m not seeing what the disadvantage of having some venues filter first is.


harry b 03.14.08 at 4:11 pm

David — am I right in thinking that JESP has a substantial subvention from USC to ensure that it will always be online, gives server space, etc? It might make sense for universities to spend money making their own subsidised journals free to all, and just cut the journals budgets of their libraries by that amount.


Martin Bento 03.14.08 at 9:50 pm

Ben, well, one disadvantage is that publication of new material must wait until the filtering is done, so the dissemination of knowledge is slowed. However, that research may well have relevance to other research that is being done at that very time that could, for example, change how the latter research is conducted. Another is that the filtering that can be done by a particular journal is constrained by the resources of that journal, including, of course, the volunteerism it can attract. If the material is already public, all of that filtering is still available, plus whatever else may come from other journals, a possibly-filtered public. Finally, for a filter-then-publish model to truly provide as much information about the filtering as publish-then-filter, it would have to provide the raw paper and all the reviews and discussion. This is much like publish-then-filter, save with a delay imposed. If it does not do this, then it is not providing equivalent information. So there are disadvantages. Now what are the *advantages* of FTP?

Also, you’re arguing that some FTP should be retained, which is much further towards PTF than the status quo, which, in this area, is almost entirely FTP. I’m not out to “ban” FTP, but I think once PTF becomes established in the ecosystem, FTP will become marginal, as PTF provides for a more rapid and open discussion.


Martin Bento 03.14.08 at 10:01 pm

Don’t know why FTP came out as a link above. I didn’t intend that and it doesn’t go anywhere.

Comments on this entry are closed.