The Chronicle has a very interesting (if long) article on the ISI citation impact index, which seeks to measure the importance of academic journals through calculating the number of citations that each article in the journal gets. Like all indices, it creates skewed incentives for people to game the system. Authors tailor their pieces to get into the top journals, while journal editors’ choice of which articles to publish may be influenced by whether or not it will get lots of citations (and bump up the impact factor of the journal).
Many other editors contacted by The Chronicle also deny making judgments on the basis of whether a paper will attract citations. But Dr. DeAngelis, of JAMA, says editors at some top journals have told her that they do consider citations when judging some papers. “There are people who won’t publish articles,” she says, “because it won’t help their impact factor.” … Fiona Godlee, editor of BMJ (formerly known as the British Medical Journal), agrees that editors take impact factors into account when deciding on manuscripts, whether they realize it or not. “It would be hard to imagine that editors don’t do that,” she says. “That’s part of the way that impact factors are subverting the scientific process.” She says editors may be rejecting not only studies in smaller or less-fashionable fields, but also important papers from certain regions of the world, out of fear that such reports won’t attract sufficient citation attention. “It’s distorting people’s priorities,” she says, “and we have to constantly fight against that.”
Some journal editors take citation-boosting to an extreme.
In other cases, research articles in a journal preferentially cite that very journal, with the effect of raising its impact factor. ISI detected a clear example of that practice at the World Journal of Gastroenterology. The company stopped listing that journal this year because 85 percent of the citations to the publication were coming from its own pages. (Despite that censure, the journal’s Web site has a moving banner that still trumpets its 2003 impact factor.) …
John M. Drake, a postdoctoral researcher at the National Center for Ecological Analysis and Synthesis, at the University of California at Santa Barbara, sent a manuscript to the Journal of Applied Ecology and received this e-mail response from an editor: “I should like you to look at some recent issues of the Journal of Applied Ecology and add citations to any relevant papers you might find. This helps our authors by drawing attention to their work, and also adds internal integrity to the Journal’s themes.”Because the manuscript had not yet been accepted, the request borders on extortion, Mr. Drake says, even if it weren’t meant that way. Authors may feel that they have to comply in order to get their papers published. … Robert P. Freckleton, a research fellow at the University of Oxford who is the journal editor who sent the message to Mr. Drake, says he never intended the request to be read as a requirement. … Mr. Freckleton defends the practice: “Part of our job as editors is making sure that our work is getting cited and read appropriately.” The policy, he says, is not an explicit attempt to raise the journal’s impact factor. But the policy has done just that, and quite successfully, according to the The Chronicle’s analysis of self-citations to one-year-old articles — which are important in the impact calculation. In 1997 the Journal of Applied Ecology cited its own one-year-old articles 30 times. By 2004 that number had grown to 91 citations, a 200-percent increase. Similar types of citations of the journal in other publications had increased by only 41 percent.
The article suggests that the problem may get better as new forms of publishing and access move the highlight from journals to individual articles. I suspect that this isn’t going to solve the problem so much as to displace it – emerging systems of figuring out which article is or is not important will almost certainly be vulnerable to gaming too. But a very interesting read nonetheless (and the Chronicle’s colloquy on the topic, starting at 1pm today should be interesting too).