Having just agreed to review a manuscript for a journal, I was greeted with this message:
When viewing the article online we recommend that you view the HTML version of the article. If the author used EndNote (or, beginning in the fall of 2007, Reference Manager) for reference management, the article HTML proof will have its references linked directly into Web of Science. This linking will save you time when ascertaining the accuracy and validity of the references. Web of Science is now also available as an ”External Search” option.
I think I’m a reasonably conscientious reviewer, but I’ve never tried to “[ascertain] the accuracy and validity” of an author’s references in my life. I just assume that either (a) they’re accurate, or (b) if more than a few are inaccurate, the author will get a flea in her ear when the proofing process commences (I’ve been on the receiving end of this). Nor do I imagine that a few iffy referenpces in the bibliography would make me change my mind about the worth of a piece (perhaps if the biblio was obviously hopelessly incompetent, but I suspect that when this happens it is one of a multitude of sins, and bad referencing is likely to be the least of the author’s problems. But am I unique in this – do others scrupulously check the endnotes etc? I suppose that this is hardly a matter of world historical significance, but I’ve always been fascinated by the details of the reviewing process.
{ 23 comments }
Jonathan Dursi 08.08.08 at 4:23 am
I think the question is does the reference say what the author is imputing to it. I certainly don’t read all the referenced papers, but this would make things much easier to double check on a suprising claim; certainly in the physical sciences it is easy to misinterpret or overstate a recent result, intentionally or otherwise.
dr ngo 08.08.08 at 5:13 am
Yes, I regularly check (some) references. If the author of a (history) paper I am refereeing makes a surprising claim – e.g., something that if true I might reasonably be expected to have encountered before, not just something I know FA about – I almost instinctively check to see what his/her source is, and if it’s something I have readily to hand, may actually go to the text to see if it supports what the author concluded.
Usually it does, and I’ve learned something new. Too often, however, the source does not in fact establish quite what the author claims, which starts to call the whole argument into question and thus affects my evaluation of the manuscript. (E.g., an argument about what Filipinos thought of a particular situation turns out to be based on what some contemporary American, perhaps with an axe to grind, asserted that Filipinos thought at that juncture.)
Ingrid Robeyns 08.08.08 at 5:17 am
I never check the content of references that I don’t know – that would simply be too time-consuming. And for the referencse that I do know — if I suspect that an author is misrepresenting someone else’s work, then I don’t need links to the articles on-line — I go to my articles-files to check it on my hardcopy. I agree with you that this is a very weird message to receive from a journal/editor.
Nitish 08.08.08 at 6:31 am
I check a couple of references per article, on average. These usually fall into two categories: First, if the result claimed is surprising/contradicts my intuition, then I want to check the source; if the claim is incorrect, that obviously matters, and if it’s correct, I want to know why my intuition is wrong. The second reason is if the reference is absolutely central to the argument; in this case, it’s worth my spending a few minutes to verify that the argument doesn’t fail completely because of an incorrect claim.
I think I’m more conscientious than average, though. Perhaps more important, my field is theoretical computer science/math. Many of our references (and practically all the ones I check) are to theorems proved in previous papers. If you use a theorem from another paper in your proof, an incorrect statement of that theorem means that your proof is likely to be wrong. I think the binary “correct/incorrect” nature of mathematical proofs makes it more important to check that a reference is inaccurate. Also, the vast majority of recent CS papers are available on the author’s website, so checking a reference is usually easy.
(And to forestall any of my colleages who might object to my “binary correct/incorrect” comment: Many of our papers are page-limited, and so we sometimes accept a technically incorrect proof that is nevertheless true in spirit, if making it formally correct would require a significant amount of extra space. Even in these cases, it’s slowly becoming more common to expect a longer, correct version to be posted to the author’s website.)
Z 08.08.08 at 7:37 am
I am with Nitish. When a proof contains the sentence “and this follows from [14] theorem 3.6”, my duty is to check whether [14] actually contains a theorem 3.6, if this theorem applies here and if one can actually deduce from it what the author claims. I think there might be a big wedge between natural sciences and social sciences with respecto to this question.
Z 08.08.08 at 7:43 am
And I would add that mathematicians can become angry pretty fast against someone who has inaccurate or invalid references. See for instance this footnote in a recent preprint:
In fact, almost all non trivial results of […] can be put in the following tripartite classification:
(a) Results for which a (sound) reference is given, but of which no proof can be found in the given reference (e.g. loc. cit., Prop. 4.2.2(i)).
(b) Results for which â€a proof will appear elsewhereâ€, and for which no proof has appeared anywhere (not even in a preprint, twenty months after the publication of […]). (See e.g. Theorem 3.2.1 loc. cit.)
(c) Results for which a reference is given, that does not exist. Indeed, many results are said to be proved in […]. This preprint does not exist, either in paper or electronic form, under this name or any other, and for obvious reasons will never do.
Otto Pohl 08.08.08 at 8:29 am
I check references if they at hand. Often they are not. If I find an unusual claim and the citation refers to a reference I can access I will definitely check it. The problem is that many or most references in many articles sent out for review are to unpublished sources. It is not possible to check restricted foreign archives or interviews with anonymous informants. In this case one can either accept the claim or note that it contradicts published information. What is very annoying is when the few references to published archival and secondary sources prove to be wrong even to the point of verbatium misquotations, but the bulk of the material is based on unverifiable unpublished sources. One can only suspect, but not prove that this matarial too contains citation errors.
Martin Wisse 08.08.08 at 9:44 am
The references are there for a reason, so I hope you’ll would check them. Haven’t there been a fair few cases in which people did try to bamboozle with references?
conchis 08.08.08 at 11:00 am
I’ll obviously check references that I know if I suspect suspect they’re being misrepresented, but I’ll also check the contents of references I don’t know, roughly on the same criteria as nitish: when (a) there’s a claim that seems counterinuitive or implausible to me; or (b) the referenced claim is crucial to the argument.
Henry 08.08.08 at 1:29 pm
Ah, I misunderstood here I think – obviously when a reference is being used to support a claim that it perhaps doesn’t make, I’ll go and check it if it smells iffy; my understanding of the message had been that it was asking us to check whether prof bumscratcher’s seminal piece really had been in issue 3 of volume 27 rather than issue 4 or whatever/
Original Lee 08.08.08 at 1:36 pm
As a librarian, I wish authors would be more careful about their citations. I cannot tell you the number of times I have been handed a list of references as a journal photocopy request and spent most of a day hunting down a citation with wrong page numbers, wrong volume, wrong year, wrong journal, wrong author. (On occasion, I’ve had to call up the author to find out what article s/he actually meant.) Errors in the references tell me the author(s) were relying on memory, not verifying that *this* article was where a particular concept came from before sending it in for review. I say hurray! for this new trend.
Jacob T. Levy 08.08.08 at 1:51 pm
Yep, I would guess that it’s there to let referees actually look at the articles referred to, not to do copy-editing work. And I’d say it counts as a genuine convenience, though not a huge one, since if the article’s online I can get to it in a few seconds anyway.
Z 08.08.08 at 2:00 pm
References I check for content, I usually also check for accuracy, so as to ensure that indeed the future reader will look at page 507 and not 577 as written in the preprint I have on my desk. The references I don’t check for content, I don’t check for assignation either, for the obvious reasons you presented in your post.
J 08.08.08 at 3:01 pm
Earlier this year, I was sent a revised paper to review (so presumably, other reviewers had already seen it).
One page in the “Methods” section seemed oddly out of place (the quality of the writing was different, there were terms used that are not native to our discipline, etc.)
The whole page turned out to have been plagiarized, word for word, from a very obscure paper by another author (not cited in the MS that I was reviewing). Oddly enough, the plagiarist had added a citation at the end of the section to a textbook by a different author.
I assume this was intended to muddy the waters a bit. The problem with citations to books is that, in the absence of a specific page number, it can be difficult to wade through a book and figure out whether the source says what the cite-er claims it says.
Eszter Hargittai 08.08.08 at 3:09 pm
I check references for substance sometimes, as stated by others. Also, if references mentioned in the document aren’t in the bibliography I start getting suspicious. (Okay, if it’s just 1-2 missing, that’s one thing, but if it’s something like what J mentions, that’s another.)
I also get suspicious when someone cites one of my pieces incorrectly – as in the author uses it to back up a claim that I know my paper doesn’t back up -, because then I have to start wondering how many other citations don’t do what they claim them to do.
Matthew Shugart 08.08.08 at 5:36 pm
Oh, I see a lot of sloppy referencing, as when someone wants to drop a name but the context of the sentence in which the name being dropped is misleading as to what the cited author said.
However, I have never systematically checked references, only noted (so to speak) those that stood out as inappropriate.
I am rather horrified by the presumption that editors might expect reviewers to follow up in the way the quoted excerpt suggests, or would be encouraging the purchase of a specific software product as part of the duties of a good reviewer. (Bad enough that so many journals have now gone to on-line reviewing systems in which one must use Acrobat because the submission is posted in proprietary FDF, rather than the more interchangeable PDF, format.)
Henry 08.08.08 at 5:51 pm
I also thought the same thing as Matthew about the implied suggestion that authors should use specific software products – I would have been a lot less concerned if they had, say, included BiblioTex or some other non-proprietary solution instead or as well. More generally, I do recognize Lee’s concerns as a librarian – but not enough to motivate me to go through someone else’s bibliography line-by-line. I imagine that this has gotten to be less of a problem as scholars have started to use Google Scholar and the like to download refs – but I think that this is a problem in search of some more complete automation (my vague impression is that there is discussion of collective standards, but that it is not working out as well as it should b/c of the conflicting interests of different publishers).
Colin Danby 08.08.08 at 7:02 pm
I’m with others above — I’ll look up things that seem fishy or really critical, but I don’t see a complete check as part of the referee’s job. Even at fastidious journals, as far as I can tell there is no process to do that.
Happily the interwebs have made it a lot easier to locate bungled refs.
If I ever finish my paper on Raul Prebisch, I’ll face the problem that the highly-useful fourth volume of his collected writings is, as far as I can tell, in only one library in the world and according to OCLC does not even exist…
eszter 08.08.08 at 9:00 pm
Editors are already having a hard enough time finding people to review manuscripts (I just spoke to one about this yesterday). I can’t imagine it would help to start adding admin tasks to the request.
mollymooly 08.08.08 at 9:45 pm
So how often does a plausible-looking paper get published whose shoddiness comes to light only when it is shown that the references are fictitious?
Martin Wisse 08.10.08 at 9:13 am
I’m with others above—I’ll look up things that seem fishy or really critical, but I don’t see a complete check as part of the referee’s job
The problem here is, do you always know that something is fishy? it’s probably somewhat less of a problem in scientific papers, but there have been quite a few pop-science or political books which systematically used citations to underwrite an argument not actually supported by the citations themselves.
Colin Danby 08.10.08 at 6:52 pm
Molly, I think fictitious as in whole-cloth-invented is rare, though I’m sure someone will find an example. Tendentious mistakes are more common.
Martin, ideally the referee knows the area and the cited literature well enough to catch consequential misrepresentations. A good example of this not working is
http://math.ucr.edu/home/baez/bogdanov.html — these fellas got five articles into refereed journals.
Righteous Bubba 08.10.08 at 7:36 pm
Thanks for the link Colin.
Comments on this entry are closed.