My namesake, Tom Quiggin has been in the news lately, debunking the idea that Al Qaeda cultivates sleeper agents and also tracing to its source the urban myth that Osama Bin Laden used a private fortune of $300 million to promote the group.
He’s sent me some reflections on the sloppy research that’s been used to promote some of these ideas, noting
. A disconnect between the statement in the body of the article and the sources in the footnotes which do not back up the statement being made,
2. Strong statements which are made, but which are built on weak foundations or on assumptions which cannot be shown to be valid,
3. Information from two different situations is overlapped or mixed together, leaving the reader with a false impression about the nature of a particular problem or situation,
4. In a limited number of cases, information provided in articles is simply false.
The faults he points out are, I think, found to some extent in every field (I’ve certainly found plenty of instances in economics, though the prevailing flaws are a bit different), but fields like the study of security issues have the added problem that replication and verification are particularly difficult. Processes such as peer review, replication and empirical testing aren’t panaceas, and errors will always slip through, but they work pretty well in the long run.
{ 20 comments }
JoB 03.07.10 at 10:00 am
The main method to avoid persisting error is truth. Peer review and the like just have a function of expediting the convergence of conjecture and truth. It is conceivable that a method of peer review, which is essentially conservative, delays the finding of truth – it would suffice to control the selection of peers. Is there research done on the subject of establishing peer groups that emerge a different but stable consensus in the long run?
Take the maybe relatively uncontroversial topic of brain science: there are clear signs of the most nauseating scientism there (“The comatose patient is conscious!”) that has been confirmed by peer review.
Sometimes only the laymen will be able to break a false consensus between peers.
Neil 03.07.10 at 11:35 am
Oh yes, the method of truth. JoB, do you have any argument for the claim that the neuroimaging studies you refer to are no good? The patient answered questions, for gods’ sake.
JoB 03.07.10 at 1:46 pm
For God’s sake, yes. One argument could be that in a controlled test the findings could not be replicated. Another argument could be that a claim that a certain neuroimage is found is not the same as a claim that consciousness is there.
PS: And I am sorry on ‘the method of truth’, I really am. I just think John is rather a bit too positive on the other methods; he doesn’t trust peer review in economics too much (not in the 90’s at least – and the only reason the consensus is broken is because reality is catching up with it, not because of the outcome of peer review).
kid bitzer 03.07.10 at 3:23 pm
i’d love to hear more details about the method of truth. it sounds terribly useful and direct.
what are the first steps in the method? what the later steps? are there any preliminary steps needed prior to its employment? does the method come with any self-checks to make sure it has been properly employed?
JoB 03.07.10 at 4:02 pm
4- Yep, I deserved that too ;-(
kid bitzer 03.07.10 at 5:37 pm
far be it from me to give anyone what they deserve–if that kind of thing gets around, i have the most to lose.
but if your #1 was not offering a better method, then what exactly was it offering? peer review etc. are all fallible methods. but without better ones, they’re the best we got. which was j.q.’s original point. which prompted you to say…?
JoB 03.07.10 at 6:41 pm
It was offering, in a very embarrasing way, a qualification on the power of peer review. I guess I wanted to snipe at neuroscientists specifically and peer groups in general with a tendency to shut out ‘unqualified’ critics.
I apologize for abusing John’s post.
Witt 03.08.10 at 12:22 am
fields like the study of security issues have the added problem that replication and verification are particularly difficult
In my experience, another variable that has a powerful effect on some fields is the amount of distance between the researcher and the consequences of his/her research. A cancer researcher is (at least in theory) studying something that he or she could be directly affected by.
Whereas someone who is studying immigration policy or “security” studies* is often someone who will not be personally affected if their data turns out to be wrong, exaggerated, used in service to a political agenda opposing residents of particular countries, etc.
It matters if the researcher has skin in the game. You get a different set of biases at work — I would argue in many cases a less belligerent set — if the researcher perceives him or herself to be among those potentially affected by the results of the research.
*phrase in quotes because I am dubious about its relation to security.
piglet 03.08.10 at 4:23 am
The Tom Quiggins posts don’t reference any academic literature. Therefore it is difficult to derive any judgment as to the quality of academic research on terrorism. I don’t have a hard time believing that there is a lot of shoddy research going on in politically sensitive areas like that one. At the same time I haven’t seen any evidence that academic research has played a big role in justifying crimes committed by governments in the name of anti-terrorism. The US, Canada and Britain have laws that allow the government to imprison any individual they wish without any meaningful judicial oversight. This is not an academic but a legal issue. The Canadian and US courts, and to some extent the British courts as well, have consistently ruled that whoever the government designates as a terrorist may be imprisoned indefinitely, without the right to a proper court trial, no matter the evidence. It would make sense to debate how Anglo-american courts, only they, have come to adopt a legal doctrine that is diametrically opposed to centuries of Anglo-american legal tradition.
Neil 03.08.10 at 9:22 am
JoB, I will take you through the evidence and you can tell me where the fault lies. Owen et al. (2006) measured activation in the SMA in response to an instruction to play tennis, and activation in PPC, PMA and PMC in response to an instruction to imagine walking around one’s own home. Why did they use these tasks? Because they found that they got very nice clear signals in healthy controls in these paradigms. As you of course know, SMA activation is indicative of imagined movement and PPC etc activation is involved in navigation tasks. They regarded the activation in an apparently PVS patient as evidence of instruction following – a voluntary action. Now you need to tell me why it is not. Note, first, that the responses are not habitual (it is not like a response to the patient’s own name). Indeed, the response was to an unconditioned stimulus (“start the task now”). Moreover, the task was engaged in for a full 30s. So why isn’t this evidence of instruction following? How does a failure of replication even bear on the question (and why does the successful replications, by the same ground and an independent group, not count?) In more recent work (Monti et al. 2010) the same paradigm was used to communicate with an apparently PVS patient (if the answer is ‘yes’, imagine playing tennis; if ‘no’, imagine navigating around your house). Now tell me what was wrong with this study?
You say that peer review is used to shut out unqualified critics. Just give me some reason to think that unqualified critics have something to contribute.
Zamfir 03.08.10 at 9:47 am
The uses of the word paradigm are many.
JoB 03.08.10 at 11:33 am
Neil, ‘So why isn’t this evidence of instruction following?’ I’m not saying such research should be barred or is unhelpful (although it is very difficult to set up controlled experiments). I also do not know this specific research (my snipe had something else in mind; google ‘Rom Houben’ if you want) and won’t say whether it is out of line in the typical sense of making grand claims on consciousness. Take your ‘voluntary action’; fair enough for me but if you jump from there to a claim of hardwired morality … the unqualified ones sometimes are the neuroscientists that have a tendency to think that most of the social sciences and all of philosophy can be absorbed as the byproduct of an experimental set-up. That’s all. I kind of like this line of research, by the way, if the researchers refrain from simplfying statements that get them into the spotlight.
Del Cotter 03.08.10 at 11:37 am
Did anyone ever cultivate sleeper agents, or is that just something that was made up for novels?
kid bitzer @6: POLONIUS My lord, I will use them according to their desert. HAMLET God’s bodkin man, much better! Use every man after his desert, and who shall ‘scape whipping?
Neil 03.08.10 at 11:40 am
JoB: Rom Houben was diagnosed as conscious on the basis of the revised coma scale, backed up by a PET scan. He is indeed conscious. Unfortunately, his therapist – not a neuroscientist – took the opportunity to publicize false claims about how can communicate fully. He can’t: he is minimally conscious, which is not compatible with complex communication. None of this has ANYTHING to do with hard wired morality. You are right that there are people who have imperialistic ambitions and advance sweeping claims like those that offend you. But guess where this happens? Outside the properly peer reviewed literature. None of your confusions even bear on its quality.
Zamfir 03.08.10 at 1:48 pm
Did anyone ever cultivate sleeper agents, or is that just something that was made up for novels?
A remarkable amount of spy novels and spy cliches can be traced back to the Cambridge 5, who I guess came pretty close to being ‘sleepers’. But I can’t tell whether they were typical for cold-war spying, or an atypical example on which the genre came to built.
Del Cotter 03.08.10 at 2:35 pm
I would want to start by defining two sorts of sleepers. First, those who are contacted in their daily lives, and instructed to keep going about their daily lives until further contacted. These seem fairly plausible, but I find it hard to think of them as “sleepers” in the full meaning of the word.
Second, those who are from outside the target organization, inserted into it with a fake background, and told to do nothing but maintain and add to that persona until further contacted. This seems like a difficult thing for any agency not backed by the resources of a large state to pull off, and seems to have few advantages. Do you hope a spy of your own is less unreliable than a native? But you’re encouraging them to go native!
Salient 03.08.10 at 3:06 pm
I just want to point out that Avatar is one gigantic pun on the term “sleeper agent” and not enough reviews of the film made mention of this.
ajay 03.08.10 at 3:30 pm
16: I can’t think of any examples of the second type of sleeper. The great illegals like Rudolf Abels had fake backgrounds and identities, but they weren’t just sitting there – they were running agents.
Philby was recruited before he joined SIS, so he was a sleeper, but he didn’t use a fake identity.
Henri Vieuxtemps 03.08.10 at 4:27 pm
Do you hope a spy of your own is less unreliable than a native? But you’re encouraging them to go native!
Hey, haven’t you seen Telefon? They are preprogrammed; all you need to do is to read them a short poem over the phone.
JoB 03.09.10 at 7:42 pm
Neil@14- Much as I don’t want to reminded about my activity on this thread, I don’t want to be hiding behind the excuse of ‘traveling’ not to have to own up to what I said:
“Rom Houben was diagnosed as conscious on the basis of the revised coma scale, backed up by a PET scan.”
Now we’re talking about the same case. Let me not dispute the boring details. I have no problem with coma scales. I do have a problem to diagnose somebody as ‘conscious’. It leads to mistaken assumptions (or at least non-proven – maybe non-provable) assumptions that consciousness is a thing that can be measured on a scale. This is the basic problem here. It is the basis on which the rest of the sweeping claims are based. It is the basis on which catholics in Belgium have used the research (without any neuroscientist ‘coming out’ to oppose it) to call into question euthanasia-laws over here. The bandying about of the word ‘consciousness’ (and much worse) is there – also in peer reviewed literature; and in any case neuroscientists seem particularly shy to come out & put (like you, for which I thank you) put some cold water on all this sweeping claiming.
Comments on this entry are closed.