Then, why use the phrase “Manchurian Candidate” Henry? That seems like your own straw-man of causality, doesn’t it? What Condi did before 200 is well known, at least on the face of it. And, that is that she was smack dab in the middle of Afghanistan for years. I’m not at all shocked to see her linked with Chinese power centers.

It’s like someone recently pointed out: why would you [tin-foil-hatters] assume Osama’s working for the Bush family? Why would it not be the other way around?

Even if that were totally true, that doesn’t make data-mining useless. It would be just another in a long line of useful law enforcement tools that doesn’t cut as harshly against the privileged. See prosecutors and the now Senator Kennedy. See jury trials and the now free OJ Simpson.

So the miners build a statistical analysis model (something I greatly distrust) to find bad guys, and out pops the names of a few Undersecretaries in critical positions. What do they do? Discard those names, of course, because “everyone knows” that “those people” couldn’t be suspects. Right.

And I would be reasonably certain that if we went back and ran various federal agencies’ and investment banks’ securities fraud models against the WorldCom data from 1995-2000 that Ebbers’ name would pop out as needing further investigation. For all we know it did at the time. Do you think anyone investigated as a result? As another commentor at Drum’s said, only pee-ons are targets for data mining.

But hey, I just do database, statistical modeling, and computer security work for a living. What do I know?

Cranky

Redwolf – sorry to say that you’re shit out of luck here – the politics surrounding data mining, data retention etc are a major part of my academic expertise. Doesn’t mean I’m necessarily right, of course, but I do have some idea of that whereof I speak.

Martin – the framing of the argument (which perhaps doesn’t come across clearly enough) is that Laura Rozen doesn’t quite get the engineering aspects of this; accordingly, she’s somewhat misinterpreting the politics. I think that Kevin’s take on this is the most convincing to me.

Cranky – point taken, but what is interesting about this kind of exercise isn’t that you can connect a, to b, to c, which yer common or garden David Horowitz can do. It’s that sometimes non-obvious connections can emerge. But obviously, if you want to figure out whether these connections are meaningful, you need to (a) apply some sort of reasoning (as many connections will be either spurious or irrelevant) and (b) do further research/investigation of the traditional variety, as appropriate.

The most credible accounts of how AD fingered Atta have to do with information from specific individuals, not publicly available data-bases. No amount of data-mining could have provided his name to Able Danger at that time: it was just old-fashioned human intelligence resources.

I don’t know if Drum is right about this, but it is worth keeping the possibility in mind: so far as the pros and cons of data-mining go, Atta may be a complete red herring.

A large part of the job that these tools take over is risk assessment. Whether it is credit card transactions, creation/checking of a no fly list, or a credential check for a government job.

And you can’t depend on the few humans still involved to override/further investigate these automated assessments, either because there’s too many instances to check, as in the credit card transactions, or because they would take a big personal risk to go against the system, like in the no fly list or the credential check.

Something like this is noted in the referenced article by DeRosa, but without much clues on how to resolve it.

So my opinion would be that

The most that data mining can do (and should be expected to do) is sometimes to highlight interesting and non-obvious relationships that might otherwise have escaped people’s attentions.

is a bit of a misunderestimation of what is being done with data mining and associated tools.

I am no fan of conspiracy theories, and I think the big one (the Bush family, the Carlyle Group) here is right out in the open. But if you start out down a path using a statistically-based tool; define your model; enter your data, and get results; and then discard certain results because you “know” they can’t be true – well, I am not sure why you are spending my tax dollars to do that in the first place.

Cranky

I find it interesting that after discussing data mining from, in effect, an engineering perspective focusing on its usefulness and practical limitations, in your close you switch to a bureacratic/political perspective (not, as Jerry Seinfeld would say, that there is anything wrong with that). Call me naive but the logic of your earlier discussion would seem to lead to a conclusion along the lines of:

“If Able Danger singled out Atta as one of a small group … the Pentagon has an opportunity for valuable progress against terror, even if it was sadly not taken advantage of. If Atta was one of many false positives, it looks like the Pentagon regrettably is stuck with the difficult, marginally effective, and often morally questionable intelligence methods of the past.

(I realize more or less all social phenomena can and should be described in both engineering/intrumentalist and bureaucratic/political terms. I guess its the unannounced switch from the middle to the end of the post that struck me.)