So, you’re saying that if we collect everyone’s metadata we can accurately identify people who are centrally involved in violent conspiracies to kill our troops and even overthrow our government? Sounds good to me!
ajay @ 12:54,
I admit that I had the same thought. Now, what would you do when the Tea Party [faction of the Republican Party], BoD of the NRA, abd the editor of “Guns & Ammo” magazine popped out of the analysis? Investigate them? Oh wait…
Seriously, this is a terrible argument against metadata collection. It’s like someone saying “No, of course the Royal Air Force should not have jet aircraft. Don’t you realise that if the Luftwaffe had had jets in 1940 they would have won the Battle of Britain?“
“Seriously, this is a terrible argument against metadata collection..”
I think you’re missing the distinction between ‘making an argument against’ and ‘explaining how it works’..(if this comment is directed towards the OP)
If KH’s intent was to make us feel better about metadata collection and analysis by explaining how it will work to keep us safe from violent conspiracies, then fair enough: mission accomplished.
Interesting and very much in point. But what do you believe the police now do, (and have done since the revolution?) in trying to identify people acting in concert to commit murder; or, to take a more current example, identify hedge fund managers who traded on inside information? They look for metadata connections and have always done so. Was the hedge fund manager a member of the same country club as a senior executive in the company where insider trading is suspected? Did they ever work at the same investment bank? Etc. Of course abuse is possible and may very well be occurring, a possibility that should not be ignored. And, there is much to criticize in the lack of transparency of the Obama administration. But wishing away the use of modern technology is not going to happen. Investigators have always looked at metadata.
Its like Hague saying “law-abiding Britons have nothing to fear” from this.
Is something easy to say when you define “law-abiding”. The tools will give you connections – what make you think you will get ONLY violent terrorists connections? Or that when you get a false positive it will not matter cause its Gitmo forever anyway (oh, yes, Obama is going to close it. Someday).
Basically there is nothing to prevent this for targeting you – the guys that decide WHAT patterns to look for are not going to tell you why.
If you feel comfortable with the Obama team handling this now, think if the “new” kind of Republicans get into power and what kind of filters would they use. Hey, lets use this for anti-abortion! Is murder so its terrorism, right? What about pro-enviroment activism? (Not that I doubt it is used for that now). Etc…
#12 So I guess if the redcoats got him they would have say move along citizen, no problem with you, send the rebels our regards. Tell them we are going to be late for tomorrow battle.
This is an amazing example of what can be done with what appears to be wholly benign information. (Note that part of the power of this is to select organizations which you, as the analyst, or the analyst’s boss, think you have reason to be suspicious of.) Nonetheless…
Is there a useful way to communicate the implications of this to, oh, I don’t know, Partick Leahy?
I don’t know these organizations in question, but were they public? Once you have been identified as member of the Enemies of London, the game is surely mostly over?
Took me a while to find out what the significance of Thomas Urann was though. Answer – very little. Minor attendee at big event. Wouldn’t like to be him if then was now.
You have proved that the name “Paul Revere” appears frequently on lists of names that were published in a book about Paul Revere. Roughly the equivalent of studying Tamerlan Tsernaev’s facebook friends, and concluding that the common element is that they all knew Mr. Tsernaev.
As is noted above, whether Mr. Thomas Urann was the real hero of the American Revolution, and successsfully hid his role from the world for the past few centuries, is a very interesting question.
Took me a while to find out what the significance of Thomas Urann was though. Answer – very little. Minor attendee at big event. Wouldn’t like to be him if then was now.
_That_ is a much better argument against metadata collection – if you could find someone like Urann who was plausibly connected to all sorts of rebel activity, but was in fact entirely innocent. (Not Urann himself; as a member of those groups, he probably wasn’t innocent of rebel activity.)
Like others, I’m not quite seeing the argument here. Yes, I you have identified a group of politically salient organizations, and you have lists of the members of those organizations, you can get useful information by looking at how they overlap. Is anyone surprised by this? And how does it affect the political arguments about the NSA program one way or the other?
It’s like if we were discussing the rules that should govern traffic stops, and someone pointed out that police cruisers can go really fast, with a clever explanation if internal combustion.
Also, even leaving aside the current political context I don’t see what this exercise proves. The actual result, as far as I can tell, is that there are ten or so people strongly linked to several clusters. One of those ten is a familiar figure in history books. Is this a higher proportion than for the population as a whole? Who knows. And is familiarity a good guide to importance? If you picked Paul Revere out as the key leader in the proto-American revolution would that be a good result, or a bad one, compared with other techniques you might use? Again, who knows. The fact that he’s the subject of a popular poem and a patriotic story we learn in childhood, does not seem dispositive.
Excellent explanation of network analysis, Kieran, thank you. I do not understand the political complaints. Does anyone really believe criticism of vast spying operations is or should be grounded in their failure to work? Of course they work! That is precisely the problem with them. And that and how they work (without listening to your phone calls) is what Kieran explains.
The whole point, for me, of the argument about programs like phone or internet metadata collection is that they demonstrate that the legal protections many felt they could rely on are utterly inadequate in the new world of big data. Our right to organize politically in opposition to authority – for example, our right to demand that the far-away king in London listen to our concerns – is fatally undermined by these data collection and analysis efforts. A police state where everyone is spied upon by a physical agent is easy to identify (there’s a lot of spooks). One in which some twenty thousand people (out of 300 million) with an array of computers do the same job is a new challenge.
And that and how they work (without listening to your phone calls) is what Kieran explains.
Indeed he does (and very cleverly- thank you Kieran), and as you intimated, that is exactly the answer to JW Mason. It may not be news to many of us here what you can do with metadata and pretty simple statistical methods (or the enormous scale on which you can do it if you have access to gobs of computing power), but I doubt very much that it is understood at all by John Q. Public. And it is specifically with the intent of misdirecting JQP’s attention that we get the kind of bland assurance offered by (among others) Obama that we needn’t worry because nobody is listening to our phone calls.
Also consider that Snowden was an employee, not of the government, but of a private contractor. As the leak itself indicates, the enormous reliance of the “security” establishment on private companies- another thing that is not well understood by JQP- creates its own layer of dangers.
Yep. The point is not having or not having jetcraft. The point is having or not having widespread spying on your own citizens and the rest of the world. The fact that it was not possible before and now is affordable doesnt mean is a good idea or the ethical problems of it went away.
Same as the new techniques of torture – oh, this is totally new, we dont lay a finger on them, no physical damage, so lets use it! Except for the fact that we are supposed to be the good guys that dont do torture, I dont see why not!
What this exercise shows is not who are the leaders of the movements. It shows who are the key players are that can make the most difference. Whether those players actually make use of that power is a different question, which this particular input data (and hence also analysis of it) can not answer. For that more specific intel, rather than meta data, is needed. The use of this can be very profound, because it shows that taking out just a few players (whether they are in fact real players or not) can significantly weaken a movement.
In fact, I am pretty sure that the CIA is using this type of analysis (though not exclusively) to select as targets for their drones. And we also know that collateral damage is considered by the administration as justified.
The problem is that the example misses one use of this, one that the whistleblower itself detailed in interviews.
Pick person A. You want person A to collaborate with you. Run query. Find what links are “bad”. Amplify them. Use them as blackmail. Now you got a source.
Want to infiltrate/manipulate new political movements? Here is your tool. You have dirt on anybody – years of dirt, searchable. Find the weak link. Maybe that guy that is in the middle of that graph is not really that important, but he is in the middle – so pressure him, and make him part of your ring of moles. Get info, plant misdirection, etc.
With ease of use, and private contractors access – why do you have the idea that somehow this is going to ONLY be limited to terrorism? Why dont you think a detailed map of everybody in, say, Occupy, will be of interest to some people to pick apart? Imagine tools like that in the hands of… I dont know, that Walker guy in Wisconsin, was it? Now how fast would the network of union members will be analyzed to find how to destroy them or cow them?
The process of traversing social networks for political repression is already going. It’s going like this:
1. Devote resources to chasing down an “important”, “violent”, “nonpolitical” crime, like breaking windows during a protest.
2. Find everyone in the social network that you think the protestors came from. Summon them to a grand jury, where they can’t refuse to answer questions for any reason, including the 5th. Ask them about all of their social contacts, who is friends with who, etc.
3. If they refuse to answer, put them in prison for up to 180 days, in solitary.
4. If they do answer, repeat step #2 with more people, who all of course now have an additional reason to distrust each other and not want to cooperate politically.
Computerized network analysis speeds up step #2 quite a bit. And of course that’s what it’s always been used for, and is used for — putting people with the wrong politics in jail.
The issue seems to be: does PRISM only look at metadata or content? If only metadata, then I’m not sure what the story is. Its legal for U.S. gov. to do it, end of story. Maybe it shouldn’t be, but I don’t see illegality and therefore I don’t see a scandal. If they can get at content, then scandal. But I am finding that part difficult to believe.
Re bigcitylib: So once something has been made legal (probably without even many proponents of the law having been aware of its full implications), the questions of whether it should be legal, or even whether the law should be modified to strengthen transparency, become completely uninteresting? That’s a perspective, I guess…
Who cares if something is a political “scandal”. Even asking that question means embracing the warped view of the DC establishment. The question is, what is the intelligence industrial complex doing? How are they doing it, and what are they using it for (e.g., is the meta-data used to identify human beings for elimination by drone)? Based on the information we have (due to heroic leaking despite government efforts to keep us in the dark), should they be doing it? (And just to be clear, it is not at all apparent whether this data collection is constitutional. If it was clear, why keep the programs secret so they can’t be examined by courts?)
No, I would not be in favor of this kind of law, and in fact up here in CDN popular sentiment shot down some legislation that would have granted our police what look to be similar powers. But what does the leak really tell us…that the U.S. authorities are using the tools they have been granted (ie to look at metadata)? To me that’s old news, and frankly not worth the current fuss. If they are indeed grabbing content from any email on the planet, then that’s something new and scandalous. But I find myself doubting this bit, for various reasons.
To most of us here it’s old news (and even so we had to rely on assumptions and guesswork to fill in details that have now been revealed). To the general public, not so much. The thing about l’affaire Snowden is that it has forced the news media to actually report on this stuff in a high-profile way.
OK then, we should ignore the whole thing because a newspaper engaged in a bit of hyperbole, something never before known to occur. Is there an actual point that you’re trying to make, or do you just want us to be impressed by your world-weary cynicism?
ajay: I’m pretty sure that mud man’s point was that the American Revolution did not intend the overthrow of the British government — just the independence of some of its colonies. Unless there was some extra-radical branch of patriots I’m not aware of.
And I think the point of the OP was, basically, you can learn a lot about someone’s political connections and beliefs from metadata, which rather weakens the value of the reassurances that “don’t worry, we’re only looking at metadata, not content.”
you can learn a lot about someone’s political connections and beliefs from metadata, which rather weakens the value of the reassurances that “don’t worry, we’re only looking at metadata, not content.â€
Beat me to it, but yes, this is exactly what I wanted to say.
Great explication. I’ve only started reading the referenced article, but perhaps Revere was such a good inter-cluster bridge because he was an artisan whose clients were mostly elite, and thus he was connected to both artisan and elite clusters.
Very nice piece of work! An introduction to the possibilities of social network analysis–and thanks for including the data and software. I’d only add the caveat to readers unfamiliar with the area that in practice it’s harder than it looks. For example:
He seems to bridge several groups in an unusual (though perhaps not unique) way. His name is Paul Revere.
Actually, it’s not that surprising. Here’s the simplest analysis I can think of: Sum the number of organizations each person belongs to; sort persons by the result. Paul Revere has the unique highest value, 5. (This is in part due to the choice of data.)
Samuel Peck, now, he’s interesting–he’s only in two of the organizations, but he’s in the middle of the network, and he scores highly on two of the three centrality measures. That is, there are all sorts of reasons we might find people interesting. Also:
I must ask you to imagine what might be possible if we were but able to collect information on very many more people, and also synthesize information from different kinds of ties between people! For the simple methods I have described are quite generalizable in these ways, and their capability only becomes more apparent as the size and scope of the information they are given increases.
As more information gets included, though, patterns become harder to see.
” Great explication. I’ve only started reading the referenced article, but perhaps Revere was such a good inter-cluster bridge because he was an artisan whose clients were mostly elite, and thus he was connected to both artisan and elite clusters.”
Nice – an elite artisan could have a unique social position.
Finding Paul Revere would not have been very useful, because the place was full of Paul Reveres. Would be easier to identify them by sectarian affiliation.
Well, come to think of it, I guess this is still how it’s done today. Metadata analysis is just fine tuning.
“In fact, I am pretty sure that the CIA is using this type of analysis (though not exclusively) to select as targets for their drones. And we also know that collateral damage is considered by the administration as justified.”
Absolutely nothing “imploded” and there was no “major cock-up”. If the Post made changes to its story it is due to two reasons: first, PRISM is no longer as highly-classified as it was b/c the DNI admitted its existence in order to then proceed to obfuscate (as bigcitylib demonstrates, with some success) about it; and second, the Post wanted to avoid pissing off tech companies and accepted carefully worded non-denial denials at face value. It is perfectly clear that there is an automated, assembly-line way those corporations use to allow the government access to used data and content that is worlds away from the imagined approach of considering and complying with each individual warrant separately. this automated, assembly-line way of access was described by internal NSA documents in exactly the way the Guardian reported.
I have to say that seeing liberals defend this massive spying operation justifies completely what some of us were pointing out before the 2012 elections about the difficulty of supporting Obama without illusions. Most people are simply not capable of doing so.
” Its legal for U.S. gov. to do it, end of story. ”
As a non-resident foreigner, and target of this program, with no rights whatsoever under US law, I don’t see things quite the same way.
I suspect quite a few others among the 6.7 billion or so non-US non-people in the world may take a similar or more hostile view. But, hey, it’s legal under US law. And, as we know, the US is the biggest baddest power in the world, so it can do whatever it likes – what could go wrong?
How would this type of analysis work though? In bigcitylib’s David Simon post the implication is that you input a number/s into the database and develop a series of connections from there..but if you were starting from the position of having a specifc number/s you wanted to investigate surely you wouldnt need a database this big, and could make those connections more conventionally
So is the idea that all calls coming from a specific region (it has to be originating from outside the US, right?) are scanned (or whatever) and developed into some sort of coherent network structure, and then….?
That’s a serious question to anyone, I’d be interested to know how it would work on this scale (which is quite different – I assume – from the OP which is dealing with much more specific and useful information)
On the PRISM thread, people were talking about this too — that if you’re not a U.S. citizen, you don’t even get rhetorical protection of rights. I mean, if the U.S. government wants to drone assassinate someone in Yemen and it’s a U.S. citizen, they have to say he’s a bad man. If not, they don’t have to say anything. Or if you kill a 16 year old boy who’s a U.S. citizen, you have to have an anonymous source call it “a mistake, a bad mistake” since he was born in Denver. If you kill a kid who wasn’t born in Denver, you don’t have to worry publicly about the PR mishap.
I do the rhetorical “But they’re spying on U.S. citizens!” thing as well, because I can’t imagine anything else coming close to working. But it would be hard not to feel contempt for anyone for whom it was the critical factor that got them to change from support to nonsupport of the program.
That the U.S. passed the Patriot Act is to its great shame. But this particular story started with the accusation that the U.S. gov was basically illegally decrypting every email on the planet with the help of corporate titans like Google et al. These claims now appear to be bullshit. All they’re doing is what the Patriot Act allows, and apparently all Google et al do is comply when they get subpoenaed. If we want to talk Patriot shortcomings that’s fine. But I object to transferring the outrage generated by a bullshit story to a real issue. Or, to be more accurate, I can’t sustain it.
I like KH little treaty because it illustrates the issue with what I believe is the much bigger scandal. People seem to be throwing to rather separate issues together and while they both relate to what the NSA is doing, I don’t think that’s helpful
1. The first Guardian story, FISA authority is used to compel _vast_ amounts of phone metadata, without any targeting – this isn’t the “all calls from a public phone” that D. Simon talks about – it’s metadata on all calls. This is much broader than anyone could/should have reasonably interpreted FISA authorities and having the whole “dataset” of Verizon – and presumably other carriers’ – connections. This is news – we had no idea this was going on – and it strikes me as excessive/scandalous. My suspicion is that the Udall/Wyden complaints relate to this and similar interpretations of Patriot Act and follow-up legislation.
2. The PRISM program. I don’t actually believe we know what’s up with this. There are essentially two versions of this out there – a) the version from the leaked presentation as elaborated by Greenwald et al – tech companies are cooperating with NSA by providing practically limitless access for the NSA to records as long as they’re “probably” foreigners and b) the semi-official version, as outlined e.g. by Armbinder: http://theweek.com/article/index/245360/solving-the-mystery-of-prism where PRISM is essentially the technical implementation through which tech companies comply with FISA laws by providing access to information they’re legally required (via court order and/or National Security Letter) to provide.
I think there are good reasons to believe one or the other is more plausible – I think Armbinder is right that the presentation could be describing version b) of PRISM and just glossing over technical details. On the other hand, I don’t think there is a particularly strong reason to trust the statements of government or tech companies on this, so we may very well be looking at version a) – which would be a major scandal, not least because by now we’d have both tech companies and the government blatantly lying. But if version b) is indeed what PRISM is, then there really is nothing new. This isn’t just something we “suspected” – we were fully aware that the government can and does compel user information from tech companies – the powers are specified in legislation and the fact that it happens e.g. in google transparency reports.
Sure, we can use it as an opportunity to discuss the appropriateness of the whole FISA construct – and I think FISA is indeed insane and the governments claim that it can’t be brought to court bc no one has standing an outrage – but in that case we should be talking FISA and not PRISM.
To answer my own question @50 then, I assume that you cant make use of this data in bulk, but it’s just a case of having access to it so you can use parts of it when needed..?
To anyone thinking metadata collection isn’t a big deal (not many here, hopefully): To blackmail Martin Luther King, the FBI used wiretaps, found out about his infidelity, and used it in an attempt to manipulate himself into committing suicide.
But today, with the dataset NSA has snagged from Verizon, with a simple machine learning program even I could probably identify call patterns characteristic of infidelity. God knows what the NSA could find out.
How’s that for privacy? With a simple analysis, couple of minutes on NSAs clusters, you could get blackmail info equivalent to what was used on MLK on millions of Americans. At that point it isn’t a question of whether they have abused it. Just the potential for abuse is too great to tolerate.
Also, worth pointing out as people have sneered that I’m “obsessed with infidelity” over these concerns: infidelity is just an example. It’s a rather common, rather closely guarded secret. But of course, it’s only an example – if such closely guarded secrets can be automatically identified, you can bet a lot more can be found. Perhaps stuff you didn’t even know about yourself (Think “X is susceptible to recruitment in political organization Y, and has an estimated 50% chance of joining it within two years, although he has presently not heard of it”).
” To answer my own question @50 then, I assume that you cant make use of this data in bulk, but it’s just a case of having access to it so you can use parts of it when needed..?”
I’ve seen this assumption made before (and have had to get rid of it myself) – it’s not the 20th century; terabyte databases are now common in industry, and the NSA has both the money and the tech skills to match anything in industry (and can hire from industry, as well). People have been working a lot on how to deal with massive data, and how to parallel process things, and this processing doesn’t have to be done in real time.
IIRC, the main Google computer site was supposed to have 100K computers running at any given time. And that was several years ago; anything Google learned/developed is probably either leaking out to the rest of the IT world, or openly available for hire.
And I’m sure that no matter how many computers, how many massive disk arrays the NSA wants, the computer industry will be glad to take their money.
I’ll wager that their biggest problem is sourcing materials from China; I imagine that their government, industry and private (hacker) interests would love to have some backdoors.
Actually, that’s potential storage. A yottabyte is a million exabytes, and there aren’t many exabyte storage systems around today. Here’s an article from January, 2012, announcing a 10 exabyte system:
Russ Kennedy, vice president of Cleversafe’s product strategy, said the entire system — with racks, networking equipment and Cleversafe software — would run in the “single-digits” billions of dollars.
As of 2012, at least, if 10 exabytes of storage cost $1 billion, then a yottabyte system would cost $100 trillion. I don’t think costs will come down fast enough for the NSA to be storing that much data any time soon.
1765 census shows Boston with a population of 15,500. It strikes me that this kind of data mining works a lot better with a small population.–the smaller the haystack, the easier it is to find a needle. Try picking Paul Revere out of a population of 320 million, and you might get different results. On other other hand, by 1772, the year of the hypothetical study, Revere was openly and publically involved in dissident politics–to find Revere, you wouldn’t have to data mine–just read the newspapers.
From KH’s linked post:
“I will show how we can use this “metadata†to find key persons involved in terrorist groups operating within the Colonies at the present time.”
I think that should read:
“I will shew how we can use this “metadata†to find key persons involved in terrorist groups operating within the Colonies at the present time.”
Even I’m wrong, “shew” looks more authentic to an ignoramus such as myself. (G.B. Shaw was still writing “shew” in the twentieth century.)
“As of 2012, at least, if 10 exabytes of storage cost $1 billion, then a yottabyte system would cost $100 trillion. I don’t think costs will come down fast enough for the NSA to be storing that much data any time soon.”
According to Wikipedia, world internet traffic was estimated at 21 exabytes/month, in 2010 (http://en.wikipedia.org/wiki/Exabyte#Usage_examples). So they’d need to spend $30 billion/year on storage. If the figure didn’t count compression, then they’d be down to a few-several $billion/year. Sounds doable.
OK then, we should ignore the whole thing because a newspaper engaged in a bit of hyperbole, something never before known to occur.
That “bit of hyperbole,” as you put it, allows the conventional wisdom on the story to become “In coup against democracy, Obama administration engages in unprecedented flouting of law and the Constitution,” rather than “Patriot Act and amended FISA continue to enable sweeping surveillance.” You think James Sensenbrenner isn’t delighted to join the pile-on over a Democratic administration using provisions that James Sensenbrenner personally inserted into a bill? Now we get to hear new variations on “Obama the Dictator” until a Republican president takes office and uses the unchanged legal framework to protect the Fatherland. (There is a bipartisan proposal in the Senate to make the FISA court more transparent, but “bipartisan” currently means “two Republican senators and seven Democratic senators,” which is probably insufficient to get it through the Senate, let alone the House.)
So yes, a framing that can be so easily spun into political point-scoring over Obama’s unique perfidy, while letting the laws that actually permitted it continue in force, might be considered counterproductive. There’s plenty to criticize and organize pushback around without turning it into “The Day the Republic Fell” from the Paulite hymnal.
Yes Obama is obviously in no way responsible. After all if the law he demanded be extended allows surveilance then obviously he has no choice but to order his intelligence minions to engage in it. It is really rather amazing what amount of nonsense liberals will spew to protect “their” president from being held responsible for his actions.
Also, from any reasonable point of view we – the people of this world – would be better off if only Republican administrations were allowed to engage in this massive spying operation. Because it would mean less spying. For this reason I am amazed the lesser evil crowd isn’t fully behind Sensenbrenner’s hypocrisy.
“I’ve seen this assumption made before (and have had to get rid of it myself) – it’s not the 20th century; terabyte databases are now common in industry….”
But do they have any way of getting any useful information/patterns out of all that data without an explicit starting point? (For example someone under surveillance etc as the centrepoint to build from) Or is it just there (at the moment) to be used in more conventional intel operations?
“But do they have any way of getting any useful information/patterns out of all that data without an explicit starting point? (For example someone under surveillance etc as the centrepoint to build from) Or is it just there (at the moment) to be used in more conventional intel operations?”
Remember, it’s not the 20th century anymore :)
My guess is all of the above. Start with known points, start with suspected points, start with random points. Pull patterns out and see what sort of patterns you find, and which ones look similar.
Maybe worth pointing out that the “nothing new” argument was central to the passage of the provisions of the 2012 NDAA, permitting indefinite military detention of US citizens. And the “entirely legal” defense also applies to NDAA. So, if you’re cool with the idea that some bad correlations in your phone call record could see you locked up without trial for life, “nothing new” and “entirely legal” seem to provide you with all the defenses you need.
Of course, for foreigners, it is indeed nothing new. For some time, the US state has openly claimed the right to kidnap, kill, spy on and torture anyone who is not a US citizen or located in the US, subject to no controls and with no possible redress after the fact.
“So yes, a framing that can be so easily spun into political point-scoring over Obama’s unique perfidy, while letting the laws that actually permitted it continue in force, might be considered counterproductive.”
I don’t think it’s counterproductive. Certainly the Democrats complained about Bush’s unique perfidy when he did the same things, and stopped complaining when a Democrat did them. And there aren’t enough people left for any change if we only take the ones who complained under both administrations. If half of the people who care about civil rights only care because of partisan reasons, well, you take your coalitions where you can get them.
@Rich A small consolation is that the consistent supporters of universal surveillance are also a minority. Majorities of Dems and Independents opposed universal surveillance under Bush, and, now that Obama’s in, Repubs are now evenly divided.
Undoubtedly this kind of analysis is a pretty powerful tool, which can still turn out some pretty flaky results, like those which are sure which book/movie/music you will enjoy (not).
I just read a WaPo column by Matt Miller in which he claims that ‘the Unknown Coder’ is the real hero, keeping us all safe, not Snowden, who he says is deluded.
The Unknown Coder. Has a certain ring to it, doesn’t it?
Ah yes, Matt “torture” Miller. Asserting that those who “keep us safe” allow deluded narcissists to work for them. So in order to deflect the accusation that his secret police buddies are evil he constructs a story in which they are dangerously incompetent.
For some time, the US state has openly claimed the right to kidnap, kill, spy on and torture anyone who is not a US citizen or located in the US
It’s a natural and unfortunate end point of the attitude that has seen the ubiquitous use of the word ‘Americans’ instead of ‘people’ in American political discourse for as long as I can remember.
The rest of us – we’re not really real. We’re just extras in the ongoing drama/farce that is the US of A.
I was prepared to find Matt Miller’s op-ed more or less reasonable. Whatever he’s done more recently (and what is happening to IT people in the military that they’re becoming alienated so frequently, are they being forced to stare at goats or something?), it seems obvious he was hired not because he was competent but because he had a military background and therefore met a security profile that values “military values” more highly than things like a high school education. But sadly, no.
I don’t understand the objection to using metadata. It sounds like something that’s supposed to make me antsy but it really doesn’t. In fact, I’d object because it sounds like a pretty numnuts way of spending finite [rueful laugh] resources. But an intelligence agency that was passive and only reacted to transgressions wouldn’t be of much use.
And I like the OP, all the points didn’t make sense to me, but I’m going to keep it around and look at it every so often and I bet I’ll get some insights.
JQ For some time, the US state has openly claimed the right to kidnap, kill, spy on and torture anyone who is not a US citizen or located in the US, subject to no controls and with no possible redress after the fact.
Not to defend what the US govt is doing re drones etc, I wd nonetheless like to add a bit of nuance to this. First w/r/t “spy on” — virtually all govts of any size around the world have spy agencies which spy on anyone they want in other countries prob. usu. w/o any controls but purely internal ones. So that doesn’t set the US apart. Removing “spy on” leaves “kidnap, kill, and torture”. But the official policy of the current US admin bans torture so that’s at least contestable (unacceptable practices are prob. continuing, but not as frequently). That leaves “kidnap” and “kill”. Kidnap and rendition cases seem to have been infrequent lately (or maybe just going on w/o any being revealed). That leaves “kill,” which is, above all, the drone program, which, yes, remains something one shd oppose.
Well Jeffrey good for you. I suppose you are one of those people who “has nothing to hide”. So what if the government can map out your whole life, find out where you (or at least your cell phone) has been at all times, identify everyone you are in any kind of electronic contact with, know which webpages you visit and when… As someone active in (perfectly legal at least according to those laws we are allowed to know) international solidarity work I’m not so sanguine. I’m not many steps removed from people who have been victims of grand jury fishing expeditions, people who have had their phones and computers confiscated and searched at the border,… I would be shocked if in some of those cases exactly such meta data collection and analysis hadn’t played a role in who was targeted. And yes, as a foreigner whose life could be seriously disrupted by the simple action of denying entry to my place of living and working (the US) all this makes me a bit nervous – which is part of its raison d’etre, I’d bet.
From the NYT: However, three-quarters said they approved of the government’s tracking phone records of Americans suspected of terrorist activity. Nearly the same number approved of the United States’ monitoring the Internet activities of people living in foreign countries.
Thanks for nothing, Americans.
LFC: the difference between the US and other governments that we probably all agree are often no more virtuous is that the one in the US has the ability to spy on massive numbers of foreign people. Not just spy on foreign governments, but to quote the term of the week “ordinary citizens” in foreign countries.
I implied that a majority of Americans would probably feel that way, in another comments section, and it was castigated as too horrible to mention. Many of these same Americans probably also realize that it is much more than “metadata”. In terms of effectiveness, you may be able to catch some individual young psychos who are too stupid to realize that The System is listening, but against organized terrorists it won’t work anyway. After Zero Dark Thirty, I imagine they have gone to paper courier + innocuous signals. So it will take 5-10 years to move everything into position for another clever attack, and some religionists already think in the long term. I think it might be wise for worldwide practitioners of comparative mysticism to start emphasizing that a peaceful transition to god-consciousness is possible in the practice of every religion. Unfortunately it is likely that many scientific academic types won’t have a clue as to what that last sentence entails.
” I think it might be wise for worldwide practitioners of comparative mysticism to start emphasizing that a peaceful transition to god-consciousness is possible in the practice of every religion. Unfortunately it is likely that many scientific academic types won’t have a clue as to what that last sentence entails.”
No, they’ll understand, but will be struck dumb by the idea. It’s sort of like peace in the Middle East, but moreso.
This whole ‘metadata is perfectly fine’ thing reminds me of the local campaign (well, ‘local’ like 75 miles south of here) where a church group has set up a 24/7 vigil outside a highway adult store. They camp out on the roads, on public grounds they’re allowed to occupy for N hours, and just change out the people keeping vigil. Whenever someone heads in to the store, they take photographs of driver and car, trace the license plate, and publicly post as much information as they could legally collect so that it pops up online. They look up your name and such and try to figure out where you work, and yes, they call your boss and inform them when and where you visited. It’s especially successful with truckers, whose vehicles often have individually identifying information. They then post information about your employer, trying to ensure that Google searches etc for that company also bring up the post about the employee’s transgression. (Last I checked they were looking into whether they were legally allowed to purchase web site advertisements that would put their materials at the top of anybody’s web search for the company or the person’s name.)
They’re working with metadata.
Now of course it’s obviously true that the people who drive into the store deserve to face exposure, public humiliation and should have to withstand an organized campaign to ensure they lose their jobs or at least have to answer to their bosses and colleagues about the places they visit in the evening. They have something to hide — especially the people who are disgusting enough to work there — so fuck ’em.
Here’s the problem: while I was there interviewing the vigilant, something happened twice: a car pulled in, drove around the building, and drove out again. Once, they sprang into action. Photos, the whole thing. And once, they didn’t move a muscle.
What the fuck happened? They computed an adjacency matrix in their head, more or less. I did ask about the SUV they ignored. “Oh, they probably didn’t know where they were going.” So why did they chase after the other one? “Oh, we won’t post it if they don’t go in. They were going to pull in to the store and we scared them off. But we keep things in case someone tries to come back and visit again, we can catch a pattern.” Somehow this was not a thing they contemplated. It just made intuitive sense, to the metadata collectors. Their internal algorithms guided their assessment and response.
That’s PRISM: they compiled and assessed metadata to construct a profile, to distinguish the incidental from the indicative. The difference in assessment was triggered, so far as I can tell, by the fact that the SUV driver was pretty old and the sporty car driver was pretty young. Or maybe it was the speed with which the vehicle entered vs the speed with which the vehicle left. Or maybe sporty car vs SUV. Or maybe someone who looked like a godly type vs… Fuck-all if I know. What I do know is that someone got a free pass and someone else didn’t.
People will never, ever be equally subjected to state intimidation. People will never, ever be equally subjected to state oppression. People will never, ever have equal standing before the law. And: people will make mistakes. They’ll make bad decisions. They’ll be stupid. They’ll drive into adult stores just to see what is there, even if they end up leaving horrified and put off by what they saw, or just disinterested. They’ll look into topics that . (My mom went on a serial killer biography and forensic investigator kick a few years ago. She was careful to always check out a book on how to write fiction along with the serial killer books, under the assumption that someone would be tracking, because she heard something something about libraries being investigating.) They’ll pursue weird hobbies without thinking of how it makes them look to a hypothetical person looking at everything. They’ll live lives as if they’re capable of maintaining independent identities in different communities, even if those identities would come into conflict with one another.
That’s because a degree of privacy is instinctual. It’s naturally assumed and expected. Simple proof: how does a person act when they genuinely come to believe that a potentially malevolent agent is tracking their actions? They don’t generally try to get clever like my mom did. (And when they do, they forget they’re not actually very good at deception. My mom’s ruse probably just made her look all the more suspicious.) They don’t (usually) say fuck this, I’m gonna live my life. They feel stifled. They give up. They shut down. They feel bad about themselves, about the hypothetical whole-self that is being constructed solely in order to determine how bad they are. They imagine what that whole-self looks like, with everything mashed together by someone who is specifically looking to see if there’s something wrong with them. They pull back. They disengage.
Metadata of this form collapses social spheres. It collapses who you are as a social being. It pushes you into accountability in every one of your social spheres for actions you take in any one of your social spheres.
Whenever you’re inclined to think about whether people have anything to hide, think about the legitimate things a person would decide to not do only because they anticipate those things might look bad to the metadata-analyst. They worry about being misunderstood by a system that does not and cannot hear them explain themselves. They have no control over the context. So they live as if hounded.
And that’s far more sane than paranoid. These analysis systems will have lots of false positives, in part because there’s no accountability for too many false positives. People who are not threats will look threatening. And from among those people, with exceptions rare enough to be notable, the already-disadvantaged will suffer consequences. And they know this. So they’ll live less full lives, and they’ll live more nervously.
Sorry for grinding my teeth a little about this. Maybe we’ll turn out to live in a society in which by some strange magic, agents like prosecuting attorneys never ever ever get to ever access to anything inferred from this data collection system. Otherwise we’re sort of all getting investigated by a prosecuting-attorney-type system all the time forever, with no ability to explain ourselves, to contribute to the context assembled by the metadata machine.
Your privacy is your right to context. It’s your right to exert control and discretion over the composition of your identities. That’s why it’s important. And the Paul Revere computation is an explicit demonstration of how metadata collection compromises that right — a far better example than the folks keeping vigil.
For that reason, I’d prefer a handicapped state authority that occasionally misses these intersections, to a state authority that aggressively profiles. So I think Kieran’s piece (as far as whimsical pieces go) is perfectly compelling, even if it does demonstrate the effectiveness of the method. Sorry, ajay.
After recovering from accidentally snorting beer out of my nose, I wish to point out that I raised a call for ecumenical outreach as early as 2003 on DeLong’s blog. And as I believe I said at that time, it is a damning failure of the world’s religions that they have not had enough insight and strength to meet the challenge of what is in some ways a religious crisis.
” These analysis systems will have lots of false positives, in part because there’s no accountability for too many false positives. People who are not threats will look threatening. And from among those people, with exceptions rare enough to be notable, the already-disadvantaged will suffer consequences. And they know this. So they’ll live less full lives, and they’ll live more nervously.”
One of the really important things to remember is this – there are limits, like if somebody ends up having a 100 person team shadow a person for a month for no good reason [1], but in general flagging and screwing over people will be easy to do, and 99.9% deniable and easy to get away with. Note that the no-fly list is still in existence, and nobody knows how to get off of it if you’re on it.
[1] With one obvious outcome being that the management explaining why they spent so much effort on somebody will decide that their best out is to bust them for something, even if they need to flat-out forge data.
BTW, this was from real life – a friend’s former co-worker was in a building with several people when a DEA SWAT team hit them (due to a phony call from an ex-girlfriend of one of the guys). The DEA didn’t find what they were looking for, but told them that the raid cost $25K, and that somebody was going to jail. They took in one of the guys for not paying child support.
“The DEA didn’t find what they were looking for, but told them that the raid cost $25K, and that somebody was going to jail. They took in one of the guys for not paying child support.”
Oh, great. This kind of thing is exactly what I’ve previously referred to as creating criminals through enforcement. It’s not that other countries have fewer people not paying child support. It’s that they don’t have huge amounts of money spent to put those people in jail.
And yes — I think that people who don’t pay child support should pay child support. Hopefully by having it garnished from their wages if that has to be done, not via jail. Or, if they can’t pay — they can’t pay.
This article informs me that poor people are often sent to jail for nonpayment of child support with no trial and no lawyer, because it’s a civil matter. And that people in jail for nonpayment of child support are about 2% of the prison population. That’s an incredible miscarriage of justice — basically, debtor’s prison — but I guess the people who think about it at all like it because it’s for the children, or something. Those people will be happy for us to have fewer people in jail as soon as they discover a category of crime that’s sympathetic, I suppose.
What Christian H said @86 regarding mass surveillance of foreigners.
Regarding torture and kidnapping, it’s true that the current Administration has disavowed one, and scaled down the other. But it has aggressively resisted any attempt by victims to seek redress, let alone any suggestion that those responsible should be sanctioned in any way. So, the US state maintains the claimed right to do those things, whenever it seems to the government of the day to be necessary.
That’s PRISM: they compiled and assessed metadata to construct a profile, to distinguish the incidental from the indicative.
I’m really confused by this. My understanding is that PRISM is a program to access the full content of accounts at tech companies. It has nothing to do with metadata.
The metadata comes from phone records that are obtained from wireless (and perhaps landline? Not sure – in any case wireless is more interesting/rich) carriers – presumably under the same general authority as PRISM, but is there any indication that they’re the same problem?
That’s PRISM: they compiled and assessed metadata to construct a profile, to distinguish the incidental from the indicative.
I’m really confused by this. My understanding is that PRISM is a program to access the full content of accounts at tech companies. It has nothing to do with metadata.
The metadata comes from phone records that are obtained from wireless (and perhaps landline? Not sure – in any case wireless is more interesting/rich) carriers – presumably under the same general authority as PRISM, but is there any indication that they’re the same program?
JQ: “But it has aggressively resisted any attempt by victims to seek redress, let alone any suggestion that those responsible should be sanctioned in any way. So, the US state maintains the claimed right to do those things, whenever it seems to the government of the day to be necessary.”
The US government did it and got away with it (a few enlisted men thrown in prison for several months counts as ‘getting away with it’ from the point of view of the rest of the government). The officials who ordered, conspired and committed the torture are all free and still in the good graces of the establishment, even when it’s known what they did. The legal, social and political precedents have been set.
‘ GUANTANAMO BAY NAVY BASE, Cuba — When the war court reconvenes this week, pretrial hearings in the case of an alleged al-Qaida bomber will be tackling a government motion that’s so secret the public can’t know its name.
It’s listed as the 92nd court filing in the death-penalty case against a Saudi man, Abd al-Rahim al-Nashiri, who was waterboarded by CIA agents.
And in place of its name, the Pentagon has stamped “classified” in red.
It’s not the first classified motion in the case against the 48-year-old former millionaire from Mecca accused of orchestrating al-Qaida’s October 2000 suicide bombing of the USS Cole warship off Yemen. Seventeen sailors were killed in the attack, and the prosecutor proposes to execute al-Nashiri, if he’s convicted.
Also on the docket for discussion this week is a classified defense motion that asks the Army judge to order the government to reveal information “related to the arrest, detention and interrogation” of al-Nashiri. By the time he got to Guantanamo in 2006, according to declassified investigations, CIA agents had held him at secret overseas prisons for four years during which, according to declassified accounts, he was waterboarded and interrogated at the point of a revving power drill and racked pistol.
But what makes the no-name government motion so intriguing is that those who’ve read it can’t say what it’s about, and those who haven’t don’t have a clue. Not even the accused, who, unless the judge rules for the defense, is not allowed to get an unclassified explanation of it – and cannot sit in on the court session when it’s argued in secret.’
“I’ve got as much to hide as the next guy, but when I use a public utility I don’t have a privacy expectation about that use.”
Bullsh*t. The situation is that the government has sweeping and secret powers to compel companyies to cough up information. They are making private utilities public (to them only).
The issue as a general one isn’t new. IIRC, it was before the SCOTUS in 79. And the particular kind of use was revealed back when Bush got AT&T immunity from prosecution.
It’s so public of an issue that the furor over it seems to me to be as contrived as the Benghazi hand-wringing. It’s a bit like Louie Renault in Casablanca being shocked to discover gambling at Rick’s.
Back when I studied graph theory it was widely considered a failed branch of mathematics, tucked away in the attic of obscure operations research methods, run out of hope that it would ever catch the imagination of the wider academic community.
Much later, when I asked two sociologists if they knew about graph theory one denied any knowledge, but provided me with a handout on “link analysis”, a formal method he himself had concocted which looked suspiciously like something from the first three pages of Frank Harary’s textbook, except with completely different vocabulary. The other one claimed he had heard of it but disagreed with its “rational actor approach”.
So I have to say it fills me with considerable pride to see how far graph theory has come in the hands of a new generation of sociologists. One might even say it has become just as important to the well-being of this great nation as waterboarding.
I watched kind of, sort of half-listened to a he-said, she-said Ping-Pong on the Chris Hayes cable news show, yesterday, and I could not tell you, who the Rightee or Leftee were — no one important, I’m sure; it was Chris Hayes, after all.
But, what I got out of it is that the Left’s position is faux moral outrage and deep concern, while the Right’s position is to just keep re-asserting the integrity and practical importance of the programs. The Right Guy’s gambit was to assert in a deep, manly voice that the Program (he used the singular), was designed to do good things, and ignorant people (who, it was implied, hate America and want us to be “unsafe” and “unprotected”) should just shutup. It was Andrew F, but let’s skillful, mixed in with a little Jack Nicholson aching to tell the handsome liberal screwup in a dress uniform, “he can’t handle the truth” about the tough guys standing a wall and doing what has to be done.
If you are going to be outraged, it should be about the faulty architecture of these programs, in which secrets and lies are two sides of the same counterfeit coin, being passed off as truth. Not faux outrage about insult to personhood or some such, but real outrage about secret efforts that damage the country, the world, the rule-of-law, democracy, etc. You know, little things like incredibly destructive, expensive, pointless and seemingly endless wars, which these jerkwads cannot seem to win or end.
Clapper gave away the existence of a British mole in Al Qaeda a few weeks ago, and instead of being fired and prosecuted, he created a scandal, investigating a Fox News reporter for leaks, implying in the process that the Fox News reported violated the Espionage Act.
Does anyone recall that, in George W Bush’s Administration, the three top officials in the CIA were forced to resign in a corruption scandal?
These things just seem to disappear down the memory hole, because no one on the Left appears capable of the act of memory and awareness it would take, to attack these “programs” as aspects of a completely incompetent organization.
No one on the Right or in the corrupt, complacent Center is ever, ever going to concede that there’s any moral value in your precious privacy. It is not, who they are. If you want to win the political struggle, you are going to have to go after the basic competence and instrumental integrity of the whole she-bang. Mussolini’s basic claim to fame was that he made the trains run on time. Authoritarianism, in its brutal expedience, claims to be efficient and effective. It’s a lie. To undermine the support of an authoritarian regime with people, who don’t care, per se, about brutality or expedience, you have to attack the “efficient and effective” claim directly.
Authoritarian regimes are corrupt and cover up galloping incompetence. The U.S. hasn’t been able to win or end the longest war in its history, in one of the poorest countries on earth, spending a multiple of that pathetic country’s GDP. None of these people have the slightest idea of what they are doing, or how to do it, and that’s what they want to keep secret. And, it is not secret; it is obvious.
112: their competence argument is, hey, we haven’t been hit in the homeland since 9/11. Only way it gets overturned is if we do get another terrorist attack, which will then create pressure to give away still more freedoms. Just sayin’.
Concentrating on individuals incompetence/evil nature itself, generally, just desolves in ‘faux moral outrage’.. anyway, we have seen consistent incompetence, globally, for a decade with no one held accountable, why is this narrative now going to become so successful?
“Also, from any reasonable point of view we – the people of this world – would be better off if only Republican administrations were allowed to engage in this massive spying operation. Because it would mean less spying”
Ronan(rf): Concentrating on individuals incompetence/evil nature itself, generally, just desolves in ‘faux moral outrage’..
The thing about a dialectic in which the “moral outrage” card is played, is that it depends on your interlocutor or counterparty playing along. If it doesn’t get something like the desired or expected response, you feel like your rhetoric has fallen flat, and you have to move on to some other strategy. In Freudian psychology, it used to be called transference and counter-transference; the therapist is trying to hypnotize the patient and the patient is trying to hypnotize the therapist. And, political debates, even, or especially, the staged for teevee ones and the ones that take place on op-ed pages, follow the same pattern. The argument is less about logic or evidence than about inducing a trance and piling memes onto emotionally loaded language and narratives — “moral outrage” being a particular template for emotionally loaded language.
Among the most effective techniques is to deliberately misunderstand or fail to respond. The Left gave up on arguing for fiscal stimulus or single-payer, because “no one” would engage in those conversations, or would engage only in wonky and pointless discussion of what was meant by “quantitative easing” (the idea that the “experts” have esoteric knowledge is almost as effective as the assertion of “secret” or “classified” knowledge) or would engage only by dismissing the argument as “politically impractical” (“as any smart person would just know, so I guess you are just not savvy”).
Consciously or not, you’ve done it, here, by referring to “individuals incompetence” [emphasis mine], when I attributed “incompetence” to an organization (and recommended disrespect for the efforts of that organization’s members and apologists).
Genuine moral outrage isn’t looking for a particular rhetorical response from others in conversation; it is looking for justice to be done and reason to apply. (Yeah, I know, good luck with that, eh!)
Faux moral outrage about the uses of information, which, maybe, ought to be private, but isn’t in our brave new digital world, doesn’t look to change anything. At best, it is looking to play a role as useful idiot, by reinforcing the moral narrative, which blames the state or government, as a “bad guy” and fits into the conservative libertarian narrative, by which private, corporate power is elevated and enhanced. If you want your “private” information to be private, then start thinking and working on ways to make your ignorant presumption about what is, or should be, private into an operative reality — don’t be campaigning to have the government go thru some empty bureaucratic ritual honoring an 18th century idealism, which Proctor & Gamble or Target stores don’t go thru.
Ronan(rf): anyway, we have seen consistent incompetence, globally, for a decade with no one held accountable, why is this narrative now going to become so successful?
It’s not. I’m saying stop expecting to change anything, by winning arguments with people, who are not arguing in good faith, people, who want to train you in learned incompetence.
Oswald Spengler had a great line: Optimism is cowardice.
We’re powerless. I’m powerless. They aren’t powerless, and that’s what we want to change, and have no apparent means to change. In the circumstances, we could try to tell the truth — it may not work to change anything, but it improves one’s moral condition and adds to one’s sanity.
“Consciously or not, you’ve done it, here, by referring to “individuals incompetenceâ€..”
I agree with you generally vis a vis highlighting organisational incompetence, it was just that after stressing orgnisational incompetence @112 you continued to show specific, individual incompetence/corruption (not to be arsey or as a gotcha, but just as, I think, it’s very easy to lose sight of the ‘organisation’ and attack the ‘organization’s members and apologists’ – which then allows them to write of the incompetence/corruption as specific to that individual/s)
“which blames the state or government, as a “bad guy†and fits into the conservative libertarian narrative..”
I personally do think the state deserves to be cast as the villan every now and again (although not in an ideologically Utopian Cowenesque nonsensical libertarian narrative)
It is types such as these who have no respect for the Unitary Executive and who will be demanding a Bill of Rights, or lacking that, a dissolution of the Constitution.
@Tom Slee, very interesting, thanks! Although computer science was one of the few disciplines that showed an interest in graph theory early on. Let’s hope the Big Graph does not drag up an unwanted connection between one of the co-authors and a blogger of the same last name, and the company she keeps.
@john c. halasz Ironically, the results in Kieran’s pamphlet are entirely driven by the fact that Revere was in all but two of the terrorist societies, more than anyone else in that cabal. If the RSA had just summarily incarcerated everyone who was in 4 or more of the listed societies, the Revolution might’ve been rooted out before it even started, with no need for graph theory.
This can’t be, Bruce, commenting because there are too, many commas.
More seriously: “These things just seem to disappear down the memory hole, because no one on the Left appears capable of the act of memory and awareness it would take, to attack these “programs†as aspects of a completely incompetent organization.”
Why would we want the security regime to become competent? That would be a catastrophe. So the top three officials in the CIA were corrupt. Think of what they’d do if they weren’t corrupt. Is it in anyone’s interest to call attention to this?
“We’re powerless. I’m powerless. They aren’t powerless, and that’s what we want to change, and have no apparent means to change. In the circumstances, we could try to tell the truth — it may not work to change anything, but it improves one’s moral condition and adds to one’s sanity.”
You write things like this, but then you write other things which imply that you don’t really believe it. Why do you call it “faux outrage”? Maybe the people saying this kind of thing are really outraged. Why should they adopt a tactic of talking about CIA competence when that’s not going to work either and they could care less about it?
I would like a society in which fewer people were imprisoned. Collecting data on everyone helps the people doing the imprisoning. Should I pretend that I’m outraged at Target?
The Revere exercise is interesting – and the write up is hilarious – but unless you’re starting with what seem to be a fairly focused set of names, I’m not sure the result is easy to duplicate.
As to the metadata program itself, the only thing clear at the moment is that we’re not clear on what it is. It’s possible, but unlikely, that the NSA is doing a social network analysis over every phone number in the country. The court would be unlikely to approve such a program, Congress would be unlikely to fund such a program, and I’m guessing the NSA has enough trouble processing data without adding that kind of monstrosity to the heap. And a program of that size which generates lots of false positives is worse than nothing at all, as the FBI simply wouldn’t be able to keep up with the flow of numbers to investigate.
What seems more likely is that the database is being used more as a data preservation tool than as a standalone input for data analysis. It would enable investigators, once they identify a number as belonging to someone of interest in a terrorism or counterintelligence investigation, to quickly trace out various features of social networks connected to that number. Investigators could also use knowledge of a particular organization’s operational methods to attempt to determine other relevant phone numbers, e.g. a particular type of use of temporary mobile phone, once they have established that a certain number is of interest.
I suspect that the databases maintained by communications providers are not well suited to that type of analysis. To conduct that analysis, therefore, the government would need to construct a new database with possibly relevant records.
This would then set up an argument for the government to request a particular database from a company as relevant to an ongoing investigation. But, such a request is unusual, and to be palatable to the court – and to Congress, and to the Justice Department, among others – certain privacy protections would have to be agreed to. For example, it may be forbidden for the NSA to simply look for signatures in the records without having a particular number unearthed as relevant that can furnish a starting point. It may also be forbidden for the database as a whole to be cross-referenced with other personal identifying information, except for cases in which one has developed a particular suspicion of a given number.
And this would all fit with what Senator Feinstein, among others, has said publicly about the program:
Well, because they then have what’s a telephone book of the numbers and if, through another way, information comes to the FBI that there is reasonable suspicion that a terrorist act, conspiracy, planning, carrying out, is going on, they can access those records. The records are there to access.
If that in fact is what the program is, and obviously it’s just a guess on my part, then there are still multiple questions to be answered, of course. It does reflect an expansive view of what the government can order produced as relevant; there’s an enormous question of whether the privacy protections in place around the database are sufficient. And aspects of those questions might be difficult to answer without divulging sensitive information about how the NSA connects a relevant number to other numbers, e.g. how knowledge of the operational methods of an international terrorist organization informs the NSA’s analysis. If I know the limits of the NSA’s analysis, then I can devise methods that will stand a better chance of escaping that analysis; if I don’t know, then it becomes more difficult for me to strike the appropriate balance between risking the operational security of whatever terrorist or intelligence action I’m involved in, and accomplishing that action with the limited time/resources at my disposal.
What I would like to see here, therefore, is for the government to disclose the strictly legal reasoning behind the production of the databases, and to disclose more regarding the limits on the use of the database constructed. I think that can be done without giving away information that could be used by an adversary to evade detection. As to how useful that database has been to the government, my best guess based on Senator Udall’s qualified negative argument is that it has quickened previous investigations, but that the information it produced would arguably, at some point, have been discovered via other methods (including perhaps a slower crawl through the records held by telephone companies).
Re: surveillance of foreign communications – as LFC pointed out already, this is done by every government with the capability to do so. What sets the US apart are two things. First, as has been pointed out by christian_h, the US has greater capability (for now) to do so; second, what has not been pointed out, the US does not use this ability to conduct corporate espionage, i.e. the US is not listening in on Airbus and then feeding the information acquired to Boeing. This is in contrast to other nations, particularly those with more active government interest in one or more companies, who have a policy of doing precisely the opposite. If the US does not use such intelligence for corporate espionage, imagine how much less use it has for peering into the personal information of billions of people producing obscene amounts of data who have absolutely nothing to do with national security. Unless you’re selling advertising, or maintaining a Great Firewall, you want targeted collection, not indiscriminate trawling.
RPuchalsky This can’t be, Bruce, commenting because there are too, many commas.
Are you just noticing, RP, for the first time that BW uses commas where I think many or most people wouldn’t? I don’t think that’s relevant to judging the worth of his comments, which sometimes are on the lengthy side for my taste but sometimes v insightful.
It wasn’t intended as a grammar flame, certainly not given how badly I mistype things. And yes, I think that his comments are insightful. There just did seem to be a lot more commas than usual.
Andrew F.: “What seems more likely is that the database is being used more as a data preservation tool than as a standalone input for data analysis. It would enable investigators, once they identify a number as belonging to someone of interest in a terrorism or counterintelligence investigation, to quickly trace out various features of social networks connected to that number”
That’s exactly what I wrote a few threads ago. The problems and / or benefits of data mining are not the point; the point is to dig up something on anyone who you want to target. Of course I don’t think that this is going to be retracted to terrorists. Or rather, “terrorists” will be defined as anyone who important interests dislike.
@119:
Sometimes the government is the bad guy, even if conservative libertarian types say so. As for “faux” outrage – as Rich says above, there’s nothing faux about it (most of it, anyway; Sensenbrenner is probably faking it).
Rich Puchalsky: Why would we want the security regime to become competent? That would be a catastrophe.
Hmmm. Let me think about that. . . . Oh, this is tough. . . . What a puzzler, this one is! . . . oh, wait! I remember: actual security!
And, they say, you can’t write sarcasm! Ha!
I suppose words like, “rational”, “competent” and “efficient” lose reliable connotations, let alone denotations, after a while, as they are stretched and pulled in competing narrative arguments. If you learn a story, where the hero is saved by a palsy in the hand of the state, with the state as villain, I guess you can say, “palsy is a good thing!”
Do we, thereafter, have to assent to the proposition “palsy is a good thing”? It doesn’t seem like a balanced view, concerning the usefulness of leviathan, or the nature and necessity of social cooperation and organization.
Going to war on false pretenses. Sending someone to prison on fabricated evidence. Failing to prosecute criminal banksters. These are things I can imagine real outrage about, but I have to imagine it, because there frequently isn’t much in evidence.
The CIA has nothing to do with security. When the CIA was competent, its business was destroying the left. For instance, it helped Pinochet to power, causing the torture and deaths of tens of thousands of people. That’s what you want to bring back? I think that the people from countries other than the U.S. commenting here might have another view.
And the security agencies in general have nothing to do with security for you or me. They exist in order to keep the powerful in place, both in foreign and domestic affairs. They are an adjunct of the right wing, nothing more. I don’t think that sharply scaling them back would result in any catastrophe.
Andrew F: “The court would be unlikely to approve such a program, Congress would be unlikely to fund such a program, and I’m guessing the NSA has enough trouble processing data without adding that kind of monstrosity to the heap. And a program of that size which generates lots of false positives is worse than nothing at all, as the FBI simply wouldn’t be able to keep up with the flow of numbers to investigate.
Andrew, stop lying. The FISA court approves the overwhelming proportion of requests, the judges all have to be approved by the national security apparatus, Congress sends several 10’s of $billions into a black budget, and can’t even keep track of the open budget. For the black budget the Congressmen/Senators can’t even write stuff down or tell their staffs or get any external help whatsoever, so the problem is vastly more difficult.
As for keeping track of phone calls, that’s the sort of thing which marketing researchers and data miners have been working on. It’s not the 20th century anymore; putting together thousands of computers and terabytes of storage is now casual. Outsiders have been working on analyzing vast datasets, and the NSA has been believed to have a technological edge in these matters for decades.
As for false positives, we just spent a trillion $ on a politically desired false positive (Iraq), which didn’t harm the elites one bit. And in most cases, a positive just means that a person is on secret black lists; action can be taken automatically to f*ck a person over (No Fly list, credit rating goes to sh*t, various background checks return bad stuff,…).
As well as class oppression, secrecy is, to a very large extent, about protecting the powerful from embarrassment. In the present case, the secrecy of the domestic programming arguably made it less effective as a tool for intimidating the population, and no more effective as a way of catching terrorists, but its exposure has caused big political problems for people from Obama on down. That was even more clearly true of Wikileaks.
I’m saying I am not sure I would concede that the CIA was ever competent, (or “competent” given that we obviously assign different narrative contexts, and, therefore, meanings, to the word). (Did I mention that every Soviet the CIA ever turned was killed, because the CIA could not keep a secret?) And, I wouldn’t score the overthrow of Allende and installation of Pinochet as a win for American foreign policy, though, clearly, some of those in power in the U.S. were getting what they wanted, or thought they wanted.
I’m pretty sure you and I are living on the same plane of reality, but we have differing narrative frames, and those frames have hijacked key abstract terms.
If you don’t have the power to displace the right wing in foreign or domestic affairs, you don’t have the power to scale back, as you put it, the security agencies, except in the way and manner of the right wing’s choosing. And, you think this is going to turn out well?
In a fit of absent-minded neglect to understand what is going on around us, we have managed to create a digital virtual reality, in which it is damnably hard to hide anything, true or false. [An acquaintance of mine nearly had his life ruined, when someone viciously implied in comments on some blog somewhere that the charity he founded to benefit his hometown in the Philippines was a cover for child molestation — pretty soon Google returned “child molester” to a search on his name (which was not particularly uncommon) and his employer wanted to fire him (he had a union job, luckily) and his landlord evict him. All without the NSA lifting a hand, as far as anyone knows.] Very little is operationally private, or deeply verifiable, which is, certainly a set of problems confronting the security agencies — and damn annoying, when some community college graduate, somewhere, decides to put you on list of people forbidden to fly a commercial airline, for example. But, also annoying to people, who are screened from job candidacy by credit reporting or drug testing.
And, our brilliant Lefties think “transparency” will magically solve all problems, as if the billionaire, who buys a candidate really cares, and no employer will ever check on line to see which Party or candidate, you donated to.
There’s a real political problem with the “secrecy” of the classification scheme, which tries to make it a crime to disclose information of genuine interest in a democracy. And, there’s a real problem with the system of personal identification. And, then, there’s the vicious, out-of-control plutocracy . . .
I don’t think we will be able to make things better, by weakening institutions. If that turns out to be the course of least-resistance, I would ask, why? It seems to me, in general, that our problems have as a common thread the deterioration of institutions, and the emergence of a self-interested elite with no understanding of, or sense of responsibility for, maintaining or managing institutions.
I think that, because I’m an institutionalist in my thinking, and I guess I have a bunch of narratives in my head, that make sense of those very abstract terms. What’s it mean to say that an institution is “strong” or “stronger”? It’s an ambiguous metaphor at best. I hope it means something more than, “I prefer this”. How would I handle discussing the “strength” of slavery as an institution in the antebellum South? I don’t know. I don’t like it, but it was a strong set of institutions, in legal, economic and sociological senses.
Just one of many things that make it hard to communicate. Even with lots of commas.
I think that there’s more than a problem of narrative frames here if you think that the CIA has never been competent ever in its history, and that it can be made competent. Whatever your definition of competence is, that suggests that it’s unattainable.
And you write that “some of those in power in the U.S. were getting what they wanted, or thought they wanted”, yes, that’s what competence means. Governmental agencies aren’t supposed to determine policy. But governmental agencies can have skill sets, inherent in their design and history, that make them good at some things and bad at others. U.S. security agencies are not designed to provide actual security.
“I don’t think we will be able to make things better, by weakening institutions. If that turns out to be the course of least-resistance, I would ask, why? It seems to me, in general, that our problems have as a common thread the deterioration of institutions, and the emergence of a self-interested elite with no understanding of, or sense of responsibility for, maintaining or managing institutions.”
So these institutions will fail.. It’s not like we on the left really have the power to save them anyways. Let them burn.
Or, look — sorry to double-comment, and with basically just a link, but if you actually care what I think about this I wrote it in a poem a while back. There’s no better way for me to say it than that.
Raymond Geuss points out that we “relate to other people not merely in terms of what they have done to us or are doing to us, but also with regard to what they will or could do to us. If I have certain effective powers, these may have a sufficiently intimidating effect on others that I get my way without ever needing actually to exercise these powers. So if we wish to understand how human action in a certain society comes to be coordinated, how some individuals or groups bring it about that others embark on certain courses of action or refrain from embarking on others, one of things we will need to take into account is not just who actually does what to whom, but also who has what powers, i.e. who could do what to whom for whose benefit. One must also take account not only of what powers an individual or group actually has, but also of who those powers are perceived, and what powers agents are, rightly or wrongly, thought to have by others…Given the role that intimidation can play, one important ‘power’ that an agent, whether an individual or a society, can have is the ability to control how others perceive its powers or what they imagine these powers to be.” Philosophy and Real Politics.
Some significant minority of Americans think that since almost any American could be shown to be suspicious by data mining–again the belief does not have to be true to take hold– the government will use the technology for political purposes. Today right-wing libertarians are opposed to PRISM and such programs; when Christie is President, the opposition will come from elsewhere.
Bruce: ” And, I wouldn’t score the overthrow of Allende and installation of Pinochet as a win for American foreign policy, though, clearly, some of those in power in the U.S. were getting what they wanted, or thought they wanted.”
That is a success for American foreign policy, because the people in power getting what they want *is* American foreign policy.
Barry, you’re letting cynicism drive you to conceding the whole argument. Fuck the people in power. The US foreign policy establishment has dragged the good name of Americans in the mud. Their criminal activities are criminal, whether or not they get away with them.
1 – federal judges are exempt from clearance requirements. They have access by virtue of their position. The CJ of the Supreme Court appoints the FISA court judges – no one else.
2 – it’s likely that most search warrant applications presented to judges period are approved – this is because the police and DA are very familiar with what makes for a successful and unsuccessful application, and so the population of applications has been weeded in advance of those most likely to fail under judicial scrutiny. In general, the more resources and time it takes to prepare an application to a judge, the more certain one will want to be that the application will succeed.
Let me put this another way. Most tax returns to the IRS are accepted and never audited. That does not mean that the IRS is simply a rubber stamp. It does mean that the standards as to what constitutes an acceptable and legal return are largely clear, and so filers are able to submit acceptable returns.
3 – the number of requests approved, disapproved, and modified can be deceiving. Not all requests – as certainly recent leaks have shown – are created equal. And there are a couple of cases pending asking that a 72 page decision by the FISC in 2011, finding certain aspects of a government surveillance program to be in violation of the 4th Amendment, be unsealed – so perhaps we’ll learn more.
4 – Expensive false positives don’t engender enthusiasm. You provide the example of the Iraq War, something absolutely no one is eager to repeat. A program that generates enormous numbers of false positives – too many for anyone to investigate – is simply a distraction. As to the cost, a telephone number can’t simply be cross-referenced to a name and then thrown on a no fly list and forgotten. The FBI or a grand jury must request the billing records to determine the name associated with the number. The name in those records must then be investigated. Reports have to be written. This all takes time, personnel, resources. If there is a program that emits a flood of false positives, it simply won’t be possible to do anything with the results by themselves.
Rich, you give the impression of someone who thinks that most personnel at the FBI, CIA, NSA, etc., are engaged in monitoring peaceful political dissidents in the US, sifting for dirt by which to prevent them from advancing politically.
Even if you think that some significant percentage of them are – which I don’t, given the reforms that have been added over the years since the 1960s and 1970s – you’d concede that at least most of those institutions are actually concerned with either crime (in the case of the FBI) or national security, right?
As to why you’d want strong institutions in a certain sense, it’s because you’d want FBI agents, CIA officers, etc., to be institutionally loyal, and not personally loyal. The latter leads to corruption and abuse of power; the former leads to fidelity to an institution’s rules. Obviously if those rules permit things we’d consider gross injustices, institutional loyalty isn’t much of a good thing. But the rules governing the FBI and CIA are actually pretty good; and the existence of institutional loyalty means that we are able to control the behavior of those institutions by controlling the rules, a not unimportant feature in a democracy.
“As to why you’d want strong institutions in a certain sense, it’s because you’d want FBI agents, CIA officers, etc., to be institutionally loyal, and not personally loyal.”
Loyalty is not competence. It’s possible to imagine a CIA in which everyone was loyal to the rules both in writ and in spirit, and in which everyone was even personally sympathetic, but in which the organization found itself overthrowing democratically elected foreign governments because it turns out that that’s all they’re really good at. Do we want them to be competent at that? It rather depends on whether you think that’s a good idea, even as authorized by our own democratically elected government.
Can we imagine them being competent at something else? Of course people can imagine any number of alternate histories. But in real history, prior events have already happened.
I shouldn’t oversimplify, though. If the CIA wanted to use its demonstrated ability to propagandize via art movements, I wouldn’t object.
” federal judges are exempt from clearance requirements. They have access by virtue of their position. The CJ of the Supreme Court appoints the FISA court judges – no one else.”
Riiiiiiiiiiiiiiiiiiiiighhhhhhhhhhhhhhhhhhhhhhhhhht. Non-cleared judges get to see high-level secret data.
“Expensive false positives don’t engender enthusiasm. You provide the example of the Iraq War, something absolutely no one is eager to repeat.”
Except for right-wingers, and those who’d profit by it.
” A program that generates enormous numbers of false positives – too many for anyone to investigate – is simply a distraction. As to the cost, a telephone number can’t simply be cross-referenced to a name and then thrown on a no fly list and forgotten. ”
Lie. Or course it can. Or worse, thrown on a no fly list and then flagged and f*cked with ***by machine*** in whatever other ways people can come up with.
“The FBI or a grand jury must request the billing records to determine the name associated with the number. The name in those records must then be investigated. Reports have to be written. This all takes time, personnel, resources. If there is a program that emits a flood of false positives, it simply won’t be possible to do anything with the results by themselves.”
You keep asserting that secret programs are bound by laws and ordinary procedures.
For example, to the best of my knowledge if you’re on the no fly list, that’s it. No appeal, no justification needed.
{ 150 comments }
adam.smith 06.10.13 at 6:27 am
this is awesome
garymar 06.10.13 at 11:12 am
The publick fupport our effortf, I am fure.
polyorchnid octopunch 06.10.13 at 11:53 am
Very nice. That article deserves wide distribution.
Ronan(rf) 06.10.13 at 12:39 pm
Amazing. I wish I could make graphs like that. I’d just do it all the time. About everything
ajay 06.10.13 at 12:54 pm
So, you’re saying that if we collect everyone’s metadata we can accurately identify people who are centrally involved in violent conspiracies to kill our troops and even overthrow our government? Sounds good to me!
Cranky Observer 06.10.13 at 1:23 pm
ajay @ 12:54,
I admit that I had the same thought. Now, what would you do when the Tea Party [faction of the Republican Party], BoD of the NRA, abd the editor of “Guns & Ammo” magazine popped out of the analysis? Investigate them? Oh wait…
Cranky
ajay 06.10.13 at 1:27 pm
Seriously, this is a terrible argument against metadata collection. It’s like someone saying “No, of course the Royal Air Force should not have jet aircraft. Don’t you realise that if the Luftwaffe had had jets in 1940 they would have won the Battle of Britain?“
Ronan(rf) 06.10.13 at 1:37 pm
“Seriously, this is a terrible argument against metadata collection..”
I think you’re missing the distinction between ‘making an argument against’ and ‘explaining how it works’..(if this comment is directed towards the OP)
ajay 06.10.13 at 1:43 pm
If KH’s intent was to make us feel better about metadata collection and analysis by explaining how it will work to keep us safe from violent conspiracies, then fair enough: mission accomplished.
Bob W 06.10.13 at 1:44 pm
Interesting and very much in point. But what do you believe the police now do, (and have done since the revolution?) in trying to identify people acting in concert to commit murder; or, to take a more current example, identify hedge fund managers who traded on inside information? They look for metadata connections and have always done so. Was the hedge fund manager a member of the same country club as a senior executive in the company where insider trading is suspected? Did they ever work at the same investment bank? Etc. Of course abuse is possible and may very well be occurring, a possibility that should not be ignored. And, there is much to criticize in the lack of transparency of the Obama administration. But wishing away the use of modern technology is not going to happen. Investigators have always looked at metadata.
Barry 06.10.13 at 1:46 pm
Sweet post. I was amazed at what linear algebra I can accomplish.
Jesús Couto Fandiño 06.10.13 at 1:52 pm
Its like Hague saying “law-abiding Britons have nothing to fear” from this.
Is something easy to say when you define “law-abiding”. The tools will give you connections – what make you think you will get ONLY violent terrorists connections? Or that when you get a false positive it will not matter cause its Gitmo forever anyway (oh, yes, Obama is going to close it. Someday).
Basically there is nothing to prevent this for targeting you – the guys that decide WHAT patterns to look for are not going to tell you why.
If you feel comfortable with the Obama team handling this now, think if the “new” kind of Republicans get into power and what kind of filters would they use. Hey, lets use this for anti-abortion! Is murder so its terrorism, right? What about pro-enviroment activism? (Not that I doubt it is used for that now). Etc…
mud man 06.10.13 at 2:16 pm
Paul Revere didn’t intend or participate in a conspiracy aimed at overthrowing His Majesty’s government.
Jesús Couto Fandiño 06.10.13 at 2:28 pm
#12 So I guess if the redcoats got him they would have say move along citizen, no problem with you, send the rebels our regards. Tell them we are going to be late for tomorrow battle.
Donald A. Coffin 06.10.13 at 2:32 pm
This is an amazing example of what can be done with what appears to be wholly benign information. (Note that part of the power of this is to select organizations which you, as the analyst, or the analyst’s boss, think you have reason to be suspicious of.) Nonetheless…
Is there a useful way to communicate the implications of this to, oh, I don’t know, Partick Leahy?
Zamfir 06.10.13 at 2:54 pm
I don’t know these organizations in question, but were they public? Once you have been identified as member of the Enemies of London, the game is surely mostly over?
Katherine 06.10.13 at 3:18 pm
Took me a while to find out what the significance of Thomas Urann was though. Answer – very little. Minor attendee at big event. Wouldn’t like to be him if then was now.
arthur 06.10.13 at 3:48 pm
You have proved that the name “Paul Revere” appears frequently on lists of names that were published in a book about Paul Revere. Roughly the equivalent of studying Tamerlan Tsernaev’s facebook friends, and concluding that the common element is that they all knew Mr. Tsernaev.
As is noted above, whether Mr. Thomas Urann was the real hero of the American Revolution, and successsfully hid his role from the world for the past few centuries, is a very interesting question.
.
mud man 06.10.13 at 4:12 pm
#13 That would be the point. So sorry about the accidental discharge Mr. Revere, but you did attempt to assault our officer! The officer said so!
Maximum effectiveness with minimal exposure and maximum deniability. Who will rid me of this troublesome silversmith?
ajay 06.10.13 at 4:25 pm
Paul Revere didn’t intend or participate in a conspiracy aimed at overthrowing His Majesty’s government.
Except for the American Revolution.
ajay 06.10.13 at 4:30 pm
Took me a while to find out what the significance of Thomas Urann was though. Answer – very little. Minor attendee at big event. Wouldn’t like to be him if then was now.
_That_ is a much better argument against metadata collection – if you could find someone like Urann who was plausibly connected to all sorts of rebel activity, but was in fact entirely innocent. (Not Urann himself; as a member of those groups, he probably wasn’t innocent of rebel activity.)
JW Mason 06.10.13 at 4:31 pm
Like others, I’m not quite seeing the argument here. Yes, I you have identified a group of politically salient organizations, and you have lists of the members of those organizations, you can get useful information by looking at how they overlap. Is anyone surprised by this? And how does it affect the political arguments about the NSA program one way or the other?
It’s like if we were discussing the rules that should govern traffic stops, and someone pointed out that police cruisers can go really fast, with a clever explanation if internal combustion.
JW Mason 06.10.13 at 4:31 pm
I s/b if.
JW Mason 06.10.13 at 4:42 pm
Also, even leaving aside the current political context I don’t see what this exercise proves. The actual result, as far as I can tell, is that there are ten or so people strongly linked to several clusters. One of those ten is a familiar figure in history books. Is this a higher proportion than for the population as a whole? Who knows. And is familiarity a good guide to importance? If you picked Paul Revere out as the key leader in the proto-American revolution would that be a good result, or a bad one, compared with other techniques you might use? Again, who knows. The fact that he’s the subject of a popular poem and a patriotic story we learn in childhood, does not seem dispositive.
Sure looks cool tho.
christian_h 06.10.13 at 4:44 pm
Excellent explanation of network analysis, Kieran, thank you. I do not understand the political complaints. Does anyone really believe criticism of vast spying operations is or should be grounded in their failure to work? Of course they work! That is precisely the problem with them. And that and how they work (without listening to your phone calls) is what Kieran explains.
The whole point, for me, of the argument about programs like phone or internet metadata collection is that they demonstrate that the legal protections many felt they could rely on are utterly inadequate in the new world of big data. Our right to organize politically in opposition to authority – for example, our right to demand that the far-away king in London listen to our concerns – is fatally undermined by these data collection and analysis efforts. A police state where everyone is spied upon by a physical agent is easy to identify (there’s a lot of spooks). One in which some twenty thousand people (out of 300 million) with an array of computers do the same job is a new challenge.
Steve LaBonne 06.10.13 at 5:08 pm
Indeed he does (and very cleverly- thank you Kieran), and as you intimated, that is exactly the answer to JW Mason. It may not be news to many of us here what you can do with metadata and pretty simple statistical methods (or the enormous scale on which you can do it if you have access to gobs of computing power), but I doubt very much that it is understood at all by John Q. Public. And it is specifically with the intent of misdirecting JQP’s attention that we get the kind of bland assurance offered by (among others) Obama that we needn’t worry because nobody is listening to our phone calls.
Also consider that Snowden was an employee, not of the government, but of a private contractor. As the leak itself indicates, the enormous reliance of the “security” establishment on private companies- another thing that is not well understood by JQP- creates its own layer of dangers.
Jesús Couto Fandiño 06.10.13 at 5:16 pm
Yep. The point is not having or not having jetcraft. The point is having or not having widespread spying on your own citizens and the rest of the world. The fact that it was not possible before and now is affordable doesnt mean is a good idea or the ethical problems of it went away.
Same as the new techniques of torture – oh, this is totally new, we dont lay a finger on them, no physical damage, so lets use it! Except for the fact that we are supposed to be the good guys that dont do torture, I dont see why not!
Christiaan 06.10.13 at 5:22 pm
@JW Mason:
What this exercise shows is not who are the leaders of the movements. It shows who are the key players are that can make the most difference. Whether those players actually make use of that power is a different question, which this particular input data (and hence also analysis of it) can not answer. For that more specific intel, rather than meta data, is needed. The use of this can be very profound, because it shows that taking out just a few players (whether they are in fact real players or not) can significantly weaken a movement.
In fact, I am pretty sure that the CIA is using this type of analysis (though not exclusively) to select as targets for their drones. And we also know that collateral damage is considered by the administration as justified.
Jesús Couto Fandiño 06.10.13 at 5:40 pm
The problem is that the example misses one use of this, one that the whistleblower itself detailed in interviews.
Pick person A. You want person A to collaborate with you. Run query. Find what links are “bad”. Amplify them. Use them as blackmail. Now you got a source.
Want to infiltrate/manipulate new political movements? Here is your tool. You have dirt on anybody – years of dirt, searchable. Find the weak link. Maybe that guy that is in the middle of that graph is not really that important, but he is in the middle – so pressure him, and make him part of your ring of moles. Get info, plant misdirection, etc.
With ease of use, and private contractors access – why do you have the idea that somehow this is going to ONLY be limited to terrorism? Why dont you think a detailed map of everybody in, say, Occupy, will be of interest to some people to pick apart? Imagine tools like that in the hands of… I dont know, that Walker guy in Wisconsin, was it? Now how fast would the network of union members will be analyzed to find how to destroy them or cow them?
Look at this http://wikipedia.org/wiki/Traffic_analysis
Look at all the things you can get from traffic analysis even if you dont have the content of the communication.
Now you have that, for everybody, for years. Just ready to use for what you want.
Rich Puchalsky 06.10.13 at 5:51 pm
I’ve linked to this before, but people might want to read a few back posts on this blog:
http://nopoliticalrepression.wordpress.com/
The process of traversing social networks for political repression is already going. It’s going like this:
1. Devote resources to chasing down an “important”, “violent”, “nonpolitical” crime, like breaking windows during a protest.
2. Find everyone in the social network that you think the protestors came from. Summon them to a grand jury, where they can’t refuse to answer questions for any reason, including the 5th. Ask them about all of their social contacts, who is friends with who, etc.
3. If they refuse to answer, put them in prison for up to 180 days, in solitary.
4. If they do answer, repeat step #2 with more people, who all of course now have an additional reason to distrust each other and not want to cooperate politically.
Computerized network analysis speeds up step #2 quite a bit. And of course that’s what it’s always been used for, and is used for — putting people with the wrong politics in jail.
bigcitylib 06.10.13 at 6:14 pm
The issue seems to be: does PRISM only look at metadata or content? If only metadata, then I’m not sure what the story is. Its legal for U.S. gov. to do it, end of story. Maybe it shouldn’t be, but I don’t see illegality and therefore I don’t see a scandal. If they can get at content, then scandal. But I am finding that part difficult to believe.
Steve LaBonne 06.10.13 at 6:21 pm
Re bigcitylib: So once something has been made legal (probably without even many proponents of the law having been aware of its full implications), the questions of whether it should be legal, or even whether the law should be modified to strengthen transparency, become completely uninteresting? That’s a perspective, I guess…
christian_h 06.10.13 at 6:22 pm
Who cares if something is a political “scandal”. Even asking that question means embracing the warped view of the DC establishment. The question is, what is the intelligence industrial complex doing? How are they doing it, and what are they using it for (e.g., is the meta-data used to identify human beings for elimination by drone)? Based on the information we have (due to heroic leaking despite government efforts to keep us in the dark), should they be doing it? (And just to be clear, it is not at all apparent whether this data collection is constitutional. If it was clear, why keep the programs secret so they can’t be examined by courts?)
bigcitylib 06.10.13 at 6:28 pm
No, I would not be in favor of this kind of law, and in fact up here in CDN popular sentiment shot down some legislation that would have granted our police what look to be similar powers. But what does the leak really tell us…that the U.S. authorities are using the tools they have been granted (ie to look at metadata)? To me that’s old news, and frankly not worth the current fuss. If they are indeed grabbing content from any email on the planet, then that’s something new and scandalous. But I find myself doubting this bit, for various reasons.
Steve LaBonne 06.10.13 at 6:35 pm
To most of us here it’s old news (and even so we had to rely on assumptions and guesswork to fill in details that have now been revealed). To the general public, not so much. The thing about l’affaire Snowden is that it has forced the news media to actually report on this stuff in a high-profile way.
bigcitylib 06.10.13 at 6:38 pm
Fair enough, but the biggest leak in NSA history, as the Gaurdian has reported? A coup against democracy? C’mon.
Steve LaBonne 06.10.13 at 6:40 pm
OK then, we should ignore the whole thing because a newspaper engaged in a bit of hyperbole, something never before known to occur. Is there an actual point that you’re trying to make, or do you just want us to be impressed by your world-weary cynicism?
Aulus Gellius 06.10.13 at 7:46 pm
ajay: I’m pretty sure that mud man’s point was that the American Revolution did not intend the overthrow of the British government — just the independence of some of its colonies. Unless there was some extra-radical branch of patriots I’m not aware of.
And I think the point of the OP was, basically, you can learn a lot about someone’s political connections and beliefs from metadata, which rather weakens the value of the reassurances that “don’t worry, we’re only looking at metadata, not content.”
js. 06.10.13 at 7:55 pm
Beat me to it, but yes, this is exactly what I wanted to say.
Peter Hovde 06.10.13 at 7:58 pm
Great explication. I’ve only started reading the referenced article, but perhaps Revere was such a good inter-cluster bridge because he was an artisan whose clients were mostly elite, and thus he was connected to both artisan and elite clusters.
RSA 06.10.13 at 8:37 pm
Very nice piece of work! An introduction to the possibilities of social network analysis–and thanks for including the data and software. I’d only add the caveat to readers unfamiliar with the area that in practice it’s harder than it looks. For example:
Actually, it’s not that surprising. Here’s the simplest analysis I can think of: Sum the number of organizations each person belongs to; sort persons by the result. Paul Revere has the unique highest value, 5. (This is in part due to the choice of data.)
Samuel Peck, now, he’s interesting–he’s only in two of the organizations, but he’s in the middle of the network, and he scores highly on two of the three centrality measures. That is, there are all sorts of reasons we might find people interesting. Also:
As more information gets included, though, patterns become harder to see.
Barry 06.10.13 at 8:41 pm
Peter Hovde
” Great explication. I’ve only started reading the referenced article, but perhaps Revere was such a good inter-cluster bridge because he was an artisan whose clients were mostly elite, and thus he was connected to both artisan and elite clusters.”
Nice – an elite artisan could have a unique social position.
Frank in midtown 06.10.13 at 8:46 pm
I post here to make all of your metadata link to all the other websites at which I also post. Good luck with that.
Mao Cheng Ji 06.10.13 at 9:06 pm
Finding Paul Revere would not have been very useful, because the place was full of Paul Reveres. Would be easier to identify them by sectarian affiliation.
Well, come to think of it, I guess this is still how it’s done today. Metadata analysis is just fine tuning.
bigcitylib 06.10.13 at 9:48 pm
The PRISM story has now officially imploded:
http://davidsimon.com/we-are-shocked-shocked/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+DavidSimon+%28David+Simon%29
http://www.zdnet.com/the-real-story-in-the-nsa-scandal-is-the-collapse-of-journalism-7000016570/
Steve,
A bit of harmless hyperbole? Looks like a major cock-up on the part of the Guardian and the Post. Maybe even the job-losing kind.
Tom Slee 06.10.13 at 10:20 pm
KJH Slashdotted! How 1990’s!
hix 06.10.13 at 10:28 pm
“In fact, I am pretty sure that the CIA is using this type of analysis (though not exclusively) to select as targets for their drones. And we also know that collateral damage is considered by the administration as justified.”
I am pretty surer.
christian_h 06.10.13 at 11:07 pm
Absolutely nothing “imploded” and there was no “major cock-up”. If the Post made changes to its story it is due to two reasons: first, PRISM is no longer as highly-classified as it was b/c the DNI admitted its existence in order to then proceed to obfuscate (as bigcitylib demonstrates, with some success) about it; and second, the Post wanted to avoid pissing off tech companies and accepted carefully worded non-denial denials at face value. It is perfectly clear that there is an automated, assembly-line way those corporations use to allow the government access to used data and content that is worlds away from the imagined approach of considering and complying with each individual warrant separately. this automated, assembly-line way of access was described by internal NSA documents in exactly the way the Guardian reported.
I have to say that seeing liberals defend this massive spying operation justifies completely what some of us were pointing out before the 2012 elections about the difficulty of supporting Obama without illusions. Most people are simply not capable of doing so.
John Quiggin 06.10.13 at 11:08 pm
” Its legal for U.S. gov. to do it, end of story. ”
As a non-resident foreigner, and target of this program, with no rights whatsoever under US law, I don’t see things quite the same way.
I suspect quite a few others among the 6.7 billion or so non-US non-people in the world may take a similar or more hostile view. But, hey, it’s legal under US law. And, as we know, the US is the biggest baddest power in the world, so it can do whatever it likes – what could go wrong?
Ronan(rf) 06.10.13 at 11:08 pm
How would this type of analysis work though? In bigcitylib’s David Simon post the implication is that you input a number/s into the database and develop a series of connections from there..but if you were starting from the position of having a specifc number/s you wanted to investigate surely you wouldnt need a database this big, and could make those connections more conventionally
So is the idea that all calls coming from a specific region (it has to be originating from outside the US, right?) are scanned (or whatever) and developed into some sort of coherent network structure, and then….?
That’s a serious question to anyone, I’d be interested to know how it would work on this scale (which is quite different – I assume – from the OP which is dealing with much more specific and useful information)
Substance McGravitas 06.10.13 at 11:24 pm
Yes. Border authorities are no fun and I have family to visit.
Rich Puchalsky 06.10.13 at 11:52 pm
On the PRISM thread, people were talking about this too — that if you’re not a U.S. citizen, you don’t even get rhetorical protection of rights. I mean, if the U.S. government wants to drone assassinate someone in Yemen and it’s a U.S. citizen, they have to say he’s a bad man. If not, they don’t have to say anything. Or if you kill a 16 year old boy who’s a U.S. citizen, you have to have an anonymous source call it “a mistake, a bad mistake” since he was born in Denver. If you kill a kid who wasn’t born in Denver, you don’t have to worry publicly about the PR mishap.
I do the rhetorical “But they’re spying on U.S. citizens!” thing as well, because I can’t imagine anything else coming close to working. But it would be hard not to feel contempt for anyone for whom it was the critical factor that got them to change from support to nonsupport of the program.
bigcitylib 06.11.13 at 12:02 am
John Quiggan,
That the U.S. passed the Patriot Act is to its great shame. But this particular story started with the accusation that the U.S. gov was basically illegally decrypting every email on the planet with the help of corporate titans like Google et al. These claims now appear to be bullshit. All they’re doing is what the Patriot Act allows, and apparently all Google et al do is comply when they get subpoenaed. If we want to talk Patriot shortcomings that’s fine. But I object to transferring the outrage generated by a bullshit story to a real issue. Or, to be more accurate, I can’t sustain it.
adam.smith 06.11.13 at 12:03 am
I like KH little treaty because it illustrates the issue with what I believe is the much bigger scandal. People seem to be throwing to rather separate issues together and while they both relate to what the NSA is doing, I don’t think that’s helpful
1. The first Guardian story, FISA authority is used to compel _vast_ amounts of phone metadata, without any targeting – this isn’t the “all calls from a public phone” that D. Simon talks about – it’s metadata on all calls. This is much broader than anyone could/should have reasonably interpreted FISA authorities and having the whole “dataset” of Verizon – and presumably other carriers’ – connections. This is news – we had no idea this was going on – and it strikes me as excessive/scandalous. My suspicion is that the Udall/Wyden complaints relate to this and similar interpretations of Patriot Act and follow-up legislation.
2. The PRISM program. I don’t actually believe we know what’s up with this. There are essentially two versions of this out there – a) the version from the leaked presentation as elaborated by Greenwald et al – tech companies are cooperating with NSA by providing practically limitless access for the NSA to records as long as they’re “probably” foreigners and b) the semi-official version, as outlined e.g. by Armbinder: http://theweek.com/article/index/245360/solving-the-mystery-of-prism where PRISM is essentially the technical implementation through which tech companies comply with FISA laws by providing access to information they’re legally required (via court order and/or National Security Letter) to provide.
I think there are good reasons to believe one or the other is more plausible – I think Armbinder is right that the presentation could be describing version b) of PRISM and just glossing over technical details. On the other hand, I don’t think there is a particularly strong reason to trust the statements of government or tech companies on this, so we may very well be looking at version a) – which would be a major scandal, not least because by now we’d have both tech companies and the government blatantly lying. But if version b) is indeed what PRISM is, then there really is nothing new. This isn’t just something we “suspected” – we were fully aware that the government can and does compel user information from tech companies – the powers are specified in legislation and the fact that it happens e.g. in google transparency reports.
Sure, we can use it as an opportunity to discuss the appropriateness of the whole FISA construct – and I think FISA is indeed insane and the governments claim that it can’t be brought to court bc no one has standing an outrage – but in that case we should be talking FISA and not PRISM.
hix 06.11.13 at 12:30 am
It would have been a huge surprise if non satelite communication was not monitored the same way we officially know since 2001 satelite communication has been monitored without any boundaries.
http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//TEXT+PRESS+DN-20010905-1+0+DOC+XML+V0//EN#SECTION1
Ronan(rf) 06.11.13 at 9:45 am
To answer my own question @50 then, I assume that you cant make use of this data in bulk, but it’s just a case of having access to it so you can use parts of it when needed..?
Ronan(rf) 06.11.13 at 10:10 am
..and you could find out things like this, I assume
http://www.nature.com/srep/2013/130325/srep01376/full/srep01376.html
Harald K 06.11.13 at 11:51 am
To anyone thinking metadata collection isn’t a big deal (not many here, hopefully): To blackmail Martin Luther King, the FBI used wiretaps, found out about his infidelity, and used it in an attempt to manipulate himself into committing suicide.
But today, with the dataset NSA has snagged from Verizon, with a simple machine learning program even I could probably identify call patterns characteristic of infidelity. God knows what the NSA could find out.
How’s that for privacy? With a simple analysis, couple of minutes on NSAs clusters, you could get blackmail info equivalent to what was used on MLK on millions of Americans. At that point it isn’t a question of whether they have abused it. Just the potential for abuse is too great to tolerate.
Harald K 06.11.13 at 11:55 am
Manipulate HIM. Ugh, why is there no edit button…
Also, worth pointing out as people have sneered that I’m “obsessed with infidelity” over these concerns: infidelity is just an example. It’s a rather common, rather closely guarded secret. But of course, it’s only an example – if such closely guarded secrets can be automatically identified, you can bet a lot more can be found. Perhaps stuff you didn’t even know about yourself (Think “X is susceptible to recruitment in political organization Y, and has an estimated 50% chance of joining it within two years, although he has presently not heard of it”).
Barry 06.11.13 at 12:37 pm
Ronan(rf) 06.11.13 at 9:45 am
” To answer my own question @50 then, I assume that you cant make use of this data in bulk, but it’s just a case of having access to it so you can use parts of it when needed..?”
I’ve seen this assumption made before (and have had to get rid of it myself) – it’s not the 20th century; terabyte databases are now common in industry, and the NSA has both the money and the tech skills to match anything in industry (and can hire from industry, as well). People have been working a lot on how to deal with massive data, and how to parallel process things, and this processing doesn’t have to be done in real time.
Niall McAuley 06.11.13 at 1:10 pm
The NSA’s new Utah datacentre has storage in yottabytes.
Barry 06.11.13 at 1:37 pm
Thanks, Niall.
IIRC, the main Google computer site was supposed to have 100K computers running at any given time. And that was several years ago; anything Google learned/developed is probably either leaking out to the rest of the IT world, or openly available for hire.
And I’m sure that no matter how many computers, how many massive disk arrays the NSA wants, the computer industry will be glad to take their money.
I’ll wager that their biggest problem is sourcing materials from China; I imagine that their government, industry and private (hacker) interests would love to have some backdoors.
RSA 06.11.13 at 1:43 pm
Actually, that’s potential storage. A yottabyte is a million exabytes, and there aren’t many exabyte storage systems around today. Here’s an article from January, 2012, announcing a 10 exabyte system:
As of 2012, at least, if 10 exabytes of storage cost $1 billion, then a yottabyte system would cost $100 trillion. I don’t think costs will come down fast enough for the NSA to be storing that much data any time soon.
rea 06.11.13 at 2:16 pm
1765 census shows Boston with a population of 15,500. It strikes me that this kind of data mining works a lot better with a small population.–the smaller the haystack, the easier it is to find a needle. Try picking Paul Revere out of a population of 320 million, and you might get different results. On other other hand, by 1772, the year of the hypothetical study, Revere was openly and publically involved in dissident politics–to find Revere, you wouldn’t have to data mine–just read the newspapers.
LFC 06.11.13 at 2:35 pm
From KH’s linked post:
“I will show how we can use this “metadata†to find key persons involved in terrorist groups operating within the Colonies at the present time.”
I think that should read:
“I will shew how we can use this “metadata†to find key persons involved in terrorist groups operating within the Colonies at the present time.”
Even I’m wrong, “shew” looks more authentic to an ignoramus such as myself. (G.B. Shaw was still writing “shew” in the twentieth century.)
Barry 06.11.13 at 4:14 pm
“As of 2012, at least, if 10 exabytes of storage cost $1 billion, then a yottabyte system would cost $100 trillion. I don’t think costs will come down fast enough for the NSA to be storing that much data any time soon.”
According to Wikipedia, world internet traffic was estimated at 21 exabytes/month, in 2010 (http://en.wikipedia.org/wiki/Exabyte#Usage_examples). So they’d need to spend $30 billion/year on storage. If the figure didn’t count compression, then they’d be down to a few-several $billion/year. Sounds doable.
mds 06.11.13 at 4:46 pm
Steve LaBonne @ 37:
That “bit of hyperbole,” as you put it, allows the conventional wisdom on the story to become “In coup against democracy, Obama administration engages in unprecedented flouting of law and the Constitution,” rather than “Patriot Act and amended FISA continue to enable sweeping surveillance.” You think James Sensenbrenner isn’t delighted to join the pile-on over a Democratic administration using provisions that James Sensenbrenner personally inserted into a bill? Now we get to hear new variations on “Obama the Dictator” until a Republican president takes office and uses the unchanged legal framework to protect the Fatherland. (There is a bipartisan proposal in the Senate to make the FISA court more transparent, but “bipartisan” currently means “two Republican senators and seven Democratic senators,” which is probably insufficient to get it through the Senate, let alone the House.)
So yes, a framing that can be so easily spun into political point-scoring over Obama’s unique perfidy, while letting the laws that actually permitted it continue in force, might be considered counterproductive. There’s plenty to criticize and organize pushback around without turning it into “The Day the Republic Fell” from the Paulite hymnal.
christian_h 06.11.13 at 5:29 pm
Yes Obama is obviously in no way responsible. After all if the law he demanded be extended allows surveilance then obviously he has no choice but to order his intelligence minions to engage in it. It is really rather amazing what amount of nonsense liberals will spew to protect “their” president from being held responsible for his actions.
christian_h 06.11.13 at 5:31 pm
Also, from any reasonable point of view we – the people of this world – would be better off if only Republican administrations were allowed to engage in this massive spying operation. Because it would mean less spying. For this reason I am amazed the lesser evil crowd isn’t fully behind Sensenbrenner’s hypocrisy.
Ronan(rf) 06.11.13 at 5:52 pm
“I’ve seen this assumption made before (and have had to get rid of it myself) – it’s not the 20th century; terabyte databases are now common in industry….”
But do they have any way of getting any useful information/patterns out of all that data without an explicit starting point? (For example someone under surveillance etc as the centrepoint to build from) Or is it just there (at the moment) to be used in more conventional intel operations?
Barry 06.11.13 at 6:14 pm
“But do they have any way of getting any useful information/patterns out of all that data without an explicit starting point? (For example someone under surveillance etc as the centrepoint to build from) Or is it just there (at the moment) to be used in more conventional intel operations?”
Remember, it’s not the 20th century anymore :)
My guess is all of the above. Start with known points, start with suspected points, start with random points. Pull patterns out and see what sort of patterns you find, and which ones look similar.
Barry 06.11.13 at 6:16 pm
I’d probably have a lot of my processing power ‘boiling down’ the data into such patterns; those can be compared and analyzed in seconds.
John Quiggin 06.11.13 at 8:06 pm
Maybe worth pointing out that the “nothing new” argument was central to the passage of the provisions of the 2012 NDAA, permitting indefinite military detention of US citizens. And the “entirely legal” defense also applies to NDAA. So, if you’re cool with the idea that some bad correlations in your phone call record could see you locked up without trial for life, “nothing new” and “entirely legal” seem to provide you with all the defenses you need.
Of course, for foreigners, it is indeed nothing new. For some time, the US state has openly claimed the right to kidnap, kill, spy on and torture anyone who is not a US citizen or located in the US, subject to no controls and with no possible redress after the fact.
Rich Puchalsky 06.11.13 at 8:08 pm
“So yes, a framing that can be so easily spun into political point-scoring over Obama’s unique perfidy, while letting the laws that actually permitted it continue in force, might be considered counterproductive.”
I don’t think it’s counterproductive. Certainly the Democrats complained about Bush’s unique perfidy when he did the same things, and stopped complaining when a Democrat did them. And there aren’t enough people left for any change if we only take the ones who complained under both administrations. If half of the people who care about civil rights only care because of partisan reasons, well, you take your coalitions where you can get them.
John Quiggin 06.11.13 at 8:42 pm
@Rich A small consolation is that the consistent supporters of universal surveillance are also a minority. Majorities of Dems and Independents opposed universal surveillance under Bush, and, now that Obama’s in, Repubs are now evenly divided.
http://www.people-press.org/2013/06/10/majority-views-nsa-phone-tracking-as-acceptable-anti-terror-tactic/
Sev 06.11.13 at 8:54 pm
I suspect much of what is being done is legal (and much of that shouldn’t be), a good deal is quasi-legal, and not a little would be flagrantly illegal if we had any right to know about it. via Fallows- one who resisted:
http://www.theatlantic.com/politics/archive/2013/06/a-man-who-resisted-the-security-state/276718/
Undoubtedly this kind of analysis is a pretty powerful tool, which can still turn out some pretty flaky results, like those which are sure which book/movie/music you will enjoy (not).
LFC 06.11.13 at 10:02 pm
I just read a WaPo column by Matt Miller in which he claims that ‘the Unknown Coder’ is the real hero, keeping us all safe, not Snowden, who he says is deluded.
The Unknown Coder. Has a certain ring to it, doesn’t it?
Rich Puchalsky 06.11.13 at 10:07 pm
Here’s the attack-wikileaks document by Palantir with its own primitive social network diagrams. See also the special Glenn Greenwald page.
christian_h 06.11.13 at 10:33 pm
Ah yes, Matt “torture” Miller. Asserting that those who “keep us safe” allow deluded narcissists to work for them. So in order to deflect the accusation that his secret police buddies are evil he constructs a story in which they are dangerously incompetent.
Katherine 06.11.13 at 10:40 pm
For some time, the US state has openly claimed the right to kidnap, kill, spy on and torture anyone who is not a US citizen or located in the US
It’s a natural and unfortunate end point of the attitude that has seen the ubiquitous use of the word ‘Americans’ instead of ‘people’ in American political discourse for as long as I can remember.
The rest of us – we’re not really real. We’re just extras in the ongoing drama/farce that is the US of A.
bianca steele 06.11.13 at 11:38 pm
I was prepared to find Matt Miller’s op-ed more or less reasonable. Whatever he’s done more recently (and what is happening to IT people in the military that they’re becoming alienated so frequently, are they being forced to stare at goats or something?), it seems obvious he was hired not because he was competent but because he had a military background and therefore met a security profile that values “military values” more highly than things like a high school education. But sadly, no.
Jeffrey Davis 06.11.13 at 11:41 pm
I don’t understand the objection to using metadata. It sounds like something that’s supposed to make me antsy but it really doesn’t. In fact, I’d object because it sounds like a pretty numnuts way of spending finite [rueful laugh] resources. But an intelligence agency that was passive and only reacted to transgressions wouldn’t be of much use.
bianca steele 06.11.13 at 11:41 pm
And I like the OP, all the points didn’t make sense to me, but I’m going to keep it around and look at it every so often and I bet I’ll get some insights.
LFC 06.12.13 at 12:10 am
JQ
For some time, the US state has openly claimed the right to kidnap, kill, spy on and torture anyone who is not a US citizen or located in the US, subject to no controls and with no possible redress after the fact.
Not to defend what the US govt is doing re drones etc, I wd nonetheless like to add a bit of nuance to this. First w/r/t “spy on” — virtually all govts of any size around the world have spy agencies which spy on anyone they want in other countries prob. usu. w/o any controls but purely internal ones. So that doesn’t set the US apart. Removing “spy on” leaves “kidnap, kill, and torture”. But the official policy of the current US admin bans torture so that’s at least contestable (unacceptable practices are prob. continuing, but not as frequently). That leaves “kidnap” and “kill”. Kidnap and rendition cases seem to have been infrequent lately (or maybe just going on w/o any being revealed). That leaves “kill,” which is, above all, the drone program, which, yes, remains something one shd oppose.
christian_h 06.12.13 at 12:47 am
Well Jeffrey good for you. I suppose you are one of those people who “has nothing to hide”. So what if the government can map out your whole life, find out where you (or at least your cell phone) has been at all times, identify everyone you are in any kind of electronic contact with, know which webpages you visit and when… As someone active in (perfectly legal at least according to those laws we are allowed to know) international solidarity work I’m not so sanguine. I’m not many steps removed from people who have been victims of grand jury fishing expeditions, people who have had their phones and computers confiscated and searched at the border,… I would be shocked if in some of those cases exactly such meta data collection and analysis hadn’t played a role in who was targeted. And yes, as a foreigner whose life could be seriously disrupted by the simple action of denying entry to my place of living and working (the US) all this makes me a bit nervous – which is part of its raison d’etre, I’d bet.
christian_h 06.12.13 at 12:52 am
From the NYT: However, three-quarters said they approved of the government’s tracking phone records of Americans suspected of terrorist activity. Nearly the same number approved of the United States’ monitoring the Internet activities of people living in foreign countries.
Thanks for nothing, Americans.
LFC: the difference between the US and other governments that we probably all agree are often no more virtuous is that the one in the US has the ability to spy on massive numbers of foreign people. Not just spy on foreign governments, but to quote the term of the week “ordinary citizens” in foreign countries.
Lee A. Arnold 06.12.13 at 1:22 am
I implied that a majority of Americans would probably feel that way, in another comments section, and it was castigated as too horrible to mention. Many of these same Americans probably also realize that it is much more than “metadata”. In terms of effectiveness, you may be able to catch some individual young psychos who are too stupid to realize that The System is listening, but against organized terrorists it won’t work anyway. After Zero Dark Thirty, I imagine they have gone to paper courier + innocuous signals. So it will take 5-10 years to move everything into position for another clever attack, and some religionists already think in the long term. I think it might be wise for worldwide practitioners of comparative mysticism to start emphasizing that a peaceful transition to god-consciousness is possible in the practice of every religion. Unfortunately it is likely that many scientific academic types won’t have a clue as to what that last sentence entails.
Barry 06.12.13 at 1:49 am
” I think it might be wise for worldwide practitioners of comparative mysticism to start emphasizing that a peaceful transition to god-consciousness is possible in the practice of every religion. Unfortunately it is likely that many scientific academic types won’t have a clue as to what that last sentence entails.”
No, they’ll understand, but will be struck dumb by the idea. It’s sort of like peace in the Middle East, but moreso.
Lee A. Arnold 06.12.13 at 2:10 am
What else have you got?
Lee A. Arnold 06.12.13 at 2:13 am
You hit the nail on the head Barry. That is very funny.
Salient 06.12.13 at 2:26 am
This whole ‘metadata is perfectly fine’ thing reminds me of the local campaign (well, ‘local’ like 75 miles south of here) where a church group has set up a 24/7 vigil outside a highway adult store. They camp out on the roads, on public grounds they’re allowed to occupy for N hours, and just change out the people keeping vigil. Whenever someone heads in to the store, they take photographs of driver and car, trace the license plate, and publicly post as much information as they could legally collect so that it pops up online. They look up your name and such and try to figure out where you work, and yes, they call your boss and inform them when and where you visited. It’s especially successful with truckers, whose vehicles often have individually identifying information. They then post information about your employer, trying to ensure that Google searches etc for that company also bring up the post about the employee’s transgression. (Last I checked they were looking into whether they were legally allowed to purchase web site advertisements that would put their materials at the top of anybody’s web search for the company or the person’s name.)
They’re working with metadata.
Now of course it’s obviously true that the people who drive into the store deserve to face exposure, public humiliation and should have to withstand an organized campaign to ensure they lose their jobs or at least have to answer to their bosses and colleagues about the places they visit in the evening. They have something to hide — especially the people who are disgusting enough to work there — so fuck ’em.
Here’s the problem: while I was there interviewing the vigilant, something happened twice: a car pulled in, drove around the building, and drove out again. Once, they sprang into action. Photos, the whole thing. And once, they didn’t move a muscle.
What the fuck happened? They computed an adjacency matrix in their head, more or less. I did ask about the SUV they ignored. “Oh, they probably didn’t know where they were going.” So why did they chase after the other one? “Oh, we won’t post it if they don’t go in. They were going to pull in to the store and we scared them off. But we keep things in case someone tries to come back and visit again, we can catch a pattern.” Somehow this was not a thing they contemplated. It just made intuitive sense, to the metadata collectors. Their internal algorithms guided their assessment and response.
That’s PRISM: they compiled and assessed metadata to construct a profile, to distinguish the incidental from the indicative. The difference in assessment was triggered, so far as I can tell, by the fact that the SUV driver was pretty old and the sporty car driver was pretty young. Or maybe it was the speed with which the vehicle entered vs the speed with which the vehicle left. Or maybe sporty car vs SUV. Or maybe someone who looked like a godly type vs… Fuck-all if I know. What I do know is that someone got a free pass and someone else didn’t.
People will never, ever be equally subjected to state intimidation. People will never, ever be equally subjected to state oppression. People will never, ever have equal standing before the law. And: people will make mistakes. They’ll make bad decisions. They’ll be stupid. They’ll drive into adult stores just to see what is there, even if they end up leaving horrified and put off by what they saw, or just disinterested. They’ll look into topics that . (My mom went on a serial killer biography and forensic investigator kick a few years ago. She was careful to always check out a book on how to write fiction along with the serial killer books, under the assumption that someone would be tracking, because she heard something something about libraries being investigating.) They’ll pursue weird hobbies without thinking of how it makes them look to a hypothetical person looking at everything. They’ll live lives as if they’re capable of maintaining independent identities in different communities, even if those identities would come into conflict with one another.
That’s because a degree of privacy is instinctual. It’s naturally assumed and expected. Simple proof: how does a person act when they genuinely come to believe that a potentially malevolent agent is tracking their actions? They don’t generally try to get clever like my mom did. (And when they do, they forget they’re not actually very good at deception. My mom’s ruse probably just made her look all the more suspicious.) They don’t (usually) say fuck this, I’m gonna live my life. They feel stifled. They give up. They shut down. They feel bad about themselves, about the hypothetical whole-self that is being constructed solely in order to determine how bad they are. They imagine what that whole-self looks like, with everything mashed together by someone who is specifically looking to see if there’s something wrong with them. They pull back. They disengage.
Metadata of this form collapses social spheres. It collapses who you are as a social being. It pushes you into accountability in every one of your social spheres for actions you take in any one of your social spheres.
Whenever you’re inclined to think about whether people have anything to hide, think about the legitimate things a person would decide to not do only because they anticipate those things might look bad to the metadata-analyst. They worry about being misunderstood by a system that does not and cannot hear them explain themselves. They have no control over the context. So they live as if hounded.
And that’s far more sane than paranoid. These analysis systems will have lots of false positives, in part because there’s no accountability for too many false positives. People who are not threats will look threatening. And from among those people, with exceptions rare enough to be notable, the already-disadvantaged will suffer consequences. And they know this. So they’ll live less full lives, and they’ll live more nervously.
Sorry for grinding my teeth a little about this. Maybe we’ll turn out to live in a society in which by some strange magic, agents like prosecuting attorneys never ever ever get to ever access to anything inferred from this data collection system. Otherwise we’re sort of all getting investigated by a prosecuting-attorney-type system all the time forever, with no ability to explain ourselves, to contribute to the context assembled by the metadata machine.
Your privacy is your right to context. It’s your right to exert control and discretion over the composition of your identities. That’s why it’s important. And the Paul Revere computation is an explicit demonstration of how metadata collection compromises that right — a far better example than the folks keeping vigil.
For that reason, I’d prefer a handicapped state authority that occasionally misses these intersections, to a state authority that aggressively profiles. So I think Kieran’s piece (as far as whimsical pieces go) is perfectly compelling, even if it does demonstrate the effectiveness of the method. Sorry, ajay.
Lee A. Arnold 06.12.13 at 2:31 am
After recovering from accidentally snorting beer out of my nose, I wish to point out that I raised a call for ecumenical outreach as early as 2003 on DeLong’s blog. And as I believe I said at that time, it is a damning failure of the world’s religions that they have not had enough insight and strength to meet the challenge of what is in some ways a religious crisis.
Barry 06.12.13 at 3:05 am
” These analysis systems will have lots of false positives, in part because there’s no accountability for too many false positives. People who are not threats will look threatening. And from among those people, with exceptions rare enough to be notable, the already-disadvantaged will suffer consequences. And they know this. So they’ll live less full lives, and they’ll live more nervously.”
One of the really important things to remember is this – there are limits, like if somebody ends up having a 100 person team shadow a person for a month for no good reason [1], but in general flagging and screwing over people will be easy to do, and 99.9% deniable and easy to get away with. Note that the no-fly list is still in existence, and nobody knows how to get off of it if you’re on it.
[1] With one obvious outcome being that the management explaining why they spent so much effort on somebody will decide that their best out is to bust them for something, even if they need to flat-out forge data.
Barry 06.12.13 at 3:10 am
BTW, this was from real life – a friend’s former co-worker was in a building with several people when a DEA SWAT team hit them (due to a phony call from an ex-girlfriend of one of the guys). The DEA didn’t find what they were looking for, but told them that the raid cost $25K, and that somebody was going to jail. They took in one of the guys for not paying child support.
Substance McGravitas 06.12.13 at 3:12 am
It might be that shadowing other people might be the only job around.
This was a good book:
http://en.wikipedia.org/wiki/Memoirs_Found_in_a_Bathtub
Rich Puchalsky 06.12.13 at 3:54 am
“The DEA didn’t find what they were looking for, but told them that the raid cost $25K, and that somebody was going to jail. They took in one of the guys for not paying child support.”
Oh, great. This kind of thing is exactly what I’ve previously referred to as creating criminals through enforcement. It’s not that other countries have fewer people not paying child support. It’s that they don’t have huge amounts of money spent to put those people in jail.
And yes — I think that people who don’t pay child support should pay child support. Hopefully by having it garnished from their wages if that has to be done, not via jail. Or, if they can’t pay — they can’t pay.
This article informs me that poor people are often sent to jail for nonpayment of child support with no trial and no lawyer, because it’s a civil matter. And that people in jail for nonpayment of child support are about 2% of the prison population. That’s an incredible miscarriage of justice — basically, debtor’s prison — but I guess the people who think about it at all like it because it’s for the children, or something. Those people will be happy for us to have fewer people in jail as soon as they discover a category of crime that’s sympathetic, I suppose.
John Quiggin 06.12.13 at 4:43 am
What Christian H said @86 regarding mass surveillance of foreigners.
Regarding torture and kidnapping, it’s true that the current Administration has disavowed one, and scaled down the other. But it has aggressively resisted any attempt by victims to seek redress, let alone any suggestion that those responsible should be sanctioned in any way. So, the US state maintains the claimed right to do those things, whenever it seems to the government of the day to be necessary.
adam.smith 06.12.13 at 4:46 am
That’s PRISM: they compiled and assessed metadata to construct a profile, to distinguish the incidental from the indicative.
I’m really confused by this. My understanding is that PRISM is a program to access the full content of accounts at tech companies. It has nothing to do with metadata.
The metadata comes from phone records that are obtained from wireless (and perhaps landline? Not sure – in any case wireless is more interesting/rich) carriers – presumably under the same general authority as PRISM, but is there any indication that they’re the same problem?
Am I missing something here?
adam.smith 06.12.13 at 4:47 am
That’s PRISM: they compiled and assessed metadata to construct a profile, to distinguish the incidental from the indicative.
I’m really confused by this. My understanding is that PRISM is a program to access the full content of accounts at tech companies. It has nothing to do with metadata.
The metadata comes from phone records that are obtained from wireless (and perhaps landline? Not sure – in any case wireless is more interesting/rich) carriers – presumably under the same general authority as PRISM, but is there any indication that they’re the same program?
Am I missing something here?
Barry 06.12.13 at 10:40 am
JQ: “But it has aggressively resisted any attempt by victims to seek redress, let alone any suggestion that those responsible should be sanctioned in any way. So, the US state maintains the claimed right to do those things, whenever it seems to the government of the day to be necessary.”
The US government did it and got away with it (a few enlisted men thrown in prison for several months counts as ‘getting away with it’ from the point of view of the rest of the government). The officials who ordered, conspired and committed the torture are all free and still in the good graces of the establishment, even when it’s known what they did. The legal, social and political precedents have been set.
Barry 06.12.13 at 10:57 am
Apologies for the duplicate comment – this is rather stunning, and I thought that it should be posted on this thread as well:
http://www.mcclatchydc.com/2013/06/10/193546/new-layer-of-secrecy-emerges-at.html#.UbhTbJy0JDd
‘ GUANTANAMO BAY NAVY BASE, Cuba — When the war court reconvenes this week, pretrial hearings in the case of an alleged al-Qaida bomber will be tackling a government motion that’s so secret the public can’t know its name.
It’s listed as the 92nd court filing in the death-penalty case against a Saudi man, Abd al-Rahim al-Nashiri, who was waterboarded by CIA agents.
And in place of its name, the Pentagon has stamped “classified” in red.
It’s not the first classified motion in the case against the 48-year-old former millionaire from Mecca accused of orchestrating al-Qaida’s October 2000 suicide bombing of the USS Cole warship off Yemen. Seventeen sailors were killed in the attack, and the prosecutor proposes to execute al-Nashiri, if he’s convicted.
Also on the docket for discussion this week is a classified defense motion that asks the Army judge to order the government to reveal information “related to the arrest, detention and interrogation” of al-Nashiri. By the time he got to Guantanamo in 2006, according to declassified investigations, CIA agents had held him at secret overseas prisons for four years during which, according to declassified accounts, he was waterboarded and interrogated at the point of a revving power drill and racked pistol.
But what makes the no-name government motion so intriguing is that those who’ve read it can’t say what it’s about, and those who haven’t don’t have a clue. Not even the accused, who, unless the judge rules for the defense, is not allowed to get an unclassified explanation of it – and cannot sit in on the court session when it’s argued in secret.’
Jeffrey Davis 06.12.13 at 12:32 pm
re:85
I’ve got as much to hide as the next guy, but when I use a public utility I don’t have a privacy expectation about that use.
Barry 06.12.13 at 1:08 pm
“I’ve got as much to hide as the next guy, but when I use a public utility I don’t have a privacy expectation about that use.”
Bullsh*t. The situation is that the government has sweeping and secret powers to compel companyies to cough up information. They are making private utilities public (to them only).
Jeffrey Davis 06.12.13 at 1:36 pm
re: 103
The issue as a general one isn’t new. IIRC, it was before the SCOTUS in 79. And the particular kind of use was revealed back when Bush got AT&T immunity from prosecution.
It’s so public of an issue that the furor over it seems to me to be as contrived as the Benghazi hand-wringing. It’s a bit like Louie Renault in Casablanca being shocked to discover gambling at Rick’s.
Ronan(rf) 06.12.13 at 2:02 pm
Just to add to Salient’s post above..
http://www.law.cuny.edu/academics/clinics/immigration/clear/Mapping-Muslims.pdf
Harald K 06.12.13 at 2:11 pm
Salient: Thanks, that (91) was a great post.
Steve LaBonne 06.12.13 at 2:15 pm
Another cheer for Salient @91. Eloquent and on the money.
Jesús Couto Fandiño 06.12.13 at 4:29 pm
If you want a more technical analysis of what the NSA can do
http://arstechnica.com/information-technology/2013/06/what-the-nsa-can-do-with-big-data/
ogmb 06.12.13 at 5:24 pm
Back when I studied graph theory it was widely considered a failed branch of mathematics, tucked away in the attic of obscure operations research methods, run out of hope that it would ever catch the imagination of the wider academic community.
Much later, when I asked two sociologists if they knew about graph theory one denied any knowledge, but provided me with a handout on “link analysis”, a formal method he himself had concocted which looked suspiciously like something from the first three pages of Frank Harary’s textbook, except with completely different vocabulary. The other one claimed he had heard of it but disagreed with its “rational actor approach”.
So I have to say it fills me with considerable pride to see how far graph theory has come in the hands of a new generation of sociologists. One might even say it has become just as important to the well-being of this great nation as waterboarding.
Tom Slee 06.12.13 at 6:18 pm
ogmb might also be interested in this work (PDF) on graph databases from the NSA.
Anarcissie 06.12.13 at 6:23 pm
‘Everything within the state, nothing outside of the state, nothing against the state.’ — Mussolini
We seem to be just about there.
Substance McGravitas 06.12.13 at 6:44 pm
Thanks for that NSA slideshow.
Bruce Wilder 06.12.13 at 8:01 pm
I
watchedkind of, sort of half-listened to a he-said, she-said Ping-Pong on the Chris Hayes cable news show, yesterday, and I could not tell you, who the Rightee or Leftee were — no one important, I’m sure; it was Chris Hayes, after all.But, what I got out of it is that the Left’s position is faux moral outrage and deep concern, while the Right’s position is to just keep re-asserting the integrity and practical importance of the programs. The Right Guy’s gambit was to assert in a deep, manly voice that the Program (he used the singular), was designed to do good things, and ignorant people (who, it was implied, hate America and want us to be “unsafe” and “unprotected”) should just shutup. It was Andrew F, but let’s skillful, mixed in with a little Jack Nicholson aching to tell the handsome liberal screwup in a dress uniform, “he can’t handle the truth” about the tough guys standing a wall and doing what has to be done.
If you are going to be outraged, it should be about the faulty architecture of these programs, in which secrets and lies are two sides of the same counterfeit coin, being passed off as truth. Not faux outrage about insult to personhood or some such, but real outrage about secret efforts that damage the country, the world, the rule-of-law, democracy, etc. You know, little things like incredibly destructive, expensive, pointless and seemingly endless wars, which these jerkwads cannot seem to win or end.
Clapper gave away the existence of a British mole in Al Qaeda a few weeks ago, and instead of being fired and prosecuted, he created a scandal, investigating a Fox News reporter for leaks, implying in the process that the Fox News reported violated the Espionage Act.
Does anyone recall that, in George W Bush’s Administration, the three top officials in the CIA were forced to resign in a corruption scandal?
These things just seem to disappear down the memory hole, because no one on the Left appears capable of the act of memory and awareness it would take, to attack these “programs” as aspects of a completely incompetent organization.
No one on the Right or in the corrupt, complacent Center is ever, ever going to concede that there’s any moral value in your precious privacy. It is not, who they are. If you want to win the political struggle, you are going to have to go after the basic competence and instrumental integrity of the whole she-bang. Mussolini’s basic claim to fame was that he made the trains run on time. Authoritarianism, in its brutal expedience, claims to be efficient and effective. It’s a lie. To undermine the support of an authoritarian regime with people, who don’t care, per se, about brutality or expedience, you have to attack the “efficient and effective” claim directly.
Authoritarian regimes are corrupt and cover up galloping incompetence. The U.S. hasn’t been able to win or end the longest war in its history, in one of the poorest countries on earth, spending a multiple of that pathetic country’s GDP. None of these people have the slightest idea of what they are doing, or how to do it, and that’s what they want to keep secret. And, it is not secret; it is obvious.
john c. halasz 06.12.13 at 8:22 pm
Just out of curiosity, has anyone performed this type of basic network analysis on the neo-cons, or is that too obvious?
PGD 06.12.13 at 9:05 pm
112: their competence argument is, hey, we haven’t been hit in the homeland since 9/11. Only way it gets overturned is if we do get another terrorist attack, which will then create pressure to give away still more freedoms. Just sayin’.
Ronan(rf) 06.12.13 at 9:12 pm
Concentrating on individuals incompetence/evil nature itself, generally, just desolves in ‘faux moral outrage’.. anyway, we have seen consistent incompetence, globally, for a decade with no one held accountable, why is this narrative now going to become so successful?
Ronan(rf) 06.12.13 at 9:15 pm
..*dissolves*..
Greg 06.12.13 at 9:17 pm
“Also, from any reasonable point of view we – the people of this world – would be better off if only Republican administrations were allowed to engage in this massive spying operation. Because it would mean less spying”
What evidence do you have of this?
Substance McGravitas 06.12.13 at 9:20 pm
Oh Anthrax, where are you now?
Bruce Wilder 06.12.13 at 9:22 pm
In Boston, for Patriot’s Day. duh.
Bruce Wilder 06.12.13 at 10:07 pm
Ronan(rf): Concentrating on individuals incompetence/evil nature itself, generally, just desolves in ‘faux moral outrage’..
The thing about a dialectic in which the “moral outrage” card is played, is that it depends on your interlocutor or counterparty playing along. If it doesn’t get something like the desired or expected response, you feel like your rhetoric has fallen flat, and you have to move on to some other strategy. In Freudian psychology, it used to be called transference and counter-transference; the therapist is trying to hypnotize the patient and the patient is trying to hypnotize the therapist. And, political debates, even, or especially, the staged for teevee ones and the ones that take place on op-ed pages, follow the same pattern. The argument is less about logic or evidence than about inducing a trance and piling memes onto emotionally loaded language and narratives — “moral outrage” being a particular template for emotionally loaded language.
Among the most effective techniques is to deliberately misunderstand or fail to respond. The Left gave up on arguing for fiscal stimulus or single-payer, because “no one” would engage in those conversations, or would engage only in wonky and pointless discussion of what was meant by “quantitative easing” (the idea that the “experts” have esoteric knowledge is almost as effective as the assertion of “secret” or “classified” knowledge) or would engage only by dismissing the argument as “politically impractical” (“as any smart person would just know, so I guess you are just not savvy”).
Consciously or not, you’ve done it, here, by referring to “individuals incompetence” [emphasis mine], when I attributed “incompetence” to an organization (and recommended disrespect for the efforts of that organization’s members and apologists).
Genuine moral outrage isn’t looking for a particular rhetorical response from others in conversation; it is looking for justice to be done and reason to apply. (Yeah, I know, good luck with that, eh!)
Faux moral outrage about the uses of information, which, maybe, ought to be private, but isn’t in our brave new digital world, doesn’t look to change anything. At best, it is looking to play a role as useful idiot, by reinforcing the moral narrative, which blames the state or government, as a “bad guy” and fits into the conservative libertarian narrative, by which private, corporate power is elevated and enhanced. If you want your “private” information to be private, then start thinking and working on ways to make your ignorant presumption about what is, or should be, private into an operative reality — don’t be campaigning to have the government go thru some empty bureaucratic ritual honoring an 18th century idealism, which Proctor & Gamble or Target stores don’t go thru.
Ronan(rf): anyway, we have seen consistent incompetence, globally, for a decade with no one held accountable, why is this narrative now going to become so successful?
It’s not. I’m saying stop expecting to change anything, by winning arguments with people, who are not arguing in good faith, people, who want to train you in learned incompetence.
Oswald Spengler had a great line: Optimism is cowardice.
We’re powerless. I’m powerless. They aren’t powerless, and that’s what we want to change, and have no apparent means to change. In the circumstances, we could try to tell the truth — it may not work to change anything, but it improves one’s moral condition and adds to one’s sanity.
Ronan(rf) 06.12.13 at 10:15 pm
“Consciously or not, you’ve done it, here, by referring to “individuals incompetenceâ€..”
I agree with you generally vis a vis highlighting organisational incompetence, it was just that after stressing orgnisational incompetence @112 you continued to show specific, individual incompetence/corruption (not to be arsey or as a gotcha, but just as, I think, it’s very easy to lose sight of the ‘organisation’ and attack the ‘organization’s members and apologists’ – which then allows them to write of the incompetence/corruption as specific to that individual/s)
Ronan(rf) 06.12.13 at 10:18 pm
“which blames the state or government, as a “bad guy†and fits into the conservative libertarian narrative..”
I personally do think the state deserves to be cast as the villan every now and again (although not in an ideologically Utopian Cowenesque nonsensical libertarian narrative)
Ronan(rf) 06.12.13 at 10:33 pm
“Oswald Spengler had a great line: Optimism is cowardice. ”
I knew you’d be a Spengler* fan Bruce! Our fallen social democratic Utopia and all that ; )
*Although Ive never read him and am working from a caricature
Glenn 06.12.13 at 10:47 pm
These revolutionaries must be stopped.
It is types such as these who have no respect for the Unitary Executive and who will be demanding a Bill of Rights, or lacking that, a dissolution of the Constitution.
ogmb 06.12.13 at 10:55 pm
@Tom Slee, very interesting, thanks! Although computer science was one of the few disciplines that showed an interest in graph theory early on. Let’s hope the Big Graph does not drag up an unwanted connection between one of the co-authors and a blogger of the same last name, and the company she keeps.
@john c. halasz Ironically, the results in Kieran’s pamphlet are entirely driven by the fact that Revere was in all but two of the terrorist societies, more than anyone else in that cabal. If the RSA had just summarily incarcerated everyone who was in 4 or more of the listed societies, the Revolution might’ve been rooted out before it even started, with no need for graph theory.
Revere.Paul (5)
Adams.Samuel
Barber.Nathaniel
Bass.Henry
Chase.Thomas
Clarke.Benjamin
Cooper.William
Davis.Caleb
Greenleaf.William
Hancock.John
Hoffins.John
Hopkins.Caleb
Warren.Joseph (all 4)
Rich Puchalsky 06.12.13 at 10:58 pm
This can’t be, Bruce, commenting because there are too, many commas.
More seriously: “These things just seem to disappear down the memory hole, because no one on the Left appears capable of the act of memory and awareness it would take, to attack these “programs†as aspects of a completely incompetent organization.”
Why would we want the security regime to become competent? That would be a catastrophe. So the top three officials in the CIA were corrupt. Think of what they’d do if they weren’t corrupt. Is it in anyone’s interest to call attention to this?
“We’re powerless. I’m powerless. They aren’t powerless, and that’s what we want to change, and have no apparent means to change. In the circumstances, we could try to tell the truth — it may not work to change anything, but it improves one’s moral condition and adds to one’s sanity.”
You write things like this, but then you write other things which imply that you don’t really believe it. Why do you call it “faux outrage”? Maybe the people saying this kind of thing are really outraged. Why should they adopt a tactic of talking about CIA competence when that’s not going to work either and they could care less about it?
I would like a society in which fewer people were imprisoned. Collecting data on everyone helps the people doing the imprisoning. Should I pretend that I’m outraged at Target?
Andrew F. 06.12.13 at 11:02 pm
The Revere exercise is interesting – and the write up is hilarious – but unless you’re starting with what seem to be a fairly focused set of names, I’m not sure the result is easy to duplicate.
As to the metadata program itself, the only thing clear at the moment is that we’re not clear on what it is. It’s possible, but unlikely, that the NSA is doing a social network analysis over every phone number in the country. The court would be unlikely to approve such a program, Congress would be unlikely to fund such a program, and I’m guessing the NSA has enough trouble processing data without adding that kind of monstrosity to the heap. And a program of that size which generates lots of false positives is worse than nothing at all, as the FBI simply wouldn’t be able to keep up with the flow of numbers to investigate.
What seems more likely is that the database is being used more as a data preservation tool than as a standalone input for data analysis. It would enable investigators, once they identify a number as belonging to someone of interest in a terrorism or counterintelligence investigation, to quickly trace out various features of social networks connected to that number. Investigators could also use knowledge of a particular organization’s operational methods to attempt to determine other relevant phone numbers, e.g. a particular type of use of temporary mobile phone, once they have established that a certain number is of interest.
I suspect that the databases maintained by communications providers are not well suited to that type of analysis. To conduct that analysis, therefore, the government would need to construct a new database with possibly relevant records.
This would then set up an argument for the government to request a particular database from a company as relevant to an ongoing investigation. But, such a request is unusual, and to be palatable to the court – and to Congress, and to the Justice Department, among others – certain privacy protections would have to be agreed to. For example, it may be forbidden for the NSA to simply look for signatures in the records without having a particular number unearthed as relevant that can furnish a starting point. It may also be forbidden for the database as a whole to be cross-referenced with other personal identifying information, except for cases in which one has developed a particular suspicion of a given number.
And this would all fit with what Senator Feinstein, among others, has said publicly about the program:
Well, because they then have what’s a telephone book of the numbers and if, through another way, information comes to the FBI that there is reasonable suspicion that a terrorist act, conspiracy, planning, carrying out, is going on, they can access those records. The records are there to access.
The Washington Post
If that in fact is what the program is, and obviously it’s just a guess on my part, then there are still multiple questions to be answered, of course. It does reflect an expansive view of what the government can order produced as relevant; there’s an enormous question of whether the privacy protections in place around the database are sufficient. And aspects of those questions might be difficult to answer without divulging sensitive information about how the NSA connects a relevant number to other numbers, e.g. how knowledge of the operational methods of an international terrorist organization informs the NSA’s analysis. If I know the limits of the NSA’s analysis, then I can devise methods that will stand a better chance of escaping that analysis; if I don’t know, then it becomes more difficult for me to strike the appropriate balance between risking the operational security of whatever terrorist or intelligence action I’m involved in, and accomplishing that action with the limited time/resources at my disposal.
What I would like to see here, therefore, is for the government to disclose the strictly legal reasoning behind the production of the databases, and to disclose more regarding the limits on the use of the database constructed. I think that can be done without giving away information that could be used by an adversary to evade detection. As to how useful that database has been to the government, my best guess based on Senator Udall’s qualified negative argument is that it has quickened previous investigations, but that the information it produced would arguably, at some point, have been discovered via other methods (including perhaps a slower crawl through the records held by telephone companies).
Re: surveillance of foreign communications – as LFC pointed out already, this is done by every government with the capability to do so. What sets the US apart are two things. First, as has been pointed out by christian_h, the US has greater capability (for now) to do so; second, what has not been pointed out, the US does not use this ability to conduct corporate espionage, i.e. the US is not listening in on Airbus and then feeding the information acquired to Boeing. This is in contrast to other nations, particularly those with more active government interest in one or more companies, who have a policy of doing precisely the opposite. If the US does not use such intelligence for corporate espionage, imagine how much less use it has for peering into the personal information of billions of people producing obscene amounts of data who have absolutely nothing to do with national security. Unless you’re selling advertising, or maintaining a Great Firewall, you want targeted collection, not indiscriminate trawling.
LFC 06.12.13 at 11:04 pm
RPuchalsky
This can’t be, Bruce, commenting because there are too, many commas.
Are you just noticing, RP, for the first time that BW uses commas where I think many or most people wouldn’t? I don’t think that’s relevant to judging the worth of his comments, which sometimes are on the lengthy side for my taste but sometimes v insightful.
Rich Puchalsky 06.12.13 at 11:11 pm
It wasn’t intended as a grammar flame, certainly not given how badly I mistype things. And yes, I think that his comments are insightful. There just did seem to be a lot more commas than usual.
Andrew F.: “What seems more likely is that the database is being used more as a data preservation tool than as a standalone input for data analysis. It would enable investigators, once they identify a number as belonging to someone of interest in a terrorism or counterintelligence investigation, to quickly trace out various features of social networks connected to that number”
That’s exactly what I wrote a few threads ago. The problems and / or benefits of data mining are not the point; the point is to dig up something on anyone who you want to target. Of course I don’t think that this is going to be retracted to terrorists. Or rather, “terrorists” will be defined as anyone who important interests dislike.
Suzanne 06.13.13 at 12:39 am
@119:
Sometimes the government is the bad guy, even if conservative libertarian types say so. As for “faux” outrage – as Rich says above, there’s nothing faux about it (most of it, anyway; Sensenbrenner is probably faking it).
Bruce Wilder 06.13.13 at 1:20 am
Rich Puchalsky: Why would we want the security regime to become competent? That would be a catastrophe.
Hmmm. Let me think about that. . . . Oh, this is tough. . . . What a puzzler, this one is! . . . oh, wait! I remember: actual security!
And, they say, you can’t write sarcasm! Ha!
I suppose words like, “rational”, “competent” and “efficient” lose reliable connotations, let alone denotations, after a while, as they are stretched and pulled in competing narrative arguments. If you learn a story, where the hero is saved by a palsy in the hand of the state, with the state as villain, I guess you can say, “palsy is a good thing!”
Do we, thereafter, have to assent to the proposition “palsy is a good thing”? It doesn’t seem like a balanced view, concerning the usefulness of leviathan, or the nature and necessity of social cooperation and organization.
Bruce Wilder 06.13.13 at 1:29 am
Going to war on false pretenses. Sending someone to prison on fabricated evidence. Failing to prosecute criminal banksters. These are things I can imagine real outrage about, but I have to imagine it, because there frequently isn’t much in evidence.
Seen my naked phone bill? OK, I’m not feeling it.
Rich Puchalsky 06.13.13 at 1:42 am
The CIA has nothing to do with security. When the CIA was competent, its business was destroying the left. For instance, it helped Pinochet to power, causing the torture and deaths of tens of thousands of people. That’s what you want to bring back? I think that the people from countries other than the U.S. commenting here might have another view.
And the security agencies in general have nothing to do with security for you or me. They exist in order to keep the powerful in place, both in foreign and domestic affairs. They are an adjunct of the right wing, nothing more. I don’t think that sharply scaling them back would result in any catastrophe.
Barry 06.13.13 at 2:27 am
Andrew F: “The court would be unlikely to approve such a program, Congress would be unlikely to fund such a program, and I’m guessing the NSA has enough trouble processing data without adding that kind of monstrosity to the heap. And a program of that size which generates lots of false positives is worse than nothing at all, as the FBI simply wouldn’t be able to keep up with the flow of numbers to investigate.
Andrew, stop lying. The FISA court approves the overwhelming proportion of requests, the judges all have to be approved by the national security apparatus, Congress sends several 10’s of $billions into a black budget, and can’t even keep track of the open budget. For the black budget the Congressmen/Senators can’t even write stuff down or tell their staffs or get any external help whatsoever, so the problem is vastly more difficult.
As for keeping track of phone calls, that’s the sort of thing which marketing researchers and data miners have been working on. It’s not the 20th century anymore; putting together thousands of computers and terabytes of storage is now casual. Outsiders have been working on analyzing vast datasets, and the NSA has been believed to have a technological edge in these matters for decades.
As for false positives, we just spent a trillion $ on a politically desired false positive (Iraq), which didn’t harm the elites one bit. And in most cases, a positive just means that a person is on secret black lists; action can be taken automatically to f*ck a person over (No Fly list, credit rating goes to sh*t, various background checks return bad stuff,…).
John Quiggin 06.13.13 at 2:39 am
As well as class oppression, secrecy is, to a very large extent, about protecting the powerful from embarrassment. In the present case, the secrecy of the domestic programming arguably made it less effective as a tool for intimidating the population, and no more effective as a way of catching terrorists, but its exposure has caused big political problems for people from Obama on down. That was even more clearly true of Wikileaks.
Bruce Wilder 06.13.13 at 3:01 am
I’m saying I am not sure I would concede that the CIA was ever competent, (or “competent” given that we obviously assign different narrative contexts, and, therefore, meanings, to the word). (Did I mention that every Soviet the CIA ever turned was killed, because the CIA could not keep a secret?) And, I wouldn’t score the overthrow of Allende and installation of Pinochet as a win for American foreign policy, though, clearly, some of those in power in the U.S. were getting what they wanted, or thought they wanted.
I’m pretty sure you and I are living on the same plane of reality, but we have differing narrative frames, and those frames have hijacked key abstract terms.
If you don’t have the power to displace the right wing in foreign or domestic affairs, you don’t have the power to scale back, as you put it, the security agencies, except in the way and manner of the right wing’s choosing. And, you think this is going to turn out well?
In a fit of absent-minded neglect to understand what is going on around us, we have managed to create a digital virtual reality, in which it is damnably hard to hide anything, true or false. [An acquaintance of mine nearly had his life ruined, when someone viciously implied in comments on some blog somewhere that the charity he founded to benefit his hometown in the Philippines was a cover for child molestation — pretty soon Google returned “child molester” to a search on his name (which was not particularly uncommon) and his employer wanted to fire him (he had a union job, luckily) and his landlord evict him. All without the NSA lifting a hand, as far as anyone knows.] Very little is operationally private, or deeply verifiable, which is, certainly a set of problems confronting the security agencies — and damn annoying, when some community college graduate, somewhere, decides to put you on list of people forbidden to fly a commercial airline, for example. But, also annoying to people, who are screened from job candidacy by credit reporting or drug testing.
And, our brilliant Lefties think “transparency” will magically solve all problems, as if the billionaire, who buys a candidate really cares, and no employer will ever check on line to see which Party or candidate, you donated to.
There’s a real political problem with the “secrecy” of the classification scheme, which tries to make it a crime to disclose information of genuine interest in a democracy. And, there’s a real problem with the system of personal identification. And, then, there’s the vicious, out-of-control plutocracy . . .
I don’t think we will be able to make things better, by weakening institutions. If that turns out to be the course of least-resistance, I would ask, why? It seems to me, in general, that our problems have as a common thread the deterioration of institutions, and the emergence of a self-interested elite with no understanding of, or sense of responsibility for, maintaining or managing institutions.
I think that, because I’m an institutionalist in my thinking, and I guess I have a bunch of narratives in my head, that make sense of those very abstract terms. What’s it mean to say that an institution is “strong” or “stronger”? It’s an ambiguous metaphor at best. I hope it means something more than, “I prefer this”. How would I handle discussing the “strength” of slavery as an institution in the antebellum South? I don’t know. I don’t like it, but it was a strong set of institutions, in legal, economic and sociological senses.
Just one of many things that make it hard to communicate. Even with lots of commas.
Rich Puchalsky 06.13.13 at 4:04 am
I think that there’s more than a problem of narrative frames here if you think that the CIA has never been competent ever in its history, and that it can be made competent. Whatever your definition of competence is, that suggests that it’s unattainable.
And you write that “some of those in power in the U.S. were getting what they wanted, or thought they wanted”, yes, that’s what competence means. Governmental agencies aren’t supposed to determine policy. But governmental agencies can have skill sets, inherent in their design and history, that make them good at some things and bad at others. U.S. security agencies are not designed to provide actual security.
“I don’t think we will be able to make things better, by weakening institutions. If that turns out to be the course of least-resistance, I would ask, why? It seems to me, in general, that our problems have as a common thread the deterioration of institutions, and the emergence of a self-interested elite with no understanding of, or sense of responsibility for, maintaining or managing institutions.”
So these institutions will fail.. It’s not like we on the left really have the power to save them anyways. Let them burn.
Rich Puchalsky 06.13.13 at 4:21 am
Or, look — sorry to double-comment, and with basically just a link, but if you actually care what I think about this I wrote it in a poem a while back. There’s no better way for me to say it than that.
John Quiggin 06.13.13 at 4:56 am
Bruce, if you really don’t care, why say so at such length?
John Quiggin 06.13.13 at 4:58 am
Rich, I liked your poem very much
Rakesh Bhandari 06.13.13 at 5:20 am
Raymond Geuss points out that we “relate to other people not merely in terms of what they have done to us or are doing to us, but also with regard to what they will or could do to us. If I have certain effective powers, these may have a sufficiently intimidating effect on others that I get my way without ever needing actually to exercise these powers. So if we wish to understand how human action in a certain society comes to be coordinated, how some individuals or groups bring it about that others embark on certain courses of action or refrain from embarking on others, one of things we will need to take into account is not just who actually does what to whom, but also who has what powers, i.e. who could do what to whom for whose benefit. One must also take account not only of what powers an individual or group actually has, but also of who those powers are perceived, and what powers agents are, rightly or wrongly, thought to have by others…Given the role that intimidation can play, one important ‘power’ that an agent, whether an individual or a society, can have is the ability to control how others perceive its powers or what they imagine these powers to be.” Philosophy and Real Politics.
Some significant minority of Americans think that since almost any American could be shown to be suspicious by data mining–again the belief does not have to be true to take hold– the government will use the technology for political purposes. Today right-wing libertarians are opposed to PRISM and such programs; when Christie is President, the opposition will come from elsewhere.
Barry 06.13.13 at 11:04 am
Bruce: ” And, I wouldn’t score the overthrow of Allende and installation of Pinochet as a win for American foreign policy, though, clearly, some of those in power in the U.S. were getting what they wanted, or thought they wanted.”
That is a success for American foreign policy, because the people in power getting what they want *is* American foreign policy.
Walt 06.13.13 at 11:47 am
Barry, you’re letting cynicism drive you to conceding the whole argument. Fuck the people in power. The US foreign policy establishment has dragged the good name of Americans in the mud. Their criminal activities are criminal, whether or not they get away with them.
Andrew F. 06.13.13 at 12:32 pm
Barry… a few things:
1 – federal judges are exempt from clearance requirements. They have access by virtue of their position. The CJ of the Supreme Court appoints the FISA court judges – no one else.
2 – it’s likely that most search warrant applications presented to judges period are approved – this is because the police and DA are very familiar with what makes for a successful and unsuccessful application, and so the population of applications has been weeded in advance of those most likely to fail under judicial scrutiny. In general, the more resources and time it takes to prepare an application to a judge, the more certain one will want to be that the application will succeed.
Let me put this another way. Most tax returns to the IRS are accepted and never audited. That does not mean that the IRS is simply a rubber stamp. It does mean that the standards as to what constitutes an acceptable and legal return are largely clear, and so filers are able to submit acceptable returns.
3 – the number of requests approved, disapproved, and modified can be deceiving. Not all requests – as certainly recent leaks have shown – are created equal. And there are a couple of cases pending asking that a 72 page decision by the FISC in 2011, finding certain aspects of a government surveillance program to be in violation of the 4th Amendment, be unsealed – so perhaps we’ll learn more.
4 – Expensive false positives don’t engender enthusiasm. You provide the example of the Iraq War, something absolutely no one is eager to repeat. A program that generates enormous numbers of false positives – too many for anyone to investigate – is simply a distraction. As to the cost, a telephone number can’t simply be cross-referenced to a name and then thrown on a no fly list and forgotten. The FBI or a grand jury must request the billing records to determine the name associated with the number. The name in those records must then be investigated. Reports have to be written. This all takes time, personnel, resources. If there is a program that emits a flood of false positives, it simply won’t be possible to do anything with the results by themselves.
Rich, you give the impression of someone who thinks that most personnel at the FBI, CIA, NSA, etc., are engaged in monitoring peaceful political dissidents in the US, sifting for dirt by which to prevent them from advancing politically.
Even if you think that some significant percentage of them are – which I don’t, given the reforms that have been added over the years since the 1960s and 1970s – you’d concede that at least most of those institutions are actually concerned with either crime (in the case of the FBI) or national security, right?
As to why you’d want strong institutions in a certain sense, it’s because you’d want FBI agents, CIA officers, etc., to be institutionally loyal, and not personally loyal. The latter leads to corruption and abuse of power; the former leads to fidelity to an institution’s rules. Obviously if those rules permit things we’d consider gross injustices, institutional loyalty isn’t much of a good thing. But the rules governing the FBI and CIA are actually pretty good; and the existence of institutional loyalty means that we are able to control the behavior of those institutions by controlling the rules, a not unimportant feature in a democracy.
Rich Puchalsky 06.13.13 at 1:06 pm
“As to why you’d want strong institutions in a certain sense, it’s because you’d want FBI agents, CIA officers, etc., to be institutionally loyal, and not personally loyal.”
Loyalty is not competence. It’s possible to imagine a CIA in which everyone was loyal to the rules both in writ and in spirit, and in which everyone was even personally sympathetic, but in which the organization found itself overthrowing democratically elected foreign governments because it turns out that that’s all they’re really good at. Do we want them to be competent at that? It rather depends on whether you think that’s a good idea, even as authorized by our own democratically elected government.
Can we imagine them being competent at something else? Of course people can imagine any number of alternate histories. But in real history, prior events have already happened.
I shouldn’t oversimplify, though. If the CIA wanted to use its demonstrated ability to propagandize via art movements, I wouldn’t object.
Barry 06.13.13 at 1:30 pm
” federal judges are exempt from clearance requirements. They have access by virtue of their position. The CJ of the Supreme Court appoints the FISA court judges – no one else.”
Riiiiiiiiiiiiiiiiiiiiighhhhhhhhhhhhhhhhhhhhhhhhhht. Non-cleared judges get to see high-level secret data.
“Expensive false positives don’t engender enthusiasm. You provide the example of the Iraq War, something absolutely no one is eager to repeat.”
Except for right-wingers, and those who’d profit by it.
” A program that generates enormous numbers of false positives – too many for anyone to investigate – is simply a distraction. As to the cost, a telephone number can’t simply be cross-referenced to a name and then thrown on a no fly list and forgotten. ”
Lie. Or course it can. Or worse, thrown on a no fly list and then flagged and f*cked with ***by machine*** in whatever other ways people can come up with.
“The FBI or a grand jury must request the billing records to determine the name associated with the number. The name in those records must then be investigated. Reports have to be written. This all takes time, personnel, resources. If there is a program that emits a flood of false positives, it simply won’t be possible to do anything with the results by themselves.”
You keep asserting that secret programs are bound by laws and ordinary procedures.
For example, to the best of my knowledge if you’re on the no fly list, that’s it. No appeal, no justification needed.
Bruce Wilder 06.13.13 at 5:16 pm
JQ: Bruce, if you really don’t care, why say so at such length?
I care. I’m careful.
Ronan(rf) 06.13.13 at 11:59 pm
JQ: “Rich, I liked your poem very much”
Same here
Rich Puchalsky 06.14.13 at 11:58 am
Thanks, JQ and Ronan.
Comments on this entry are closed.