The open data movement is a hammer which has gathered the support of many nails. There are the curious taxpayers, who feel their annual checks mean they deserve a peek at the interesting facts the government has collected. There are the ambitious business owners, who see an opportunity to privatize profits from work with socialized costs. And there are the self-styled activists, who believe that if we reveal the data on what the government is really doing, we will arrest corruption by exposing it to sunlight.
The coalition is a confusing mix of these very different motivations (as Tom Slee observes), and the benefits of such a tactical alliance has come with the cost of some confusion. So let’s be clear about what open data can and cannot do.
If the St. Louis Fed publishes reams of economic data, it can certainly make it easier for Mr. Yglesias to make his fantastic charts. If the MTA makes real-time subway information public, it can certainly let Mr. Ernst improve his fantastic app. And, as the talented Mr. Lee pointed out to me, his careful collection of data about members of Congress and the bills they’e passing can be an invaluable resource for professional activists.
So, if I got to choose whether the government should share the data it’s collected, I’d happily vote yes. In fact, I spent several years of my life using the FOIA laws to force it to do just that. I can’t claim my work had any particular impact, but as a curious taxpayer, it was a weirdly-enjoyable hobby.
But the open data movement often claims to be much more than that. They insist open data will not just help a few people with their jobs or a few kids with their hobbies but, as the Sunlight Foundation puts it, "make government transparent and accountable." And that I just don’t see.
I’ve outlined my theory why elsewhere, but the short version is pretty simple: people hide their crimes. Imagine you learn lots of bribes are exchanged at top of the Capitol Reflecting Pool. So you lobby Congress hard to set up bright lights and a camera to catch the perpretrators. The video would be livestreamed to the Internet so dedicated watchdogs can name and shame the bribetaking politicians. Your lobbying succeeds and, on January 1st, the lights go up and the cameras switch on.
But as an engaged citizenry tunes in, there’s is nothing but disappointment. Nobody seems to be taking bribes; just a couple pieces of litter blowing by the pool.
Was Congress really squeaky-clean after all? Of course not — the bribes just moved to the other end at the pool, out of the spotlights.
When you have time to prepare, it’s pretty easy to disguise the data. And this is exactly the pattern we’ve seen. It’s always been investigative journalism, not data mining, that’s revealed the big scandals about politicians. I, more than anyone, would love to believe that the next great Watergate is just lying in plain sight to be uncovered by a swashbuckling econometrician, but the sad fact is, it simply isn’t so.
But it’s also worth pausing to ask: what was any of this supposed to achieve? Imagine, for some strange reason, members of Congress didn’t bother avoiding the spotlight. Every day, we saw them, in full HD video, taking money from prominent businessmen. Do we really think even this (far-fetched) instance of transparency would change much? After all, most Americans already think Congress is corrupt. Most Americans think money actually buys politicians’ votes. Seeing it happen in video might be striking, and maybe make for some good segments on the evening news (or, these days, some viral YouTube videos), but would it really change anything?
After a couple weeks of chatter, and perhaps a few grandstanding legislative proposals, I suspect it’d just fade into the background. More dramatic examples are not exactly what’s most missing from the reform debate — Lessig’s recent book has enough to last us a couple decades. Structural reforms have failed because of the incompetence of reformers, not because there’s a lack of evidence that there’s a problem. (Free tip to structural reformers: get state legislators to sign on to your constitutional amendment. They’re very susceptible to public pressure, there’s a lot of them (so you’ll have a constant narrative of progress), and they’re the ones you’ll ultimately need to actually pass the amendment.)
But maybe open data was supposed to improve politics in other ways. Structural reform is an ambitious goal — maybe the open data proponents wanted something much more modest. But all the more modest stories suffer from a similar excess of naïveté. Whenever geeks turn their eyes to politics, they always have the same reaction: There’s so much inefficiency! And they naturally propose the obvious ideas for reducing it — for example: If only it was easier for citizens to read bills, citizens with relevant expertise could assist Congress by sharing their hard-earned wisdom!
The fact is, Congress isn’t interested in availing itself of your wisdom any more than the sausagemaker needs your help tidying the floor. Lawmaking is The Wire, not Schoolhouse Rock. It’s about blood and war and power, not evidence and argument and policy. (I have one friend who was startled to learn that when members of Congress debate an issue on C-SPAN, they’re speaking not to each other but to cameras in a largely-empty room.)
I don’t want this to sound overly harsh. The truth is, it’s really hard to do effective philanthropy. With a little work, you could mount a similar critique of the vast majority of our bumbling efforts to do good. Most ideas for helping people that seem reasonable in the abstract, turn out to fall apart upon close confrontation with reality. The real question is what happens then. There’s no shame in admitting your mistakes, learning from them, and trying again. Indeed, as my old professor Carol Dweck has shown, that’s the only real route to success. But most of us are too vain or too proud to take that route. We insist that the purity of our intentions reduces the need for careful scrutiny of our effects. Or we try to make ourselves feel better by grasping at any factoid that suggests we had an impact.
I have no particular interest in correcting people’s pride or vanity. This movement is populated by my friends and I respect them enormously and wish them well. Throwing darts at their day jobs has only made my life worse. But this stuff matters — funders and volunteers face tough choices about which causes to pursue. It’s important that they know the case for opening up data to hold government accountable simply isn’t there. (And that they should invest in metaresarch, inxluding open scientific data, instead.) It’s nothing personal — just trying to help everyone do their best. I dearly hope that if anyone ever has a similar critique of the causes I pursue, they will be even more blunt in pointing out my folly.
{ 33 comments }
rea 07.03.12 at 3:23 pm
Is decision-making by accountable officials really improved by having all of the advice they receive be public? Don’t we get better decision-making by keeping the advice confidential, and holding the decsion-maker accountable for the outcome?
Doctor Memory 07.03.12 at 3:36 pm
There are the curious taxpayers, who feel their annual checks mean they deserve a peek at the interesting facts the government has collected.
There’s quite a fascinating implicit understanding of the relationship between government and citizenry bound up in that little aside.
Metatone 07.03.12 at 3:47 pm
It’s easy to make a case being cynical – and certainly no single thing, e.g. open data, is a panacea. However, there are benefits to shining light and making them move their payments to darker waters. It increases their risk, which gives more chance of them being caught.
As for the cynicism around “everyone does it” – tell it to Rupert Murdoch. Yes, everyone does it and recording it doesn’t magically make it a cause for a downfall. But evidence gathered can build a case, even if it takes years to build that case. It it actually more valuable to give up on politics and focus on improving science? Maybe, but one could equally be cynical about the chances for success of the meta-research agenda…
ponce 07.03.12 at 4:45 pm
“Every day, we saw them, in full HD video, taking money from prominent businessmen. Do we really think even this (far-fetched) instance of transparency would change much?”
This Vanity Fair article on Romney’s aquicition of his fortune made me feel the same way:
http://www.vanityfair.com/politics/2012/08/investigating-mitt-romney-offshore-accounts
But there is now some data showing that exposing Romney’s financial hooliganism is damaging him in the polls.
Bruce Wilder 07.03.12 at 5:01 pm
rea @ 1: “Is decision-making by accountable officials really improved by having all of the advice they receive be public? Don’t we get better decision-making by keeping the advice confidential, and holding the decsion-maker accountable for the outcome?”
No and no.
Aaron Swartz: “It’s about blood and war and power, not evidence and argument and policy.”
It’s about blood and war and power,
notand evidence and argument and policy.Over and over, the posts — and many of the comments — have made the same basic point about the fatal, stubborn naivete buried in the moral idealism of the open data movement.
Knowledge is power, it was said at the beginning of the Scientific Revolution. A narrative, a fable even, that had some attachment, some definite reconciliation of its meaning with facts and functional relationships was and is powerful, more powerful than faith in miracles and ritual, and reciting a common creed. But, it is not magic. The whole point of the Scientific Revolution and the Enlightenment is exactly: it is not magic! And, while faith and ritual and common creed reinforce hierarchical authority, the authority of facts is fundamentally democratic, egalitarian and distributed; logic and observational evidence are accessible to every person.
Politics is strategic conflict among individuals and groups. If you want officials to make good decisions in the public interest, the critical, political element is their strategic freedom: they have to be at the nexus of strategic conflict, and in a position to achieve freedom in balancing that conflict. It is not a question of whether bribes are taken; it is a question of whether the politician is dependent on the bribes of one interest, and, therefore, controlled by that singular interest. The politician, who can take the bribe and cannot be held accountable to those, who paid the bribe, because he also took a bribe from their opponents — well, that politician is free to think and act. People are going to argue and debate and deliberate in earnest only when it matters, and it will only matter when power has neutralized itself in conflict, sufficiently for the argument and debate and deliberation to matter.
At the beginning of the Bush-Cheney Administration, Cheney famously assembled in secret an Energy Task Force to advise the new Administration. Secrecy wasn’t the problem, per se; it was a means to a strategic end. The problem for good government was the elimination of conflict, or, at least, the elimination of organized and effective representation of some interests and points of view.
There’s a dynamic to the development of these things. I watched the Nora Ephron movie, Julie and Julia recently. She had a great insight into power, which I may have missed the first time. Julia Childs and her cookbook achieved their great influence in conflict and reaction to that triumph of post-WWII processed food: the TV dinner. The same forces of economic development that make possible the centralized manufacture of homogenized powerless passive conformity also make possible something else.
I’ll know that the Open Data folks understand what they are doing, when NextBus and its competitors let people report service failures. People know whether the bus arrived, whether it was crowded, etc. This information is distributed in its origin; it isn’t preternaturally centralized in an LA Metro database. The problem isn’t to free good centralized data from unjust imprisonment; the problem is to recognize that data is not centralized, to begin with.
Boy: Do not try and bend the spoon. That’s impossible. Instead only try to realize the truth.
Neo: What truth?
Boy: There is no spoon.
Neo: There is no spoon?
Boy: Then you’ll see that it is not the spoon that bends, it is only yourself.
Aaron Swartz 07.03.12 at 5:04 pm
Tom Slee sent me this illustration:
http://youtu.be/hGo5bxWy21g?t=1m16s
Jerry Vinokurov 07.03.12 at 6:06 pm
Actually, in Pittsburgh at least, there is an app which does this. It’s called Tiramisu (like the Italian dessert) and it allows you to say that the bus failed to show up on time or was too full to be boarded or just plain didn’t stop.
No one 07.03.12 at 7:25 pm
Maybe of interest? “Open house? freedom of information and its impact on the UK Parliament”
Not all parliaments are subject to Freedom of Information (FOI), and no one has studied what difference it makes to the working of a parliament. This article evaluates the impact of FOI on the UK Parliament by reference to its main objectives. It finds that FOI has made Westminster slightly more open and accountable; but it has not increased public understanding, participation, or trust. Westminster was already very open and other initiatives have helped increase participation. Trust was badly damaged by the MPs’ expenses scandal, triggered by FOI requests. The House of Commons’ hesitant response to those requests revealed problems of corporate governance.
http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9299.2012.02024.x/abstract
Tom Lee 07.03.12 at 7:37 pm
Aaron@6: the flip side to that video, of course, is the limitations it puts on Stringer: he can’t archive what’s happening for future reference, use a transcript to adjudicate resulting conflicts, or involve remote actors; all because any record he makes could be discovered and used against him by those looking for it.
Does this mean it’s impossible for him to commit crimes? Obviously not. However, it does make his criminal enterprise much less efficient than it otherwise might be and, it’s not unreasonable to think, capable of doing less harm.
But more on this when my piece runs (hopefully not while everyone’s out picnicking…)
L2P 07.03.12 at 9:52 pm
I don’t think so.
IMX, FOIA and the CPRA has meant that a lot of decisions and advice that in the past were formalized and written are now only formalized to the bare minimum. Legal advice to government agencies that 30 years ago would have been written in a memo is now given orally, and simply noted in the agency records as “acting on advice of counsel” or not noted at all.
IMX, it’s the worse possible result. No one really knows why their government is taking action, and the government is taking action without solid advice because no one wants to pin anything down that could be used against them either in court, or in the next election cycle. Would you, as the Mayor of Big City, want a report saying that your next project might bring in a million jobs but would cost billions over 30 years? Not particularly. So you’ll kill the report and just have oral testimony that a million jobs will come in from the project, and move on. If a report is required, it’ll be vague stuff about “job benefits far exceeding any potential blahbity blah.” Let the Times try to sort it out three years from now, after you’re Senator and cost overruns make Big City fire half the cops.
Why do you think the “big stories” the Times and the Post publish are penny ante BS like expense account problems or minor billing issues? Those are usually cases where the government didn’t think or know it was doing anything wrong, or else there wouldn’t have been anything to find. There’s huge corruption at every level of government, but it takes real reporting to find it, not just filing FOIA requests and looking at expense reports and budgets. Nobody wants to do that that I’ve seen. I’ve got my name on reports about more shady deals than a tree salesmen and no reporter has ever called me about them. Not one.
I don’t have a good answer to this. Without public records acts a lot of governments would turn into secret empires, obviously a bad solution. But it’s important to acknowledge that there is a cost to virtually every governmental communication being an open record.
Carl Weetabix 07.03.12 at 10:24 pm
You need to think about “layered security”. That is, if you’re in the security trade you take what you can and layer it on top of other security and so on. Eventually the layers make it so difficult the bad guys move on somewhere else easier, even if no part of the security itself is perfect.
The point is, even if some step, say trying to create transparency, isn’t perfect, if you keep making these overlaying efforts that make the bad guys lives harder, eventually they give up. So it’s true, transparency efforts alone aren’t the solution, but if you add these to other layered small changes at some point it becomes to much of a PIA that they move on to other easier prey. At least that’s the hope.
In short every little bit helps, as long as you don’t have the illusion that any is the “end all” in itself.
CB 07.03.12 at 11:42 pm
I agree with everything you write here (though maybe not the tone of it), except for this:
“The real question is what happens then. There’s no shame in admitting your mistakes, learning from them, and trying again. Indeed, as my old professor Carol Dweck has shown, that’s the only real route to success. But most of us are too vain or too proud to take that route. We insist that the purity of our intentions reduces the need for careful scrutiny of our effects. Or we try to make ourselves feel better by grasping at any factoid that suggests we had an impact.”
Just because we haven’t solved the problem yet doesn’t mean people working in this space haven’t been open to critique, self examination, iteration, and re-focusing.
Tim Dymond 07.04.12 at 12:04 am
Bribe taking moving ‘out of the spotlights’ isn’t necessarily a failure on open data’s part if it is making the bribe taking more difficult to pull off i.e. increasing the transaction costs of bribe taking by forcing them to change time, place etc.
That is Julian Assange’s argument for Wikileaks – not that it stops the conspiring of the powerful, but it reduces their ability to act with impunity.
‘How can we reduce the ability of a conspiracy to act?…We can split the conspiracy, reduce or eliminating important communication between a few high weight links or many low weight links.’
http://www.huffingtonpost.com/urizenus-sklar/understanding-conspiracy-_b_793463.html
Tom T. 07.04.12 at 12:21 am
Re 8: “Not all parliaments are subject to Freedom of Information (FOI), and no one has studied what difference it makes to the working of a parliament.”
Presumably people have studied these differences in the context of securities disclosure laws over time or across different countries, though, haven’t they?
bianca steele 07.04.12 at 12:40 am
Belle, Have you seen An Education? Not quite Mandy level but I was shocked.
I don’t think it’s a big secret what R. Kelly’s songs are like. I would find it hard to go back to an artist who’d, for example, presented a novel about child rape as if he identified with the victim, even as if it was based on his own victimization, and it turned out (after the statute of limitations had run out) that he had been the rapist. I might also start looking, in the same case, for descriptions of the circumstances of the rape that made them sound like a good thing.
bianca steele 07.04.12 at 1:24 am
sorry, wrong thread.
Phil 07.04.12 at 7:32 am
he can’t archive what’s happening for future reference, use a transcript to adjudicate resulting conflicts, or involve remote actors; all because any record he makes could be discovered and used against him by those looking for it.
Criminal conspiracies need open data too. Except that it would destroy them, so… Hmm.
This is interestingly reminiscent of the Fuller/Hart debate about the procedural morality of law, and more importantly how far that gets you. F & H agreed that any legal system should respects its subjects as rational individuals, and should be equally binding on government, but Hart denied that this added up to a set of moral criteria and argued that what was being described was simply a technically efficient legal system. Fuller maintained that a procedurally moral legal system could not be used for immoral ends – a good legal system would in and of itself obstruct an evil regime. The debate isn’t as easy to settle either way as it might sound; it outlived both disputants and still rumbles on now, mainly in the form of laborious counterfactuals involving Nazis.
So is open data purely a technical good – something that’s good for getting stuff done, whoever you are and whatever you have planned? Or is it in part a moral good – something that will make it easier to get the right kinds of thing done, but will get in the way if all you want to do is rip people off and break things?
Phil 07.04.12 at 7:32 am
Modded! Is it cos I is mentioning Nazis?
Phil 07.04.12 at 7:33 am
Well, that’s weird. Mods, could you fish my previous comment out of the moderation sink-trap?
andrew 07.04.12 at 8:34 am
Without open records and open meetings laws, and reporting requirements and stuff like that, the line “you’re taking notes on a criminal fucking conspiracy?” all too often would become something like “you’re taking notes on a city fucking council meeting?” [Feel free to substitute the deliberative body (and expletive) of your choice for “city fucking council”.]
gordon 07.04.12 at 10:22 am
I wonder whether Mr Swartz would hang Julian Assange or give him a medal.
Phil 07.04.12 at 10:56 am
Or buy him a coffee! Or ask him for an interview. Or deliberately tread on his toes. Or buy him a coffee and *then* tread on his toes. Or deliberately maintain a distance of about ten feet, then catch his eye and give him an enigmatic stare through narrowed eyes, as if to say I know something of great importance which you don’t know… Or do you?… No, you quite definitely don’t! Ha!. Then, and this is the really good
bit, leave the room but watch from behind the door. Maybe he’d do that. That’s what I’d do, anyway.
Barry 07.04.12 at 11:43 am
Aaron: “The open data movement is a hammer which has gathered the support of many. nails.”
The only I interpretation which I can see here is that many of the supporters of the Open Data movement would be specifically harmed by it.
However, I don’t see much support for this position.
Anonymole 07.04.12 at 4:14 pm
“Inxluding”? Was that a Aristotlean slip of the ring finger? Including while at the same time excluding?
I never considered that FOIA was going to expose government in any enlightening way. I always thought it was a doctrine of sharing. We paid for that information, research, census, coalesced nuggets of knowledge, so why not have access to it. How much was spent funding the purchase of national monuments, parks, open space, forests in 2011? Who, exactly, received those funds? What has been done to the land since then? Those kinds of once opaque bits of information which would help the public know what’s being done in government. Not the why, or the how, just the what.
blah blah 07.05.12 at 12:28 am
I remember reading an article in one of the following magazines back in the late 70’s early 80’s, playboy, penthouse, or cracked. It was about Pinocchio being president along with many other Disney and looney (now Disney as well) characters. The point of the article was that all the characters decided to make Pinocchio president, why cause they would know when he lied. The pros, well he couldn’t do anything wrong even if was for the right reasons. You know those little white lies about trying to encourage a fat person to keep on their attempts at loosing weight. Course he couldn’t know any military secrets either. Open government data is a good thing, but remember those that are in office are not really trusted by anybody already. and just replacing them all means it will take time to figure out who has come up with new ways to screw the people. Personally I’m OK with open data with of course the exception of many a military program. Course their are some diplomatic programs that need to be classified as well.
So the question is really open data or not, that answer is yes, it will help take a bite out of government crime. But the next question is how do we go about arresting and prosecuting and imprisoning those criminals once we found them out and are able to prove their crime. Remember there is a constitutional amendment that states every law congress comes up with they are to live by it as well. But we all know it doesn’t work that way here in the US. So yes to open data (exceptions as noted, course who gets to review that data for real accountability), but what is to be done to get the criminals charged and jail and stop their paychecks. The is currently one republican and one democrat in jail today, rumor has it they are still collecting they paychecks as if they were in office.
Lorelei Kelly 07.05.12 at 3:35 am
Thanks for this piece. I worked on the Hill for many years looking at how to modernize its support system for expert knowledge. The lack of accountability concurrent with transparency that you rightly mention isn’t caused by the open data movement–it is revealed by it. Americans have just begun to glimpse how much of our civic life has gone down the memory hole. We don’t actually practice an evolved form of democracy in this country…and citizens need to figure out how to reinvent approaches to accountability locally…Capitol Hill will not reform itself. This is absolutely not impossible. Technology makes it more available than ever. When I worked on the Hill doing foreign policy and national security issues– the people with the best relationships and highest value information (not based on self-interest or financial gain) NEVER showed up in a way that was useful, timely or helpful in the congressional environment. Where are our land grant schools? Where are the universities? Where are the funders for improving standing civic/accountability infrastructure outside of elections? When are public advocacy groups going to stop with their “surround and punish” tactics with Members and staff and build actual relationships? The open data movement is a clarion call to modernize our civic life. It can’t do this alone. Just know that Congress is less venal and corrupt than it is obsolete and incapacitated. These things can be fixed.
piglet 07.05.12 at 6:17 am
I’m confused what this is about. FOIA in the USA isn’t exactly new. It has been around for decades if I am not mistaken. What exactly is the point of this blog post at this time in history?
Davos 07.05.12 at 7:54 am
It really depends what data we start collecting. In the big data world we could be storing things like CCTV footage and start capturing data from all kinds of sensors. Door swipe card records, etc etc
The sheer volume of data possible makes this a forensic issue not one of ideology. Every crime leaves a trace and in the future those traces might be in a handy database ripe for scraping.
Brian Balke 07.05.12 at 4:27 pm
Having watched trolls take over and destroy vibrant on-line communities, I have trouble imagining that more interaction is going to lead to better predictive decision-making by our legislative bodies.
What has happened in the past, and will always be the case until other mechanisms are brought to bear, is that government will be reactionary. It will rise to address the needs of the citizenry only when the number of people that have nothing to lose – and therefore that have no reason not to engage in social disruption – becomes large enough that they are able to disrupt the economic cycle. Then legislative bodies will be able to step in to create mechanisms that redistribute power to those that need it most.
This is why BP spent billions in the Gulf of Mexico. This is why Johnson was able to push through the “Great Society”.
What Congress should be seeking to do, as I see it, is to create opportunities for the people to see what motivates decision makers that have real influence on events in real time, so that we can decide where to invest our good will. That means calling them in to testify before Congress, not necessarily with the aim of evolving policy, but simply with the goal of ensuring that the public is able to punish them in the marketplace if it turns out that they have abused our trust.
Anonymous Coward 07.06.12 at 7:40 am
You’re right, open data won’t uncover the next Watergate. But it will at least point a lot of investigative journalists in the right direction, and make their work easier.
IMO, the only thing preventing failures in democracy and corruption is a well educated population, able, willing and used to constantly educate and inform itself on matters of public interest. Do you think that’s the case in the US? In fact, do you think that this is the general direction in which the world is heading? For some time, it seemed some countries in Northern Europe were going in that direction, but I’m not so sure anymore.
Dan Nguyen 07.06.12 at 5:54 pm
Aaron is right to be pessimistic. Data should never be seen as an end in itself, but the start of where inquiry can begin. The problem for government agencies who publish data but attempt to obfuscate it is that there are methods to discovering the probability of fakery, something that can’t easily be done when no data exists in the first place. Data is itself just one (meta) datapoint in any investigation, but it can still be extremely useful.
Luke 07.06.12 at 7:28 pm
The entirety of a real scandal like Watergate is never going to show up in a database, and anyone who thinks it is is foolish. (More foolish still is anyone who thinks that just because they know how write a computer program to process a CSV, they’re an expert on the topical matter that a spreadsheet depicts, and equipped to offer advice after running some totals.)
But it’s a lot easier to deny something verbally in the present tense than it is to go back and modify a wide and long paper trail with varying bits and pieces. Scandals like Watergate don’t emerge one day in their full-blown form, they gradually build up over time, and politicians don’t have the foresight to keep it entirely away from the record in the early days. Any time a paper trail is formed in real time, there is a record that spinmeisters can’t deny and an opportunity for red flags to emerge. They won’t capture the worst of it, but they can highlight what you need to look at.
bianca steele 07.07.12 at 4:11 pm
In the off chance that John H. is reading this, and because I’ve been reading Dreyfus (and John has brought this up in the past IINM), does the problem with computer programmers’ (putatively) thinking they can answer advice (beyond what the output of the computer program states) have to do with the limits of logic, or does it have to do with the difference between having a program run through the logical steps and someone’s doing it in his or her head? (This is almost but not quite a Chinese Box question, I guess.)
Comments on this entry are closed.