Searching for John Snows

Sometime in the early 1840s, a British doctor and statistician named William Farr took control of the Weekly Returns Of Births And Deaths, a publication of the Registrar General’s office where Farr worked. Variants of the Weekly Returns had been published by the state for at least two centuries before Farr took over, but for most of that time the Returns recorded only the name of the newly born or newly dead, and the parish where they resided. But Farr was what we would now call an Open Data advocate, and over time he greatly expanded the information disseminated through the Weekly Returns. By the mid 1850s, the Returns tracked age, cause of death, occupation–even the elevation of the dead’s primary residence. (Farr believed that people living in higher altitudes had healthier lives.) Inspired by a debate with one of his contemporaries, the Soho doctor John Snow, Farr even added information on the deceased’s regular source of drinking water.

I knew nothing about William Farr, or indeed the Weekly Returns, until I sat down to research my book _The Ghost Map_, which tells the story of John Snow’s brilliant solution to the riddle of cholera, as it emerged in the middle of devastating outbreak in the summer of 1854. Snow is rightly famous for creating a map of the outbreak that helped convince authorities of his waterborne theory of cholera’s origins. But it turned out that Snow was greatly assisted by the data that Farr had accumulated in the Weekly Returns. Indeed, it is an open question whether Snow would have been able to make his case to the authorities—and thus likely save hundreds of thousands of lives around the world—without the additional information he drew from Farr’s dataset.

Farr’s rationale for releasing that data is very much in sync with the argument for open data today. No, it was not a solution in and of itself, and its successes were unpredictable (and often indirect, as in the case of Snow and cholera.) But Farr recognized—as I think many of us have come to understand in a contemporary context—that a much larger network of minds existed outside of government, outside the public health authorities, minds that might perceive patterns in the data that escaped the eyes of the authorities. John Snow happened to have one of those minds: a classic 19th-century amateur intellectual, pursuing the great mystery of cholera as a hobby while he kept his day job as a local doctor and anesthesiologist. Most of the official public health establishment had ignored his ideas about cholera, but Farr’s data helped Snow change their minds in the end.

When I look at most of the Open Government initiatives today, I can’t help but see them as a kind of search probe for all the John Snows out there across the country, unaffiliated with government, but willing and able to solve some small piece of the puzzle. This is good news for at least three reasons. The first is simple enough: we will have better ideas inside of government, and a sharper understanding of the problems that confront us, if more people are focused on those problems, even if that focus comes in their spare time, on someone else’s payroll. This is a core principle in Henry and Cosma’s notion of cognitive democracy; with the right tools, when we expand the density and diversity of minds engaged in solving problems, we get better solutions.

The second benefit is almost as direct. By creating platforms that encourage the John Snows of the world, we make more John Snows. In other words, we expand the ranks of the semi-pros and the hobbyists, the people who spend some part of their live trying to improve their government, beyond just voting every couple of years. The line that divides the politicians/bureaucrats from ordinary citizens becomes more porous. And as the class of part-time participants widens, it attracts more people who have other, equally useful, talents to share. The end result is more engagement, more civic participation, and an increased awareness of the services that states provide, and the challenges they face.

Open data has an additional benefit that is worth mentioning, given “the future of news” debates of recent years. I’ve argued elsewhere that there is a great deal to be optimistic about in terms of long-term journalistic trends, even if part of that story is the slow demise of Newspapers As We Know Them. But I think skeptics like Paul Starr are right to be worried about the fate of the investigative journalist, the city hall reporter that unearths corruption (or, on occasion, showcases civic achievements.) Open data can subtly help us avoid this bleak scenario—not by paying for investigative journalism directly, but rather by making it cheaper. When public data is actually public, the investigative side of being an investigative journalist gets a lot easier, or at least it gets more easily crowdsourced by a large group of amateurs and hobbyists who want to help out. Yes, information abundance meant that the newspapers lost their local advertising monopolies to Craigslist and Groupon, but it also means that the crucial data they used to have to unearth by hanging around City Hall for months is now available to anyone with a Web browser or an API key. We may well have fewer investigative journalists on the payroll of newspapers, but if we play our Open Data cards right, we might well end up with more investigations.