Does digital data disappear?

Redundant storage is the key survival strategy for digital records. Our own DNA is massively replicated, so nature has already told us how best to preserve critical data. As storage moves to cloud-based systems, the service providers will replicate data on a vast scale so as to protect against loss-related legal liability. As long as data storage technology continues to improve exponentially, the cost of taking this brute force approach will be negligible.

JP, then stop sitting around your house and get out there and make some copies! :)

The short answer is yes, it does. The long answer is so does everything else. The time frame varies, of course — books printed in the Eighteenth Century, for example, are likely to survive those printed in the Nineteenth. (See Slow Fires for a good overview, and a healthy cultural frisson.)

As long as enough representative bits of former glories remain, we should do fine. It would be nice if the Parthenon still looked as it did on the day it was first dedicated, we could still peruse the stacks of the Bibliotheca Alexandrina, or dine at the Brown Derby, but time marches on — and so do we.

Yes, I’m sure publishers who are blithely replacing their blogs’ native comment systems with the eye-bleed-inducing majesty of Disqus have never heard of Haloscan.

Honestly thought we’d grown beyond the desire for fragile, third-party javascript-based commenting systems.

There’s a fair bit of digital data (in absolute terms) that was created back before software was widely sold or otherwise standardized. It seems to me that fear of undecipherable digital data mostly comes from extrapolating (e.g.) harrowing experiences with records created on long-obsolete machines with proprietary, internally-developed software at big corporations or NASA in 1970. It’s hard to find working hardware to even read the tapes, the tapes may have suffered in storage, and the only “native” application for it was a 50 year old FORTRAN or COBOL program running on a long-gone mainframe. It may be an effort finding even a single source code or binary copy of the software that created or used the data.

It seems to me that digital preservation alarmists have incorrectly projected that experience to widely-distributed software and its associated data formats. If retrieving an overlooked reel of Apollo data takes herculean effort in 2010, reading an old Visicalc file created on the Apple II will be similarly difficult by 2020, the logic goes. If you wait until 2020 to read your 1980 vintage files, the magnetic storage medium may indeed be too deteriorated. I migrated my Commodore 64 files in the early 1990s; avoiding deteriorating storage media is the most important part.

If you have the data preserved, just fire up your emulator and read the software with its original application. You can’t get easy data interchange with modern computer systems that way (no copy/paste rows out of Visicalc and into Excel!), but neither can printed paper records. Millions of people used Visicalc and collectors have helpfully placed archives of its various versions online for free. This is technically a copyright violation but no publishers are losing money over it so nobody really cares. The NYT article discusses and dismisses emulation in one paragraph, basically because it’s unwieldy, but it’s much cleaner and simpler than trying to shoehorn past-popular software formats into the constraints of currently-popular software.

Finally, if your data is stored in a widely understood format with several implementations, there’s no reason to fear that you’ll even have to dip into an emulator. TIFF and PostScript files created in 1986 are still readable today with zero extra effort. XML, PDF, JPEG, GIF, ZIP, MP3, MPEG2, and other formal and informal file standards are likely to remain readable with mainstream software decades into the future. If not, there’s always emulation again.

There’s one point about digital archives that curiously wasn’t in the NYT piece. We have higher expectations for searching digital archives than for searching paper. We don’t expect to walk into a rare book room, shout “Isaac Newton” and instantly see the books with relevant text leap into a relevance-ordered stack for reading. But we do expect similar magic when it comes to searching digital archives. This is where it gets tricky with emulation: if you have an interesting document in obsolete Wordperfect 2 format, you can easily snag a copy of old Wordperfect 2 from the pirate-preservationist cloud and start reading in minutes under an emulator. But if you’re relying on an automated search system to find interesting documents, you need a bridge between that old Wordperfect document and the document indexing process before you even discover that its contents are interesting. Nobody has bothered to write bridge software for most obsolete/proprietary formats. In the future, either people will write bridging tools so that all formats can be searched in a uniform way or the oldest and least popular formats will sink to the level of paper books, only revealing their contents when deliberately opened by humans.

Tremendously kludgy idea for rapid, uniform bridging of obsolete document formats: associate each document with an appropriate software emulation environment. Set up genre-specific macros to systematically go through documents (scroll down and across in spreadsheets, scroll down through pages of text in word processors, etc.) and capture screen shots for each screen-full of information. Then run OCR on the screen captures just like you would for scanned pages of a paper book, and index them the same way. Just as Google Books captures pictures of book pages and recognizes words and numbers in them, do the same with software, capturing it all as pictures and relying on OCR to provide good-enough indexing so it will be noticed if further study is warranted.

Matt@10: No, we don’t have reliable or even approximate estimates of the proportion of written materials from c. 1000AD that has survived. The number of written documents increased dramatically shortly afterwards, i.e. in the central Middle Ages, during and after the great economic boom of the 1050-1250s AD and the concomitant rise of literacy among the general population. I would guess that about one million pages written in Western script, mostly on parchment (paper, less durable but cheaper, became gradually more common), written before 1500, have survived.

The printing press of the late fifteenth century generated yet another explosion of written materials but again, mostly on paper.

By the way: parchment, as opposed to paper, is highly resistant to insects and mold; even water does not usually leave lasting damage if it’s good quality parchment. Fire is another story: much more would have survived if not for the horrendous devastations in the two World Wars of the 20th century.

I am intrigued by your suggestion for “really durable digital storage.” But there is no money in it, is it?

Haven’t some works of Archimedes been recovered from parchment copies which had been repurposed for contemporary use? That would put their survival nearer two thousand years.

Being connected can have important implications for the survival of information. If you’re recording an atrocity, the authorities can’t confiscate the data if it’s been uploaded beyond their reach. Replication is perhaps the best general approach to preserving information.

(JP Stormcrow: three kids? Strictly speaking, with two or more all you know for sure is that it’s between half and one, at least if you have a son or you’re female [assuming you’re a mammal; it’s the reverse if you’re a bird].)

As this genre goes, the article isn’t actually all that alarmist; unfortunately, it has an alarmist headline, and the Sterling example might make it seem like the author shares his view, which I don’t think she does. I also don’t think it’s aimed at people who already pay attention to how and what they store digitally. One of the main points of the article is that you have to do ongoing maintenance on your digital files, so if you already do that it’s not going to seem remarkable. Matt’s comment about migrating Commodore 64 files in the 1990s is exactly one of the points of the article.

But I bet a lot of people didn’t do that and aren’t doing the equivalent today: I have nothing from my Commodore 64 and none of the programming I did in Pascal or Logo or Basic when I was a kid, and my parents didn’t save any of that – unlike some of my paper-based work. I also have none of my e-mail from my university sponsored account. Of course I could have kept almost all of this if I’d thought about it. I may even find some of the old disks, though I’d have to track down a 51/4″ drive.

Last summer, I came across some old 3.5″ floppies, and after running an old version of word on an old laptop with an old floppy drive, I was able to migrate all of the papers I wrote in college in wordperfect 5.0 into a more recent (but still proprietary) file format without much problem. (Although I do miss the old blue screen.) But that was because it occurred to me to do something.

What digital archivists – that is, the ones who are actually working with digital materials – have run into in recent years is a breakdown in the old ways of acquiring and processing donated materials. You used to have people show up with boxes of papers, on paper (including photos, etc.) and maybe some magnetic media in the form of audio/video stuff and that was pretty much it. But now you either get donors who don’t think the digital stuff is “archival” enough to donate, but who have it, or you get boxes of material that include stuff dating back to the 1980s that hasn’t been migrated and hasn’t been stored properly. And then they have to recover it. That doesn’t mean it’s impossible, of course.

I take this article to be an attempt to keep that from happening over and over down the line. And to get people who don’t plan on donating anything anywhere, but who want to preserve stuff that’s important to them, to think about how they’re going to do that.

On the flip side, and maybe this should get into print more often, you see people argue that as long as we keep our digital preservation systems running, we are very likely to be able to save a higher percentage of material than was possible in the past. And I suspect they’re probably right. But I also suspect that a lower percentage of that is going to be preserved by accident, in the sense of discovering a great-ancestor’s bundle of papers in an attic.

At the risk of missing (ignoring) the point, surely this is all partially good news from a privacy point of view, no? There are no doubt some people who don’t want permanent digital archives (scholarly or otherwise).

In speaking of digital data engraved in stainless steel, I was inspired by the Long Now Foundation’s Rosetta Disk concept. Instead of the spiraling design and natural language you could use constant-scale markings to store compressed digital data. Standard laser engraving machines should easily be able to make the markings. You might need to include a natural-language cover sheet or two explaining the encoding process. Due to the greater information density achievable with compact symbols and data compression, I think the cost can be competitive with paper records, despite the much higher price of stainless steel. But this is sadly faint praise compared to cheap consumer-grade storage costs of about 5 cents per gigabyte.

I have vague past recollections from Usenet of people talking about occasionally printing out must-retain data on archival quality paper when digital storage wasn’t trusted. I don’t think there is any common solution to the problem of passively preserving digital data for the long term. Most organizations either don’t have digital records that they expect to need in 200 years or they expect to maintain and migrate the records as necessary every few years. Almost nobody is clamoring for a solution where they can produce a physical artifact representing the records, lock it in an unused vault for a couple of centuries, and expect their descendants to easily read it 10 generations later.

I wonder how well hard drives would serve for century+ storage. How rapidly do the magnetic patterns randomize over time if the drives are stored with as much care as rare books? If the degradation is reasonably slow, predictable, and uniform, you can initially store the data with whatever degree of error-correction encoding is necessary to survive X centuries. I wouldn’t expect anyone to have a SATA cable in 200 years, nor would I expect the mechanicals to work after so much idleness, but the drive casing might serve as reasonably cheap and robust environmental isolation for the data-dense platters. A few centuries from now the drives could be opened up in a clean room and the platters read with whatever equipment is then current, as long as the platters retain data well enough.

Having all the bits to your thesis in, say, NisusWriter will do you no good if you don’t have something that reads NisusWriter.

One thing I’m always curious about when this topic comes up is why cryptography principles don’t apply. While not a trivial problem, or one that casual archivists would undertake, surely it’s no harder in principle to “translate” an obsolete text format than to crack a moderately straightforward code. With the added benefit that, for the most part, the format won’t have been designed to be hard to crack. And, quite possibly, the older the format, the less complicated the analogous code.

bad Jim@13: JP Stormcrow: three kids? Strictly speaking, …

Yes three. I tend to speak “likely” rather than “strictly”. Although strictly speaking your calculation did not include a discount factor for the possibility of my not actually being their father. All I know for sure is nothing.

Dump everything into gmail, like I do. What can go wrong?

One of the more famous cases of data-rot (or format rot and hardware rot) was the BBC’s Domesday Project, designed to commemorate the 900th anniversary in 1986 by collecting accounts of local communities from schools and colleges. Unfortunately, the method of distribution (laser discs and highly customised BBC Master System computers) went from being Tomorrow’s World to Today’s Junk very quickly, and it’s taken the best part of the last decade, and the hard work of very many people, to get the emulation right and finally put the data online.

Looking at just the [print/MS] survivors doesn’t give a good idea of durability.

At best, it gives a sense of priorities. As someone whose archival research only stretches back 300 years, it became very obvious through the manner of their archiving that many pamphlets survived thanks to the personal interest of a handful of collectors. The great libraries of the Elizabethan era regarded playtexts as ephemera; private collections supply us with the bulk of the theatrical archive. The intention of digital archivists is to diminish the need to prioritise at the outset.

I’d agree with Clay’s survey, and am reminded of how Jason Scott, who now works for Brewster Kahle’s Internet Archive, recently put out an alert and obituary for 5.25in floppies. In addition, one of my ongoing interests is the shifting proportion of people’s digital vaults that can be classed as ‘commodity data’ (commercial multimedia, OS files, applications, etc.) for which there’s a solid presumption of duplicability and retrievability, as opposed to personal data that requires either sharing or some form of backup in order to survive catastrophic failure. If ubiquitous streaming access or some kind of referential storage alleviates the burden of copying or backing up one’s music collection or samizdat archive of obscure BBC documentaries, then perhaps it also permits greater focus on preserving personal data, but such a move would mean transferring both the access rights and the archival duty to third parties, and the stories of the BBC’s past backup-wiping habits may give one pause.

I also wish that I’d been more active in keeping personal copies of some of the earliest websites: it’s a chunk of social history that looks more and more like an archaeological project.