Archiving

There’s no expiry date on old caches, right?

I think there is — I believe if the page changes at all the cache will eventually be updated and the old one will be replaced. If you want to keep a record, then you need to save it somehow yourself (screenshot, web archive, pdf, whatever) and keep it somewhere you control, whether in your fireproof safe or on your website.

Use the Wayback Machine: http://www.archive.org/web/web.php

That way, you can also (more closely) support APA/MLA style for citations, right?

Of course, there’s no guarantee the Wayback folks will stay solvent either.

My suggestion? Use Wayback for most everything, and local HD for most everything else (you can save whole Web pages in IE/Firefox/Safari) & get one of them terabyte arrays Henry (was it?) was bragging about the other day. I noticed LaCie now has a $400 1TB external hard drive recently–well within the price range of all but the poorest CT readers.

‘someone who new’ gggrrrrrrr…not even the spell check on Firefox 2 can save me.

I think what you want, though you don’t know it yet, is WebCite.

I’m pretty sure Google doesn’t have an archive, just one recently cached copy of some pages, and that will go away fairly soon after the page does. The Wayback Machine is a better bet, though unfortunately if the domain changes hands it seems the Internet Archive allows the new owners to determine whether they continue to archive the pages — that is, if the new owners put in a robots.txt file that excludes the archive.org bot, the earlier pages will be removed from the archive.

And of course TinyURL URLs will be completely useless if the tinyurl.com site disappears.

So enterpeneurs could and should create an archiving service, like the Wayback but specific and on demand, could be for a fee, or ad revenue-driven. Tiered solutions, with a kind of… yeah, like those cryogenic services that freeze your brain for near-forever! Yeah! Content cryogenics! Big money! Hey!

Actually, I meant http://www.furl.net/. Lets you keep the page as it is.

Oh I see. “Never” having the usual startup meaning, “At least until the next round of VC funding.”

It depends how secure you want your copy to be; the only certain answer is to keep copies yourself, or to have a very large and reliable archive keep the copy. The usual meaning of “cache” in computing is a place for temporary local copies, so I wouldn’t be relying on Google’s cache.

Hunh. I wonder if there is an “archive” URN standard? A way to say “page found at this URL on this date and time”? That would, if it existed, be similar to a library catalog reference; publisher, date, serial number. Of course, you’d still have to actually store the document.

(pause to research)

well…there’s an expired internet draft for something called a “DURI”–dated URI. It was written by Larry Masinter of Adobe, but it doesn’t seem to be going anywhere. Getting in moving would probably be a good project for some library science degree candidate.

A service that doesn’t quite fit your use case, but is nonetheless handy for keeping around snapshots of webpages is Furl (now owned by Looksmart). With Furl, you can create your own snapshots of webpages, which will remain indefinitely on Furl’s servers. You can also download a zip file containing all of your stored content. I don’t think it gives you the ability to publicly surface a page you’ve previously cached, however, so it might not be of much use to you.

I would use the Web Archive, as suggested above. But that solution assumes the page of interest exists on it.

Alternatively, you could create a pdf of the page and put it on your own site. What I might do in such a case is create a separate HTML page on which I link to the original, but then note that if that’s no longer available, the linked-to pdf below should work.

Rather, see: http://www.archive.org/about/faqs.php#1

Thanks, Erik.

This linked page suggests eight weeks:
http://www.alexa.com/site/help/webmasters#crawl_site

“Alexa will include your site in the next crawl of the web, usually within 8 weeks of submission.”

Of course, that’s likely still too long to meet John’s current needs.

Many pages rendered by your browser do not exist in HTML form until they are rendered– they are the output of a query against somebody’s database run through formatting software.
Amazon, ebay, or flickr “pages” are like this, for instance, as are “pages” from services that display scientific data. Keeping your own copy or paying someone else to archive data whose format and organisation is your responsibility is really the only way to preserve this kind of data, ideally in cooperation with the current curator.

Data archives (libraries, among others) have considered the problem also; you’re not the first to ask this question.
When has a page changed?

Keep the page yourself, with the Scrapbook Firefox extension – its so much better than a stack of printouts.

You can save it, edit it, crop it, print it, search it, attach annotations, etc… Get a single page, or follow several links deep into a site. It will save the original URL for you, with internal links structures intact. If the page disappears, you’ve got your own copy – and I think that would qualify as “fair use” if the original is no longer available.

If you need your Scrapbook to be linkable for the public, put it on your site.

abb1: uh… no. Preserving the output medium is exactly the point, because that’s what you’re interested in directing your readers to. Ideally, you’d want to be able to point them at the state a page was in at a particular time in order to evade the author changing or removing the page.

If you think that reconstructing output pages solely from the database “shouldn’t be too difficult” then I suspect that you haven’t thought about the problem hard enough.

Many pages rendered by your browser do not exist in HTML form [..] flickr “pages” are like this

Hmm.. can you elaborate? Flickr tends to have very stable URLs. Sure, results to a search may change and yes, comments can be added changing some content on the page, but generally speaking, a photo’s, a set’s or a group’s URL seems constant on Flickr. (The issue of people being able to replace photos on a page is a different one, I don’t think that’s what you were addressing here.)

Nah, what you really care about is the data, not all the zillion snapshots of various presentations of this data.

Flickr, like many services, generates base URL in a way closely connected to the way underlying objects are stored in the db; in flickr’s case (as with most services), there’s a core object the page is associated with. I was actually thinking of added comments or edits, both of which change the content of the page displayed; capturing the human reasoning “the photo is key; same photo, same URL, everything OK” in a program is not easy. That intuitively stable mapping may not exist; consider the document http://del.icio.us/recent,
or a vendor’s “you may also like” recommendations.
The core attribute linking the entries on that page is the result of a query against an ever-changing db. Or consider this very exchange– are comments not written by the entry’s author a part of this page or not? What if the interface was richer and links to threads with similar keywords were displayed– would those links be part of the page? What about stable URLs associated with an astronomical object that just blew up, or a gene for which there’s a newly reported functional assay? Same query, same URL, same “core” entity, but a qualitatively different page. These are not marginal cases– pages with a single definite human author and denumerable edits by that author are already rare.

Static documents are at least superficially indistinguishable from dynamic objects that are pretty different from them, making scope and synchrony questions essential for any internet archiving beyond saving a small amount of html locally. This leaves aside the unformalizable question of format/content, which is not one of HTML’s strengths.

On a local note– are the Shakespeare gardens behind Vogelback being kept up?

Yeah: Furl, Scrapbook, and Spurl are all good for making archive copies. With Furl and Spurl you also get to see who else has made a copy… so you can, like, ask to borrow theirs.

Sfisher, sure, clearly if what you have is a bunch of documents – you need to archive the documents.

And it’s certainly true that archiving databases often isn’t simple.

I’m just saying that conceptually you’re probably better off with a copy of the database than with a bunch of reports based on this database.

1. Archived copy (with Refresh) – With Furl, you can save a copy of a web page and it is archived for you. This means that you can access that page and read it any time you need to, even if the web site is down, or the page has changed on the original web site, or even if the page is no longer accessible for free. In those cases where the web page has changed and you would rather have the new content, you can refresh the archived copy at will. [furl]

2. Full-page capture programmes, like Zotero. Some other browser plugins can also do it.

WebCite looks pretty sweet.

Here is an example I just made:

http://www.webcitation.org/5MSlJEAAW

Here’s a recent bibliography from a presentation title “New Tools for Preserving Digital Collections”, given by Tracy Seneca of the California Digital Library at the Library & Information Technology Association’s 2006 Forum. Go here for the Forum’s main page.

Then there’s the January 2007 issue of Ariadne, with relevant articles and conference reports.

That information is oriented to organizations (such as libraries) seeking to archive web pages. But you are going to have many of the problems they’ve already spent a lot of time analyzing.

If you’re only interested in “pretty good” persistence for someone else’s document, and you want to be able to point people to it with a URL, then webcite seems to be the way to go. Assuming the pages aren’t already being archived by a national library.