Netflix Weirdness

by Kieran Healy on November 23, 2008

There’s an article on the Netflix Prize in the Times today. You know, where Netflix made half of its ratings data available to people and offered a million bucks to anyone who could write a recommendation algorithm that would do some specified percent better than Netflix’s own. What tripped me up was this sentence about one of the more successful teams:

The first major breakthrough came less than a month into the competition. A team named Simon Funk vaulted from nowhere into the No. 4 position, improving upon Cinematch by 3.88 percent in one fell swoop. Its secret was a mathematical technique called singular value decomposition. It isn’t new; mathematicians have used it for years to make sense of prodigious chunks of information. But Netflix never thought to try it on movies.

Can this possibly be true? I’d have thought that just about the most obvious way to look for some kind of structure in data like this would be to do a principal components analysis, and PCA is (more or less) just the SVD of a data matrix. PCA is a quite straightforward technique (evidence for this includes the fact that I know about and use it myself). It’s powerful, but it’s not like it’s some kind of slightly obscure method that isn’t ever applied to data of this kind. And there’s a whole family of related and more sophisticated approaches you could use instead. If you’d asked me about the prize before I read this article, I would naively have said “Well, it’s this effort to get people to help Netflix do better than I guess anyone could using something like bog-standard PCA.”

Maybe the article just got written up in a way that misrepresents the contribution of the team who introduced the method to the data. Or maybe I am misunderstanding something. I guess I should page Cosma and see what he thinks.

Center-Right Nation?

by John Holbo on November 23, 2008

A little something about the whole ‘Obama needs to be cautious because this is still a center-right nation’ thingamajig. (Hilzoy derides it; Sirota has been tracking it; Ramesh Ponnuru questioned the intelligibility of the proposition. No doubt you’ve noticed some of this discussion going around.)

Way back two years ago, I blogged a review of Micklethwait and Wooldridge’s Right Nation. Here was my verdict: “The authors basically have a Louis Hartz ‘liberal consensus’ argument. Do a change-all ‘liberal’ to ‘conservative’. Which is really a substitution they ought to think through a bit harder. Since they cite much of the same evidence Hartz cited for his thesis way back when.”

Consider “Liberalism in America: A Note for Europeans” by Arthur ‘vital center’ Schlesinger, written in 1956, anthologized in The Politics of Hope [amazon]. He takes a Hartzian view. “In a sense all of America is liberalism.” That’s the first line, establishing a certain ‘who’s your daddy?’ dominance. Then what follows is ostensibly more moderate: [click to continue…]

Print, pixels and prescriptivism

by John Q on November 23, 2008

This post on a question-begging argument in favour of carbon taxes and against an emissions trading scheme, naturally raised (!) the question of whether the correct interpretation of a phrase like “begging the question” is determined by the predominant usage or by its original derivation as a technical term in logic or maybe by some other criterion such as the efficiency of communication.

That set me thinking and I turned to the usual research tools Wikipedia and Google to look at how this phrase and a couple of other standard items for debate (“aggravate” and “metholodogy”) are actually used.

[click to continue…]