Nate Silver’s relaunched [FiveThirtyEight](http://fivethirtyeight.com) has been getting some flak from critics—including many former fans—for failing to live up to expectations. Specifically, critics have argued that instead of foxily modeling data and working the numbers, Silver and his co-contributors are looking more like regular old opinion columnists with rather better chart software. Paul Krugman has been a prominent critic, [arguing that](http://krugman.blogs.nytimes.com/2014/03/26/data-as-slogan-data-as-substance/) “For all the big talk about data-driven analysis, what [the site] actually delivers is sloppy and casual opining with a bit of data used, as the old saying goes, the way a drunkard uses a lamppost — for support, not illumination.” Silver has put is tongue at least part way into his cheek and [pushed back a little](http://fivethirtyeight.com/datalab/for-columnist-a-change-of-tone/) with an article titled, in true Times fashion, “[For Columnist, a Change of Tone](http://fivethirtyeight.com/datalab/for-columnist-a-change-of-tone/)”.
> [Krugman] has expressed substantially more negative sentiments about FiveThirtyEight since it left The New York Times, according to a FiveThirtyEight analysis. … [Krugman] has referred to FiveThirtyEight or editor-in-chief Nate Silver 33 times on his blog. FiveThirtyEight classified each reference based on whether it expressed a favorable, unfavorable or neutral sentiment toward FiveThirtyEight. … Mr. Krugman referred to FiveThirtyEight or Nate Silver on seven occasions during its independent period. Four of these mentions were favorable, two were neutral, and one was unfavorable. … During FiveThirtyEight’s tenure with The New York Times, Mr. Krugman referred to FiveThirtyEight or to Nate Silver 21 times. Over all, 15 of these references were favorable, as compared to five neutral references and one unfavorable one. … But Mr. Krugman’s views of FiveThirtyEight have changed since it re-launched March 17 under the auspices of ESPN. The columnist has mentioned FiveThirtyEight four times in just nine days, all in negative contexts.
He prints this table:
Silver goes on to say, “To be sure, the difference in Mr. Krugman’s views could reflect a decline in quality for FiveThirtyEight … While it can be easy to extrapolate a spurious trend from a limited number of data points, the differences are highly statistically significant. At his current pace, Mr. Krugman will write 425 more blog posts about FiveThirtyEight between now and the 2016 presidential election.”
I think new sites always take a little time to settle down, and I hope FiveThirtyEight can get rolling with the kind of quick and accessible, model-based and data-driven analysis that made Silver justly famous. Maybe this post would be a good place to start. Given that he says the differences are “highly statistically significant”, a natural question to ask is whether there is any omitted variable bias in the model. I went and looked at the original columns, which Silver helpfully links to in his table, and quickly coded up two new variables: the topic that Krugman was referring to when he mentioned Silver, and whether the item mentioned by Krugman had a worked-out statistical model underneath. That gives us a new table:
It seems to me that either the topic Krugman was writing about or the model-based nature of Silver’s discussion might be driving the statistically significant results in the original model. Science marches on.
{ 63 comments }
snuh 03.26.14 at 11:42 pm
ok that is pretty spectacular
Cranky Observer 03.27.14 at 12:10 am
Brilliant. And a perfect illustration of the (potential) problem.
Cranky
Sam Tobin-Hochstadt 03.27.14 at 12:20 am
Isn’t there an NYT house rule against criticizing other Times writers?
adam.smith 03.27.14 at 12:38 am
Sam @3 – no, not on their blogs – Krugman has occasionally critically addressed Brooks by name in his blog, e.g. here on big data. The rule only refers to the opEd page.
Tiny Hermaphrodite, Esq. 03.27.14 at 12:40 am
@ Sam
Silver isn’t at the times anymore, he has started his own gig together with ESPN. And I have definitely read Krugman gently chiding other Times writers, chief among them David Brooks,
Patrick C 03.27.14 at 1:16 am
pwned
Bloix 03.27.14 at 1:16 am
Silver’s table shows precisely what’s wrong with the new 538. It’s ridiculous to argue that Krugman, whose every other article is a veiled attack on Brooks, is Times op-ed loyalist. But Silver’s got a table, so away we go.
The new 538 is completely different from the old 538. It’s a lot more like Wonkblog – a bunch of half-digested “interesting” pieces that take way too much time once you realize you can’t rely on them.
Donald A. Coffin 03.27.14 at 3:14 am
I’ve been fairly critical of what’s appeared at 538 so far, largely because the posts in my area of expertise (labor economics) have had three common characteristics:
1) Lack of a clearly defined model of whatever process is being discussed.
2) No *original* research or results being presented.
3) When original research has been *referred to* (NOT actually presented), there’s no way to assess the validity of what’s being said, because all that’s presented are the conclusions, not the methods or the actual analytical results.
Other than that, I don’t have any issues…
alexh 03.27.14 at 4:12 am
Which is worse? Silver has not the faintest idea what “statistical significance” means (and does not)? Or that he does, but is deliberately trying to bamboozle his audience?
mbw 03.27.14 at 4:55 am
Silver’s section on climate in his Signal and Noise book was lousy. He repeatedly violated his own general rule of good Bayesian behavior- try to get some specific knowledge before weighing in. I noticed 3 or 4 basic physics errors, not directly relevant to his confused statistical analysis but still egregious. The most important confusion in his analysis was a misunderstanding of what a model is. It’s a function to generate predicted temperatures given some inputs (e.g. anthropogenic CO2 production and volcanic activity) where the inputs themselves are not part of the model. Silver treated models as simple predictions of temperatures, so he overestimated model errors by attributing uncertainty about input predictions to errors in models.
I used to say he was a closet Bayesian. That proved correct. I now think he’s a hedgehog in fox’s clothing. He knows some general data-handling techniques but has trouble recognizing that there are other things to know outside his favorite areas.
Thers 03.27.14 at 4:57 am
Silver’s “I know math!” fundamentalism is making me nostalgic for vulgar materialism.
js. 03.27.14 at 5:44 am
A Healey post is a thing of joy.
oldster 03.27.14 at 11:23 am
Can you help me out on the second added variable?
“whether the item mentioned by Krugman had a worked-out statistical model underneath.”
By “item” here, do you mean an object in the world, (e.g. Paul Ryan, the Eiffel Tower), or do you mean something like “a published column or essay,” (as in “I see MoDo has a new item up in which she makes fun of Al Gore”)?
If the second, then do you mean it was an item written by Nate Silver? I.e., is the idea that Krugman mentioned a piece of writing that Silver had done, and that Silver’s writing was not based on a statistical model?
That’s what I am slowly coming to think, but it was really not clear to me on the first couple of readings of that sentence. So is this your point: every time Krugman has criticised one of NS’s columns, the NS column has not been based on a statistical model?
That clarification in your prose would help. But then I wonder about how determinate the idea of “being based on a statistical model” is. For instance, how about this new snarky NS column, the one titled “For Columnist, a Change of Tone”: that includes a nifty chart, but would you say that it is “based on a statistical model”? If Krugman criticises it, how should it be coded? NS might claim that it is “based on a statistical model,” since it has a nifty chart. But given how paper-thin the “model” is, Krugman would be within his rights to say that it is not, or not meaningfully so.
Trader Joe 03.27.14 at 11:45 am
Clearly the recent five thirty eight pieces have lacked the depth and detail of some of Silver’s prior work some of which was quite brilliant.
I’d put this down to ESPN and an interest in getting the ball rolling with some “quick hit” pieces, written at the mass market level – not the college educated level, just to let the world know he’s out there and back from hiatus.
Time will tell whether he returns to the higher quality of data and analysis that had historically been the norm or whether analytics about how under 6 foot 3 point shooters perform against match-up zone defenses becomes the height of scholarly work.
Said differently, with ESPN as a sponsor one might expect that sports driven analysis becomes the meat and potatos and political work becomes the occassional appetizer or side dish that lacks in an assortment of ways.
politicalfootball 03.27.14 at 12:54 pm
The first sign that something had gone badly wrong with Silver* was when he chose the fox as his logo – showing that he completely failed to understand the nature of his success, or the moral of the original fable.
In fact, Silver at one time knew one big thing: That statistical analysis could be applied in journalism to stunning effect in situations where the data is rich and the pundits are stupid. Now he apparently knows many things, and he’s as dumb as any other pundit fox.
*okay, maybe the second sign after the global warming chapter in his book.
Barry 03.27.14 at 1:11 pm
Trader Joe, it’s not just that; hiring Pielke was not a start-up mistake, especially after Silver was justifiably trashed for getting climate science wrong in his book. It’s either Silver refusing to accept criticism, or being deliberately dishonest.
Barry 03.27.14 at 1:13 pm
mbw: “I used to say he was a closet Bayesian. That proved correct. I now think he’s a hedgehog in fox’s clothing. He knows some general data-handling techniques but has trouble recognizing that there are other things to know outside his favorite areas.”
The way I see Krugman’s diagnosis, it’s that Silver was able to arbitrage the intellectual gap between polling + poli sci (high intellectual level) vs. election punditry (0 intellectual level). He added some very nice work, but the gap was critical – perhaps his greatest contribution was to assess two fields, realize that one was complete BS while the other was good, and to have the intellectual courage to go with that.
With climate science in particular, he’s doing the exact opposite.
Bloix 03.27.14 at 2:07 pm
#15 – “when he chose the fox as his logo”
I was surprised at that, too. I thought Silver understood that he was the hedgehog par excellence.
thesteelgeneral 03.27.14 at 2:57 pm
I know he misquoted Michael Mann in his book , and Mann was pretty livid about that.
Who can tell me what else is wrong with that chapter?
and with Pielke’s piece.
CJColucci 03.27.14 at 3:33 pm
I hope Silver gets rolling for utterly selfish reasons. I handle cases in which one of the issues is whether the decision-maker conducted a sufficiently rigorous analysis, often of the sorts of things Silver was good at explicating. In fact (and legally), the required level of rigor is not very high, but litigants always say it wasn’t rigorous enough and argue that X,Y, or Z, or all of the above, should have been done. One of my stock responses has been, especially since the elections, “Yes, there has to be rigorous review, but anything can always be more rigorous. That’s not the standard. [The decisionmaker] doesn’t have to hire Nate Silver. What [the decisionmaker] did was as rigorous as the law requires, and that’s enough.”
mattski 03.27.14 at 3:51 pm
With what motive?
TM 03.27.14 at 3:52 pm
The wider problem I think is still that you need to show a level of perceived contrarianism to make such a media enterprise viable. And while there are plenty of opportunities to be contrarian to the mainstream and get it right, you can’t all the time and you can’t make it a business model. It’s no coincidence that Freakonomics and Nate Silver both failed spectacularly with Climate Science: the scientific mainstream is right and the contrarians are wrong, period.
Further, the contrarian business model obviously precludes any real contrarianism in the sense of radically questioning society. The Freakonomics and of course Gladwell brands of contrarianism rely on the art to sell the public what they already want to hear while making it appear that they are learning something new and insightful. Silver cannot successfully implement this strategy while actually remaining “true to the data” (and neither could Freakonomics) because, um, you do have to make stuff up from time to time if you want to remain media darling.
thesteelgeneral 03.27.14 at 4:08 pm
@mattski 03.27.14 at 3:51 pm
>>or being deliberately dishonest.
>With what motive?
Motive is always money. See all the black people on FoKKKs tV. Both of them have discovered there’s money to be had in selling the image of the evil lazy black man to KKKonservatives.
Lee A. Arnold 03.27.14 at 4:42 pm
Nate, Nate, I wonder whether it’s a sustainable business model. To keep big advertisers on board, you need lots of people coming back, every day. The idea of modeling social trends etc. is laudable, despite the additional, non-numerical, analyses that 538 needs to be putting into it (which was evident to me, the first time I looked at it. Occasionally HIRE part-time contractors for 300-700-word guest spots (such as Krugman or lesser lights, etc. Because numbers ain’t everything, and the market for intellectual information in our sordid and benighted world is probably less than 50,000 people a day. Tops.) To survive, the website probably will have to split down the middle, and devote the entire right side (or even 2/3rds) to sports, gambling, big horseraces, etc., all the forms of handicapping. Slide the contributors column to the very bottom, out of sight, because we don’t really care what people look like, unless they look like Gwyneth Paltrow, and prioritize that sports column by game dates. In the same column underneath that, do the larger trends in the leagues, plus statistics on enhancer drug use, coach compensation, etc. etc. The left half (or 1/3rd) of the webpage could do the politics, medicine, social trends, climate, etc.. I don’t even follow sports very much, and I can see this coming. Build an expert database alongside, to match your predictions to events and criticize yourselves (very important!), and optimize the headlines and titles so it stays google-searchable, so in 2-3 years you can join up with Ezra’s Vox and put Bezos out of business, or else get bought-out for megabucks. But don’t listen to me — I am always right, and so now I have to go crawl under somebody’s house and fix the pipes for a living.
adam.smith 03.27.14 at 4:46 pm
Adding to the irony, a Bayesian worldview should warn you against contrarianism. (Standard examples would be that a Bayesian would/should be very skeptical about results such as Benn’s telepathy article or the higher than the speed of light particle motion that some people published).
I’m not sure you have to be contrarian to be successful, though. One can argue about the merits of Ezra Klein’s work (I think it’s mostly quite good), but I don’t think anyone would call him – or people like Khiff, Matthews, and Plumer he brought along from Wonkblog – a contrarian (Matt Yglesias perhaps a little more so, but still not very much). Personally, I see the most promise in the NYT/David Leonhardt’s “The Upshot” project. They’ve hired great people, the NYT has the best data visualization department in the industry, their concept seems more promising, so I’m looking forward to that
Barry @17 – nice point about Nate Silver’s model as arbitrage. That’s spot on.
Bloix 03.27.14 at 4:46 pm
I’ve never understood the fox and hedgehog line to be pejorative of the hedgehog. Berlin didn’t. I think Silver doesn’t understand the saying.
BTW – the lead today is another bit of climate derpery: “Can Evolution Outrace Climate Change?” (answer: no.) The research it describes is perfectly reasonable but the question-mark headline is an outrage.
and PS – Here’s what I think about Silver: he’s not a scientist; he’s not a statistician; he’s not an academic; he’s not an activist; and he’s not a journalist. He’s a gambler. He got his start in on-line poker and moved to sports books, and he found that applying stats to cards and sports allowed him to win.
Then he found that doing a really obvious thing – applying the same simple tools to political competitions, with a narrow focus and a willingness to put in the work – gave a lot better predictive results than punditry, and he rode that insight to fame in 2012.
But he’s now reached his Peter Principle limit. He’s been promoted over his head and he’s got no good ideas on how to be a general-interest news and comment website, so he’s just copying what other people do.
TM 03.27.14 at 6:44 pm
I think Silver is under pressure to show that he’s “better than the others” – after all, having made the best election predictions is his one and only claim to fame. So he has to come up with something surprising at least once in a while (which he did! We are talking about him!). I haven’t read Ezra Klein in a while but he was always quite boring and that’s ok for his readers, they explicitly want the boring stuff. Am I off?
adam.smith 03.27.14 at 7:19 pm
I guess my hope was that Silver would do something like The Guardian’s data journalism – lots of data that’s made available openly and then maybe as the 538 “trademark” add some more modelling to the data – say include the occasional regression or matching analysis or so rather than just simple descriptive stats. That was probably unrealistic – doing this well takes a lot of work and probably doesn’t allow for the rapid pace that 538 envisions, but with all the hype one can dream?
boring is subjective, but it’s always been (and still seems to be) Klein’s business model and worldview that policy details matter. It seems to work for him, which I find very impressive.
Colin Danby 03.27.14 at 7:55 pm
I always read Archilochus as favoring the hedgehog.
mbw 03.27.14 at 8:27 pm
@thesteelgeneral
Here’s a passage from a review I wrote (but decided not to publish) right after the book came out:
mbw 03.27.14 at 8:29 pm
#19 thesteelgeneral
Here’s a passage from a review I wrote (but decided not to publish) right after the book came out:
What had started as a sophisticated discussion rapidly turns dicey when he gets to the actual physics: “…gases absorb solar energy that has been reflected from the earth’s surfaceâ€, an assertion that’s wrong on several levels. The gases transmit most solar energy and absorb much of the energy from the earth because it is not reflected but radiated by the earth at much lower frequencies. The effect he describes wouldn’t even cause warming. Then he writes that without this effect some 30% of the incoming solar energy would be reflected out into space. Again, the reflection is a different process caused by clouds, ice, etc. and it is not much reduced by greenhouse gases. As for the overall energy flow, essentially all of the energy ultimately gets back out into space, the gases only affecting how rapidly it escapes. He then writes “The [greenhouse] prediction relies on relatively simple chemical reactions ….†when the calculation he’s describing relies on simple spectra for the absorption of infrared light, not a chemical reaction at all. He thinks that CO2 lasts about 30 years in the atmosphere. The actual concentration changes trail off on a wide range of time scales extending to more than a millenium. At this point one realizes that he’s making just the sort of mistake he describes in other contexts, trying to play poker without knowing anything about cards. You might win, but it depends too much on luck, and you’re vulnerable to hustlers.
Wonks Anonymous 03.27.14 at 8:35 pm
I think Silver’s choosing the fox was inspired by Tetlock.
mbw 03.27.14 at 8:44 pm
OK, this motivated me to put that draft review up:
.
mbw 03.27.14 at 8:46 pm
OK, this motivated me to post that old review of Silver here:
Silver’s Signal and Noise
Consumatopia 03.27.14 at 9:31 pm
That’s what makes it such a brilliant troll! Krugman probably doesn’t want to say “It’s ridiculous to argue that I’m a Times op-ed loyalist. Look at all my veiled attacks on on other Times writers.”
mbw 03.27.14 at 9:51 pm
#19- re Pielke’s piece.
Here’s a nice discussion: title-“Climate article”.
Pielke is right that the economic signal of increased storm damage is lost in the noise. He incorrectly then jumps to the conclusion that it’s zero. And he ignores non-economic measures that show that certain types of extreme weather (large tropical storms) expected to worsen are indeed worsening. Some other types of extreme weather may not be worsened by climate change.
TM 03.27.14 at 10:29 pm
Excellent review 33. Also liked http://www.nakedcapitalism.com/2012/12/cathy-oneil-why-nate-silver-is-not-just-wrong-but-maliciously-wrong.html.
Bloix 03.27.14 at 10:56 pm
#33 & 36 – those reviews are just great. Something they both imply without quite saying is that gamblers are not liberal in temperament. Gamblers choose to live in a zero-sum world. Taking money from other people makes them happy, not just because it makes them rich, but because it proves that they are smart and the other guy is stupid. This is not a liberal outlook on life, and we shouldn’t expect Silver to be particularly liberal politically.
Bloix 03.27.14 at 10:57 pm
#31- you’re clearly right. Thank you.
#34 – lol
JW Mason 03.27.14 at 11:09 pm
mbw’s review is very good.
js. 03.27.14 at 11:12 pm
Just joining the chorus here: the mbw review linked @31 is great. Thanks!
GiT 03.27.14 at 11:25 pm
“Time will tell whether he returns to the higher quality of data and analysis that had historically been the norm or whether analytics about how under 6 foot 3 point shooters perform against match-up zone defenses becomes the height of scholarly work.”
Hm, is election forecasting really such a high and noble pursuit compared to sports forecasting?
Ronan(rf) 03.27.14 at 11:34 pm
The naked capitalism/mathbabe link is interesting as well.
A question for anyone who might know – Kathy O Neill says in the link that she’s writing a book on modelling that complicates Silver’s book (Id assume for the layperson, showing how people can misuse models, when they’re useful etc)
Does anyone know of any books/papers that fit this description available at the minute ? Non technical as much as possible, more the theory/history behind mathematical modelling (or what have you) than a ‘how to’ guide ..
Tabasco 03.28.14 at 2:49 am
I’m going to ask a dumb question. Krugman has mentioned Silver six times since he moved to ESPN. How can anything based on a sample of six be “highly statistically significant”?
Tabasco 03.28.14 at 2:50 am
Sorry, that should be sample of four.
faustusnotes 03.28.14 at 4:00 am
i think the sporte stuff would be interesting. but that stuff is hard and beyond nate’s ability.
adam.smith 03.28.14 at 4:39 am
@Tabasco – this is a back-of-the-envolope calculation and there are more format/correct ways of doing this, but here goes:
Let’s for simplicity only look at posts where Krugman made a fav/unfav judgement – there were 16 of those during the NYT period, 15 of them positive. So our the estimate of the probability of unfav based on that period is 1/16 or ~.06
If you want to, you can calculate a confidence interval around that – the standard deveiation is sqr((1/16*15/16)/16) which is also around .06, so we get as the upper bound of the 95% confidence interval around .2 (.6+2*.6 with some upward rounding to make up for downward rounding before.)
Now statistical significance is based on the idea “how likely would these results be if my null hypothesis were true”? Our null hypothesis is that the probability of unfav stays the same after Silver leaves the NYT. Recall our upper bound for the probability of an unfavorable mention was .2. Given that, what’s the likelihood to get for unfavorables in a row? Simply .2^4= .0016 – i.e. fantastically unlikely. So we say we reject our null hypothesis that the probability of an unfavorable mention has staid the same.
We don’t always need large sample for statistical significance. If an effect is strong enough (and, obviously, excluding the possibility of measurement error), even tiny samples can yield very high significance.
(needless to say – this is not meant in any way as a defense of Silver’s substantive claims. It’s purely to illustrate statistical significance).
Tabasco 03.28.14 at 5:17 am
@ Adam.Smith
If I understand you correctly, there were 16 coin tosses when Silver was at the Times and 15 times they came up heads, there have been 4 since he left and four of them came up tails, and you conclude it’s highly unlikely the same coin was used.
That’s all very well, but how do you know that the probability of a favorable mention is/was 50%?
adam.smith 03.28.14 at 5:41 am
Your summary is fine, but translating this into coin tosses might have thrown you off. Nowhere do I mention or use that the probability of favorable/unfavorable is 50%. Why do you think that? All I’m using is that it’s a binary choice – a mention is either favorable or unfavorable and that’s by definition since I excluded all neutral mentions.
GiT 03.28.14 at 5:43 am
Tabasco –
The 16 tosses at the NYT were used to estimate the probability represented by the coin. At no point was it assumed that the coin was 50/50.
Your first para belies that with “it’s highly unlikely the same coin was used.” Each set of events gives us an estimate of the likely values of the underlying “coin” (read: probabilistic process, or something like that. I don’t think I’m actually clear on the ontology here…) The question is how likely two sets of events emerge from the same underlying “coin.”
So if one set gives us a probability estimate of 15/16, and another gives us one of 1/4, how likely is it whatever’s going on underneath is the same?
A different question would be,’ how likely is it that either set of outcomes would occur from a 1/2 process?’
Tabasco 03.28.14 at 6:25 am
OK, forget the coin analogy.
But the statement “what’s the likelihood to get for unfavorables in a row? Simply .2^4= .0016” implies a series of independent events, does it not? If Silver is writing nonsense time after time at his new site, maybe for the same reason, such as he has to dumb down his articles for the ESPN readership, then Krugman is consistently going to criticise him, and four critical mentions in a row might not be so unlikely.
adam.smith 03.28.14 at 6:33 am
sure. Silver actually makes that very same point (“To be sure, the difference in Mr. Krugman’s views could reflect a decline in quality for FiveThirtyEight”). The “statistically significant difference” just refers to the fact that the probabilities for an unfavorable mention are different. It doesn’t tell you why they have changed.
Chaz 03.28.14 at 7:29 am
I used to read 538, but even when I read it I never got all the Nate Silver hype. I saw him in three distinct phases: predicting the state by state outcome of the Clinton-Obama primary (blogging under the alias Poblano at Daily Kos), predicting the outcome of the Obama-McCain general election at fivethirtyeight.com, and then predicting the outcome of the British House of Commons election soon afterward.
The results of his first expedition into political forecasting were pretty bad. In the Clinton-Obama race, he built up a regression model where he tried to account for the results in completed state primaries by demographic data–he broke voters down into a ton of categories, like race, region, etc., got the population share of each category in each state (from the census I guess), and then tried to derive the Obama-Clinton split in each subgroup (like 50 you know!) by leaving them as variables and fitting to the election data. Oh, the data I mean here is the headline vote total from ~8 states. That’s right, he DID NOT use any polling data that actually had info on how those subgroups voted, he just guessed from the official vote count. He was using ~50 variables to fit a curve to ~8 data points. Half the comments in his blog were statisticians screaming that he was overfitting, but he soldiered on! The results were astoundingly bad. Every pundit beat him.
Next he did the general election. Lots of polling data, which he averaged to make predictions. And he converted the response percentages into odds. Well done, Nate, but not groundbreaking by any means. Everyone acted like he had some wizardry but he was literally just compiling polls, and I never got all the fuss. Oh, and he had a formula for weighting which polls were especially credible, I think based on results from past elections. Basically just cut the weighting of Rasmussen. Smart move. BUT WAIT! It wouldn’t be Nate Silver without a regression! So sure enough, he reused that same, awful, utterly debunked (and strictly proprietary) demographic model from the primaries, used it to predict each state, and mixed result in with the polls. The 538 secret sauce no other aggregator has! But he was a smart boy this time; he only gave his own model like a 5% weight, and had its weighting decrease as the election came closer too, so it couldn’t screw things up. Good move; the only better move would have been to not have it at all. This model got excellent results, because it just parroted polls and presidential general election polls are pretty reliable. Although to be fair his predictions were better than the network news or TPM (ahem) which were listing Montana and New Jersey as “too close to call!!!!”
Then he needed to keep the party going so he modeled Britain. He came up with a spiffy new model called Proportional Swing which got results somewhat worse than the (pretty similar) slightly more mundane Uniform Swing model the press had been using for years.
Oh and then he did the 2012 general, yeah? I believe he “modeled” it exactly the same way as he did 2008 and got pretty accurate results. (Honestly, are we going to call it a “model” when the input data is literally a prediction of the outcome you are predicting?) And this time it was suddenly his big breakthrough? I would say the breakthrough was 2008, which is why he had a NYT section in 2012.
Peter K. 03.28.14 at 1:50 pm
Yeah I remember it as Chaz @ 53 does. Silver predicted the 2008 Democratic primary. Then he confidently contradicted 2012 conservative pollsters who said Romney would win. His book “The Signal and the Noise” had a great title. What does one do on the Internet except filter out the noise in a fruitless search for the signal?
From the Naked Capitalism review: “Silver has gone to great lengths to make his message simple, and positive, and to make people feel smart and smug, especially Obama’s supporters.”
Lolwut? Oh how so very contrarian. NC has too much anti-Obama bias noise.
His message seemed to be – from the book title for example – that he was following the data and there wasn’t any “agenda.”
His bio suggested as much, from the mbw review @34 “Silver describes getting interested in politics partly as an outgrowth of the effect of legislation on his career extracting money from gullible “fish” as a poker player. ” But he blogged at Daily Kos.
Krugman among others were pushing Sam Wang for Princeton as an alternative back in 2012 if I remember correctly. My guess was that Silver’s chapter on Rumsfeld and the Iraq war wasn’t vitriolic enough for them, but that hasn’t come up.
Chatham 03.28.14 at 2:05 pm
It’s odd. The 11/21/12 article Silver lists as “favorable†merely shows a clip from the Adam’s Family with Krugman saying one of the kids looks like Silver; the 8/5/13 article listed as “neutral†has Krugman saying he’s a Nate Silver fan. I only checked about 4-5 of them, so I wouldn’t be surprised if there were more that he mislabeled. Not a good sign for a supposedly data driven site.
I also think people are forgetting what made Nate Silver famous when they talk about how he did such a great job predicting the 2012 election. Sam Wang and Drew Linzer had more accurate predictions than Silver. What Silver was good at was getting visibility and writing a steady stream of posts that the masses would enjoy (compare that with, say, Sam Wang’s site that is hosted at Princeton and hasn’t been updated in half a year).
Random Lurker 03.28.14 at 2:48 pm
Hey I didn’t follow closely Silver’s tracking of elections, however:
Suppose that I know nothing of USA elections, I only know that there are candidate A and candidate B, and one will be the president.
With this only information, I have 50% chance of making the right call.
So predicting an election result isn’t such a great feat, the real question is (if it is true that most pundits got it wrong) how is this possible that pundits are on the aggregate less smart than a coin toss.
The answer to this question is likely political (it is known since the fifties that surveys that give a candidate an advantage tend to influence voters in favor of that candidate).
TM 03.28.14 at 3:49 pm
Agree 53. It is mysterious why Silver became so hip. Must be because Reps were trying to discredit him so he gave liberals something to be smug about. Now he’s predicting a Rep Senate takeover so maybe liberals will be less smug this time. One should note another underlying problem with the whole hype. Why is it important to have predicted the outcome of Tuesday’s election on Monday? Why not just wait until Tuesday? It’s the kind of prediction that actually shouldn’t matter and it does matters only for the wrong reasons – because reliable polls drive campaign spending and the US electoral system is so screwed up.
adam.smith 03.28.14 at 4:20 pm
RL @55: In fairness, Silver predicted the outcome for 49 of 50 states. One can argue about how impressive that is, but if you knew nothing about US politics, it’d be impossible.
but more importantly:
TM @56 – I think all of this misses a good part of Silver’s popularity, which had two components.
1. As Bruce puts it nicely @17, it was a contrast to the hapless coverage of the campaign by “insiders” (think Mark Halperin and the like), which turned every little event into a big affair. If anything, Silver’s model was probably still too sensitive to day-by-day events. But that alone wouldn’t do it – as others have pointed out, Silver wasn’t the only person doing this type of stuff. There was also
2. Silver is a really engaging writer about statistics. He was able to make the write-ups of his model/polling updates interesting to a lot of people. Obviously being right was important to the appeal, but it was the combination of being right and being entertaining that made him a star.
Bloix 03.28.14 at 5:10 pm
#55-
“What Silver was good at was getting visibility and writing a steady stream of posts that the masses would enjoy”
You write this as if communicating with a broad audience in a lively and persuasive way is a contemptible thing.
Sam Wang had zero success in reducing the flow of garbage that pours into the stream of public debate. Nate Silver did do it and he served a lot of cool, clear information to a audience that had been living on toxic waste for years.
Trader Joe 03.28.14 at 7:07 pm
Maybe I’m the only one that reads them here, but the sports related commentary has been pretty decent – particularly for football and baseball.
I realize this is of a little less importance than getting global warming ‘right’ but the vast majority of Silver’s audience has already formed a concrete view on global warming and he doesn’t add credibly to the topic whatever his data might have suggested.
However, plenty of his audience has fantasy baseball drafts this weekend and will value his take on Spring Training stats or participate in NCAA Basketball tournament brackets and value his weighted probabilities for each of the 63 games….
John Quiggin 03.29.14 at 12:37 am
I think the big problem for Silver’s new venture is that, outside politics and sports, there’s not a lot of low-hanging fruit. These are areas that have long been dominated by innumerate pundits relying on their supposed intuitive grasp of the subject matter, so it’s easy to beat them using tools as simple as Ordinary Least Squares and an understanding of statistical significance, or, in Bayesian terms, the relationship between priors and posteriors.
But economics and climate science (to take the scenes of his most spectacular debacles) aren’t like that. Of course, there are plenty of innumerates weighing in, but there are enough serious professionals that you can’t just take a data set and a stat package and set everyone straight.
Add to that some unforced errors like giving space to Roger Pielke Jr, and you’ve got a recipe for failure.
Barry 03.29.14 at 12:27 pm
Trader Joe: “I realize this is of a little less importance than getting global warming ‘right’ but the vast majority of Silver’s audience has already formed a concrete view on global warming and he doesn’t add credibly to the topic whatever his data might have suggested.”
I like the way that you say that, as if there’s no data available.
TM 03.31.14 at 2:44 pm
“I think the big problem for Silver’s new venture is that, outside politics and sports, there’s not a lot of low-hanging fruit.”
There’s also the fact that in sports, fans do generally agree who won and who lost the game, and they mostly agree on the rules of the game and the procedures for interpreting them. Further, there don’t seem to be powerful lobbies with a vital interest in distorting the public discussion of sports. And there is this thing they call a “level playing field”.
Comments on this entry are closed.