Comments on: Attractive Models

By: derek

derek — Sun, 24 Sep 2006 19:59:42 +0000

Paul, it is not actually true that negative results aren’t valuable: they are. The trouble is that they aren’t sexy. My lab tutors hammered it into me that to only report your positive results is only slightly short of fraud.

They were probably being polite to say “slightly short”. In fact it is a known scam to send stock predictions or racing tips to a number of potential marks, then use the credibility you gain with the minority of them to fleece said marks. After all, they don’t know about the other potential victims you sent your (wrong) predictions to, who were not as impressed.

By: jprime271

jprime271 — Thu, 21 Sep 2006 23:55:04 +0000

Great post! And great article! Between this and his GOTV stuff, Alan Gerber is one of the best scientists ever to recently set foot into social science. I hope this piece gets read widely! Negative results are the vitally important to moving scientific debates forward. I’ve had several papers rejected, partly based on the complaint that they reported negative results. How else does one disprove theories if one cannot publish negative results?

By: vivian

vivian — Thu, 21 Sep 2006 02:13:08 +0000

In political science, a worse problem is the paper that fits lines to the outliers, not the paper that ditches them, usually with some discussion. We tend to have lots of problems with not-quite-homogeneous cases aggregated with more-or-less justification, to get a population large enough to generalize about. When they don’t quite fit the hypothesized relationship, it often is that they don’t really belong. And if you’re professionally concerned, you will read the discussion of which cases are omitted, and why, and rant about it in another article, or student lounge, or whatever.

But it’s kind of scary to disagree with Daniel, and so forth. Just thought I’d say. Also, it’s in a journal that has mandatory publication of replication data and code, so anyone who cares, follow the link and play around for yourselves.

By: dipnut

dipnut — Wed, 20 Sep 2006 21:20:15 +0000

What? No tits?

By: rcriii

rcriii — Wed, 20 Sep 2006 21:07:05 +0000

I prefer the term “data-dredging” to “data mining” in this context because there are non-pejorative senses of “data mining”. Also a miner sometimes strikes gold but a dredger only ever stirs up mud.

On the contrary, David, dredgers dig plenty of stuff besides mud. Plus dredgers usually get paid for everything they dig up, while miners often end up with fools-gold.

By: elliottg

elliottg — Wed, 20 Sep 2006 20:33:26 +0000

My favorite is the multiple regression against more than 10 variables that reports 1 is significant.

By: Jacob Christensen » Publication Bias in Political Science?

Jacob Christensen » Publication Bias in Political Science? — Wed, 20 Sep 2006 18:01:20 +0000

[...] Crooked Timber – Kieran Hiely: Attractive Models [...]

By: Paul Orwin

Paul Orwin — Wed, 20 Sep 2006 15:51:17 +0000

I see your point Daniel (and Kieran), and yet, I can’t quite let go of this issue. I think the problem is more generally the opacity of highly specialized research (in any field, but I suppose in this context more in the social sciences) that precludes reviewers from critically analyzing methods. So, you get a paper to review, and it says something like “the data were analyzed using the such-and-such procedures modified to suit the unique nature of our data set”, and that leaves you with the questions Daniel is asking (i.e., what pts did they leave out to get that sig value to 0.05).

So, an alternative hypothesis is that rather than “tweaking” the model (to be charitable), instead people take large data sets, test lots of different models till they find one that gives them p

By: Tim McG

Tim McG — Wed, 20 Sep 2006 14:44:42 +0000

Don’t political scientists look more sceptically at the just barely significant results? Especially if they claim importance? Or is the real dirty secret of all academia that everyone just reads the abstracts?

I’m inclined to be Very Sceptical that mischief like this will compound itself into larger error; rather, someone may attempt to build further on barely-significant results and find nothing.

By: Easily Distracted » Blog Archive » Failure and Knowledge

Easily Distracted » Blog Archive » Failure and Knowledge — Wed, 20 Sep 2006 14:28:09 +0000

[...] It seems to me that an increasing number of people are also rethinking the way that failure is a necessary part of the production of knowledge, and asking where the good failures have disappeared to. Look Kieran Healy discussion of a fascinating study at Crooked Timber (also discussed by Kevin Drum.) [...]

By: JL

JL — Wed, 20 Sep 2006 14:10:45 +0000

Many years ago, when I was in graduate school, one of my professors, an experimental social psychologist, proposed that journals evaluate papers solely on the basis of research design. Researchers would submit all the preliminary stuff including the design but not the results. If the journal accepted the paper, researchers would then provide the data regardless of p values. I’m not sure how this method would work with non-experimental research.

By: Daniel

Daniel — Wed, 20 Sep 2006 14:08:27 +0000

leederick: but the trouble is that you don’t know whether the cheating is done consistently. You don’t even know whether the propensity to cheat is uncorrelated to the conclusions of the paper and in most cases you might suspect that it wasn’t.

By: leederick

leederick — Wed, 20 Sep 2006 14:03:52 +0000

How terrible is it that about 5% of results are bumped from one side of an arbitrary line to the other? Instead of an arbitrary 5% significance level, you’re working with a de facto arbitrary 6%ish significance level. So long as you all cheat consistently, aren’t you in the same position as you would be if you’d put your arbitrary line in a different place?

By: bethany

bethany — Wed, 20 Sep 2006 13:59:49 +0000

daniel, for “yer” you get 5 style points, for confusing Kieran’s word “statistics” with your concern for empirical measurements (the stuff that one would feed into statistical models), you lose 2. I let you keep 3, so you can keep playing because I think your underlying gripe is correct in that usually measurement and statistics go together and when people screw it up they cause harm—as in Kieran’s complain about death rates. I’d just say that in this case, you’re wrong.

By: eweininger

eweininger — Wed, 20 Sep 2006 13:39:32 +0000

Of course, in my effort to clarify, I reversed left and right in the above comment. Good thing I don’t teach today.