Data mining

Basically your intuition is right; data mining in marketing is what you do when you’ve got a mountain of data that you’ve collected because it “seemed like a sensible thing to do” and want to know what to do with it. Like Amazon.com’s sales data. Also note that marketing people typically use nonlinear fitting techniques rather than stepwise regression these days, because they’re not as hung up as economists on t-ratios greater than 1.96 (they are, correctly, more concerned with practical significance than statistical significance).

I like the phrase “data dredging” to describe the pejorative sort, because a miner occasionally strikes gold but a dredger just stirs up sludge.

There are respectable data mining competitions where the competitors are given half a huge data set and told to develop a model on it. Then the models are tested on the other half.

Also, as Daniel said, nonparametric methods are now very well established, and increasingly common in the social sciences… I think the data-mining side of things complements the development of model validation methods of various sorts (like jackknifing, extreme-bounds analysis, resampling methods and so on) that help you check the value of your model. They’re both consequences of the computational revolution in statistical modeling that’s taken place in the past 20-odd years.

Frank Harrell, linked to in Chris Lightfoot’s comment above, is the author of the excellent Regression Modeling Strategies, which is one of the books I try to live up to when doing my own data analysis.

working link

Barry: There is nothing wrong with multiple testing as long as you’re very careful about interpreting the results (after all, it would otherwise be utterly impossible to ahve a valid model of GDP since so much work has already been done on the time series). Have a look at Doornik’s responses to criticisms of PcGets.

I think most sociologists look at data mining in a negative light as well. But it may depend on your training and how you feel about use of quantitative data. I was taught – and prefer this approach – that you should usually theorize even what are often simply referred to as “the usual control variables” so I prefer to have conceptual reasons for including variables in an analysis and that’s what I’m teaching my students as well.

Somewhat related is a point that came up in class yesterday when we were discussing a piece by an economist. One of the students noted that the author had not controlled for the usual variables when looking at how computer and Internet use at work may be influencing hours worked. I noted to the student that the author had, in fact, controlled for many many factors but did not include these in the table. Rather, they are listed in the appendix (not in a table with coefficients, just as a list of controls included in the analyses). I don’t ever recall seeing that done by a sociologist. (Granted, this is a working paper, I don’t know how often this is done in econ in a final publication.) In case anyone’s curious, the piece we were discussing was Richard Freeman’s NBER working paper on “The Labour Market in the New Information Economy“.

Epidemiologists and biostatisticians (e.g., as Kieran notes, Frank Harrell) worry about data mining in the perjorative sense for the same reasons that economists do. However, we are also trying to see what can be learned from the marketing techniques (data mining in the positive sense): we want to find problems in clinical practice that we aren’t currently aware of by spotting data anomalies (artificial example: “how come there so few imaging charges associated with appendectomies in the last six months? can they possibly be opening folks up without looking first???”).

Another thing you failed to mention is that pretest estimators are equal to neither the OLS or RLS (Restricted Least Squares) estimators. This makes is hard to evaluate pretest estimators.

The sampling distributions are stochastic mixture of the unrestricted estimator and the restricted estimator. Pretest estimators are also discontinuous functions of the data and a small change in the data can bring a switch in the results.

Also many economists use pretest estimators.

Economists reject data mining based on methodology. The purpose of econometric analysis is hypothesis testing. Those hypotheses must come from theory. Without theory, there is no hypothesis to test. I suppose you could use mining to find hypotheses to test, but you would then have to “start over”, developing a theory which calves off that particular hypothesis, and then testing that hypothesis on yet another data set – the first is contaminated. (The regression statistics wouldn’t mean what you would be claiming they meant, if you used the first data set.)

There isn’t much concern on either side of the divide because the marketeers view the economists’ concerns as foolish, and economists view most business research as garbage. (Hence the low esteem generally given to economics departments in business, rather than academic, colleges.)

There was once a Dilbert cartoon about data mining.

Perhaps a Bayesian view would help.

In Bayesian analysis, you always test one hypothesis against others (real ones, not imaginary null and alternative ones :-). For example, if you were fitting a line to data on a 2D graph, you might start with the idea that all slopes and all intercepts were equally likely before you saw the data, then do calculations to see how observing the data changed those probabilities. So, after seeing the data, you would know how likely each possible line was.

For example, take the following model. This model says that there are ten possible slopes (m=1,2,3, … 10) and the intercept of the line equals zero. So there are ten possible lines we are considering.

Each possible line starts with 10% probability (called the prior probabilities, since they are the probabilities you have prior to getting the data). Then, when we see the data, that probability changes (it is multiplied by a factor that depends on how close the data fit each possible line).

After seeing the data, we would then have a new set of 10 probabilities, one for each line, describing how likely each line is now.

Now, let’s do some data mining; let’s look at another model, using the same data. This model, instead of assuming the intercept equals zero, gives it ten possible values (0,1,2,…,10). Now, instead of ten possible lines, we have 100.

This has two effects:
1. It (most likely) makes the best line fit better; after all, instead of just 10 lines to choose from, you have 100.
2. It lowers the prior probabilities; now each line starts with 1% probability rather than 10%.

So, the second model, with two parameters, may give worse results than the first, with only one. For example, if you could only fit the data, say, two times better (in some sense), then it doesn’t outweigh the tenfold loss of prior probability.

So, in comparing the two models, the first one would be better; even though the “best fit” line of the second model is better than the first, the second model ends up with a lower probability. The calculations effectively took Occam’s razor to it, and cut it down to size.

So, Bayesian analysis automatically makes the trade-off between fitting the data and introducing new parameters into a model.

My understanding of stepwise regression is that this doesn’t happen; am I correct? In other words, every time you put a new variable in, the p-value gets better (take one out, it gets worse).

Does this solve the problem of data mining you are having? I’m not sure it does.

Let’s say you test 1000 models on the same set of data. In classical terms, even if there is no “connection” whatever, about 5% will be statistically significant at the 5% level (hey, 50 publishable papers, I’m on my way to tenure :-). This is the original problem with data mining, correct?

What happens in Bayesian terms? You end up with a probability for each model being the correct one.

This assumes that one and only one is correct. The “only one” part is fine, I’m not so sure about the “one” part. In other words, you are assuming that, among the possible hypotheses you have, one is the correct one. Any problems with that? Perhaps not; maybe the outcome of a statistical analysis should be something like “Of the hypotheses we considered, this one is (say) 20 times more likely than all the others combined.”

Then, you assign appropriate prior probabilities to each model. This addresses the “sure-thing hypothesis” problem with data mining. Once you see the data, you could come up with a model that predicts “That data is the only possible thing that could have happened.” This is clearly the winner in terms of fitting the data, but the prior probability of such a hypothesis is too small to consider.

Then, for each model, the analysis above will take into account (a) how well the model fits the data and (b) how many parameters it took to do so. This isn’t a vague, subjective trade-off; it is built into the calculations themselves, as my example above suggested.

So, what are the difficulties with this Bayesian data mining? Real question, BTW; I’m no expert. One problem is that most statistics courses are too “cookbook” and like to use techniques that don’t require much actual understanding of what is going on. For similar reasons, such a method won’t be included in a simple statistical package. Another problem might be calculational; if the above analysis required 1000 nested integrals, then we would have to turn to other methods (MCMC, perhaps). Others?