If I’d only known…

This may be overly simple, but the key lesson I learned from my (undergrad, economics) major paper was, “Pay very close attention to your data.” Does it mean what you think it means?

All the tricky stuff is trivial if you don’t get the basics right–a representative sample, questions that are understood and answered accurately are the most basic basics.

The very obvious one is draw lots of graphs.

This lesson is almost always learned too late, but it can never be emphasized enough: Always check your equipment before recording an interview!

God, that 2 hour interview I lost because I didn’t check my tape that one time still haunts me. If my reminder prevents just one future cock-up then I will consider my loss to not have been in vain. Obviously this is an addendum to #5 above.

Also, always carry writing materials with you, or at least when you’re in the midst of research. Ideas pop up at some odd times and you’ll be glad you managed to jot down that flash of inspiration you had when you were in the canned goods aisle.

I can’t think of any more advice offhand. Try not to work at home? Definitely a bad idea for certain personality types.

(Don’t forget to plug the book, and maybe point us to your intro online when it’s done!)

Periodically during the project, look into your soul. What are you glossing over, handwaving about, deferring to think about later? Now face the hard fact that by not addressing these now, you’re setting them up to be show-stoppers.

To expand on dsq’s excellent advice, look for patterns in any way you can think of – if not graphs, then look for differences between subgroups, try unlikely combinations just in case they’re interesting. “Marinate yourself in the data” a prof used to say. It’s not just quant advice, it’s not just visualization, it’s about knowing the material, the data, the background.

Leave time for the unexpected. Sometimes it’s a disaster, but sometimes it’s an unexpected insight leading to fruitful but different kinds of analysis/conclusions.

and of course, Until you have the data imported into your analytical software on your machine (or whatever), cleaned and ready to crunch, you don’t actually have the data yet.

There seems to be a “round up the usual variables” attitude in my field, criminology, and I imagine in others as well. So we “know” that delinquency is strongly associated with poverty, low educational achievement, single-parent households, etc., but there is less willingness to look at the micro-level variables, at what causes this person to become delinquent while another person with the same characteristics does well. Different people face different challenges and opportunities in their lives, but these differences are often not picked up in very many studies.

Strength and quality of a research methodology is almost always inversely proportional to the complexity of the statistics involved.

This comes from health, so I don`t know if it`s relevant to “empirical social science”, but it got me once … never believe on face value the claims from collaborating service centres on things like the number of subjects they might see a day, how likely they are to interview subjects as part of their routine service provision, how likely uninvolved staff are to cooperate, and what factors may hinder the data collection process in their service. Particularly if there is grant money or publication credit involved, or the subjects are … complicated … or the topic of the research is … challenging.

(And if one is doing what witt suggests, one will see early on that this is an issue, and can find a way to get around it – but a research assistant may not tell you that recruitment is impossible if they think it will reflect poorly on them).

Aw dang, I thought the elaboration of #4 was going to be “more pies and shorter hours and bollocks to almost everything else”…

Eudoxis:
“This is, undoubtedly, common knowledge, but it is one of those lessons that seem difficult to learn. Consult a statistician during project design, not after data collection, even if you know some statistics.”

As a statistician, I really, really support this. There’s an old saying, ‘post hoc is post-mortem’, or ‘when the statistician is consulted after an experiment is run, frequently the only thing that he can do is to confirm the cause of failure’.

In addition, good data management will save a lot of time, and might make the difference between success and failure. Having an idea of what variables are to be collected, how data is to be linked between/within subjects, times, locations, facilities, interviewers, etc., might save a person-year or so of re-do work. .

Last: if you’re scaling up (e.g., from a single research/facility to multiple researchers/facilities), don’t assume that you can do things in the manner which worked for smaller, simpler studies. You wouldn’t approach building a large house in the same manner as building a garden shed; it works similarly in research.

Another in the “obvious, but often missed” category: if your study could be applicable to a political controversy, be aware of the critiques made of studies by partisans on both sides. Yes, some of them are ridiculous nit-picks–but often some very good, but non-obvious, points are well-known to partisanns and ignored elsewhere. (For example, the wedge between “wages” and “compensation” has grown very fast in the last 20 years.)

Another one I learned the hard way: Before you start the project, practice writing your final report. Seriously. I have found more gaps in my data collection by realizing that I wanted to be able to write a sentence that said “The average participant got a loan of $3000, although the loan program is nationwide and the impact of that $3000 varied greatly….” ooops. Maybe I ought to think about cost-of-living differences around the country and whether those need to be factored in.

I suppose both of these are trivial too, but they’re probably the most important lessons I’ve learned over the years.

1.) When designing an experiment, keep in mind that it’s probably not going to be the only one you need to run. Make sure you leave yourself directions to go next.

2.) Related to 1), when designing a study, try to imagine the criticisms you’ll get. I usually think to myself, “What would reviewers say is missing, or needs to be controlled, what follow-up experiments would they want to see, etc.” Thinking about the flaws that other people will see is a good way of taking a step back and looking at a study with fresh eyes.

The most important thing is to try to arrange your study so you get a result which other scholars will find interesting no matter how the data or evidence comes out.

That can be tricky in some circumstances. If you are doing routine tests of drugs and you find your drug cures cancer, that’s a real result – but if you find your drug doesn’t cure cancer, people are going to be less interested. The outcome you get, and whether you end up on the front page of Nature or page 4568 of Annals of the Properties of Obscure Compounds: Series H, basically depends upon luck.

Social scientists have an easier time of it because the theories they look at are usually more complicated than whether something has a treatment effect or not. If you think hard enough about the problem, you can often arrange things so your data will say something new or interesting about some aspect of the theory you’re trying to address however it comes out. And you get to say something worthwhile to say to your peers either way.

So my advice would be to be very wary of standard research design from the experimental sciences and what the statisticians you consult will say. Standard NP-hypothesis testing and study design will set you up for a situation where you are basically playing Russian Roulette for an interesting result. You’re making a gamble, and if you lose and the results say accept the null or don’t have a large enough effect size you’re not saying anything interesting.

The other thing I would say is to be prepared to be methodologically eclectic when if comes to statistics. Statisticians can’t agree on a theory of inference amongst themselves, so there’s no reason you should feel obliged to stick to one. By all means set up a hypothesis test before you collect the data, but be prepared to throw that out the window and focus your study around exploratory data analysis, or post trial statistics, or Bayesian analysis if this will give you a more interesting result to publish.

Robert Chambers, in his book “Rural Development: Putting the Last First” (Longman, 1983), has a nice, intemperate discussion of the practical issues which impact the collection and analysis of social science research data, and which are usually ignored in textbooks. His focus is on research in developing countries but the lessons apply more generally, particularly since the (sub-)culture of the researcher is almost never that of the research subjects.

And the (second or third) language skills you think are sufficient seldom are.