This one was one that my group dodged during one class, but every other group in the course was burned by it. During an IC design class, we completed the design work early in the semester (the other 2 wanted to take time off to visit family in the middle of the semester, so we had to start early). The software license for the design software expired about halfway through the semester, so that we were the only group that managed to complete the design work that semester. Because the other groups waited until about the last month of the semester to start, the expired license wasn’t rectified until after the semester was over (most serious CAE software expires annually). I’m sure the EE dept was ticked off that most of the course (all but 3) got an Incomplete that semester.

Like Sarapen, I also carry a notebook of some kind all the time. And like Daniel, I’ve had some embarassing incidents horribly underestimating how long it takes to perform some tasks.

That can be tricky in some circumstances. If you are doing routine tests of drugs and you find your drug cures cancer, that’s a real result – but if you find your drug doesn’t cure cancer, people are going to be less interested. The outcome you get, and whether you end up on the front page of Nature or page 4568 of Annals of the Properties of Obscure Compounds: Series H, basically depends upon luck.

Social scientists have an easier time of it because the theories they look at are usually more complicated than whether something has a treatment effect or not. If you think hard enough about the problem, you can often arrange things so your data will say something new or interesting about some aspect of the theory you’re trying to address however it comes out. And you get to say something worthwhile to say to your peers either way.

So my advice would be to be very wary of standard research design from the experimental sciences and what the statisticians you consult will say. Standard NP-hypothesis testing and study design will set you up for a situation where you are basically playing Russian Roulette for an interesting result. You’re making a gamble, and if you lose and the results say accept the null or don’t have a large enough effect size you’re not saying anything interesting.

The other thing I would say is to be prepared to be methodologically eclectic when if comes to statistics. Statisticians can’t agree on a theory of inference amongst themselves, so there’s no reason you should feel obliged to stick to one. By all means set up a hypothesis test before you collect the data, but be prepared to throw that out the window and focus your study around exploratory data analysis, or post trial statistics, or Bayesian analysis if this will give you a more interesting result to publish.

1.) When designing an experiment, keep in mind that it’s probably not going to be the only one you need to run. Make sure you leave yourself directions to go next.

2.) Related to 1), when designing a study, try to imagine the criticisms you’ll get. I usually think to myself, “What would reviewers say is missing, or needs to be controlled, what follow-up experiments would they want to see, etc.” Thinking about the flaws that other people will see is a good way of taking a step back and looking at a study with fresh eyes.

A regression analysis usually consists of several stages such as variable selection, transformation and residual diagnosis. Inference is often made from the selected model without regard to the model selection methods that preceeded it. This can result in overoptimistic and biased inferences.

The technical content is about regression analysis, but there are good general lessons as well.

1. When entering data from surveys one has administered on one’s own, don’t worry excessively about how to code responses to questions. Settle on a consistent coding scheme that captures the relevant variation in the data, and THEN when you are ready to analyze the data, let your software do the work of creatingn appropriate variables (e.g. dummy variables) by using logical statements to create new variables.

2. Don’t “hardwire” your data when you enter it. Subject to memory capacity and similar constraints, keep the data in as flexible a form as possible, because as one begins to do the analysis, one may often find a need to create new variables, and this is easiest to do when data are not entered into a pre-existing format. This can save time back-tracking later one.

As a statistician, I really, really support this. There’s an old saying, ‘post hoc is post-mortem’, or ‘when the statistician is consulted after an experiment is run, frequently the only thing that he can do is to confirm the cause of failure’.

In addition, good data management will save a lot of time, and might make the difference between success and failure. Having an idea of what variables are to be collected, how data is to be linked between/within subjects, times, locations, facilities, interviewers, etc., might save a person-year or so of re-do work. .

Last: if you’re scaling up (e.g., from a single research/facility to multiple researchers/facilities), don’t assume that you can do things in the manner which worked for smaller, simpler studies. You wouldn’t approach building a large house in the same manner as building a garden shed; it works similarly in research.

Yes. Look at your data.