Why are all required statistics courses essentially the same? They start off with bland assurances from the instructor that no knowledge of maths is required and that the concepts involved are pretty easy to grasp – all you need to do is turn up in class and do lots of practice questions. Oh, and have a positive attitude. Yeah, right.
I’m about to take the third stats exam of my life. As with the two before, failure is a barrier to continuing my ‘real’ studies. And, though this is my third tour of duty through histograms to simple regression, failure is a distinct possibility. The null hypothesis, that Maria has sufficient knowledge, nerve and luck to once again pass stats by the skin of her teeth, looks like being rejected. Of course I don’t blame myself, not entirely. I’d rather blame the teachers, or perhaps the subject itself.
The first compulsory (is there any other kind?) stats course I ever took was as an undergraduate. The lecturer – there were no tutorials, and no textbook that I can recall – clearly hated the job. I don’t blame him. Unlike teachers in any other discipline I’ve studied, statisticians genuinely seem to have been dragged blinking and unwilling to the lectern and struggle to communicate their wisdom to the herd. (Though I’ve never studied third level maths or hard sciences so maybe this quality is typical of all quant jocks.)
When the exam rolled around at the end of the year, I couldn’t understand a word of it and barely attempted any questions. Which didn’t surprise me, but my classmates who actually had a clue were completely stumped too. There were tears all round and everyone resigned themselves to autumn repeats. But when the results came out, everyone had passed, even me and the other no-hopers. What had happened?
The rumour was that the lecturer had resigned / been denied tenure, and to annoy the department he had set the first years an exam none of them could pass. Which would have meant enrolment for second year was precisely nil. By the time it was discovered, we’d all left for our J1 summers in Long Island and Martha’s Vineyard, and the only thing for the department to do was pass us all. Which was a boon to me as I’d never have gotten through otherwise. So that time it was pure luck that got me through.
The next time was at LSE. This time, the lecturer couldn’t have been nicer and more apologetic about the fact that we had to pass the class to get a degree. Don’t worry, he said gently, it will all be geared to teaching you how to understand and interpret the statistics in social science literature. Which was only fair and probably would have been quite useful. But there was still any amount of chi-square, one tails, two tails, and all the rest of it which went completely over my head. By now, computerised statistical analysis was all the rage, and we had ‘lab’ sessions where we were supposed to apply the theoretical insights gained in our weekly lectures. And interchangeable and grindingly bored doctoral students trying to teach us how to use SPSS and answer the questions you can’t ask in a lecture theatre of 500 people, i.e. all of them.
The questions statisticians dread; ‘but WHY did you do that?’ and ‘what does it MEAN?’. Why is it, that when you ask a statistician one of these questions they look at you as if you’ve addressed them in ancient Greek? When it’s painfully obvious that they’re the one speaking nonsense in an obscure language…? And when they do respond, it’s as if they’re a demented computer stuck in a sub-routine. They simply repeat the steps of the procedure several times more, never saying why they did it or what the outcome signifies.
I think understanding statistics must be like visiting a faraway land that only the chosen few may enter. Like, for example, North Korea. Or maybe it’s like being stolen away by the fairies. When they come back, well, they don’t ever really come back, do they? Their eyes are glassy and their responses are just a bit off. Or perhaps it’s like a faustian pact where you gain tremendous secret knowledge but lose the ability to explain or share it with anyone. Except other statisticians, of course. Perhaps they intermarry?
In any case, my second bout in the ring was decided by a take-home exam which we all thought was very American, complete with an all-nighter of pizza and Coca-Cola in the residence computer room. It was pass-fail too, though you could apply to the department for your actual mark. I think it was sheer nerve that got me through this time, and simple sugars. When I passed, I decided that was enough, and it would be pushing my luck to see if my mark was more than the bare minimum.
And now here I am facing stats exam number three, which I’ll have finished in exactly 24 hours. And, if the conceit of this blog entry is to hold, I need to get through it based on my acquired knowledge of the discipline.
All the usual tics and blind spots of the statistics course were present:
· The emphasis on teaching us how to use statistics to make informed decisions, not train us to be practitioners of some dark art.
· The promise in week 1 that basic arithmetic was all that was needed, giving way to a dismissive aside in week 4 that anyone who couldn’t graph a quadratic equation should do an emergency maths course in the 5 days remaining before the exam. (OK, I can do that and so can everyone else in the class. But still.)
· The claim that statistics’ biggest enemies are ‘blockheads, fools, morons, idiots, prigs and authoritarian personalities’. Well, that’s me done then.
· And above all the assurance that it would be over quickly and hardly hurt at all.
I’ll admit I’m easily in the bottom 25% of the class, and was never going to be a natural with this stuff. And I’m sure it does the ego no harm to feel like a remedial case from time to time. But this time I did the exercises and had a positive attitude and really thought I’d crack it.
So, I have to ask. do stats courses for non-practitioners really have to be so painful and so obscure? They seem just as unpleasant for us students as they are for the teachers…
Three courses, three exams, and I know that if I manage to pass this one, I’ll never open that book again.
{ 58 comments }
J. Ellenberg 10.13.04 at 10:50 pm
For what it’s worth, I think teaching statistics sounds fun.
Anand Hattiangadi 10.13.04 at 10:52 pm
All the problems you describe – the bored teachers, the promise to keep it simple (basic math only, or simple calc only, or something like that) the inability to describe the meaning of what your supposed to do rather than the process – are related to the class being “not for practitioners”. I think this is especially true for quant classes, but that might just be because humanities classes for non-practitioners are just very rare outside of freshman distribution requirements in the US. My recommendation – take the classes for practitioners. They might be harder, but at least the lecturer will be animated and interested, and the meaning of what you’re learning will be the focus of the class.
dk.dk 10.13.04 at 10:54 pm
Hah! Oh Maria that post made me laugh in ways usually only Kieran can – like being stolen away by fairies indeed.
I sometimes feel that my relationship with statistics is like Sarah Jessica Parker characters relationship with Mr. Big – all the promise of something great in the beginning, then the realisation that I’m just being abused for some obscure introverted cause some visits later.
My latest nihilistic episode has been with categorical regression models – and it bears all the hallmarks you mention: the first 4 lessons spent ‘just casually revisiting’ page after page after page of proof – leaving some of us really questioning the infamous slogan/mantra ‘minimal technical content’ …
Lisa 10.13.04 at 10:55 pm
The average intro stats course (in my experience) is not particularly hard compared to most college level math courses. It’s true that if you do lots of practice questions, and actually try to understand the material (as opposed to blindly plugging numbers into formulas) then you should have no problem passing the exam. I’m kind of curious as to how many hours a week you actually spent on this class (outside of lecture).
Personally, I think anyone who wants to seriously study social science, or ever plans to use statistics in an argument, or ever forms opinions based on other peoples use of statistics (20% of children blah blah blah) absolutely MUST have some detailed knowledge of stats.
dsquared 10.13.04 at 10:56 pm
For what it’s worth, the only way to learn statistics in a reasonably painless way is to learn a decent amount of linear algebra first. That’s how I finally managed it anyway.
Now if someone could come up with a painless way of learning linear algebra, we’d be away.
pgl 10.13.04 at 11:17 pm
As a Ph.D. in economics, I had to learn statistics at some point and did OK as I’m a math geek. But as I started practicing and teaching economics, I realized that much of the boring rigor in stats classes was really unnecessary, while much of the practical insights got lost in the boring rigor. As in all things, it depends on having a teacher who knows how to related. Fortunately for me – my grad econometrics teacher was wonderful at this. Hope you find an instructor who can relate.
nick 10.13.04 at 11:19 pm
I did ‘pure maths and statistics’ for my A-levels, thinking that the alternative of ‘pure maths and mechanics’ was too, um, ‘mechanical’ for someone whose other subject choices were firmly stuck in the humanities.
It’s something I regret, in spite of the fact that I’ve forgotten most of my A-level maths. I loved the pure maths elements of the course, especially calculus, and mechanics would have meant more of that, rather than the squishiness of confidence intervals…
(That said, if I’d gone down the history route, not the Eng Lit one, I’d probably have benefitted from it.)
I’ve just seen the amount of stats that my wife has to learn (or re-learn) for her psychology licensing exams. Ugh.
Cosma 10.13.04 at 11:22 pm
As the non-statistician husband of a statistician, let me suggest, in all seriousness, Larry Gonick and Wollcott Smith’s Cartoon Guide to Statistics.
Michael 10.13.04 at 11:23 pm
This is all very familiar.
I remember my first statistics exam. About three people passed first time around, out of 29. I admitted defeat rather quickly and ran off to listen to a Chopin piano recital in the auditorium.
Our teacher had a master in Mathematics and assured us that statistics was quite easy and logical. Of course, he then started filling the blackboard with gibberish, occasionally commenting on how easy and logical everything was. Easy for someone whose idea of fun is solving a few differential equations.
I think it is very often forgotten that, unless a student is dealing with equations on a daily basis, statistics is (too) difficult to grasp. I once asked why the formula for a certain correlation coefficient had the number 6 in it. The teacher replied that it would take an hour to explain and be beyond my comprehension.
The course consisted of a practical where you had to design a questionaire and feed the results into SPSS and an exam. The practical was useful. The emphasis was on drawing conclusions from the collected data. The exam was a total disaster and totally unnecessary. There is really no compelling reason to be able to write down a complete chi-square calculation and the average student isn’t going to understand why the formula for chi-square is the way it is anyway.
I believe the exam has recently been removed from the course requirements.
Zaoem 10.13.04 at 11:25 pm
I have taught statistics to practioners. It starts out sort of like convincing your child to eat broccoli or brussels sprouts for 3 hours a week. The problem is convincing students that it is actually good for them. Telling them doesn’t help. I try to create an assignment early on where students have to interpret simple tables or graphs from a relevant report, such as the World Development Report. More often than not, students find out that this is hard but that they would have a head start if they could do it well. In general, I never stray too far from very practical applications. Some students will of course never take to it. But in my experience a good number are actually quite pleased to actually develop some skills.
dsquared 10.13.04 at 11:30 pm
If you end up retaking the bugger, btw, the best textbook to use is the relevant chapter of “Quantitative Methods in Finance” by Parramore and Watsham. It is just about the only zero-bullshit, no-proofs, how-to-get-it-done textbook out there.
Cranky Observer 10.13.04 at 11:33 pm
Try taking either Engineering Statistics or Engineering Probability in your University’s Engineering School rather than Arts & Sciences. I have found those courses are generally better structured and better taught that “Stats for Poli Sci Majors Who Don’t Give a Damn”.
Cranky
me2i81 10.13.04 at 11:33 pm
Now if someone could come up with a painless way of learning linear algebra, we’d be away.
Linear Algebra will never be painless, but it helps to have an instructor who actually understands the material. At Prof. Gilbert Strang’s linear algebra coursepage on MIT Open Courseware, there are video lectures, and they’re good for anyone who wants to understand linear algebra at an elementary level.
Donald A. Coffin 10.13.04 at 11:36 pm
I’m an economist and I occasionally have to deal with statistical concepts, working with people who have completed required stats classes. I don’t, myself, teach stats classes. What I tell them is that there are only two concepts in stats–variation and distribution–and only one (type of) question.
Suppose EVERY observation about something (e.g., response to a dose of a drug) were exactly the same. How many people would we have to give the drug to?
But if not every response is the same–if there’s variation, what’s the DISTRIBuTION look like?
And, finally, all statistics comes down to helping answer one (type of) question–is this number DIFFERENT FROM that number by enough that we care? (Is the response of people in the clinical trial who actually receive the drug different enough from the people who receive the placebo that we can reasonably think the drug has an effect?)
skippy 10.13.04 at 11:52 pm
What Donald Coffin said.
G. Svenson 10.14.04 at 12:08 am
The Staticians… I saw them, on the Plains of Leng… And I shall tell you, their “standard deviations” were neither standard, nor deviations. They were something worse altogether, and I thank The Worm that I never truly understood what… Beware of the Polygon with a thousand young!!!! (especially if you are questioned about its annual increase in feritily rate).
Bleh, it actually sounds as if I got off easy with my stats course…
Kieran Healy 10.14.04 at 12:53 am
For what it’s worth, I think teaching statistics sounds fun.
Jordan, that is because you are a professor of mathematics at Princeton University.
As for Maria’s post, I sympathise with everything in it. I’ve found myself in a similar situation many times, with the difference that I really _want_ to learn this stuff, because I think it’s important and also pretty cool. But thanks to persistent math anxiety and a lack of practice between the ages of 15 and 25, I have to approach the topic obliquely. At its best, this technique involves learning the theory via writing code to do actual data analyses. At its worst, it amounts to buying Springer and Cambridge statistics texts and leaving them around the place in the hope that their content will seep into my brain by osmosis.
Mike Kozlowski 10.14.04 at 1:25 am
They emphasize all that stuff about not needing to know much math and the concepts being reasonably graspable because 1) they know that non-technical people are scared of math and will need to be reassured to get over their mental hurdles, and 2) because compared to other math courses — say, “Techniques in Ordinary Differential Equations” or “Elementary Topology” — it’s true.
(Disclaimer: I haven’t actually taken either of those courses; my math career ended after Numerical Fucking Analysis; but based on the courses I did take, I assume they’re completely incomprehensible to mortals.)
fiat lux 10.14.04 at 1:26 am
I’m taking a stats course at UC Berkeley this fall as a prerequisite for graduate school and yes, it is tough. My teacher does seem to care about his subject and has obviously put a lot of thought into how he presents the material, but he is really poor at one on one interaction with students. So although he works very hard to give careful, step by step explanations of things, when he makes a leap and people ask him about it, he has a hard time responding. He’ll give you this look like you’re from another planet, and can seem at a loss how to explain what must be blindingly obvious to him.
It is frustrating, to be sure. I did OK on the first exam, a take home, but am very concerned about the next 2, which are in class.
Marc Valdez 10.14.04 at 1:32 am
Statistics is too wordy for math people and too mathematical for word people. There is too much of a backstory to intelligently convey in a typical course, so what happens is the story gets rushed in a barrage of confusing, mostly unnecessary information: who the hell is Wilcoxon, who the hell is Student, etc. Statisticians need to pare down their material!
In other words, it’s not your fault!
roublen vesseau 10.14.04 at 1:36 am
I have a pretty good intuition for statistics, and when I tried to teach statistics to people I tried to figure out where it came from. Here’s my best guess:
statistics books:
Statistics a textbook by Freedman, Pisani, Purves. The best statistics textbook I have seen, except 1)it is bloody expensive 2)Even though there are a lot of problems, with answers, people complain there is too much handwavy discussion, with not enough concise summaries, formula lists, and step by step, “cookbook” instructions
Statistics Without Tears, by Derek Rowntree A very good, informal book, but people complain it only deals with high-level concepts, and not enough detail and formulas. Rowntree wrote another book, “Probability Without Tears”, which I have not read.
How to Lie With Statistics by Darrell Huff, not a textbook, again, informal concepts, high level discussion
Non-Statistics books:
Books by Andrew Tobias:
“The only investment guide you’ll ever need”, “My vast fortune”, “The invisible bankers”. Not directly related to statistics, but a lot of skill in using numbers with common sense, and they were important for me developing how to think about these things.
Books by Paul Krugman:
“Peddling Prosperity”, “The return of depression economics”, “The accidental theorist”, “Pop internationalism”, et al. Same deal.
A Book by Robert Cialdini: “Influence: The Psychology of Persuasion”. This one was very important, I think for learning about psych experiments, which obviously involve some statistics . I read it in high school, and never learned so much from one book.
With the exception of the first “Statistics” textbook, none of these books get deep into formulas and explaining in a step by step way how to solve problems. I don’t really know of any such books, but there are a few chapters on statistics in Harold Jacob’s great math textbook “Mathematics: A Human Endeavor” and in Paul Foerster’s book “Algebra and Trigonometry”, both written for high school students.
Obviously all this is a bit late for this particular course, but Cialdini, in particular, is worth reading anyway. Good luck!
roublen vesseau 10.14.04 at 1:43 am
I should also mention that the Tobias books are funny, and the Huff book tries to be.
g 10.14.04 at 2:11 am
When I did my mathematics degree, there were no courses in Numerical Fucking Analysis or indeed in any other aspect of the analysis of copulation. I suspect attendance at lectures might have been better if there had been.
One reason why statistics is difficult, and why explanations of statistics tend to be unsatisfying, is that the intellectual foundations of statistics, as conventionally practised, are a bit of a mess. (The alternative approach known as “Bayesian statistics” is more coherent, but intimidating in different ways.) When learning statistics, it’s helpful to be skilled at the suspension of disbelief.
Chris Genovese 10.14.04 at 2:43 am
Maria, the problem you describe is sadly familiar, but it’s also changing, if slowly. Over the past decade here in the US, there’s been a quiet revolution in introductory statistics education that addresses the issues you raise and others.
In my (Statistics) department, our introductory course is a core requirement for all humanities and social science majors. A lot of work has gone into making it an effective and approachable course. “Lab” work doing data analysis on real data sets is central; guided practice and group work are designed into the curriculum; and there’s a big emphasis on good communication and writing. This is a non-mathematical course in the way you were promised, but we also have a sequence of “bridge” courses that lead humanities students toward the more mathematical material in downstream courses, which is nontrivial. This approach carries into other parts of our curriculum as well; most of our undergraduate courses have a hands-on lab component. Since we’ve been doing this, the number of humanities students who have signed on to be Statistics majors has risen sharply.
This trend is not limited to my university. My wife is a cognitive psychologist who studies learning and expertise, and she’s heavily involved in similar efforts at other institutions. Eventually, the old approach will become the exception.
Besides a recognition that the old approach wasn’t working, I think the main prompting for the new ideas was a change in the culture of Statistics. Although there were great applied statisticians, Statistics was viewed by most as a sub-discipline of mathematics, and it’s my impression that many academic statisticians of decades past had little experience with actual data. As more powerful computing has become available, data analytic methods have extended beyond the simplest theoretical framework and asymptotic justifications. Over time, applied work has become more valued, for both career and funding, and graduate training has come to emphasize data analysis much more than it once did. It’s still true that mathematical research is critical, but most Statistician’s now have more experience with real data, applied problems, and collaboration beyond simple consulting. Statistics is no longer viewed (by Statisticians anyway) as a sub-discipline of mathematics but rather as a mathematical discipline with a distinct blend of theory, praxis, and art (not dark).
This is starting to “trickle down” to education, undergraduate and graduate. Students can study interesting data sets for class work and get good experience at statistical practice, and introductory statistics no longer needs to be taught as a Mathematical Statistics lite.
None of this is happening quickly, and my sense is that these changes are happening more slowly in Europe. So, I’m afraid I can offer little solace. Demented computers can last for a long time.
A few reasonably good introductory books are Introduction to the Practice of Statistics by Moore and McCabe, Intro Stats by Deveaux and Velleman, and Statistics by Freedman, Pisani, and Purves. A highly non-mathematical book that I’ve heard good things about (but haven’t read) is Seeing Through Statistics by Utts. I’d also second Kristina’s recommendation of the Cartoon Guide. At a more mathematical level, I’d strongly recommend All of Statistics by my colleague Larry Wasserman.
Oh, and a plea from the practitioners of the dark arts: can we give Disraeli a rest…
Good luck!
one 10.14.04 at 3:39 am
Get the Cliff’s Notes. It’s five dollars or so and has the whole course in about 100 crystal clear pages.
John Quiggin 10.14.04 at 5:23 am
‘g’ has it right. The whole apparatus of significance tests, confidence intervals and so on is a load of mumbo-jumbo.
And the Bayesian approach (the correct one in my view) can’t be comprehended without a fairly sophisticated understanding of concepts like conditional probability.
Jim Harrison 10.14.04 at 6:02 am
I used to edit college-level math and stat books and always asked about which courses were the hardest for undergrads. Intro stat always won the difficulty competition—calculus was a distant second.
I think the hard thing about mastering statistics is not the mathematical part—you can grub through that. The statisticians I knew actually expected their students to understand statistical concepts. That was the rub. It doesn’t help that most people find the fundamental notions of statistics down right perverse while those who “get it” can’t remember the time before their conversion experience when it all seemed as screwy to them as it does to the average undergrad.
roublen vesseau 10.14.04 at 6:36 am
I also want to mention a book by Douglas Hofstadter, Metamagical Themas which is a collection of great essays, some of which deal very intuitively with probability & game theory, and how they relate to real world issues. There is also a great parody essay of William Safire, on sexist language.
clew 10.14.04 at 7:12 am
I profited in intro stats by thinking to myself, in spare moments, So say I want to fudge the data, what would be the most subtle and effective way? It adds that touch of human interest, of narrative.
Unfortunately, you do have to be quick enough at algebra to see what changes to the data would cause. (For that matter, you have to be quick enough to have spare moments.)
Randolph Fritz 10.14.04 at 8:03 am
“do stats courses for non-practitioners really have to be so painful and so obscure?”
[It’s late & this is disjointed. Sorry.]
Maria, I think you’ve hit on a serious pedagogical problem. I wonder if the obscurity is partly a result of a lack of the *why* of statistics. Does it help if I write that statistics are how we sort out “true, maybe, false, we don’t know” from data about large numbers of people? Statistics are thus necessary for evaluating hypotheses about society.
I wonder if visualization techniques might not be of value? It is possible to do quite effective structural engineering with surprsingly little mathematics (Allen & Zalewski, *Shaping Structures: Statics*). When I have a bit more time, I want to take a look at Mandelbrot’s *Misbehavior of Markets*, which aims to explain Mandelbrot’s difficult pricing models entirely with graphical methods.
Tracy 10.14.04 at 8:59 am
I learnt during my engineering degree that mathematicians and statisticians are the greatest liars on the planet. They leave marketing in the dust. Satan, the Father of Lies, should sit at their feet to learn.
Either that or they don’t speak English, but some weird language which sounds like our language but there’s a negative correlation between meanings. E.g. “it can be easily shown …” means “after 4 hours hard work, assuming you don’t make any mistakes” and “it is obvious” means “it’s obvious if you spent the last 10 years working in the field, if you picked this book because it has ‘introduction to’ in the title you’re basically stuffed.”
So now you should be able to guess what “you won’t need any maths for this course” means. This won’t help you in passing the course, but remember it in the future.
Alex Fradera 10.14.04 at 12:21 pm
I guess I fall in the middle of the range of ability given here (well, the median – the proficiency of professors of mathmatics undoubtedly skew the distribution ;) ) in that I did a little maths at school and enjoyed it, whilst not being a genuine maths kid. As a consequence while an undergrad I struggled with Stats at times but when it clicked found it very satisfying; the deepest and most durable click was when I did an advanced stats course in my final year and we got taught it from Matrix algebra on up – when the whole endeavour made sense and ceased being esoteric and fetishistic, and became pragmatic and commonsensical. (I’ve since forgotten enough Matrix algebra to make this stage a shimmering ideal…)
From my perspective, you really do need a good teacher, lots of examples, and to work through simple, intuitive problems to get a grip on stats. My recommendation for a handbook comes from Chris McManus, doublewinner of both the Aventis science writing prize AND the ignobel prize (in 2 consecutive years, IIRC), so an obviously reliable source!
Statistics as Principled Argument, by Robert P Abelson, is very maths light, uses tons of fun examples and gets to the heart of what stats is about. It’s drily humourous and was bus/bedtime reading – a good candidate for making you like statistics, if not so much for outright passing exams.
jam 10.14.04 at 1:44 pm
I trained as a mathematician, sufficiently pure as to not have taken a stat course. So all the stat I know has been encountered in some practical context: the Poisson distribution in the context of computer networks, for example.
As a result, I have the notion, probably completely wrong, that statistics consists of a bunch of tricks to model reality together with a set of heuristics to decide which trick to use.
The advantage of this view is that you never bother about such questions as Where does the 6 come from? It’s simply part of the trick. Nor do you ask What does it MEAN? It’s a quasi-empirical fact that this sort of situation tends to behave in this sort of way and this sort of math will model it. Accept that and use it when you encounter this sort of situation.
Scott Spiegelberg 10.14.04 at 2:36 pm
My exposure to statistics was in an undergraduate Analytical Chemistry course, and a graduate Data Analysis course taught by a psychology professor. The first was very practical: calculate this variable, look it up in this table, write down the variance in your lab book. The second course was more theoretical, though still with the idea that we would be doing many ANOVAs ourselves. I thought this was a very good class, perhaps because it was taught by a non-mathematician/statistician.
wes 10.14.04 at 2:36 pm
I think jam is right that the battery of formal statistical methods is mostly a set of tricks coupled with heuristics on how to use them. (The AP Statistics program here in the US does an admirable job of trying to explain what the tricks mean, but if you try very hard you come up against the “mumbo-jumbo” John Quiggin referred to: most of the explanations of the tricks are not very persuasive). The trick-and-heuristic description is especially appropriate when the methods are being used by people without advanced training in statistics. Statistics is one of the few subjects primarily practiced and taught by people without an advanced degree in the subject. This makes educational reform difficult.
What Deirdre McCloskey said of economics applies I think to statistics as well: it is not a subject that draws on the natural interests of 19-year-olds. Instructors of history, literature, music, and maybe even mathematics have it a little easier here. Even students who thrill to baseball statistics are unlikely to be fascinated by the logic of formal statistical inference. I don’t think this justifies Joe Student’s refrain that statistics is “boring,” but the cry of “boring” is so widespread that I have to think there’s something to it.
And I second the recommendation for Abelson’s _Statistics as Principled Argument_, but I agree with a reviewer who described it as too subversive to be used often as a textbook.
Bill Tozier 10.14.04 at 5:07 pm
Well, I have to both echo and refute Maria’s experiences.
Yes, introductory statistics classes are interminable, and in my experience relatively useless. This is true (also in my experience) about “advanced” mathematics classes of the sort that introduce vector calculus and differential equations, too. In general the classes I’ve sat through (several times, now, and each time “introductory” — don’t ask) have focused on mere explication of methods and proofs while rarely discussing application and analysis. The closest one gets to seeing how to approach and discuss a problem is typically a “word problem”; these are often graded from the point where the words have been eliminated by the student.
As an actual engineer-type person, of the researcher subclass, let me tell you how much this sucked. It sucked. I’d have given an eyetooth to get it, twenty years ago when it could have made a big difference. Nobody delivered.
But in hindsight, I see now that it’s a cultural phenomenon. Something about the culture of the academy, or perhaps just of those tapped to teach the basic concepts of statistics, is correlated with a fundamental disability to communicate knowledge.
But.
In the last three years, I’ve had the opportunity to hire academic statisticians (inhabitants of fairyland themselves) to do stuff for me on consulting gigs. This is the nominally “advanced” stuff — exploratory data analysis, machine learning, modeling, working with exotic assumption-breaking data, running R, and so forth.
It was great! They were great! I learned more in one two-month project about “basic” statistical concepts like linear dependence, goodness-of-fit and the foundations of principal component analysis than I did in four preceding classes. We had to visualize distributions, re-visualize them, swivel them around in many dimensions and get to know them, and in the process I picked up a good deal of the tacit knowledge that our academic crowd doesn’t even know they have.
And that’s the problem, fundamentally. Statistics is a practice, not a toolkit. It is steeped in tacit knowledge, depends on it, and no amount of explicit training in the explicit bits can offset the need to convey some of that tacit understanding.
Alas, instructors who try to teach statistics (or proof-based mathematics) rarely know they even have that tacit side. They see the “what next” as obvious, and never even note the need to try to pass it along. And so we in the audience all muddle through, perhaps eventually saying we “don’t do math.”
Except that we all need to be able to “do math.”
So my advice is this: go hire (or solicit in whatever other way presents itself) a good fairy or two, pick a real problem, put them through their paces, and pay attention to how they do it, not what they do. It’ll help.
Anarch 10.14.04 at 5:09 pm
Intro stat always won the difficulty competition—calculus was a distant second.
Part of that, as mentioned by several people above, is that the why of statistics isn’t explicable within the confines of your average intro-to-stats course. The Central Limit Theorem, for example, isn’t even meaningful without two or three semesters of calculus under your belt.
[I actually understood the CLT about two years after I took the stats course, my mathematical skills having developed enough by then. The only other distribution I ever *really* understood was the Poisson distribution, and that only for a few weeks, because it happened to relate to my thesis.]
Frankly, I’m with jam (although I did actually do the A Level Statistics module): intro to stats is just collection of models plus a set of heuristics telling you when to apply which model. Anything beyond that is, as we like to say, “technical.”
nirtak 10.14.04 at 5:30 pm
Maria,
while I have only very fleetingly been exposed to a course of statistics (and don’t remember the first thing about it), I can relate very well to your observation that statisticians seem to be
“…speaking nonsense in an obscure language … They simply repeat the steps of the procedure several times more, never saying why they did it or what the outcome signifies…”
I have a suspicion that they are not actually talking nonsense, but I certainly feel they are speaking a language that nobody ever bothered to explain the grammar of, during my entire career at school and after.
If mathematics are a useful tool for describing the world as it is, then there must be basic rules that govern how relations between things get translated into math-speak. But somehow I never progressed from imitating mathematical phrases parrot-fashion to actually being able to frame my own ideas.
I feel like a tourist in a foreign country with only a phrase-book, and heaven help me if somebody says anything that is not in the book! ;-)
If only mathematics were taught like a language, I might even have found it interesting – now I’m basically a mathematical illiterate, much to the amusement of my engineer husband!
Another Damned Medievalist 10.14.04 at 6:25 pm
I never took statistics. This appalls my other social science (here, history is SS, not humanities) colleagues. In grad school, only some of the history specialties required stats and/or demographics, and the students had to do the math and a bunch of SPSS stuff.
And my colleagues, who are appalled, ask me how I can do my job. What, they ask, did you have to do, if not statistics. Gee, I dunno. Exams in French, German, and fucking classical AND medieval Latin AND a paleography course. Not enough? I can also fight (with a dictionary and a lot of time) my way through articles in Spanish and Italian, thanks.
But there must be statistical analysis, they say. To which I respond: How. Many. Fucking. Documents. Do. You. Think. I’ve. Got. To. Work. With? For what I do, I use a lot of land transactions. 8th c. land transactions. Some years, there might be, in a given codex, as few as two transactions. So if I’m focusing on one area, and a relatively short time period, and there are only maybe 60 documents pertinent to the study — and I’m studying something that doesn’t always appear in the documents, or onlyu appears in the documents written by one monk, (and again, I’m not an expert) do I have a large enough sample to come up with statistical analysis that is in any way relevant or even true? Do I really need statistics to tell me that 3/4 of the properties donated to X have strings attached? Can I use that to extrapolate trends that actually stand up to the evidence?
It’s not that I have anything against statistics, per se, but can anyone see my point?
CalDem 10.14.04 at 6:48 pm
There is a book by Kennedy I think Intro to Econometrics or something similar. Its almost all intuitive explanation and its the best single book out there explaining statistics and how they are used. its a paperback, cheap, and thin.
Opus 10.14.04 at 7:41 pm
Maria, you are my soul mate and partner in being tortured. My midterm stats exam (for, yes, my third stats course) was Tuesday. At least 90% of the class is doing much better on the weekly homework than I am, and *no one* finished the exam in the time allotted. I am walking around under a cloud o’ doom.
Fortunately, when I went and sat, stupefied, in my advisor’s office, she told me that although a C won’t allow me to count the class toward my doctorate hours, I can just take something else to fulfill those credits and the fact that I took the class will fulfill the stats requirement. I immediately cancelled my plans to jump off the top of the university campanile.
Opus 10.14.04 at 7:43 pm
Maria, you are my soul mate and partner in being tortured. My midterm stats exam (for, yes, my third stats course) was Tuesday. At least 90% of the class is doing much better on the weekly homework than I am, and *no one* finished the exam in the time allotted. I am walking around under a cloud o’ doom.
Fortunately, when I went and sat, stupefied, in my advisor’s office, she told me that although a C won’t allow me to count the class toward my doctorate hours, I can just take something else to fulfill those credits and the fact that I took the class will fulfill the stats requirement. I immediately cancelled my plans to jump off the top of the university campanile.
Matt McGrattan 10.14.04 at 7:43 pm
I found this pretty helpful (and they also have a similar book for Business and Economics.. Intro to stats for business and economics, I think).
T.H. Wonnacott and R.J. Wonnacott. 1990. Introductory Statistics. (New York: Wiley)
{It was recommended to me by an economist friend and is quite user-friendly…)
Also, I liked Ian Hacking’s book The Emergence of Probability — for an interesting historical/philosophical overview of the birth of statistics. It won’t help you pass an exam, though, but it might help explain a bit about why statisticians started doing what they do.
I have to confess I still don’t really have a handle on statistics but the Wonnacott gave me enough of a handle to work on the things I really did need to do at the time (Bayesian stuff, philosophical bases of probability, etc.) and understand some real world examples.
agm 10.14.04 at 8:31 pm
I gotta go with jam on this one. I’m actually surprised to hear people say that stats was/is considered a subfield of math, because back home it was regularly mentioned that the two fields were mortal (or at least methodological) enemies. If I understand correctly, the only people mathematicians hate more for the abuse we do to math is we physicists (think chain rule, or the derivative as a ratio of two differentials, or the integral of zero must be zero, or …).
numerical fucking analysis; (or solicit in whatever other way presents itself): ROTFLMFAO.
It’s criminal to teach statistics without mentioning that Student was on Guiness’s payroll.
And just to show that I’m an evil physicist, two words: statistical mechanics. Enjoy =).
Cranky Observer 10.14.04 at 9:16 pm
One more observation: of the 100 or so undergraduate and gradute classes I have taken in my life, I use two weekly if not daily: Engineering Probability and Intermediate Political Theory. I do more technical work than most, but nonetheless having a basic understanding of probability is critical to job success IMHO.
Cranky
Patrick 10.14.04 at 9:43 pm
Matt…Wonacott & Wonacott is a classic! I second your choice.
agm, I’m with you. I’m a Research Consultant working with academics in a public university in Texas. The mathematicians I work with on computing issues regularly laugh at me and my background in social science statistics (econometrics, political methods, and psychological methods). To them, statistics is some voodoo that detracts from number theory and matrix algebra. All of our statistics courses are taught in the different departments (i.e. econometrics in Economics, political methods in Political Science, and psychological methods in Psychology).
When I started my Master’s about 6 years ago, I was horrified by statistics. I sucked at them despite having a background in engineering stats from my undergrad days. Now I work with them on a daily, comfortable basis and I write code for several different stats platforms. It is odd how the worm can turn…
Jonathan Goldberg 10.15.04 at 12:04 am
It’s hard to answer the question: what do you do when you’ve given a logical explanation of something mathematical, such as a statistical concept, and the student still doesn’t get it. The usual approach is to repeat the same explanation more slowly. This seems to be an application of the Ned Land school of linguistics. In the opinion of Mr. Land (the harpooner in Twenty Thousand Leagues Under the Sea) “It’s a mighty stupid native who can’t understand English spoken loud and slow.”
It works about as well here as it usually does.
I’m pleased to see from some of the postings above that real work is being done on this problem.
One problem with books written by mathematicians is that mathematicians are poets. It pains them to put on paper anything not strictly necessary to make the point at hand. This results in the “lapidary style,” which is wonderful for them and torture for everyone else.
But there are exceptions. This post was prompted by the mention of Gilbert Strang’s open courseware Linear Algebra site. I haven’t seen it, but I’ll bet it’s wonderful.
I think this because, after having taken (and even passed) a course in it, I learned what I know about Linear Algebra from Strang’s Applied Linear Algebra. This is THE BEST MATH TEXTBOOK EVER. It’s astonishing how much Strang can clarify things with a few sentences or even a few words of motivation: “the natural way to think about this type of problem is this because…” I think that anyone writing a text on a mathematical subject for anyone other than other mathematicians should be required to memorize Strang’s book first.
I was especially impressed with his development of the usual least squares formula without calculus, starting from the idea of “the line closest to the data.” I’d always seen this done using the usual calculus-based minimization method. Doing it without calculus was a revelation I still remember twenty years later.
Ethesis 10.15.04 at 1:31 am
Ok, I enjoyed statistics, at least the second class, I found the first class’s grading frustrating.
Now, I’ve got to confess they designed the advanced statistic’s final just for me (I was ten points ahead of the next highest grade out of any of the sections who took the final, who was ten points about the “A” grade cluster), but …
Statistics can be taught without being terribly painful. Our school used, for the most part, Econ grad students who taught for 3-4 years and who were always fresh, invigorated and engaged (and the only non-tenured people teaching any intro class to Econ. My other intro class profs included the guy who became department chair and a full professor).
The key is to love the subject.
I had been out of school for two years (doing volunteer church work) before that first stats class, and a speech major before that, so I wasn’t a math geek going in, in spite of any aptitude for the subject.
But what made it work, and what led to a 90% pass rate or so (in a department that had an average gpa of 1.98 — so they weren’t exactly passing everyone) was engaged instructors who liked the topic.
And good textbooks. If you want to make tenure, use good textbooks.
roublen vesseau 10.15.04 at 1:33 am
Last post, but I thought of two more books which might be helpful in developing a statistics intuition: Richard Fenyman’s two autobiographies, “Surely You’re Joking, Mr. Feynman!” and “What Do You Care What Other People Think?”. They’re both very funny, but I think there is a lot of probability-thinking in it, especially in his discussion of the failure of the Challenger shuttle, and the differences in mindset between the top managers and the lower level engineers.
Patrick 10.15.04 at 1:59 am
My personal pet peeve here-
The reason statisticians often give you mathematical gibberish in response to a social science student’s question of “but what does it all MEAN?” is because the mathematical gibberish is about 100 times better at answering the question than anything they could put into words.
I’m reminded of a class conversation I had in law school. There were some numbers about different stop and frisk policies, and there was dispute as to whether these numbers showed an increase of 2% in a particular effect, or a 20% increase. This seemed to make a lot of difference to people.
It was the dumbest discussion ever. The raw data was 10% originally, then 12% afterwards. Increase of 2%? Or 20% increase?
BOTH. Come on. It just depends on how you want to articulate what the numbers say. And frankly, knowing the numbers is a LOT more informative than just hearing “20% increase.” 10% to 12% is a 20% increase. It’s also an increase of 2% raw. It makes a difference how you describe these things in how people will understand the data. They hear 2%, they think small increase. They hear 20%, they might think big increase. But, neither explanation is inherently better than the other.
So many people were convinced that there was a “right” way to articulate this. There wasn’t. The closest thing to a “right way” would be to repeat the original numbers.
Statisticians know this, and they get leery of telling you something in words when they think you might draw a false conclusion. So try to give them some credit on it. They’re not being obscure, they’re being accurate. This may be frustrating for social science students who want the verbal answer, but sometimes the verbal answer is less useful if you’re goal is accuracy.
Oh, and graphing a quadratic is taught to college preparatory students at 10th grade, and some students as early as 7th or 8th. So the instructor was probably being honest when he made his comment about basic arithmetic.
Maria 10.15.04 at 9:25 am
Hi all,
sorry to have stepped out of comments – just the tiny matter of actually doing that exam… which I’m not sure I passed. Oh well, if I have to re-sit, maybe next time I’ll start to get it!
Alex Fradera 10.15.04 at 1:15 pm
Agree Feynman is another good way to get some statistical thinking.
Some statistical namedropping (pointless I know, but when else do you get a chance to do this?) – Jonckheere hangs out at my department and I have workshopped stats with him (and THAT is a fun way to meet the subject head on) – a living legend of stats.
Our dep is quite cool for its stats heritage, what with Spearman to boast about (and Fisher down the road at the genetics dept).
Jason Grossman 10.15.04 at 7:20 pm
I agree with this:
“‘g’ has it right. The whole apparatus of significance tests, confidence intervals and so on is a load of mumbo-jumbo.”
But I’m not just posting to say “me too”. I’m posting to say that if we’re right that the apparatus of statistics (currently) makes no sense, no wonder it’s no fun to learn.
I think what we have is NOT mainly a pedagogical problem. Sure there are pedagogical problems, but they exist in other subjects too, and in some cases are just as bad. What makes statistics so special is that your teachers don’t KNOW why the methods they’re teaching you make sense, and the reason they don’t know that is that the DON’T make sense!
Jason Grossman 10.15.04 at 7:24 pm
Oh, P.S. I write about this stuff for a living (I’m on the faculty at a good university), and I’ve taught it to undergrads, so even if you suspect I’m wrong you probably find it interesting to know that there are people who do understand statistics and still think it’s gibberish (as currently practised).
Fluffy 10.16.04 at 3:03 am
I enjoyed reading Maria’s comments. My perspective about teaching technical topics to non-technical people is:
1. Don’t lie. If you really need algebra, then tell students they need algebra. If they need calculus, tell them you need calculus. Be up front.
2. Present a few formulas, but not too many. Any Ph.D. student in a decent American university in social sciences should be able to handle some algebra. For crying out loud – they are a “Ph.D.” student!! Asking for some algebra is really not all that much. Have some standards!! Expect a high school level education from people.
On the other hand, don’t get lost in equations. Statistics is about “inference” – got that? If you want math, go to a different department. We do statistics because it helps us reason about a complex world. So don’t drench the class in equations, but explain how statistics helps us make choices.
3. Do a lot of “real” examples. Download some GSS or NES data. Don’t use silly text book examples. Use data that social scientists might really be interested in.
That’s my $.02
Cogan 10.17.04 at 8:24 am
Huh. I must have been really lucky. I took a few lower-division stats courses at UC Berkeley and found them surprisingly interesting and always well taught. I didn’t have too much trouble grasping the basic concepts or learning the little bit of linear algebra required, and I’m not even a math geek. The Freedman/Pisani/Purvis textbook is good at explaining the concepts but skimps too much on those very useful formulas. Other good textbooks are Multivariate Statistics by Sam Kachigan, and Modern Elementary Statistics by John Freund, both of which give you the formulas. I also enjoyed The Pleasures of Probability by Richard Isaac. Maybe I’m weird, but I found stats and prob downright fun, because it’s so damn counterintuitive. Its results often defy your commonsense expectations. I regret now I didn’t take more stats courses.
Tobias Schwarz 10.18.04 at 12:10 am
The questions statisticians dread; ‘but WHY did you do that?’ and ‘what does it MEAN?’. Why is it, that when you ask a statistician one of these questions they look at you as if you’ve addressed them in ancient Greek?
Strange, I had to take stats classes at the LSE, too, and I liked them much more than anything I had done previously in this area not just because the teaching was great but also because they usually provided answers to the questions you mention here.
LarryH 10.20.04 at 11:43 am
It appears that you’ve taken stats courses which are geared to humanities-oriented bullshitters. My background is in science and technology(B.S.Ch.E.) Would you believe this?… people in technical areas can actually make halfway SENSE out of various statitical criteria!
A corollary to the expression “Get a life” might be summed up by “Get a meaningful perspective.”
My advice: find a stats course that’s not populated by student and instructor SLACKERS.
Comments on this entry are closed.