Having been involved in the debate over schools policy for quite a few years, I’m enjoying a bit of schadenfreude following the publication of a couple of regression analyses showing that students at charter schools (publicly funded US schools operating independently from the main public school system) score worse on standard tests than students at ordinary public schools[1]. I don’t have a particularly strong view on the desirability or otherwise of charter schools, but I have long been critical of one of the most prominent rationales for charter schools and other programs of school reform[2].
This is the claim that “regression analyses show that students in small classes do no better than those in large classes”. If you believe this claim, you should believe the same claim with “charter schools” replacing “small classes” since both are supported by the same kind of evidence.
The class size claim goes back to the pioneering attempts of James Coleman to estimate “educational production functions”, which came up with the conclusion that almost everything is insignificant. A lot of other studies had similarly negative findings, and the natural interpretation, that this was because there was no effect, was promoted vigorously by a number or writers, notably including Ric Hanushek, who collected a bunch of studies and summarised the results as showing no effect.
An alternative interpretation is that the dependent variable in these regression analyses is so noisy as to make the results highly suspect. We want to know how much the kids learnt in a given period in school, but what we can typically measure is the difference between two test scores, before and after the relevant schooling[3]. A test score is a pretty noisy measure of how much someone knows; the difference between two test scores is so noisy as to be close to useless.
Of course, problems of this kind arise in lots of contexts, and there’s a big literature on meta-analysis as a way of extracting meaningful results from disparate studies. Hanushek’s “counting studies” approach is a very crude kind of meta-analysis. A more sophisticated version, undertaken by Hedges, Laine and Greenwald[4] shows that there is a significant relationship.
Although not everybody at CT likes it, there’s also revealed preference to consider. If small class sizes are educationally valueless and are adopted as a response to, say, teacher union pressure, we’d expect to see a very different allocation of resources in private schools. In fact, though, wealthy private schools typically go for small classes – the share of budgets going to teaching staff doesn’t vary much between public and private schools.
Anyway, I’ve been hammering these points for years, without much of an impact. But now, I expect, a lot of people are suddenly going to discover that comparisons of school performance are tricky, that regression analysis is not infallible, and that there’s more to educational outcomes than test scores. I hope they will be consistent enough to revise their prior beliefs about class sizes.
fn1. As you might expect, there’s another study that gets the opposite result.
fn2. As always, I use the term ‘reform’ to mean ‘structural change’, without any connotation of approval or disapproval.
fn3. Sometimes, there’s only one test score, which makes everything even worse.
fn4. Hedges is one of the big names in meta-analysis
{ 11 comments }
dsquared 11.25.04 at 12:05 pm
Further to John’s footnote number 2, I’d suggest that this neutral meaning of “reform” is stylebook for CT, and that the definition I used to use on D-Squared Digest (“a repulsive attack on the interests of the common man, made to sound less disgraceful by calling it a reform”) will only be used if specifically flagged.
JennyD 11.25.04 at 2:07 pm
As a statistician doing educational research at a big fancy state university, I agree with much of this. All the analyses are noisy for several reasons.
But I would argue that the analyses involving charter schools and class size are likely quite valid. Why? Because neither of these variables is actually a measure of “teaching” or “education.” These are some kind of measure of “school organization.”
So consider class size. If you have two teachers, one who uses very effective teaching strategies and who uses ineffective teaching strategies, and you give them both smaller classes, your coefficient for the effect of small classes should be the same as with larger classes. By putting fewer kids in the room, you’ve simply minimized the number of kids exposed to the teaching of these teachers. So one group of kids would learn more and one less. (This assumes that the classes of kids are equal on either measures, like SES, etc.)
Now suppose you take the same analysis with charter schools. Yes, there are more covariants to control for, but still, a charter school is just a measure of school organization, and nothing more. It is often assumed by analysts that charther school is a code for “better teaching.” But that’s an assumption.
I’m actually working on my dissertation and I’m trying to build better measures of instruction so I can use these in some kind of analysis. Probably an HLM model using groups of kids in a classroom, and the teaching being the level two, or level three variable.
tc 11.25.04 at 3:25 pm
I don’t think so. What the NYT story doesn’t say is that the differences between charters and regular schools were no longer statistically significant after controlling for race.
Giles 11.25.04 at 4:04 pm
I think that the interesting revealed preference is that black Americans and Hispanics were both over represented in the charter schools – suggesting that they as a group perhaps place greater weight on things other than standardized test scores.
MC 11.25.04 at 4:45 pm
That also brings up the possibility that race is a mediator of charter schools and success. If this were the model, then of course differences between charter school would become insignificant. I don’t know the theory and haven’t seen the numbers, but if such a model makes sense, it’s something to consider.
harry 11.25.04 at 7:27 pm
I can’t get at the NY Time article, so take this with a pinch of salt. Charters probably do have less teachable students, because the political and other pressures for charters is higher where schools have children with lower performance. This pressure is a response, I would guess, to student quality rather than school quality, because people think that schools with good students in them are good and schools with bad students in them are bad. Furthermore there is consistent opinion poll data showing that support for choice, vouchers, and charters is much higher among blacks, for example, than among whites.
John’s point about class sizes is interesting, but I discount it a bit (only a bit). Huge swathes of well-educated professorial types that I know are convinced that small class size have a huge effect, and have no idea there is a serious empirical debate about it. They pressure for small classes for their kids, and would choose them for them in private schools, for that reason. But there is also a non-educational reason for choosing small classes (and small schools) which I would guess the private sector responds to: it is simply a recipe for a more pleasant experience for both teacher *and* students, to be in a a smaller class, regardless of the educational effects (understood in terms of test scores). We’ve obliquely talked about this before, and I still promise a thoughtful post about it, John. But next time you post on an educational topic, please don’t do it on a US national holiday!!!
rd 11.25.04 at 7:50 pm
I take the point about the difficulty of catching what we want to know with statistics. But, having followed the link to the NYT article, I still found its report of the results mildly infuriating. We’re told that in two of the five states studies, Texas and Colorado, charters performed worse than mainline public schools even after taking greater race and poverty into account. Which means that in the other three states….what? No difference after taking race and poverty into account? Could charters actually be performing better in those states once you take race and poverty into account? The Times sees no need to bother with such petty details, since they don’t further the obvious agenda of the article.
Tom West 11.25.04 at 9:53 pm
people think that schools with good students in them are good and schools with bad students in them are bad
How *isn’t* this true? Students are more influenced by peers than anything else. As many people have pointed out, many private schools exist to help parents avoid bad students, rather than anything wonderful about the school itself. And they’re right. Young people conform to their peers.
So, as a parent, I can’t see any better criteria for evaluating a school than by how good its (incoming) students are. It’s probably a very good stand-in for how good my child will be coming out.
harry 11.25.04 at 10:22 pm
As a parent you have absolutely no interest in improving the overall quality of schooling, only in making sure that you’re own kid gets the optimal level of support, and yes, peer-effects will be significant (though maybe not as important as you think). But as a policymaker you have no interest in any particular kid, but in improving overall performance under given budget constraints. Peer effects are then just one variable. Your comment, though, Tom, might help explain why people are so ignorant; they confuse the (more-or-less) rational judgement that ‘X school is better for my child because it contains lots of high-performing kids’ for the mistaken global judgment ‘X school is a better *school* because it contains high performing kids’.
Tom West 11.25.04 at 11:07 pm
I’ve always had a lot of admiration for those who actually want to work in the ‘tough’ schools. They’ve got everything arrayed against them: student attitudes, home environment, low budgets, and public opinion. You have to fight tooth and nail for every advance. Meanwhile, schools in higher socio-economic areas get so much for free. Even a mediocre teacher is going to be able to get results from a class that gratify the teacher and impress the parents…
I swear I don’t know how they do it. Those who can give it their all in such environments, day after day, really are heroes.
Ralph Squillace 11.29.04 at 5:39 am
I read the NYT article, and while the smaller classroom thing is there, I too was interested in the ethnic distribution (I was surprised) and the performance difference between some states and others that wasn’t elaborated upon.
An aside: Do people have a good starting place of the assessment for the rise of educational costs over the course of the century? As an historian of politics, I’ve not delved into the economics and public policy of public and private education before, so a good starting place would be appreciated.
Comments on this entry are closed.