For various reasons I’ve been thinking a lot about schools that beat the odds lately (schools with high need student populations but which get high achievement out of them). My inclination has been to agree with people who think that if we could figure out which schools do beat the odds we might learn something useful from them (the best argument against this was recently put to me by an expert on whole school reform who said that finding out what naturally occurring schools that beat the odds do would not tell us anything about what policymakers could actually make schools do, which is the useful thing we are trying to learn). But are there, in fact, any schools that beat the odds, and if so do we know which ones they are?
It seems that the answer to the second question really is no.
Let’s start with a brief summary of Richard Rothstein’s rather devastating critique, in Class And Schools, of the “schools that beat the odds” gambit.
Rothstein is responding to three lists of schools that beat the odds—the Heritage Foundation’s list of “No Excuses” schools (taken from No Excuses : Lessons from 21 High-Performing, High-Poverty Schools), the Education Trust’s list of “High Flying” schools and the so-called 90-90-90 schools. His argument against the evidence from these schools has three basic components:
i) In every case of a school that is said to beat the odds, there’s something demonstrably different about the children in these schools, therefore there is no evidence that they are beating the odds. Almost all practice some form of selection. Very few practice what might be called cherry –picking (though some do); but many of them avoid enrolling the students most likely to perform very badly indeed and most likely to consume high levels disciplinary attention. The KIPP schools, for example, in requiring parents to opt in and to sign a contract – and then in enforcing the contract by expelling students who display major behavioural problems – exclude the most difficult students who are, therefore, concentrated into other schools. The Pentagon schools exclude the most disadvantaged students (whose parents cannot gain admission to the military) and enjoy the benefits of a social-democratic style welfare state by virtue of their parents military service. (An anecdote: the Superintendent of a large urban district recently told me the cost of the most expensive student in that district: $350k per annum, which is paid by the district to a correctional facility in another state, in response to a court order. These kinds of costs are not shared by charter schools).
ii)In some cases, there may be something remarkable about the schools, but investigation of the details makes for skepticism that the desirable features can be scaled up. These schools rely heavily on a limited supply of young teachers without families (again, the KIPP schools appear to do this) and have found exceptional principals. One of the striking flaws of Jay Mathews account of David Levin’s and Michael Feinberg’s achievement in creating the KIPP franchise, Work Hard. Be Nice., is his failure to consider the possibility that these two young men whom he seems to be so struck with might have a similarly magnetic effect on high quality teachers who may thereby concentrate in particular schools, producing remarkable effects at a cost to other schools where they would otherwise be located.
iii)Some of the schools just aren’t producing high-achievement anyway in a systematic way. Rothstein reports from his systematic examination of the Education Trust’s 1,320 schools, at least half of whose students were both poor and minority, and whose test scores in math and reading were in the top third of their states that “only a third of the high-flying schools had high scores in both reading and math. Only a 10th were high in reading and math in more than one grade. Only 3% were high in reading and math in at least two grades for two years running. Less than half of one percent of these high poverty and high minority schools were truly high flying, scoring well consistently”. [my italics]
It gets worse, thanks to my colleague Doug Harris, in his paper, “High flying schools, student disadvantage, and the logic of NCLB”. Using the School-Level Achievement Database and the Education Trust’s definition of high-flying schools (high-performance in either reading or math in the grade and year selected by ET for analysis), Harris has estimated how many schools remain high flying over time, and what the characteristics of those schools are. Using a sample of 18,365 schools he finds that when the definition of high performing is changed to require consistency over time fully 93% of schools identified as high-performing for a year drop out of the category. And whereas low poverty schools are only 3 times more likely to be high performing than high poverty schools on the single-year definition, they are 22 times more likely to be high-performing on the definition that requires consistency over time. Using data on schools with high minority populations he finds that, on the more demanding definition, the “likelihood that a low-poverty-low-minority school is high-performing is 89 times greater than for a high-poverty-high-minority school”.
What’s the upshot? It is not that there are no schools that beat the odds; rather it is i) that very few schools do beat the odds, few enough for us to wonder whether there is very much to learn from them, ii) that we don’t have any reason to think that the schools identified as beating the odds are actually doing so and iii) that we haven’t identified whatever schools are, actually, beating the odds.