Two from the FT

Thanks for the tip! It quotes one of my favorite educational statisticians, Harvey Goldstein, who has done yeoman’s work for years on the league tables and value-added statistics.

To bob b, I’d point out that there is a difference between transparency (which is good) and the misleading implication that statistics are more accurate than they really are (which is bad). There’s nothing to prevent governments from adding error bars and confidence intervals to league-table-like results, and yet they never do.

“If you look at any league table, you see leafy selective schools at the top, large under-resourced urban schools at the bottom.”

That is a convenient myth. Several of the London boroughs placed lowly in the league table of local education authorities (LEAs), based on school leaving exam attainment, are actually big per capita spenders on schooling whereas top rated LEAs are only middling spenders.

Part or much of the basic trouble is that some local politicians running LEAs gained an insight years ago that maintaining poor school standards had reassuring, predictable electoral consequences according to the equation:

Poor school standards = uncertain job prospects for school leavers = vote Labour

It is to Blair’s credit that he is trying to undo this well-recognised recipe, which helps to explain why he is in so much trouble with the many Labour MPs who work on the principle: if it ain’t broke, why fix it?

The league tables are necessary to show up which LEAs and schools are failing. We can then look in depth to see whether failing is due to low spending, to local affluence/poverty, poor teaching, local politics, cultural factors or sheer inertia.

The enduring potency of cultural factors is not to be under-estimated. Unfortunately, the picture painted by George Orwell in chp.7 of The Road to Wigan Pier (1937) is still recognisable in places:

“And again, take the working-class attitude towards ‘education’. How different it is from ours, and how immensely sounder! Working people often have a vague reverence for learning in others, but where ‘education’ touches their own lives they see through it and reject it by a healthy instinct. The time was when I used to lament over quite imaginary pictures of lads of fourteen dragged protesting from their lessons and set to work at dismal jobs. It seemed to me dreadful that the doom of a ‘job’ should descend upon anyone at fourteen. Of course I know now that there is not one working-class boy in a thousand who does not pine for the day when he will leave school. He wants to be doing real work, not wasting his time on ridiculous rubbish like history and geography. To the working class, the notion of staying at school till you are nearly grown-up seems merely contemptible and unmanly. The idea of a great big boy of eighteen, who ought to be bringing a pound a week home to his parents, going to school in a ridiculous uniform and even being caned for not doing his lessons!”
http://www.george-orwell.org/The_Road_to_Wigan_Pier/6.html

Just to show that I am sadly not mistaken in this, consider this recent BBC news report:

“Almost half of 17 year olds in some parts of England have dropped out of full-time education or training, government statistics reveal. . . . The statistics, which were issued in response to a Parliamentary question from the Liberal Democrats, confirm England’s poor international standing for staying-on rates in education. The Organisation for Economic Co-operation and Development ranks England’s drop-out rates as among the worst among industrialised countries.”
http://news.bbc.co.uk/1/hi/education/4481300.stm

The fact that England shows up poorly on stay-on rates in education after 17 by international comparisons among OECD countries rather proves that commitment to education and staying on in full-time education is not just a simple matter of the deprived inner-urban areas versus the leafy suburbs.

What makes that international ranking especially paradoxical is that by EU standards, the UK is more affluent in terms of GDP per capita than most other EU15 countries according to this:
http://epp.eurostat.cec.eu.int/pls/portal/docs/PAGE/PGP_PRD_CAT_PREREL/PGE_CAT_PREREL_YEAR_2005/PGE_CAT_PREREL_YEAR_2005_MONTH_06/2-03062005-EN-BP.PDF

“…each figure is based on a limited sample, and thus inherently uncertain.”

Bollocks. Position in a league table is not based on a sample, it’s based upon the results from the entire population. There is no uncertainty in the result, every single person is counted. There’s some statistical (or journalistic) slight of hand going on when people suggest that the “true performance figure” is different to the performance figure you get when you count all of the results.

The main point – that there may be no substantial difference between the results of school in first place on a league table and the school in last place is perfectly valid (i.e. that the differences between schools may be the result of change, or so small to be inconsequential). It is perfectly true that changes in ranking may be meaningless.

It’s a shame they didn’t focus on regression toward the mean more. The government likes to select underperforming institutions, put special measures in place, and then declare a success when they improve. That’s basic to New Labour. It’s a shame more people don’t realise just how stupid it is.

Soru;

They’re not saying the results are totally random (i.e. that a complete reversal could take place). But they are saying that the random element is so great you can’t be sure that the school in top place is any better that most the other schools in the table. If that’s the case, that does kind of destroy the purpose of a ranking.

Position in a league table is not based on a sample, it’s based upon the results from the entire population

Not quite right; I would say it’s more usefully thought of by saying that the result of each year’s cohort is a sample of the underlying unobservable “performance” of the school, with noise introduced to the process by the stochastic “quality” of the kids they had to start with. The FTSE100 index is based on the population of the entire 100 stocks in the FTSE, but it’s still noisy over short time periods.

Even 300 would make it a smallish suburban or urban american school. But certainly in England and Wales (sorry, I should not imperialistically have used Britain) 10-15 form entry (whcih is what you are describing) is considered huge, and quite outside the norm.

They’re not saying the results are totally random (i.e. that a complete reversal could take place).

Well, the article does say that, even if it didn’t mean it:

so wide that they overlapped those of every other school, making a mockery of any attempt at ranking them

The relative ranking of two schools could easily be 99% certain while their 95% error bars overlapped. Not many things in life are 99% certain, it would be pretty foolish to mock those that are.

In any case, the concept of ‘error bars’ based on sampling error seem pretty dubious – is he really assuming the academic success of pupils in the same class, doing exams on the same day, is statistically independant?

It really does seem to me that either the article simplifies or distorts the professor’s views, or someone is attempting to deliberately mislead through misuse of statistics.

That aside, there is clearly some kind of valid underlying point here, that league tables are more precise and cleaner than the underlying data supports. A school bouncing about between, say, #1 and #3 in a league table almost certainly means nothing in terms of actual standards, but probably causes the head no end of phone calls from irate parents.

Sport aside, not many other areas of life evaluate things so brutally. Kitchen salesmen and japanese school pupils may get ranked strictly, everyone else gets a much vaguer grade, more like the school grades ‘A+’ to ‘F’.

Perhaps some qualified statistician could come up with a scheme where schools with a ranking that was genuinely statistically uncertain were ranked equal in the table?

soru

Harry,

We are learning to live with proliferating numbers of league tables for all sorts of things from football teams through “international competitiveness” to the Global 500 and the ranking of hospitals for surgery success and infection rates. Police forces now have to cope with regular performance rankings. The genie is finally out of the bottle and if there aren’t official league tables, the likelihood is that the press will invest resources to build their own league tables for the public services because their readers – the consumers of public services – want to know and won’t be put off.

Hospital league tables generated a parallel critical discussion to education league tables. Challenging issues arise in ranking hospital departments – the major teaching hospitals tend to attract referrals of especially difficult cases with inbuilt risks of higher failure rates so adjustments need to be made for that factor in making comparisons but the league tables illuminated some worryingly high failure rates in some hospital departments which then became the focus of closer investigation and remedial measures where appropriate. It will virtually impossible to coax that genie back into its bottle now.

Much the same applies to local education authority and schools league tables and for similar reasons. For all the many flaws in the tables, there are substantive reasons for continuing concern about raising and sustaining schooling standards in Britain as I have tried to argue and document above.

We need to recognise situations where particular schools are failing in order to focus investigation and introduce remedies. The fact is that many stereotypical explanations are often actually incorrect – I can recall a paradoxical case of an 11-18 (comp) school with miserable attainment rates at GCSE for 16 year-olds but above national average attainment rates at A-level for its small sixth form. How come? There is no obvious, credible explanation in terms of resourcing or poor teaching – I fear the Orwell factor applied.

Can’t say I’m in any way qualified to judge whether the WHO no smoking thing is bad or good publicity but it certainly isn’t “both idiotic and anti-liberal”.

You can’t be a smoker and effectively lobby for implementing the FTCT, which is an important part of the WHO business.

It would sound like “Hey, I can’t convince myself to quit, but surely some of those 1 billion Chinese will fall for my great anti smoking PR line!”.

Presumably, a similar critical analysis must apply to healthcare league tables, possibly with even greater force on sampling issues. No? But then recent news reports on hospital care such as this suggest very persuasive reasons why monitoring surgery success and infection rates are essential for identifying problematic situations:

“LONDON, Dec. 2 (UPI) — One-fifth of post-surgical patients in Scotland were infected by bacteria such as MRSA, a government report said. The Scottish Audit of Surgical Mortality quantified deaths related to hospital superbugs for the first time and said that of 1,854 people who died after an operation, 376 of them were infected by methicillin-resistant Staphylococcus aureus, known as MRSA despite Health Department campaigns to control infections.”
http://www.upi.com/NewsTrack/view.php?StoryID=20051202-021024-2659r

Hi Dave: Good to see another of your posts again – I won’t mention the lamentable performance of the Eurozone since we exchanged forum messages about that elsewhere in March 1999 after Lafontaine resigned from Schroeder’s government.

Having been through a cardiac [diagnostic] surgical procedure at a major London teaching hospital almost exactly a year ago, I can assure you that some hospitals take precautions against MRSA infection very seriously. In addition to maintaining rigorous cleaning and staff hygiene practices, all surgical patients are routinely subject to swab tests for MRSA infection before surgery.

Fortunately, I had no subsequent MRSA complications – but I was nevertheless much relieved to be advised that as the result of the procedure, the need for further cardiac surgery wasn’t indicated. This contrasts with the previous experience of someone I know well who twice contacted MRSA infections after cardiac surgery at other hospitals. I also know several people who live locally and who lost spouses through MRSA infections following surgery.

The hazards of post-surgery MRSA infection are really not to be under-estimated. I can promise you that it is not just a tabloid scare but a very real issue.

“Your anecdotes aren’t statistics.
Do these Scottish hospitals test non-stiffs for MRSA?”

But I was quoting a news agency which claimed to be quoting from, “The Scottish Audit of Surgical Mortality”.

I can’t speak for the practices of all British hospitals but I have testified as to my own, very specific experience of a surgical procedure in a London teaching hospital where MRSA swabs were taken before surgery.

What I can also say from following the news and from personal contacts with (mainly retired) health service professionals is that post-surgery MRSA infection is certainly a challenging – and so far largely intractable – continuing issue for hospitals in Britain.

Googling will retrieve huge repositories of news reports, studies, audits, ministerial announcements and the like. The problem has international dimensions – not least because this malignant strain of Staphylococcus aureus is alleged to be deeply entrenched in a particular province of Canada, and with significantly lower infection rates elsewhere in Canada.

This is arguably important because it suggests that a silver bullet approach – eg “it’s all due to poor hospital cleaning” because of private cleaning staff – is unlikely to be viable as an explanation for the incidence of MRSA or the basis of a successful remedy.

In the present context, I am also saying that hospital league tables were challenged on similar grounds to school and local education authority league tables but they have become a valued tool for illuminating (often life-critical) issues in healthcare and professional competence. Whatever their flaws, these tables won’t be pushed back into the bottle.

Btw on school league tables, I have a CD, issued recently as a freebie by The [London] Times, with an all schools database supposedly showing the examination results of each and every school. The protective cover displays in bold type: Parent Power.