The US News College Rankings Scam

I think he missed the point.

Think American Football – only one team wins the Super Bowl and it isnt necessarily the best team, but it all that matters. College rankings are similar – only one top 10 list in USNews, regardless of what the “real, statistically valid” measurement would say. The difference between Pomona and Claremont is what, exactly and why is it relevant to whom?

The real shame about this is college administrators, faculty and students have sat by and let excellence and ranking by determined by US News and have not done the more challenging and honest assessment of asking “are we doing a good job”? And the follow-up “Can we do a better job”? Since that might upset the applecart, it is easier just to live-and-die by US News … I know when I advise students applying to college, prestige is an almost impossible barrier to overcome when selecting a school – the only thing that I’ve seen that can do it is $$$…

One of the nicer things about Google Reader is that it leaves those annoying extras off so you can cut and paste without being called a pirate (Arrrr!). Not actually being a pirate, I don’t need the reminder to add the URL along with the fragment of text I thought worth sharing.

Understudy@1 ‘The real shame about this is college administrators, faculty and students have sat by and let excellence and ranking by determined by US News and have not done the more challenging and honest assessment of asking “are we doing a good job”?’

My experience is in the UK rather than the US, where we have a slightly different set of newspaper league tables. FWIW, that experience is that managers [administrators], academics [faculty] and governors [trustees], far from having ‘sat by’, are the most eager customers for rankings because they have so completely internalised the hierarchy that the ranking expresses. Some students also care.

They will need mutliple copies of Introduction to Statistics in Hell. Or perhaps that too will be part of the punishment.

Well they have to be better than the revealed preference rankings some professors crazily tried to institute a few years ago.

I long for the time when there were no rankings, just the patronage of the Ivy League, Seven Sisters and small private New England colleges by the nation’s WASP plutocrats and cultural elite. Distinctions of reputation, quality and character were not voiced in publications any person on the street could purchase for a few dollars, but instead made in hushed tones in wood-paneled rooms on Fifth Avenue and Beacon Hill. The Scotch tasted better back then, too.

Now, what next? Biannual rankings of philosophy departments, to see which ones philosophize better, and make better philosophers?

In theory, I am not against the idea of rankings. Some things are better than other, like things in way that we can articulate and that have meaning to a wider audience. Brown is a better school than Claremont McKenna in important ways that make sense, and not just because one of them apparently has an administration full of liars. But the idea of an individually enumerated top twenty, or fifty, is ludicrous. Instead of correcting wider ignorance about higher educations, these types of rankings perpetuate them.

On the other hand, I think there would be surprisingly widespread agreement about which 15 colleges and universities are the top in the nation, which 25 follow in the second tier, and then probably which 50 are in the third, at least as far as undergraduate education is concerned. Judgments at the margin would be difficult, but that is not a reason not to make them. The rest, as long as they are accredited, would be perfectly serviceable places to get a degree from, or to work.

My (also UK-based) experience, unlike Andrew Fisher’s, is that most university staff haven’t internalised the rankings’ hierarchy, but feel ourselves powerless against it. Many therefore think that gaming rankings as far as possible is a tactical rather than a moral problem: after all, since the rankings are always going to be nonsense, what’s wrong with influencing the nonsense to our own survival and benefit, especially since others are going to be doing the same?

That’s orthogonal to Understudy’s point about thinking about whether we’re actually doing a good job, and how to do a better one. Again in my experience, many of us are deeply concerned about this, but don’t think that the rankings have much to do with it.

Rereading myself at 9, I realise that I come across as saying ‘hey, I game the rankings all the time, what’s the problem?’, which is not actually what I meant. I don’t have the power to do that. I meant: I understand why e.g. Case Law School got to the point described in the OP.

in the endeavours where I earn my daily crust, a pernicious form of rankings/metrics known as KPIs (Key Performance Indicators) is venerated. Unfortunately it appears that any KPI or set of KPIs can be maximized in an unproductive way. Management prefers not to know this.
This is only another way of repeating Goodhart’s law,
any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

Presumably it is the aspect of control which produces the reality-distortion field around the metrics. It is a pathological type of gamification – careful design can produce Foldit, profit-maximizing strategies conflated with obfuscated metrics gets you the US News rankings and related shenanigans. I prefer the Washington Monthly rankings.

In related analysis, a careful disassembly of Boston Magazine’s dissembled school rankings is at furialog. It is a thing of beauty.

Steven @7: I think there would be surprisingly widespread agreement about which 15 colleges and universities are the top in the nation, which 25 follow in the second tier, and then probably which 50 are in the third, at least as far as undergraduate education is concerned.

Probably. But that’s not the value of rankings. There are on the order of 2,000 four-year undergraduate institutions in this country. 99.997% of applicants aren’t going to even apply to the top 15. What is needed is some unbiased guidance distinguishing between the schools that aren’t household names. Is Mary Baldwin better or worse than Mary Washington? In what ways?

The standard college guides don’t attempt to judge institutions. US News at least tries, even if it does a terrible job. Someone ought to try to do better.

“Let’s be honest, the U.S. News rankings are more about prestige than educational quality. Rankings of educational quality would necessarily be (sometimes strongly) dependent on major.”

And if prestige is one of the most important parts of having a college degree, they’re ranking the important thing.

Now I’m willing to agree that prestige ought not be one of the most important things, but I strongly suspect for most practical purposes that is how a college degree is used. In places I’ve worked, college degrees were important for only three major reasons. First, it fulfilled a listed Human Resources requirement for a job. In that case pretty much any college would do. Second, it fulfilled a Human Resources ass covering requirement. In that case prestige was important (i.e. we hired from Harvard, who could have known the guy didn’t know anything useful). Third, it fulfilled a networking requirement. In that case the prestige was vital.

The concept of “we know that such and such a college has a particularly good program that is well suited for exactly the job under consideration” was such a distant fourth place that I think I’ve only seen it twice. [Though I’ll admit that it *may* be more common in technical fields].

Prestige is a relative concept. So there will be some sitting on top of the prestige ladder [deserved or undeserved] and those fighting over the middle. It is too bad that more people don’t realize that unless you go to one of the very top schools the prestige factor difference of your middle tier schools isn’t going to be worth worrying about and for the most part your degree will only function as the HR checkbox. If they realized that, they could go to whatever middle tier school made sense for them.

[There is a caveat. In a lot of locations around the country there is an upper middle tier school with excellent local networking. They will get a ‘meh’ from most people around the country or a vague feeling of recognition, but they will be great for you so long as you stay within the area. Think Georgetown in the DC area, USC for certain professions in the LA area, etc.]

Steven @7:
The rest, as long as they are accredited, would be perfectly serviceable places to get a degree from, or to work.

Having recently worked at a 4th tier institution, I have to disagree. It was a glorified diploma mill, useful only as a place where third tier professors lacking tenure could retire with ease and newly minted professors who didn’t aspire to much could pad their resumes. The undergrads were wholly under served, if not outright ignored, especially in the liberal arts. And this institution has a historical reputation as a Liberal Arts school. A few years ago, in order to stay solvent, they started a grad program focusing on the health professions. The university’s primary goal is to feed barely prepared workers into the bloated health care industry, emptying their pockets of loose change as they walk out the door.

Being so low in the standings (by any metric) they are useless. They don’t provide a useful education, nor do they contribute much in the way of research. All they do is siphon off excess money from wealthy idiots who couldn’t get into the nearby 2nd tier state schools. Then they flood the job market with barely qualified applicants who have been deluded into thinking they can compete with their counterparts from Fill-in-the blank ivy league school.

And don’t get me started on working at a place like this. All the problems of Academia with none of the perks. The place is run by career administrators who graduated from similar schools and so have no idea what a quality education even smells like.

Now, what next? Biannual rankings of philosophy departments, to see which ones philosophize better, and make better philosophers?.

Was your question rhetorical? Philosophical Gourmet Report has been doing exactly that, in more detail than you’d believe possible, for a decade or so.

NYU #1, Rutgers #2, Connectircut and Missouri tied for #50. All quantitative and objective, and reflexive and self-fulfilling too. And screw Penn State and the New School.

Reed deserves a lot of credit for being one of the few institutions to opt out of the US News BS.

I can think of few endeavors more pointless and noxious than the U.S. News rankings. Any information they provide is outweighed by the misconceptions they reinforce. All the institutions that agree to cooperate with U.S. News and furnish it with figures etc. are culpable.

In 1999, I began my studies as a law student at the University of Victoria, Canada, when David Cohen was the outgoing dean of Law there. He went on to be the dean of Pace Law right afterward. While he was dean at UVic Law, that school went quickly from a mid-ranked school to a top 3 school, primarily because of a jump in the Maclean’s rankings for UVic (the Macleans rankings being the Canadian equivalent of the US News rankings).

I interviewed him for the student newsletter as he was preparing to leave, and I raised the question of the Macleans rankings. While he didn’t openly admit to gaming the rankings, he did say that he “found their methodology extremely questionable and simplistic,” and stated that they probably didn’t actually reflect any substantive value. It was pretty clear to everyone that he had a lot to do with UVic’s Macleans position, and it wouldn’t surprise me in the least that he further refined his techniques of gaming the questionable methodologies at Pace.

My intake year was the first year after UVic reached #1, and many of my classmates had made an extra effort to go to UVic because of that ranking. But they were corporate-commercial types looking for a boost into Toronto firms, or possible Oil & Gas law in AB. UVic, being a left-leaning enviro-aboriginal school, was a very bad fit for them, and they threw a lot of tantrums as a result.

Since Cohen’s departure, UVic has found its natural place closer to the upper-middle of the pack, and while this is a bit galling to me as an alumni, it’s not too disappointing, since it means that more people will choose UVic for its faculty and specialties, as opposed to its sheer marketability.

“I’m surprised how little discussion there’s been at the ridiculous role the rankings play in many students’/parents’ college searches (this of course drives the system).”

Considering how many college and universities there are in the country, what should guide perspective students’/parents’ searches? 50 years ago, perhaps the recommendation of a teacher, counselor, etc. would be considered sufficient, but considering this is the second most expensive consumer good most families will ever buy (after a house), they should and do take the search very seriously, even if the information out there is not the most helpful.

Imagine if toothpaste cost $1,000 a tube, and was VERY IMPORTANT. I’m sure US News and its equivalent would start rankings, talk about which celebrities use which brand, what % of millionaires use product X or Y… would you spend more time thinking about which toothpaste YOU use?

It’s probably in large part about jockeying for position to get into the limited number of positions near the top of the heap. Some of the criteria look much more to do with social srtatus and networking than academic excellence.

Yet another convoluted causal loop arising from reification of metrics: the point of concultin these rankings looks as though it’s at least in part to assess the reputation of a college in the eyes of employers etc. – of which the rankings are a partly self-fulfilling indicator.

From the website of my alma mater, http://www.reed.edu/apply/news_and_articles/college_rankings.html

“Since 1995 Reed College has refused to participate in the U.S. News and World Report “best colleges” rankings….

“Reed College has actively questioned the methodology and usefulness of college rankings ever since the magazine’s best-colleges list first appeared in 1983, despite the fact that the issue ranked Reed among the top ten national liberal arts colleges. Reed’s concern intensified with disclosures in 1994 by the Wall Street Journal about institutions flagrantly manipulating data in order to move up in the rankings… This led Reed’s then-president Steven Koblik to inform the editors of U.S. News that he didn’t find their project credible, and that the college would not be returning any of their surveys…

“The year the college refused to submit data, the magazine arbitrarily assigned Reed the lowest possible in several categories and relegated the college to the lowest tier in its category, the most precipitous decline in the history of its ratings. The following year, responding to widespread criticism of its retribution, the magazine trumpeted Reed in its “best colleges” press release as being new to the “top tier” of national liberal arts schools. After that Reed was relegated to the “second tier” until this year when it was returned to the top tier in a tie for 47th place, even though the magazine’s sources rate the college’s academic reputation as high or higher than half of the top-ranked schools…”

I recall reading about another problem – those producing the report are under pressure to shuffle the rankings about, to make the annual issue newsworthy.

In an environment where you could do a pretty good ranking of the top 100 or so using the algorithm (Ranking 2012 = ranking 1900), this is likely to produce a very low signal/noise ratio