The US News College Rankings Scam

by Henry on February 8, 2012

Stephen Budiansky, via Cosma Shalizi’s Pinboard feed.

Back in ancient times when I worked at esteemed weekly newsmagazine U.S. News & World Report, I always loathed the annual college rankings report. Like all cash cows, however, the college guide was a sacred cow, so I just shut up about its obvious statistical absurdities and inherent mendacity. As a lesson in the evils of our times, it is perhaps inevitable that the college guide is now the only thing left of U.S. News.
A story in today’s New York Times reports that Claremont McKenna college has now been caught red handed submitting phony data to the college guide to boost its rankings. But the real scandal, as usual, is not the occasional flagrant instance of outright dishonesty but the routine corruption that is shot through the whole thing. … To increase selectivity (one of the statistics that go into U.S. News’s secret mumbo-jumbo formula to produce an overall ranking), many colleges deliberately encourage applications from students who don’t have a prayer of getting in. To increase average SAT scores, colleges offer huge scholarships to un-needy but high scoring applicants to lure them to attend their institution. (The Times story mentioned that other colleges have been offering payments to admitted students to retake the test to increase the school average.)
… One of my favorite bits of absurdity was what a friend on the faculty at Case Law School told me they were doing a few years ago: because one of the U.S. News data points was the percentage of graduates employed in their field, the law school simply hired any recent graduate who could not get a job at a law firm and put him to work in the library. Their other tactic was pure genius: the law school hired as adjunct professors local alumni who already had lucrative careers (thereby increasing the faculty-student ratio, a key U.S. News statistic used in determining ranking), paid them exorbitant salaries they did not need (thereby increasing average faculty salary, another U.S. News data point), then made it understood that since they did not really need all that money they were expected to donate it all back to the school (thereby increasing the alumni giving rate, another U.S. News data point): three birds with one stone! (I gather the new Case law dean has put an end to these shenanigans.)

Worth reading the whole thing (even though Budiansky’s site has one of those annoying and anti-social ‘if you cut and paste text from my site, you will get unasked for cruft about how you ought to click on the original link added to your pasted text’ installations).

{ 36 comments }

1

understudy 02.08.12 at 2:36 pm

I think he missed the point.

Think American Football – only one team wins the Super Bowl and it isnt necessarily the best team, but it all that matters. College rankings are similar – only one top 10 list in USNews, regardless of what the “real, statistically valid” measurement would say. The difference between Pomona and Claremont is what, exactly and why is it relevant to whom?

The real shame about this is college administrators, faculty and students have sat by and let excellence and ranking by determined by US News and have not done the more challenging and honest assessment of asking “are we doing a good job”? And the follow-up “Can we do a better job”? Since that might upset the applecart, it is easier just to live-and-die by US News … I know when I advise students applying to college, prestige is an almost impossible barrier to overcome when selecting a school – the only thing that I’ve seen that can do it is $$$…

2

straightwood 02.08.12 at 3:05 pm

After most of the American private universities price themselves out of existence, these ratings will have a certain nostalgia value, rather like comparing the virtues of Packard and Duesenberg automobiles.

3

Don A in Pennsyltucky 02.08.12 at 3:16 pm

One of the nicer things about Google Reader is that it leaves those annoying extras off so you can cut and paste without being called a pirate (Arrrr!). Not actually being a pirate, I don’t need the reminder to add the URL along with the fragment of text I thought worth sharing.

4

r. laughlin 02.08.12 at 4:21 pm

you can opt out of that Tynt scruft [the Copy tracking] by going to:

http://www.tynt.com/tynt-users-opt-out

5

Andrew Fisher 02.08.12 at 4:23 pm

Understudy@1 ‘The real shame about this is college administrators, faculty and students have sat by and let excellence and ranking by determined by US News and have not done the more challenging and honest assessment of asking “are we doing a good job”?’

My experience is in the UK rather than the US, where we have a slightly different set of newspaper league tables. FWIW, that experience is that managers [administrators], academics [faculty] and governors [trustees], far from having ‘sat by’, are the most eager customers for rankings because they have so completely internalised the hierarchy that the ranking expresses. Some students also care.

They will need mutliple copies of Introduction to Statistics in Hell. Or perhaps that too will be part of the punishment.

6

Jerry Vinokurov 02.08.12 at 4:42 pm

Not only do the rankings not measure anything relevant to actual education, but those irrelevant figures are easily gamed. Wonderful.

7

Steven 02.08.12 at 4:51 pm

Well they have to be better than the revealed preference rankings some professors crazily tried to institute a few years ago.

I long for the time when there were no rankings, just the patronage of the Ivy League, Seven Sisters and small private New England colleges by the nation’s WASP plutocrats and cultural elite. Distinctions of reputation, quality and character were not voiced in publications any person on the street could purchase for a few dollars, but instead made in hushed tones in wood-paneled rooms on Fifth Avenue and Beacon Hill. The Scotch tasted better back then, too.

Now, what next? Biannual rankings of philosophy departments, to see which ones philosophize better, and make better philosophers?

In theory, I am not against the idea of rankings. Some things are better than other, like things in way that we can articulate and that have meaning to a wider audience. Brown is a better school than Claremont McKenna in important ways that make sense, and not just because one of them apparently has an administration full of liars. But the idea of an individually enumerated top twenty, or fifty, is ludicrous. Instead of correcting wider ignorance about higher educations, these types of rankings perpetuate them.

On the other hand, I think there would be surprisingly widespread agreement about which 15 colleges and universities are the top in the nation, which 25 follow in the second tier, and then probably which 50 are in the third, at least as far as undergraduate education is concerned. Judgments at the margin would be difficult, but that is not a reason not to make them. The rest, as long as they are accredited, would be perfectly serviceable places to get a degree from, or to work.

8

DCA 02.08.12 at 4:52 pm

Worth noting that this has metatisized beyond college; my local hospital’s “listen while you wait” phone message includes the information that three of their doctors are in the top
something-or-other of the “US News Top Docs” list.

9

Sam Clark 02.08.12 at 4:53 pm

My (also UK-based) experience, unlike Andrew Fisher’s, is that most university staff haven’t internalised the rankings’ hierarchy, but feel ourselves powerless against it. Many therefore think that gaming rankings as far as possible is a tactical rather than a moral problem: after all, since the rankings are always going to be nonsense, what’s wrong with influencing the nonsense to our own survival and benefit, especially since others are going to be doing the same?

That’s orthogonal to Understudy’s point about thinking about whether we’re actually doing a good job, and how to do a better one. Again in my experience, many of us are deeply concerned about this, but don’t think that the rankings have much to do with it.

10

Jerry Vinokurov 02.08.12 at 4:56 pm

Funny that Brown got mentioned; having actually attended it (as a graduate student) I wouldn’t say that the quality of education you would get there is necessarily superior to the quality of education at most other respectable institutions, including large state schools. There are other advantages to attending Brown, but they have little to do with the overall “quality” of education you would receive there.

11

Sam Clark 02.08.12 at 5:09 pm

Rereading myself at 9, I realise that I come across as saying ‘hey, I game the rankings all the time, what’s the problem?’, which is not actually what I meant. I don’t have the power to do that. I meant: I understand why e.g. Case Law School got to the point described in the OP.

12

Alex 02.08.12 at 5:10 pm

because one of the U.S. News data points was the percentage of graduates employed in their field, the law school simply hired any recent graduate who could not get a job at a law firm and put him to work in the library.

There’s a movie in this; today’s Mann ohne Eigenschaften. At the very least, a Doonesbury subplot.

13

Doug K 02.08.12 at 5:14 pm

in the endeavours where I earn my daily crust, a pernicious form of rankings/metrics known as KPIs (Key Performance Indicators) is venerated. Unfortunately it appears that any KPI or set of KPIs can be maximized in an unproductive way. Management prefers not to know this.
This is only another way of repeating Goodhart’s law,
any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

Presumably it is the aspect of control which produces the reality-distortion field around the metrics. It is a pathological type of gamification – careful design can produce Foldit, profit-maximizing strategies conflated with obfuscated metrics gets you the US News rankings and related shenanigans. I prefer the Washington Monthly rankings.

In related analysis, a careful disassembly of Boston Magazine’s dissembled school rankings is at furialog. It is a thing of beauty.

14

adam.smith (was Sebastian(1)) 02.08.12 at 5:15 pm

So is it unethical to try to game a stupid ranking? And what type of gaming is unethical? Personally, I’m inclined to give Case Law credit (or at least no grief) for what they did.
I feel though you need to be honest in how you game the system – just lying about test scores is, well, cheating.

15

jim 02.08.12 at 5:31 pm

Steven @7: I think there would be surprisingly widespread agreement about which 15 colleges and universities are the top in the nation, which 25 follow in the second tier, and then probably which 50 are in the third, at least as far as undergraduate education is concerned.

Probably. But that’s not the value of rankings. There are on the order of 2,000 four-year undergraduate institutions in this country. 99.997% of applicants aren’t going to even apply to the top 15. What is needed is some unbiased guidance distinguishing between the schools that aren’t household names. Is Mary Baldwin better or worse than Mary Washington? In what ways?

The standard college guides don’t attempt to judge institutions. US News at least tries, even if it does a terrible job. Someone ought to try to do better.

16

David Hobby 02.08.12 at 5:36 pm

First, I’ve always been a fan of Stephen Budiansky. He doesn’t post that often, but always makes good sense.

Let’s be honest, the U.S. News rankings are more about prestige than educational quality. Rankings of educational quality would necessarily be (sometimes strongly) dependent on major.

17

Sebastian 02.08.12 at 5:55 pm

“Let’s be honest, the U.S. News rankings are more about prestige than educational quality. Rankings of educational quality would necessarily be (sometimes strongly) dependent on major.”

And if prestige is one of the most important parts of having a college degree, they’re ranking the important thing.

Now I’m willing to agree that prestige ought not be one of the most important things, but I strongly suspect for most practical purposes that is how a college degree is used. In places I’ve worked, college degrees were important for only three major reasons. First, it fulfilled a listed Human Resources requirement for a job. In that case pretty much any college would do. Second, it fulfilled a Human Resources ass covering requirement. In that case prestige was important (i.e. we hired from Harvard, who could have known the guy didn’t know anything useful). Third, it fulfilled a networking requirement. In that case the prestige was vital.

The concept of “we know that such and such a college has a particularly good program that is well suited for exactly the job under consideration” was such a distant fourth place that I think I’ve only seen it twice. [Though I’ll admit that it *may* be more common in technical fields].

Prestige is a relative concept. So there will be some sitting on top of the prestige ladder [deserved or undeserved] and those fighting over the middle. It is too bad that more people don’t realize that unless you go to one of the very top schools the prestige factor difference of your middle tier schools isn’t going to be worth worrying about and for the most part your degree will only function as the HR checkbox. If they realized that, they could go to whatever middle tier school made sense for them.

[There is a caveat. In a lot of locations around the country there is an upper middle tier school with excellent local networking. They will get a ‘meh’ from most people around the country or a vague feeling of recognition, but they will be great for you so long as you stay within the area. Think Georgetown in the DC area, USC for certain professions in the LA area, etc.]

18

Jerry Vinokurov 02.08.12 at 6:05 pm

And if prestige is one of the most important parts of having a college degree, they’re ranking the important thing.

But then just say so. It’s dishonest to pretend you’re ranking something you call educational quality when a) your rankings don’t measure any such thing, and b) to the extent that they measure anything, it’s virtually all prestige-oriented. The honest (but sadly unprofitable) thing to do would be to say, “Hey, Harvard, now that’s a name!” and end it right there.

19

Keith 02.08.12 at 6:25 pm

Steven @7:
The rest, as long as they are accredited, would be perfectly serviceable places to get a degree from, or to work.

Having recently worked at a 4th tier institution, I have to disagree. It was a glorified diploma mill, useful only as a place where third tier professors lacking tenure could retire with ease and newly minted professors who didn’t aspire to much could pad their resumes. The undergrads were wholly under served, if not outright ignored, especially in the liberal arts. And this institution has a historical reputation as a Liberal Arts school. A few years ago, in order to stay solvent, they started a grad program focusing on the health professions. The university’s primary goal is to feed barely prepared workers into the bloated health care industry, emptying their pockets of loose change as they walk out the door.

Being so low in the standings (by any metric) they are useless. They don’t provide a useful education, nor do they contribute much in the way of research. All they do is siphon off excess money from wealthy idiots who couldn’t get into the nearby 2nd tier state schools. Then they flood the job market with barely qualified applicants who have been deluded into thinking they can compete with their counterparts from Fill-in-the blank ivy league school.

And don’t get me started on working at a place like this. All the problems of Academia with none of the perks. The place is run by career administrators who graduated from similar schools and so have no idea what a quality education even smells like.

20

Steven 02.08.12 at 7:35 pm

“Funny that Brown got mentioned; having actually attended it (as a graduate student) I wouldn’t say that the quality of education you would get there is necessarily superior to the quality of education at most other respectable institutions, including large state schools.” -Brown is apparently good at producing modest graduates, among other things. This “false modesty factor” should count in the rankings.

Kieth (19)- You are probably right. I was being amicable, being unable to paint with a broad brush once you get past the first 100 or so colleges and universities. It would have been better to have remained silent.

I deliberately declined the idea of ranking below a third tier (i.e., below about 100 schools, and even then this first 100 only in broad tiers themselves) because it is indeed impossible to distinguish the Marys, Baldwin and Washington, in some important way. That’s what campus visits, alumni interviews, and the rest should be about. If you’re below the third tier, just find a school you can afford and that seems appealing to you, for whatever personal goals you have in mind.

Alas, we should just get it over with and rank the schools by how much money their graduates have made, mean and median, at the end of each decade of their lives after graduation. Or maybe by those figures, minus what their parents made at the same point in thier lives, adjusted to today’s dollars. That is the list everyone is really after anyway.

21

Antonio Conselheiro 02.08.12 at 8:13 pm

Now, what next? Biannual rankings of philosophy departments, to see which ones philosophize better, and make better philosophers?.

Was your question rhetorical? Philosophical Gourmet Report has been doing exactly that, in more detail than you’d believe possible, for a decade or so.

NYU #1, Rutgers #2, Connectircut and Missouri tied for #50. All quantitative and objective, and reflexive and self-fulfilling too. And screw Penn State and the New School.

22

Bruce McCulley 02.08.12 at 9:49 pm

Out of idle curiosity, I checked to see how my old school, Eckerd College, fared in the rankings, and found it tied with six others at #144. I was, like, totally grossed out.

23

ben w 02.08.12 at 10:24 pm

Reed deserves a lot of credit for being one of the few institutions to opt out of the US News BS.

24

Jerry Vinokurov 02.09.12 at 1:23 am

Brown is apparently good at producing modest graduates, among other things. This “false modesty factor” should count in the rankings.

I’m not sure where you picked up any false modesty in anything I said. I was being completely serious in my (admittedly anecdotal) observations of the school.

My larger point is that what dictates “quality of education” depends on many things; some of those things, like for example the quality of equipment one uses in the physics lab, are areas where a school like Brown has an advantage. Just to lay my cards on the table, Berkeley is my undergrad alma mater. I got an excellent education there, but the equipment that we used in our experiments was either a) decades old, b) broken, or c) both. This was not fun! When I came to Brown and got a look at the undergrad labs, I was impressed by the fact that they were using reasonably up-to-date equipment.

But even taking all that into account, knowing how to use one piece of equipment isn’t going to make that large of a difference in one’s educational experience (and in any case you’ll learn that on the job or in grad school or wherever the need calls for it). I’d be hard-pressed to say that one’s, say, physics education (because that’s what I know) would be dramatically different between Brown or Berkeley or some other reasonably respectable institution. I don’t see any reason why Reed, for example, wouldn’t have just as good of an undergraduate physics (or, really, any science for that matter) program as anyplace else. After all, you’re doing mostly the same work out of the same textbooks (indeed, if you go to Reed, you might take a class with David Griffiths himself, for those to whom that name means something) so that’s not going to affect things too much.

In my (again, anecdotal) experience, the things that really affect education are not made-up metrics like “percentage of alumni giving” (when I was a grad student Berkeley called asking me for $1000; I laughed at which point the poor student on the other end of the line lowered the request to $25) but rather institutional strictures. For example, the thing that was most frustrating about my undergraduate experience was the near-complete absence of any sort of sensible advising (one area where Brown would almost certainly be better than almost any state school) and constant red tape to overcome to get what I wanted. I think that anyone who cares to get the most education out of their college experience should care about things like that, and those things are going to be highly context-dependent and not necessarily susceptible to simple measurements that you can compile from the university’s “About Us” web page.

25

LFC 02.09.12 at 2:18 am

I can think of few endeavors more pointless and noxious than the U.S. News rankings. Any information they provide is outweighed by the misconceptions they reinforce. All the institutions that agree to cooperate with U.S. News and furnish it with figures etc. are culpable.

26

ben w 02.09.12 at 2:31 am

My alma mater’s letters to me asking for money occasionally mention the importance of having a high degree of alumni “participation” (without ever saying that, specifically, they mean “participation in giving money to us”), because, of course, that’s a factor in the US News rankings—something that does quite little to make me want to give them money.

27

anonymous for this purpose 02.09.12 at 4:30 am

In 1999, I began my studies as a law student at the University of Victoria, Canada, when David Cohen was the outgoing dean of Law there. He went on to be the dean of Pace Law right afterward. While he was dean at UVic Law, that school went quickly from a mid-ranked school to a top 3 school, primarily because of a jump in the Maclean’s rankings for UVic (the Macleans rankings being the Canadian equivalent of the US News rankings).

I interviewed him for the student newsletter as he was preparing to leave, and I raised the question of the Macleans rankings. While he didn’t openly admit to gaming the rankings, he did say that he “found their methodology extremely questionable and simplistic,” and stated that they probably didn’t actually reflect any substantive value. It was pretty clear to everyone that he had a lot to do with UVic’s Macleans position, and it wouldn’t surprise me in the least that he further refined his techniques of gaming the questionable methodologies at Pace.

My intake year was the first year after UVic reached #1, and many of my classmates had made an extra effort to go to UVic because of that ranking. But they were corporate-commercial types looking for a boost into Toronto firms, or possible Oil & Gas law in AB. UVic, being a left-leaning enviro-aboriginal school, was a very bad fit for them, and they threw a lot of tantrums as a result.

Since Cohen’s departure, UVic has found its natural place closer to the upper-middle of the pack, and while this is a bit galling to me as an alumni, it’s not too disappointing, since it means that more people will choose UVic for its faculty and specialties, as opposed to its sheer marketability.

28

Ben Alpers 02.09.12 at 2:09 pm

I think a lot of academics and administrators simultaneously feel that the rankings are important but don’t actually mean much. It’s an attitude that positively encourages gaming the system. And I guess I agree with those upthread who’d distinguish gaming from simply cheating…and that gaming should be judged on the basis of its other costs and benefits to the university. Creating more small classes to improve one’s US News rankings, e.g., might positively benefit students, even if the principal purpose of the move is gaming the system.

Of course the rankings measure prestige, and insofar as people take them seriously, the rankings do so entirely tautologically, so the particular absurdities of the algorithm are irrelevant to the system “working” or not.

I’m surprised how little discussion there’s been at the ridiculous role the rankings play in many students’/parents’ college searches (this of course drives the system).

But optimally we’d all be better off if colleges and universities simply refused to take part, which is the only way I could imagine them ceasing to matter. The only way to win is not to play.

29

understudy 02.09.12 at 2:24 pm

“I’m surprised how little discussion there’s been at the ridiculous role the rankings play in many students’/parents’ college searches (this of course drives the system).”

Considering how many college and universities there are in the country, what should guide perspective students’/parents’ searches? 50 years ago, perhaps the recommendation of a teacher, counselor, etc. would be considered sufficient, but considering this is the second most expensive consumer good most families will ever buy (after a house), they should and do take the search very seriously, even if the information out there is not the most helpful.

Imagine if toothpaste cost $1,000 a tube, and was VERY IMPORTANT. I’m sure US News and its equivalent would start rankings, talk about which celebrities use which brand, what % of millionaires use product X or Y… would you spend more time thinking about which toothpaste YOU use?

30

Ben Alpers 02.09.12 at 2:58 pm

Imagine if toothpaste cost $1,000 a tube, and was VERY IMPORTANT. I’m sure US News and its equivalent would start rankings, talk about which celebrities use which brand, what % of millionaires use product X or Y… would you spend more time thinking about which toothpaste YOU use?

But education isn’t toothpaste. It’s not outside the realm of possibility that there is a single best toothpaste (though surely subjective factors like taste would play a role in one’s choice). But individuals educational needs and desires differ incredibly. The problem is not just that US News ranks institutions incorrectly. The problem is that the very notion of such a one-dimensional ranked list of schools is ridiculous.

So what should parents of college-age kids (a group that will include me in four years) do? Abandon the hope of finding the single best school for their child (there probably is no such thing). If one’s goal is simply to find a school that will serve your child well, the huge number of possibilities makes the task easier. See the incredible range of choices and the easy access to a vast amount of information — good and bad — about colleges and universities as a feature, not a bug. Finally, realize that if you have the means to actually take advantage of these choices you are part of an increasingly small sector of our society and be grateful for that fact. (I think this sort of thing is what the #firstworldproblems hashtag was created for.)

31

Tim Wilkinson 02.09.12 at 3:15 pm

It’s probably in large part about jockeying for position to get into the limited number of positions near the top of the heap. Some of the criteria look much more to do with social srtatus and networking than academic excellence.

Yet another convoluted causal loop arising from reification of metrics: the point of concultin these rankings looks as though it’s at least in part to assess the reputation of a college in the eyes of employers etc. – of which the rankings are a partly self-fulfilling indicator.

32

Tim Wilkinson 02.09.12 at 3:24 pm

Sorry, hadn’t read all previous comments

33

Bloix 02.09.12 at 7:03 pm

From the website of my alma mater, http://www.reed.edu/apply/news_and_articles/college_rankings.html

“Since 1995 Reed College has refused to participate in the U.S. News and World Report “best colleges” rankings….

“Reed College has actively questioned the methodology and usefulness of college rankings ever since the magazine’s best-colleges list first appeared in 1983, despite the fact that the issue ranked Reed among the top ten national liberal arts colleges. Reed’s concern intensified with disclosures in 1994 by the Wall Street Journal about institutions flagrantly manipulating data in order to move up in the rankings… This led Reed’s then-president Steven Koblik to inform the editors of U.S. News that he didn’t find their project credible, and that the college would not be returning any of their surveys…

“The year the college refused to submit data, the magazine arbitrarily assigned Reed the lowest possible in several categories and relegated the college to the lowest tier in its category, the most precipitous decline in the history of its ratings. The following year, responding to widespread criticism of its retribution, the magazine trumpeted Reed in its “best colleges” press release as being new to the “top tier” of national liberal arts schools. After that Reed was relegated to the “second tier” until this year when it was returned to the top tier in a tie for 47th place, even though the magazine’s sources rate the college’s academic reputation as high or higher than half of the top-ranked schools…”

34

David 02.10.12 at 3:16 am

The annual Playboy survey of best party schools was always inherently more honest.

35

John Quiggin 02.10.12 at 3:37 am

I recall reading about another problem – those producing the report are under pressure to shuffle the rankings about, to make the annual issue newsworthy.

In an environment where you could do a pretty good ranking of the top 100 or so using the algorithm (Ranking 2012 = ranking 1900), this is likely to produce a very low signal/noise ratio

36

Tom M 02.10.12 at 12:03 pm

While researching colleges with our daughter, I came across an article (which I cannot find now) by a woman who worked in admissions for Duke. Her self-described job was to travel the Midwestern US visiting high schools to encourage applications to the school which she knew would be turned down so as to boost the “selectivity” factor in the US News rankings.
Our daughter attended Dickinson College in part because the president, William Durden, seemed so outspoken in opposition to such metrics. The school is one member of the Annapolis Group which organized now 130 liberal arts colleges to come up with a sort of counter programming.
Any idea if that group is still around and active?

Comments on this entry are closed.