Burton Weisbrod and Evelyn Asche have an entirely sensible oped in yesterday’s IHE on the fuss about Clemson University’s shocking attempts to game the US News and World Report rankings:
Watt, according to reports, literally drew gasps from her audience when she revealed that when Clemson administrators fill out U.S. News reputational rankings survey, they rate other universities lower than Clemson across the board. Why not? Reputation accounts for fully 25 percent of a school’s ranking score. Watt’s statement that she was confident that other colleges do the same is perfectly plausible.
Inside Higher Ed reported Monday that the University of Southern California inflated the number of National Academy of Engineering members on its full-time, tenure-track faculty. Because the number of NAE faculty is a criterion for U.S. News rankings, USC has good reason to include NAE faculty who are not full-time or tenure-track.
If we step back from higher education, we will see the same dynamic of gaming a performance measurement system in many other spheres. Hospitals receive “report cards” that measure their performance in many areas, including their mortality rates. A little thought reveals the easiest way to improve the mortality rate is to keep terminally ill patients from being admitted to the hospital in the first place or discharge them prior to death. In fact these events do occur. Nonprofit hospitals receive large tax exemptions but are expected to provide charity care to indigent patients in return. Their substantial tax benefits are currently being scrutinized in the courts and in Congress, so hospitals are certainly scrambling to alter their accounting procedures to increase their charity care levels
Academics based in the UK can correct me, but it seemed to me when I was there that both the RAE process and the system of for evaluation the quality of quality control of teaching were designed to be gamed; US News and World Report has to be aware of the many ways that Universities and some of their component parts (those I’m especially aware of are Business and Law Schools), game their rankings: a narrative account of the ways they suspect the rankings are being gamed might go some way to discouraging the most egregious and visible tricks. (I don’t really see how to eliminate the incentive artificially to undermine one’s close rivals when ranking them).
{ 29 comments }
Tracy W 06.10.09 at 1:07 pm
In defence of government bureaucrats’ ranking systems:
1. Coming up with a system of performance evaluation that can’t be gamed is incredibly hard.
2. Doing without a transparent system of performance evaluation opens bureaucrats up to all sorts of accusations of political knavery, from personal favouritism to racism to political correctness and probably beyond.
3. And in health care, pressure for spending keeps rising apparently endlessly, placing pressure on the rest of the government’s budget, which creates political pressure to try to find efficiencies.
US News and World Report could more plausibly just not offer a college ranking service, but on the other hand with the vast numbers of colleges in the USA I can see the demand for it, even if the ranking system is poor.
John 06.10.09 at 1:08 pm
While they would never admit it, I wonder if US News assumes that if everybody attempts to game the system, the end result is a wash.
Tom West 06.10.09 at 1:53 pm
We all use metrics of some sort to guide our decisions, and they’re all subject to being gamed. If I speak out a little more in meetings when the boss is present, I’m attempting to game his internal metrics on his employees and hopefully keep my job when the next round of layoffs occur.
So, of course metrics will be gamed. The real question is not “are metrics perfect”, but “overall, are we making better decisions with these metrics than we would without them”. On that, I think the answer is *almost* always ‘yes’.
Chris Bertram 06.10.09 at 2:16 pm
There are two sorts of assessment here, one of which is useful and the other of which is usually quite distorting and damaging. Tom West’s comment doesn’t distinguish.
The first asks whether what we do around here is up to scratch, both absolutely and comparatively. Is our research solid? Is our teaching sound? Comparative information is important here, just so that we know that our own view of the matter isn’t wildly out of whack with what everyone else thinks. (Of course, we may come to the conclusion that everyone else is wrong, after thinking things through …. )
The second asks whether we are in the top N of institutions, sets as an explicit goal to be in that top N, and chases funding geared to rank order. That sort of assessment can be really damaging, since all the other goals that we might have get subordinated to this central one. Also, the goal itself gets transformed into a drive to score highly on the ranking rather than to score highly in what the ranking purports to be a ranking of. UK universities are (alleged by their critics) to subordinate teaching to research. That might be sort-of true. But what is actually true (allegedly) is that they chase what they take RAE panels to be valuing, and that they chase high rankings in the National Student Survey (partly by means of impression management) rather than trying to ensure that the teaching is what it should be.
deliasmith 06.10.09 at 3:22 pm
This is surely a Gerald Ratner event
Salient 06.10.09 at 3:45 pm
Also, the goal itself gets transformed into a drive to score highly on the ranking rather than to score highly in what the ranking purports to be a ranking of.
Chris,
Coincidentally I found myself in a fair bit of social-pressure trouble yesterday for stating essentially the content of your third paragraph to a colleague within earshot of an administrator. (The U.S. university I’m located in has a regionwide reputation for bloviation about its infeasible goal to be a Top N university by Year X while cutting overall expenditure by p% per year.)
The dressing-down admonishment I received from said administrator bordered on incoherence, but was essentially: “One may reasonably define Best Practices as ‘that which is done at the Top 25 Universities.’ Therefore, one ought to emulate Top 25 universities wherever possible, presumably so that one may tout one’s resemblance to a Top 25 university. Practices which are not to be found in Top 25 universities are not likely to be Best Practices, because otherwise the Top 25 universities would be using those practices. QED.”
Might as well call this the Late Adopter Thesis — do what’s tried and true. I’d like to hear your thoughts on it. I can see problems with the model, but… specifically, is this viewpoint a satisfactory answer to the statement I quoted up top?
Chris Bertram 06.10.09 at 3:56 pm
My experience suggests that administrators are highly selective and opportunistic when it comes to adopting characteristics of the top N universities. The drive to present themselves as in the top-whatever percentile involves “branding” themselves in ostentatious ways (which genuine top institutions don’t need to do) and imposing serious top-down management on academics (which Harvard or Oxford don’t do). Larry Summers (at least qua university boss) could only dream of the power of a “Howard Newby”:http://en.wikipedia.org/wiki/Howard_Newby .
Steve LaBonne 06.10.09 at 4:06 pm
Salient- sounds like another data point to support my long-held suspicion that there is a maximum IQ requirement (somewhere in the double digits) for academic administrators. ;)
Salient 06.10.09 at 4:41 pm
sounds like another data point to support my long-held suspicion that there is a maximum IQ requirement (somewhere in the double digits) for academic administrators.
Poking fun aside, I suspect it’s more of an institutional constraint than any personal lack of acuity. From what I’ve seen there’s a need to ‘network’ and form alliances that I would find incomprehensible to navigate: the requirement that one must know, and be favorably known by, so many people. Most of whom are, in my thus-far-limited experience, actually quite contemplative and clever.
The institutional constraint’s quite simply the predominant need for a Good Face. This makes blunt honesty impossible in any public forum or potentially public forum, and it’s an automatic consequence of having administrative control. This political criterion has the same sorts of policy consequences that an inability to say “This United States has acted shamefully” has on detention policy.
It’s this political constraint which I’d find less incomprehensible than suffocating: in a very real sense, I suspect that I have insufficient IQ to be an administrator! (Where IQ is taken to mean ‘institutional quiescence,’ of course.)
You and I and etc. can assert our perspective bluntly, and explicitly state what we see as clear problems with an institution, because we don’t have to worry about (1) attempting to reweave the dense social fabric to lay enough anticipatory foundation to ensure that proposing something controversial will be socially acceptable, so that the proposal isn’t DOA, and (2) we’re not official spokespersons, so we don’t have to worry about various local/state media reporting that Salient trashes U. of X: calls the Top 25 by 2025 plan “infeasible” and “silly” in tomorrow’s papers and evening TV.
Tom West 06.10.09 at 4:58 pm
#4, I don’t see the difference in how the assessment is done, only the purpose to which the metric is put (ranking or pass/fail).
If you are selecting only a few of the number (which university do I attend, which employees do I let go), then a ranking seems to me inevitable. People desperately want one, so they’re going to get one (or invent one).
What a ranking *does* mean is that you never escape the metric, i.e. you can never say you are “good enough” and thus there will always be pressure to game the system (or as the measurers would prefer to believe, pressure to improve).
Steve LaBonne 06.10.09 at 5:04 pm
The problem, of course, is that what’s functional for an individual’s climb up the greasy poll can be highly dysfunctional for said individual ‘s institution, which will more often than not be in crying need of blunt assessments of what it should be doing better and how (as opposed to how things look.) So (to be in all likelihood very unfair to what you meant) I would tend to be a little less complacent about the real value and necessity of those political constraints. After all, the system that appears to require those constraints was set up by exactly the kinds of people who function “well” (in a self-serving way) within them.
Steve LaBonne 06.10.09 at 5:05 pm
Not to mention the greasy pole. I will use preview, I will use preview, I will use preview…
Holmes 06.10.09 at 5:15 pm
Certainly designing a system that cannot be gamed is very difficult. However, in the case of the US-News university rankings, it seems that there are easy ways to make the system harder to game.
The proportion of classes with at most 20 students is manipulated by distributing students unevenly across sections. The mean class size, or size-biased mean (i.e.\ mean from the point of view of an average student) are much harder to manipulate. Specifying detailed guidelines on what components of salary can be included in average salary reports can also help.
More interesting though, re evaluation of rival universities, what would be the effect of just publishing more of the raw data? Would it make a difference if the filled (anonymized?) questionnaires were available online? At the very least it would make extreme forms of such manipulations public knowledge.
Salient 06.10.09 at 5:15 pm
I would tend to be a little less complacent about the real value and necessity of those political constraints.
Well yes, and we could be less than complacent about the real value and necessity of helium in our atmosphere. An administrator’s preoccupation with how one’s institution looks, perhaps to the detriment of how one’s institution operates, strikes me as an inherent property of administration.
A sensible solution IMO is to sustain demands for enough transparency, access to information, etc. to optimize independent assessors’ ability to call an institution to task for problematic or destructive policy. We can’t reasonably expect bluntly honest assessment from administrators, but we can try to influence the extent to which bad policy results in bad face, and good policy results in good face.
[Also: Preview? What preview? I thought Preview vanished right along with access to the topical archives.]
Tom West 06.10.09 at 5:18 pm
I have to agree with Salient, that being an administrator is a tricky game. The fundamental dissonance is that if the population as a whole believed that universities were exactly what those teaching in them believed they should be, they’d defund universities entirely and demand something completely different.
So an adminstrator’s job is to reconcile what those involved in university life want the university to be with what the voters are actually willing to pay for, mostly by lying to all sides :-). Certainly not a job I’d take in a million years…
David Wright 06.10.09 at 5:21 pm
Reading the actual accusations, I find it hard to be very scandalized by them.
The University favored reducing a class of 20 by 1 over increasing a class of 40 by one? The manginal utility change for the students probably favors the same move.
The Univeristy included benefits in compensation numbers, as U.S. News asked them to? And they will also report disaggregated numbers, if that’s what you want to know? Woop-te-do.
They changed their admission process to accept students with higher SAT scores, students move likely to be in the top of their class, and students more likely to graduate? This seems to be what really irks their accusers — that Clemson moved away from being a school for everyone and toward being a school for the elite. But being elite is precisly what the rankings are supposed to measure — doing this isn’t “gaming,” its purposely changing the nature of your school. You’re welcome to believe that they shouldn’t have made that choice, but making that choice isn’t “cheating”.
Salient 06.10.09 at 6:09 pm
But being elite is precisly what the rankings are supposed to measure—doing this isn’t “gaming,†its purposely changing the nature of your school.
Observing this dissonance between being “elite” and being “the best” also got me in aforementioned administrator’s ungood graces: Does this state need an elite Top 25 school, or does that goal actually conflict with what would best serve the state?
[Lessons learned: (1) never suggest to an administrator that the conventional metric for determining “the best” might not help one determine “what’s best for [the state],” because this will only be taken as an insult to the state; and (2) never interpret a dressing-down as an opportunity to converse…]
Tom West 06.10.09 at 7:07 pm
Salient, thanks for (2) – the chuckle was appreciated.
On the other hand, I wouldn’t want to be the administrator who tried to explain to the funding politicians in 25 words or less why a state’s university shouldn’t be part of the elite (if he in fact got the full 25 words before he got his funding slashed in favour of the others who were ‘trying’).
Really, the only way out of the metric mess is to get a high profile group or individual to start your own metric that weighs various factors the way you’d like them weighed. In general, any metric wins against “we’re all doing are best”. But with competing metrics, it’s usually the prestige of the sponsoring organization that determines which metric gets the attention and which gets ignored. The only other way is to have so many metrics that they *all* get ignored.
Steve LaBonne 06.10.09 at 7:17 pm
Actually, many legislators would be actively HOSTILE to that goal- they (rather sensibly) want the state university to fulfill its intended function of educating ordinary citizens rather than diverting resources to competing in academic penis-measuring competitions. It’s the faculty and administrators who really want this, not the folks who pay the bills. In my state (Ohio) the total cost to middle-class families, AND average debt load to the student, of sending a kid to a state university (I have to know these things, mine is just finishing 11th grade) is typically MORE than to a good private school especially given the near-impossibility of graduating from a state school in 4 years. In effect my taxes are subsidizing rich families who won’t get financial aid at any school and therefore will be out of pocket far less at a state school than at a private one. I don’t especially like that, and I’m not inclined to pay higher taxes just to give Ohio State more bragging rights in stupid university rankings.
rea 06.10.09 at 9:42 pm
I don’t see how USN&WR has any incentive to have a rankings system that isn’t routinely gamed. After all, if the magazine ranks Clemson ahead of, say, Louisiana State, there is no objective way to falsify that result. Nobody will know the difference, and people will still buy the magazine.
Tom Hurka 06.10.09 at 9:46 pm
Elaborating on something at the end of #16:
University rankings like in U.S. News and World Report or (in Canada) Macleans are done from a potential student’s point of view: *assuming you got in*, what would be the best university for you to go to? From that point of view a small, highly selective student body is desirable. You get small classes, high prestige for your degree, etc. That other people aren’t getting those things doesn’t matter. It’s assumed that *you* got in.
But ranking universities from a population’s or a government’s point of view is very different. Then it may be and surely is desirable for a university to admit a large number of students: then it’s serving its community better, educating more of its members, etc.
What’s puzzling is why populations and governments think rankings done from a student’s point of view are at all things they should care about.
Salient 06.10.09 at 10:12 pm
Actually, many legislators would be actively HOSTILE to that goal- they (rather sensibly) want the state university to fulfill its intended function of educating ordinary citizens rather than diverting resources to competing in academic penis-measuring competitions.
Oh, how I wish that were true. You have momentarily forgotten the ease with which a legislator, quite understandably mind you, can get swept up in the rhetoric of Quality. Also, the rhetoric doesn’t usually come with dollars attached: it’s “become Top 25 while enduring yearly budget cuts.”
Salient 06.10.09 at 10:16 pm
I don’t see how USN&WR has any incentive to have a rankings system that isn’t routinely gamed.
They have a strong incentive to have a rankings system that isn’t thought to be gamed, at least not by their audience: middle-class parents of students in the college-applications process, who believe the rankings correspond at least roughly to the chance of competing with similar students from other schools on the job market. That is, their audience is largely parents who believe that anyone who hires people also pays close attention to the same report each year.
Steve LaBonne 06.10.09 at 11:41 pm
Well, I can only speak to the Legislature I know, but that’s not the general attitude in Ohio, with a partial exception for Ohio State. The legiscritters’ interest in research, even for the most part research at OSU, is much more oriented to applied research that they can be persuaded might create jobs than it is to “prestigious” stuff. And attempts to institute more selective admissions at state schools require careful political diplomacy- after all, the dumbass legislators went there and what’s good enough for them… Miami is the one university that gets a pass on having fairly competitive undergraduate admissions, just as OSU is the only one allowed (but not exactly pushed) to aspire to a major research reputation. Here it’s more often governors and their appointees that (sometimes) want to push for more “prestige” and legislators that are skeptical.
(But there is one big exception: they’re VERY interested in OSU beating Michigan in football. Now FOOTBALL is where they actually demand “elite” status. ;) )
But then this is Ohio, the State of Mediocrity. Sigh.
Chris Lloyd 06.11.09 at 1:28 pm
“I don’t really see how to eliminate the incentive artificially to undermine one’s close rivals when ranking them” I do. It’s easy. The ranking algorithm just ignores your assessment of any rival within say 10 ranks of thyouy. So each of say 100 competing schools would recieve 90 ranks (between 1 and 90) from those school that are not their close competitors.
You don’t even need to advertise this to the school. Just let them think they are being clever down grading their close rivals and they will probably rank the others relatively objectively.
Salient 06.11.09 at 1:39 pm
I do. It’s easy. The ranking algorithm just ignores
A ranking system that takes into account the opinion of persons with a clear conflict of interest, who are likely to have never visited the school or seen its operations? If we’re starting from that vantage point, how about eliminating the reputation rankings from university faculty and replacing them with rankings received from human resources managers at large and growing corporations? That’s the information parents want to know anyhow: between schools A and B, which gives the competitive edge on the job market?
(Such a metric would have the same destructive effects as the current system, but the potential for a school to game that aspect of the metric would plummet.)
Robert 06.11.09 at 4:05 pm
Not about university rankings, but about USN&WR’s hospital rankings, here and here.
Ben Hyde 06.11.09 at 4:41 pm
The choice to rank any group’s members using “objective metrics” is to create a game. The parties that establish the game assume a powerful role. Maybe they deserve that role, maybe not. There are serious ethical questions there. That the players start learning the rules and adapting their behavior is exactly what the game designers wanted. It is extremely disingenuous for them to complain if when of those adaptions fail fufill thier power fantasies.
This is a pattern that repeats it self over an over again. Using metrics and raising stakes is an extremely popular managerial technique. There is always a presumption that the social contracts can be manipulated by the powerful without the players responding. These schemes are often called objective or rational, and it’s odd to insist that the players forgo rational tactics.
I once sat on a committee to select talks: three out of ten submissions. We asked a dozen people to vote; a scale from 1-10. Most voters submitted votes that were normally distributed with a narrow variance. One guy gave talks 10, and the others 1. Only his signal survived the sums. That changed how I fill out surveys. The ethics of that is questionable, but will you change how you fill out surveys?
Will you play the game well, or do you feel that each play must hew to the behavior he assumes the other players adopt? If so then should he assume the other players will play in conformance with some ethical frame. Is that ethical frame the one the game’s umpire is modeling?
norm 06.16.09 at 5:05 am
I heard rumors that Boston University hired unemployed recent graduates to work fake jobs in the career services office in order to boost their “percent employed one year after graduation” score on U.S. News. At first, I was disgusted. But then I thought, “well, all other things being equal, I’d rather go to a law school that guaranteed me a job in career services if all else fails!”
Comments on this entry are closed.