Defending the NRC Rankings

There’s a qualified defense of the recent NRC rankings of universities by the rather magnificently named duo of E. William Colglazier and Jeremiah P. Ostriker in the “Chronicle today”:http://chronicle.com/article/Counterpoint-Doctoral-Program/125005/?sid=at&utm_source=at&utm_medium=en

bq. Stephen Stigler, a professor from the University of Chicago, offered a thoughtful critique, saying that the project “was doomed from the start” when reputation was downplayed. Rankings based on reputation were provided in the 1982 and 1995 NRC reports, but they were excluded from the charge given to the committee for the 2010 report. We agree that reputationally based rankings contain important information, which was discussed in the report, but they also contain seriousness weaknesses.

bq. The decision not to include rankings based purely on reputation arose from three considerations. First, pure reputational rankings can contain “halo effects,” meaning a program’s ranking may be skewed by the university’s overall reputation, or it may lag behind because of its past reputation. Second, reputationally based rankings were not supported by many universities whose participation was needed to collect the data. And third, this study was intended to provide a comprehensive, updatable collection of data on characteristics associated with perceived program quality that would allow faculty, students, administrators, and other stakeholders to assess programs based on their own values, and thereby become a transparent tool to strengthen doctoral programs. The reputationally based rankings that received the most attention in previous reports did not provide a means to achieve this last important objective.

The NRC rankings have come in for a lot of methodological flak. But this at least seems to me to be a reasonable claim, and one that could indeed be strengthened a little further. One of the reasons that reputational rankings plausibly lag actual changes in departments’ quality is because they tend to reflect individuals’ internal models of what _others in the discipline_ think about a given department, rather than personal information that the individuals themselves have about a department’s actual quality. The fact that _x_ department is perceived as a flagship program counts for more than the fact that I know professors a, b and c at that department, and they don’t seem all that smart or productive. Public perceptions plausibly trump private information.

Cognitive science would suggest (or at least hint) that having a source of non-reputational information about departments’ actual quality rather than their perceived quality, even if it is rather noisy, could usefully help to unsettle these collective beliefs. It can make people more likely to reveal their private information, and to include that information (rather than their perception of what everyone else thinks) in their judgments as to which department is _actually_ good or in fact not as hot as its reputation warrants.

In short – rather than thinking about innate-quality-based rankings (however flawed) or reputational rankings (however biased toward the status quo) as simple alternatives, one can think of them as possible stages in an iterated search for better information. If the information provided by a quality-based ranking such as the NRC is really off-base (as perhaps it is with respect to some departments), one would expect that it will at most have a very moderate impact on people’s beliefs (as it will contradict _both_ their private information and their perception of the general wisdom of the field). If, instead, the information more accurately reflects the actual performance of a department, it will arguably have a greater impact on people’s judgments, since it will reinforce their confidence in their private information as opposed to the received wisdom.

I’ll note that I’m not an unbiased observer here. My department is one that has (according to the possibly-themselves-biased perceptions of those within it, including myself) performed less well than it should on reputational surveys, but “done quite well indeed”:http://www.themonkeycage.org/2010/09/the_nrc_rankings_of_political.html in the NRC ranking. But if I’m right (and if my colleagues are right too), one would expect that this will result in a significant bump in my department’s reputation when the next reputation-based survey happens. The NRC rankings will have jarred people’s perceptions a bit, so that they do not depreciate their own private knowledge so much. We’ll see what happens.