Comments on: Ranking UK philosophy departments: RAE versus Leiter

By: noone

noone — Wed, 05 May 2004 02:38:20 +0000

From what I remember when I was studying philosopy 15 years ago, it was difficult to think of 15 “good” departments, maybe only 7 or 8, but I do remember York University was one of them.

By: Jo Wolff

Jo Wolff — Sun, 02 May 2004 16:21:59 +0000

How about a completely different, market based, approach, based on numbers of graduate students who apply? We would have to discount non-academic CV enhancing factors, of course, as well, as attractiveness as a place to live. So to do well, you would have to argue that your department gets loads of applications, but the non-academic visibility of your institution is low and it really isn’t that great living here. In the US this would put Rutgers way ahead, I’d imagine. I don’t want to say much about the UK but I hope my friends at Sheffield won’t take it the wrong way when I say that I think they would probably come top in this table.

By: Michael Otsuka

Michael Otsuka — Sun, 02 May 2004 13:23:26 +0000

Chris: I didn’t notice that you had made another post in the meantime while I was typing my above post. The ‘pot calling the kettle black’ remark is a response to the following comment of yours: ‘I don’t see where you get the data for your second comment, though, since a UoA is not = to a department.’ I don’t think the table on which we’ve just converged distorts on account of not taking size into account. Rather, it distorts on account of giving the same rating of ‘5’ to 16 departments which must have varied quite a bit in terms of the average quality of their submissions.

By: Michael Otsuka

Michael Otsuka — Sun, 02 May 2004 13:10:46 +0000

Chris, My second table is derived from the RAE’s submissions data on its web page. It is, you are right, data about ‘units of assessment’ (UofA) rather than ‘departments’. But note that the RAE data that you rely upon in arriving at your RAE ranking of ‘departments’ is also data about ‘units of assessment’ (UofA). So, as I was just following you in eliding the distinction between a department and a UofA, I think your last remark is an instance of the pot calling the kettle black. (Note to non-anoraks: A ‘UofA’ is a construct of the RAE. Philosophy departments and philosophy UofAs are in fact, however, roughly co-extensive, though the latter sometimes includes philosophers in a university who aren’t in the philosophy department.) I take your point that departments/UofAs should get credit for submitting a greater proportion of their philosophers. And that’s what impelled me to make my second post, which is simply lists the total number of philosophers in a given university, irrespective of whether their work was submitted to the RAE for assessment. Note how much more closely your RAE-derived rankings correlate to that list rather than the following list which is probably as close as we can come, simply on the basis of RAE data, to a ranking of departments based on their assessment of the average level of quality of publications of the members of the department as a whole: 1. Cambridge (tied) 1. Kings (tied) 1. LSE (tied) 4. Oxford 5. Edinburgh 6. Birkbeck (tied) 6. Bristol (tied) 6. Sheffield (tied) 6. St Andrews (tied) 10. Durham (tied) 10. East Anglia (tied) 10. Essex (tied) 10. Nottingham (tied) 10. Reading (tied) 10. Stirling (tied) 10. Sussex (tied) 10. UCL (tied) 10. Warwick (tied) 10. York (tied) 20. Leeds (tied) 20. Middlesex (tied) (Method: I ranked departments which were rated 5* above those which were rated 5. Among those which were awarded the same rating, I ranked those which submitted a greater proportion of their staff for assessment above those which submitted a lesser proportion.)

By: Chris Bertram

Chris Bertram — Sun, 02 May 2004 12:52:59 +0000

Actually Mike, if you think size is unimportant, then the following RAE ranking would seem reasonable: rank first by quality score and then according to the proportion of staff included. That gives us: Premier League (5*A) LSE, Kings and Cambridge. Nationwide division one(5*B) Oxford Nationwide division two(5*C) Edinburgh Nationwide division three (5A) Sheffield, Birkbeck, St Andrews, Bristol Vauxhall Conference (5B) Reading, Stirling, Nottingham, Essex, UEA, Durham, Sussex, UCL, Warwick and York. I think you’ll agree that one of the perverse effects of not taking size into account is the low ranking that UCL would achieve.

By: Chris Bertram

Chris Bertram — Sun, 02 May 2004 11:47:53 +0000

Mike: Right about Reading/Durham (I’m not sure how I messed up on that). But surely wrong on size, since it is up to UoAs how large a proportion of their staff they submitted. A UoA, such as my own, that excluded nobody and still got a 5 ought to get due credit for that. I don’t see where you get the data for your second comment, though, since a UoA is not = to a department.

By: Michael Otsuka

Michael Otsuka — Sun, 02 May 2004 08:53:27 +0000

In case it’s relevant to my post above, the following is a list of the ten largest UK departments when one counts every philosopher, including those whose work wasn’t submitted for RAE assessment: 1. Oxford 2. Cambridge 3. Leeds 4. Kings 5. UCL (tied) 5. Glasgow (tied) 7. St. Andrews 8. Warwick 9. Edinburgh 10. Bristol

By: Michael Otsuka

Michael Otsuka — Sat, 01 May 2004 20:59:26 +0000

One big difference between Leiter’s ranking and Chris’s RAE-based ranking is that Chris’s builds in a big advantage to large departments. Chris takes the RAE rating—which is roughly a reflection of the RAE panel’s assessment of the average quality of the work of the philosophers submitted for assessment—and multiplies that by the number of philosophers whose work was submitted. Leiter, by contrast, asks his surveyors to come to an overall judgment of the ‘quality of the faculty’ of a department. Leiter’s surveyors may, if they wish, take size into account, but they need not. While surely at least some of them took overall numbers as well as average quality into account, I doubt that any of them did so in Chris’s mechanical fashion. If they all had, then Notre Dame, which ranks 14th on Leiter’s survey, would surely have ranked above small departments such as Harvard, MIT, and UCLA, which each tied for 8th. This is because Notre Dame’s department is larger than the latter three combined. The 8 largest UK departments measured simply in terms of the sheer number of philosophers they submitted to the RAE (and ignoring the RAE’s assessment of the quality of their publications) are: 1. Oxford 2. Cambridge 3. Leeds 4. Kings 5. St Andrews 6. UCL 7. Warwick 8. Bristol Note how closely this size-based ranking correlates with Chris’s RAE ranking. It is, in fact, identical, except that Kings and Leeds swap places. So Harry it’s probably not owing to any inability on Reading’s part to submit a persuasive RAE report that they score so low on Chris’s RAE table. But rather it’s that Reading’s department is relatively small. In fact 16 departments submitted more philosophers to the RAE than did Reading. By the way, it appears that, by Chris’s formula, Reading’s RAE ranking should be tied with Durham’s, and Durham ranks 15th, according to Chris. It is, I think, not entirely coincidental that Reading and Durham both submitted the same number of philosophers to the RAE —i.e., they both submitted fewer people than 16 other Departments. Once again, there’s a fairly close correlation further down the scale between sheer size and Chris’s size-weighted RAE ranking. So if you think size may not be everything but it matters a great deal, then Chris is your man.

By: Jo Wolff

Jo Wolff — Sat, 01 May 2004 15:04:21 +0000

Harry, I entirely take your point that the most important member of department from the point of view of intending graduate students may well be someone who either has stopped publishing or never did. In the recent past we can mention Dreben and Albritton as perhaps the most prominent. I think the most important criticism of all of the RAE is that such people would count as ‘non-research active’ and are likely to come under pressure to publish even if they have nothing they want to commit to print at the moment and even if their time would be better spent, say, supervising graduate students. What I don’t know, though, is whether the US is currently more tolerant of this. I don’t know who your colleague is, but all the names that come to my mind are either retired or dead. But it may be that this confirms your point: this information doesn’t circulate very much.

By: Matt

Matt — Sat, 01 May 2004 14:53:32 +0000

Harry, I’m pretty sure the problem you mention w/ the Leiter report about Buchanan can’t come up, at least now. As I read the methodology, a list of current faculty is given to the rankers, not a list of departments. So, if Buchanan was on the list, you surely got credit for him (if such things were even going then.) As I understand it, the lists don’t include the names of the departments, so as to try and force people to actually look at the lists. (This is all taken from the “methodology” section of the PGR web page.) If anything this sometimes gives more credit to depts who list people w/ tenious connections to the dept., such as the bio-ethics people at Penn.

By: harry

harry — Sat, 01 May 2004 14:13:03 +0000

Chris and Jo—you’re both right about the RAE: but wrong (I think) about Leiter. My understanding is that Leiter surveys people’s opinion about other departments. People’s views of other departments are almost always out of date. EG, my own department contained Allen Buchanan for four years. Hardly anyone else registered he was here till after he left—we had no reputational benefit while he was here, but gained some later. Leiter himself seems indecently up-to-date on these things, but his input must be limited; there’s a significant time lag in his reports, probably as significant as in the RAE. The other thing is that as jdsm wisely says reputations come from publishing rather than philosophising. RAE, of course, is only about publishing; and Leiter might have the edge in that there are at least some people who are famous for being excellent philosophers who are great to have around. But, again, one of the best philosophers in my own department (the best?) has hardly published for years, although he continues to write, teach, discuss things with colleagues, and is the one person I tell every graduate student they have to take a seminar with if they took the trouble to come here. I doubt we get much reputational benefit from his presence. But there’s nothing can be done about this; ranking requires publically accessible information. I think, in a way, this makes ranking groups of philosophers sillier than ranking individual philosophers. But, of course, ranking is a fact of life. Th Brits, btw, should simply get rid of the RAE for philoosphy and hire a consultancy to do a survey of non-UK-based philosophers; and bas e the excercise simply on that. It’d be way cheaper and would get equally good if not better results.

By: Chris Bertram

Chris Bertram — Sat, 01 May 2004 12:33:40 +0000

I’m sure you’re right there Jo. I certainly don’t want to give the RAE more credibility than it deserves and you and I both know what a massively distorting role it has played in British philosophy.

By: Jo Wolff

Jo Wolff — Sat, 01 May 2004 11:36:34 +0000

Well, Chris, although I have no desire to pick at old sores, I feel I have to here. The myth of the RAE is that a small group of people can conduct a speed-reading peer review of the entire range of philosophical output of the UK. From my discussions with RAE panelists over the years my feeling is that they often have had to read submissions that they are not in a good position to judge (for example, that no journal would ask them to review). Furthermore the speed with which the panelists have to read the vast volumes of submissions leaves them too dazed to make philosophical judgements except in the narrow specialist areas in which the panelists are genuinely up with the current literature. For the departmental rankings as you suggest Leiter may be even less reliable, as it is a reputational survey, but the area rankings are based on the judgements of people active in sub-fields, and this may be the best guide of all, even if there is no uncontroversial way of turning it into a ranking. I understand that the process is about to start to produce a new Leiter report. I’m not sure when it will be completed.

By: Chris Bertram

Chris Bertram — Sat, 01 May 2004 10:31:27 +0000

“The advantage of Leiter is that it is based on who is teaching in the department now, rather than up to seven years ago, as will eventually be the case with current RAE scores.” True, Jo, but I think I’m right in saying that the current Leiter ranking is about 2-years-old now, which makes it roughly contemporaneous with the RAE. While Leiter ranks people who were teaching in departments a couple of years back, the RAE will “eventually” go out of date. Not much to choose between them in 2004 then, although Leiter will be a better measure on its next update. The other thing that the RAE has going for it (as against Leiter) is that it is based on a peer-review-panel actually reading the submissions. Leiter is based on peer opinions of fellow professionals and there may well be some reputational bias involved there. (For instance in the case of X who is a “big name” but hasn’t produced much of note recently.)

By: Jo Wolff

Jo Wolff — Sat, 01 May 2004 09:45:23 +0000

Very nice of enthymeme to rank my dept UCL higher than Kings, but in fairness I should say that Hampshire has been emeritus since possibly the 1960’s and Honderich for around 5 years. On the other hand Sorabji, O’Shaughnessy and Gilles have retired from Kings, and Sainsbury is only part-time. One further thing that may have affected KCL’s position is that they submitted some world class linguists as part of the Philosophy submission which may distort the Bertram RAE ranking both in terms of quantity and quality. One lesson is that often departmental reputations outlast the people on which those reputations are based. The advantage of Leiter is that it is based on who is teaching in the department now, rather than up to seven years ago, as will eventually be the case with current RAE scores.