Law reviews and meritocracy

by Henry on August 8, 2005

This paragraph in a post by Andrew Morriss, guestblogging with the Volokhs, struck me as saying some pretty odd things about publication policies in the legal academy.

There are huge problems with the U.S. News rankings but there is no question that they are important. For example, a friend recently told me that she had been called by a law review about one of her manuscripts. The articles editor apologized for rejecting the manuscript and explained that the rejection had been made without reading the paper because the editors had mistakenly misclassified my friend’s school as being in a lower tier law school. Now that they realized their error, the editor told her, they wanted to consider the article on the merits. I don’t know how widespread this type of screening is, but that it occurred at a well-ranked, but not top journal is at least moderately disturbing.

Now I know that there are a lot of complaints among legal academics about the dominant role of student-edited law reviews. But (if this is at all representative), I hadn’t realized quite how bad the problem was. It’s not (as Micah has argued previously ) the fault of the students editing these reviews – when you have so many article submissions that you’re literally unable to read them all (thanks to the practice of multiple submissions), you’re inevitably going to have some more-or-less unfair metric for deciding which ones to read, and which ones not to. The problem is a structural one. But it does suggest that it’s going to be a lot more difficult for a smart legal academic in a second or third tier school to improve her position by publishing material in good journals, than it would be if she were, say, a political scientist or a sociologist. If her school’s position in the rankings counts against her chances of getting published, she may find herself in a Catch-22 situation; the only way to get published in good journals is to improve her personal name-recognition (since her school won’t help), but the only way to improve her personal name-recognition is to get published. Blind peer review, which is the norm across most of academia, does serve as at least a modest corrective to mutually reinforcing hierarchies in journals’ publication records and department rankings. It’s possible, albeit difficult, to publish your way up. As an aside, I wonder whether the importance of name-recognition to legal academics helps explain why so many of them have started blogs – blogging is a cheap and easy way to make your name familiar to the editors of law reviews.

Update: Steve Bainbridge is seeking to test whether law review editors do indeed read law blogs, by soliciting “bids” for a recent article that he’s written. He’s already received an offer from a journal in the top 35-40 range.

{ 25 comments }

1

Bertrand 08.08.05 at 3:09 pm

I’d like to see someone knowledgable build a case against the whole post-bachelor’s educational system (above all in the humanities. Various sorts of corruption seem to be in place. Everyone complains about something, but as far as I know no one has gathered the various complaints of the various types into a well-thought-out piece making a case that the overall problem is very serious.

Having a devil’s advocate make such an argument would be a prerequisite to refuting it. Right now we really don’t know how serious the overall problem is, since the case hasn’t been made.

Of course, I **would** say something like that.

2

Jane Adams 08.08.05 at 5:08 pm

This is a pretty extreme example of problems in the current journal system. Others are delay in publication and the fact that only the wealthiest universities can afford many journals.

I think the solution is public posting of papers. The federal government and other sources can root out the sums sent to current journals. They can use their editorial superiority to select papers considered most important and mantain existing structures even as a thousand new journals bloom and new methods of organization and selection burst forth. The fact that a number of key journals may referance the same paper is a strength, not a weakness.

Information by and large paid for by the people and theorectically open to all according to the idealism of science subjected to all kinds of weird controls by structures resembling mercantile guilds .

Put it out there and then orgaize it. If we don’t do this then some of the rising science centers will and the result will be their research having a much better chance of becoming central because it’s “turnaround time” is reduced by 6 months to a year (even more if we add a system which encourages work in progress) is more widely availible and better organized (both electronically and in terms of the vast “critical communities” which can have at it.)

3

Tom T. 08.08.05 at 10:29 pm

Ironically, law school ranking is one of the primary discriminants used by law firms and many other legal employers in determining where to interview and which students to hire. I suppose it’s not surprising that the students would adopt the same tactic in a situation where they are “hiring”.

4

Tad Brennan 08.08.05 at 10:53 pm

That is a deeply weird glimpse of how it works in law journals.

But it is *nothing* like that in philosophy or classics, the two areas I know something about directly.

I referee quite a number of papers every year, and *all* of them are rendered completely anonymous: not only do I not know whether the author is at a “second tier” school or not, I don’t know their name, their employment status, whether they are a full professor or a grad student or a dog. All I have is the contents of the paper. I have no choice but to judge it on its merits.

Ditto for the APA conferences. I have been reading submitted papers to appear at an APA conference, and these too are *completely* scrubbed of any identifiers.

So, I mean, this is evidence of a deep corruption in the law journals. But it is no part of “a case against the whole post-bachelor’s educational system above all in the humanities” (to quote bertrand above) because as far as I know, the humanities just don’t work this way. At least, in my corner of them they sure don’t.

(I will say that every now and then I get a paper where the journal didn’t do a complete scrub–e.g., they tore off the cover page, but forgot that the institutional affiliation was on the back page. Then I include a rant with my refereeing, telling the journal editors that they have to do a better job. Doesn’t happen much.)

5

John Emerson 08.08.05 at 11:12 pm

I have seen it said that at every level of the university the best single predictor of success is the quality ranking of the previous institution attended. Specifically, the students from the best undergrad school do best at the grad level, and PhD’s from the best grad programs do best in hiring and advancement.

This of course could mean that the present system is Panglossian, having attained the BOAPW, or it could mean something different.

A friend at Texas told me that it was clear from Day One who the successful philosophers in his cohort would be, and that oddly enough they were well-recommended students from the high-ranking schools. This could be meritocracy, or it could be the old-boy network.

We also have Brian Leiter’s fine-tooth rankings of philosophy schools and specific programs within the schools, which to me seem weighted down with fake precision. A study also showed that there’s general agreement within the profession about these rankings. I believe that this is supposed to show us that philosophy is now mature as a science, the way physics is.

Perhaps corruption is less necessary in philosophy, since one rather narrow school with well-defined standards has attained a stranglehold on hiring and promotion.

6

Michael Kochin 08.08.05 at 11:49 pm

1. The journal editors, who assign the paper to referees, see the author’s institutional affiliation. Since experienced editors do try to influence the decision by the choosing the “right” referees, blind refereeing in itself can’t eliminate institutional prejudice.

2. The American Economic Review study showed that professors at top institutions had even greater advantages under blind refereeing than when the institutional affiliations of authors were revealed to referees. This suggests that:

A. Institutional affiliation of an author is a pretty good proxy for the quality of an article.
(Highly ranked institutions may have their deadwood, but qua deadwood they aren’t dubmitting many papers!)

B. “Corruption” in the refereeing process when the author is known to the referee may be a matter of helping friends and harming enemies, and not institutional prejudice.

3. In my cohort at Chicago it was clear from day one who was going to make it as an academic political scientist, but the single most successful of us came from outside the old boys network, as a mature student from a not particularly prestigious public University. His talent was nonetheless immediately recognized by students of professors alike.

7

PanJack 08.08.05 at 11:56 pm

RE
“I referee quite a number of papers every year, and all of them are rendered completely anonymous…”

Even without knowing the NAME it is pretty easy for me in my field to have an idea of who the author is or who the author is associated with.

8

Matt McGrattan 08.09.05 at 1:43 am

I’m currently in the process of reviewing a small number of papers for a graduate philosophy conference and, as with journal submissions, I have no idea who they are from or which philosophy program the participants are affiliated with. All I have to go on are a set of uniformly formatted Word documents identified by a number.

Unlike panjack I don’t think I could even take a guess at the institution to which the authors are attached.

The law reviews may face the problem of more submissions than they can handle but it does seem like some other metric can be used, one that offers less potential for accusations of corruption, than the affiliated school of the author.

9

Matt McGrattan 08.09.05 at 1:48 am

I’m currently in the process of reviewing a small number of papers for a graduate philosophy conference and, as with Tad’s comments above re: journal submissions, I have no idea who they are from or which philosophy program the participants are affiliated with. All I have to go on are a set of uniformly formatted Word documents identified by a number.

Unlike panjack I don’t think I could even take a guess at the institution to which the authors are attached.

The law reviews may face the problem of more submissions than they can handle but it does seem like some other metric can be used, one that offers less potential for accusations of corruption, than the affiliated school of the author.

10

Matt McGrattan 08.09.05 at 1:49 am

Sorry about the double post. The first returned a series of .php errors — I assumed it hadn’t made it through.

11

Noah Snyder 08.09.05 at 3:46 am

My impression in mathematics (from several discussions with professors, I’m just a grad student) is that although the referees might not know who the author is, the editors of the journal definitely do. And so knowing an editor makes a large difference in your chances of publication. Of course this might be because you submit to a particular editor, and so that editor is guaranteed to know who you are even if no one else in the process knows.

12

vn 08.09.05 at 7:38 am

There are a lot of (justified) criticisms of student-edited law reviews, though I believe more of the blame should be laid on law professors who appear happy with a system which lets them slough off responsbility for editing their own scholarly journals.
That said, when I was head articles editor of our law review (top-20), we paid close attention to articles from scholars at lower-tier schools. Our reasoning was that a submission from a prof at, say, UCLA was probably going to attract attention from higher-ranked law reviews, whereas an article from a prof at a low-ranked school could be a diamond in the rough that would be overlooked by higher-ranked journals but would wind up being widely cited once published. So I guess we were gaming the system, too, but acting on the assumption that editors at *other* law reviews were going to go with a scholar’s school prestige over the merits of the article.

13

harry b 08.09.05 at 8:48 am

Philosophy again — my experience is almost like tad’s and matt’s, but not quite. I keep up with the literature on several issues, and about 1 in 4 of the submissions I referee are pretty easy to place. I have also heard people say things like “if I’d known it was by so-and-so I’d have recommended it for publication”. Also, it is worth noting that the big journals at least do not send out all submissions for refereeing, and they do know the affiliation of the author (or they can know it).

But otherwise my refereeing is truly anonymous. The editor of one journal I frequently referee for is extremely conscientious about matters of mix, and about getting longer and more helpful comments for rejections when the paper is less senior or from a lower tier school, by the way. This reflects a very particular view of the function of the refereeing process.

14

Hektor Bim 08.09.05 at 9:16 am

In physics, when you referee a paper, you know everything. Who wrote it, their institution(s), and even what previous reviewers said.

Interestingly, the most successful people I knew in physics grad school were all from public universities.

15

cranky 08.09.05 at 9:33 am

Jane Adams wrote I think the solution is public posting of papers. The federal government and other sources can root out the sums sent to current journals. They can use their editorial superiority to select papers considered most important and maintain existing structures…

Eeek. The last thing academia needs is BushCo — or, for that matter, any single policing organization, whether governmental or private, elected or appointed — deciding which papers are “most important” and which “existing structures” to maintain.

16

Silent E 08.09.05 at 9:39 am

Two things to understand this situation (and indeed, nearly everything else about law schools):

1. The only thing that differentiates a law school is the student body.

2. Law students (and lawyers) are natural game players.

Conclusion: law schools are abstract signifiers of the quality of the student – not the education. The initial admissions decisions are only mediated by final class rank (and making law review). Everyone understands this. Students, professors, and those who hire them react accordingly. The only real questions are of the “big fish little pond vs. little fish in big pond” variety.

There is also a very strong class bias here. Like the “Anna Karenina Principle” (“Happy families are all alike, unhappy families are each unhappy in their own way…”) the students at the top law schools generally had to have everything turn out right – for which an upper class background helps immensely. In turn, they don’t interact with very many students who have harder luck – and thus fail to realize the degree to which luck and birth have selected them for their positions. This leads them to over-estimate the importance of skill and ability in outcome, both their own and that of their peers. Because this group then goes on to become the next crop of professors and hiring partners, the biases get entrenched. Sad, but true.

=

Detailed Discussion:

1. The only thing that differentiates a law school is the student body.

I mean really, law isn’t that tough to teach. There’s two parts of law school: the first year where all the incoming students learn to “think like a lawyer”, i.e., making logical arguments supported by facts and evidence. It’s called “critical thinking” and it’s depressing that so many bright and motivated 22-year-olds can be so bad at it. 1Ls also learn HOW to do legal reseach – how to find and cite the correct cases and how to present effectively their arguments in written form.

First years learn the “core subjects” of the law, in part because every lawyers needs to know about Constitutional Law and Property and Torts and Criminal Law, but also because those subjects are areas that students may already have some familiarity with and where the fine legal distinctions have been carefully hashed out over decades (or centuries!) and so can be taught to beginning students.

The second part, called “the rest of law school” is all elective: it involves taking courses in subjects that interest the student, or which may be necessary for the specific practice area the student wants: intellectual property, securities, business associations, federal regulatory and enivronmental law, etc. But these courses are pretty simple: “Black letter law” and the major or recent cases. Almost ALL of it comes out of the textbooks.

There’s little real effort to teach students HOW to practice law: drafting contracts, getting clients, networking, answering discovery, litigating, lobbying, or negotiating. So it’s not like the quality of the faculty at the school is really gonna affect your education.

What matters is your fellow students: better students mean that classroom and hallway discussions will be more informative and will do a better job of developing reasoning skills.

More importantly, since all law schools grade on similar curves, the only difference between top tier school “A” students and lower tier “A” students is the quality of the curve. It’s the “who did you beat” issue.

2. Law students (and lawyers) are natural game players.

Law students, professors, and recruiters, and the big firms that hire new grads all understand that law schools are just symbols – arbitrary signifiers of percevied quality. So the best students all want to go to the best schools. They all think they’re hot stuff – nobody goes to Harvard planning to get Cs.

And, for the most part, most students who can get in to the best schools can go because of financial aid and the promise of very large post-graduation salaries to pay off loans. That said, the law school admission process is not perfect: many outstanding students don’t go to the best schools – maybe they need to stay closer to home, have families, can’t get aid, didn’t do well as undergrads, or didn’t have the particular mix of grades, LSAT score, activities and leadership for the schools they applied to, etc.

Not really an exception: lower tier schools with high-reputation, very specialized programs within a specific sub-field, i.e., patent law, or complex financial instruments.

17

Silent E 08.09.05 at 9:49 am

Other considerations:

1. Law is about subjective values and argument: there isn’t much of an objective standard by which to measure law review papers. This is less true for “law and economics”-type pseudo-mathematical papers.

2. Law student reviewers are not sufficiently informed to make penetrating critical judgements about the arguments. This is ESPECIALLY true for mathematical-statistical papers.

3. Most law review articles are of two types: (a) recent court decisions have said X, and (b) future court decisions should say Y. How do you choose which descriptive or normative article is “best”? Nearly all the descriptive articles will be factually true, but most attorneys and law professors don’t get their descriptive information about the law from journals, they get it from weekly or monthly summaries or cases in local Bar Reports. As for normative articles, well… there’s no shortage of opinions out there.

18

nik 08.09.05 at 11:13 am

It’s not … the fault of the students editing these reviews – when you have so many article submissions that you’re literally unable to read them all … you’re inevitably going to have some more-or-less unfair metric for deciding which ones to read

Couldn’t they just toss a coin and throw away all the papers that get tails? This is arbitrary, but it isn’t “unfair”, as only a limited number of papers can be read and everyone gets the same chance of making it to this stage.

19

John Emerson 08.09.05 at 11:49 am

There is also a very strong class bias here. Like the “Anna Karenina Principle” (“Happy families are all alike, unhappy families are each unhappy in their own way…”) the students at the top law schools generally had to have everything turn out right – for which an upper class background helps immensely. In turn, they don’t interact with very many students who have harder luck – and thus fail to realize the degree to which luck and birth have selected them for their positions. This leads them to over-estimate the importance of skill and ability in outcome, both their own and that of their peers. Because this group then goes on to become the next crop of professors and hiring partners, the biases get entrenched. Sad, but true.

Not just in law schools.

A lot of the most successful academics seem to have an undue willingness to accept the discipline uncritically in its established form. (Yes, academics think critically, but not about themselves. And there are internal debates, but within the paradigm).

“Don’t argue with the teacher” isn’t a universal rule, but it’s a pretty good default principle for ambitious young things.

20

Brian 08.09.05 at 12:11 pm

Law school is odd. It doesn’t do much as far as teaching you professional skills you’ll need like drafting contracts, negotiating, trial practice, etc. But it goes on a lot longer than it needs to if it’s just bar exam prep. You could teach the courses on the bar exam and legal research/writing in two years or less. The third year is a waste if time.

I also don’t really understand the point of law reviews. In doing legal research and writing I’ve found them consistently irrelevant. I guess they’re just ivory tower exercises?

21

Henry 08.09.05 at 1:15 pm

I should say that I agree with the commenters who have pointed out that blind review often works better in principle than in practice – that’s why I said in the original post that it worked only as a “modest corrective.” Still seems better than the alternative.

22

Silent E 08.09.05 at 2:04 pm

Law school is odd. It doesn’t do much as far as teaching you professional skills you’ll need like drafting contracts, negotiating, trial practice, etc. But it goes on a lot longer than it needs to if it’s just bar exam prep. You could teach the courses on the bar exam and legal research/writing in two years or less. The third year is a waste if time.

This is known to the law schools. If students leave even a semester early, the law schools lose out on 17% of tuition income. That’s why they go to great lengths to structure their “requirements” to prevent as many students as possible from leaving early. E.g., requiring an arbitrary number of credits high enough to force students into a sixth semester. They might offer a very important or popular class only in the Spring semester, every other year, e.g., Environmental Law or Bankruptcy. Some important subjects are broken up into three or more courses (Intellectual property: Patents, Trademarks, and Copyrights, or the UCC subject courses), some of which may require completion of another, or which are not offered every year. Many firms want to know class rank, and of course, a class rank can’t be determined until the whole class graduates. You can’t be a senior editor of Law Journal if you leave a semester early.

Hiring firms like the three year system because it lets them audition prospective associates for the summer after the 2L year. For students, the third year is a chance to take courses relevant to their probable job after graduation, after they have some exposure to legal work over the 2L summer. It also is a chance to take smaller courses not frequented by the top 5% of students in an effort to pad out GPAs which might have been bloodied in the larger 1L mandatory courses.

I also don’t really understand the point of law reviews. In doing legal research and writing I’ve found them consistently irrelevant. I guess they’re just ivory tower exercises?

Path dependence. The student-run law reviews evolved that way, and it serves everybody’s interest at the top. They are published by students, lending prestige to the editors – what would be considered mindless proofreading drudgery is elevated to high-status “experience”! The student labor is free.

Faculty do not have to spend time reading galleys or peer reviewing the authors. Of course, as others mentioned, this extends turn-around and publication times, leading to the need for multiple simultneous submissions and serious quality control issues.

This also means that practicing attorneys who have acheived some recognition or expertise (not necessarily both) in their field can write articles and get them published in niche journals – and Law Review publications are always impressive on a non-academic’s resume.

In turn, the inexhaustible supply of potential articles (professors have few requirements and can always secure a student research assistant) and eager students (everyone wants to put “Law Review” on their resume) is feeding an ever-growing number of niche journals. Most law schools now have at least three journals, and some have ten or more. This then leads savvy hiring professionals to “rank” the various journals at a single school in an attempt to combat “journal inflation”.

Some, but by no means all, journals are purchased by law school libraries. Nobody except tenured faculty over 60 who can’t use Lexis actually bothers to read the paper editions of the journals. They are uploaded into Lexis-Nexis, making them available to researchers – who, as you find, end up wading through mountains of pedantic uncitable crap in the vain hope that maybe there’s something useful in there

Whether or not anyone actually reads the crap is irrelevant.

23

save_the_rustbelt 08.09.05 at 2:34 pm

The entire post-grad system is a mess. The primary objective is to protect the comp, perks and status of those who already have tenure, as well as the administrators.

I have watched the whole process (as a lecturer not concerned with tenure) and it is as petty and ridiculous as anything I have ever seen.

In accounting, the Ph.D.s have done such a good job of restricting the market there is a serious shortage of new profs and some programs will be in danger come time for accreditation (a class taught by an incompetent Ph.D. is automatically rated higher than classes I teach).

I don’t know of a single practicing CPA in the entire country who reads the Accounting Review or who would find anything in it of value. This is the leading journal for accounting profs.

It is worthwhile to pull out “Prof Scam,” written about 15 years ago. Good read.

24

Jedmunds 08.10.05 at 11:26 am

There is also a very strong class bias here. Like the “Anna Karenina Principle” (“Happy families are all alike, unhappy families are each unhappy in their own way…”) the students at the top law schools generally had to have everything turn out right – for which an upper class background helps immensely. In turn, they don’t interact with very many students who have harder luck – and thus fail to realize the degree to which luck and birth have selected them for their positions. This leads them to over-estimate the importance of skill and ability in outcome, both their own and that of their peers. Because this group then goes on to become the next crop of professors and hiring partners, the biases get entrenched. Sad, but true.


There’s something to that, but on the other hand, you take a middle class kid who’s overcome a tad bit of adversity and put him in an ivy league setting, and he becomes pretty insufferable. Successful people tend to think the system worked. You could take the class out of it.

25

Anderson 08.11.05 at 4:09 pm

Just to add to the empirical issue: I was on a 2d-tier school’s law journal, and the editors definitely were interested in publishing articles from top-tier schools, preferring them to “better” articles from lower-tier profs.

The argument being that only by publishing these top-tier folk could we lift ourselves out of the 2d tier. I’m not endorsing this reasoning, folks, just passing it on.

So, are there no peer-reviewed, blind-submission law journals? If not, why not? What does that say about the legal academy?

Comments on this entry are closed.