How To Write Comments On Student Papers

by John Holbo on October 5, 2011

I’ve been grading papers half my life, so I think I know a thing or two about how it should go. Here’s a simple point that, I think, is not always clear to the grader him or herself (I’ve found it necessary to explain this to newbies, when advising them about how to do their jobs); that is almost never clear to the students themselves; that really ought to be to made clear – and made explicit – to all involved. There are two basic functions comments on papers can serve.

1) Explaining/justifying to the student why she got the grade she got, not the higher grade that, perhaps, she hoped for.

2) Communicating something significant that will teach the student to be a better writer/thinker.

I think graders try to do 2 but feel vaguely obliged to make 2 do double-duty as 1. And students typically expect 1, although many of them are also healthily open to 2. But 1 and 2 often come apart. It’s damned hard to provide anything that would really be sufficient to accomplish 1 in a general way. And even harder if you’re trying to do 2, too. And 2 is more important, and do-able, so basically you should just do 2. Clear your head of the vague feeling that you should be doing 1, except a bit around the edges, in the natural course of doing just 2.

Since students expect 1, you need to make it very explicit that you are doing 2 instead. ‘I’m going to pick something – your writing, your organization, your understanding of some point, something – and I’m going to spend my time and energy trying to give you a good lesson in how to do that one thing better. So if my comments consist entirely of nit-picks to the effect that your introduction is badly written, that shouldn’t be taken to imply that I didn’t read the rest of the paper, or that nothing was wrong except the introductory paragraph, or that the grade was a pure function of what went wrong in the first paragraph.’ (Obviously there’s no need to be a puritan about this. You can include a few general gestures towards 1. But don’t dissipate that precious, terribly finite quantity of time and energy, per paper, you need to accomplish 2.)

Students often want 1, more than 2, because they want to feel that the process – your standards – are reasonably transparent and fair. Students also want to be able to come in and ask about grades. All this is totally reasonable. So you need to accompany this statement that you will only be doing 2 with suitable assurances that 1 is available in a wholesale way, in office hours, etc., but it just isn’t provided, retail, in individual comments on papers. You grade fairly, but you don’t provide a separate, sufficient proof that you grade fairly, on each individual paper.

This semester I’m trying something a bit different. I’m telling students that they have to ask me, specifically, for help on particular aspects of their papers. Writing problems, thinking problems, something specific. Every paper should conclude with a little self-criticism (needn’t be a lavish self-flagellation or anything like that) indicating where they think they need improvement. My comments will be directed accordingly. If they fail to provide a little self-criticism, they get skimpy comments. So far what I’ve learned is that next time I should encourage more specificity in specific self-criticisms. (I’m getting a lot of ‘dear prof., please comment on anything that you think needs work!’)

Possibly this whole do 2) not 1) business is very obvious to most of you out there who are my fellow graders. But students don’t get it, so you need to give them a little ‘how the enemy thinks’ talk, explaining why it makes sense for things to go this way. I’ve found it really helps.

But, since this is the internet, it will probably turn out that I am very, very wrong. I am probably not a competent grader at all, since I have said these outrageous things! We shall see!

{ 221 comments }

1

Neville Morley 10.05.11 at 7:27 am

This assumes, of course, that you have the freedom to offer the kinds of feedback and comments that you feel are most useful and important, rather than having to comply with a sheaf of guidelines that tell you what feedback must be provided or else, and insist that all aspects must be covered. Ah, the joys of UK higher education.

More substantively, I think you may need a distinction between 2a and 2b: communicating something significant that will help the student become a better writer/thinker, and communicating something significant that will get them a better grade next time. In theory, doing 2a will also do 2b, but it isn’t guaranteed, and at any rate I don’t think we can afford to ignore entirely the fact that – at least in the UK – an awful lot of students are interested in 2a only insofar as it achieves 2b, and will take comments on board only if convinced that this is what will happen.

2

Sam Clark 10.05.11 at 7:48 am

Thanks for this: I try to give type 2) comments, although I have a tendency to completism which is less helpful than picking one issue and commenting on it in detail. But like Neville Morley and other UK academics, I’m under pressure, from administration and from students, to justify my marks by reference to published, allegedly precise criteria. I suspect that the wholesale model of type 1) comments wouldn’t satisfy the smart and ambitious student who comes to my office hours to contest her B+, and if I haven’t written grade-relevant comments while marking, I’m unlikely to remember that particular essay now.

3

Andrew Fisher 10.05.11 at 7:56 am

I agree with Neville on both points: (a) Feedback needs to help the student improve their performance, and for most that means primarily their performance in future assessments. I don’t want your esteem as an ‘interesting’ student, I want to pass the course. And (b) the idea that you can get away without sufficient proof that you grade fairly on each individual paper relies on you having a level of power over the students which assessors in other contexts just don’t have – hence the need for highly prescriptive forms of feedback in the UK.

4

maidhc 10.05.11 at 8:16 am

How does that work when the student’s reason for asking for a higher grade is not the fairness of your grading, but “if I don’t get a B in your course I will be disqualified from the university, and I will have to go back to my own country, and my uncle who has borrowed money to finance my education will lose his house?”

How does that work when you have to grade 150 papers per week?

Commenting more closely on 2), if a large percentage of students are having difficulty with a particular area, it might be more efficient to deal with it in the lecture rather than individually paper by paper.

My father once set an exam with just one question: “Compose a question on the subject material in this course, and answer it. You will be graded on both the question and the answer.”

You would think that the students would like that, because you could concentrate on the area that you had really studied up on. But he said he got more complaints about that exam than any other in some 40 years of teaching.

5

William Timberman 10.05.11 at 8:22 am

O tempora, o mores! Measure it by the pound, or by the inch, by the milliliter or by the hectare, but by God, you will measure it, and measure it in units any bean counter anywhere in the universe of our discontents can compare with the standards in his wee book. Or else.

I vote for the or else, and let the rest of the machinery look to itself as best it can. There must be some stony desert somewhere bereft of professors. I propose we convene there in a fortnight, or a century, and start over. Those who can’t bear the uncertainty should find a suitable cave to shelter in, and thereafter growl at anyone who appears at the mouth of it, supplicant or otherwise.

6

John Holbo 10.05.11 at 8:24 am

“(b) the idea that you can get away without sufficient proof that you grade fairly on each individual paper relies on you having a level of power over the students which assessors in other contexts just don’t have”

I think it’s important to ensure as much fairness as possible in this way. And as much transparency. But it’s also important to see that it’s totally unrealistic to try to articulate a truly sufficient, positive sense of why essay x deserves the mark it got when that is substantially a matter of comparative judgment relative to y and z and so forth. We markers can recognize that x is more clearly written than y but less clearly written than z. It isn’t feasible to try to completely articulate the basis for that relative judgment, in non-relative terms, to someone – the author of x – who can’t even be allowed to see y and z (unless the authors of those other works gave separate permission.) Since my mark is based in large part on comparisons like this, I can’t articulate a sufficient proof, in comments, that the paper deserves the mark it got, not a better one. I can, of course, say things like ‘you didn’t really address the topic’. Or ‘you didn’t make an argument’. And, of course, one should say those things. (If I seem to be suggesting otherwise, in the post, then let me make it clear: big important things about someone’s paper that can be stated briefly should, of course, get said.)

I can, of course, have a little checklist of criteria. And one should make clear what expectations are. But the trouble is: people can easily just form their judgment of what the paper deserves – seems like a B! – and then check the list in some way so it comes out getting a B. The checklists aren’t really effective ways of ‘binding’ graders to be fair if they aren’t independently inclined to be/capable of being fair-minded, aren’t independently willing to work hard to be fair. Checklists of criteria don’t do much to protect students from unfairness. Requiring graders to act as if they are demonstrating, in words, that each grade is fair, won’t actually make the grading fairer, on average. Other approaches are better.

7

Andrew Fisher 10.05.11 at 9:07 am

John @6

I think you are completely missing the point, which is not about fairness but about power. Of course there is an irreducible element of judgement involved in assessment, but in a UK context that judgment is subject to challenge and review. If you were to tell the OIA that a less procedural approach to securing fairness was ‘better’ than a more procedural one it would be roughly similar to telling the British police that driving on the right side of the road is ‘better’.

Anyway, the other point is just as important it seems to me. Who is the professional and who is the client in this relationship? When you tell your students that the feedback you want to give is better than the feedback they want to receive, it amounts to telling them that the purposes they have for coming onto their programmes and submitting their assessments are the wrong ones. If you were to treat me like this I would consider it straightforwardly impertinent.

8

Seeds 10.05.11 at 9:25 am

Anyway, the other point is just as important it seems to me. Who is the professional and who is the client in this relationship? When you tell your students that the feedback you want to give is better than the feedback they want to receive, it amounts to telling them that the purposes they have for coming onto their programmes and submitting their assessments are the wrong ones. If you were to treat me like this I would consider it straightforwardly impertinent.

“The customer is always right”?

It’s a shame that students are now clients, and that higher ed has to be a market. Isn’t it just possible that the students have come onto the course and submitted their assessments for the wrong reasons? By implicitly accepting that they are just in it for the grades, not for self improvement, you are surely making the point that they need to be taught how to learn, rather than how to buy pieces of paper for the purposes of hanging above their desk, thereby improving their economic prospects.

9

John Holbo 10.05.11 at 9:29 am

“I think you are completely missing the point, which is not about fairness but about power. “

I get your point, Andrew. But I don’t agree with it. My response was a rebuttal not just to the point about fairness but also to the point about power. (Both things are important, of course.) My point is that obliging teachers to make a show of doing things in comments that can’t, realistically, be done in comments, won’t address either the problem of fairness or the problem about power. This ends up being the grading equivalent of ‘security theater’. You shouldn’t do things unless they actually make sense. Even if there’s a serious problem.

“When you tell your students that the feedback you want to give is better than the feedback they want to receive, it amounts to telling them that the purposes they have for coming onto their programmes and submitting their assessments are the wrong ones.”

First of all, as I think the post makes clear, I actually ask the students to tell me what sort of feedback they would like to receive. I ask them what they think they need help with. My philosophy is that I should give the client what she wants! (when feasible.) I presume the client wants to get better at what I’m teaching. (If she doesn’t want that, then there’s a bigger problem here.) But sometimes students have rather confused ideas about how commenting can and should work. Then I try to explain to them how I’m going to do it, and why it makes more sense my way. And how, if they are not satisfied with their grade, they can get satisfaction by coming to talk to me about it, and I’ll do my best to explain why their grade is what it is, and if, on reflection, I have been unfair, I will change their grade.

10

Kim Weeden 10.05.11 at 9:46 am

More on the fairness issue than the content of comments …

I teach a large undergraduate course on inequality, where large = capped at 200 students. I administer in-class essay exams, 4 per term (although students can and usually do opt out of one). Errors in assessment are inevitable, both with the sheer volume of grading and students’ expectations that they’ll get their exams back within a week to ten days.

To make grading as fair as possible and opaque as possible, I double-grade every exam. (In addition to the graduate student TA assigned to the course — typically just 1, sometimes 2 for the 200 students — I hire a team of advanced graduate students by the hour to grade.) The grad students type their grades and comments into a spreadsheet so that the 2nd grader doesn’t see what the 1st grader wrote, so it’s blind double-grading.

Any exam with a full letter-grade discrepancy or greater gets triple graded, by me. I switch around teams and halves-of-the-alphabet so that students’ essays are graded by different teams throughout the semester.

This may seem like a lot of “extra” grading, and it is. Some of the time is recouped in that I rarely, if ever, get students who complain about their grades. At an institution like mine (US Ivy) and this many students, that’s rare.

But I also think that its time well-spent in that students are getting, and appreciate that they are getting, as fair a shake as possible. Double-grading was an eye-opener the first time I went this route: even with an extensive “crib sheet” for the graders that laid out my expectations for an “A” answer on each question, an “A-” answer, etc, the percentage of essays that needed to be triple graded was much higher than I had anticipated (10-15%).

11

John Holbo 10.05.11 at 9:49 am

It may be that in my post I seem to be reveling in not telling students why they get the grades they get. I’m advocating some sort of inscrutable ‘wax on/wax off’ style of paper commenting where the students don’t know why the hell I’m telling them to do what I’m telling them to do. Well, no. If it sounds like I’m being weird like that: no.

12

Sam Clark 10.05.11 at 9:50 am

A possibly-useful distinction between two ways that the student/customer might be wrong: (1) she might want something different from what would be best for her to have, e.g. she might want a qualification rather than an education; (2) she might want what would be best, but be wrong about the best means of getting it, e.g. she might think that education is best served by a point-by-point comparison between her essay and explicit marking criteria. I’m interested to know if Andrew thinks that it would be equally impertinent for John (or me) to tell a student that she’s mistaken(2) as that she’s mistaken(1), because I took John’s point to be about (2).

(By the way, Andrew’s blog is well worth reading, and particularly valuable for giving admin-sceptical academics like me some insight into how the other half lives).

13

Bill Benzon 10.05.11 at 9:50 am

My father once set an exam with just one question: “Compose a question on the subject material in this course, and answer it. You will be graded on both the question and the answer.”

I once took a final something like that — What did you learn in this course?. Loved it, but is a was a mofo because you really had to think.

14

John Holbo 10.05.11 at 9:57 am

Kim is right about how large classes should be handled. I don’t have any tutors at the moment, but when I was teaching our 400+ intro course, with several tutors, rather than establish elaborate checklists of criteria (tried that years ago, didn’t think it worked), all papers were read twice – once by the student’s own tutor, once (blind) by another. Then, in cases of serious discrepancy, I made the final call. What I learned from that experience is that the major source of bias is tutors who are too nice to a few of their own students.

Also, make sure all tutors are giving about the same average grades (or else find out the reason why not).

Of course, this isn’t practical when it’s just me, myself and I doing the grading.

Making people write more – or different – styles of comments is, in general, not a way to improve fairness in grading.

15

Andrew Fisher 10.05.11 at 10:00 am

John@9. On point (a) I’m afraid that you hadn’t made that point clear to me, at least. I apologise for misunderstanding you. I was attributing to you the kind of attitudes that Seeds shows @8, and having myself experienced such attitudes from university teachers both directly as a student and indirectly as a colleague I was too quick to make assumptions.

On point (b) I think we are still talking past each other. I think you are effectively saying that it doesn’t ‘make sense’ to operate within the legal and regulatory constraints that UK academics & their institutions do, in fact, have to operate within, which doesn’t make much sense to me. So I am probably misunderstanding you on this point too.

16

Seeds 10.05.11 at 10:06 am

While we’re reminiscing, my favourite lecturer at my university gave me the best feedback I’ve ever had, and it was definitely of the 2) sort. The grade was good, but not alluded to in the comments section, which gently pulled me up on everything from my abilities at counting electrons to my use of hyphens in compound structures.

The same lecturer was in charge of student examinations and set one of the finals – having freely admitted in a tutorial that his technique was to set something interesting, see how everyone did (“everyone” being the five of us taking that particularly course) and then grade us accordingly.

I still occasionally have strange dreams about his particular question, which was a full page diagram of one metal atom, inside a dodecahedron of other metal atoms, inside a god-knows-how-many-sided shape of yet another sort of metal atoms and the instruction to pick any three obscure spectrographic techniques that we liked and to explain what we would see and why. Brr.

17

J. Otto Pohl 10.05.11 at 10:12 am

Well I have only been grading for a little less than 10% of my life. But, given that I have 80 mid-terms for Early Modern European History to grade I would prefer not to do it. The purpose of grades seems to have become to provide outside employers with a way of evaluating potential workers without having to expend any resources on the process themselves. I would rather just turn over my tests to the Ghanaian Foreign Ministry or the Ghana Commercial Bank, or to the various graduate schools in the US and UK to grade since they seem to be the ones that actually use the grades.

18

Seeds 10.05.11 at 10:24 am

Andrew at 15 – fair enough, John’s answer was certainly more interesting and reasoned than mine. (Which isn’t to say that I don’t stand by what I said.)

I also thought Sam’s distinction at 12 was useful, and John’s comment at 9 makes it clear that we are talking about the best way to help students/clients who do want education rather than qualifications.

So taking that as given, is it unreasonable to attempt to improve the quality of feedback, if you are still marking with legal and regulatory boundaries and your justification for grades is available to students that want it (just not written on the actual piece of work)?

19

Andrew Fisher 10.05.11 at 10:32 am

Sam@12. I was thinking primarily in terms of (1), not (2). (1) is what seems to me to be impertinent. If I value and respect your perspective and want to learn how to be what you think a ‘better’ thinker or writer is, and you have the time and willingness to mentor me in that way, then that’s great. But I may not have that much respect for you, just as you may not have that much time for me (almost certainly don’t if you work in a UK HEI…). Likewise if you and I differ about what is ‘best’ for me to have, well frankly it is up to me whether I value your opinion on that or not and the default position is surely that you keep your opinion to yourself unless I ask you for it. This is just common courtesy. Obviously I erred in reading the OP to imply anything like (1), for which I have already apologised, but John’s presumption @9 that students are in his class to get better at what he thinks he is teaching them, whilst no doubt fair and reasonable in his own case surely can’t be generalised. HE teaching certificate courses for new lecturers in the UK might be an impish example, but I’m going to use it anyway.

Your example in (2) I find a bit implausible, so I may be missing another point here. Of course students, like anyone else, can be wrong about the best way to achieve their goals. In an ideal world one would offer them professional advice and let them make their own choices. An analogy would be that when I hire an architect I’m open to him or her suggesting a better design for my home improvement but ultimately I expect to decide what work will be done – even if the Architect thinks I’ve made the wrong decision. But this kind of analogy fails because of the radically different constraints operating on professional architects and professional academics.

20

Andrew Fisher 10.05.11 at 10:50 am

Seeds@18

See Sam@2, I think.

21

Steve LaBonne 10.05.11 at 11:21 am

(1) is what seems to me to be impertinent.

Well, yes. Offering students a real education, from the time of Socrates (whose notorious impertinence I should think makes the point obvious) onward, is indeed an act of impertinence, precisely because it is predicated on the notion that the master knows better than the students what they need by way of intellectual nourishment. (As I used to remark in my academic days, “If the students know better than we do what they need, then why are they paying us?”) And so we have largely dispensed with education altogether on those grounds, since customers dislike impertinent merchants. Can you look at our society and tell me with a straight face that this has been a good thing?

22

Matt 10.05.11 at 11:23 am

I think you are effectively saying that it doesn’t ‘make sense’ to operate within the legal and regulatory constraints that UK academics & their institutions do, in fact, have to operate within, which doesn’t make much sense to me.

There’s a way to read this that makes sense, something like, “these are the rules, and you have to follow them, and if you don’t, it will be a lot of trouble.” That’s often so in life, of course. But I take the people objecting to be suggesting that the rules here are bad ones, and there should be better ones. How could that not “make sense”? It might be wrong, but it’s surely not nonsensical. Given the way that, to an outsider, British higher education seems more and more to be turning into a nightmare of bureaucracy, administrative bloat (even more than in the US, where there’s a very similar problem), and highly dubious “objective” measures, the idea that these standards ought to be questioned seems very reasonable.

23

Andrew Fisher 10.05.11 at 11:32 am

Steve @21

I you are going to cite Socrates in defence of the practice of being paid to teach, then clearly educational standards have fallen somewhat – or at any rate the curriculum is getting hollowed out.

24

Seeds 10.05.11 at 11:36 am

Hi Andrew – sorry again for my earlier ill-humour, I find your point really fascinating (no sarcasm intended).

If we are considering students who simply want a qualification, no learning, then when you say:

(1) is what seems to me to be impertinent. If I value and respect your perspective … But I may not have that much respect for you…

the student is surely asking for respect (i.e. lack of impertinence) without providing any in return (i.e. willingness to be educated).

The architect analogy fails I think, not because of the different constraints operating on architects and tutors, but because of the different kind of service offered by them. The client in both cases is aware of their ignorance of the subject, but in the case of the architect, they are employing someone so that they do not need to personally overcome their lack of domain knowledge – their ignorance matters no more at the end of the process than the beginning. In the case of the tutor, the employee is there to give them the knowledge, and so the client’s ignorance relative to the architect is pertinent. To put it another way, the client who employs an architect to build a house can know nothing about how their house was designed at the end of the process, and still have a new house. The client who employs an architect to teach them architecture cannot remain ignorant at the end of the course and still be a new architect.

Unless you take the strong position of universities offer qualifications, learning as an optional extra (in which case, fair enough, but let’s be clear).

I think Sam at 2 is describing impracticalities with John’s suggestion, due to type (1) students – but the impracticalities are not an argument against the concept of providing better feedback in terms of learning rather than grades, so long as the academic’s back is covered in terms of their legal obligations.

25

Sam Clark 10.05.11 at 11:40 am

So, it looks like everyone agrees that a student could be wrong about the best means to her goals, and that a teacher may properly correct her about them, and offer – with explanation – feedback which isn’t what she was expecting.

But now I’m interested in my (1), because I don’t think that the disagreement between my views about what education is for, and the views of some of my students, is just a clash of preferences.

Expanding for my own UK-based experience: some of my students have been taught, especially by their A-levels, to have an instrumental and formal approach to education. They see the point of what they’re doing as finishing qualifications in order to move on to the next bit of their lives; they expect those qualifications to be distributed according to explicit formal criteria. They want to be able to see just how much work of what kind they need to do to get a 2.1. This isn’t any sort of criticism of them, by the way: it’s the predictable result of their education up until they get to university.

I, on the other hand, think that education – especially at university, and especially in my subject (philosophy) – should be an open-ended process of self-development. That requires frustration and lack of closure. It requires reshaping oneself by engaging with troubling works of genius – Hume’s second Enquiry, for example (just because I’m about to start teaching it again). This process has different and maybe even clashing means, because it has different goals, from the instrumental and formal idea above.

The distinction between these two isn’t just ‘Sam likes chocolate icecream, but others prefer strawberry’. It’s a difference between ideals of the good life. I therefore don’t think that polite tolerance of difference is the appropriate response to it, especially when the practical outcome of that tolerance is to give in to the instrumental ideal.

26

John Holbo 10.05.11 at 12:04 pm

Just for the record: if it turns out my approach is illegal in the UK, I’m not advocating civil disobedience. I’ll leave that up to the individual consciences of UK graders. But my experience is that there is not much value in replacing a simple ‘you got a B’ plus comments with some sort of ‘objective’ matrix where the student is informed she got a

2 for organization
2 for clarity
2 for demonstrating understanding of the material
3 for general content
2 for construction of argument
1 for addressing the topic
0 for originality
for a total of
12 (which – consult table – is a B)

Because now, instead of wondering why you got a B, and your friend got a B+, you end up wondering why you got 3 for ‘general content’ when your friend got 4, or whatever. And you wonder what’s the difference between ‘content’ and ‘argument’, or ‘organization’ and ‘clarity’, or why your take didn’t count as ‘original’. (There are always imponderables, where these sorts of schemes are concerned.)

If you are required by your department or school or country to do something of the sort, to ‘ensure quality’, then you have to do it. But, realistically, it won’t ensure quality. What happens when these sorts of schemes are imposed is that graders tend to grade the paper, decide it’s a B, and then assign numbers accordingly. So the whole thing ends up backwards. It doesn’t do what it appears to do. It doesn’t ensure consistency across graders, and it doesn’t defend students against capricious grading. It isn’t, in the end, helpful to students. It makes work for graders, whose time is better spent trying to write something that would actually help the student learn to write better, even it that something didn’t do double-duty as an attempt to explain why the student got the grade she got, rather than the one she wanted.

27

Steve LaBonne 10.05.11 at 12:09 pm

If you are going to cite Socrates in defence of the practice of being paid to teach, then clearly educational standards have fallen somewhat – or at any rate the curriculum is getting hollowed out.

Which is what I was saying, yes. Treating the offering of actual education, as opposed to credentialing, to students as an impertinence to be deprecated, is bound to have that result. What I find curious is that, unless I’m badly misunderstanding you (which is entirely possible), you seem to regard this as a good thing.

28

Steve LaBonne 10.05.11 at 12:23 pm

And note that, in case you thought I didn’t get your little joke, one most certainly doesn’t have to be Socrates to be a real teacher as opposed to an assembly-line credentialler. So while it may seem clever to attribute the decline to the fact that universities are staffed by mere mortals, that’s not the case.

29

tomslee 10.05.11 at 12:23 pm

What happens when these sorts of schemes are imposed is that graders tend to grade the paper, decide it’s a B, and then assign numbers accordingly.

I’m sure this is a general phenomenon; at least, exactly the same thing happens on performance reviews in employment situations.

30

Andrew Fisher 10.05.11 at 12:31 pm

Seeds@24. Please don’t apologise: I’m enjoying myself.

‘the student is surely asking for respect (i.e. lack of impertinence) without providing any in return (i.e. willingness to be educated).’

I think you mislead yourself into false equivalence here. Lack of impertinence is the routine level of respect I expect to accord to and be accorded by everyone (except maybe on the internet) whereas a willingness to educated in the strong sense of that word you wish to use (rather than just the sense which suffices to pass exams) is a very high level of respect which I accord to few and never expect to be accorded to me at all. I’m not saying that the students are paying so they can do anything they like, but rather that your expectation of respect from your students should be lower – you have a right to common courtesy and to be treated like a professional, but no right to be regarded as a mentor.

And clearly the actual fact of the matter is that universities do offer qualifications, learning as an optional extra. I can certainly respect your regret that this should be so, but as a statement of fact it is incontrovertible.

The architect analogy is a poor one, and I don’t want to press it further, but to clarify the point that I was trying to make my architect does two things for me (a) ‘you’d be better off taking out this wall rather than that wall, as you’ll get more light’ and (b) ‘if you remove that load-bearing wall you’ll need such-and-such a steel beam to replace it’. My analogy was tring to get at A, not B which I agree isn’t even a poor analogy for education.

Sam@25. Isn’t Steve@21’s reference to Socrates appropriate here? Socrates refused to teach in exchange for money because, like you, he had a higher view of what teaching was than is compatible with providing a service in exchange for money. Unlike Socrates, I don’t actually myself think there is anything wrong with teaching for money, but I do think that if you are going to take the money you have to provide the service that is asked for even if it is not the ideal you would have aspired to outside the cash nexus.

31

Steve LaBonne 10.05.11 at 12:41 pm

Unlike Socrates, I don’t actually myself think there is anything wrong with teaching for money, but I do think that if you are going to take the money you have to provide the service that is asked for even if it is not the ideal you would have aspired to outside the cash nexus.

The problem is that the service you’re discussing isn’t teaching. It’s a combination of somewhat rote training with credentialling. That’s why you and a number of other commenters on this thread are talking past each other. As charitable as I try to be, I can’t help but read all of your comments as suggesting that it’s wrong for professors to try to provide actual teaching when many of the students think they’re paying for something else. Pardon my density, but if that’s not what you’re saying, a clearer explanation of what you are trying to say would be useful. And if it is what you’re saying, the word “impertinent” is doing far too much work and needs some unpacking.

32

John Holbo 10.05.11 at 12:53 pm

“I’m not saying that the students are paying so they can do anything they like, but rather that your expectation of respect from your students should be lower – you have a right to common courtesy and to be treated like a professional, but no right to be regarded as a mentor.”

Actually, ‘mentoring’ is one of the things we educators are standardly supposed to do. Sometimes I’m actually assigned as a faculty ‘mentor’. It depends on what you mean, of course, but that’s the word that gets used in a lot of cases. (Hey, it’s a school. What did you expect? Teaching/mentoring. You say potaytoe, I say potahtoe.)

You seem to be saying, Andrew, that teachers – or teachers in higher ed? – should confine themselves to providing the instrumental means for students to reach their ends. But they are not qualified (permitted, professionally?) to dictate what ends a student shall strive for? Is that about right?

33

Andrew Fisher 10.05.11 at 1:04 pm

John@32
What have qualifications got to do with it? On what grounds could you possibly be permitted to dictate what ends any other adult human person should strive for? And who could give permission, except for the human individual in question?

If you pay me, you might reasonably dictate what I do during the time for which you pay. But the students are paying, aren’t they? So if anyone is in a position to dictate it is them.

If you don’t wish to work on such terms then of course you are free to refuse employment in the dehumanising contemporary university. Or you can strive to rescue such shards of true learning as may be possible by identifying a few students who feel as you feel and offering them a proper education *on top of* the standard service.

But currently no-one is trying to deny that the service the vast majority of students think they are paying for is the instrumental one. You are engaged in trying to persuade me that it is ethical to take their money and not give them what they have paid for.

34

David Moles 10.05.11 at 1:14 pm

Andrew, if I understand your position, it’s that the university exists to provide a service, i.e. the granting of a certificate indicating that the bearer has jumped through a certain series of hoops (possibly set by statute); the professor should confine him or herself to holding the hoop at the specified height and angle, and if a certain amount of kibitzing is inevitable the professor should nonetheless not expect the student to look to him or her as a source of hoop-jumping expertise, or take any heed of his or her advice on the student’s hoop-jumping technique, which nattering is in any case a distraction from the student’s goal of making it to the next hoop. Am I getting this right?

I’ve certainly seen (and taken) some courses that worked this way, generally introductory lecture courses in the sciences, nominally taught by research professors. It served the university’s purpose, namely weeding out of the sciences all those students not sufficiently talented, prepared and driven to learn the material on their own without aid of instruction, but as pedagogy or even as customer service it was balls.

35

Steve LaBonne 10.05.11 at 1:16 pm

Well, clearly I did not misread Andrew F. I don’t know a lot about what has happened in Britain (though what I do know sounds pretty dire), but here in the US we still have a lot of colleges and universities that quite clearly advertise themselves as educational institutions, and some of them actually continue to deliver on that promise at least some of the time. Students who attend such institutions have no cause to claim they are being defrauded when the institutions attempt to deliver the goods that they advertised. Furthermore, the prestige of the better colleges and universities, which is vital to the value of their degrees as credentials, is closely bound up with the fact that they have not yet thrown education completely overboard.

36

David O. 10.05.11 at 1:22 pm

A Pedant Writes:
Please use ‘they/them’ instead of ‘he/she/him/her’ when it’s desirable to avoid getting into a tangle over gender. I was taught this back in the 1950s in a London ‘grammar’ school (state) with a long and elitist tradition. So it must be correct usage!
As with economics – it’s amazing how much learning gets forgotten…
Now, what were you saying before I get distracted?

37

Steve LaBonne 10.05.11 at 1:27 pm

It served the university’s purpose, namely weeding out of the sciences all those students not sufficiently talented, prepared and driven to learn the material on their own without aid of instruction, but as pedagogy or even as customer service it was balls.

Indeed, and it is no accident that in the US the holders of undergraduate degrees from liberal arts colleges, which are not entirely free of this sort of thing but where it is at least not as bad, are significantly overrepresented among those who go on to earn Ph.D.s in the sciences as compared to graduates of large universities.

38

Andrew Fisher 10.05.11 at 1:48 pm

David @34
If you mean by ‘hoop-jumping’ what I think you mean, then no I don’t mean that.

Sam’s post @25 puts it really well – he has a vision of what learning Philosophy is or can be which is not just a bit different from what his students (mostly) think learning Philosophy is – they are not even related ideas. The processes that might deliver Sam’s vision are – to say the least – unlikely to be the same as those which deliver what his students think they want.

It therefore can’t be right for him to provide what he thinks is good and right rather than what they think is good and right. As he says himself, this isn’t a mere question of flavours such as chocolate or strawberry, but of him imposing his idea of what the good life is on them, and I would add under false pretences since we all agree that this is not what the students (or the vast majority of them) wanted or expected when they parted with their money.

Now one can imagine some completely different institution than the contemporary university in which different kinds of students gather for different purposes. You might prefer to work in that institution, and so might I, but the point is that none of us do. The way you could properly behave if you worked there is quite different from the way you can properly behave in the real, existing contemporary university.

In the contemporary university it is precisely expert assistance in hoop-jumping which most students want and which meets most of their purposes. You have every right to be horrified by that, to oppose it, to campaign against it: but not on time which the students have paid for.

39

John Holbo 10.05.11 at 1:54 pm

“On what grounds could you possibly be permitted to dictate what ends any other adult human person should strive for?”

Well, I might be a teacher, to pick an example not wholly at random. Suppose I am running a guitar school and no one can graduate without learning some scales and chords. I say so. The student merely desires the end of getting that precious, precious piece of paper, saying he graduated. I could give him the piece of paper without bothering about the whole scales and chords thing. I could decide it’s enough if he can play eight extra songs by heart. He thinks that should be enough. But I’ve decided I’m going to insist on the scales and chords, too. I am, in a certain sense, dictating an end towards which he now must strive. Yet he is an adult. Is this intolerable behavior on my part, imposing my ‘must know not just songs but scales’ standard on him?

In general, how is your prohibition on dictating ends to any other adult consistent with establishing any sorts of curricular standards in adult education? Shouldn’t all adults be able to set their own curricular standards – decide for themselves where the hoops should be – on pain of having someone else decide for them. Which would be intolerable?

But that seems absurd, no?

40

Steve LaBonne 10.05.11 at 1:56 pm

Andrew, that would work equally well as an argument for why physicians should provide patients with the treatments which they think they want, not the ones which the physician thinks they need. In medicine we would rightly regard this as highly unethical. You need arguments, not merely ex cathedra pronouncements from your lofty perch as a “higher education planner”, for why things should be very different as regards education. The next such argument you produce will be the first.

41

mpowell 10.05.11 at 2:20 pm

Kim, that’s fascinating to hear. I’ve never heard of this being done, although it makes a ton of sense. I’ve definitely experienced or heard of classes where it seems highly likely that there is some substantial grader to grader variation in standards and this is obviously quite unfair.

42

Seeds 10.05.11 at 2:26 pm

Speaking of physicians, doctors are certainly a profession in which I would prefer to know that the paying client had received an education, rather than merely a qualification. Unless you have faith that no further knowledge or ability is required to be a good doctor than that required to pass an exam at medical school, it follows that the education is there to provide more than simply the qualification. Doctors need to be well-educated for the good of society, rather than simply for the good of undergraduate medical students looking forward to a lucrative career in proctology.
A less extreme form of this argument could be made for any social good that the university provides (making well-rounded students or whatever). Treating education as a simple business transaction is a rather arbitrary and short-sighted view.

43

Seeds 10.05.11 at 2:31 pm

Also, although it seems facetious – paying clients presumably don’t like failing, either. By taking only the students’ views into account, you end up with a situation in which grade inflation makes these expensive qualifications less meaningful to both students and employers. I’m told that this has become an issue for US universities.

44

Andrew Fisher 10.05.11 at 2:32 pm

John@39 I agree, and I erred in attributing to you the kinds of view that Sam set out in his comment 12 case 1. You set out the curriculum, the means of assessment and so forth and you support the student in achieving the necessary marks in the assessments which have been set. If the student think it is enough to learn 8 songs by heart to pass the course, and that isn’t what the module specification says, this is an error of Sam’s comment 12 type 2, just as if students don’t know the difference between the subjunctive and the dative this is an error and a teacher should correct them.

The problem arises if the students are signing up to learn guitar, and you are trying to revolutionise their whole lives as in Sam’s comment 25. But you have never offered this position, and I only attributed it to you via misunderstanding.

Steve@40. I’m not clear what I did to deserve that ad hominem. Whilst I am not an expert on medical ethics by any means, when I have been operated on in the past it has been with my consent. I have never argued that students are entitled to anything they want: clearly if I were to want accupuncture for a broken arm my surgeon might refuse to provide it. I have only argued that you cannot take my money for one service, whilst providing something different which I do not want. It would be unethical to agree to give me accupuncture, take my money for the procedure, and then do something else which I have never consented to.

If students want to use their higher education a a mere means to some other end you are more than welcome to refuse their money; but if you take the money you must provide the service requested and not some service which isn’t wanted and which, as Sam says in his comment 25, has radically different goals and outcomes. Higher Education can’t be likened to emergency medicine in which you have to operate without the patient’s consent.

45

Sam Clark 10.05.11 at 2:37 pm

If the situation were that my institution had advertised lessons in hoop-jumping leading to a certificate in advanced hoopla and nothing else, and our students had paid our price, and then on the first day I’d told them that I had a personal moral objection to hoop-jumping, and that I was instead going to force them to practice my own favoured hobby of invisible ring-juggling – then Andrew would be right, and I’d be in the wrong.

This is not the situation. The situation is that our students are paying (mostly through loans) to come to university, which they’ve heard is important. They have a variety of expectations, desires, and plans about what that will be like and where it will lead; they’re admirably open to being surprised by joy in something different; they’re young, clever, and ambitious for… something, they’re not sure quite what.

My options are: (1) run them through the credentialling routines, collect my pay-cheque, go home; or (2) offer them something that challenges them and me to develop ourselves in ways which go beyond that. (1) would be easier. (2) strikes me as the honest thing to do, even though it sometimes puts me in conflict with the expectations of some of my students and the demands of my university’s administration. It’s certainly not the case that these expectations and demands are entirely consistent and clearcut. It’s also certainly not the case that my only moral options are to do exactly what I’m told, or to quit.

46

Steve LaBonne 10.05.11 at 2:51 pm

I have only argued that you cannot take my money for one service, whilst providing something different which I do not want.

The problem here is again a word that’s doing too much work, in this case the word “service”. There are services- medicine and education are two- which are quite different from more “mundane” services in that the service provider is an expert and when I pay for that service I do so precisely for the purpose of placing myself in the hands of someone who, in a particular domain, understands my needs better than I do. (Yes, my consent is required; who here has advocated shanghaiing students into universities?) If a student enrolls in an educational institution when what s/he really wants is narrowly vocational training plus a certificate, the student has made an error, which is properly rectified, not by destroying all educational institutions and turning them into purveyors of what is done better by (in the US) community colleges and “career institutes”, but by the student leaving to seek a more appropriate service provider. And I do in fact hold the obvious corollary of this, that there are too many students attending traditional colleges and universities from which they are not equipped to benefit. (I of course must plead guilty to being able to hold this position, whose implications are admittedly rather callous with respect to the career prospects of academics, comfortably because I long since ceased to be one myself.)

I note as well that you continually draw back from the obvious implications of your stated position whenever challenged, as you did with respect to John’s #39. This is a sign of insufficiently thought-through ideas.

47

christian_h 10.05.11 at 3:04 pm

Let’s just be clear that Andrew’s premise – that it is students who are paying the educator – is, in most cases, utterly wrong. It’s taxpayers, parents, alumni etc. who pay. Even if we bought into Andrew’s capitalist vision of the university, the students wouldn’t be our “customer” any more than the readers are the customers of newspapers.

48

ptl 10.05.11 at 3:11 pm

Andrew, till you agreed with John’s 39, I suspected you of trolling. I now think Steve (46) is right, you simply haven’t thought this through.

Let me ask how your revised

“The problem arises if the students are signing up to learn guitar, and you are trying to revolutionise their whole lives”

can in any way justify the kind of stuff to which Neville (1) is — I infer — subjected.

And let me object in the strongest possible terms to your

“but not on time which the students have paid for”

Would you give extra time to the overseas students admitted simply for financial reasons, who pay higher fees? (I’d give them equal time.) Would you give less time to students whose fees are waived? (I’d give them the same amount.)

Last, did students demand, e.g., course descriptions that include a section on Aims and a section on Objectives (etc.)?

49

Jonathan Mayhew 10.05.11 at 3:14 pm

To get back to original topic (not that other issues aren’t relevant), I grade mainly to provide opportunities for improvement. If I do this right, it also justifies the grade along the way, so I don’t have to do that separately. I agree with Holbo’s comment 26 that rubrics tend to justify the grade one would already be giving.

50

subdoxastic 10.05.11 at 3:16 pm

@ Prof. Holbo:

Your discussion of x paper being better than y paper but not as good as z paper is (I assume) one of the typical, and traditional approaches to grading normatively. Which of course raises all sorts of difficult problems about the purposes/aims of education. Recognizing that, many have advocated a switch from norm referenced assessment to criterion reference. Of course, criterion referenced assessment is often guilty of the vagueness you describe in your “2 for clarity, 2 for organization” example you provide above.

But this need not be a fatal flaw– it merely requires a more specific and detailed list of criteria (what is counted as elements of clarity, etc.). Yes, this is difficult to do but it might go a long way to addressing the needs of students and the needs of institutions as well as a much better job of incorporating both examples 1 and 2 of the grading you discuss in the O.P.

51

Aulus Gellius 10.05.11 at 3:41 pm

Andrew Fisher @44: “if you take the money you must provide the service requested and not some service which isn’t wanted and which, as Sam says in his comment 25, has radically different goals and outcomes.”

Well, it gets a bit complicated, because universities of course don’t individually negotiate terms with each student. But, at least in the US, it’s quite clear that a lot of students DO expect life-changing, wise, mentor-y guidance on a very general level, and colleges quite explicitly advertise themselves as providing it. When university presidents give speeches to incoming freshmen, they don’t say “we are going to teach you a limited set of concepts, of course leaving your core beliefs about what is important untouched.” They talk about teaching citizenship and inspiration and community and blah blah blah. So I think a student who made your complaint would be in the position of a customer who failed to read the extremely large print on the agreement.

52

John Holbo 10.05.11 at 3:45 pm

“Yes, this is difficult to do but it might go a long way to addressing the needs of students and the needs of institutions as well as a much better job of incorporating both examples 1 and 2 of the grading you discuss in the O.P.”

Well, I don’t want to deny that a truly well-thought-through matrix approach might work, ok, but – then – a truly well-done version of my approach obviously works ok, too. I think we are obliged to be a bit more pessimistic than the best case scenario. The problem with adding more specifics and detail is, in my experience, that the problem gets worse rather than better. Suppose you have a matrix with (let’s say) 5 basic areas of assessment and (for neatness) five different sub-areas within those 5. Organization turns out to be a function of five sub-aspects of organization. So now you really know what ‘organization’ means!

So now you have 25 little boxes to put some value in. And you add it all up and that’s your grade.

I just don’t think that you can realistically get graders to think rigorously in such terms. They won’t be able to separate those sub-aspects of organization in their mind, much less consistently identify them in the papers. They won’t all agree about what these sub-aspects amount to. They will have a better immediate of the overall organization of the paper than of the presence or absence of these sub-varietals of organization whose presence or absence is supposed to be criterial of organization overall. They will have a better sense of the overall worth of the paper than of its overall organization. So it won’t really be the case that you have standardized things by mandating a fine slicing process.

The basic problem is that values get more elusive, not more ‘basic’, when you slice them this fine.

I’m not saying it could never work, but it doesn’t sound like a good plan to me. It sounds like a recipe for bureaucratic make-work.

53

Jonathan Mayhew 10.05.11 at 3:46 pm

Soul-deadening social-science jargon aside, if you “reference” based on “criteria” and that gives you a different result than if you had “referenced” based on “norms,” then that is unfair. You can’t give a better grade to a worse paper because it turns out better when you strain it through the rubric. That means you’ve applied to rubric wrong or the rubric is flawed to begin with. So what most graders would do is adjust the scoring in order to make the better paper come out on top anyway.

54

Bloix 10.05.11 at 3:48 pm

#10 – Prof Weeden – is the first professor I have ever heard of who makes a genuine effort to grade fairly. At least this level of objectivity is absolutely required for any degree of reliability. If you believe that you can run through 200 essays over a period of a week, reading each one once, and rank them fairly, you are kidding yourself.

Any fair grading system would absolutely require at least the following:
1) a set of impartial standards, requiring scoring for different criteria and a weighting system for the importance of those criteria, devised at the time the test or essay is devised, that is applied rigorously to the results.
2) separate grading by two graders skilled in the field (not TA’s with a year of grad level course-work that is vaguely related to the course topic).
3) an initial read of the work to score by the criteria.
4) A review of all the papers to make sure that the scoring has been done consistently.
5) A tallying of scores, a ranking, and an assignment of grades based on the ranking.
6) A comparison of the two sets of grades and a re-grade in the event of discrepancies.
7) Random review of a certain number of papers by a third person to make sure the two graders are not both making the same errors in applying the criteria.

And none of this deals with an even more important issue, which is that grading as between different professors teaching similar material is wildly inconsistent. To deal with that, grading standards would have to be imposed at the departmental level. Tests and paper assignments would have to be vetted by committees, and grading by one professor would have to be reviewed by other professors.

Prof Weeden’s system, which she funds herself, apparently, is a giant step in the right direction, but it’s not nearly enough. For example, she checks only those papers that are a full letter grade apart. This practice has at least two flaws:
1) if the TA’s err consistently in the same direction (e.g. grading papers that show rhetorical skill higher than those that are substantively superior) Prof. Weeden will not detect it.
2) At most top-tier schools, the vast majority of papers are given A’s and B’s. Let’s be conservative and say that in Prof. Weeden’s classes all but a few failing papers are given A’s, B’s, or C’s. This means that effectively, there are 9 grades. A difference of a full letter grade (A- to B-) is a 1/3rd difference. So a discrepancy can be over 20% (e.g., A- to B, = 2/9 = .22) and Prof. Weeden won’t check it. Presumably she averages it – something she would never do, I would guess, if data for her own research revealed discrepancies of that magnitude. So, for example, one grader could be 20% harder than the other, or one could show wider variability than the other, and Prof. Weeden would trust in averaging to smooth out the discrepancies.

This isn’t at all to criticize Prof. Weeden, who does more than any professor I’ve ever heard of to be fair and consistent.

Universities are chock-full of people who know how to do study design in order to gather data fairly and how to aggregate, analyze, summarize, and interpret that data, but apparently none of that knowledge is ever applied to the major data-creation-and-analysis function of the university itself: grading.

Why not? I would give three reasons:
1) Doing it right would be ungodly expensive and time-consuming. It’s cheaper to do it wrong, and there’s no constituency to force you to do it right. The consumers of the data (grad schools and employers) apparently don’t care that the results are not reliable.
2) Doing it right would infringe on the autonomy of professors, who don’t want anyone else to tell them how to test and grade.

But students are well aware that the system is arbitrary. This is why they come to your door and whine and wheedle. It would be one thing if the grade had been assigned by a process that is transparent, objective, and reliable. But it wasn’t. So why shouldn’t they try to get a better grade?

55

bianca steele 10.05.11 at 3:49 pm

I should start out by saying that “grade s/he wants” seems like a peculiar turn of phrase to me.

I suppose what you need to do may depend on your students and what their problems are.

For me, many of my courses as an undergraduate began with a little speech about grade grubbers who come to office hours to beg for a half-grade raise on a test, which of course, communicated not to come to office hours.

I only complained twice, I think, once for a grade that was mathematically impossible even if I’d gotten a 0 on the final exam, and once in a computer science course with essay assignments–in that case, seeing that the TAs had checklists of topics covered, and that the topics matched the textbook, was very clarifying. (Though I’ve no doubt a decent high school produces college students who already know that.)

56

Bloix 10.05.11 at 3:49 pm

Oh dear. You know, there are 3 kinds of people in the world – those who can count, and those who can’t.

57

John Holbo 10.05.11 at 3:51 pm

“I grade mainly to provide opportunities for improvement. If I do this right, it also justifies the grade along the way, so I don’t have to do that separately.”

This is often the case, but not always. Maybe I exaggerated the ‘not always’ for post purposes.

58

bianca steele 10.05.11 at 3:51 pm

I was incidentally reminded, by the OP, of an interview I did where my last question was, “What are you best at, what are your best skills?” and got the answer, “I’m best at everything,” which two or three follow-up questions couldn’t narrow down. So I’m willing to believe some students have room for improvement in the area of self-criticism.

59

Andrew Fisher 10.05.11 at 4:03 pm

Christian_h@47. I obviously can’t speak for all places, institutions and jurisdictions, but for the part of the anglosphere I know, the idea that the student pays is a simplification, rather than ‘utterly wrong’. There are certainly very few students studying at English universities without paying substantial fees.

Sam@45, you have moved your perspective quite some way from your comment 25. Of course one can overemphasise the degree of tension as well as the degree of alignment between student and staff goals.

ptl@48. I have admitted to enjoying myself so can hardly resent your suspicions. I am not trying to offer a justification of institutional feedback policies in the UK. they exist, and they exist for reasons, and I could explain those reasons; but I fully accept John’s criticisms of such approaches – the reasons are not necessarily powerful educational reasons. I think your objection to ‘time the students have paid for’ a little overblown. We don’t bill students by the hour, and I haven’t suggested that we should. What I am saying is that we should prioritise their goals, their ideals and their understanding of what the good life is during the time that we set aside to their professional service.

60

Matt 10.05.11 at 4:09 pm

I’ve graded a lot of papers, though surely not as many as John. Often, I write a lot of notes in the margins while I’m doing it, and underline things, and put little checks and the like. Normally, these are as much for me as for the student, so that I can remember what I thought was good, bad, confused, etc. about the paper. I then write a paragraph or so on the back trying to sum up the comments, say what was good or bad, and offering advice. This process has gotten easier in some ways by having students submit papers electronically and using the “comment” button from track changes- easier to read what I write and so on, and easier to change the comments afterwords to make them more useful for the student and not just me.

As for Bloix’s comment, I expect that time and money are the big factors preventing grading more like you’d like. Grading is extremely time consuming, if done right, as-is. Hiring more people to do it would be expensive. But, some approximations can be done. When I first was a TA, the professor for whom I graded had me give her some representative papers from various grade points, and she read them again and discussed them with me. It was very useful, and I learned a lot. When I’m grading, I usually assign a provisional grade, and then go back over the papers before assigning a final one to try to make sure I haven’t let me standards move around during the process. It’s not perfect, but it does help take out some variation.

61

piglet 10.05.11 at 4:12 pm

“I think graders try to do 2 but feel vaguely obliged to make 2 do double-duty as 1. And students typically expect 1, although many of them are also healthily open to 2.”

My experience is slightly different: graders almost never write any comments (I’m referring to graduate research papers here – undergrads usually don’t even get their work back and nobody has time to read it anyway).

Maybe things are different in philosophy.

62

Bloix 10.05.11 at 4:23 pm

#46- “I do so precisely for the purpose of placing myself in the hands of someone who, in a particular domain, understands my needs better than I do. “

Does anyone believe that the average university professor understands the needs of students better than the students do? What training does a professor have in the needs of students? What incentive does a professor have to learn about the needs of students? Are professors hired, rewarded and retained for their ability to understand the needs of students?

Most professors’ understanding of students comes from their own experiences as students. But professors were unusual students – they were students who wanted to become expert in a field of academic study. Most students don’t want that: they want to enter some non-academic profession: business, industry, law, medicine, journalism, civil service, whatever.

But professors generally know very little about the world outside academia. They know how to be academic specialists and they know, more or less, how to teach others to become academic specialists.

Universities used to have core curricula. Those that still do – in the US, afaik, that means two, Reed College and Columbia University – can make a claim that they are teaching students general knowledge that they need to know as citizens of a democratic society. Those schools can say, we know the students’ needs better than they do. How a professor at any other school can make that claim is beyond my comprehension.

63

mpowell 10.05.11 at 4:26 pm

Andrew, I’m sorry but your argument is just terrible. Even if we grant that a university is nothing more than a credential granting service providing expert advice on how to achieve the best credential, your argument fails. You cannot take each class in the university as it’s own credential island. In fact, for instructors to achieve the purpose of actually providing expert guidance to students on how achieve the best final set of credentials they must at times actually provide instruction that they believe will best lead to student learning. After all, if you don’t understand and learn the material in the introductory level course, it doesn’t really matter what grade you get. You are going to get crushed in the next go round.

And this is how top tier universities function, at least in the United States. They advertise themselves to students on the overall educational experience they provide and the immense value of the credentials they grant. If a student arrives on campus and decides, ‘hey this isn’t exactly what I want’, it is entirely appropriate to respond, ‘hey, if you want our credentials, this is the way you go about getting it’. Because I think Harvard knows quite a bit more about how to sustain it’s institutional reputation and the value of it’s credentials to the students who attend than does each individual student.

If you want to run a churn and burn bottom tier diploma granting house, sure, just provide guidance to the students on what they need to accomplish to get the grade they want and let them work it out on their own. But that is quite apparently not the path to providing a truly premier degree. Everybody in the middle can then make up their minds towards which goal they wish to strive.

64

mpowell 10.05.11 at 4:28 pm

Bloix @ 62: As ignorant as the Professors are, do you really think 19 year old kids know better? How many have you talked to recently? And anyhow, it is not just professors but a set of professional educators who are theoretically shaping the experience students receive. The most well regarded universities most certainly have an opinion on matters and act on it. If they were all wrong, probably they would not be so highly regarded.

65

LFC 10.05.11 at 4:41 pm

Universities used to have core curricula. Those that still do – in the US, afaik, that means two, Reed College and Columbia University – can make a claim that they are teaching students general knowledge that they need to know as citizens of a democratic society.

You left out U. of Chicago and St. John’s College. There may be a few others. Of course many institutions have things they call core curricula or General Education, but they don’t match what you are talking about.

66

bianca steele 10.05.11 at 4:50 pm

@46
Hopefully this is not too far offtopic for the post–I have nothing against the idea that a teacher is assumed to know where each student ought to go, but . . . What is the difference between “credentialing” and “education” in this case? Is “education” just “credentialing” with tacked-on courses in “philosophy of X” and “sociology of X”? Or is it the difference between being able to do a proof in geometry and being able to write a sentence defining “SAS” as a form of triangle congruence?

67

Andrew Fisher 10.05.11 at 5:08 pm

Bloix@62 An ally at last – thank you. Of course what teachers do have is greater subject knowledge in their specific field and this I have never sought to challenge (although Steve at least thinks I have, so I may have been unclear).

Mpowell@64 at my institution the average new undergraduate is about 25, and a significant minority, especially of the black students, are in their mid or late thirties. I was myself most recently a student at the age of 38, and may well be one again before I die. Kids, we are not.

68

LFC 10.05.11 at 5:15 pm

undergrads usually don’t even get their work back and nobody has time to read it anyway

I think this is not generally true in US universities. In big courses grad students, teaching assistants etc will read the papers, in small courses the instructors themselves will.

69

Sam Clark 10.05.11 at 5:27 pm

Andrew: I don’t think I’ve moved my perspective much. ‘Some students think X’ is compatible not only with ‘Other students think Y’ but also with ‘Some students think X, Y, Z…’. I was just trying to complicate your account of what students want.

But perhaps the more important issue is this: you complain, if I understand you, that I have a utopian vision of what’s good in life that powers my teaching (you can’t just be complaining that I’m forcing my preferences on my students – as Steve LaBonne has already pointed out, no-one’s talking about press-gangs here).

But you also have a utopian vision: it’s the liberal capitalist fantasy of human public life as consisting only of individuals making and honouring free agreements with one another, whether those agreements are about education or laptops or biscuits. It’s a powerful ideal – not one I share, but one I recognise and have some respect for. But it isn’t a neutral position, and you do need to argue for it, not just to assume it. That’s what I was getting at in saying that there are different visions of the good life in play here.

70

Steve LaBonne 10.05.11 at 5:28 pm

Re Andrew’s #67, in my previous, academic life I had opportunities to teach older returning students and found that, on average, their expectations were consistently better aligned with what a real institution of higher education has to offer than those of the typical 19 year old (not surprising, really, given their greater experience of life); and their motivation was correspondingly better. So the prevalence of such students doesn’t necessarily reinforce Andrew’s worldview.

71

William Timberman 10.05.11 at 5:37 pm

This thread reminds me of the relevance wars in American universtities of the Sixties. I was a student then, and on the other side — meaning that in my not so humble opinion a) the curriculum, even in the sciences, was too compartmentalized, too outdated, and too formulaic, b) many of the professors took the my way or the highway approach to students, even when pontificating outside their own area of expertise, and c) we had a right to expect more of our education than a ticket of admittance to the administrative classes.

All of which I continue to think was true at the time, and remains true today, with, as we say, some notable exceptions. It’s also true that now, forty or more years later, I seem to have switched sides. That is, I don’t think that academics themselves were — or are — responsible for most of what was — and is — wrong with higher education. In fact, I think that professors have, by and large, been remarkably successful as individuals and as committees in subverting the nastier aspects of education-as-commodity. The fact that they’ve also remained as successful as ever in imposing their personal quirks on the poor serfs under their charge seems less a burden to me now than it did then, and not just because I escaped their clutches decades ago.

Andrew Fisher is largely wrong, I believe, but he’s also in the majority. The only unit of measurement with any relevance here is dollars (pounds, euros, renminbi.) John’s agonies of conscience over grading, and his honesty in identifying what’s making him uneasy about the justice of the process is commendable, but sadly, I don’t think that his particular circle can be squared as long as the sound of money being counted in the next room is allowed to filter into his office.

In the end, education has all the qualities that are being claimed for it here. It’s a service, a commodity, and a discipline, as well as a catechism that points us in the direction of mental mastery of this or that subject. It’s also damned expensive, and as we all know, the people who provide the money call the shots, at least to the extent that they’re aware of the shot available to be called. (This is probably why physicists can get away with more than philosophers. Everybody with a few bucks fancies himself a competent philosopher. After all, if he weren’t, he wouldn’t have all those bucks, now would he?)

Given higher education’s tangled history, and it’s even more tangled definitions of itself, threads like this always seem to wind up being inconclusive, but they’re always interesting, much as looking at the light coming out of a prism is in some sense more interesting than light projected from a lens.

72

Bloix 10.05.11 at 5:57 pm

#64- “As ignorant as the Professors are, do you really think 19 year old kids know better? How many have you talked to recently?”

I’m the father of two college students, who I talk to frequently – and to their friends, from time to time. Obviously they don’t know what they need, except that they do know they need a college degree in order to become adults. Among my kids and their friends and others of their social class, the only reason not to go to college is that you’re mentally or physically ill.

So what is it that they do need? Well, from my point of view, they need to learn to think rigorously, testing their ideas against serious criticism. They need to locate themselves and their society in the sweep of civilization. They need to learn to participate in the public civic debate in an informed way. And it would be nice if they developed some sort of aesthetic sense while they were at it.

What they often get is a narrow slice of an academic discipline based on what the professor is thinking about that year.

But oh dear, I didn’t mean to be hi-jacking this thread away from the point about comments and grading. I agree with just about everything Prof Holbo says, except for this:

“If they fail to provide a little self-criticism, they get skimpy comments.”

What you are doing here is devoting your time to those students who need your help the least.

In the Passover seder, there is the parable of the four children, from the wise, who asks a detailed question about the ins and outs of observance, down to the “one who does not even know how to ask.” Don’t ignore that one – that’s the student who needs you the most.

73

bianca steele 10.05.11 at 6:01 pm

me @ 66
Of course, there is the level of knowledge above mere “skill,” including ability to do research. I’m not sure how widely a BA is understood to include the rudiments of academic research in the subject.

I’ll add that I’ve known people who[1] had a background in “sociology of” but whose knowledge was more at the “defining terms” level than I found appropriate. They were often not happy people.

But way OT.

[1] Judging from what they spent most of their time discussing.

74

Sam Clark 10.05.11 at 6:27 pm

This fine piece by Stefan Collini in the LRB is relevant.

75

christian h. 10.05.11 at 6:32 pm

I’m sorry but what is “fair” grading anyway? Bloix and others seem to think it should mean that there are “objective” criteria laid out beforehand that the students could – given time – check their own work against. This may work in calculus (we who teach math are lucky that way), but it cannot possibly be done in humanities, beyond some very basic, rough rules that are – as usual – proved by the exception. Imagine Beethoven – getting a “D” on his composition homework because he failed to satisfy all the rules of symphonic composing correctly. What nonsense. Any grading will amount to making a subjective judgment.

The conclusion I’ve long drawn from this is that grading beyond “pass/fail” is a fool’s errand. And it’s done for the consumers of corporate education that are not the students – the corporations hiring, or medical schools admitting, students. It’s also done to reinforce societal stratification, enforce social norms etc. The belief that assigning grades has a role to play in the education of human beings properly understood (the kind of education Bloix would like for his kids, core curriculum fetishism aside) is, in my opinion, spectacularly mistaken.

76

Meredith 10.05.11 at 6:42 pm

This semester I’m blessedly free of paper-reading (what I try to discipline myself to call it, rather than paper-grading) because I’m teaching only language courses. Fresh from Plato’s Apology, I would just comment that in addition to taking no fee for anything he did, Socrates claimed that he didn’t in fact teach, either. (Although in the Protagoras, he acknowledges that he rather does end up switching positions with Protagoras on the question of whether it’s possible to teach civic or moral excellence — but that’s the aporia into which that dialogue settles and, at any rate, not Socrates’ position on what it is that he does.)

This comments section could rapidly turn into a Graeber-esque discussion of moneticization and the imagination. And maybe it should. Certainly, I think, Socrates’ concerns about fees and whether teaching is possible can only be understood in the context of his society’s being in the midst of profound economic and other socio-political transitions. And some of the tensions in the comments here about the teacher’s relationship to the student (client?!!? please, spare me!) strike me as products of our confused consumerist-cum-democratic ways of thinking. (I include myself in that “our.”)

I’d just say that surely a teacher can — indeed, must — possess authority AND humbly practice listening.

As for me, the teacher is like the Iliadic hero in that (1) she teaches because she’s mortal and wants to do something that will in some sense make her immortal = something of larger social benefit, and (2) precisely because she is mortal, she also expects to be rewarded for her efforts, while she still lives, with at least enough to live on.

As for paper-reading and paper-grading (which I’ve been doing for 35 years now, often in very large courses), I have no simple formula. So much depends on the level and size of the course (and if you have TA help — something I’ve never had and always envied, though I know that coordinating TA’s is also a big job). Also, students’ collective needs shift and change over time. I have found that less may depend on the kinds of comment I provide on individual papers (though I am a great scribbler) than on the way I structure a course and how I integrate papers, of various types, into that structure. Last spring I taught a writing-intensive course to 19 students (so discussion-oriented) and assigned a series of graded “short analysis” papers on very narrow topics that required students to anaylze a text very closely. I was a bear about concise writing — slash marks everywhere. These assignments were interspersed with ungraded “response papers” on movies they watched for the course — opportunities to write however the hell they felt like writing, and for me to respond more personally, even whimsically. Finally, three 5-page essays on a topic each student had to devise (and get my approval, in an individual conference). Because of the short analysis papers, students understood the concise writing I expected on the 5-pagers as well as the kind of detailed textual analysis they should undertake. But I also expected each of them to come up with their questions to explore.

Next semester I’ll be teaching the same course to some 40 students (so more lecturing, less real discussion). No way I will be able to structure paper-writing assignments so artfully. But I like John’s idea of worrying less about justifying my grade in my comments and disciplining myself to identify one or a few things I can help each student with in his or her writing.

77

mdc 10.05.11 at 6:45 pm

These discussions always make me think about education without grades. I haven’t put a grade on a paper for many years, and it’s wonderful.

78

MS 10.05.11 at 7:24 pm

I would love to see examples of other people’s comments on papers. (1) and (2) are right.

The (3) you don’t add is that you are creating an intellectual dialogue with a student. This is why it is useful to assign papers early in the semester, in order to pull them into a private discussion with you and get them to see how their ideas can matter. I am realizing that I treat ungraded papers early on as if a student did a blog post, in fact–although I try to give better comments than you get on most blogs.

Like most people I hate to grade but unlike most people I actually enjoy writing comments on papers when I have very small classes or seminars (i.e., infrequently). If you make sure your students see you are engaging in an intellectual back and forth with them, you will get better papers–it’s also more fun for the professor.

I also agree with Bloix–You really can take the student who doesn’t know how to ask and get some discussion going on with them. It takes some creativity to do that–to take that tiny spark of originality and turn it into a fire. Also, it takes time a lot of us don’t have. The results can be very interesting. College students aren’t that different from kindergartners in that their view of what they can do is affected by what the professor thinks they can do. I sometimes act like they were giving me a well-thought out position even when they probably weren’t. ‘Oh THAT’S your view! But what about…??’ The next paper is always better (sometimes only marginally, but sometimes by alot.) I interpret lack of self-criticism very charitably in my comments (not in my grading)–it may make me a sap in most cases but I hope I catch those who genuinely need help.

This also leads me to 2* which is to get the student to see they may have done something worth doing even if they aren’t getting an A. Good comments reward their effort (a teeny bit).

That direct personal feedback is the best way to advance student learning is why we desperately need small classes in college–but that’s a different can of worms.

79

Andrew Fisher 10.05.11 at 8:15 pm

Sam @69 I agree that both your perspectives are equally valid and I was trying – I see now with rather I’ll grace – to concede this very point. I certainly simplify for the purpose of discussion (this is the Internet!), but I hope I do so consciously.

I don’t wish to complain about your utopian vision. Your vision is yours. As it happens I am an anarchist by inclination, but when I come to work in the university I accept that it is a hierarchical institution: I accept orders from some and give instructions to others. This is what I am paid to do. Whilst it is some way short of the press gang, there is also a very real power imbalance between the teacher and the students in a university setting, as in any other. Every day I meet teachers who talk about ‘good’ students and ‘bad’ students, meaning precisely those who comply and those who do not comply with the teachers’ vision of what education is about and what a student ought to be. Often I find myself engaged in policy discussions (and as we highlighted at the top of the thread, assessment is an institutional policy issue in the UK, not a private issue for professors as I believe it is in the US) with colleagues who want to advantage the ‘good’ students and disadvantage the ‘bad’. To me, this feels like an abuse of power: as Bloix says at 62 the ‘good’ students are those who resemble the staff themselves so we see an elite in the process of replenishing itself. And we permanent academic and management staff in universities are an elite, let’s not beat about the bush there.

This has real effects on real people. Right now as I type, my own institution is getting whiter and (slowly) posher as it pursues ‘good’ students at the expense of ‘bad’, whilst the gap between white and black student attainment – already large – is widening. None of my colleagues,I am sure, is personally racist or even unduly prejudiced against poor people but they blind themselves to the effects of their action through an ideology similar to yours.

I don’t wish to accuse you – or anyone else I know only through some internet discussion – of any of this. Some of the most pernicious drivers, such as League Tables, you probably disapprove of just as strongly as I ever could. But the league tables gain so much of their power over institutions precisely because the senior managers and academics see something there that they recognise about what a ‘good’ university ought to be. In my view, if you have never seen any pressures like this at work then you are either wilfully blind or work at an institution from which the undesirable have already been practically excluded, and have hence become invisible.

So I am passionate about arguing precisely for the rights of students who do not value education in the way we value education, who don’t admire us or want to emulate us, who have some other purpose for their lives in which we are merely incidental. And I use the language of the marketplace because we have chosen to bring our labour to the market, so we might share a language there. When I tried up at about comment 33 to argue on mere grounds of human equality, I don’t think I even got the disagreement I have gotten in the rest of this thread – just incomprehension.

I don’t wish to comment on the Collini piece, because my comments would not be sufficiently civil.

80

piglet 10.05.11 at 8:17 pm

“I think this is not generally true in US universities. In big courses grad students, teaching assistants etc will read the papers, in small courses the instructors themselves will.”

Let me throw out a question here that I think is more than relevant to this thread: how long does it take an experienced instructor to quality-read and grade, with meaningful comments, a student paper? There’s plenty of expertise in this thread. I’m looking forward to your responses.

81

ScentOfViolets 10.05.11 at 9:25 pm

Yet another problem that Andrew hasn’t thought through and which goes along the lines of his decidedly odd notions of differential respect is that institutions that perform the sort of service he is demanding already exist. We call them “trade schools” or “community colleges”.

Andrew is perfectly free to go there and get the education he demands. Or failing that, he could perhaps hire a private tutor.

But it seems that what Andrew really wants is not only an education and credentialing tailored to his demands; he wants it to be from a fairly elite, fairly high-powered institution as well. Iow, he wants to go to Cornell, demand to be taught as he deems he needs to be, and then get his sheepskin from a rather prestigious school and ipso facto actually carry a significance it doesn’t really have.

As been said of so many other things, you get to pick any two of the three ;-)

82

mpowell 10.05.11 at 10:01 pm

Andrew @67, Okay, I can see the argument for a significantly different structure of university when your students are much older. They probably are not signing up for an immersive 4 year experience and so it would make more sense to offer something slightly different and more sensitive to the likelihood that they are pursuing very specific and possibly narrow educational goals related to career needs they have identified. I don’t honestly know what the demographics of the average American university student look like, but I am thinking of your stereotypical big 4 year school bringing in 90% of it’s undergraduates directly from high school. And there I think it is clear that the school is going to be deciding what kind of experience students will have.

Responding to some other comments, I think it is possible and likely that the school will frequently get it wrong with poorly designed curriculum or stale subject matter. Not everything is perfect and certainly no general process is. I don’t think it would improve things in many cases to instead just do exactly what each student wants.

83

piglet 10.05.11 at 10:32 pm

This is fascinating:

In 1964, “the new Labour government set up the Department of Education and Science (DES)… In 1995 …, the DES, was dissolved and replaced by a new ministry, the Department for Education and Employment. In 2001 this was superseded by the Department for Education and Skills, which was itself broken up in 2007 and replaced by the Department for Innovation, Universities and Skills. Most recently, in 2009, responsibility for higher education passed to the new Department for Business, Innovation and Skills.”

One would think that Education and Science was a perfectly good name for the authority overseeing the University system. Business, Innovation and Skills? Really?

84

socratic_me 10.06.11 at 12:47 am

There is also at least one major flaw in Andrew’s argument and it has been bugging me since I first popped into this thread in the early AM. Just because you pay someone money does not make them your servant. “I am giving you money.” moves you nowhere in “you must do exactly as I wish now.” For instance, I don’t get to call up my phone company and demand that they send me a pizza pronto on the basis of our monetary pact. I cannot do that even if I think that phone service should include pizza to be decently called phone service. They offer a set of services and, if I pay them, I get those services. If I am not fond of how they provide those services, I can take my money and try to find another phone company that will deliver pizza, by God. But I don’t just get to insist that the phone company do what I demand because I paid good money for phone service. (Well, I can, but people will point in laugh.)

The educational institution offers an education. If students want mere credentialing, they can find an institution that offers such. Lord knows there are plenty. But a student’s money doesn’t make them king and their educators servants, just as it does not make the workers of a phone company, pizza parlour or other place of business servants.

85

Bloix 10.06.11 at 1:04 am

#81 -“I am thinking of your stereotypical big 4 year school bringing in 90% of it’s undergraduates directly from high school. And there I think it is clear that the school is going to be deciding what kind of experience students will have.”

In point of fact, schools are extraordinarily incurious about the kind of experience the students will have and do very little to create anything approaching a valuable pedagogical experience. Read My Freshman Year: What a Professor Learned by Becoming a Student by ‘Rebekah Nathan’ (2005, Cornell University Press).

86

ScentOfViolets 10.06.11 at 1:08 am

I’m sorry but what is “fair” grading anyway? Bloix and others seem to think it should mean that there are “objective” criteria laid out beforehand that the students could – given time – check their own work against. This may work in calculus (we who teach math are lucky that way), but it cannot possibly be done in humanities, beyond some very basic, rough rules that are – as usual – proved by the exception.

As it happens, I teach math, in fact, just finished a night class in what’s called “Discrete Math”. Yes, it is easier to grade something like math or physics (or conversely, actually assigning grades for something that can’t meet some sort of objective criteria – even if it is subjectively graded – is just so many angels dancing on the head of a pin.) But even in math there is often a “right” way to go about solving a problem vs a “wrong” way, even if both ways give the correct answer. Iow, giving points for style is perfectly reasonable.

And let’s face it, most of the time there is a pretty good correlation between students who have mastered the mechanics of writing on a particular subject and how well they know it. Yes, I’m sure that there is an odd Beethoven or two who might not technically get a passing grade in a subject based on how well they’ve “demonstrated mastery of core concepts.” But they are very few and far between, and in any case, fairly recognizable.[1] The inarticulate oaf who is actually a sublime genius is only a little more common than hens teeth.

[1]That is to say, while “demonstrating mastery of core concepts” might be one way to earn a passing grade, few teachers are so straight-jacketed as to think that this is the only way to get a passing grade.

87

ScentOfViolets 10.06.11 at 1:13 am

I don’t honestly know what the demographics of the average American university student look like, but I am thinking of your stereotypical big 4 year school bringing in 90% of it’s undergraduates directly from high school. And there I think it is clear that the school is going to be deciding what kind of experience students will have.

Or rather, what the mythical median student will have. Yes, it’s quite possible for a student to come back 20 years later and complain that he was ripped off – all he ever needed to know post-graduation he could have learned in three semesters, four at the most. He’s quite right, of course. But the trick is knowing that before matriculating, not 20 years after. And if you know how to pull that one off, let me know – I know a way to turn that into millions, billions.

88

Kim Weeden 10.06.11 at 2:07 am

Bloix @ #54: I completely agree that my system isn’t perfect. I don’t always have a large enough pool of willing grad students who study inequality to hire as graders, although I do always hire advanced grad students. (I have input on the TA, but I don’t choose him/her.) And, yes, I could trigger triple-grading with a lower tolerance for grade discrepancy. Some years I have.

But there are diminishing returns, both to the students in the aggregate — they get better lectures if I spend fewer hours grading — and to me professionally. As you note, getting my students’ grades absolutely perfect does not help at all at promotion time. It’s also not why I went to grad school, took a job in an R1, and write grant proposals (some of which fund hands-on training for undergraduates).

I didn’t mention this in my first comment, but one of the reasons why I let students drop one of the four exams also has to do with equity, indirectly. In a large class, the odds are that some students will genuinely fall ill, experience a family emergency, observe a religious holiday that falls on or adjacent to an exam day, or be overwhelmed to the point of doing themselves harm (sadly, I’m not exaggerating).

It seems inhuman not to make some allowances, and it seems unfair to only give dispensation to students who ask for it. After all, we know that some students come to college with much more experience, and cultural capital suited for, negotiating with educational authorities to get what they want or need. See, e.g., Annette Lareau’s work on class differences in same.

(Now that I think about it, my concern with creating equitable grading structures is part occupational hazard. I can’t turn down a request to answer a survey, either; market researchers have me on speed dial. And occupational brainwashing is evidently permanent, because I still overtip wait staff.)

89

Michael Harris 10.06.11 at 2:07 am

On marking/grading, I wish I had the resources to do what Kim Weeden described — hiring multiple markers and cross-checking.

On “what students want” versus what professors think they should want/get, a fair bit has been written/debated already. Some of that has referenced the information asymmetry between new students fresh out of school and the professors who think they know what’s best for them. I haven’t, however, seen discussion in the thread of how “what students want” changes during and soon after the completion of their formal education.

Some anecdotes that have helped inform my own views:

1. In the undergraduate degrees I teach into, we do keep reasonable track of a bunch of our students upon completion and after they graduate and move into the workforce. We used to have meetings with them at the end of their final year to get direct face-to-face feedback. They have a much more informed view ex post about what the degree was about and why it was the way it was than they did ex ante, and their views are generally and in the main, favourable.

2. When I started this particular job, I was put in charge of the fourth year research program — that is, coordinating the final year undergraduate thesis-writing class. I gave the assembled students a speech about how this task would be fundamentally their responsibility, and their supervisor’s job would be to assist them in the process, but not to micromanage it. One student became agitated and insisted to me that he and his peers were paying for their education and deserved to be instructed in detail as to what was expected and how they should proceed, step-by-step. I held my ground. At the end of the year, as the submission deadline approached, he made a point of stating in the final class that I’d been right and he now saw why I’d said what I’d said, and what I meant by it.

3. Later on, I taught a 3rd year “research methods” unit that was intended to prepare students for their upcoming research tasks. I filled it full of all sorts of interesting things (maybe too much) which some students liked but many resented. I, as usual, refused to play the “what’ll be on the exam?” game? Several students mentioned to me the year after that much of what they’d seen and done in that unit was helpful and made more sense now that they were doing research, even if it had not been obviously relevant to them at the time.

4. I’ve occasionally had “mediocre” (average or below) students who clearly didn’t like my style of teaching all that much when suffering through it come to me later on (sometimes after graduation) to tell me that I’d had a significant impact on their education and that they were seeing the purpose of their education differently than they had when they started out.

Of course, students who are deeply dissatisfied usually drop out/transfer, so we miss those data points. But I do see enough affirmative data to know that even when our goals in teaching bump up against immediate student expectations (in their “customer” mentality), we don’t have a significant mismatch that leads to long-term problems.

Our bigger problems in meeting student expectations tend to be driven by resource constraints, not by some fundamental pedagogical incompatibility.

90

Peter T 10.06.11 at 3:31 am

I’m sorry to have to tell you, but there’s a right and good way to machine a part or tune an engine, and a wrong or bad way. And people who go to trade schools get shouted at very loudly when they insist that, as they paid, they have a right to do it how they like. Or even, in their first year or so, to waste people’s time arguing about it.

On “objective” criteria, I’m with John Holbo. I spent time in a bureaucracy watching supposedly unfair or opaque “subjective” judgements be replaced with “objective” check lists, formal criteria (with more sub points each year) and so on. All it did was either provide an impenetrable cover for unfairness or allow the stupid or lazy a way to justify the wrong decision. Oh, and sometimes good people made the wrong decision because, you know, the objective criteria came out that way, even though their subjective judgement was screaming “wrong, wrong!”.

One driver is the hubristic belief that we can make explicit and codify precisely all the factors that go into complex judgements. I can see the next stage – “yes, we know you can hit the ball, Miss Williams, but we can’t qualify you unless you can tell us exactly how you plan to hit a fast left hand return over the net. We have to have objectivity and transparency in tennis – the fans have paid to watch, after all”.

91

David Moles 10.06.11 at 3:50 am

Andrew, I think I see part of your point, inasmuch as many introductory courses in many disciplines at many universities are taught very much as if all the students in them were expected to go on to major in those fields, obtain graduate degrees in those fields, and eventually do original work in those fields, which the vast majority of those students manifestly will not; and those students could almost certainly be better served by a different course design.

That said, you’re not going to get that course design out of giving a bunch of 19-year-olds what they say they want, because what they want is a shortcut route to an A grade in Astonomy 101 that doesn’t involve actually learning the material or learning how to demonstrate that they’ve learned the material. (Cf. “Eat less and exercise.”) And I submit that it’s at least worth consedering the possibility that their professor might know something about how the material is best learned, or at least how it might be better learned than by whatever slapdash means the students might invent themselves out of blind fumbling and misinformed common sense.

Alternatively you could argue that the very idea they should have to learn the material is unjust, that the system is in general a sham and professors should facilitate the quick and easy granting of honors degrees to the student-customers who have after all paid for them. There’s a case to be made there, but I wouldn’t expect any great number of professors to hear you out while you make it.

92

Meredith 10.06.11 at 3:53 am

It’s scary to read John Holbo@26, that UK matrix. Similar projects are afoot in the US, as my husband can attest (he teaches at a state college — public institutions get much more hung up on this stuff than private colleges like my own do, though we fret about “grade inflation”). But still, nothing so rigidly bureaucratic as that (easily manipulated) matrix.
I love the way verbs for teaching in Latin and Greek take two accusatives. Less clear in English what’s up grammatically, and maybe conceptually. Does agent deliver object to indirect object? It shouldn’t, and can’t, really work that way.
Teachers should be agents who disappear in the transaction between material/skills and the next generation. Between past and future. But they are crucial nonetheless. (Sort of like money.)

93

Bloix 10.06.11 at 4:23 am

#26- no, these sorts of lists can’t possibly work if the grader actively rebels against systemization and says, I know what a B is – I know one when I see one.

The thing about grading is that it’s a no-feed-back system. You send the grade off into the world, and you never get any return to tell you whether your grades are fair or whether they’re arbitrary. There’s no corrective mechanism. You can grade for 30 years and you feel that you’ve become expert. But that’s an illusion. You don’t really know if you’re fair or even competent. You can’t know if you’re not tested in some objective way.

Suppose you have 200 papers to grade. Suppose you recorded the grades, not on the papers, but on a separate sheet, then – oh, 3 days later – you pulled 20 at random, shuffled them, and regraded? Have you ever done that? If you haven’t, how do you know that you’re consistent?

Would that be too much work? Well, you never send an article off to a journal without reading it over a dozen times and checking every calculation and conclusion over and over again. Why? Because you know to a moral certainty that there are errors that can only be caught on repeated review. Yet you think that you can grade accurately on the first pass. How can that be true?

And how do you know that you don’t have a pattern – say, you give higher grades when you grade after dinner, or lower grades when you’re hungry? One study showed that judges tended to give longer sentences just before lunch. Do you think you’re immune from that sort of bias? What in your 30 years of grading have you done to check?

And why is it that you can’t articulate what makes a B paper? Perhaps you just give higher grades to the students who are better essay writers – perhaps being a good bullshitter is worth at half-grade or more at your hands. Shouldn’t you take steps to guard against rewarding fluency over substance?

And even if, although you never had a day’s training in it, you happen to be a natural, a truly wonderful grader – fair, consistent, accurate – you wouldn’t say that about all your colleagues, would you? Some of them are terrible graders, aren’t they? Harsh, lazy, biased, or worse. And it’s not your problem that your department doesn’t do something about it – it’s the students’ problem that they’re treated arbitrarily, right?

94

F 10.06.11 at 4:52 am

But as a practical matter, requiring criterion-based grading does nothing to improve the quality of grading. Because, as pointed out above, it is trivial to circumvent the system, consciously or subconsciously. Besides requiring instructors to be angels, what practical means can be used to require thoughtful and accurate grading?

95

ScentOfViolets 10.06.11 at 4:59 am

But as a practical matter, requiring criterion-based grading does nothing to improve the quality of grading. Because, as pointed out above, it is trivial to circumvent the system, consciously or subconsciously. Besides requiring instructors to be angels, what practical means can be used to require thoughtful and accurate grading?

Hmmm . . . maybe this is representative of why the sciences – the harder the better – get more respect than the humanities. When I grade an exam or a quiz or homework, I’ve got a pretty good idea of what gets points added or points subtracted. And I can justify the relevance. More importantly, I can use the grading procedure as a diagnostic and write corrective comments accordingly. I get the sense that people like Holbo simply can’t do this, which is just . . . weird.

I must be misunderstanding the general argument if the results are as changeable as all that.

96

F 10.06.11 at 5:05 am

SoV

Agreed, but if you’re not careful it can be equally arbitrary in the sciences. Although the distinction between correct and incorrect is clearer, the design (or lack thereof) of the grading rubric can completely undermine that kind of certainty. It’s better, but not categorically so.

97

Meredith 10.06.11 at 5:53 am

Okay, I do two kinds of grading.
In language courses (especially lower level), the criteria are very clear to me and to my students: learn these forms, that vocabulary, these (to me) rudimentary translating skills — to one degree or another = grade. Yeah, we could argue about the basic framework for all this, but we don’t. I have knowledge/wisdom/skills to impart, students appreciate that (it’s why they signed up for this course), and we’re all on the same page — they’ve in the course to achieve easily defined (though not easily achieved) goals that I can help them meet. Easy, re grading. (Not easy to construct good and fair tests of this kind of knowledge, btw! But still. The project is clear, and shared.)

Upper level language courses and literature courses where texts are read in translation (usually lower level, but not always): another ball game, not just in terms of grades but in terms of goals, which have become more diffuse, or grandiose, or — well, the goals are more each student’s, even as my goals are more mine. Much harder to define what those goals are, though they’re not less important (by a long shot). Certainly, though, much harder to associate progress toward those goals with grades. (Here I feel like Socrates: I don’t teach! Whatever I’m doing, it’s not teaching in the way I teach the forms of Greek verbs!)
Finally, though, the goals ARE the same in both kinds of course. And they lie beyond the course itself — yet are embodied in it: some huge portion of what goes on in a classroom (or paper or test) is a teacher — a caring adult who isn’t a parent — who cares about each student in that class (even if the teacher can’t know each personally and well). About who each of them is now but, equally, about who each could or might become. And that those questions are tied up with writing well or analyzing some text or other intelligently and imaginatively, and with experiencing connection to other people in the past or present. Something as simple (and as hard) as that.
Labor intensive, too.
If I may return to Socrates in the Protagoras (I’m not even a Plato fan, but I’m with him here): these are the souls of young women and men that have been placed in our care.

98

John Holbo 10.06.11 at 6:05 am

“I get the sense that people like Holbo simply can’t do this, which is just . . . weird.

I must be misunderstanding the general argument if the results are as changeable as all that.”

Is it really that hard to see why, when I grade essays, there is a certain amount of qualitative squishiness to the assessment process, whereas when I write an mcq-style final exam I can not just say what the right answer is but actually program a machine to grade the tests! I would not have thought it was weird in the least that this difference exists. Does it truly surprise you that grading essays and marking mcq questions are such different sorts of activity, SoV?

Perhaps the problem is this: you are inferring from the fact that I say I have trouble explaining to students why they got the grade they got that I myself don’t feel I know why I am giving the grades I give. So therefore the grade is ‘changeable’, you infer. I feel a deep B-ish feeling welling up in me when I read x, so the student gets a B. But tomorrow I might feel more A-ish in response to x, an equally inscrutable and unanalyzable internal condition of my grading self. So it’s just the student’s bad luck that she got a B not a A.

I don’t want to dismiss the sorts of real worries about fairness in essay grading that are expressed above but, to put it mildly, this is not the sort of picture of the grading process I was seeking to get across. It isn’t like consulting the Dephic oracle, with the grader as manic, un-self-conscious Pythia, emitting inscrutable verbal riddles as to what is wrong, and what must be done to improve the situation.

I know an unclear paper when I read it, usually. But often it is difficult to bring a student to see faults like unclarity or bad organization in their own work. They just don’t see the problems, so simply saying ‘you got a B because the organization was weak’ – although a true, general statement about what went wrong, hence why the grade was given – is often no more enlightening to students than just plain ‘you got a B’. Often the way to get the student to see how to improve is, for example, to point out, laboriously, 10 small things going wrong in the intro. (‘Why did you start this sentence this way? Why doesn’t this sentence come first? Why did you say ‘yet’ rather than ‘and’ here? What job is this intro paragraph supposed to do? What reader would be informed by this paragraph, and is that reader also the reader who would be informed by what you have to say in the rest of the paper?) This sort of sentence-by-sentence trench warfare is nothing you can wage across a whole 5-page paper. I’m not made of time, you know. But, often, there is no way to make the student see what you mean except to get nitpicky about some little bit. But then you end up not really providing ‘general comments’. My point is that it’s important to say what you think will help the student see what went wrong. Often, saying ‘the paper is badly organized’ is not the way to do that – not because it’s only a subjective judgment, or debatable, or of indeterminate significance (although these may be real issues for graders, I’m not denying it) – but because the student can’t recognize bad organization, so telling the student that it is there does not induce recognition of it.

99

Roa 10.06.11 at 6:52 am

#96 nails it! The thing to really emphasize here is time. I could in detail explain every problem in the paper that justifies the grade the paper receives. But to do so in a complete manner for every paper in a way that would actually be grasped clearly by the student would require a lot of time. Many days! It here helps to think of the total time per student available during the entire course. More time spent writing detailed comments on everything in the paper (time spent in addition to that needed to form a considered judgment on the grade) means less time on lectures, seminars, preparations and so on. So here is an open questions to those reacting to John H: how many lectures, seminars and so on are you prepared to cut from the course in order to fund the level of extensiveness and generality in grading comments that you wish for?

100

Neville Morley 10.06.11 at 8:13 am

Fascinating discussion, and I’m sorry to have to dip in and out every twenty-four hours or so rather than participate properly. There’s a risk, therefore, that this comment simply repeats points made earlier, and at any rate that it’s out of place in the discussion, but I don’t think it’s been put in quite these terms…

Instrumentality v. idealism in education; largely in response to various of Andrew Fisher’s comments and responses to them. The formal position is that I am employed to teach my subject to students who have sworn blind on their application forms that they wish to study this subject as an end in itself. In theory, therefore, I could teach them – in curriculum design and content, assessment, feedback etc. – as if they were all going to become academic specialists in this subject. However, I know perfectly well that most if not all of them will not become academics and don’t wish to anyway, that some of them are simply interested in getting the qualification and regard the subject itself as a more or less pleasant and interesting means to that end, and that most of them are juggling, more or less unconsciously, instrumentalist concerns about the need to get ‘their’ 2.1 degree and a genuine interest in the subject. I therefore tailor both course structure and feedback to acknowledge this fact. I do believe that as a result of studying Roman economic history my students are given a valuable training in research skills, critical analysis, construction of arguments, presentation skills etc. – but I don’t agree that I should not try to teach anything more than those practical, ‘transferable’ skills, let alone that I should organise everything around getting the students to pass the qualification regardless of their actual grasp of the subject, even if I suspect that for some of them this is their main if not sole motivation. After all, they could have gone elsewhere to do a different subject; if they’ve chosen to add the lustre of my sort of institution to their qualification, they have to accept the constraints of how the subject gets taught at such an institution.

The imposition of external constraints and guidelines is a slightly different matter. I’ll concede Andrew’s point that some academics previously gave only subject-specific feedback, and that it is a good thing that students should receive feedback that’s actually useful to them in a range of ways. My personal experience is that it’s unhelpfully formalised and restrictive; I’d be giving this sort of feedback anyway, but now I have to do it in specific ways according to specific formulae – and I entirely agree with John Holba at #26 and #96 about the logical incoherence of a lot of this stuff.

Further, for all that many of my students do clearly have a somewhat instrumentalist view of the purpose of their studies, the real pressure to turn all degree programmes into vocational training courses has come from elsewhere, above all government and various quangos. Virtually all my students do have a genuine interest in the subject, and are thoroughly bemused by all the obsession with transferable skills, specification of learning outcomes and the like; they may be orientated towards getting the qualification at the end, but they are happy with the idea that they’ll get it by doing well in their subject. To an alarming degree, we in the UK are now effectively training our students to think in these terms because of the ways in which we have been required to describe and organise our programmes.

101

Sam Clark 10.06.11 at 9:48 am

Andrew, I think you’re now making me stand in for some bugbears of your own. I haven’t said, and don’t believe, anything about ‘good’ and ‘bad’ students of the kind you’ve suggested, and I’m aware, as any even minimally decent teacher is, that many of my students aren’t like me.

One more go at what I think is the central point: you don’t really believe that the instrumental and the transformative ideas of education are ‘equally valid’, any more than I do. And you do want to complain about my utopian vision: you think it has the ugly and unjust consequences you sketch. If you could convince me that you were right about that, I would change my views about what I should do as I teacher. But my view, at the moment, is that it’s your vision which has the ugly consequences, and mine which points towards a way out of them. As I’ve said already, I don’t think that the situation we’re in is just one where different people have different desires, where nothing can be said in favour of or against any desire, and where the only question is who has the power to impose their own desires. I think part of the point of studying philosophy is to develop reasons for and against our desires: to find out what’s actually worth wanting. I don’t have a list of such ‘correct’ desires that I want to impose on my students (‘Beethoven and Hume, and nothing else’): my job is to help them do the work of developing themselves.

102

ptl 10.06.11 at 11:14 am

Bloix, 93

of course you’re right. The answer’s the old UK system, still, presumably, largely existence, of blind double marking and external examiners, and examiners’ meetings to review marks. But I doubt very much Sam — for example — objects to that system.

Neville, yes. Quite.

Andrew, yes there are academics whose view of “good” and “bad” students disadvantage some, well, good students! And yes — though you didn’t quite say this — an overemphasis in university admissions on A Level results will further entrench social and economic and racial inequality. But your hyperbole and your support of mindless bureaucratic “feedback” mechanisms, and “never mind the quality, weigh the paperwork” assessment, hamper understanding of your true cause.

103

Sam Clark 10.06.11 at 12:36 pm

ptl: indeed not. If we’re going to grade, we should do it fairly, and anonymity, blind double marking, and external review are important means for doing so.

104

Emily 10.06.11 at 1:10 pm

Bloix (93)> You don’t seem to have much of an idea about how grading can work (in my case DOES work). I teach at a small university in the US. I am the only one in my field in my department (English), so in that sense, I’m the only one to say what is right or wrong about, say, the Wife of Bath. There is a historian in my area, but not anyone else in the English Dept. That said, we DO discuss how we grade. We sit around every year at the end of the year, for our assessment, and choose about 30 random essays from across the second-semester composition classes, and we do “grade norming.” We grade them, talk about our grades, discuss outliers (if most folks gave it a “c” why did I give it an “A” and bob over there give it an “f’?). There are surprisingly few outliers, and usually they can be explained by differeing views of standards. Once we’ve “normed” we grade pretty similarly. So, a paper will get an average of a B, let’s say. Some folks will give a B+, some may go as low as a C, but we are in the same range. As far as I know, this process is fairly normal. SO, one doesn’t usually go 30 years without having any idea if they are grading “wrong” or not.

I, for one, was questioned about my grades by my school Dean and department chair because they felt my grades in composition classes (and perhaps overall) were a little on the high side.

And I can’t grade all the papers three times. With at least four classes each semester, and at least 70 students, (40 of which are in composition), that is impossible. That said, I grade a group before putting grades on them. I read them, comment, and consider what I’ve got in terms of papers, standards, etc. If I see a clear misunderstanding, that is, the same mistake over and over that is likely to be my fault, I consider it when giving grades.

We also have departmental definitions of what makes an “A” paper, etc. that are (pretty much) departmentally agreed upon AND available to our students–they are part of the handbook students get at the beginning of the year, and I go over them in class.

So, grades are not entirely arbitrary. I will say that they aren’t entirely uniform, either, of course. They do vary based on professors, just as content varies some, too.

And, I try to do what you want for your kids: teach them the skills–critical thinking skills–that will serve them regardless of their majors, careers, etc. The truth is, it’s the hardest skill to learn, and often the one that is the least immediately reconizable. As a post said above, students sometimes come back later and say “now I get what you were trying to show me then…”

105

bianca steele 10.06.11 at 1:10 pm

@John Holbo: because the organization was weak

Isn’t the problem here that they don’t know the words organization and weak? Is this something that could be addressed by special orientation programs for kids from poor schools, or by some kind of Routledge-short-intro style crib sheet, if undergraduates are expected to understand this kind of language (or by Shirky-style volunteers on the Internet, I’ll bet there are scores of one-time philosophy students who would be glad to share what they’ve learned, or alternatively to redeem earlier failures by preventing others’ making them in future)? Or is the problem, really, that they think their problems have to do with content and such but really are mechanical?

106

bianca steele 10.06.11 at 1:14 pm

(FWIW my undergraduate experience in almost all my philosophy and other humanities courses, 25 years ago, was excellent, as were almost all the grad school courses I took later in history and history of science. But most of the professors tended to the minimal comment school in most cases.)

107

John Holbo 10.06.11 at 1:27 pm

“Isn’t the problem here that they don’t know the words organization and weak?”

No. They know the words ‘organization’ and ‘weak’, obviously, and they know that organization is ‘how things are put together’. And ‘weak’ means bad. So they know that if I say ‘the organization is weak’ that means somehow things are put together badly. But they are unable to recognize that, and how, their organization is weak. Just saying ‘the organization is weak’ doesn’t, in itself, cause the scales to fall from their eyes. They don’t need a dictionary. There isn’t any crib sheet that will do it. (I don’t know how better to explain it. If you are in the forest and the trained naturalist sees the tracks and you don’t, he isn’t going to hand you a dictionary with the page open to ‘tracks’. It doesn’t work that way.)

108

mpowell 10.06.11 at 2:42 pm

@106 Bianca’s comment has me completely baffled. Does someone actually believe this is the way it works? I remember getting papers back as an undergraduate- “this paragraph is good, why can’t all of your writing be this clear and concise”. And I’m thinking: “I know, I know, but how!”

109

Zehou 10.06.11 at 2:47 pm

Back to the original post for a minute:

What John recommends here is advice that is very familiar from my days as a TA (a decade ago): Grading is one thing, and commenting is a completely different thing. Grade slowly and seek out whatever aids to objectivity you can find. But never, never, ever write comments that have anything to do with the grade. If students want to talk about how to get higher grades or why they earned the grade they did, they’re welcome to seek you out in office hours. But the focus in comments should be on their pursuit of wisdom.

I’m generally a fan of this approach, but I don’t think it’s perfect. Nor do I think it’s alternatives are especially awful. For one thing, office hours mean something very different at residential colleges vs. commuter colleges. The in-office explanations and justifications are very available to students in the former, but largely inaccessible to students (especially those with full-time jobs) in the latter. So pushing the justification and explanation to office hours affects different student populations quite differently.

Writing a really good grading rubric for a very specific kind of writing assignment can be done well, I think. Harry’s colleague Peter Vranas has a nifty example on his webpage, for example. And even if it leaves students unsure about what’s really wrong with their paper, it can be the beginning of a helpful conversation about writing in office hours the same way comments about writing can be the beginning of a helpful conversation about grades in office hours.

110

ScentOfViolets 10.06.11 at 3:06 pm

Is it really that hard to see why, when I grade essays, there is a certain amount of qualitative squishiness to the assessment process, whereas when I write an mcq-style final exam I can not just say what the right answer is but actually program a machine to grade the tests!

Well that’s certainly not what you’ve been saying. It’s more like this:

Perhaps the problem is this: you are inferring from the fact that I say I have trouble explaining to students why they got the grade they got that I myself don’t feel I know why I am giving the grades I give. So therefore the grade is ‘changeable’, you infer. I feel a deep B-ish feeling welling up in me when I read x, so the student gets a B.

Yes, that’s exactly what you and others have been saying. Look up at the discussion where people are saying the time of day the paper is being read can effect the final grade, or the discussion where what might be a “C” one day could morph into a “B” next week.

If you disagree with those sentiments, now it the time to do so. But I think that here is the real nub:

This sort of sentence-by-sentence trench warfare is nothing you can wage across a whole 5-page paper. I’m not made of time, you know. But, often, there is no way to make the student see what you mean except to get nitpicky about some little bit. But then you end up not really providing ‘general comments’.

My jaw just hit the floor with that one. John, none of us our made out of time. But we’re paid to do our jobs, and grading and evaluating our students in a way that gives them relevant feedback is part of that job. I grade something like 1,000 to 2,000 problems each week. I spend a lot of time outside of the official workday grading each and every one of these problems, and doing so in that picky line-by-line detail that you just don’t have the time for. In fact, that’s the first activity that comes to mind when opponents of public schooling cite the easy hours and long vacations that teachers get as justification for cutting their already less-than-generous pay.

My point is that it’s important to say what you think will help the student see what went wrong.

Well, on that part we’re agreed. As I said earlier, maybe this is one of the reasons the sciences get more respect than the humanities; I don’t know a single math or physics guy who doesn’t put in long hours grading in that picky line by line fashion that you don’t have time for. Very, very few of my students have ever expressed dissatisfaction in what I’ve marked wrong in their work (how much I take off for those mistakes is a different story.)

It’s comments like yours that make me realize all of a sudden why some people might hold the teaching profession in rather low regard. I’m still blinking in amazement that you would say such a thing.

111

Jonathan Mayhew 10.06.11 at 3:36 pm

I think it’s more a question of raking one or two paragraphs over the coals and then having the student realize that you could have done the same with every other paragraph. The student has to extrapolate from the corrections of those two paragraphs what sh/e was doing wrong in the rest of it. In other words, you don’t have to spend more time grading a paper than the student spent writing it. The student is developing the capacity to be self-critical, responsible. That won’t happen if the professor notes every single thing that is wrong. Even vague comments like “organization is weak” can be strategic in this sense.

112

John Holbo 10.06.11 at 3:51 pm

“Yes, that’s exactly what you and others have been saying. Look up at the discussion where people are saying the time of day the paper is being read can effect the final grade, or the discussion where what might be a “C” one day could morph into a “B” next week.”

I think you are confusing two things, SoV. It’s one thing to say that grading may be inconsistent and that something you give a C one day could turn into a B if you only had more coffee. That’s a real concern and you have to take some steps to avoid that sort of thing (like not grading when you haven’t had enough coffee, and not grading too many hours in a row, and a bunch of other stuff). But it’s wrong to infer from the fact that there are consistency problems (there are) that graders must ‘weirdly’ have no idea why they are giving the grades they give, and (by additional inference) that this is the reason why I think shouldn’t try to explain why they give the grades they do. Namely, because they themselves don’t have any real notion why they are giving the grades they give. If I seem to be saying the opposite upthread – I don’t think I did – then I hereby set the record straight.

More generally, I am very much of the view that, as Zehou says, grading is one thing and commenting another. There are mechanisms for seeking fairness and consistency. But I don’t think insisting on trying to write comments as explicit justifications for the grade received is a good mechanism. It’s time-consuming, not very educational for students, and doesn’t end up ensuring fairness or consistency. Of course you are free to disagree, but even if you disagree you shouldn’t assume that, just because I don’t seek to ensure fairness or consistency in this way you favor, that I don’t seek to ensure fairness and consistency in other ways. Or don’t think it’s possible, or don’t care about it, or any other still more extreme and unlikely thing.

“My jaw just hit the floor with that one. John, none of us our made out of time. But we’re paid to do our jobs, and grading and evaluating our students in a way that gives them relevant feedback is part of that job.”

Well, yes. But it would probably take 2+ hours per paper to do what I would consider a full job. Probably more like 4 hours. That’s not practical. This week it so happens I have 70 short papers to mark for two classes. But I can’t, realistically, spend 140+ hours over the next 10 days just grading this batch – much less 280. In an ideal world I would have fewer students or more TA’s or some combination of the two. But we have to do the best with what we’ve got. Realistically, I can read every paper. And write some comments on every one. And then read them all again, doing suitable crosschecks for consistency. That takes at least 30 minutes per paper. Sometimes a bit more. So I have to figure out what the most efficient way to give relevant feedback is. It’s no good saying ‘I’ll give the style of feedback I would ideally give if I spent 4 hours per paper’ if I know that, in fact, I can’t spend 4 hours per paper. It’s better to give the style of feedback more suited to the time and effort I can, realistically, put in. Theory of the second best and all.

113

ScentOfViolets 10.06.11 at 4:51 pm

I think you are confusing two things, SoV. It’s one thing to say that grading may be inconsistent and that something you give a C one day could turn into a B if you only had more coffee. That’s a real concern and you have to take some steps to avoid that sort of thing (like not grading when you haven’t had enough coffee, and not grading too many hours in a row, and a bunch of other stuff).But it’s wrong to infer from the fact that there are consistency problems (there are) that graders must ‘weirdly’ have no idea why they are giving the grades they give, and (by additional inference) that this is the reason why I think shouldn’t try to explain why they give the grades they do. Namely, because they themselves don’t have any real notion why they are giving the grades they give.

Huh? You might want to re-edit what you just said, given what you’re having me say immediately beforehand.

More generally, I think you’re confused about what people have been telling you. Reread those comments; they’re not suggesting anything so crass as not enough sleep or coffee or grading too many hours in a row. They’re suggesting more subtle biases that can vary from day to day or person to person. Iow, as before, there is no “inference” on my part. This is what people have been saying. And it’s wrong of you to infer that they “really” must mean something else. Looking back, in fact, you say this yourself:

I just don’t think that you can realistically get graders to think rigorously in such terms. They won’t be able to separate those sub-aspects of organization in their mind, much less consistently identify them in the papers. They won’t all agree about what these sub-aspects amount to.

And:

The conclusion I’ve long drawn from this is that grading beyond “pass/fail” is a fool’s errand.

And:

And none of this deals with an even more important issue, which is that grading as between different professors teaching similar material is wildly inconsistent.

And:

Why not? I would give three reasons:
1) Doing it right would be ungodly expensive and time-consuming. It’s cheaper to do it wrong, and there’s no constituency to force you to do it right. The consumers of the data (grad schools and employers) apparently don’t care that the results are not reliable.
2) Doing it right would infringe on the autonomy of professors, who don’t want anyone else to tell them how to test and grade.

But students are well aware that the system is arbitrary. This is why they come to your door and whine and wheedle. It would be one thing if the grade had been assigned by a process that is transparent, objective, and reliable. But it wasn’t. So why shouldn’t they try to get a better grade?

And so on and so forth.

“My jaw just hit the floor with that one. John, none of us our made out of time. But we’re paid to do our jobs, and grading and evaluating our students in a way that gives them relevant feedback is part of that job.”

Well, yes. But it would probably take 2+ hours per paper to do what I would consider a full job. Probably more like 4 hours. That’s not practical. This week it so happens I have 70 short papers to mark for two classes. But I can’t, realistically, spend 140+ hours over the next 10 days just grading this batch – much less 280. In an ideal world I would have fewer students or more TA’s or some combination of the two. But we have to do the best with what we’ve got.

Iow, this is just like the drunk searching under the streetlight for his keys instead of by his car where he dropped them “because the light is better over here”. As someone else said upthread and which I just quoted: “But students are well aware that the system is arbitrary.” If you don’t have the time, you don’t have the time. But that makes a mockery of any sort of feedback process that grading supposedly gives.[1]

Don’t think I’m not unsympathetic, btw: I am, deeply. But this is part and parcel with any sort of “rating” system for students, or their teachers for that incentive “merit pay”, or general employee evaluations. Doing it right is expensive and time consuming. Doing it wrong is less than useless.

[1]This doesn’t hold in the hard sciences, of course, where you can apply a rubric with some sort of objectivity and a lot of effort.

114

Steve LaBonne 10.06.11 at 5:13 pm

This doesn’t hold in the hard sciences, of course, where you can apply a rubric with some sort of objectivity and a lot of effort.

However, the issue of inconsistency in grading between professors remains. And it’s not a simple issue since it’s affected both by the type / difficulty of exam questions and by each professor’s individual notion of what a reasonable grade distribution looks like. Speaking as a scientist myself, I think the self-righteousness in your last couple of comments is a wee bit over the top.

115

Jonathan Mayhew 10.06.11 at 5:13 pm

Using an objective-seeming rubric is actually faster. You could quickly go down and assign scores in different areas and add them up. It is not hard to assign grades and justify them, and the variation due to subjectivity leads to only slight variations. Between a B and B+, for example. A C paper never turns into a B the next morning just because I’m less tired. It seems odd on this thread that people are objecting to giving students more than they ask for, substantive comments rather than a simple numerical score, but then complaining that these substantive comments need to be even more extensive.

116

ScentOfViolets 10.06.11 at 5:39 pm

However, the issue of inconsistency in grading between professors remains.

True. But I was addressing Holbo’s claim about what he thought people were saying as opposed to what they were actually saying, so that’s a different issue.

And it’s not a simple issue since it’s affected both by the type / difficulty of exam questions and by each professor’s individual notion of what a reasonable grade distribution looks like. Speaking as a scientist myself, I think the self-righteousness in your last couple of comments is a wee bit over the top.

Unless the class being offered is a first- or second-time offering, the grade distribution is what it is. And um, . . . I certainly don’t see any self-righteousness being expressed. Look again. What I am saying is that the students I’m familiar with – even the artsy types who have no use for math or science – certainly respect the science-types more, even if they don’t like them. Otoh, I hear a lot of talk about sucking up to certain humanities professors just because they have some idea of how arbitrary the grading is.

And I’m sorry, but when I hear some prof say they just don’t have the time to really grade in a manner that is transparent, objective, and gives good feedback, it’s kind of difficult for me to defend them from the public-school-is-evil-and-ineffective-and-teachers-are-featherbedding-incompetents pro-voucher types.

Yes, grading and giving good feedback is hard. It’s also what we get paid for. Deal with it.

117

ScentOfViolets 10.06.11 at 5:45 pm

Using an objective-seeming rubric is actually faster. You could quickly go down and assign scores in different areas and add them up. It is not hard to assign grades and justify them, and the variation due to subjectivity leads to only slight variations. Between a B and B+, for example. A C paper never turns into a B the next morning just because I’m less tired.

And that is exactly the way I do it. I also do things like grade everyone’s exam problem one at the same time before proceeding to exam problem two, using exams with a coversheet which I turn back immediately so I don’t see the particular student’s name or ID before I start grading, etc.

It seems odd on this thread that people are objecting to giving students more than they ask for, substantive comments rather than a simple numerical score, but then complaining that these substantive comments need to be even more extensive.

Indeed.

118

socratic_me 10.06.11 at 6:01 pm

SoV,

As someone who spent some philosophy grad school time grading essays and a lot of high school math teacher time grading tests by hand (so as to give partial credit and better commentary that students can use to improve their thinking about math), I think the distinction between hard and soft science grading is being overstated in your comments. I know that I have to double check my early grades against my later grades when doing tests because I tend to understand what mistakes my students are making and why they are making them better (and thus become more generous) the more I work through papers. I have various mechanisms I use to help ensure more accurate and consistent grading occurs, but they are very similar to the type of steps I took as a grad student to make sure that I wasn’t letting my expectations fall off as I was increasingly worn down by common mistakes.

The interesting thing is that rubrics tend to make the grading unfair in a different way. I used to teach AP Calc and in prepping kids for the written response portion of the exams, I got very good at training them to show off any little bit of info they had by manipulating the way that we knew the grading rubrics were used. It ends up creating a situation where those who are savvy about test taking and gaming the system do much better than those who aren’t with very little respect to their actual mathematical knowledge or ability.

All of which is to say, as a math teacher, I think that teachers of math and other hard sciences often overestimate just how objective and fair their assessment is as compared to the soft sciences.

119

Steve LaBonne 10.06.11 at 6:02 pm

Unless the class being offered is a first- or second-time offering, the grade distribution is what it is.

And “what it is” can be quite different 1) between different courses at a similar level, and 2) among different sections of the same course in a large university, so I’m afraid this can’t be waved away quite so easily. Your hard-working science professors in fact have typically put little or no item and effort into rectifying even pretty gross inconsistencies of this kind. So I must once more dissent from the science triumphalism of your comments.

120

ScentOfViolets 10.06.11 at 6:14 pm

All of which is to say, as a math teacher, I think that teachers of math and other hard sciences often overestimate just how objective and fair their assessment is as compared to the soft sciences.

That’s a fair cop, Guv :-) But I think – speaking as one math teacher to another – that you really do go through each and every problem line by line, right? And that you grade everyone’s problem number one first before proceeding to everyone’s problem number two just so you can go back and correct any deficiencies in your grading – maybe you realize after the same wrong answer five times in a row that you’ve asked an ambiguous question, say, or there’s a typo on the exam that you missed.

This seems to be a standard part of good practices everywhere I’ve worked, so I’m guessing that what I’m outlining is no surprise to you, and in fact by far the norm rather than the exception.

121

ScentOfViolets 10.06.11 at 6:29 pm

And “what it is” can be quite different 1) between different courses at a similar level, and 2) among different sections of the same course in a large university, so I’m afraid this can’t be waved away quite so easily.

All I can say is that in my experience when such discrepancies arise – and yes, of course they do, it would be silly of me to deny it – people certainly hear about it. Especially if it’s your 2). That’s way not cool. And that’s apparently the difference between the sciences and the humanities, to judge from this isolated discussion.

The one thing we’ve found over the years is people taking a 7:50 or 3:50 slot just don’t do as well on average as those who take an 8:50 or 9:50 slot, despite our best efforts to make sure this doesn’t happen. Following their stats as best as we can, the department has concluded, reluctantly, that these students simply are not as good in these particular subjects. Graders from later sections, for example, still grade the earlier sections lower in double-blind grading.

Your hard-working science professors in fact have typically put little or no item and effort into rectifying even pretty gross inconsistencies of this kind. So I must once more dissent from the science triumphalism of your comments.

And I must once more point out that you haven’t exactly documented this triumphalism. If it’s there, quote it. For that matter, you’re simply asserting that these people have put little to no effort into rectifying these inconsistencies, which has not been my experience. In sum then, sometimes if the evidence points to an unwanted conclusion, well, the sometimes the evidence points to an unwanted conclusion.

122

Steve LaBonne 10.06.11 at 7:14 pm

All I can say is that in my experience when such discrepancies arise – and yes, of course they do, it would be silly of me to deny it – people certainly hear about it. Especially if it’s your 2). That’s way not cool. And that’s apparently the difference between the sciences and the humanities, to judge from this isolated discussion.

This is simply not true at the large state university my daughter (an engineering major) attends.

And I must once more point out that you haven’t exactly documented this triumphalism. If it’s there, quote it.

I did quote a typical example, above.

123

ScentOfViolets 10.06.11 at 7:22 pm

This is simply not true at the large state university my daughter (an engineering major) attends.

Anecdotal data and all that. However two points – you made the claim, so you back it up. I merely pointed out this has not been my experience. The other point is that your daughter is a student (and where you presumably picked this one up); I’m on the other side of the desk.

I did quote a typical example, above.

You’ve quoted a number of things now, none of which seem to have that note of triumphalism that you find so objectionable. So you’re going to have to be specific.

124

socratic_me 10.06.11 at 7:25 pm

But I think – speaking as one math teacher to another – that you really do go through each and every problem line by line, right? And that you grade everyone’s problem number one first before proceeding to everyone’s problem number two just so you can go back and correct any deficiencies in your grading – maybe you realize after the same wrong answer five times in a row that you’ve asked an ambiguous question, say, or there’s a typo on the exam that you missed.

It is indeed. It was also pretty common practice when I was grading essays. In fact, based only on my purely anecdotal observations, I would say it was more prevalent among the philosophy graders than among my fellow math teachers because the math teachers assume objectivity that just isn’t there.

125

Steve LaBonne 10.06.11 at 7:26 pm

Anecdotal data and all that. However two points – you made the claim, so you back it up.

It took a remarkable lack of self-awareness to type that.

126

andthenyoufall 10.06.11 at 7:28 pm

Apologies if I’m repeating something that was said up-thread… that’s 120 longgg comments.

Having students do peer assessments has seemed to me a useful way to accomplish “1”. It’s hard to actually give an analytic account of why a paper gets a certain grade (although I do show them a rubric which I use as a heuristic). But if they’re forced to think critically about a peer’s paper, they grades they give are uncannily close to the grades I give, and they have a much stronger grasp of what kind of judgment went into evaluating their own paper. Then my written comments can focus on something important.

127

Salient 10.06.11 at 7:42 pm

This doesn’t hold in the hard sciences, of course, where you can apply a rubric with some sort of objectivity and a lot of effort.

(Responding because, in a suitably limited sense, I agree with you. I might choose to not follow up on it.) The problem that I encounter is: to be anywhere remotely objective requires one to either [1] relax rubric-standards until there is some amount of subjectivity available for me or [2] be really fucking vicious. “This person has written something that makes no sense, but seems to somewhat closely resemble something that could make quite a lot of sense, under some suitably expansive definition of closely and could” is either [1] partially creditable or [2] not creditable.

If it is creditable, then it’s just as hard and subjective to grade as an English paper. If it is not creditable (i.e. if I grade according to achievements listed on a pre-formed rubric with no flexibility) then quite a lot of folks will receive no credit for having some poorly-formed but potentially evolving-toward-better-formed intuition that happens to not align well with the preconceived standard. (Which hey, making this kind of hmm-maybe-they’re-getting-there quality work creditable what pre-exam quizzes are supposed to be for. Please don’t lose sight of the fact that I’m expressing some agreement with you.)

I want to give my students credit for moving from a crappy to a measurably-less-crappy sense of logic and pattern. But they’re still going to be speaking a lot of absurd nonsense at that stage, frankly. (That’s an empirically verified fact.) So gauging their progress according to objective standards is quite hard, because, like me, students are inventing new ways to not quite make sense every waking minute of the day. :)

I don’t expect students to leave my 101 classes suddenly capable of ‘doing’ mathematics any more than, say, a professor of psychology would expect their students to be capable of ‘doing’ psychology. (That is, I don’t expect them to demonstrate mastery of the field, I expect them to leave with demonstrably improved facility with some tools of the trade, and with some sense of the craftsmanship involved in pattern-finding technique.) Judging their progress toward facility and craftsmanship, in a way that acknowledges fuzzy and ill-defined improvement, is every bit as hard as I imagine it would be for a humanities course. (And on top of that, as a graduate student I’m still in the process of developing extensions of the craftsmanship of others, and constructing and establishing mastery of a subsubfield.)

So hey, if you’re just super duper awesome at grading and assessment, ok. A fair amount of what you are saying makes sense in theory. But quite a lot of the rest of us struggle more than you do–and it’s not a question of disparity of “effort” I don’t think. Especially as I don’t feel like I learned any better sense of the mastery of technique from your comments– #95 just sounds like bragging. (No, I can’t “prove” to you that you were bragging. Or at least, I have no intention of attempting to do so.)

128

Meredith 10.06.11 at 7:45 pm

Just to point out that John is right about discursive essays: unit by unit, they take much longer to read and comment on than do things like problem sets, if I may compare the latter to correcting homework assignments, daily quizzes and tests in lower and intermediate level foreign language classes. They just do. One (among many reasons): each essay has its own voice, its own distinctive approach, and perhaps its own distinctive mode of argumentation, and as its reader you have to orient yourself to all those things before you can even begin to evaluate how successful the essay is or how it may have failed in its own purposes. Even the five-paragraph “canned essay” leaves infinite room for legitimate variety in all these areas. Once you think you understand what the writer is trying to achieve, it can still be very difficult to identify precisely what she needs to do to improve, and you may need to read several different essays by the same student to discern the habits of constructing sentences or organizing paragraphs that she needs to modify, in accordance with her very individual voice and mode of thinking.
It’s often more useful to the student to isolate, say, just one sequence of sentences or one paragraph, and to go over the problems there exhaustively. A paper with a million things marked up is just dismaying to the student. Meeting with a student about a paper — especially if you’ve already met with them as they were preparing to write it — is also hugely valuable. For one thing, by that time they’ve gained some distance from what they’ve written and
One thing that hasn’t come up here: revising/rewriting. We all do that with anything serious we’re working on, usually after having gotten feedback from colleagues (and then again from readers and an editor). Having students revise an essay may be the most effective tool at our disposal for guiding them in improving their writing. The main reason we don’t do more of it, I think, is time — students’ time spent rewriting and our own time spent rereading, when time is needed for giving full attention to the next portion of the course. One reason advanced seminars in the major or senior theses can be so valuable to students is that these are the contexts in which rewriting and revising can (or, in the case of theses, must) be built in.

129

piglet 10.06.11 at 10:13 pm

SoV: “I grade something like 1,000 to 2,000 problems each week.”

“Grade”? Do you mean “grade”, or do you mean “check a result that could also have been checked by a computer”? If you mean something involving a serious evaluation of work submitted, I can’t imagine doing a decent job in less than a minute per item, and that is physically almost impossible since you’d spend most of your working time grading (33 hours per week in your example).

I am slightly baffled. Everybody here claims to be investing serious time and energy to fairly and impartially grade student work, going the extra mile and all that, and yet we all know that most academics don’t have the time that would be required to really do an appropriate job evaluating student work and giving meaningful feedback. Either everybody except for myself here is living in academic Utopia or there is a serious cognitive dissonance. I have seen a lot of grading up close at a not too terrible public University and I find it hard to believe that my own admittedly anecdotal experience should be a total outlier. In my experience, in large intro classes, student work is NOT returned. If only for practical reasons – rules don’t allow passing large piles of papers through class because students are not supposed to see other students’ grades. Also, most of the grade in large classes is based on scantron. There’s no point making comments and returning multiple choice answer sheets. Obviously there are courses that work differently but arguably they are the exception and not the rule.

130

piglet 10.06.11 at 10:18 pm

Also, remind yourself how some of the “grades” that really count are made – GRE and the like: Computer algorithms plus some essay grading by lowly paid students, resulting in scores that are known to correlate with essay length. Those scores are far more relevant in terms of career impact than whatever you all are doing.

131

ScentOfViolets 10.06.11 at 10:19 pm

Anecdotal data and all that. However two points – you made the claim, so you back it up.

It took a remarkable lack of self-awareness to type that.

Sigh. Really? Because it looks to me like it took a remarkable amount of laziness to type that. Let’s hit the instant replay, shall we?

This doesn’t hold in the hard sciences, of course, where you can apply a rubric with some sort of objectivity and a lot of effort.

And you replied with:

And “what it is” can be quite different 1) between different courses at a similar level, and 2) among different sections of the same course in a large university, so I’m afraid this can’t be waved away quite so easily. Your hard-working science professors in fact have typically put little or no item and effort into rectifying even pretty gross inconsistencies of this kind. So I must once more dissent from the science triumphalism of your comments.

So, uh, it looks like you better chill just a bit. And, as I said, produce some evidence for your assertions.

Next time, review the comments before posting.

132

john c. halasz 10.06.11 at 10:27 pm

SoV:

You might find this radio program of some interest:

http://americanradioworks.publicradio.org/features/tomorrows-college/lectures/

133

ScentOfViolets 10.06.11 at 10:44 pm

“Grade”? Do you mean “grade”, or do you mean “check a result that could also have been checked by a computer”? If you mean something involving a serious evaluation of work submitted, I can’t imagine doing a decent job in less than a minute per item, and that is physically almost impossible since you’d spend most of your working time grading (33 hours per week in your example).

As always, specific examples are enlightening. I gave an exam last week in my fundamentals of algebra class where a typical question was (I’m using LaTeX conventions):

Find the value of the following expression if $x=-2$ and $y=5$: $\cfrac{x^2+1}{4x+5y^2}$.

That’s a 5-pointer, and takes maybe five seconds to grade, tops, if it’s right. A typical mistake – for which one point is deducted – is substituting in -2 for x in the x^2 bit and getting -4. Needless to say, it doesn’t take one minute to grade a problem like this. More like ten seconds, on average.

Also needless to say, most of those 1,000 questions are questions of this type with varying degrees of complexity. If it’s a question in my stat class, say dealing with conditional probability and a tree with four branches, it takes a bit longer, on average about a minute. But I only have on the order of a hundred of those to grade. And if it’s a problem in linear algebra, say using eigenvectors to find an orthogonal basis for a general ellipse, well, those can take longer still, but there are correspondingly fewer problems (and the mistakes aren’t the kind where you’re drowning in nickel-and-dime stuff).

I don’t think these figures are inaccurate or wildly out of line with what a lot of other math teachers experience. But yes, while it’s not 33 hours, it’s definitely considerably more than ten hours a week grading outside of class.

Now I happen to think that’s a lot of time spent grading and as I already said, when certain anti-teacher types start to go on about thirty-hour work weeks with three months of summer vacation, I do my best to disabuse them of this notion that us academic types got it soft. So when someone says grading is hard but they just don’t have the time to do it right so you go with what you got, well, that does tends to undermine my defense of how hard we work :-(

134

John Holbo 10.06.11 at 11:40 pm

SoV: “More generally, I think you’re confused about what people have been telling you. Reread those comments; they’re not suggesting anything so crass as not enough sleep or coffee or grading too many hours in a row. They’re suggesting more subtle biases that can vary from day to day or person to person. Iow, as before, there is no “inference” on my part. This is what people have been saying. And it’s wrong of you to infer that they “really” must mean something else.”

Well, perhaps it’s simplest for me just to say (again): while I admit that the consistency problem is real and serious, I haven’t been talking about it, apart from a few incidental remarks, because it is my view that the way to address it isn’t to mandate a particular style of comments. So what I say about how to write comments should not be read as a callous refusal to admit that there are serious issues concerning how to ensure consistency in grading.

I was responding to what you thought was weird in what I said. It seemed you thought I was saying that I think graders really have no idea why they give the grades they do. That is not the case. You have suggested that I am exhibiting a shocking disregard for basic standards. But you are simply assuming that if the grading process is not being done perfectly, to the highest imaginable standards, that it must be done ‘arbitrarily’. But that’s not a reasonable assumption. (‘The perfect is the enemy of the good’ diagnoses a fallacy. It is not an assault on common sense and decency.)

Perhaps you will think I am misreading you. Well, possibly. Then please just explain what you think is so shockingly ‘weird’ about what I am proposing, and why you think it entails that ‘the system is arbitrary’.

135

ScentOfViolets 10.06.11 at 11:59 pm

Just to point out that John is right about discursive essays: unit by unit, they take much longer to read and comment on than do things like problem sets, if I may compare the latter to correcting homework assignments, daily quizzes and tests in lower and intermediate level foreign language classes. They just do.

For something like this as opposed to a project problem, I’d certainly agree.

But that doesn’t mean you can say that since it’s really, really hard you can just say that your grades and comments are accurate, transparent, give good feedback etc.

That’s just wrong.

136

Jon 10.07.11 at 12:11 am

Scent, I’m confused. You’ve convinced me (in #133) that, in your Discrete Math class, you can give exams that are objective and easy to grade fairly. But the exams I give (I teach in a US law school) are not objective, and are much more difficult to grade fairly. Further, I could not give an objective exam and still capture the things I’m looking for. I think you get this. What I don’t understand is [1] why you think it’s “weird” (#95) that people who give exams that are hard to grade objectively, have a harder time grading them than do people who give exams that are easy to grade objectively; and [2] why you’re convinced that people who give exams that are hard to grade objectively, work less hard at grading than you do. We face real challenges in grading stemming from the nature of our disciplines (challenges, as you point out, that you don’t face). We acknowledge those challenges. That doesn’t mean that we work less hard than you do. Some of us do, I expect, and some of us don’t (the world is like that). I’ve surely got no reason to believe that Holbo does.

137

Meredith 10.07.11 at 12:18 am

Could I ask: when we create assignments beyond the daily reading/study preparation for class, are we assigning essays, problem sets, tests (and so forth) in order to be able to give grades at the end of the course? Or to promote students’ learning? I realize that the latter goal, which I take to be THE goal (with grades almost an incidental product), has somehow been lost in much of this discussion.

138

Ken Lovell 10.07.11 at 1:10 am

Piglet @ 129 gives me heart that my university and I may actually be quite forward-thinking in our assessment practices. I haven’t accepted an assignment in hard copy for years (apart from final exams, on which no feedback is given) and all assignment comments are made using the wonders of modern technology such as Microsoft Word.

John your opening statement was reminiscent of what so many business students say, especially MBA students: ‘I’ve been managing people half my life, so I think I know a thing or two about how it should go.’ The inference is that people who actually study management, viz academics, have nothing valuable to impart. The thread here has been notably lacking input from teaching and learning scholars. I’m not one, but as a professional teacher I try to stay abreast of teaching and learning scholarship, and there are sound reasons why the subjective, I-know-a-good-essay-when-I-see-it approach has been thoroughly discredited.

A well-designed rubric (which will look nothing like the illustration #26 but will have extensive criteria directed at the specific learning objectives of the assessment task) makes function 1 in the OP comparatively straightforward and takes little time to apply. It does however take quite a lot of time and effort to design in the first place, mainly because it forces academics to think through exactly what learning objectives they want to assess and what evidence they will use to make their evaluation. To say “Oh that’s just too hard and academics won’t/can’t do it” is simply a judgement of widespread professional incompetence, which may be accurate in some places but not in my experience. The objections to rubrics that I have observed are usually nothing more than the usual resistance to change, coupled with a resentful feeling that they are somehow undermining the status (and power?) of the professor.

Much of the thread implies an either/or outcome, as if giving students a comprehensive explanation of the basis of their mark precludes ‘communicating something significant that will teach the student to be a better writer/thinker’. Using rubrics makes the first task quite fast, as per #115. The latter kind of feedback can be tailored to meet the needs of individual students provided proper learning resources have been provided in the first place. For example, ‘you should download the advice on report writing from [URL] and consult it frequently next time you have to write this kind of assignment’, or ‘you have misunderstood the nature of path/goal theory – I suggest you watch the recorded lecture again starting about 30 minutes in, or read the textbook pp x-y’. Of course many (most) students will ignore this kind of advice but it doesn’t take long to write. Those who sincerely wish to learn will benefit.

Finally, what happened to asking a student to drop by for a chat or give you a call? Interactive feedback is often more effective than anything you can write, and it lets students who want this kind of help self-select. At least at my institution, students enrol with very diverse aspirations and expectations. Trying to apply a one-size-fits-all approach to assessment comments is simply not appropriate.

139

John Holbo 10.07.11 at 1:13 am

“But that doesn’t mean you can say that since it’s really, really hard you can just say that your grades and comments are accurate, transparent, give good feedback etc.

That’s just wrong.”

SoV, I’m not saying that, because my approach isn’t perfect that necessarily it’s good. I’m saying that, just because it isn’t perfect, it doesn’t follow that it can’t possibly be good. You are saying that if I’m not doing my job perfectly I must be like the guy who is looking for his keys under the streetlight when he lost them somewhere else. That is, you are saying that non-perfect efforts are, necessarily, completely vain and valueless. I’m pointing out that this attitude seems lacking in common sense (unless you know better, and I am missing something here.)

You also seem to be suggesting that, unless I have solved the consistency problem to perfection, I can’t have solved the ‘how to comment’ problem at all. That is, I can’t possibly be providing helpful feedback on papers unless I’m grading them with perfect consistency because (do you think this?) the feedback itself should, as it were, contain a transparent proof that I’m being consistent overall. I don’t think that follows at all, to put it mildly. Obviously one shouldn’t strive to do this, but I think it would be possible to be a quite inconsistent grader and yet be a very good paper commenter. You have the knack for spotting problems, for really putting your finger on the sore spot and making good, instructive diagnoses that help people learn to write, but you are erratic about giving consistent weightage to these factors, in arriving at grades. I think it’s important to see how separate are the activities of giving useful comments and assigning fair grades.

140

John Holbo 10.07.11 at 1:33 am

“all assignment comments are made using the wonders of modern technology such as Microsoft Word.”

I use MS-Word, too. If my post seemed obscurely luddite in spirit, I herely avow that I am very pro modern technology. Softcopy submissions only!

“A well-designed rubric (which will look nothing like the illustration #26 but will have extensive criteria directed at the specific learning objectives of the assessment task) makes function 1 in the OP comparatively straightforward and takes little time to apply. It does however take quite a lot of time and effort to design in the first place, mainly because it forces academics to think through exactly what learning objectives they want to assess and what evidence they will use to make their evaluation.”

Obviously #26 was not well-thought out. Having spent time in the past designing such rubrics, I know it well. It’s not a worthless thing to do, by any means. But it doesn’t really eliminate the problem of potential inconsistency, it just diffuses it. We agree to give 2/3 weight to ‘content’ and 1/3 to ‘form’, say, but – at the end of the day – people don’t agree in the ways in which they notionally abstract form from content, so there is room for the very slippage we wanted to avoid. And if we further subdivide ‘content’ and ‘form’ and make things even more complicated, we just create more chances for people to have different private senses of how these abstractions of paper elements are supposed to be achieved.

Even the well-designed ones have the same problems as #26, in the end, unless the assignment itself is very tightly constrained. And I don’t always want to give assignments that are that constrained. I want to give students some leeway to approach the topic from different angles, or perhaps part of the assignment is to devise a topic, come talk to me about it, then write the paper. Then everyone is writing on different topics and rubrics are harder to apply. You get more apples and oranges type problems. In such cases, there are better tools for reducing inconsistency. Mostly, making sure that you reread and cross-check and, if there are several tutors, do suitable cross-checks. Obviously it isn’t perfect.

One issue at this point is the following: you might argue that, since fairness is important, you should never assign an essay topic on which students are permitted to take significantly different approaches – and you should certainly not allow students to pick their own topics. You have to lay down the path along which they will write very narrowly in advance (otherwise you end up with apples and oranges problems, and that means inconsistency in applying any rubric you’ve got.) There’s something to be said for this attitude, but it does constrain the kinds of assignments you can assign.

“Finally, what happened to asking a student to drop by for a chat or give you a call? “

I advocated it!

141

John Holbo 10.07.11 at 1:39 am

“Much of the thread implies an either/or outcome, as if giving students a comprehensive explanation of the basis of their mark precludes ‘communicating something significant that will teach the student to be a better writer/thinker’.”

It’s not really right to suggest that the thread doesn’t consider that these things may go together. Rather, the post begins by positing that they are likely to come apart (which is consistent with hoping that they will, all the same, go together). It’s nice when they go together, as well they may. But they can come apart, and the ‘comprehensive explanation of the basis’ is quite likely to be an elusive goal. So it should be let go, the better to focus on the other. Just try to teach better writing. If you end up providing as well something like a stand-alone mini-proof of why the paper got exactly the grade it did, then that’s a welcome bonus.

142

Ken Lovell 10.07.11 at 2:01 am

John I think some of the disagreement in a very fascinating thread arises from the fact that people are coming to the discussion from different disciplinary backgrounds, and therefore conceptualise the purpose of assessment differently. However there is no reason why rubrics cannot be designed to cover multiple approaches to a topic. Always the test is what learning outcomes are we trying to assess? What evidence will we use to evaluate the extent to which students have achieved them? If we are clear about these things we ought to be able to express them systematically. If we can’t answer the questions then it suggests either the assessment task or the course objectives have been poorly designed.

I agree that rubrics cannot guarantee consistency but they are very useful in moderating assessment amongst a team of graders. If academics are giving different grades it means there are inconsistencies in the way they are interpreting the marking criteria, and the rubric then becomes the agenda for finding out where the differences are arising. This still involves a considerable amount of time and discussion, and there’s still plenty of scope for problems to arise, but at least it avoids long aimless debates about the purpose of the assessment task.

The discussion is something of an pointless exercise for business schools – if they want international accreditation, which most do, they have no choice. They will have to use rubrics, not only for individual assessment tasks but also to link to overall program learning outcomes and graduate attributes. I certainly agree that we should always try to go beyond this and give feedback that will help individual students learn to do better, while acknowledging that many students don’t want this kind of advice and it’s not our role to try to persuade them to change.

143

ScentOfViolets 10.07.11 at 2:25 am

What I don’t understand is [1] why you think it’s “weird” (#95) that people who give exams that are hard to grade objectively, have a harder time grading them than do people who give exams that are easy to grade objectively; and [2] why you’re convinced that people who give exams that are hard to grade objectively, work less hard at grading than you do.

[1] is definitely not what I think. What I think is weird is that despite being harder to grade, little to no effort is being made to use it as some sort of diagnostic or relevant feedback to students. You apparently get your A or B or C on your paper, a few vague comments that some teachers here have freely admitted aren’t terribly helpful, and away you go. I can see why students have problems with this style of grading.

[2] is kinda sorta but not really – I don’t think I’ve said anything one way or another on that one. What I have said is that if it is hard . . . so what? It’s your job to do this sort of thing (at least, as I interpret my job on the math/science side). If it’s not possible to do this and you’re going off the “I know a B paper when I see one” rubric, I don’t see how you can justify to your students that you’re being fair and consistent.

Yes, I do have students who will come up to me and demand an explanation as to why they received different grades for the “same” work. And most of the time it’s trivially easy for me to show why. I’ve had very, very few students dispute a grade (except of course for end-of-semester grades) once I’ve explained my reasoning and shown them my rubric.

144

ScentOfViolets 10.07.11 at 2:47 am

Could I ask: when we create assignments beyond the daily reading/study preparation for class, are we assigning essays, problem sets, tests (and so forth) in order to be able to give grades at the end of the course? Or to promote students’ learning? I realize that the latter goal, which I take to be THE goal (with grades almost an incidental product), has somehow been lost in much of this discussion.

Of course we’ll all say (and mean it) that it’s to promote student learning. My kin around the table when we meet on certain occasions are somewhat taken aback when I say I’m relatively uninterested about what students get right. Well, of course I am; once they’ve learned the material to my satisfaction, my job is done and they get handed off to the next set of guys on the education treadmill.

What I am much more interested in, what is really my bread and butter, is what my students get wrong, and why. Over the years I’ve found it’s not so much that my kids aren’t learning as much as their “mislearning”. Accordingly, exams, homework, etc. are diagnostic tools. A right answer is a right answer, mostly. But there are many ways to go wrong. Spotting a pattern in the way wrong answers accumulate actually tells me something about what’s been “mislearned”.

Another example: a mistake some students make is the improper multiplication of integrals. They think that because you can freely pull out sums from under an integral and integrate them separately that you can do the same thing for products. Well, you can, sometimes.

Now, you’d think that this would be easily corrected but once I cogged onto it, something much interesting was happening: turns out that a lot of these students had the wrong idea of what it meant for a function or an operation to be what we in the biz call “linear”. Catch that error, correct them on it, and it’s amazing how your exasperation at these rumdums who just couldn’t get it and were guessing apparently at random changes into “Gee, I really should make sure before we do differentiation and integration that all of my students really know what the term linear means.”

I guess this is my long-winded way of saying that not everyone regards quizzes, essays, exams etc as primarily a diagnostic tool.

145

Meredith 10.07.11 at 3:38 am

By diagnostic tool, I assume you mean a tool for grading. (Of course these exercises are diagnostic tools for other things, no? Where students are strong, what they still need to work on, where the teacher needs to devote attention in class or in individual conferences, where the teacher needs to develop better ways to present or explain the material, or better ways of modeling the kind of analysis the students are expected to be learning.) And more than that: quizzes and tests not only motivate students to study (grade as motivator) but also, and more important, are themselves modes of learning. (Information-retrieval as reinforcement, and all that.) Essays are less about information-retrieval from memory, but they, too, give students the opportunity to focus their attention and, most of all, do research and analysis on their own: a chance to speak (especially important in larger classes).
I’m more concerned that, in getting caught up in figuring out how to provide students crystal-clear criteria for our grading (and that can be very hard to do for many kinds of essay in the humanities and the humanistic social sciences, no matter how much time you might devote to it), we may end up putting the cart before the horse (or letting the tail wag the dog), that is, we may find ourselves designing assignments in terms of the clear grading criteria they enable us to provide students rather than to serve other goals that may be more important. It’s those UK guidelines — and similar things sometimes proposed in the US — that make me nervous. As John says in his post, “And 2 is more important, and do-able, so basically you should just do 2. Clear your head of the vague feeling that you should be doing 1, except a bit around the edges, in the natural course of doing just 2.” I think he’s got it right there. I’d only add that, often, in order to help students write better essays, we need to think beyond the kinds of comment we may write on them and consider how we structure the whole course, and the place of writing assignments within that structure.

146

Zehou 10.07.11 at 4:30 am

Ken: “there is no reason why rubrics cannot be designed to cover multiple approaches to a topic. Always the test is what learning outcomes are we trying to assess?”

Allowing multiple approaches in some cases means that different students are pursuing different learning outcomes, right? If Sammy wants to try to develop a novel, plausible, and charitable interpretation of an argument in such-and-such work of philosophy, and Susie wants to try to refute that argument, understood according to the standard interpretation of it, then there isn’t one learning outcome that we’re trying to assess by means of the paper.

Now, you can just make your rubric more complicated, of course, by including a lot of disjunctive criteria. But then you might decide that the maximum points limit in one line of the rubric is just right for the refutation project but not for the interpretive project. So then you discover you need to make the rubric flexible–2 points or so for this–or you need two different rubrics. But then Joan comes in, and she has an interesting plan to contextualize the argument by comparing and contrasting it with one found in a little-known earlier work philosopher so-and-so would have known. So then you need a third rubric. And a fourth and a fifth. But finally everyone has their plan approved, and you have your five rubrics (or one really, really complicated one) ready to go on the due date of the paper, only to discover that Joan and Sammy wound up radically changing their plans in the days before the due date, but wrote very carefully prepared papers of yet unconsidered types nonetheless, so your set of rubrics is now inadequate. You can make more. Or you can refuse to accept Joan and Sammy’s papers, and vow to make many more rules about what the paper has to be like in future.

Or you could quit with the rubrics. They’re useful aids to objectivity, and they’re sometimes aids to efficiency, but they aren’t always worth the cost in time you could spend . . . giving students helpful feedback on their work. And not of the “reread such-and-such pages” sort, but comments that are much deeper–comments that help students see what a smart, skeptical reader is thinking as she tries to work through some important part of the paper’s argumentative or interpretive or contextualizing project (or whatever else the student comes up with).

147

John Holbo 10.07.11 at 4:57 am

“What I think is weird is that despite being harder to grade, little to no effort is being made to use it as some sort of diagnostic or relevant feedback to students. You apparently get your A or B or C on your paper, a few vague comments that some teachers here have freely admitted aren’t terribly helpful, and away you go. I can see why students have problems with this style of grading.”

Well, but you were applying the ‘weird’ label to what I said, SoV. I take it you recognize that, at the very least, my intent is NOT to do this ‘give a few unhelpful comments and basically call it a day’ thing. Explicitly, my approach is targeted at avoiding that, of all things. My feeling is that the reason you do 2) not 1) is precisely to avoid this weird and unsatisfactory result. Because I think trying to do 1) is, in practice, this thing you deplore. (Because to really do it would take 4 hours per paper, but I don’t have 4 hours per paper. So rather than doing it badly, do something else well.)

Let me make another comment against what Ken Lowell wrote, above: “The objections to rubrics that I have observed are usually nothing more than the usual resistance to change, coupled with a resentful feeling that they are somehow undermining the status (and power?) of the professor.”

My considered view is that this is exactly upside down and backwards (and if the advanced scholarship on the subject says otherwise, I would be curious to see the argument, but I am gravely suspicious.) If what you really want is freedom to do what you like without being challenged, as a grader, then rubrics are honestly your best friend. Because they provide you with the appearance of rigor, a quantitative veneer, without actually constraining you. I can always say ‘the reason you got the grade is because you got a 3 on content, a 2 on organization, etc. and it all added up to your grade.’ No one can gainsay all these individual numbers I have assigned and they make it sound like I have been very precise and careful. The rubric provides the very thing that Ken hopes to deprive the grader of: status and pseudo-authority. Honestly I might have just slapped down these 3’s and 2’s haphazardly, or at least rather idiosyncratically. Who’s to know? By contrast, requiring that graders hit a grade curve, or check each others’ work – that has real teeth. And it has nothing to do with the comments you write on the paper. Again: comments should aim at teaching. Separate their character from the issue of enforcing grading consistency.

148

Ken Lovell 10.07.11 at 5:00 am

Zehou it sounds to me like the learning outcomes you want to assess are (1) depth of understanding of a text, and (2) the capacity to develop a coherent argument about it. A single well-designed rubric can be used as a framework for assessing those; there is no need to design multiple rubrics just because students will adopt different approaches to the assessment task. Most of my assessment tasks are designed to test students’ understanding of theory and capacity to critically evaluate it, and it’s not terribly hard to design an appropriate rubric.

Like I said before, these are issues that have been extensively researched by teaching and learning scholars and there is a huge amount of published material available. It’s a bit disconcerting to read so many people trying to make a case from personal experience, as if the body if existing knowledge doesn’t exist or isn’t worth consulting.

149

John Holbo 10.07.11 at 5:15 am

I re-responded to Ken before reading his second comment, above. So I’ll add something in response to that:

“If academics are giving different grades it means there are inconsistencies in the way they are interpreting the marking criteria, and the rubric then becomes the agenda for finding out where the differences are arising. This still involves a considerable amount of time and discussion, and there’s still plenty of scope for problems to arise, but at least it avoids long aimless debates about the purpose of the assessment task.”

I don’t agree that academics will only give different grades due to inconsistencies in understandings of marking criteria and/or learning outcomes. I think that’s a doubtful assumption. And I don’t agree that tightening up the marking criteria – i.e. settling on a further verbal formula that settles what the first verbal formula said – will even have a regular tendency to lessen the degree of inconsistency due to differences in understanding. You are, as likely as not, just infecting the fresh formula with whatever disagreement afflicted the first formula.

In general, I agree with what Zehou says about different rubrics and their discontents.

150

John Holbo 10.07.11 at 5:20 am

“It’s a bit disconcerting to read so many people trying to make a case from personal experience, as if the body if existing knowledge doesn’t exist or isn’t worth consulting.”

Be it noted, I’m not trying to make the case from personal experience. I’m making a more conceptual point about how a rubric – a verbal formula – will predictably be subject to the very sorts of problems it is being introduced to solve. (Of course, I think my two decades of grading experience is not exactly chopped liver either, but I can’t exactly present it in evidence.)

But if you want to post some citations showing that the thing I think is a newbie misconception is in fact the bleeding edge of science fact – well, I’ve been wrong before. And I’ll be wrong again. I hope I am amenable to being corrected even about the stuff I think I know quite well. (But now I have some papers to grade. Even I am not silly enough to neglect my grading for the sake of getting into righteous argument about how properly to grade. So I may not be back for a day or so.)

151

Meredith 10.07.11 at 5:54 am

What is the problem people are trying to solve here?
I’m not automatically hostile to all “outcomes assessment,” the way many people I know are, only because I worry that we liberal arts folks (including the physical scientists and mathematicians among us) can get sloppy in various ways for various reasons, including: look at this beautifully designed program full of wonderful courses, these fine young people, and this committed and talented ME and my colleagues! Isn’t this glorious? We academics can fall prey to business-flow-chart-think and packaging/”branding”-mentality and celebrity-think as much as anyone else. So, serious thinking about our goals and whether we’re actually approaching meeting them is, I think, really important.
But the business-mentality, or 12-step program, or if-it’s-got-a-number-attached-it-means-something-serious way of thinking, or whatever it is that so many outcomes assessments become (hey, here in Massachusetts good old Mitt Romney’s business-dominated education council — the acronym escapes me, and my husband, who would know, has wisely retired to bed — got the ball rolling on all this in our public colleges and university) — all this strikes me as, well, problematic.
Have we forgotten that our “problem” or “challenge” is not to create better student-widgets on the assembly-line or a better “bottom-line” of “student results” or a prettier picture to project on some god-knows-who’s-watching screen (death of god for the narcissist?)? Sorry, but I am teaching the Apology in Greek right now (intermediate level, with lots of emphasis on, hey, what’s the mood and tense of that verb? I can grade that!), and though I don’t think of myself at all as a Platonist or Socrates fan, I keep coming back to Socrates. In some important way, like him, I do not teach! I engage in dialogue with real and fragile and vital and strong people, called students. (Hello, Bakhtin.)

152

Zehou 10.07.11 at 6:57 am

Ken: “it sounds to me like the learning outcomes you want to assess are (1) depth of understanding of a text, and (2) the capacity to develop a coherent argument about it.”

It makes sense to assess the one paper for “depth of understanding of a text,” but not the other one, since the other one doesn’t aim at deep understanding of a text, but rather at refuting an interesting and important argument (largely) assumed to be the one in the text. So there’s no deep understanding on display, but something else useful and relevant to the course gets done. And you can’t assess the interpretive project for not offering and defending a verdict about the success of the argument, since offering and defending such a verdict was no part of the student’s project, though something else useful gets done.

Of course, you can just vow never to allow multiple approaches in this way, and thereby ensure that you can stick with one, tidy rubric for a writing assignment that has a lot of constraints. I’m not saying that is or isn’t the right way to go. But you were telling John that he wouldn’t have to give up allowing multiple approaches. I thought that sounded like telling him he can have his cake and eat it too. Hence the examples.

153

Myles 10.07.11 at 7:17 am

But I also think that its time well-spent in that students are getting, and appreciate that they are getting, as fair a shake as possible. Double-grading was an eye-opener the first time I went this route: even with an extensive “crib sheet” for the graders that laid out my expectations for an “A” answer on each question, an “A-” answer, etc, the percentage of essays that needed to be triple graded was much higher than I had anticipated (10-15%).

Thank you, Professor Weeden. I’ve had research papers graded by single TA’s before, and it was not a good experience. (I got an A, but the laziness of the professor in actually not doing grading at all was extremely unattractive.) I appreciate enormously professors who do take the extra effort to ensure fairness in their grading, because the grades issued do matter. My sincere kudos to your efforts.

154

Tony Lynch 10.07.11 at 8:06 am

Mark & be Damned.

Know it.

Do your best.

Forget yourself, not the subject.

Help others.

& Fail.

155

Ken Lovell 10.07.11 at 8:50 am

Zehou by referring to ‘something else useful and relevant to the course’ I think you disclose the conceptual differences I mentioned earlier. IMHO good teaching starts with learning outcomes and works backwards. There may be courses where the learning outcomes are malleable and subject to change as the teaching period progresses but I suggest they are a tiny minority and can be disregarded for the purposes of the general argument. Hard cases make bad law, as they say, and outliers can be found to create apparent problems for any general approach to assessment.

Assuming the course learning outcomes are known in advance and are capable of being reduced to practical assessment activities, the role of the course assessor is to design a series of assessment tasks that collectively test ALL the learning outcomes. Each assessment task should test one or more of the required learning outcomes, and by the end of the teaching period they should all have been assessed. If a student misses the point and does something that demonstrates they have met one of the other learning outcomes, which I assume is what you mean by ‘something else useful and relevant to the course’, that should not be taken into account in THIS assessment task. They will get the opportunity to demonstrate they have met that other learning outcome in another assessment task. This assessment task has been designed to see if they have met one specific learning outcome and following your example they clearly have not; therefore they fail.

If I understand your example correctly you are talking about a student who ‘doesn’t aim at deep understanding of a text'; but if deep understanding of the text is exactly what they need to meet the learning outcomes of the course then they should fail this assessment task. It’s not to the point that they made a great argument about something else – they can copy and paste it into the other assignment where it’s relevant, and get credit for it there. This assignment was trying to find out the extent of their understanding of the prescribed text, and on your example they failed to do it and should get graded accordingly.

It’s a bit like asking in an exam question ‘what is 2+4′, and getting the answer ‘2×4=8′. Certainly the student has demonstrated they know how to multiply, which may well be relevant to the course, but that isn’t what you wanted to know when you set the question. You want to know if they can add. On the evidence, they don’t even understand the concept, and accordingly they should get zero marks for the question. If you are a competent assessor you have asked other questions in the exam to see if students can multiply and this student will presumably do well there; but that is not relevant to the earlier question trying to establish if they can add.

156

Peter T 10.07.11 at 10:46 am

The word “objective” is getting a lot of exercise here. It seems mostly to be used to mean “impartial”, but the use allows the overtone of “invariantly true – regardless of observer” to creep in. I think it was the philosopher Alasdair MacIntyre who observed – following on from Aristotle – that all bodies of learning are crafts, with standards of procedure, correctness etc set by the current practitioners, and the aim of learning is to show someone how to be a practitioner – and thereby earn the right to participate in and influence the ongoing evolution of the standards.

This makes judgements on student progress less “objective” than “intersubjective”, and allows (indeed demands) a certain amount of variation in views on standards etc. These may be indicative not of inexcusable sloppiness but of a field that is intellectually alive.

157

John Holbo 10.07.11 at 2:55 pm

“IMHO good teaching starts with learning outcomes and works backwards. There may be courses where the learning outcomes are malleable and subject to change as the teaching period progresses but I suggest they are a tiny minority and can be disregarded for the purposes of the general argument. Hard cases make bad law, as they say, and outliers can be found to create apparent problems for any general approach to assessment.”

These cases really aren’t a tiny minority. They are more the norm than the exception, at least in the humanities: essay topics that are broad enough that they don’t have a narrowly prescribed right answer, relative to which you can backform a determinate path of correct approach, hence a rigorous rubric. It’s rare for me to assign an essay topic to which I am unprepared to accept different answers, even somewhat different styles of answer.

158

Zehou 10.07.11 at 3:19 pm

Ken: “by the end of the teaching period they [all of a course’s learning outcomes] should all have been assessed.”

I don’t think I denied this, nor does anything I said rule out endorsing it, right? We’re talking about a particular paper assignment, where we needn’t assume it is the only thing students are required to do in the course.

“IMHO good teaching starts with learning outcomes and works backwards. There may be courses where the learning outcomes are malleable and subject to change as the teaching period progresses but I suggest they are a tiny minority and can be disregarded for the purposes of the general argument. Hard cases make bad law.”

I don’t remember saying anything about the learning outcomes for the course being malleable. We were talking about a particular paper assignment, right?

In any case, what you say here is exactly what I was trying to get you to admit: it’s not ok to allow multiple approaches, after all, according to you, since that makes the instructor’s job of assessment too hard.

159

nick 10.07.11 at 4:52 pm

Rubrics, no doubt, have their place. For anything but the most basic essay assignment, they function to produce pseudo-objectivity; they may be necessary in a complex bureaucracy where ass-covering is systemic, but they are not actually pedagogically useful. And then there’s the problem of rubrics driving assignments: when the goal becomes to produce something that dovetails nicely with the language of assessment…..

It’s the duty of those teaching in the humanities and sciences, I think, to push back against voices like Ken Lovell’s. The production of data for bureaucratic purposes, however necessary it is, is neither teaching nor learning; those who promote assessment and talk of outcomes, however necessary they are, are neither scholars nor teachers.

160

bianca steele 10.07.11 at 4:54 pm

Meredith @ 151
Why is this numbers-based, objective approach associated with business more than, say, science and so-called “liberal hubris”? Or is it? (Among managers outside academia who’re subjected to something similar, it is frequently seen the opposite way, and it seems peculiar that something originating largely in universities themselves is rejected out of hand by academics as alien in that way.)

(As for the Delphic oracle thing, there’s a comment on a philosophy paper–an exam, though, not an essay–that I’m still wondering about the meaning of. Did it say, “i can’t fault this answer for what it is but you aren’t cut out for further work in philosophy,” or did it say, “interesting but could be better”? No I’m not hoping John will interpret it for me, just a data point.)

161

socratic_me 10.07.11 at 6:03 pm

It makes sense to assess the one paper for “depth of understanding of a text,” but not the other one, since the other one doesn’t aim at deep understanding of a text, but rather at refuting an interesting and important argument (largely) assumed to be the one in the text.

I am not actually sure how one refutes an argument in the text without demonstrating a fair depth of understanding. I would say that is a niggly point, but your refutation seems to depend on it. Primarily, you seem to be taking an extremely narrow parsing of how a rubric could be used and/or what “depth of understanding” means and then creating overly narrow and restrictive rubrics from that narrow parsing.

Simply put, it seems that the ability to understand and critically evaluate X’s argument put forth in piece Y, that should be addressed by all three papers, along with fun objectives like “create coherent arguments” (because a negation is most definitely still an argument, and a comparison to an earlier work should certainly contain an argument as well, unless it is a purely historical work, in which case it really may not be a good paper topic, depending on the class). That doesn’t require a ton of varied rubrics so much as a discussion as to what you expect students to accept in their papers. And if you wanted to check their ability to do historical analysis of philosopher’s works in addition to critical analysis, make that an objective and note that you expect at least one paper of each type. You will need two rubrics, but that is because you intend for them to do both things.

Or actually in short, what Ken said the first time.

162

Meredith 10.07.11 at 6:22 pm

bianca steele, Of course I have no problem with numerical or statistical measurements when they actually are measuring something, and something of use to be measured, and when they are interpreted intelligently (e.g., in concert with other information and with due caution about the reliability of the measuring instruments themselves). But not everything can be evaluated/understood well, or if well then sufficiently or efficiently, through numerical measures. Seems pretty obvious to me, and not just when I’m reading students’ papers. I spent a good deal of the last four-six weeks reporting on assistant professors’ teaching, basing my analyses on the results of student course surveys (lots of numbers, percentage groups — with huge standard mean deviations, btw, even in large courses — and several poorly worded questions), reports on class visits made by tenured professors, interviews we conducted with students, and discussions about all these things with my tenured colleagues. I found all these instruments of evaluation useful, when put into conversation with one another. In writing the final reports, though, I had to keep in mind that the faculty committee reading them would look at the student course evaluations (some nice graphs that make the “bottom line” on key questions easy to take in at a glance) before even reading my reports and might already have latched on to the “numbers” in those surveys as somehow “the real story.” Not because these faculty don’t trust my judgment or my departmental colleagues’, but because of a naive reflex, to which most of us are prone, to accept numbers or graphs as somehow being “objective” and more accurate or useful than other modes of analysis.
My guess, for what it’s worth, is that business probably uses numbers to measure things like sales and labor costs well, but that they may waste a lot of time, energy and money devising various schemes for improving sales or office efficiency or whatever through their misuse of numbers and statistics. I’m pretty sure that the decisions made by successful upper management on big questions involve many judgments that aren’t driven directly by numbers but by shrewd anaylsis of many factors and by, well, vision. Steve Jobs went to Reed, right?

163

Meredith 10.07.11 at 6:35 pm

Oh, and I learned yesterday from a friend who went to Reed that Steve Jobs audited a course from a calligraphy teacher who was legendary at Reed. (Hence Apple’s long interest in good font design.) I wonder how that calligraphy teacher graded his students’ work?

164

bianca steele 10.07.11 at 7:09 pm

Meredith, I had in mind, actually, not sales and accounting, but areas that are more qualitative or based in the physical world (especially engineering and manufacturing): new numerical measures and rules for monitoring performance and process, usually imposed from the outside by paid consultants (either because management wants to use a new technique, or because government and industry bodies impose those measures) who are trained to get companies up to speed with the requirements of what is unfortunately often just the next new thing, or sometimes from within large organizations by internal specialists.

For example, there is a good body of evidence that in developing new software, the number of bugs found tracks a curve of a certain shape. There is also a good body of evidence that the specifics of the curve vary greatly. The average of one important number, across every project ever studied, ever, in the history of computers, which I’ll call B, is IIRC 8. It makes perfect sense to track the value of B for the group, and to see where they are on the curve; it is less than helpful to mandate they will hit B=8. It makes even less sense to miscalculate B because a manager believes to calculate the number differently will make people work harder.

165

Meredith 10.07.11 at 7:51 pm

bianca steele, I think we’re on the same page — we’re thinking of the misuse or faulty application of numerical/statistical/technological tools. I’m not sure where I’d expect that to happen more. In business or manufacturing or engineering, the costs and the consequences (outcomes) are usually more immediate and clearer than in academics, you would think, so that people working in those areas should be motivated to get things right. But then, you would expect academics to get these things right because they’re the people teaching these skills to future business people, engineers…. In the end, I suspect, our romance with quantification (even sloppy quantification — love is blind) has very deep cultural roots.

166

John Holbo 10.08.11 at 3:30 am

“And if you wanted to check their ability to do historical analysis of philosopher’s works in addition to critical analysis, make that an objective and note that you expect at least one paper of each type.”

The problem is that historical analysis is not self-evidently distinct from critical analysis. Suppose I pose a vanilla topic like ‘critically assess the arguments of Descartes’ First Meditation’. Now I can additionally stipulate things like: and don’t gimme no history! This is not in fact a totally crazy thing to do: to stipulate that you want a non-historicist-style paper. But it is also fairly normal to leave it up to the students to approach a topic like ‘critically assess’ which might take them in an historicist direction or not. Given that my lectures on D. will be a combination of historicism and argumentative analysis, their papers might, likewise, be hybrid productions.

And that’s just the first layer of problems. Now you need additional stipulations. Suppose the student’s first sentence reads: ‘In this paper I’m going to focus exclusively on the dream argument. It has problems. When we see that, we see what goes wrong with all the arguments in the 1st Med.’ Now if I’ve got a rubric and little checkboxes for all the arguments in the first med, this paper is going to fail, because it doesn’t check all the boxes. But it might be a reasonable intellectual production. And now it gets difficult: it isn’t unreasonable to try to see the whole Meditation through the lens of the dream argument. All the same, there are interesting aspects of the first meditation that, perhaps, can’t be seen through that lens (this is something reasonable people can disagree about, I think – it’s a function of what you find intellectually interesting). To put it another way, we have a trade off of intellectual economies. You can see a lot of the First Med through the dream argument, so this reductive view has real virtues. It’s simple, clear … but maybe it’s not really complete. So a completist can fault it. Better to have a complex, more nuanced, less clear but more complete picture. Apples and oranges.

So now one grader – sticking with the rubric – gives a C. (On the ground that the paper patently failed to address the topic.) And another grader – sticking with the rubric – gives an A. (On the ground that the student has examined all the arguments through the lens of dream argument.) And another grader – sticking with the rubric – gives a B (on the ground that the student came close, and managed to see quite a bit through this lens, but in the end missed some things she was obliged to cover.)

And everyone was following the rubric to a T. They just disagreed about what counts as ‘to a T’. And thus we see the glories … not of garbage in/garbage out (that will have SoV on my tail again, accusing me of confessing to total negligence) … but of inconsistency in/inconsistency out. A phrase like ‘critically assess the arguments of x’ is much too malleable to be rubric-able in ways that will eliminate potential inconsistencies in grader judgments. And again, I’m not saying that grader judgments will be an inconsistent riot. Just that I predict that the rubric will not reduce inconsistency. It may actually increase it. How so? This paper I am imagining might get a C, due to failure to check off some basic boxes. Or it might be deemed to check off those boxes and get an A. Big inconsistency! What does it stem from. A strict vs. a narrow application of a bureaucratic rule. If we hadn’t been working from a rubric, this particular source of inconsistency would never have arisen.

You can try to be more specific in writing your topics, can try to nail it all down. Maybe tell everyone ‘don’t try to see everything through the lens of the dream argument, just mechanically proceed through the Meditation, summarizing one argument after another. Don’t get fancy.’ That’s not a crazy option. But it does mean you are greatly restricting the sorts of assignments you can assign, the sorts of writing you can allow. You always have to have a ‘don’t get fancy, trying to make some bold new approach to the material’ clause and that means, in effect, mandating mediocrity along one axis, for the sake of assessment standardization along others. Most of the types of paper topics that are currently standard fare in the humanities – which are typically hopeful invitations to ‘original thinking’ – would be off the table. I say it a third time: that’s not crazy. Typically papers aren’t that original. Maybe it would be better for paper writing to be a highly restricted, drill-like activity, on the grounds that this would inculcate basic writing skills. But I have my doubts.

167

John Holbo 10.08.11 at 3:49 am

Also, just for the record: I still regard these issues about assessment and consistency and rubrics as substantially separate from the issue of how to comment on papers. So what I just said neither follows from, nor is an argument for, what I wrote in the post, except in the following sense: the reason I think it’s hard to do 1) – per the post – so you should do 2), is closely related to my skepticism about the utility of rubric-style box-checking. Which I have now articulated at greater length. The same perils that make for inconsistency-in/inconsistency-out problems, for assessors, make for inability-in/inability-out problems, for students. Students start with an inability to recognize bad organization, in their own writing, so merely sticking a low number on it does not cause them to understand why they got the mark they got, or help them do better next time.

168

Ken Lovell 10.08.11 at 6:31 am

John I’m not sure if you are deliberately missing the point or if I am expressing myself badly. You do seem to be responding to individual points out of context instead of trying to understand my whole argument.

Of course I don’t try to write marking criteria for an assessment task like an essay that assumes there is ‘a narrowly prescribed right answer’. That would utterly frustrate the whole aim of encouraging students to engage critically with the course content. The rubric would cover firstly depth of understanding of the relevant course content, and secondly the ability to develop a coherent argument that responds to whatever question you asked. I am not interested in whether they give the ‘correct answer’ (because there isn’t one) but in whether they are able to use evidence and logic to make a persuasive argument. That’s what I incorporate in the rubric.

In a math assignment you DO want the ‘correct’ answers, but the principle is the same – first you work out the purpose of the assignment and then you work out what evidence you will rely on to evaluate whether students have achieved the objective/s. You seem to be assuming rubrics have to be inflexible things that have a checklist of acceptable answers; that’s simply not correct. If you want to know whether students know legal principles, you use a quantitative correct/incorrect approach; if you want to know whether students can write creative verse you use a qualitative approach. Either way you are forced to explain exactly what purpose the assignment is serving and what evidence you are going to take into account in awarding a mark, which is (1) good discipline for the academic; (2) valuable guidance for students, and (3) ethically highly desirable to promote fairness and transparency in assessment.

There seems to be an assumption in a lot of the comments that rubrics are some kind of numerical or quantitative tool. There is nothing inherently quantitative about them, nor do I use them that way. They are flexible tools, which have plenty of difficulties associated with them, but if used properly they should produce superior results to alternatives such as norm-referenced assessment. This is really so well-accepted now in teaching and learning scholarship that I’m surprised to see them so vigorously attacked here. Especially when, if used effectively, they free up some time to give the kind of constructive advice to students that everyone from the OP on agrees is an important part of learning.

169

John Holbo 10.08.11 at 6:49 am

“You do seem to be responding to individual points out of context instead of trying to understand my whole argument.”

You haven’t made an argument, Ken. You have assured me that there is an argument available in some unspecified scholarship. That’s as may be. Nothing wrong with making claims about the existence of an argument in a comment box. But I can’t respond to the argument until you have at least sketched it, or cited it more specifically.

“Of course I don’t try to write marking criteria for an assessment task like an essay that assumes there is ‘a narrowly prescribed right answer’. That would utterly frustrate the whole aim of encouraging students to engage critically with the course content.”

Your example of ‘what is 2 + 2?’ was rather ill-chosen, then. And this is a significant problem for your position because, in emphasizing the virtues of the rubric approach, you need to emphasize its virtues in dealing with ‘2 + 2′ type cases, i.e. cases in which it is quite obvious that answering a different question gets a fail. If you are not thinking about this kind of case at all. If, instead, you are thinking about cases like ‘critically assess x’, then it is not clear to me what response you have to my, and zehou’s, objections. We’ve made our arguments. What’s yours?

“They are flexible tools, which have plenty of difficulties associated with them, but if used properly they should produce superior results to alternatives such as norm-referenced assessment. This is really so well-accepted now in teaching and learning scholarship that I’m surprised to see them so vigorously attacked here.”

How is their flexibility consist with the sort of rigidity that is seemingly necessary to eliminate inconsistencies between graders? If I tell grader A and B that they must not merely follow some general norms but follow a specific rubric – which I then add is ‘flexible’ – then how do I know we aren’t back where we started, in terms of getting A and B on the same page? Why are flexible rubrics superior to norms? Why should I believe that they are even anything different than general norms by other names?

I realize this sounds dismissive, but I’m just very skeptical. You are telling me that there is some reason that my skepticism is unwarranted. Indeed, there is a ‘generally accepted’ reason. Well, I didn’t get the memo. So lay it on me.

170

Ken Lovell 10.08.11 at 7:57 am

John you are clearly now aggressively supporting a position, rather than engaging in a discussion; your prerogative of course (your blog etc) but also the reason I rarely bother to comment on blogs these days. Combativeness for its own sake gets tiresome after a while.

To anyone who would like to explore the issues in greater depth than can be done in a comments thread I commend Jonsson, A & Svingby, G (2007), ‘The use of scoring rubrics: Reliability, validity and educational consequences’, Educational Research Review, 2, 2, pp 130-144. Those who have time might also like to read Stevens, D & Levi, A (2005), ‘Introduction to rubrics’, with the caveat that it is an unapologetic argument in favour of rubrics and needs to be balanced by consulting other literature, of which there is an ample supply (contrary to what one might think from reading this thread, assessment in higher education is not exactly a virgin field of research). Rubrics, or assessment based on criteria and standards to use a more cumbersome but informative term, are not a cure for all the problems ailing assessment in higher education, but an approach that when properly implemented overcomes some of the deficiencies of more traditional, holistic grading systems.

171

John Holbo 10.08.11 at 8:53 am

Well, thanks for the citations. I am, I know, quite the prickly one. But you have to appreciate that if you march into a thread and tell people they are wrong because of some well-known, unspecified results somewhere – and if you furthermore hint that taking the other side is symptomatic of a bad impulse to cling to unaccountable power – you are going to get challenged. What’s the argument? What’s the evidence? You act as though my failure to accept your claims must be symptomatic of a disbelief in the existence of published literature. But that’s not really a reasonable assumption about why I’m skeptical. (What academic could ever be skeptical about the existence of a mound of published literature somewhere?) I’m skeptical because you’ve said something that, honestly, seems wrong to me, and I’ve tried to articulate, substantively, what my reasons for doubt are. Your responses seem to me to have been quite non-responsive on that front. (No doubt I could have been less snarky. But glass houses and first stones and all.) But again, thanks for the cites. I’ll give a look.

172

John Holbo 10.08.11 at 10:07 am

Actually, it occurs to me there way be a way to salvage the discussion – or at least for me to articulate my point in an alternate way. What I am looking for is a way to distinguish assignment specific rubrics from general norms. Ken writes: “Of course I don’t try to write marking criteria for an assessment task like an essay that assumes there is ‘a narrowly prescribed right answer’. That would utterly frustrate the whole aim of encouraging students to engage critically with the course content. The rubric would cover firstly depth of understanding of the relevant course content, and secondly the ability to develop a coherent argument that responds to whatever question you asked.” Nevertheless, despite this admirable flexibility, rubrics are supposed to be considerably stiffer-spined than traditional, more holistic, or informal norm-based grading systems.

There is an ambiguity here: if the rubric concerns just what it means to have deep understanding, and what it means to develop a coherent argument, then the rubric will be the same whether I am asking for a critical assessment of the First Meditation or Kant’s First Critque. If this is the way of it, then I fail to see that there would be much profit in the graders putting their heads together, retail, assignment by assignment. We are just talking about very general wholesale disciplinary norms – in philosophy or whatever discipline we are in. General ideas about intellectual quality and argumentative standards. In short, the rubrics turn out just to be holistic norms, in bureaucratic get-up.

But if we are actually settling a rubric for The First Meditation that will apply only to the critical assessment of it, not to the critical assessment of the First Critique, then we are back with the problem that you have to specify the ‘right’ answer, and then backform a ‘correct’ path to it/justification of it. After all, we don’t think answers about Descartes should be different than answers about Kant because the standards for ‘deepness’ or ‘good argument’ differ, from Descartes to Kant. Rather, Descartes and Kant say different things, so the answers as to what they say should differ. So the Descartes rubric should encode standards of accuracy about what Descartes says. More briefly: the rubric is the correct answer as to what Descartes says. (Just as 4 is the correct answer as to what 2 + 2 might be.) But now we get the problem sketched above. Namely, different answers are possible, different approaches are possible, different styles of approach, and it is hard to see how a ‘right answer’ rubric can be flexible enough to accommodate that pluralism unless it effectively falls back – once again – on being effectively only a general articulation of norms of ‘being critical’ and ‘good argument’ that would apply, equally, to a Kant paper as a Descartes paper. Good old holistic grading. (After all, it is because of these norms that different answers can count as defensible. So, in admitting this pluralism, we are falling back on the norms.)

So either way, it seems to me, we end up seeing how rubrics either turn out to be holistic grading in disguise, or else to be something genuinely different, that would restrict the sorts of assignments one could make, and that will carry risks of bureaucratic deformation – hence increased not decreased inconsistency – even if we narrow ourselves in that way.

Pardon my blathering on. But, having blathered this far …

173

Zehou 10.08.11 at 12:06 pm

socratic_me: “I am not actually sure how one refutes an argument in the text without demonstrating. . . .”

Here’s how: you lay out the standard interpretation, show how there is at least moderate textual support for it and show that it isn’t uncharitable in any immediately obvious way, explain how the argument is interesting, worth considering, and popular whether the interpretation of the text is correct or not, and then you go about critically assessing the argument. Is it deductively valid? How plausible are the premises? And so on. That is, as I made explicit in the bit you quoted, you address an argument “(largely) assumed to be the one in the text.” Trying to figure out what is the most plausible and charitable interpretation of the text is a quite different thing, and doing that might involve being (largely) agnostic about whether the argument is valid, sound, interesting, popular, etc.

“because a negation is most definitely still an argument, and a comparison to an earlier work should certainly contain an argument as well.”

Yes, that’s part of the point: these are all different types of argumentative essays, so of course they all involve argumentation, but the skills involved in developing the arguments are quite different in each case, and one student might be skilled at writing one paper but not the other.

“you seem to be taking an extremely narrow parsing of how a rubric could be used and/or what “depth of understanding” means and then creating overly narrow and restrictive rubrics from that narrow parsing.”

What gives a rubric real potential as an aid to objectivity (as opposed to the pseudo-objectivity decried above) is precision. This seems to be how these conservations go: The experts advocate rubrics as convenient, efficient, and cost-free tools that maximize objectivity in grading. If you suggest respects in which they are not objective, the reply is immediate: your rubric criteria are too broad. If you suggest that introducing the precision threatens efficiency, the reply is immediate: change your course so you can keep your rubric.

Rubrics are great, but there are costs to using them, and it isn’t obvious that the costs always outweigh the benefits, especially when we care them to other aids to objectivity such as the ones John and others have mentioned.

174

Roa 10.08.11 at 12:09 pm

Two things:

1. CT should change the coding so that post numbers never change as posts are injected further up after moderation. Let the post moderation posts get unique, suffixed numbers (“#54a” instead of taking “#54″ from some other post. Isn’t there some technical way to do that?

2. abstract argumentation might not be very effective on this topic. How about John H provides a sample question and sample paper and let Scent of Violet grade it and time that grading. The rest of us can the grade THAT grading performance.

175

Meredith 10.08.11 at 3:21 pm

In today’s NYT (http://www.nytimes.com/2011/10/08/world/us-envoy-peter-van-buren-takes-caustic-pen-to-iraq-war.html?_r=1&ref=world), Steven Lee Myers on Peter Van Buren’s book We Meant well, on the US “reconstruction” in Iraq: “The day-to-day reconstruction projects, he argues, were done as much to satisfy the bureaucratic need to demonstrate measurable progress as actually to make measurable progress.”

176

bianca steele 10.08.11 at 8:09 pm

Meredith, I meant both numeric and qualitative evaluations, which are imposed by the same sorts of consultants, trained in the same way.

Also, Peter W. @ 156: I think it was the philosopher Alasdair MacIntyre who observed – following on from Aristotle – that all bodies of learning are crafts, with standards of procedure, correctness etc set by the current practitioners, and the aim of learning is to show someone how to be a practitioner – and thereby earn the right to participate in and influence the ongoing evolution of the standards.

I thought he was saying this but on last reading I couldn’t find it exactly, so either I misremembered the place or misread it. He talks about practice in some places, craft in other places (e.g. bridle-making), traditions of dialectic in others.

177

Ken Lovell 10.09.11 at 12:26 am

Heh … ‘March into a thread’ John? What is the approved practice on your threads – sidle in with a shy smile? Request permission to speak? Serve a probationary period to demonstrate the commenter is One of Us?

I’m making this final comment purely for the purposes of clarification. I wrote at #168 that ‘The rubric would cover firstly depth of understanding of the relevant course content, and secondly the ability to develop a coherent argument that responds to whatever question you asked.’ If you believe this is correctly paraphrased as ‘the rubric concerns just what it means to have deep understanding, and what it means to develop a coherent argument’, then it explains why we have been talking past each other. But I suggest on any reasonable reading of what I wrote, your version is a complete misrepresentation.

178

Salient 10.09.11 at 12:59 am

What is the approved practice on your threads – sidle in with a shy smile?

Normally new entrants are required to bust a move unprecedented in its scope and grace, Ken, but you’re welcome to an exception: you really need only provide a link to a sample rubric of your devising, or a sample rubric that receives your stamp of approval.

179

John Holbo 10.09.11 at 3:03 am

Ken, the manner of your entry was fine. Perhaps ‘march’ wasn’t quite the word for it, but marching is fine, so that wasn’t the source of the difficulty in any case. I was just explaining why people didn’t readily believe what you said, the way you had evidently hoped. Again, it wasn’t the marching, it was the lack of argument/evidence.

Let me try to state my point once again, more clearly, in case you really haven’t understood my concern. The rubric cannot just describe the general aspirations of the liberal arts education: we want students to acquire the capacity to read texts with depth of understanding, make coherent arguments, so forth. This is the desired learning outcome of most assignments in the humanities and it won’t change much, if at all, from a Descartes paper to a Kant paper. I don’t think that a rubric that aims at goals that exhibit this generality will really do much to constrain graders in the requisite way. Everyone will just pour into this nominally assignment-specific matrix their holistic dreams for the liberal arts. It will be a holistic norm in light bureaucratic garb. (Not the worst thing in the world, but not much use for avoiding the troubles of holistic norms.)

Or the rubric will really be something more specific. But then how will it avoid the problems that I and Zehou have pointed out? namely, it will say that the right answer regarding Descartes is X? But honestly we are prepared to accept X, or Y, or Z … so now the rubric becomes this unwieldy, ever-more-disjunctive thing. And that, again, is no way to ensure consistency. Not to mention everyone is obliged to spend a lot of effort on the fine and delicate topiary art of snipping all these disjunctive branchings with an eye for keeping the whole tree looking … like it holds the shape of the ideals of the liberal arts as a whole: depth of understanding, coherent argument, so forth. So what have we gained for our efforts, except to get back where we started?

This is a very practical concern and you have brushed it aside, first, as something that can’t be dealt with, but that is an unusual problem (bad cases make bad law); second, as something that can be dealt with in a sufficiently obvious way that you trust it is obvious that I, and Zehou, are just misunderstanding how things go. You act as though my skepticism about what you say must be due to my not believing people have written about this, or my ire at your comment style. Why not consider that my skepticism is due to the fact that I think I have outlined a systematic problem with the solution you propose. Either there is a fairly easy way to deal with this problem, systematically, or there is not. If there is, what is it? If there is not, well rubrics can still be useful but we need more truth in advertising about how they work and what they are good for. Your advertising was misleading.

180

John Holbo 10.09.11 at 4:22 am

“How about John H provides a sample question and sample paper and let Scent of Violet grade it and time that grading. The rest of us can the grade THAT grading performance.”

This isn’t going to happen but I think just thinking about how it might is instructive. One sample paper won’t do because the issue concerns how you handle a stack of, say, 45 papers consistently and fairly. (I’m not going to write 45 sample papers, and SoV isn’t going to grade them.) But suppose we took – oh, say, the most recent 45 substantial CT posts that contain substantive arguments of one sort or another. I’ll bet that SoV could 1) write substantial qualitative comments on each of them – good argument! unclear! 2) identify which 10% he thought were the best (or the least terrible, as the case may be). But it would be much harder to 3) write substantive qualitative comments on all 45 that do double-duty as 45 stand-alone mini-proofs of each post’s either being in the top 10% of CT posts, quality-wise, or not. It’s when you reflect on the relative easy of doing just 1) and 2) without attempting 3) that you see the deep and humanistic wisdom of my post!

181

Neville Morley 10.09.11 at 10:36 am

A further thought on rubrics that perhaps helps bring the discussion back to the original question of how best to write effective comments – or at any rate helps me to clarify why I do find this an issue worth worrying about.

Rubrics of the general and holistic sort that are appropriate to discursive essays and analyses in the humanities and various areas of the social sciences almost invariably suggest a wide range of different things that might contribute to the overall assessment and so, by implication, ought to commented upon. I don’t remotely have time to comment in any detail on every single aspect, or even every relevant aspect – and I would also suspect (I’ve never risked trying it) that detailing every respect in which an essay failed to meet the published criteria adequately would be entirely useless as feedback, as I can’t imagine any student being receptive to such a barrage of criticism – tears at best, violence at worst, one might fear.

So, I do feel a constant tension, even leaving aside the question of how much time I can devote to the task, between my sense of what and how much feedback will be most useful and effective (which also runs into the question discussed originally of what purpose the comments are intended to serve – explain the mark, point the way forward etc.) and the implication of the institution of assessment rubrics that I should in fact be commenting on everything.

My department attempts to negotiate this with a standard feedback sheet, where I have to rate each of five general attributes (Information and Use of Evidence, Structure of Argument etc.) between ‘Perfect’ and ‘Oh Dear God’, or words to that effect, with or without additional comments, and then identify ‘Best Features of Essay’ and ‘Areas for Improvement’, in addition to written comments on the work itself, where I can then follow the same sort of approach that John describes above of focusing on what I consider to be the most important and useful points. In other words, the student gets some sort of feedback on all the major things, but not necessarily qualitative comments. How far this actually works is another matter; my sense, from talking to students who then turn up to discuss their work, is that they focus entirely on the mark sheet and don’t even read my comments half the time, so perhaps I should save myself the trouble…

182

Jonathan Mayhew 10.09.11 at 4:16 pm

Holism is itself a defensible goal, and one that rubrics can undercut. In other words, forcing me to separate aspects of a paper that are inseparable, or to draw arbitrary lines between categories like organization and argumentation, or clarity of argumentation and clarity of style. Holism means that I can lay out some general categories, like intellectual rigor, tight organization, and marvelous writing, and then offer extensive comments on all of those. I can give or take away credit for any of these aspects in any proportion. This is fair as long as (1) students know what my categories are and (2) I don’t give a worse paper a better grade than a better one. My “learning outcome” is that the student become an intellectual.

A rubric flexible enough to be holistic is no longer a rubric, right? Once you introduce flexibility back in then you also increase the possibility of variability of results. Are rubrics a deliberate assault on the ideal of holism?

183

ScentOfViolets 10.09.11 at 10:34 pm

Meredith, I meant both numeric and qualitative evaluations, which are imposed by the same sorts of consultants, trained in the same way.

Yet Another Problem, one which I cover in my first or second class period in stats: the confusion of ordinal data with interval or ratio data.

As much as possible, I try to stick with the old 90/80/70 rule for A/B/C grades. So in my classes, I can say that someone getting an A is doing very roughly 15% better than someone getting a B.

Not so in other classes! I remember taking freshman comp many many moons ago; while you didn’t have to do that much better to get a B as opposed to a C, going from a B to an A was much, much harder. Even as an 18-year-old, I could compare my scrubby B paper to someone’s A paper and admit “That’s a lot better than mine.”

I’m guessing that us guys in the harder discipline tend to treat grades more like a direct translation from some percentage scale, those in the humanities much less so.

184

ScentOfViolets 10.09.11 at 11:03 pm

This is a very practical concern and you have brushed it aside, first, as something that can’t be dealt with, but that is an unusual problem (bad cases make bad law); second, as something that can be dealt with in a sufficiently obvious way that you trust it is obvious that I, and Zehou, are just misunderstanding how things go. You act as though my skepticism about what you say must be due to my not believing people have written about this, or my ire at your comment style. Why not consider that my skepticism is due to the fact that I think I have outlined a systematic problem with the solution you propose.

The problem here John (and I know that you know this) is – surprise! – your cavalier treatment of burden of proof requirements. Don’t you think you have some obligation to try to convince people that your way is the correct way? As opposed to the tired old “Prove to my satisfaction that what I say is wrong”? You know, since you’ve abandoned the original premise, which seemed to be soliciting some constructive suggestions?

You can be as prickly or rebarbative as you like with whatever you deem the requisite justification for that stance. But if you go that route – with not very much justification that I can see – other people are then free to hold you to a strict and strictly rational accountability. you haven’t met that standard yet.

185

John Holbo 10.09.11 at 11:36 pm

“The problem here John (and I know that you know this) is – surprise! – your cavalier treatment of burden of proof requirements. Don’t you think you have some obligation to try to convince people that your way is the correct way?”

How can I convince people that my way is better than some unspecified rubric-style x approach before I know what rubric-style x is? It seems to me that it’s actually reasonably constructive to say: here’s a dilemma that we face in designing our rubric. Either we’re just doing the old thing, in disguise, or we’re doing a new thing that is predictably more rigid than we would like in certain ways. Even if we can’t solve it, it’s helpful at least to see what the structural difficulties are. I don’t think you could really design a rubric, competently, without being aware of the practical bite of the problems I and Zehou have been sketching out. In any case, I don’t really think it’s fair to fault me for failing to prove a very difficult negative: perhaps there is some rubric-style I haven’t thought of that overcomes these difficulties. Obviously yes.

186

Salient 10.09.11 at 11:41 pm

Don’t you think you have some obligation to try to convince people that your way is the correct way?

This? Again? Please, spare us. By this point, we are all intimately familiar with the standards you would like to hold us to; if we’re not meeting them, just assume that we never will, and go on with your life. Here it’s especially egregious to press us — the persons saying “this is uncannily tricky and difficult and possibly intractable” have quite a hell of a lot less to ‘prove’ or ‘convince’ interlocutors of, than the persons who assert something to the effect of “WTF is y’alls problem, not only am I super awesome at this, but also I think super-awesomeness is so easy to achieve that I scornfully call into question your professional competence relative to mine, just because you have asserted that this is really hard and possibly intractable”

…cue you demanding that I prove to your satisfaction that you’ve done this, of course. The only open question is whether you’ll type a wink smiley or just write “Sigh.” (I do like when you don’t even encase the sigh in a parenthetical, as if it’s something you’re exhaling with declarative exasperatory force rather than wincingly releasing under your breath. It’s honest.)

But whatever, as with pretty much any open thread in which educators try to hash out the trickier and tougher aspects of the trade, it only takes a few people popping in to let everybody know how easy it all is for the competent souls of the world (to the point of demanding that we convince you it’s hard!) to make the other commenters defensive and testy and derail whatever hope of productive contemplation existed. The requisite humility is unsustainable in the face of scorn.

187

ScentOfViolets 10.10.11 at 12:37 am

I don’t think you could really design a rubric, competently, without being aware of the practical bite of the problems I and Zehou have been sketching out. In any case, I don’t really think it’s fair to fault me for failing to prove a very difficult negative: perhaps there is some rubric-style I haven’t thought of that overcomes these difficulties. Obviously yes.

Pish-Tosh; throwing out “you can’t prove a negative” to quell any demands that you back up what you say just won’t cut it.

One more time: other people in other fields use rubrics and associated paraphernalia with some degree of success. So why can’t you?

Don’t just baldly say you can’t. Iow, show, don’t tell. Surely you as a teacher recognize this basic dictum, yes?

So far we’ve gotten stuff like you could grade with a modicum of transparency and with some good feedback, but that it just isn’t worth your time. We’ve also gotten that you could restrict an assignment – which would have the salutary effect of making it easier to design a rubric – but you for whatever reason just don’t want to restrict your assignments in that fashion. Iow, for whatever reason, transparency in grading and providing good feedback is rather far down on your list of priorities. Now, for all I know, you’ve actually got a good reason for doing things this way. But you haven’t exactly been very forthcoming on why. And you certainly haven’t justified it.

Look. I know that grading is boring. A necessary evil that I would just as soon fob off on somebody else if I could. But the fact of the matter is that I can’t. It appears that you can’t either but that you’ve gone in a somewhat different direction than the one I went in adapting to these circumstances.

188

ScentOfViolets 10.10.11 at 12:40 am

Don’t you think you have some obligation to try to convince people that your way is the correct way?

This? Again? Please, spare us. By this point, we are all intimately familiar with the standards you would like to hold us to; if we’re not meeting them, just assume that we never will, and go on with your life.

The problem here, Salient, is your purblind insistence that this is somehow “my” particular and idiosyncratic standard.

It’s not, not by a long shot. It is in fact SOP for people in the hard sciences.

I’ll stick with that standard, if you don’t mind. If you do, well, no one is forcing you to participate in this discussion.

189

Salient 10.10.11 at 2:43 am

Nobody’s forcing you to be so obnoxious, either. Nor is anyone forcing you to be so persistent–since you arrived on the scene with comment #81, you’ve posted 20 of the subsequent 82 non-Holbo comments (plus or minus a couple as I’m unmotivated to take a precise count). You’re suffocating us.

And no, acting the ass and then demanding that people prove to your satisfaction their claim that you’re being obnoxious when they complain of your obtuseness is not “SOP for people in the hard sciences” –thank. the. gods. for. that.

190

ScentOfViolets 10.10.11 at 2:46 am

Nobody’s forcing you to be so obnoxious, either.

. . . either. Thank you.

191

MPAVictoria 10.10.11 at 2:52 am

“As it happens, I teach math…”
You don’t say?

/I kid ScentofViolets, I kid.

192

ScentOfViolets 10.10.11 at 3:19 am

Yes, MPA, we know your feelings are paramount, whatever the actual facts are . . .

/I kid Victoria, I kid.

193

MPAVictoria 10.10.11 at 3:29 am

“whatever the actual facts are.”
So you are not a math teacher? You are some sort of impostor? That is just sad man.
/Also who is Victoria?

194

ScentOfViolets 10.10.11 at 3:35 am

You’re losing it . . . but in any event, do you actually have anything of substance to contribute to this thread? You know, about how to grade papers as opposed to drive-by snark?

/Not kidding.

195

MPAVictoria 10.10.11 at 3:43 am

“You’re losing it . . . but in any event, do you actually have anything of substance to contribute to this thread? You know, about how to grade papers as opposed to drive-by snark?

/Not kidding.”

Since you asked I used to TA for professors while doing my graduate work. I found marking papers consistently extremely difficult even with extensive examples and descriptions of what the professor was looking for. I have nothing but sympathy for people who have to do that sort of grading all the time.

196

Meredith 10.10.11 at 4:38 am

I’m not sure how to think about comments on just one paper by a student. The first paper I read by a student, even if an ungraded response paper of some kind, is always exciting and fraught for me. Who is this person I am about to meet? (There is something very intimate about a paper, even if you’ve already come to know its writer in, say, a small discussion class. Especially undergraduate papers — god, do they reveal more than they realize, so much of the time.) Btw, because I want to get to know someone a bit through this intimate exchange of paper+comments (which is different from the intimacy of classroom or conference exchanges — far more intimate, really), even in a seminar with one major paper near course’s end, I like to assign shorter written assignments of some kind earlier.

John’s post and comments, along with many others, have gotten me thinking about that: the importance of reading more than one paper by a particular student before being able to grapple thoughtfully with his or her writing/thinking. (Which may raise problems for the understandable — laudable — techniques described way above for coordinating paper-reading and paper-grading among multiple TA’s in a very large, university course.) One paper may not tell me enough….

Something that hasn’t been mentioned about commenting (I think it hasn’t): engaging the substance of the student’s ideas in some way. Whatever else we may have to say about clarity or organization or accuracy or argumentation or all such vital matters, we can always pick up on some observation or insight the student has had and just respond to it in some longish scribble in the margins or in the final comments — a response that takes them seriously as thinkers, as people. Something that affirms the value of their grappling and shows our interest — shows them that their engagement with a text or an issue has excited our imaginations. I don’t mean the “good” or “excellent” in the margins, which may indicate nothing more than “you have met some standard I may never really have made clear to you!” (though it’s important to sprinkle those in, where deserved) but rather, “I’m learning from you” or “my imagination is being excited by you,” the way we learn and are excited in exchanges with colleagues and friends. Such comments may take the form of expanding on the student’s idea in a learned but enthusiastic way, pointing to a related idea, posing a counter-argument…. Many forms. (Time-consuming — maybe one of the places to put the limited time available for commenting on any one paper.)

Finally, no comment is made in isolation from a larger curriculum/experience. We can’t predict or control where students will go with our comments. We can only do our best with each of our own students AND work to build a larger curriculum (in our departments, in our whole college) that will enable each student to make something valuable and useful out of our individual efforts. The old: “I imagine my world without my favorite teacher/mentor, and I am bereft; I imagine my world with only that favorite teacher/mentor, and I am bereft.”

197

Meredith 10.10.11 at 5:21 am

I should have made explicit: our students will teach themselves when we motivate or inspire them to. A little encouragement can go a long way.

198

John Holbo 10.10.11 at 7:08 am

SoV, I am arguing that the rubric approach is less effective at securing consistency and transparency than its advocates think. Your objection seems to be that I must not care about consistency and transparency. This doesn’t follow.

I also make the point that you might be able to get more consistency by narrowing your assignments, giving students less leeway to take different approaches. But that means making a trade-off. You infer that I “for whatever reason just don’t want to restrict [my] assignments in that fashion.” That is, my motive would seem to be sheer callousness. But I quite explicitly did not say anything of the sort. I said the opposite.

I feel sort of odd saying this but perhaps we could just stipulate, for argument purposes, that I am an alarmingly lazy, callous grader. I am utterly cold to all virtues of fairness, transparency, and pedagogic professionalism. I do not see that you have any reason to think so, yet that it is so seems to be a strong – and growing! – article of faith for you. Fine. Let us move on to consider whether I might not be an omni-incompetent or moral monster who has, all the same, put forward intellectually considerable points about the rubric approach. (I am rather loath to stipulate to these initial conditions, even for argument purposes. But such is my faith in philosophy that I will pursue an argument even from an unflattering start point of auto ad hominem, since it is not as though I am taking my own virtuosity and conscientiousness as a premise or anything like that.)

“other people in other fields use rubrics and associated paraphernalia with some degree of success. So why can’t you?”

The basic concern is that grading the sorts of essays we assign is a function of applying rather holistic norms – quality, consistency, clarity – and that attempts break the hold of the holistic approach end up being the holistic approach by other means. This isn’t the argument, mind you. This is the concern behind the argument. I gave the argument above.

199

Neville Morley 10.10.11 at 8:17 am

@Meredith #196: “…the importance of reading more than one paper by a particular student before being able to grapple thoughtfully with his or her writing/thinking. ” Yes, at least within the fuzzy, subjective humanities where the aim of assessment is not only to to tick off the attainment of specific outcomes within that particular task but to guide the student’s intellectual progression through their whole course. My comments will be far more useful if I can say something like “your detailed readings of the text are significantly improved and better supported, but the overall structure of the argument is still rubbish”, giving the student a sense of how they’ve improved relative to their last effort rather than relative to an abstract set of criteria.

Except that another of the consequences in the UK of the drive towards ever more regulated and supposedly objective assessment practices – based on precisely the conviction of the superiority of practices in science and engineering (ignoring the differences in the nature of what is to be assessed) and the unprofessional amateurism of everyone in the humanities illustrated above – is that I can’t do this. All our assessments are now anonymised, because obviously if I know the identity of the student then the objectivity of my assessment, such as it is in the first place, will inevitably be compromised. So my comments can only ever relate to the abstract rubrics rather than being tailored to the particular needs of particular students.

I can see the arguments for this approach, and will readily concede that the old system of non-anonymous assessment was open to abuse (I don’t believe that it was actually abused very often, or even inadvertantly discriminatory very often, but it’s impossible to substantiate that argument in the face of the assumption that humanities professors can’t be trusted). It’s just another example of unacknowledged tensions within the system: the enhanced objectivity of the result comes at the expense of the usefulness of the exercise for the student. In theory, I could set a lot of formative assessment tasks alongside the formal anonymous assessment, where I could adopt an approach focused on the needs of individual students, but – quite apart from my own sense of how much time I have to spare – my university’s line is that we all over-assess already, so I wouldn’t be allowed to do it unless I can set it up as an automatic on-line exercise, which wouldn’t meet the students’ needs at all.

I can’t help feeling that the whole dynamic of the system is pushing us towards doing exactly what ScentOfViolets proposes: imitating the hard sciences in narrowing the scope of what is assessed, so that it is assessable with clearly-specified rubrics and near-automatic grading. Problem is, I honestly can’t imagine how most of my subject could be assessed adequately in that manner; it would effectively have to become a different subject. Now, from a hard science perspective this doubtless looks like clear evidence that the knowledge generated by the humanities is at best inadequate and at worst useless, and it’s difficult to frame opposing arguments without starting to sound like Martha Nussbaum…

200

kidneystones 10.10.11 at 10:26 am

I never write comments on students’ papers. Mind you I only grade several hundred each year. Others may do more. My feeling is that if the students can’t be bothered to make time to meet and discuss their work, I should ensure that I invest a comparable amount of energy in feedback. I’m always available by email and in person to discuss paper preparation and revision before the work is turned in. I enjoy assisting students with their research. If the first comment they receive from me about their work arrives after the paper been completed, no more need be said. If the paper is part of an ongoing discussion, then I ask them to critique their own work: how well was the question framed? Does the introduction clearly identify sources to be cited in the discussion. How fairly is the counter-argument presented and where? The technical elements of writing can be dealt with in groups. Seminar students respond extremely well to this sort of approach. Large class lecture students self-select for detailed responses. Not every student cares much about the work they do. Our efforts should directed to those who clearly do. But that’s just me.

Shaping young minds and all that. Go for it.

201

post46 10.10.11 at 11:14 am

I think a large part of the problem is the fact that we use numbers in the first place, when what matters is the classification that they get turned into. If you’re marking a stack of undergraduate essays you have an idea of what you’re looking for – you probably have a much better idea of what you’re looking for after you’ve marked the first 10 or so, but that’s another matter. On that basis, you have an idea of what a First-ish essay is going to look like, and the same for 2.i-ish, 2.ii-ish and Third-ish essays. The numbers come next, and they only really come at all for administrative convenience. 55 isn’t a mark, it’s the number-ese for “solid 2.ii”.

Of course, marking schemes – particularly the really prescriptive “10 marks for presentation” variety – are impossible to reconcile with this approach to marking; in practice something’s got to give, and in practice what gives is has to be the marking scheme. Or rather, what gives has to be the marking scheme. You could mark in strict accordance with the scheme, but you’d inevitably end up with perverse marks that you felt unable to justify or defend – which would be a pretty miserable experience at best, and could be much worse if some of the marks were challenged.

One attraction of the marking scheme is that it gives students something to work with – if the aggregate mark is 48 and referencing scored 6 out of 10, bingo, there‘s what you need to change in order to get a better mark. Of course, what they really need to do is write a better essay, but learning how to do that takes a lot longer and a lot more work.

Having said all of that, I do think the OP and some commenters are talking past each other, because they’re arguing on the basis of very different experiences. When I hit a First-ish essay, worrying about whether they’ve answered the question the way I wanted it answered would just be a distraction. When I’m reading 2.ii-ish essays – which I do much more frequently – answering the question the way I suggested could make the difference between a high 2.ii and a high Third. Teaching in the less rarefied strata of HE, you learn to teach students how to jump the right fences in the right order, and you’re gratified when they manage it. You’re always looking out for the students who don’t need this kind of approach and can benefit from being encouraged to strike out on their own, but they’re not the majority; they’re not even a large minority.

[longtime commenter, temporarily anonymised for obvious reasons]

202

ScentOfViolets 10.10.11 at 11:17 am

SoV, I am arguing that the rubric approach is less effective at securing consistency and transparency than its advocates think. Your objection seems to be that I must not care about consistency and transparency. This doesn’t follow.

No, that is not my objection. I’ve said that you do not care enough about consistency, transparency, and feedback to put the requisite comments on each individual paper. This is something you’ve explicitly said, so you can hardly go on about my “interpretation”. If you like, I will quote you on this one.

I feel sort of odd saying this but perhaps we could just stipulate, for argument purposes, that I am an alarmingly lazy, callous grader. I am utterly cold to all virtues of fairness, transparency, and pedagogic professionalism. I do not see that you have any reason to think so, yet that it is so seems to be a strong – and growing! – article of faith for you.

Sigh. You know, it wouldn’t hurt you to practice a little bit of that rigor you’re supposedly grading your students on. Let me quote you, again:

They just don’t see the problems, so simply saying ‘you got a B because the organization was weak’ – although a true, general statement about what went wrong, hence why the grade was given – is often no more enlightening to students than just plain ‘you got a B’. Often the way to get the student to see how to improve is, for example, to point out, laboriously, 10 small things going wrong in the intro. (‘Why did you start this sentence this way? Why doesn’t this sentence come first? Why did you say ‘yet’ rather than ‘and’ here? What job is this intro paragraph supposed to do? What reader would be informed by this paragraph, and is that reader also the reader who would be informed by what you have to say in the rest of the paper?) This sort of sentence-by-sentence trench warfare is nothing you can wage across a whole 5-page paper. I’m not made of time, you know. But, often, there is no way to make the student see what you mean except to get nitpicky about some little bit. But then you end up not really providing ‘general comments’.

I even included the part I originally bolded (and was in part what prompted me to respond.)

These are your words, John. I’m not twisting them, I’m not giving them any sort of extra significance. They are what they are. If you want to retract what you said, fine.

But this sort of innuendo on your part, with your “seem to think” or “seem to feel” is just pathetic. Please cease and desist.

203

ScentOfViolets 10.10.11 at 11:29 am

I can’t help feeling that the whole dynamic of the system is pushing us towards doing exactly what ScentOfViolets proposes: imitating the hard sciences in narrowing the scope of what is assessed, so that it is assessable with clearly-specified rubrics and near-automatic grading.

Let me be very clear on this: I have not said that humanities should resort to a rubric. I have said that the grading should be transparent, impartial, and provide some sort of good feedback to the student so that they can correct their errors. Nothing more.

If it takes – per John’s description – hours and hours of blow by blow comments per paper, so be it (though I suspect that’s rather inflated; at those rates maybe he should just schedule explicit appointments for each student to discuss their individual papers. In fact, I know several teachers who do just that.) If it takes a restriction of topics and forms to facilitate grading, so be it.

Anything rather than “I know a B when I see it, but I don’t have the time to explicitly say why.” Students have a right to be peeved at that sort of system.

It seems that what’s really going on here – and I agree, emphatically – is that the administration doesn’t want to give us the time, the resources (the money) to grade in anything like a comprehensive manner.

204

John Holbo 10.10.11 at 1:05 pm

SoV: “I’ve said that you do not care enough about consistency, transparency, and feedback to put the requisite comments on each individual paper.”

The thing you quote doesn’t say anything like that, SoV. I’m said I’m willing to admit for the sake of argument that I’m a bad person. But I draw the line at admitting you have evidence to that effect. (Be reasonable!) Can’t we get back to talking about how to comment?

All the bit you quote says is that there’s a limit to how much time I can – and can be expected to – devote to grading. But that’s fairly trivial. (No one is suggesting that I grade more hours than there are in the day, for example.) How does it follow that what I’m doing isn’t the requisite amount? Along whose yardstick, ‘requisite’? Do you have some antecedent notion about how much time is needed, absolutely? And my half hour per paper is categorically not enough? That seems unlikely. Are you just assuming that it is requisite to do what you think would best ensure consistency, fairness, education and all the rest, rather than what I think would work best? But that is question-begging. You are, again, crossing the issues of best practices and bad motive. You are welcome to believe I have bad motives, but please recognize that the fact that we have disagreement about best practices is not, per se, evidence of bad motive on my part.

Now you write: “I have said that the grading should be transparent, impartial, and provide some sort of good feedback to the student so that they can correct their errors. Nothing more.”

Everyone agrees that best practice is to optimize the supply of these values, SoV. The question is not whether, but how. (You are, I will say it one last time, conflating the issue of best practices with the issue of bad motives.)

205

John Holbo 10.10.11 at 1:06 pm

And on that note, I’m off. I shall not return until the morrow. Do not take my silence amiss.

206

kidneystones 10.10.11 at 1:15 pm

In praise of rubrics. I just pulled two texts off the shelf that some here will likely be familiar with: John Creswell’s Research Design – Qualitative, Quantitative and Mixed Methods Approaches; and Bennet and George’s Case Study and Theory Development in the Social Sciences. I did my undergraduate degree in English and American Literature in the early nineties and graduate work in the social sciences.

Hard as it is for some here to believe, not every student finds professors keen to hear dissenting views. As a teacher, I really try my best to avoid putting students in a similar position. To that end, presenting students with simple rubrics can be very helpful. Rubrics can vary. For example, we need not always grade for punctuation or spelling, unless the paper is being prepared as an exercise to demonstrate and practice these skills. I’m much more interested in the clarity of expression and the logic and organization of the argument. Work for graduate school is a different matter.

The key idea underpinning both texts is that there are some fundamental elements that make a good research paper. When these elements are present and are presented in a reader friendly fashion, the reader comes away with a clearer understanding of the topic in question. I don’t happen to like Robert Frost and his poetry, but I’m always pleased to read why others do. I express my own dislikes and likes openly in class and invite students to demonstrate where I’m wrong. These, of course, are the best kind of papers to receive and read. Writing and grading papers needn’t and really shouldn’t be tedious.

207

bianca steele 10.10.11 at 6:26 pm

SoV: “either . . . either”–hilarious

I think John is right in the OP that rubrics can make things worse. I’ve become interested in how people misunderstand rubrics and similar formalized criteria. Sometimes people just can’t believe it’s as simple as it seems. They conclude that what they’re being taught is some kind of obscure or esoteric “code.” Sometimes they seem to focus on apparently irrelevant elements of the training session: the examples, the presentation, the division of labor between the presenters, whatever. Though I’d be less worried this would happen with grad student TAs than with undergraduates (or with older learners who’ve been persuaded they were wrong to take everything so straightforwardly for so long).

208

John Holbo 10.11.11 at 3:03 am

OK, I’m back. I’ve been grading Descartes papers, it so happens. And, it so happens, I just graded a paper that raised exactly the issues that zehou and I discussed, upthread. So let me be fairly concrete about one case, in the hopes that this will actually clarify things.

The topic was basically an ‘assess the first and second meditations’ thing (there’s more detail than that but it doesn’t matter). And one student took an unusually historical approach that wasn’t mandated by the topic, but wasn’t forbidden either. I ended up writing ten or so brief comments ‘not clear’, ‘this isn’t right because …’. I only made one really general comment, which ran about like so: ‘the paper tries to do two things: 1) establish the historical context; 2) state and criticize the arguments. That’s a reasonable plan. I like the history stuff, too. And trying to do two things at once, that mutually inform each other, is ambitious. But 1 and 2 just end up jostling each other, trying to get through the door together. The history you include is scattershot and somewhat misleading; the arguments not stated fully enough to make the criticisms compelling …’ A bit more in that vein.

Now, what grade did I give the student? You don’t know, do you? B-, a B, a B+, maybe even an A-? It could be any of these, depending on how serious I took the stated, general problem to be (and how serious the other 10 incidental problems were, and what I thought about other aspects of the paper I didn’t end up commenting on, and how flawed all this stuff is relative to what was wrong with other student papers.) The student knows what grade he/she gets, of course. But he/she can’t really know WHY he/she got that particular mark, on the basis of my comments, for the same reason you don’t know.

Now, in a sense it follows that the grading is not ‘transparent’. Why B, rather than A-? The comments themselves do not prove one grade is absolutely more appropriate than the other. (To the extent that SoV has a reason for his conviction that I am history’s greatest grading monster, this would seem to be it.) What could I do instead? Well, I might craft a rubric. The first step will be to abstract form from content and find a way to express that, quantitatively, so that the student can then know why the numbers added up to the grade he/she got. The problem is that this doesn’t really work very well. I think this student’s problem is more form-related than content-related. So I should mark the one down while giving points for the other. But if a different reader deemed it more content-related than form-related, I could see that, too. My view is that an inapt organizational scheme kept content from coming through clearly. But perhaps it is rather the case that a weak grasp on content induced bad organization. Maybe what looks to me like unclarity of expression is truly just intellectual confusion. This is a very fine question about the genesis of the problem I saw with the final product.

So with the rubric you offer something that looks like an explanation of why the student got the grade he/she got but, functionally, it really isn’t. The rubric is ‘transparency theater’, as you might say (on the analogy of security theater, you understand.) It has numbers that add up. But since different numbers could have been assigned, it is only pseudo-technical that way.

If I had to guess how other graders (and myself on a different day) might differ about this paper, my guess would be the following: there would be reasonable consensus about the grade that it deserves. But there would be some genuine intellectual disagreement about how good the paper really is. (That’s the way it goes.) In the event that we were all being professionally obliged to put little numbers in boxes, there would be quite wild oscillations, due to some people slotting as a form problem what other people saw as a content problem. There would be more consensus on the correct final number than on the individual numbers attached to different aspects of the paper. (The latter would tend to be deduced from a prior judgment of what the correct final number should be.)

A better approach is to offer rich enough qualititative comments that the student can actually see how and why some particular thing/aspect went wrong, and needs work. Writing cogent, qualitative comments on an aspect of a student’s paper actually gives the student some evidence that you read and understood the paper – it’s not proof, but it’s actually better proof than a checklist with numbers. Which could just be made up, after all.

And you do various other things, not in individual comments on papers, to minimize inconsistency in grading and inform students about standards, and uphold them as best one can. That should go without saying, but probably I should say it. (Drink enough caffeine, but not too much. Invite students to come talk to you in office hours. Say clear things to the whole class about the assignment. The list goes on.)

It may be that I really am history’s greatest grading monster, for taking this approach. I am, in a sense, giving up on one possible goal: including in each paper a bespoke, tailored proof that the paper deserved the grade it got, none other. But from where I sit, it looks like the most I could really do would be pretend to do that, not really do it. It’s better to spend my time doing something that I really can do. So, per the post, I do 2 not 1.

209

LFC 10.11.11 at 4:16 am

…engaging the substance of the student’s ideas in some way. Whatever else we may have to say …, we can always pick up on some observation or insight the student has had and just respond to it … a response that takes them seriously as thinkers, as people. … I don’t mean the “good” or “excellent” in the margins… but rather, “I’m learning from you” or “my imagination is being excited by you,” the way we learn and are excited in exchanges with colleagues and friends.

Well put, imo. In years of writing papers, I believe on only one occasion did a reader write on a paper of mine that he had learned something from what I had written; others implied it now and then, but only one person explicitly said it (now, I admit that many of the papers I wrote were pedestrian and so I wouldn’t have expected such a comment very often at all, but once out of scores of papers perhaps reveals something about those doing the commenting [?]). The one occasion I refer to was one of the first papers I wrote in college (many years ago), and I still have it.

210

Meredith 10.11.11 at 4:40 am

John@208, makes sense to me.
Maybe just a quibble. “But he/she can’t really know WHY he/she got that particular mark, on the basis of my comments, for the same reason you don’t know.” I should think he/she probably does have some idea why she got that mark, assuming that you don’t evaluate students’ work wildly differently than your colleagues do, or than other teachers in other contexts (e.g., secondary school) have graded this student (and commented on his or her work). Most of the time, students seem to get it, in my experience. Maybe not why they got a B rather than a B+, but they get the broad strokes of grading. Somebody way upthread, as I recall, commented on having students evaluate one another’s work — and how much harder they tend to be on one another than we are. Sometimes I have shared sample paragraphs of students’ work with the whole class, to illustrate standard problems and also strong writing and argumentation. Most students get it, very quickly. (Some teachers do a lot of that sort of thing, and do it effectively. Not my thing most of the time, but I can see its virtues.)

Rather than push the question, would you have commented differently were this the second or third paper you’d received from this student (I’m assuming it wasn’t), I’d like to ask something a little different. Say this was the second or third paper from this student in this course, or you’d taught him or her previously. If this paper basically and honestly struck you as a C+ paper, but you knew the student had been struggling to improve certain elements of her writing and thinking and had, in fact, improved on those, and you judged that, even more than your commenting on the improvements, giving her a B- would give her just enough encouragement to keep her trying, would you give her the B-? (Assuming this wasn’t somehow going to commit you to giving her an undeserved final grade.) I know I would (though for some reason, I probably wouldn’t give an A- to a B+ paper under the same circumstances). Probably very sinful of me, in the eyes of some.

211

Zehou 10.11.11 at 9:51 pm

John: “My view is that an inapt organizational scheme kept content from coming through clearly. But perhaps it is rather the case that a weak grasp on content induced bad organization. Maybe what looks to me like unclarity of expression is truly just intellectual confusion. This is a very fine question about the genesis of the problem I saw with the final product.”

These are, as you make explicit, competing hypotheses about “the genesis of the problem [you] saw with the final product.” Now, is there a way to characterize that problem in a neutral way? Perhaps: how the parts of the project are supposed to fit together is unclear, though your reader can’t be sure whether the cause was a bad plan (for expressing something you understood well), bad execution of a good plan (for expressing something you understood well), or a plan for expressing something you simply didn’t understand well.

If there’s a way to accurately describe the problem whilst being completely agnostic about the cause, then we might think the rubric (on the assumption we’re considering using one) should be written so that there’s a way to indicate the presence of the problem regardless of the cause, in which case it doesn’t matter if different graders are inclined toward different causal hypotheses. Grading papers is not grading students, we might say: the paper could be bad to degree x, but the student’s achievement in writing it could vary dramatically depending on which causal explanation for the badness of the paper is right. But the rubric is a means of grading the paper, not the performance, and so using it needn’t involve us in thinking about the cause at all.

My suspicion is that you think there’s something wrongheaded in this view, but I’m not sure quite what you think it is.

212

Meredith 10.11.11 at 10:09 pm

Zehou, what use is it to the student if the paper is graded, not the performance (a distinction I’m not sure I understand), without any reflection and comment by the reader on the (possible) “cause”? Seldom do I read a student paper that I think, gee, the world should see this, for its insights! (And that’s the one context in which I can imagine thinking only in terms of “the paper.”) If you’re commenting on the paper in order to help the student (and not just in order to grade the paper), don’t you need to think about causes? Maybe you’re only thinking about grades, but this obsession with grading distracts, it seems to me, with our primary activities when we’re reading papers.

I agree, though: grading a paper is NOT grading the student, something very important to convey to students.

213

Zehou 10.12.11 at 3:00 am

Meredith: “what use is it to the student if the paper is graded . . . without any reflection and comment by the reader on the (possible) ’cause’?”

The point of commenting is to help the student (or so I think, anyway), and writing comments informed by what you think went wrong or right in the student’s process seems to me a great way to be helpful.

But that’s commenting. Grading is the process that aims at discovering and assigning the correct grade. Grades on papers are supposed to be correct, and that’s all. Right?

214

Bloix 10.12.11 at 3:01 am

215

ckc (not kc) 10.12.11 at 3:08 am

Grades on papers are supposed to be correct, and that’s all. Right?

…or – grades are ranks, with an arbitrary cutoff at “adequate”

216

Meredith 10.12.11 at 5:04 am

Having been listening obsessively in the last week to different performances of Dido’s lament (“Thy hand, Belinda…. When I am laid in earth…”) in Purcell’s Dido and Aeneas, I suddenly realize we’ve left out the performing and visual arts in this discussion. Gets me to thinking. As my (almost) first and (certainly) most influential violin teacher counseled me before a recital (a million years ago), “Just make your performance convincing. While your audience is listening to you play, they will respond: This is the only way it can be played.”
So, grading. You know, you audition, and someone has to decide, yes or no, first chair, middle chair, last chair, not at all. Presumably, those decisions have something properly to do with things that matter in terms of future performance for a larger audience. (Cf. grad schools, some employers.)
Papers and their comments. More like the lesson you’ve prepared for with all your heart (or not much at all) that week. An ongoing conversation/struggle/love affair (the last most of all) between the music and the musician (with that teacher in there, prodding, criticizing, encouraging — the disappearing teacher! but no musician ever forgets her teachers!). I guess I’m saying, yes, some kind of evaluation (grading) has to be done. But that whole project is subsidiary to the large project: the music.

217

John Holbo 10.12.11 at 5:49 am

Some further thoughts …

Zehou is right to suspect that I don’t really like this idea (even though I see the appeal, because it promises greater consistency, which would be great):

“If there’s a way to accurately describe the problem whilst being completely agnostic about the cause, then we might think the rubric (on the assumption we’re considering using one) should be written so that there’s a way to indicate the presence of the problem regardless of the cause, in which case it doesn’t matter if different graders are inclined toward different causal hypotheses. Grading papers is not grading students, we might say: the paper could be bad to degree x, but the student’s achievement in writing it could vary dramatically depending on which causal explanation for the badness of the paper is right. But the rubric is a means of grading the paper, not the performance, and so using it needn’t involve us in thinking about the cause at all.”

The problem with this is that I think grading has to be grading students rather than papers. A paper that has solid content, just poorly expressed, is worth more than a paper that is just plain confused (because I weight content more than form). So you have to try to tell the difference. Here’s another way to think about it. If you really just graded what actually got down on paper, rather than mucking about guessing what the student was trying or attempting or had in mind, then unclarity in writing would basically be a capital offense, grade-wise. In fact what we do is give half-credit, as it were, for unclear content. But that means guessing at the content. That is, guessing at something that, in a sense, isn’t in the paper

There is an important corollary of this awkward truth: namely, correct grades for unclearly written papers are inherently indeterminate to a potentially highly significant degree. To assign any grade (other than a grade that would be too low) you have to give credit for something that didn’t really make it in (what was meant, rather than said; what was attempted, but not really achieved). But there is just no solid method for deciding what this absent element should be deemed to be. You just have to make your best guess and grade accordingly.

Let me tell you how I handled this, administratively, in my really big intro module, where I had lots of tutors (I’ve retired from this duty for the time being). We did blind double-grading of final papers. When there were serious discrepancies, I made the final call. Sometimes one tutor seemed right and the other wrong. But often, when the paper was particularly unclear, I could have gone with either of two significantly different grades. It all depends on what you decide to imagine going on behind the scenes. Something bad or something not so bad. Then it seemed best to split the difference.

I also allowed students to appeal their tutor’s mark. But the rule was: if the mark is reasonable, it stands, even if it would be equally reasonable to give a higher mark. I try to be transparent, so I would actually tell students: this paper could have gotten a B, but your tutor gave it a C, and the grade stands. The student is properly shocked to hear this. But then I explain that the reason is this: it gets a B if the tutor imagines that the student really understood it all, but just tragically failed to express it clearly. It gets a C if the tutor thinks the student just has no idea what he/she is talking about. The tutor obviously thought the latter, and that was reasonable. The paper is really very unclear, and it seems possible the cause of that unclarity is deep, deep confusion about the content. The student wants to know why he/she can’t get the benefit of the doubt. I point out that, if we always assumed the best in these cases, then writing unclearly might even be a good strategy. We would always just assume that the author of every unclear paper was a misunderstood genius. That’s worth an A, probably. But that seems absurd. Then the student says: why not just grade me on what I wrote. Why not be objective? Because then you’d get a D or an F. Because you didn’t actually succeed in getting anything down, without help from charitable assumptions by the reader.

In a sense this ought to be obvious. Writing badly is risky. You may be misunderstood. And if you are, you will get double-dinged, grade-wise. Once, for writing unclearly. And again, for being wrong. This is unfair, in a certain sense. But there is no procedural remedy. So the moral of the story is: if you don’t want to risk being graded down for not understanding stuff that you actually understand, don’t write unclearly. Or major in math. Writing unclear essays means rolling the dice and hoping for a charitable reader. Best of luck to you!

I’m putting the point somewhat extremely, for illustration purposes. It isn’t like I think papers whose proper grades are radically indeterminate are the norm. But saying the sort of thing I just said is part of the transparency process. You need to make the students understand what you are doing and how it works. And, again, I think rubrics tend to conceal this more than reveal it.

218

John Holbo 10.12.11 at 6:34 am

Bliox, yes I noticed that post, too. I don’t know what to think about it having not read the book in question.

219

Neel Krishnaswami 10.12.11 at 10:01 am

Or major in math.

This won’t help them — math majors do proofs, and grading proofs is a lot like grading essays. Your essay @217 absolutely precisely describes my experience teaching proof-based mathematics courses. Almost all student proofs are unclear, incomplete, or wrong, and you really have to judge whether the student understood the argument or not.

The same goes for computer programming courses, since again, the programs don’t work or are horribly written, and you have to judge how much the student understood what they were doing.

220

Salient 10.12.11 at 10:59 am

Why not be objective? Because then you’d get a D or an F. Because you didn’t actually succeed in getting anything down, without help from charitable assumptions by the reader.

+1. I’d go further: The numerical grade encodes a measurement of how much charity the reader needs to expend reading through from beginning to end. Even if that’s irrelevant to how the grade was arrived at, the correspondence is nearly guaranteed.

221

John Holbo 10.13.11 at 1:57 am

Let me add a slightly longer response to Bliox, whose comment far upthread (54) was certainly worth reading and thinking about. I didn’t say much about it because I consider it tangential to my point in the post (not that there’s anything wrong with that.) Bliox concludes by saying that students know grades are often quite arbitrary and it’s a cryin shame. I just gave a defense of arbitrary grades, in certain not uncommon circumstances, so it seems worth closing the circle in response to that earlier Bliox comment. I have argued that a significant degree of arbitrariness is not a function of flawed grading, paradoxical as that might seem. That one tutor gives a C and another a B is cause for concern but not a proof that something has gone wrong. But that’s not to deny that all the methodological hazards Bliox outlines are really potential problems – and often actual problems. They are. The problem is that it’s easy to imagine a system that would be a lot fairer than the system we’ve basically got. But that system would be hugely more costly. So for a lot of resources expended we could get a lot of improvement. Is it worth it? Maybe. It depends how much ‘a lot’ and ‘a lot’ turn out to be. And that’s of course hard to calculate. Maybe I should go read that book Matt Y talked about.

Comments on this entry are closed.