In the thread on community colleges (which morphed into a discussion of more general education and management issues), someone mentioned Kahneman on the “halo effect” in grading (or marking) student work. Thinking Fast and Slow has been on my to-read pile since Christmas, but I got it down from the shelf to read the relevant pages. Kahneman:
Early in my career as a professor, I graded students’ essay exams in the conventional way. I would pick up one test booklet at it time and read all the students’ essays in immediate succession, grading them as I went. I would then compute the total and go on to the next student. I eventually noticed that my evaluations of the essays in each booklet were strikingly homogeneous. I began to suspect that my grading exhibited a halo effect, and that the first question I scored had a disproportionate effect on the overall grade. The mechanism was simple: if I had given a high score to the first essay, I gave the student the benefit of the doubt whenever I encountered a vague or ambiguous statement later on. This seemed reasonable … I had told the students that the two essays had equal weight, but that was not true: the first one had a much greater impact on the final grade than the second. This was unacceptable. (p. 83)
Kahneman then switched to reading all the different students’ answers to each question. This often left him feeling uncomfortable, because he would discover that his confidence in his judgement became undermined when he later discovered that his responses to the same student’s work were all over the place. Neverthless, he is convinced that his new procedure, which, as he puts it “decorrelates error” is superior.
I’m sure he’s right about that and that his revised procedure is better: I intend to adopt it. Some off-the-cuff thoughts though: (1) I imagine some halo effect persists and that one’s judgement of an immediately subsequent answer to the same question in consecutive booklets or script is influenced by the preceding one; (2) reading answers to the same question over and over again can be even more tedious than marking usually is. I thing it would be even better to switch at random through the piles; (3) (and this may get covered in the book) the fact that sequence matters because of halo effects strikes me as a big problem for Bayesians. What your beliefs about something end up being can just be the result of the sequence in which you encounter the evidence. If right (and it’s not my department) then that ought to be a major strike against Bayesianism.