Two-point scales

I can’t think of any citations, but I wouldn’t be surprised if there’s been analysis of, say, the ‘stars’ awarded in Rolling Stone‘s album reviews. Or any kind of subjective review using either five stars, a 1-10 scale, or a percentage value.

If he’s referring to the UK, the top level has been used, for about a month after 7/7.

Simon, the passage John Q is referring to is discussing the US threat-alert system. The British government has only just made its own system public. It will be interesting to see how it is used.

Star ratings in music magazines use the entire spectrum, although the extremes are naturally rarer than the middling ratings. Music reviewers also like to occasionally declare that something they’ve been listening to*) sucks to high heaven, so they have an incentive to give 1- or zero-star reviews. But then, they have no disincentives either way; they can’t fire an album that sucks, and don’t have to do followup interviews with someone who has just made a brilliant album.

*) probably a very charitable assumption on my part, that.

Maybe it’s because the threat hasn’t been reduced. This seems to be yet another paranoid conspiracy theory. There’s a lot of it about, both left and right.

At UW Madison we grade students on an A, AB, B, BC, C and D scale. My understanding is that A is widely used, and D rarely. I don’t understand why there is no CD option, by the way.

The PG-13 rating was added to the MPAA’s movie content rating system in part because there was a perception that the system had effectively shrunk to two useful ratings, PG and R: after a period in the 1970s when G and X ratings were actually given to well-respected movies for adults, usage had shifted so that G was only given to obvious kiddie movies and X to obvious porn.

Since the introduction of PG-13, PG has been gradually turning into a second kiddie rating, though it seems as if the system hasn’t yet completely degenerated to two ratings again. I suppose having two grades of kids’ movies does impart some information.

Two observations (and sorry none of this is of the technical nature you asked for):

On grading, I still can’t get over my wife’s contention that, at grad school, everyone gets an A except for a few Bs, and C is effectively failing (others have backed up this assertion; sorry if you all think it’s simply not true, I don’t know). She argues that only exceptional students – A students – belong in grad school, while I argue that grades are always relative – even in the Manhattan Project, there was a bell curve of talent, however far over on the scale of general human achievement.

As for stars, I can speak as a professional (free-lancer, anyway): as a restaurant reviewer, 90% fall between 2.5 and 3.5 stars on a 4 star scale. A lot of this is selection bias – why would we subject ourselves to a crummy meal? – and a lot of it is the consequences of giving 1 star (0 is not an option). 1 star seems to suggest near-inedibility, and certainly results in angry calls to the editor. Meanwhile, 4 stars are reserved for essentially perfect meals (not necessarily haute cuisine – a cafe brunch recently got 4 stars), perhaps one every 6 months. So everyone else gets shunted into this absurdly narrow range. Our predecessor actually gave everyone 3 stars as a protest against the star system.

Bell curve

steve: it’s not about future events, but what some people want to portray to US citizens as its prognosis of future events. That much should be obvious.

(But hey, this is Bush we’re talking about! Every word from the Bush administration must be interpreted in the most charitable light possible, while every word from opponents of Bush must be interpreted in the most uncharitable light possible. To do otherwise will be tantamount to high treason.)

I remember that a friend had a “card trick” in which he held up a card reading:

“PICK A NUMBER

1 2 3 4”

I picked “3”, and he turned the card over, and it said, “I knew you would pick 3!”

I asked what the trick was, and it turned out there was no trick. I watched him show the card to several other people, and they all picked “3”, too.

I’m not sure how, if at all, that relates.

So just about everyone gets a 2 or a 3, yielding, in effect, a two-point scale.

This is sort of the opposite of what might be called the Spinal Tap Principle of metric-ology, whereby the introduction of extreme values into an (arbitrary) conventional scale makes it more attractive: “but it goes to eleven, man!”

I’ve worked as a music and theatre critic for newspapers who use a five-star scale. I tried to reserve my 5s for really mindblowingly outstanding things (ie one or two a year), but other critics were giving 5s out regularly to stuff they clearly (by the text of their reviews) thought was “merely” very good, so I looked ungenerous giving the same things 4, which in others’ usage apparently meant just “good”. And what did 3 mean? Average or mediocre or quite good or actually not very good? Any of the above, depending on the critic. Reviews given one star, meanwhile, were the most likely to be spiked, unless they were of highly anticipated or puffed things. The whole system of awarding points out of 5 to artistic endeavours is, of course, a philistine farce.

while I argue that grades are always relative – even in the Manhattan Project, there was a bell curve of talent, however far over on the scale of general human achievement.

This is probably not the case. When you have a selected population like the physicists on the Manhattan Project, or the students in a top-ranked graduate program, you’re left with the only the left tail of the curve. If you rate your people on a 1-5 scale, you should expect the most 1s, fewer 2s, even fewer 3s, etc.

richard bellamy, are you sure that’s what your friend actually said? I’m guessing you have added one thing that they didn’t say, because there is a well-known “trick” in which the tester says
“pick a number from 1 to 4”

“Three” is the response of something over 85% of the subjects tested.

The reason? The power of verbal suggestion. In saying “one to four,” we hear “one two four” and most subjects dive for the “safety” of the only number not named in the test: three

Try it on a friend sometime, works like magic. Don’t even need a card to do it, but the card is a nice touch.

Speaking of card tricks, has anyone ever had you call the “wizard”? Far and away the best card trick and the one that most baffles and amazes that I’ve ever performed.

has anyone ever had you call the “wizard”?

That used to be my favorite too. I learned it from my uncle when I was a kid. Haven’t thought of it in years.

The British government has only just made its own system public. It will be interesting to see how it is used.

I worked for the police for about six months last year, during which I had access to the threat classification (it was no great secret internally, being posted on the inner door as we entered the building). The threat has mostly been severe general (second highest) since 9/11, but was infamously lowered to substantial about a month before 7/7 then raised to the maximum afterwards, where it remained for a further month. They also had separate classifications for particular groups, eg dissident and indeed non-dissident Irish Republicans, for both of whom the threat was raised to substantial for a brief period after the IRA bank robbery.

Thanks to this morning’s news, the threat is back to critical.

I note the sad irony that we’ve been partially overtaken by events on this one as well. The Brits have actually gone to “critical,” although it’s not clear whether it’s system-wide. As you note, we’re at red for transatlantic flights and orange generally.

The iTunes five-star ratings system seems flawed for similar reasons, because (unlike reviewers) people who have music on their computers are unlikely to keep stuff around that’s at the bottom end of the scale.

has anyone ever had you call the “wizard”?

That used to be my favorite too. I learned it from my uncle when I was a kid. Haven’t thought of it in years.

All of my family know to expect a call to the wizard at any time, day or night. Even wakened from his or her slumbers at 3 AM, the wizard has always come through.

I’m glad someone else knows the wizard.

Antti wrote:

Hi,

The magical number is seven, plus or minus two.

http://www.well.com/~smalin/miller.html

Have a nice day,
Antti
This result is irrelevant. The seven +/- two rule is for remembering bits of data, not for making rankings of things.

As the Spinal tap example hints, and as the old “on a scale of one to ten, she’s a twelve” joke also indicates, you should ignore the arbitrary boundaries often placed on these scales. Are movies rates from 1 to 4 stars (actually seven ranks, from 1 to 4 by 1/2 steps)? Then occasionally hand out a 6, or a -3. You know in your heart that some are just that bad, or that good.

Likewise with every other ranking/judging metric you have ever encountered.