Comments on: Estimation

By: David Margolies

David Margolies — Mon, 30 Aug 2004 20:02:23 +0000

45%. I would have done much better (pehaps 9 more points) if I said the English Civil War started in 1641+-1 rather than 1641 exactly (answer 1642).

By: Joshua W. Burton

Joshua W. Burton — Sun, 29 Aug 2004 06:51:32 +0000

“Conversely, if you’re accidentally dead on, but overestimate your error, you’re hardly penalized, if at all. (That happened to me a couple of times.)” I don’t know how that could have happened to you, when I guessed “400 +/1 400” for the weight of the Boeing, which was off by less than 1%, and got 0 points. I answered 4 +/- 130 BCE for Jesus’s birth, fully expecting to be penalized for being a smartass, but got full marks. On the other hand, 800 +/- 700 billion stars (another “too correct for the quiz” answer I was proud of) scored nil. In general, it’s awkward to mix intrinsic errors with measurement uncertainties, since they are statistically independent and therefore add in quadrature. Perhaps questions like these that don’t have real answers (within an order of magnitude) should be stricken from the next iteration.

By: M Kochin

M Kochin — Sat, 28 Aug 2004 19:45:11 +0000

58% Would have been better if I hadn’t read Tony Blair for Tony Benn :>( http://roughly.beasts.org/scripts/quiz?_eq_web_session=a37a1095bdd1b918

By: Tracy

Tracy — Sat, 28 Aug 2004 12:37:00 +0000

I was puzzled by working out the error bands for the UK asylum seekers benefit. I vaguely recalled reading that it was about £40 a week, so that was my best guess. But, when it came to error bands, obviously asylum benefits can vary upwards without limit and without breaking the laws of arithmetic, but if the UK started charging asylum seekers for staying in the country it’d be called a tax or a fee, not a benefit. Which makes for a lop-sided error band, but you can’t specify that in the question. So I gave a wide band and got a 0, despite being very close to the right answer. Anyway, 37%. Not helped by me thinking of things in metric and then forgetting to convert. (700 k is roughly the distance between major NZ cities, it would have been a better guess if I’d turned that into miles) Incidentally, the Magna Carta was signed several times by various kings of England. To be pedantic, the question should be specified as when it was first signed. Tracy

By: plover

plover — Sat, 28 Aug 2004 09:12:24 +0000

50% for me. 3 of those percent are directly attributable to Glenda Jackson, who happened to mention the number of MPs in a radio interview a few days ago. Being a Yank and thus not having a clue who Tony Benn might be, but deciding that the name couldn’t go too far back I thought 1800±200 quite reasonable, but I got 0 points. However, my wild guess of 2000±1000 for the UK GDP got 3 points even though the actual answer isn’t even in the range. Do you suppose the scoring system assumes you know that Tony Benn is a recent figure? (And maybe also that he’s an adult—I set my range to include the idea that he might be a child.) The quiz now says that carrier bag means shopping bag, which I assume implies it’s been altered since most people here took it.

By: vivian

vivian — Sat, 28 Aug 2004 02:38:12 +0000

From the brief blurb “how well you know what you don’t know” I thought that some of the scoring would be on how well you choose margins of error-in-guessing. That is, no matter how large the error, or random the guess, if the respondent is at least aware of how wide to make the confidence interval, that would count relatively more than being right with the guess. Not sure exactly how that would be done, but it ain’t done here. For the Tony Benn question I guessed 1925+/-10 and the answer was 1925 exactly, and you took off a point, as though you think I thought his mother’s labor lasted about ten years when it was mere hours. In other words, your margins appear to be inherent in our current ability to measure the quantity, not in the test-takers awareness of the vagueness of their recollections. Orders of magnitude also should count for more, esp. with the numbers that aren’t dates. Getting within a factor of ten is good enough for a Friday night not cheating, while getting within a factor of two is pretty good for a casually informed guess. Getting within one’s own margin of error, close to the center of it, is good regardless of one’s knowledge of the facts, but narrower ranges are good too. Perhaps these are best brought out with different dimensions of scoring – or maybe you don’t really care about how well we estimate our own uncertainties, but then you might consider changing the cover blurb. I’m pleased with 36% though, as a merkin.

By: Dan Simon

Dan Simon — Fri, 27 Aug 2004 23:02:17 +0000

I think the problem may be that reasonable error ranges are highly dependent on the question. For example, I seem to recall doing very well by answering “50 +/- 5” degrees for the latitude of London—even though ten degrees is a huge range for anyone who knows that London is neither in the arctic nor the tropics. In other questions, of course, accuracy within 10% would be pretty impressive.

By: Cryptic Ned

Cryptic Ned — Fri, 27 Aug 2004 21:36:03 +0000

“Conversely, if you’re accidentally dead on, but overestimate your error, you’re hardly penalized, if at all. (That happened to me a couple of times.)” I don’t know how that could have happened to you, when I guessed “400 +/1 400” for the weight of the Boeing, which was off by less than 1%, and got 0 points.

By: Cryptic Ned

Cryptic Ned — Fri, 27 Aug 2004 21:34:24 +0000

42%, without knowing who Tony Benn is, and missing the astronomical questions by at least one order of magnitude, and having my entire knowledge of British history drawn from “Quicksilver” and “God’s Secretaries”. Accuracy on the geographical and populational questions, and a lucky guess on the number of UK counties, helped me out. Grrr…the answer is 396.9 tonnes…I say “400, plus/minus 400”, and get 0 points.

By: jam

jam — Fri, 27 Aug 2004 21:28:56 +0000

46%. Which I thought was pretty lousy ‘til I read these comments. The only one I’d really quibble on was the birth of Christ question. The answer Chris gave privileges the Matthian story over the Lucan. Since it’s likely both are pure invention, I don’t see the basis for his preference.

By: Xavier

Xavier — Fri, 27 Aug 2004 21:04:11 +0000

This would be a lot more interesting if it kept track of the answers people give. I would be curious to see what the range of guesses are on some of these questions.

By: Dan Simon

Dan Simon — Fri, 27 Aug 2004 20:02:13 +0000

What makes the quiz unsatisfying, I think, is that it attempts to combine two completely different skills: 1) Accumulating general quantitative knowledge, and applying it to answering quantitative questions about the physical world; and 2) Estimating the reliability of one’s own knowledge and/or reasoning skills. These are two completely different abilities, and it’s unclear how they’re weighted in the quiz score. For example, it appears that if you grossly overestimate an answer, but use a 100% error margin to indicate a recognized complete lack of knowledge, then you get zero—just as if you were supremely confident in your wrong answer. Conversely, if you’re accidentally dead on, but overestimate your error, you’re hardly penalized, if at all. (That happened to me a couple of times.) On the other hand, the test did make me think carefully about how to construct a good estimate. I started off by picking my best estimate and then an error bound, but as I progressed through the questions, I shifted to a strategy of thinking in terms of the extreme ranges of what I thought plausible, and using that as my guide. I think it improved my score, although I was occasionally insufficiently confident in my estimates. Oh, yes—for what it’s worth, I got 57%.

By: andrew cooke

andrew cooke — Fri, 27 Aug 2004 19:23:22 +0000

yeah, sorry.

By: Chris Lightfoot

Chris Lightfoot — Fri, 27 Aug 2004 18:35:20 +0000

Thanks—yes, that makes sense. Perhaps I should do version 2 of the quiz…. (You mean “root mean square” not geometric mean, btw—the geometric mean is sqrt(dxdX), and so is zero in the case of an exact answer—not desirable.) Interesting user-interface suggestion. I think the hypothetical version 2 could just ask for a percentage error.

By: Paula

Paula — Fri, 27 Aug 2004 18:25:32 +0000

35%. Appalling. I do, however, congratulate myself on knowing what a carrier bag is.