Estimation

I don’t exactly know what carrier bags are (grocery bags?). And I think I should get points if I guess English History dates within a hundred years (Would it have made a difference if I had widened my MOE?) And who is Tony Benn and should I feel stupid for not knowing? A disappointing 34%

Surely everyone knows there’s 50 states in the US?!? Or that the union of England and Scotland was in 1707? 51% here.

Is 20 percent of the UK adult population really functionally illiterate?

I had guessed Pride and Prejudice at roughly twice its actual length. Reminds me of the review of the movie The Hours. “Hours? It seemed like days.”

Otherwise 44 – not bad for an American taking such an Anglocentric quiz…

I was helped by hoping that Tony Benn was a misspelling of Tony Bennett, since they were born a year apart.

I don’t object to questions that reveal my ignorance of geography, literature, and history, but a question depending on the Australian meaning of “carrier bag” is just silly. Would a poor swagman use one as a tucker-bag?

The scoring presumably is larger when the exact answer is in the answer interval, and smaller when the relative width of the interval is large. It would be nice to know.

I assume the “Wisdom of Crowds” would reveal something here.

i got 10 points on the number of stars in the galaxy, but was off an order of magnitude on the length of the river nile and number of carrier bags in austrailia. so sad

then i guessed 1100 +- 200 for the union of scotland and england, and pretty much bombed every other UK question other than latitude

but I did get the magna carta, un, all the science questions right (7+), so I can live in pleasant ignorance, not knowing what I’d receive if the questions were not so UK centric

I can’t believe the UK has 650 MPs but only 12,000 gas stations. That’s a remarkable contrast to the US.

the trouble with the scoring is that if you have no idea what a number is, you are often estimating the magnitude. in such cases, it makes more sense to describe the error in the log of the value (or the ratio). so you might let the user enter either a linear error (as you have now) or a factor (within a factor of 2, say).

also, your scoring doesn’t seem to account for the error estimates entered. or maybe i don’t understand what you’re trying to score. if i were you, i would not try to score each answer, but instead calculate the normalised error of each estimate. then score how well that is distributed.

for example, if i estimate x+/-dx and the real value is X, then my normalised error is (x-X)/dx (use logs if ratios are specified). now, if i’m good at estimating, the (x-X)/dx for all the answers, taken as a population, should have zero mean and unit error. you can give a score based on some test for that.

wow, doesn’t posting take a long time? anyways, forgot to add, you probably want the score to be based on the log of the probability that the distribution is as described, otherwise we’re all going to be scoring near-zero.

Gosh, I thought my 23% was good, given that I’m a USAian and many but the most general UK-centric questions gave me much trouble.

But compared to the scores above, I’m pathetic. Darnit. A couple I got wrong but really shouldn’t have. I know, dammit, Earth to Moon is aprox quarter-million miles, but didn’t remember to think about it that way and so estimated half that. And I typed in 5,000,000 for words in Austen when I meant 500,000—still wrong, but not, you know, insanely wrong. :)

>>Surely everyone knows there’s 50 states in the US?!?

Actually it could be construed as a trick question. Only 46 of the states are actually “States” – Maassachusetts, Pennsylvania, Virginia, and Kentucky are officially “Commonwealths” – the safest answer would have been 50 +/- 4.

31%, I completely miffed many of the UK-centric questions and I wasn’t sure what a carrier bag is.

There really should be an ‘I have know idea what you are talking about’ catagory for a couple of questions. I can’t estimate things that aren’t even in my realm of knowledge at all. (Though it did give me some points for my random-wild-ass-guess on Tony Benn.)

I got 52% and was low because of values I thought I knew, and consequently put a small +/- figure on – like I’ve always “known” that light takes 7.5 minutes to reach Earth from the Sun, London’s 50.5 degrees north. Zero points each. Bah.

Chris: this is how I estimated the number of carrier bags used in Australia.

Population of Australia: 20 million.
Each person buys something needing a bag 6 days a week, or 300 times a year.
300 * 20 million = 6000 million.
Add a suitable fudge factor for humility, and there you go.

I got the gas station question wrong, though, because I guessed that cars can go about 10 days between refuelings, and was off by a factor of 2 as a result. That’s what I get for not owning a car. :)

I’m not certain how you incorporate information from the real error in the answer in that model, though.

hi. if both values have error estimates then you normalize by the geometric mean. so if i estimate x+/-dx for a value “known” to be X+/-dX then the normalised difference is (x-X)/sqrt(dx*dx+dX*dX) (it’s all classical stats assuming normal distributions etc etc; i’m not sure how you’d go about combining a log-based answer with a linear target, though – probably move the linear target to logs and work there).

i agree that adding factors complicates the user interface. what you might do, although it’s hardly intuitive to the user, is switch to factors if the error is more than half the value. for example, say one of my answers was 500+/1000. you might take that as 500 within a factor of 3 (using the upper bound, since the lower bound is -ve…).

(but we all know the only thing that really matters is getting noticed, provoking debate etc; these are just nerdy details ;o)

Thanks — yes, that makes sense. Perhaps I should do version 2 of the quiz….

(You mean “root mean square” not geometric mean, btw — the geometric mean is sqrt(dxdX), and so is zero in the case of an exact answer — not desirable.)

Interesting user-interface suggestion. I think the hypothetical version 2 could just ask for a percentage error.

What makes the quiz unsatisfying, I think, is that it attempts to combine two completely different skills:

1) Accumulating general quantitative knowledge, and applying it to answering quantitative questions about the physical world; and

2) Estimating the reliability of one’s own knowledge and/or reasoning skills.

These are two completely different abilities, and it’s unclear how they’re weighted in the quiz score. For example, it appears that if you grossly overestimate an answer, but use a 100% error margin to indicate a recognized complete lack of knowledge, then you get zero–just as if you were supremely confident in your wrong answer. Conversely, if you’re accidentally dead on, but overestimate your error, you’re hardly penalized, if at all. (That happened to me a couple of times.)

On the other hand, the test did make me think carefully about how to construct a good estimate. I started off by picking my best estimate and then an error bound, but as I progressed through the questions, I shifted to a strategy of thinking in terms of the extreme ranges of what I thought plausible, and using that as my guide. I think it improved my score, although I was occasionally insufficiently confident in my estimates.

Oh, yes–for what it’s worth, I got 57%.

46%. Which I thought was pretty lousy ’til I read these comments.

The only one I’d really quibble on was the birth of Christ question. The answer Chris gave privileges the Matthian story over the Lucan. Since it’s likely both are pure invention, I don’t see the basis for his preference.

“Conversely, if you’re accidentally dead on, but overestimate your error, you’re hardly penalized, if at all. (That happened to me a couple of times.)”

I don’t know how that could have happened to you, when I guessed “400 +/1 400” for the weight of the Boeing, which was off by less than 1%, and got 0 points.

From the brief blurb “how well you know what you don’t know” I thought that some of the scoring would be on how well you choose margins of error-in-guessing. That is, no matter how large the error, or random the guess, if the respondent is at least aware of how wide to make the confidence interval, that would count relatively more than being right with the guess. Not sure exactly how that would be done, but it ain’t done here. For the Tony Benn question I guessed 1925+/-10 and the answer was 1925 exactly, and you took off a point, as though you think I thought his mother’s labor lasted about ten years when it was mere hours. In other words, your margins appear to be inherent in our current ability to measure the quantity, not in the test-takers awareness of the vagueness of their recollections.

Orders of magnitude also should count for more, esp. with the numbers that aren’t dates. Getting within a factor of ten is good enough for a Friday night not cheating, while getting within a factor of two is pretty good for a casually informed guess. Getting within one’s own margin of error, close to the center of it, is good regardless of one’s knowledge of the facts, but narrower ranges are good too. Perhaps these are best brought out with different dimensions of scoring – or maybe you don’t really care about how well we estimate our own uncertainties, but then you might consider changing the cover blurb.

I’m pleased with 36% though, as a merkin.

I was puzzled by working out the error bands for the UK asylum seekers benefit. I vaguely recalled reading that it was about £40 a week, so that was my best guess. But, when it came to error bands, obviously asylum benefits can vary upwards without limit and without breaking the laws of arithmetic, but if the UK started charging asylum seekers for staying in the country it’d be called a tax or a fee, not a benefit. Which makes for a lop-sided error band, but you can’t specify that in the question. So I gave a wide band and got a 0, despite being very close to the right answer.

Anyway, 37%. Not helped by me thinking of things in metric and then forgetting to convert. (700 k is roughly the distance between major NZ cities, it would have been a better guess if I’d turned that into miles)

Incidentally, the Magna Carta was signed several times by various kings of England. To be pedantic, the question should be specified as when it was first signed.

Tracy

_“Conversely, if you’re accidentally dead on, but overestimate your error, you’re hardly penalized, if at all. (That happened to me a couple of times.)”_

_I don’t know how that could have happened to you, when I guessed “400 +/1 400” for the weight of the Boeing, which was off by less than 1%, and got 0 points._

I answered 4 +/- 130 BCE for Jesus’s birth, fully expecting to be penalized for being a smartass, but got full marks. On the other hand, 800 +/- 700 billion stars (another “too correct for the quiz” answer I was proud of) scored nil. In general, it’s awkward to mix intrinsic errors with measurement uncertainties, since they are statistically independent and therefore add in quadrature. Perhaps questions like these that don’t have real answers (within an order of magnitude) should be stricken from the next iteration.