Comments on: Nice results

By: Crooked Timber » » People’s Web-savvy (or lack thereof)

Crooked Timber » » People’s Web-savvy (or lack thereof) — Thu, 21 Jul 2005 13:02:11 +0000

[...] Regular readers of CT may recall that all of this is close to my interests as an important aspect of my work is looking at people’s Internet skills. My paper examining proxy measures of actual skill is coming out this Fall. (I’ve mentioned it here before.) In it I show that the types of knowledge items on which the Pew researchers just collected data are better predictors of people’s actual skill than traditional proxies such as amount of Internet experience or even self-perceived skill (a very common proxy in the literature). [...]

By: eszter

eszter — Mon, 27 Sep 2004 16:00:41 +0000

Dsquared – see above. But yes, it seems like that would certainly be one way to approach it.

By: dsquared

dsquared — Mon, 27 Sep 2004 13:40:14 +0000

I love pontificating about social sciences research … while watching my son at playgroup, I jotted down a few notes. My guess is that you did a PCA of the scores on the skills tests and took the first principal component as a measure of skill. Then you carried out a regression of the survey questions on the skill measure to get a set of weights to apply to the questions, giving you a weighted average of the survey questions which would be an instrument for the skill score. Or at least that’s what I think I would have done. Am I right?

By: eszter

eszter — Mon, 27 Sep 2004 06:17:13 +0000

PG - I just gave people tasks to perform, I didn’t decide ahead of time which ones were more difficult than others. The tasks were usually general enough that you could find relevant information on all sorts of Web sites. There was never just one correct response. More details are in the paper(s). David – Your comments remind me of something I’ll want to blog separately another time.. how people with different abilities draw the line of what counts as skilled computer knowledge. But my study focused on average random users so I didn’t end up with too many IT professional types who would classify related knowledge on a whole other spectrum. I’d be happy to post the survey here sometime.. but that would require approval from my Institutional Review Board for Human Subjects research.. a pretty tedious process. (I guess if I wasn’t going to use the data for research purposes it might not, but if I’m going to collect data, I’d like to have that option available.) Stay tuned.:)

By: David Tiley

David Tiley — Mon, 27 Sep 2004 01:56:57 +0000

I want sociology geekdom. I want historian geekdom too. They hurt my brain less than philosophy and I am sure you have a heap of readers like this. I would have been amaaazed if there was much correlation between self-assessed skills in any areas of computers and shared datum lines of ability. For a start, we have particular areas of expertise (applications) where higher levels of ability seem simple, and a nodding acquaintanceship with others where we seem by comparison to be inept. That is, we are attuned to small ability differences which are magnified. And our own geek subcultures accept levels of skill as normal which seems magical to outsiders. There is also a tendency, across the broad computer domain, for a significant tribe of people to simply not understand that other areas of expertise exist at all. Ask your IT folk about interface design. If you wanted to post the tests, we could all have a go. I would be fascinated. Have I learnt anything from these late night skitterings across the world’s databases? I don”t know.

By: PG

PG — Sun, 26 Sep 2004 20:47:11 +0000

Were there certain searches that you categorized as more difficult than others? for example, was finding the fulltext of a NYT article understood as a greater accomplishment than finding a fulltext Washington Post piece?

By: eszter

eszter — Sun, 26 Sep 2004 20:06:37 +0000

I didn’t simply run a PCA on all the survey measures to figure out what variables to include in the index, if that’s what you’re asking. There were some other issues I wanted to consider while constructing the index variable.

By: dsquared

dsquared — Sun, 26 Sep 2004 19:17:16 +0000

But just in general terms, are we talking about a principal component here?

By: eszter

eszter — Sun, 26 Sep 2004 18:58:54 +0000

Thanks. The paper that describes all of the above in detail is not available online because I am about to send it off for review (and I don't post things online at that stage). I have a paper out about the details of how the data were collected (i.e. recruitment, technical specifications, types of questions asked). I have another paper out about how I coded and classified users' online actions. That's less directly relevant although it does show how I have the data by second on every action, which is relevant given that one of my skill measures is time-to-completion of task.

By: dsquared

dsquared — Sun, 26 Sep 2004 18:47:39 +0000

Which is the specific paper where the methodology is described? It all sounds very interesting.