From the category archives:

Research

Or “I thought Science was a serious peer-reviewed publication…”

A study published today in Science by Facebook researchers using Facebook data claims to examine whether adult U.S. Facebook users engage with ideologically cross-cutting material on the site. My friend Christian Sandvig does an excellent job highlighting many of the problems of the piece and I encourage you to read his astute and well-referenced commentary. I want to highlight just one point here, a point that in and of itself should have stood out to reviewers at Science and should have been addressed before publication. It concerns the problematic sampling frame for the study and how little prominence it gets in the publication (i.e., none, it’s all in the supplemental materials).

Sampling is crucial to social science questions since biased samples can have serious implications for a study’s findings. In particular, it is extremely important that the sampling methodology be decoupled from the substantive questions of interest in the study. In this case, if you are examining engagement with political content, it is important that sampling not be based on anything related to users’ engagement with politics. However, that is precisely how sampling was done here. I elaborate below, but in sum, although the study boasts 10 million plus observations, only seen in the supplementary materials is the fact that only a tiny percentage (single digits) of Facebook users were eligible to make it into the sample in the first place. These are folks who explicitly identify their political affiliation on the site, i.e., people who probably have a different relationship to politics than the average user. They are also relatively active users based on another sampling decision, again, something confounded with the outcome of interest, i.e., engagement with political materials.

Not in the piece published in Science proper, but in the supplementary materials we find the following:

All Facebook users can self-report their political affiliation; 9% of U.S. users over 18 do. We mapped the top 500 political designations on a five-point, -2 (Very Liberal) to +2 (Very Conservative) ideological scale; those with no response or with responses such as “other” or “I don’t care” were not included. 46% of those who entered their political affiliation on their profiles had a response that could be mapped to this scale.

To recap, only 9% of FB users give information about their political affiliation in a way relevant here to sampling and 54% of those do so in a way that is not meaningful to determine their political affiliation. This means that only about 4% of FB users were eligible for the study. But it’s even less than that, because the user had to log in at least “4/7 days per week”, which “removes approximately 30% of users”.

Of course, every study has limitations. But sampling is too important here to be buried in supplementary materials. And the limitations of the sampling are too serious to warrant the following comment in the final paragraph of the paper:

we conclusively establish that on average in the context of Facebook, individual choices (2, 13, 15, 17) more than algorithms (3, 9) limit exposure to attitude-challenging content.

How can a sample that has not been established to be representative of Facebook users result in such a conclusive statement? And why does Science publish papers that make such claims without the necessary empirical evidence to back up the claims?

Can publications and researchers please stop being mesmerized by large numbers and go back to taking the fundamentals of social science seriously? In related news, I recently published a paper asking “Is Bigger Always Better? Potential Biases of Big Data Derived from Social Network Sites” that I recommend to folks working through and with big data in the social sciences.*

Full disclosure, some of my work has been funded by Facebook as well as Google and other corporations as well as foundations, details are available on my CV. Also, I’m friends with one of the authors of the study and very much value many of the contributions she has made to research.

[*] Regarding the piece on which I comment here, FB users not being nationally-representative is not an issue since the paper and its claims are only concerned with Facebook use.

This Is Only a Test

by Belle Waring on January 9, 2014

BREAKING MUST CREDIT BELLE BREAKING

What if I linked to my favorite song by everybody’s favorite Australian punk band, and merely pointed out that Ross “I Would Do Anything For Love But I Won’t” Douthat is only 8 1/2 days away from misusing found alien technology, not to better the world, because he’s a selfish monster, but to better his own life by becoming infinity percent cooler?

Ross-Douthat1

And look, he put out some great music, in his defense, right?

saints2

But that’s not at all what Culture tech is for and I can tell you, without spoilering the 21st Century at all, that both I myself and way MOAR KNIFE MISSLES get involved, so, it all evens out.

Are you going to go on and on about The Saints like a bunch of little bitches, or what? Not that there’s anything wrong with that! Oh, wait, now I’ve like entangled myself in my own quantum wossnames, haven’t I? COMMENT IN THE NEW THREAD. If you’re going to go on and on about something do it in an interesting fashion. Don’t feed boring trolls poorly-prepared victuals. Make the trolls earn their meagre fare. Then, eventually, you may reward them with fun-size Milky Way or something. God.

UPDATE: I failed. Because I could have written the post in a comprehensible way. But I did not. OR WAS I FAILED?!
1. Look, people have been going on about whether the Jim Crow South was maybe, possibly, kinda racist for like 425 comments down there in John’s thread. IT WAS. Also, DON’T FEED LOW CALIBER-TROLLS. “But Belle, Mao Cheng Ji started it!” STFU bitch.
2. People were not enough with the loving Breakwater’s “Say You Love Me Girl,” from below. Let’s grant they just don’t like that kind of thing, which was why they didn’t like it. OK GRANTED. Why didn’t they love Shuggie Otis more? Was it…racism? Oh, no, actually. Probably it was just liking this one other kind of music that they do like. Which is fine an all but…
3. As inestimable non-trollish commenter Michael Sullivan pointed out, if this had been a male poster reminiscing about some post-punk stuff he might well have gotten 100 comments in an hour. So, was it–sexism? No, actually, people just don’t appreciate good music sometimes. So then I was listening to The Saints. And then I noticed, damn, Ross “I Would Do Anything For Love But I Won’t” Douthat looks just like that one dude in The Saints! So I suggested an humorous vignette in which Meatloaf (not “Mr. Meatloaf”; the New York Times regrets the error; like many Indonesians, Meatloaf only uses one name) was transported back in time using the technology of Iain M. Banks’ Culture. But would that have been satisfactory? No, so I had to specify that I subsequently hunted him down and killed him with knife missiles which, if you haven’t read the books, are EXACTLY WHAT THEY SAY ON THE TIN. Then people expressed variously, bafflement (here, sorry) and the objection that The Saints, being a seminal punk band, can hardly be post-punk can they, missy? (EXPECT KNIFE MISSILES.)
I don’t know why you people make everything so difficult.

A recent study by the Washington University School of Medicine in St. Louis found that women using IUDs and other methods like under the skin implants or Depro-Provera injections were much less likely to have an accidental pregnancy than women using ordinary birth control pills, the trans-dermal patch, or the vaginal ring. (CT readers who are not up-to-the-minute on ladyissues may be interested to learn that the ring is a polymer, well, ring, which is inserted into the vagina, and then releases hormones over the course of three weeks. The birth control type is replaced after four weeks. Another version is used to treat the effects of menopause and has a different schedule.)

The women using the pill etc. were, in fact twenty times as likely to have an accidental pregnancy as the other group. “We know that IUDs and implants have very low failure rates — less than 1 percent,” says Brooke Winner, MD, a fourth-year resident at Barnes-Jewish Hospital and the study’s lead author. “But although IUDs are very effective and have been proven safe in women and adolescents, they only are chosen by 5.5 percent of women in the United States who use contraception.” In this case the study provided the various types of birth control at no cost. Worth noting, when the cost barrier was removed, the percentage of women choosing long-acting contraceptives went way up, to 75%.
[click to continue…]

Hey Look, Some Sexist Bullshit at Slate. No Wai!

by Belle Waring on January 16, 2012

Oi, this is so dumb and irritating that I pretty seriously considered not writing about it at all, in part because I worry the comments thread will develop a fetor of glib ev-psych nonsense. Uplift the human race, people, and surprise me with your intelligent thoughtfulness and concern for the feelings of other commenters. Who, I would like you to note, are actual human beings. Ya Rly!

Moving on, Slate has proffered for your attention an article by one Mark Regnerus, if that is his name. It is entitled thusly: “Sex is Cheap: Why Young Men Have the Upper Hand in Bed, Even When They’re Failing in Life.” I’m actually concerned that the stupid is going to burn my screen, and that readers of the article should perhaps be provided with an old-timey screensaver to avoid this. Flying toasters, say.
[click to continue…]

Ghostwriters of Science

by Henry Farrell on May 27, 2011

Via Randolph Fritz, a “very interesting article”:http://www.guardian.co.uk/science/2011/may/20/drug-companies-ghost-writing-journalism about how extensive the pharmaceuticals ghostwriting industry is:

bq. The planning companies are paid to implement high-impact publication strategies for specific drugs. They target the most influential academics to act as authors, draft the articles, and ensure that these include clearly-defined branding messages and appear in the most prestigious journals. … There are now at least 250 different companies engaged in the business of planning clinical publications for the pharmaceutical industry … Current Medical Directions, a medical communications company based in New York, promises to create “scientific content in support of our clients’ messages”. … n a flow-chart drawn up by Eric Crown, publications manager at Merck (the company that sold the controversial painkiller Vioxx), the determination of authorship appears as the fourth stage of the article preparation procedure. That is, only after company employees have presented clinical study data, discussed the findings, finalised “tactical plans” and identified where the article should be published. … “We’ve never done ghostwriting, per se, as I’d define it”, says John Romankiewicz, president of Scientific Therapeutics Information, the New Jersey firm that helped Merck promote Vioxx with a series of positive articles in medical journals. “We may have written a paper, but the people we work with have to have some input and approve it.”

I used to think that political scientists were lucky, in that no-one cared enough about what we had to say to try to suborn our reputations via dubious endorsements. And then I read about the “Gadaffi and political science scandal …”:http://www.guardian.co.uk/world/2011/mar/04/the-monitor-group-gadaffi-pr

Odds

by Belle Waring on October 24, 2010

I congratulate journalist Megan McArdle for having the good fortune to encounter such a talkative fellow passenger on the D.C. bus the other day.

Yesterday, I rode the bus for the first time from the stop near my house, and ended up chatting with a lifelong neighborhood resident who has just moved to Arizona, and was back visiting family. We talked about the vagaries of the city bus system, and then after a pause, he said, “You know, you may have heard us talking about you people, how we don’t want you here. A lot of people are saying you all are taking the city from us. Way I feel is, you don’t own a city.” He paused and looked around the admittedly somewhat seedy street corner. “Besides, look what we did with it. We had it for forty years, and look what we did with it!”

He’s a little off, because I think black control of Washington D.C. officially occurred only in 1975 when Parliament’s “Chocolate City” was released.

A fresh look at the left and right political blogospheres

by Eszter Hargittai on April 28, 2010

It’s exciting to see a paper about blogs across the political spectrum that goes beyond the by-now rather common practice of looking at who talks to whom among bloggers (e.g., whether there are any cross-ideological conversations going on). Yochai Benkler, Aaron Shaw and Victoria Stodden of Harvard’s Berkman Center for Internet & Society have just released “A Tale of Two Blogospheres: Discursive Practices on the Left and Right” showing some significant differences in types of blog platforms used (with different affordances), co-authorships and levels of participation among blogs of different political persuasions. Here is one example of specific findings (based on analyses of 155 top political blogs):

Over 40% of blogs on the left adopt platforms with enhanced user participation features. Only about 13% of blogs on the right do so. While there is substantial overlap, and comments of some level of visibility are used in the vast majority of blogs on both sides of the political divide, the left adopts enabling technologies that make user-generated diaries and blogs more central to the site to a significantly greater degree than does the right. (p. 22.)

There are lots of other interesting results in the paper so I highly recommend reading it [pdf].
It’s very clearly written and summarizes related literature well so in case this is not an area you’ve been following, this is a good piece with which to start to familiarize yourself with related debates. If it is an area that you’ve been following then this is a must-read to see some truly original contributions to the literature.

For more on this elsewhere, Ari Melber has an interview with Yochai Benkler on this research in The Nation.

How can schools use research?

by Harry on February 9, 2010

Madison School Board member Lucy Mathiak, with a lament that, presumably, all thinking school board officials in the US share:

For years, MMSD staff have advocated for their proposals and programming choices by arguing that they are research-based data driven best practices. At times, I have wondered whether the research selected has undergone critical review. That is, do the people selecting the research stop to ask whether the research is methodologically sound with verifiable results, much less whether it was conducted on populations or under conditions that are comparable to the Madison public school district.

I’ve also wondered at an understanding of research that ignores entire bodies of data or work that falls outside of the narrow educational research paradigm. (Prime examples of the latter case include the district’s unwillingness to consider the considerable body of research on how children learn to read that is carried out by cognitive psychologists, linguists, and communicative disorder researchers. But that’s another post.)

What follows is my longwinded response, which builds up to a plea for Districts (or groups of districts) and States to establish local versions of the Consortium on Chicago School Research.

Mathiak’s particular concern is that the only source concerning underrepresented minorities mentioned by name in a report on TAG developments is by Ruby Payne, who is not a researcher, and self-publishes. Whatever the merits of this particular instance of the worry, it is a shared worry for a reason. Educational research (broadly construed as it should be) is voluminous, to say the least, and even much of the best of it is not designed, or written, to be readily accessible to non-academics. Educational leaders, whether at the school or district level, are not trained in the consumption of educational research: in fact, they are not even presented with a great deal of it during their training, even for the purpose of learning what it says. Preparing them would be quite difficult, for a couple of reasons. First, education is beset by a culture of deference to ideological commitments, which makes it quite difficult to have some kinds of discussion in a way that is really sensitive to the evidence. Consider inclusion – the policy of including children with special educational needs in the regular classroom – which is, in some quarters, a matter of faith of such strength that evidence is really irrelevant. It is similarly difficult in some districts and schools to have an evidence-sensitive discussion of racial achievement gaps. When you do have the discussion, furthermore, it is not necessarily the discussion you think you are having! (The most unnerving conversation I had with a superintendent was one in which the superintendent told me that his district uses Ronald Ferguson’s work to design their policies around the racial achievement gap, which I would think was a pretty good idea had he not just told me, as truth, a whole bunch of claims that I had, the previous day, read a Ronald Ferguson essay disproving). Training leaders to conduct such discussions in these circumstances, in which some of them have, themselves, made the particular commitments of faith, is no easy task.

[click to continue…]

Two years ago, as part of a collection of articles researching social network site uses, I published a piece (blog post here) about the different predictors of Facebook and MySpace use among a diverse group of first-year college students. Some of the reactions to that paper suggested that the the differences by race/ethnicity and socioeconomic status identified in the data were only temporary and would soon change.

Change in Facebook and MySpace use by race/ethnicity among a group of college students, 2007-2009I now have some new data to consider possible changes over the past two years. I haven’t written this up in any formal way yet (nor do I have more elaborate statistical analyses to share right now), but I do have some figures suggesting that the differences I identified two years ago persist today.

Note that this is a new cohort of first-year students (i.e., not the same students resurveyed two years later) at the same universitywhere I conducted the study in 2007. (See details about the data collection and sample descriptives at the end of this post.)

Change in Facebook and MySpace use by parental education among a group of college students, 2007-2009There are two main findings here. (Click on the images for larger versions or see the table below.) First, there is a general increase in use of Facebook and a general decline in use of MySpace across the board. In 2007, 79% of the study participants were using Facebook while in 2009, 87% of the sample reports doing so. In contrast, while in 2007, 55% of the group reported using MySpace, in 2009, only 36% do so. [click to continue…]

ZOMG! Facebook use and student grades

by Eszter Hargittai on April 13, 2009

It started last night: links showing up on Twitter and elsewhere to articles about how Facebook users do worse in school. It’s not hard for people then to jump quickly to the conclusion that Facebook use results in worse grades (e.g., Study: Facebook Hurts Grades). Unfortunately, I know of no data set out there that could help us answer that question. The few people who have relevant data sets could establish correlation at best. I myself have not found such a connection in my data, but let’s back up a bit. [click to continue…]

The real world

by Eszter Hargittai on December 19, 2008

On Tuesday, I had the opportunity to go to DC and meet with some people on the Presidential Transition Team. I got to talk about my research on Internet uses and skills with people who seemed genuinely interested in what we know about this topic and how it might apply to future initiatives. It was an exciting experience.

It is great to see an administration again that cares about information technologies (see related comments in Obama’s weekly address from two weeks ago). However, it’s important to realize that achieving a knowledgeable Internet citizenry is not simply a technological problem and thus cannot be resolved by a solely technical solution. There is plenty of research now that shows how mere access to the Internet does not level the playing field when it comes to achieving universal Internet literacy. Rather, coupling technical access with education about uses is an important part of the puzzle. Of course, even if one accepts all this, solutions are far from obvious. I got lots of really good questions from the people in the room and was thrilled by the conversation.

Afterward, walking down the hall, I saw on the doors the names of lots of people who have been in the news recently. It’s wonderful and encouraging to see the number of smart and knowledgeable people on this team.

Do you remember the first time?

by Henry Farrell on July 25, 2008

Not the Pulp song; Siva Vaidhyanathan is looking for people to “tell him”:http://www.googlizationofeverything.com/2008/07/can_you_remember_your_first_ti.php about the first time that they used Google, if they can remember it. Personally, I can’t – there was a vague process of transition beginning with exclusive use of Alta Vista (remember that?) and finishing with exclusive use of Google, and I’m not sure what came in between.

Blogs, Participation and Polarization

by Henry Farrell on July 1, 2008

Eric Lawrence, John Sides and I have just finished writing a paper which looks at the first decent dataset that allows us to figure out what blog readers look like. This isn’t a final version (there are comments from Eszter and a couple of other readers that we want to incorporate – further comments and criticisms welcome), but it is just about fit for wider human consumption. The paper is “available at “:http://papers.ssrn.com/sol3/papers_LAB.cfm?abstract_id=1151490 SSRN (if you’re signed up with them, we’d love you to download it from there cos it’ll bump up our hit count), and at “http://www.themonkeycage.org/blogpaper.pdf”:http://www.themonkeycage.org/blogpaper.pdf if that’s more convenient. So what do we find?
[click to continue…]

Gender differences in sharing creative content online

by Eszter Hargittai on June 25, 2008

This ArsTechnica write-up of some recent research of mine has received numerous votes on the recommendation site Digg in the last few hours. I wonder if it will make the front page of Digg, although as a Twitter contact of mine noted, since it’s not a top-10 list (nor, if I might add, does it cover Google or Apple), that may be unlikely.

The post reports on a study in which we found that male college students are more likely than their female counterparts to share creative content online even though both men and women in the sample are equally likely to create such content. However, when controlling for online skill, the gender differences in posting go away.

Gina Walejko and I published the paper “The Participation Divide: Content Creation and Sharing in the Digital Age” this Spring in the journal Information, Communication and Society. We examine the extent to which college students share creative content online and whether we can identify any systematic differences by user background. In particular, we looked at whether students create and share the following types of material: poetry/fiction, artistic photography, music, and video (both completely own and remixed in the case of the latter two), including both private and public sharing. [click to continue…]

Mapping the political blogosphere

by Henry Farrell on June 23, 2008

via “Ethan Zuckerman”:http://www.ethanzuckerman.com/blog/2008/06/23/pdf-visualizing-the-political-blogosphere/, a really nice visualization (with clickable information) of the political blogosphere “here”:http://presidentialwatch08.com/index.php/map/.