Why doesn’t Science publish important methods info prominently?

“This means that only about 4% of FB users were eligible for the study. But it’s even less than that, because the user had to log in at least ‘4/7 days per week’, which ‘removes approximately 30% of users’.”

Let me guess that people with strong political opinions may partake of Facebook less than those with less conviction, so that the study was left with an even more skewed subset of FB users.

I’m not entirely sure what your central point is here. If it’s that the paper seems to have slipped potential flaws past the reviewers on the way to publication – well, it happens, and a manuscript that more openly admitted these flaws probably wouldn’t have caught the reviewers or editor napping, so while your complaint is valid it’s also a bit circular. I think the more interesting and open questions have to do with the use of online supplements for the Methods.

Such use of the “online supplementary information” as a place to put some or even all of the Methods and a lot of the data has been an issue since this became possible with widespread use of the web by journals (I’d place this about 15 years ago; the first biological journal to have full text online started 20 years ago). Indeed, this capability is often abused – and by the journals as much as by the authors, with Science and its mirror image Nature being the worst offenders. Because these are short-format journals who publish high-impact papers often reflecting vast amounts of work, they’ve been delighted that they can shift in some cases all of the Methods information online.

That said, the situation is complicated, and this is not necessarily a bad development. The supplementary online material is often open-access even when the article itself isn’t (basically to make it easier on people who have gotten a copy of the article by legitimate means but don’t want to be hassled while checking the online supplement), and I’m not sure you understand just how egregious Nature and especially Science were in the days before the online supplement. Nature at that time did allow a section of the manuscript for Methods, carved out from within the already stingy space limitations on a Nature letter. Science just didn’t. Notoriously, the Methods for a Science paper were often found in the endnotes, mixed in among the references, and published in tiny print. For both of these short-format journals, the extreme length limitations made for terribly composed, ludicrously unreadable papers: often great science but unjustly crammed into a few thousand words.

The ability to give unlimited space to the Methods and various fine points of the work while shifting them out of the actual article and into an online supplement has made the papers more readable and also made the methods and other supplemental material vastly more readable. There is I think a real, if not wholly convincing, argument to be made that the casually interested don’t need to double-check the methods but can read the paper trusting peer review has done its job, while those sufficiently engaged that they actually want to ensure every t has been crossed and every i dotted will have the motivation to go the extra step and look online, where they will find a more complete presentation than could ever have fit in the article.

Where this become particularly abusive is that people are now expected to publish vat amounts of data in the supplement; often what would be entire additional papers are contained there. And it’s not necessarily even material directly related to the manuscript at hand – I know of one case offhand in which an anonymous peer reviewer successfully demanded the authors of a manuscript under review at a prestigious journal add to the online supplement a set of experiments attempting (and failing) to replicate the key findings of a paper published by another group, in the same field but in no way closely connected to the manuscript in question. That’s an extreme example, but it’s now common for reviewers to make extreme requests, knowing it can all go in the supplement.

So: I’ve been longwinded as I often am, but I think there are important issues with the u of online supplements, but that they really aren’t the issues you raise.

Sampling on the dependent variable is crappy social science, whether you state the original sample (note: not population) on the first page or not, or whether it’s in the supplementary appendix or not. It’s something every first year grad student in sociology, demography, and political science learns, or at least should.

Dean — yes, this isn’t selecting on the dependent variable, but it also is not just non-random sampling, but non-random sampling with a very plausibly very strong sampling bias, which isn’t acknowledged anywhere in the paper. Meanwhile, as Eszter points out, the paper does make very strong claims about the entire population of US Facebook users.

Moreover, while you’re correct that the size and the basic methodology of the sample are stated, specifics of the sampling methodology are not–but that’s what’s really crucial for understanding to what degree we should believe any statement that goes beyond the 10million people actually covered by the study. In other words, this information is crucial for understanding whether the central claims of the study are true. The fact that that information isn’t included in the study is, indeed, very puzzling. I’d expect this to be covered even in a good journalistic summary of it, but most certainly in the study itself.
Do you really not find that problematic? You’re clearly a much more accomplished statistician than I am, but all the serious statistician’s I’ve met are obsessively cautious about such things, so I’m genuinely flabbergasted by what seems like a pretty callous attitude on your part to basic problems with this paper.

@Warren Terra — I take Ezter’s point to me that we’re seeing the modern day re-occurrence of the 1936 Literary Digest election poll that’s prominently covered in (almost) all statistics courses: People get excited by large sample sizes, neglecting the quality of the sample.

@#8
Sure, but this sort of lapse in peer review has nothing to do with the use of online supplements. If anything, the supplements make more complete inspection much more possible for the peer reviewers and for sufficiently interested readers. And yet the title and the first bit of the post is all about the use of online supplements for Methods.

Not a social scientist, never so much as took a statistics class. But based on Eszter’s OP, and adam.smith’s and Bloix’s comments especially, I’m sitting here wondering: Are the authors such incompetent social scientists that they didn’t know that their conclusions and the basis for reaching them were seriously flawed? Or did they think no one would notice?

Of course, there are never only two possibilities. I’ve been having recent personal experience of the way the tail (sales and marketing) increasingly wags the dog (the actually grunty work that the rest of us have to do) in the business world. And with a behemoth like Facebook, I suppose there’s a strong bubble effect as well.

@Freddie — I am an active researcher and I am very confident that nothing like this would be published in a Top 10 journal in my field (political science) without a strong caveat about external validity and likely some attempt to justify the sample as being reasonably representative, if possible. Sure, you can’t random sample for everything, but if you don’t, you can’t go out and make statements about some larger population without significant qualifications. I’ve been aghast by some of the quantitative stuff I’ve seen published in education, so if that’s what you’re talking about, that’s possible, but let’s not overstate how bad things are. Most researchers are careful in not overstating their findings.

JanieM — the folks who published this are very capable. They’re aware of this. I’m actually quite curious why they decided to downplay it to the extent they did.

That should be easy to settle:
If you’re able to cite 5 recent journal articles from a field for which the following three things are true, I’ll agree you’re right for that discipline:
1. Published in a top journal (depending on size of the discipline top 5 or 10)
2. Using a convenience sample of some type to
3. Make claims about a larger population without a discussion about generalizability of the sample to said population in the paper itself.

I’m making a basic point about populist critiques of research methodology.

Then make it.

Freddie,
what you started out with here was:

these comments demonstrate the common condition of people who aren’t researchers radically overestimating what common research standards of publication are.

which I hope you realize is super condescending and also rather ridiculous given that a) a lot of the people here are active researchers and b) Eszter, who wrote the original post, is one of the pre-eminent researchers working on the effects of social media.

As for Twitter, in my feed I see a lot from Zeynep Tufekci, who is also a widely published and respected researcher on social media and the public. So, as someone with no relevant research experience and really not terribly impressive academic credentials to come around and explain to the plebes how real research works is pretty rich.

“which I hope you realize is super condescending and also rather ridiculous given that a) a lot of the people here are active researchers and b) Eszter, who wrote the original post, is one of the pre-eminent researchers working on the effects of social media.”

I said nothing whatsoever about Eszter or her credentials.

“As for Twitter, in my feed I see a lot from Zeynep Tufekci, who is also a widely published and respected researcher on social media and the public. So, as someone with no relevant research experience and really not terribly impressive academic credentials to come around and explain to the plebes how real research works is pretty rich.”

1. That’s not what I did. That’s not remotely what I did. And it’s very telling that you’ve projected that baggage into an innocuous and not-at-all-insulting expression of mildly dissenting opinion. (From my first comment, literally: “I don’t disagree….”)

2. Literally every argument of yours here is purely asserted arguments to your own expertise, which has involved you claiming to be a big shot, making fun of other people’s fields, and grandstanding about what a big deal you are.

3. All of this is straight from Crooked Timber’s playbook: fake a kind of milquetoast warm milk liberal egalitarianism, but the second you get alternate opinion on the slightest issue, start talking about your Ivy League credentials. I don’t play that. I’m not impressed by you, your anonymity, your asserted credentials, credentialism in general, your totally uninformed take on me or my abilities, your cloying condescension, or your very conservative endorsement of straight-up meritocracy mythology. If you are really this hurt by alternate opinion, it’s hard to understand how you function in the world at all.

Keep putting the “less” in bloodless, Crooked Timber. The next time you guys like to posture about your lefty cred, remember that this is a place that lives and dies based on who is more shameless in flogging their degrees.

That was, indeed, the mild point I was trying to make: that when the flaws of any given study are suddenly thrust into public conversation, they tend to get compared to an idealized standard of perfectly random samples that aren’t found in real research.

But unlike Adam Smith, I didn’t go to Yale College Harvard, so we can feel free to disregard my opinion.

I see The Internet’s Most Self-Important Commenting Section hasn’t lost its edge in recent years.

Coming from you Freddie, that is very, very funny. So much so that I’d almost suspect you of a delightfully self-deprecating sense of humor, if I didn’t know better.

Hey, it’s Henry “I’m a bigger deal than you” Farrell, here to enforce that conformity I was just talking about!

that when the flaws of any given study are suddenly thrust into public conversation, they tend to get compared to an idealized standard of perfectly random samples that aren’t found in real research.

That overlapped and to bring this back on topic, I tend to disagree with this on both an empirical and a normative level.
Empirically, I continue to maintain that it is _not_ in fact common practice in the social sciences to draw unwarranted inferences from non-random samples. That doesn’t mean I’m claiming that research looks like some type of textbook ideal–it certainly doesn’t–but I would argue that most published research does in fact not have flaws that anyone who has taken Stats 101 can easily see. I am genuinely surprised you disagree with that, so I was asking for examples.

On a normative level, I think it is perfectly OK to put studies with major implications under closer scrutiny, especially when they’re published in one of the most visible science publications anywhere. And given the impact of Facebook algorithms on the daily life of many people, this study would certainly qualify. So even if similar criticism do apply to other studies, I don’t think there’s any downside to highlighting them in publicly visible research.

This gives me the opportunity to report that I just held my statistics final exam and the results were as always slightly depressing (I have only myself to blame though – I should “lower my expectations” as older colleagues told me right away). I’m not teaching students likely to engage in a scientific career but I think the pattern is pretty widespread – most textbooks discuss hardly anything outside the bare cookbook recipes and students have learned for so many years to disregard context and focus on doing computations, it’s an uphill battle to convince them that understanding what they are doing and how to interpret it is far more important than following the recipes. The issue highlighted in the OP should have jumped out to researchers and reviewer but reviewers are easy to satisfy as long as the formalities are checked and the p-values are there.

Oh, great, a back-and-forth about Freddie, and his complaints, rather than anything to do with the various actual subjects under discussion. Quelle surprise.

Yeah, it’s crazy how, when I make an anodyne point that began “I don’t disagree,” but which might, if you squint the right way, amount to something like dissent, and this blog’s amoeba of comity fetishists lay into me with straight credentialism and big-timing, I might say, “hey, that’s lame.” I must be some kind of kook!

Oh, great, a back-and-forth about Freddie, and his complaints, rather than anything to do with the various actual subjects under discussion. Quelle surprise.

I was going to suggest that people stop engaging, on the grounds that Freddie deBoer considers any conversation that is not already all about Freddie deBoer (or perhaps, at a pinch, the sinuous complexities of Chairman deBoer Thought) to be definitionally an outrage. Happy to have been anticipated.

Freddie – please stop being a pest. This thread isn’t about you and your pet peeves about the Crooked Timber comment section. I warmly encourage you to write an extremely long post over at your own blog about the self-satisfied liberal commentariat, the horrid way in which people treat you, how I’m always telling you how learned and all-knowing I am, and how you and you alone truly appreciate and understand what it means to still be leftwing in these days of tribulation. But you’re making yourself a bit of a nuisance here, and getting in the way of a conversation that other people would like to have, a conversation which is manifestly Not About Freddie. I suggest you desist.

Freddie, I like your blog a lot, but you’re way out here. It’s true that populist critiques of sampling are often way out (I actually coined the phrase “Devastating Critique” on this very blog to describe “a critique of a piece of statistical work based on points made in the methodology section of the actual paper”). But this is not the sort of “close-to-the-line” case that John McEnroe used to get angry about. It’s a self-selected sample, specifically self-selected because of their political identification, used to generalise about all Facebook users, when the generalisation is “Facebook users do X, Y and Z based on their political identification”. You’ve gone off at half-cock here – this would be a valid point in a lot of contexts but actually in this case Eszter is dead right.

I’m not sure why I should need to add this, but it’s trivially easy to disagree with Ezster’s claims and say you think they’re false without implying that she is wrong because she is not a researcher/doesn’t know enough about the literature/is taking it the wrong way/is too sensitive/maybe people chose the shirts themselves, did you ever think of that missy?/that’s not how computers work/maybe fewer women than men submitted papers for the conference, did you ever think of that, missy?/etc., etc., …

Sorry, I hadn’t seen Belle’s comment. As usual she says everything in a much more interesting way. Now to mark my papers and listen to Alabama Shakes.

40: Why should the merit and relevance of research be measured by how much money it cost to produce, rather than by its scientific value? Oh wait… are you saying …?

None of the comments that disagree or downplay Eszter’s criticism argue that she is mistaken when she says that the sample is non-representative of the population and therefore the conclusions are worthless. Some people seem to be saying, Don’t you understand?? ALL published Social Science research is not worth the glossy paper is printed on!!!!

Is that really the argument? That Eszter is a fool for assuming that the point of publication is to disseminate knowledge, and not merely to create a c.v. that will get someone tenure?

Having briefly looked at the study, it appears both flawed and relatively banal. People’s facebook friends tend to share their outlook? How surprising, rush that to Science Magazine. Except that one wonders why moderates have so few moderate friends. Even a casual reader at that point should wonder about sampling issues (interestingly, the article doesn’t give an ideological breakdown of their sample).

I do think that social science is underrepresented in Science and when something does get published, it is often faddish like this (internet! social media! network graphs!!!)

I’m not sure where anyone got the idea that Science is a serious peer-reviewed publication. It’s an academic ambulance-chaser. It’s more important that the results be sensational than that they be right. Speaking as a professional statistician, the standards of statistics in Science are generally abysmal, and they have no interest in correcting the methodological errors.

> Or is there a more subtle point here that I am missing?

Confused language reflects confused thoughts. If someone’s thoughts are a muddle, then an accurate reduction of their thoughts into words would also be a muddle: if the words were to convey any coherent meaning they would cease to reflect what the speaker thought.

So I read the post, thought “yeah that’s pretty bad, they should be on the lookout for things like this considering they’re a top journal”, saw FdB’s first comment and thought “yeah, that’s a good point, people tend to reference everything against these ideals that are increasingly passe in the corrupted juggernaut we call academia, while not ceasing to point out bad practices we should keep perspective about it – probably a more useful approach in the long term to fixing these problems”, and then wondered how on Earth things devolved so quickly.

There must be some sort of Dave Graeber-esque bad blood here, right? Maybe I don’t read the relevant radicals: I know CT is liberal and not, say, anarchist, but that doesn’t stop me from reading it any more than it stops me from checking out Krugman’s arguments against austerity. I haven’t heard of a liberal Politburo here, just horrible reactionary trolls like Brett Bellmore.

OK, going back to an extended thought I had when reading FdB’s first comment: first, is it possible to reform academia at this point? The publish or perish syndrome, the increasing risk-averse bureaucratization, the specter of bad practices (especially statistical ones) hiding behind buzzword shields like Big Data, the Bayesian arguments about the literature being full of false positives – do these mark the final stages of modern academia as we understand it, are they a symptom of the wider problems of society or are they simply unrelated problems with discrete, plausible solutions? If the latter, and zooming in to the problem at issue, what’s the best way to go about fixing it?

Should we tacitly pretend that there is elsewhere the Statistics Ideal being practiced (if even by omission of a discussion of the state of the average publication etc when it comes to this stuff) when taking Big Shiny Papers out behind the woodshed, for maximum effect and critical attention? Or would it be better as a whole to keep emphasizing the fact that this is really a terrible problem affecting tons of papers basically everywhere, and that almost every journal is guilty to some degree, and Twitter storming about a particular paper is not as effective as, say, getting some sort of organized Retraction Watch-style group to shed light on the issue? I’m honestly not sure.

Thanks Eszter for the thought-gunpowder.

Thomas Leeper’s post about responses to the study, including this one, may be of interest.

He says, in response to this post, regarding the study’s generalization to the larger Facebook population, “In this case, it is not obvious that the authors intend to generalize to other groups – maybe they do, maybe they don’t, they’re not really clear on the question.” That is, he doesn’t seem to notice, or maybe care, that the sample conditions were described only in the supplementary material, which was a particular point here.