I don’t know much about “grievance studies”, but I do know quite a lot about fraud, having written a book on the subject and spent two years researching it (and now three months more researching some additional bits for the US edition, out in 2019). So just a further observation after Henry’s post on the subject – one thing that I think is underappreciated in a variety of contexts is that the susceptibility of a system to intentional deceit is not by any means a good indicator of the underlying health of that system. In fact, I would even go so far as to say that the equilibrium level of fraud is determined entirely separately from the equilibrium level of quality and output, and that therefore the ease with which X can be faked is irrelevant to your assessment of whether there are severe problems with X.
In my book I call it “the Canadian Paradox”, after the well-known possibly fictional North American country which has the surprising combination of a high-trust society, a strong economy, an equity investment culture and an absolute rats’ nest of securities fraud. High trust economies (and high trust systems in general) are usually very vulnerable to fraud, precisely because trust and checking are substitutes for one another. The opposite pole of the spectrum would be somewhere like Greece, which has quite low levels of fraud against parties other than the government[1], but achieves this low-fraud equilibrium by having an economy where close circles of personal trust are prevalent and economic activity outside these circles is reduced. The main cost of fraud to the economy is not the amount stolen, nor even the amount of money wasted on checking and auditing activities; it’s the amount of business that never gets done at all.
I could even extend this point to say that not just the vulnerability but the actual prevalence of fraud is not closely associated with the health of the system. We have low-fraud-low-growth equilibria like Greece, but there are also high-fraud-high-growth equilibria. You get epidemics of mining frauds during real gold rushes, for example. In the mid 19th century, roughly one sixth of all companies floated on the London Stock Exchange were frauds, but the railways did get built.
What’s going on here is that fraud is, informationally, something that comes from outside the system (or outside its domain of feedback and control). If you try to bring it into the system and optimise for the combination of costs and fraud losses, you tend to get something like Medicare in the 1990s, where at the peak fraudulent claims reached as much as 30% of the total expenses. That’s because frauds are designed around the set of controls, to exploit the fact that it’s not possible to check on everything and that every decision on what to check is, implicitly, also a decision on what not to check. All that a fraud demonstrates is that the gatekeeping function is not perfect.
And in the case of academic publishing, it’s hard to see why anyone should expect peer review – a system whereby people contribute uncompensated labour to a multinational company in order to create output that they will then be charged monopolistic prices for – to be done well. It’s a miracle that the system works as well as it does. And the function of peer review is to produce quality-controlled academic output, not to spot fakes. Even the definition of what constitutes quality control in context is not obvious to me.
When a venture capitalist funds a hundred startups, in the knowledge that ten of them will be frauds and eighty of them poor-quality flakes, but the hope that nine of the remaining ten will pay their way and the final one turn into Facebook, we understand what they’re doing. We also realise that the main risk of that investment strategy isn’t that you get twenty rather than ten frauds, it’s that you miss the one-in-a-hundred unicorn. Different academic disciplines are going to take different risk/reward tradeoffs; one wouldn’t want to see a journal of hepatology taking a speculative moonshot approach, but in fields that are addressing controversial areas where nobody really understands the questions let alone the answers, move fast and break things might be a better approach. How many lesser oddballs and lobster-fondlers did the anti-PC culture warriors have to sift through before they found Jordan Peterson, after all?
[1] The government is a special kind of victim for crimes of dishonesty, because it’s an entity which (by definition) can’t be incorporated into a close personal trust network, but which (also by definition) has to be involved in the economic life of nearly everybody. Ordinary businesses can turn you away if they don’t like the look of you, but the government has to prove there’s something wrong with you first.
{ 47 comments }
Benquo 10.05.18 at 2:24 pm
While perhaps people have overstated what the hoax can tell us (I haven’t been following closely), it seems to me like you’re tacitly equivocating between “the hoax tells us nothing” and “everyone already knew what the hoax tells us.” For instance:
While the belief that peer review functions as an effective filter against falsehood is not a reasonable view to hold, it is sadly not a strawman. And telling us that peer review in many disciplines is as weak as a reasonably savvy person would expect is an important result of the hoax.
Likewise, the famous Replication Crisis only demonstrated empirically what people like Richard Feynman had called out analytically decades before – but it was still helpful to have empirical validation.
mpowell 10.05.18 at 2:37 pm
I think this is a very good observation on the question of fraud. But there is also a question of quality control. If fitting an easily matched ideology allows you to promote bad or silly ideas, this is not a good thing. But the proof is not as strong as simply saying, “hey, these papers were a hoax, they should never have been accepted”.
Linnaeus 10.05.18 at 2:44 pm
Here’s Yascha Mounk piling on:
dilbert dogbert 10.05.18 at 2:55 pm
I wonder if the author discusses Fraud By Complexity. Cell phone contracts come to mind.
Sebastian H 10.05.18 at 3:06 pm
Calling this a hoax in the sense you’re using isn’t understanding what they did. They didn’t bamboozle with fake data. They bamboozled with fake reasoning, and clearly fake reasoning at that. The dog rape study for example shouldn’t have been published because the purported data, even if correctly tallied, has nothing to do with the conclusions.
That is the kind of thing that reviewers should be able to pick up if peer review is working anywhere close to properly.
Brian 10.05.18 at 3:45 pm
the susceptibility of a system to intentional deceit is not by any means a good indicator of the underlying health of that system
This is remarkably underappreciated, even in academia.
And I’m not sure it matters whether it was fake data or fake reasoning. It’s really not clear if, in the accepted papers, what was fake about them was something peer review should address. Similarly, some of the ‘fakeness’ of the papers may look quite different when reading with the assumption of good faith on the part of the author.
ph 10.05.18 at 3:48 pm
Read the link via Linneaus (cheers) @3. It’s ugly stuff. I can’t frankly understand how anyone with even ordinary reading skills could fall for this stuff. From the link:
…The sheer craziness of the papers the authors concocted makes this fact all the more shocking. One of their papers reads like a straightforward riff on the Sokal Hoax. Dismissing “western astronomy†as sexist and imperialist, it makes a case for physics departments to study feminist astrology—or practice interpretative dance—instead:
“Other means superior to the natural sciences exist to extract alternative knowledges about stars and enriching astronomy, including ethnography and other social science methodologies, careful examination of the intersection of extant astrologies from around the globe, incorporation of mythological narratives and modern feminist analysis of them, feminist interpretative dance (especially with regard to the movements of the stars and their astrological significance), and direct application of feminist and postcolonial discourses concerning alternative knowledges and cultural narratives.”
This is worse than I feared. I’m sure some fine academics have published in these journals. Whatever must these poor souls be thinking now? Most probably expected their editors/reviewers to be at least able to read. Worthy company? Not so much after this.
How could get through the cited passage and not laugh? I mean, really?
Patrick 10.05.18 at 3:57 pm
I think you are deflecting rather than grappling with this issue seriously. The thing that peer review was supposed to catch wasn’t that the papers were fraudulent. It was that they were bad papers.
I’ve worked in a job where I had to, among other things, review legal documents for whether they contained certain notations and markings and signatures. It would have been STUNNINGLY easy for someone to defraud me in certain ways. For example, if someone who was not John Doe Notary stamped a paper with John Doe Notary’s stamp, I wouldn’t know the difference from looking at the paper. But if they didn’t notarize it at all, or if the date of the notarization was different from the date of the signature, or if the notary stamp was just a Tinkerbell sticker, I damn well would have caught it.
If some hoaxer ran a bunch of faked but plausible signatures past me, that would have proven nothing about the system that we didn’t already know. Me looking at a signature of someone I’ve never met isn’t a setup designed to detect that.
But if some hoaxer ran a bunch of gibberish past me that any fool should be able to look at and recognize as faulty, that would be a very good point and a very concerning one for my employer and for the legal system.
I know the aim of the game right now is to dissemble as much as possible and try to distract from the central thrust of the hoax.
But the hoax looks a lot like that latter scenario to me. It looks like a bunch of material that never should have passed peer review, did. And it looks like a bunch of material that should have failed peer review for certain obvious reasons failed it for other reasons that omitted key points that peer review should have caught.
I don’t think any defense of these fields against the charge of the hoax is going to be possible without 1) accidentally burning down the village you’re trying to defend by arguing that peer review is just generally garbage and we should stop caring about published research, or 2) arguing that the papers that were accepted were actually good papers and the comments on them were good comments.
I hope 1 isn’t true, and I’m pretty sure 2 isn’t true.
And what’s worse is that I’m worried that even if someone wanted to take a shot at arguing (2), they would literally not have a standard or methodology or test or anything that would show whether or not they made their case.
faustusnotes 10.05.18 at 4:00 pm
This is at the root of my objection to this stupid hoax. I peer review a lot of papers, and when I peer review them I do so on the assumption that a) the researchers are serious b) they aren’t lying about their data or their institution, c) they really do want to get published and d) everyone’s work is worth trying to get published if it is done well and it matches the interests of the journal. I’m not looking for fraudulent data, or liars and tricksters, because I don’t have the time or energy to search for con artists and even if I did there’s no way I would be able to prove it and anyway, what are the chances? It’s super easy to trick people who are assessing your work at a remove and assuming you’re honest.
The most obvious example in my field is data. I can get access to the raw data and I can investigate that data as much as I want, but if you told me that you went to a hospital to recruit 300 people and you didn’t, but instead just invented a dataset of 300 observations that says what you want, there’s nothing I can do about that. That’s what Andrew Wakefield did and people are dead because of it. Peer review is a defense against poor quality reasoning but it’s not a defense against fraud, lies and misconduct – which is what these guys did. It would be trivially easy for me to hoax my own field with 20 papers using fake data, and it would prove precisely nothing except that I’m an arsehole.
map maker 10.05.18 at 4:01 pm
Well, the authors did completely make up the dog rape data, but dressed it as reasonably researched and consistently measured. I’m not sure how you can be so confident that this data doesn’t support the article’s conclusion. While the hoaxers might think so, perhaps the joke is really on them, that, while their data is bogus, their conclusions may be right. Perhaps replication is in order?
Linnaeus 10.05.18 at 4:50 pm
Looks like the authors of this hoax have their own grievance ready to go:
JRLRC 10.05.18 at 5:16 pm
And that intentional deceit is not necessarily a good indicator of the underlying health of a system doesn´t indicate that the system is in good health. The academic system is not healthy.
Sebastian H 10.05.18 at 5:47 pm
Faustusnotes, your defense is fine in theory but doesn’t deal with the facts as they are.
I could run a study on varying colors of trees in Seattle based on cloudiness levels to prove that Trump is a lying turd. If I somehow properly measure the varying colors, and additionally properly measure the cloudiness levels, and if I do it in Seattle, I haven’t actually done anything useful in proving that a Trump is a lying turd.
So yes, it is very hard for reviewers to know whether or not I correctly identified the colors of the trees, or the various cloudiness level each day, or even if I’ve ever been to Seattle.
But it isn’t hard for reviewers to know that those questions aren’t relevant to whether or not Trump is a lying turd.
So even if they think Trump is a lying turd, in fact even if they are certain that he is (which to be clear I certainly think he is), they should be able to notice that the questions investigated have little to do with the conclusions.
As we can see from reading the review notes, even a number of the rewrites and rejections fail that test. And certainly the acceptances fail it. That can’t be waved away with something like ‘you can’t expect me to rerun their experiments’.
Cian 10.05.18 at 5:58 pm
I’m amused that all the rationalists think these fradulent papers prove that there’s something rotten in the fields of gender studies.
Obviously a true rationalist would know that it proves that you can submit fraudulent papers to some of these journals, and that more research is required to find out why and how significant it is.
Also, what is their control group here? Basic scientific rigor – that’s all I’m asking for.
Doug K 10.05.18 at 6:12 pm
Good points, thank you.
Follow on to faustusnotes’ comment –
One of the reviewers on Twitter, David Schieber:
Andrew Thaler Retweeted David Schieber
This grad student worked hard to provide a constructive critique of one of the fraudulent “grievance studies” papers before rejecting it. The hucksters used his good-faith effort to be systematic and thorough to mock his field.
Anyways, I guess I could be more critical in the future, but I assumed a grad student had written a confusing paper and I tried to be constructive. I’m embarrassed that I took it as seriously as I did, I’m annoyed I wasted time writing a review, and I’m glad I rejected it. 5/5
Cian 10.05.18 at 6:15 pm
Full disclosure I know little about gender or cultural studies, and don’t particularly care about the health of the field.
That said, what pisses me off about the hoax is the smug certainty of the authors that they know what is an acceptable thing to study. Thus:
What if we write a paper claiming that when a guy privately masturbates while thinking about a woman (without her consent—in fact, without her ever finding out about it) that he’s committing sexual violence against her? That gave us the “Masturbation†paper. What if we argue that the reason superintelligent AI is potentially dangerous is because it is being programmed to be masculinist and imperialist using Mary Shelley’s Frankenstein and Lacanian psychoanalysis? That’s our “Feminist AI†paper. What if we argued that “a fat body is a legitimately built body†as a foundation for introducing a category for fat bodybuilding into the sport of professional bodybuilding?
One of these strikes me a pretty interesting thing to study (the AI paper), one I’m skeptical about but seems like a reasonable thing to debate (the “masturbation” paper) and the final one I actually find rather delightful, particularly given how cultural constructed the whole concept of healthy/unhealthy bodies is.
If anything it seems that a truly healthy field in a more philosophical field would public at least some nutters just to make sure it didn’t end up as narrowly focused as modern political economy. Now if the whole field is nutters there may be a problem, but this piece of ‘research’ couldn’t possibly demonstrate that.
Oh and also these guys are in position to criticise other people’s research methods.
Cian 10.05.18 at 6:33 pm
ph @7:
What is wrong with that quoted paragraph in isolation? Maybe in context it’s all the thing that you say, but in isolation it seems reasonable. Archeoastronomy is a thing, and whatever one thinks of it astrology it is a cultural fact, and one which affects the world around us. It is also something which historically had an influence on astronomical knowledge.
Can you point to a place in the paper where they “western astronomy†as sexist and imperialist? Or are you just taking the hoax authors at their word? Trusting hoaxers to be truthful seems risky.
Also, this paper didn’t get published. Without checking the reviewer comments you don’t know how much of the original paper would have made it in.
Cian 10.05.18 at 6:36 pm
I also think it’s pretty disingenous of the authors to reference papers that got rejected (like the masturbation paper) as if they prove something about the field.
a 10.05.18 at 6:54 pm
Sure, but there’s an obvious mechanism by which those 90 poor-quality startups soon get weeded out. If 90% of publications got retracted within a few years it would be a different story
Jonas Holl 10.05.18 at 8:05 pm
This is the most dead on analysis of this latest attempt to “debunk” bleeding edge social science research.
Carl Irvine 10.05.18 at 8:26 pm
Cian,
As other commentators have noted a control group would would only really be necessary if your questions is whether bullshit is more or less likely to be published in these fields than others. If your question is, can we get bullshit published in these fields, than you don’t really need a control group.
Moreover, no doubt inadvertently, they have included a control group – none of the submissions to sociology journals were published. Which suggests that some disciplines are better than others at detecting bullshit (which, honestly, doesn’t come as a much of a surprise).
Brian 10.05.18 at 8:28 pm
Also, validity of the hoax aside, if the three authors are really concerned about their futures in academia, they might have spent their time doing actual scholarship.
Even if the hoax shows what they say it shows, a prank is just not research, and trolling cultural studies journals isn’t scholarship.
On the other hand, I suppose they did replicate previous findings…
bill benzon 10.05.18 at 8:37 pm
@Cian: “That said, what pisses me off about the hoax is the smug certainty of the authors that they know what is an acceptable thing to study.”
Yes. That’s a very tricky thing.
I’m interested in literary form, in actually describing the form of literary texts. While literary critics talk a lot about form, it turns out there’s little consensus in what it is, much less in how to describe it. So, but the standards of standard issue literary criticism (for the last 60 years), what I do is odd at best, if not crazy.
In fact, one of the things I’m interested in is something called ring-composition. It’s studied mostly by Biblical scholars, classicists, and medievalists. It was the late Mary Douglas (the anthropologist) who got me interested in it. Well, a bit after the turn of the century there was an article that dismissed the entire enterprise more or less on ideological grounds. But the author of the article didn’t actually demonstrate or even attempt to demonstrate that any of the examples he offered were wrong. That is, he didn’t attempt to argue that the formal descriptions those “ringers” (his term) produced were wrong. He simply assumed they had to be.
Assumptions about what are the legitimate boundaries of inquiry need to be questioned every now and then, on principle.
Fake Dave 10.05.18 at 10:02 pm
I don’t think the analogy of this hoax to securities fraud really works here. The editors and peer reviewers seem more equivalent to regulators and auditors than just Johnny Hedgefund betting on Worldcomm. The journals had full access to “the books” as it were and while they can’t know for sure if the numbers are real or faked, they can damn well be expected to check the math and see if it all adds up. Maybe you can’t blame a private investor for buying junk bonds, but people can and do blame the SEC or the ratings agencies when they let obvious junk get listed or certify it as AAA. My memory is a bit hazy, but I seem to recall people having a lot to say about that topic about ten years ago, in fact. The core argument that hoaxes don’t really prove anything about the health of a system may still be right, but given that these peer-reviewed journals are supposed to serve as gatekeepers to protect the public, likening it to an economic system where negligent gatekeepers have continually exposed the public to unacceptable risks seems like a poor rhetorical strategy.
Eli Rabett 10.05.18 at 11:15 pm
Before everybunny gets too shirty about grievance studies or whatever, look up Jan Schoen.
https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal
AcademicLurker 10.05.18 at 11:52 pm
Eli Rabett@25: Scientific fraud seems like a different sort of thing. Fraudsters make significant efforts to not get caught. As I understand it, Schon crafted his fake data to agree with most plausible theories about organic materials that he picked up from his theory colleagues at Bell Labs.
These hoaxes seem to me more like the old story of the philistine who looks at a piece of modern abstract art and says “That’s just a bunch of paint splattered everywhere! I could do that in 10 minutes! Why is this guy a famous artist?” Now generally the philistine is wrong and he couldn’t just splatter paint on a canvas for 10 minutes and convince critics that the result was a significant work. But suppose he could. Would that prove anything about the prevailing standards in the art world?
It seems to me that what these hoaxers believe they are doing is the equivalent of splattering paint around for 10 minutes and convincing the art world that the result is, if not a masterpiece, at least deserving of wall space in a respectable museum.
I haven’t read the hoax articles, so I take no position on whether or not that’s what they’ve actually accomplished.
ph 10.06.18 at 12:43 am
Cian, I enjoy your comments and feel no need to agree, but “what’s wrong with…” the first frigging sentence screams nonsense: “Other means superior to the natural sciences exist to extract alternative knowledges about stars and enriching astronomy” including ethnography and other social science methodologies, careful examination of ‘blah-blah…’
The ‘careful examination’ line is the hoaxer figuratively waving his/her bum in our faces.
Laughter is the only appropriate response, the hoaxers were announcing their fraud. When you’re reading a sentence that ‘includes other means superior to the natural sciences exist’ in an article purportedly about astronomy, this line confirms the farce.
I deal mostly with primary sources, and when I read modern commentary I rarely/never read anything impressive other than by those who are experts in their fields, a group that does include graduate students. But crap is crap.
That’s pure crap you’re defending, and I don’t believe for a moment you can’t discern the difference. And if you really can’t (which I doubt), inform your dean, your peers, and your students. All deserve to know. If anyone still wonders why some treat cultural and gender studies with more than ordinary skepticism, your defense of this nonsense (more than the hoax itself) is exhibit A.
Faustusnotes 10.06.18 at 1:08 am
Cian I make the point at my blog that the masturbation paper is covering an interesting topic. If society doesn’t have some kind of notion of metasexual violence we can’t punish upskirting videos or revenge porn. A coherent theory underlying cultural and judicial responses to things like that would be helpful. Sometimes ideas that seem facially stupid aren’t.
I can imagine 18th century bloggers opposed to this newfangled “science” laughing at preposterous hoaxers who almost published a paper on using milkmaid’s skin sores to protect against smallpox. Ridiculous!
J-D 10.06.18 at 2:17 am
Sebastian H
The purported evidence presented in the ‘Dog Park’ paper would, if it were not entirely fabricated, support the stated conclusion of the paper.
Matt 10.06.18 at 2:50 am
If society doesn’t have some kind of notion of metasexual violence we can’t punish upskirting videos or revenge porn.
It’s orthogonal to the main point, but I don’t see why this is so. The old tort of “invasion of privacy” would normally allow for a civil suit in these cases, and a criminal charge based on a similar rights invasion seems straight forward without having to invoke the idea of “metasexual violence”.
SusanC 10.06.18 at 8:29 am
I was mildly amused that tis posting comes directly after one about the REF.
It’s well acknowledged in many fields that there are junk journals that will publish anything you send them. The implied claim by Sokal et al is that he hard sciences are better han the social sciences at rehecting junk. It’s unclear tha this is true, and if Sokal et al wanr to inply that claim they should at least test it by submitting junk papers to a predatory journal in physics.
If we take it as given that junk journals ate everwhere, he stronger claim is: do any nin-jink jiurnals exist, and do we know which they are. Can you get a 4* rating in the REF with one of these hoax papers.
Bill Benzon 10.06.18 at 10:41 am
@Faustusnotes #28: I can imagine 18th century bloggers opposed to this newfangled “science†laughing at preposterous hoaxers who almost published a paper on using milkmaid’s skin sores to protect against smallpox.
Don’t forget Ignaz Semmelweis, who was widely and soundly mocked for the empirically based suggestion that, by washing their hands when dealing with patients, doctors could reduce the number of fatalities in hospitals. Semmelweis’s antiseptic practices were only accepted, years after his death, when Pasteur’s germ theory of disease became established.
Brandon 10.06.18 at 11:16 am
So the control group here could be what is called a *synthetic control.* When a researcher in the field can’t do her own randomized controlled trial, she compares the outcomes of her experimental cases to the outcomes of the natural cases where the population was left to its own devices in instances where it was contending with the same environment of independent variables in addition to the one that was manipulated for experimental purposes.
You use a synthetic when you can’t create an actual control group because you as a researcher are incapable of identifying or designating them. Say US states W and T have practically the same gun laws, economies and cultures, *and are observed to have basically the same outcomes as well,* but suddenly T gets rid of nearly all if its gun laws and starts giving away guns to nearly anyone who wants one. State W is the synthetic control for “experimental” state T in rigorously studying the outcomes of gun law changes.
The hoax authors here cannot create a control group themselves because they have not been certified by the scholarly community’s recognition standards (ie, the degree) as capable of creating such control scholarship. I guess they could get people with the PhDs to submit legit papers as a control but the worry there is a placebo effect of some sort if the journal editors became aware of the experiment. Plus I doubt grievance any profs would cooperate.
So the null hypothesis here is that there is no difference between the likelihood of “well-researched, professionally-written” and “nonsensical, apoplectic, pedantic, fraudulent” papers being published in the class of journals in the experiment.
The synthetic control is the natural success of the former type of papers in very similar or the same journals.
One thing appears to be clear, even though we don’t have the precise stats on the control outcomes: we don’t have cause to reject the null hypothesis.
Cian 10.06.18 at 4:29 pm
The synthetic control is the natural success of the former type of papers in very similar or the same journals.
They’re not similar. One set of journals are from a single discipline, the other set of journals are interdisciplinary. Also, they made no attempt to use the sociological journals as a control group, we have no particular reason to think all the other variables are controlled, etc, etc (not to mention that synthetic controls are problematic in practice, which is why just retroactively designating one group as a control doesn’t really work).
Hidari 10.06.18 at 4:36 pm
Good article in Slate on this pointless heartless ‘hoax’.
‘If you really take the time to read through Pluckrose, Lindsay, and Boghossian’s 11,650-word essay, along with the bogus papers they produced, you’ll find the project fails to match its headline presentation. The hoaxers’ sting on academia is supposed to have exposed the “sophistry†and “corruption†that exist across a broad array of research fields—those built around the “goal of problematizing aspects of culture in minute detail in order to attempt diagnoses of power imbalances and oppression rooted in identity.†The authors call these “grievance studies†and say their disregard for objective truth has yielded to a widespread “forgery of knowledge.†Yet these grandiose conclusions overstate the project’s scope and the extent of its success. They also serve as cover, in a way, for what appears to be the authors’ lurking inspiration: not their problems with the scholarship of grievance, but with that of gender (specifically, feminist studies)…
Let’s analyze the hoax a bit more carefully. The team wrote up 21 bogus papers altogether. (The essay starts by saying there were only 20; according to Lindsay, that’s because two of the papers were largely similar to one another.) Of those 21, two-thirds never were accepted for publication. The Areo essay dwells on several papers that had been rejected outright, including one suggesting that white students should be enchained for the sake of pedagogy, and another proposing that self-pleasure could be a form of violence against women. They take it as a sign of intellectual decay that such papers managed to elicit respectful feedback from reviewers, even short of publication. (One of those has since explained that he was just trying to be helpful.) But I think we can all agree that it’s neither telling nor newsworthy when a bogus paper fails to get into an academic journal, however offensive or inane it might have been.
What about the seven papers that were accepted for publication? One was a collection of poetry for a journal called Poetry Therapy. Let’s be clear: This was bad poetry. (“Love is my name/ And yours a sweet death.â€) But I’m not sure its acceptance sustains the claim that entire fields of academic inquiry have been infiltrated by social constructivism and a lack of scientific rigor.
Another three plants were scholarly essays. Two were boring and confusing; I think it’s fair to call them dreck. That dreck got published in academic journals, a fact worth noting to be sure. The third, a self-referential piece on the ethics of academic hoaxes, makes what strikes me as a somewhat plausible argument about the nature of satire. The fact that its authors secretly disagreed with the paper’s central claim—that they were parroting the sorts of arguments that had been made against them in the past, and with which they’ve strongly disagreed—doesn’t make those arguments a priori ridiculous. But hey, that’s just my opinion.
That leaves us with three more examples of the hoax. These were touted as the most revealing ones—the headline grabbers, the real slam dunks: the dog-rape paper, the dildo paper, the breastaurant research. They also share a common trait: Each was presented as a product of empirical research, based on original data. The dog-rape study is supposed to have resulted from nearly 1,000 hours of observation at three dog parks in southeast Portland. The dildo paper pretends to draw from multihour interviews with 13 men—eight straight, two bisexual, three gay—about their sexual behaviors. And the breastaurant research claims to have its basis in a two-year-long project carried out in northern Florida, involving men whose educational backgrounds, ages, and marital statuses were duly recorded and reported.
How absurd was it for such work to get an airing? It may sound silly to investigate the rates at which dog owners intervene in public humping incidents, but that doesn’t mean it’s a total waste of time (as psychologist Daniel Lakens pointed out on Twitter). If the findings had been real, they would have some value irrespective of the pablum that surrounds them in the paper’s introduction and discussion sections. Indeed, one can point to lots of silly-sounding published data from many other fields of study, including strictly scientific ones. Are those emblematic of “corruption†too?
It’s true that Pluckrose, Lindsay, and Boghossian tricked some journals into putting out made-up data, but this says nothing whatsoever about the fields they chose to target. One could have run this sting on almost any empirical discipline and returned the same result. We know from long experience that expert peer review offers close to no protection against outright data fraud, whether in the field of gender studies or cancer research, psychology or plant biology, crystallography or condensed matter physics. Even shoddy paste-up jobs with duplicated images and other slacker fakes have made their way to print and helped establish researchers’ careers. So what if these hoaxers did the same for fun? These examples haven’t hoodwinked anyone with sophistry or satire but with a simple fabrication of results.
Pluckrose, Lindsay, and Boghossian employed this made-up-data method for five of their 21 papers, and three of those were accepted for publication—yielding a hoax-success rate of 60 percent. When they wrote up papers without this added layer of deception, just four of 16 were accepted.
Even if we push the made-up-data papers to the side, those results are still quite grave: Twenty-five percent of bullshit papers made their way through peer review. But what, exactly, does it prove? It would be nice to know how often counterfeit research makes its way into the journals of adjacent fields; e.g., ones that touch on race, gender, and sexuality, but are uncorrupted by radical constructivism and political agendas. Sadly, we may never know, because the field of humanities hoaxing appears to suffer from several of the flaws it aims to expose. For one thing, it’s politically motivated, in the sense that its practitioners target only those politicized research areas that happen to annoy them. For another, it’s largely lacking in scientific rigor. Most (but not all) hoax projects lack meaningful controls, and they’re clearly subject to the most extreme variety of publication bias. That is to say, we only hear about the pranks that work, even though it’s altogether possible that skeptic-bros are writing bogus papers all the time, submitting them to academic journals, and ending up with nothing to show for their hard work. How many botched Sokal-style hoaxes have been tucked away in file drawers and forgotten because they fail to “prove†their point?
It’s even harder to assess this week’s sting because its authors are so weirdly coy about their targets. Whom, exactly, have they stung? It’s never clear: The essay starts by pointing at “certain fields within the humanities†that are motivated by grievance and that examine topics of “gender, race, sexuality, culture.†In the YouTube video released in tandem with the essay, Lindsay says the project has revealed a deeply concerning and pervasive corruption “among many disciplines, including women’s and gender studies, feminist studies, race studies, sexuality studies, fat studies, queer studies, cultural studies and sociology.â€
That does sound pervasive. But if you look at the details of the hoax, its targets aren’t really that diverse; they’re clearly focused on the fields concerned with gender. Among the 21 academic journals named in the essay, almost half describe themselves on their websites as venues for “feminist†research; three more refer to gender. (By contrast, just a handful say they’re dedicated to the study of “race,†“sexuality,†or “culture.â€) The sham papers, as written, show an even clearer version of this tilt: Going by their abstracts, almost all the fakes (18 of 21) make silly or parodic claims concerning gender; just eight mention race or sexuality.
Going off these numbers, one might presume that “grievance studies,†as the sting would have it, is best defined as mostly feminism and gender stuff. That interpretation would seem to fit the authors’ prior work and interests too. This week’s project turns out to be the offspring of another hoax from Lindsay and Boghossian, published in May 2017. That one, called “The Conceptual Penis as a Social Construct,†was billed as an attack on gender studies in particular. “We suspected that gender studies is crippled academically by an overriding almost-religious belief that maleness is the root of all evil,†they wrote at the time. “On the evidence, our suspicion was justified.â€
…
How timely… that this secret project should be published in the midst of the Kavanaugh imbroglio—a time when the anger and the horror of male anxiety is so resplendent in the news. “It’s a very scary time for young men,†Trump told reporters on the very day that Pluckrose, Lindsay, and Boghossian went public with their hoax. Both express a fear of ‘false’ attacks on men, whether levied by regretful sluts, lefty liberals, radical academics, or whoever else.’
Etc.etc. etc.
Whatever one thinks about this hoax (and the whole Slate article is worth reading) it has little to do with Sokal’s concern that the academic left was being sidetracked. These authors do not want a resurgence of academic leftism (which is what Bricmont and Sokal wanted). They want large swathes of academia, specifically cultural studies, to cease to exist, precisely because it leans to the left (or at least is perceived to). To be even more precise, they want academic feminism to cease to exist, because they feel it is all about ‘grievance’ (I mean, goodness me, what could women possibly have anything to feel grieved about).
Needless to say, the Usual Suspects (e.g. Pinker) of the tiresome ‘right wingers posing as left wingers’ stylee have piled in demanding that pseudo-science is cracked down on (but not the pseudo science of, say, Evolutionary Psychology. Other pseudo sciences).
As the Slate author points out, the fact that this ‘hoax’ was published in the wake of #Metoo and the Supreme Court fiasco, is not a coincidence. The same thought processes are at work.
cian 10.06.18 at 4:40 pm
PH @ 27
but “what’s wrong with…†the first frigging sentence screams nonsense: “Other means superior to the natural sciences exist to extract alternative knowledges about stars and enriching astronomy†including ethnography and other social science methodologies, careful examination of ‘blah-blah…’
To extract alternative knowledges about stars. I.E non-scientific knowledge about stars/heavens – historical, cultural, etc. Which is how anyone working in the field would read this paragraph. It’s highly probable that the pranksters meant it the way that you interpreted it – but that’s simply because they’re not familiar with the field they’re pranking.
Patrick 10.07.18 at 5:55 am
Hidari wrote:
“. The Areo essay dwells on several papers that had been rejected outright, including one suggesting that white students should be enchained for the sake of pedagogy, and another proposing that self-pleasure could be a form of violence against women. They take it as a sign of intellectual decay that such papers managed to elicit respectful feedback from reviewers, even short of publication. (One of those has since explained that he was just trying to be helpful.) But I think we can all agree that it’s neither telling nor newsworthy when a bogus paper fails to get into an academic journal, however offensive or inane it might have been.”
They did not argue that it was bad that they got respectful feedback. They argued that the papers were praised for the parts that they, the creators, thought were stupid.
Patrick 10.07.18 at 6:16 am
The analogy between this and Kavanaugh suddenly struck me.
I’ve been half frustrated about the Kavanaugh thing, and half resigned. Frustrated because whatever norms we try to establish with respect to decades old crimes from childhood, his inability to behave with dignity ought to disqualify him. But also resigned because I know that whoever might have replaced him would have been just as bad, both politically and in terms of temperament, because that’s what the people picking him want. Lying about his opinions on abortion and torture was one of the job requirements, so there was never a chance of getting a good judge.
And frustrated because no matter how much people shouted and screamed and hoped the Democrats could somehow win this, I knew that in the end, as it always does… all that mattered was the Senate vote and there was no real chance of winning that. Protest might shift public opinion but it wasn’t going to shift any Republicans. Protest had little to no chance of shifting the real levers of the structure that was making the decision.
And with respect to this hoax… part of the structure of academia is that fields self police. If one academic goes rogue, the “solution” is that peer review and reputational harm torches his ability to influence others and spread his poor scholarship. But if a field goes rogue, say, by becoming a place where people just publish provocative think pieces and patting each other on the back for how fashionable they’re all being together, there’s no mechanism to fix that. Even if everyone who looks into the field from the outside can see it, the field itself peer reviews its own work. Plus there are a myriad of smoke screens to be used. Political arguments: “we’re just being attacked by people infected with bad politics and ‘isms!” Relying on the general glow of academia: “calling our published, peer reviewed academic work is like disagreeing with global warming or evolution!” Obscurantism: “the real problem is that you don’t understand our field, until you truly understand, say, the divine mystery of the holy trinity, you can’t disagree with it!”
Or just make stuff up about your critics and refuse to address the issue. “We’re concerned that you said our terrible idea was promising and you hoped we’d resubmit it once we tidied up the essay.” “Oh, so you’re saying your mad that people were polite?!?”
I guess its always been like this. Theology departments still exist. Alvin Plantinga has a career. I guess it makes sense that there’s no easy fix for this.
But its depressing.
There was never a way for this hoax to be successful or useful. Because at the end of the day all of the academics involved will just go back to work, and when the next round of half baked pseudo philosophy think pieces hits the review stage, they’ll review them the same way they always do.
Just as with Kavanaugh, the levers of power needed to fix the problem are in the hands of the people who created it, and they don’t want to fix it.
So I guess I should just do my best to stop caring.
SusanC 10.07.18 at 10:08 am
“This paper is nonsense. The authors are idiots.” Is generally considered to be not acceptable as a referee’s report. (Sure, you can say that in the confidential section that only goes to the editor, but not in the bit that goes back to the authors). I have on occasion as a referee been asked by the journal editor to be more tactful in my feedback to the authors. You’ll always get a respectful reply, no matter how stupid your paper is; getting one to a hoax paper is not an idication of poor refereeing.
casmilus 10.08.18 at 5:11 am
In case no one’s mentioned it: Stefan Collini used “grievance studies” as a jibe against what he saw as bad trends in Cultural Studies, in an essay collected in “English Pasts”. I think that dates to the early 90s. I very much doubt he has any link to Nick Land, though Land might have pinched it off him.
Talking about “journals” is pretty futile unless the dimensions of prestige and “impact” (for want of a better term) are also accounted for. All those poor creationists who seem to think the ICR produces “academic work” in a “scientific journal”.
DCA 10.09.18 at 9:50 pm
Perhaps too late for your research, an entertaining compendium of US mining fraud (and the occasional good-luck story):
Bonanzas & borrascas / Richard E. Lingenfelter
Vol 1. Gold lust and silver sharks, 1848-1884
Vol 2. Copper kings and stock frenzies, 1885-1918
Fake Dave 10.10.18 at 2:36 am
That Slate article is deeply suspect because it claims that all but 7 out of 21 papers were rejected. This the least charitable possible interpretation of their success rate and completely ignores what the authors themselves said about having to end the project early and that their belief that some of the remaining papers were likely to be accepted. Even if we think that is unlikely, it is just wrong to count papers still early in the review process as being rejected. Either they misinterpreted the data or they were deliberately trying to minimize it for some political agenda.
That these papers are supposed to be about “greivance studies” in general but mostly focus on gender and sexuality is a damn good point though. A few explanations that spring to mind are that that is a big field with lots of journals to target (as opposed to, say, Fat Studies), that they targeted feminist literature because it’s the origin of the standpoint epistemology they are targeting, and that they invested a huge amount of time in learning the specific theory and lingo of Gender Studies and would have had to add months to their project before they could fake other topics like race and ethnicity.
None of those are particularly unreasonable explanations, but the hoaxers really should have been honest about who they were impersonating and targeting and why. The fact that they weren’t open about that and even muddied the waters with jibes at fields that had nothing to do with with this hoax is grounds for suspicion.
faustusnotes 10.10.18 at 3:23 am
Actually Fake Dave I think that it is honest to say they only got 7 published. The original hoaxers were trying to claim that 7 publications is enough to get a professor tenure, but they were a team of three, so it’s reasonable to expect that they should be able to publish 21 papers in a year. They couldn’t because their papers were shit. Some got three rejection notices. You could argue, yes, that if they kept going for another year they would be able to get the other 14 published, but you would also then have to argue that they had significantly changed the content by the time they did get published. Indeed one of the 7 that did get published had all its fake data stripped out (according to the hoaxers), so in order to get it published they already had to completely change their work.
It depends on the field, but if a team of three authors writing separate first author publications to journals in their field can only get 7 papers published in a year, they’re not very good at what they’re doing, and certainly not at the level of a tenurable prof as they claim in their little rant. If you google the profile of any decent tenured academic you’ll find they have more than 2 papers published in one year. I think it’s legitimate for Slate to treat the numbers presented as a robust assessment of their efforts.
SusanC 10.10.18 at 9:35 am
If it was as easy as that, we’d all have tenure…
Also, for promotions/tenure, your publications have to be in good journals. And at least in the departments I’m familiar with, trying it on with publications in junk journals will be considered worse than publishing nothing.
Patrick 10.10.18 at 10:34 pm
Can we quote the people in this thread in the future when dealing with these journals, and with the academic credibility of anyone published in them?
Or is this one of those facile mean kids lunch table things where we all agree that these journals are terrible for a week until we forget about this hoax, then go back to treating them like normal?
Is anyone in this thread who is an actual academic willing to have their real names and real identities attached to statements like, “Hypatia is junk and publishing in it is worse than not publishing at all?”
…if we eliminate all the journals we’re now agreeing are junk (presumably any journal that would provide similar results, or any journal with less reputation as the journals we’re calling junk, counts as junk as well), what do these fields have left?
Faustusnotes 10.11.18 at 2:42 am
SusanC I was quoting them, not establishing facts. I just want to show that their output was not actually large and certainly was below what you would expect from a group of three tenure-worthy academics working for one year without teaching responsibilities.
J-D 10.11.18 at 5:35 am
Patrick
I suppose you can quote people if you like, but I haven’t seen anybody on this thread commenting on the academic credibility of people published in any journals, so I’m not sure what you’re referring to.
I haven’t seen in this thread that we all agree that these journals are terrible. I’ve seen people in this thread disagreeing.
SusanC has suggested that junk journals do exist and that publishing in them is considered worse than not publishing at all, but she didn’t mention Hypatia as being one of those journals.
I have seen SusanC’s suggestion that some journals are junk, but I haven’t seen agreement on this thread about which journals are junk.
Comments on this entry are closed.