Reverse Turing Tests

by Henry Farrell on April 13, 2005

“Tom B,” commenting at Making Light, points us to the Automatic Computer Science Paper Generator, which uses context-free grammar to generate papers, complete with graphs, figures and citations, which can then be submitted to conferences with low or no standards for the papers they accept. Its creators (MIT pranksters) have already succeeded in getting accepted by one conference – if they can raise the money, they intend, Yes Men style, to go there and deliver the paper with straight faces. It seems to me that pranks of this sort (the Atlanta Nights affair also qualifies) have the logic of a reverse Turing test – any conference (or publishing house, or journal, or whatever) which is stupid or unprincipled enough to accept this sort of nonsense is revealing itself to be a fake.

{ 25 comments }

1

KCinDC 04.13.05 at 1:20 pm

I thought this was going to be another post about comment spam.

Doesn’t publicizing this before the conference is held ensure that it will be presented only if the conference is even faker than required to accept it in the first place?

2

Jake 04.13.05 at 1:29 pm

What I find disturbing (in an amusing way) is that such a generator, if used in some of the humanities or social science fields where only an abstract is required for admission, might be able to generate admission to conferences which actually have standards.

3

bi 04.13.05 at 1:34 pm

Andrew Bulhak did something similar with his Postmodernism Generator, and there’s also the _Social Text_ affair. Last I know, the crazy postmodernists are still alive and well, and were making up all sorts of lame excuses for their acceptance of the bogus article.

Then there’s the Bogdanov affair. I think this affair could be handled better, but at least the people involved aren’t busy blaming other people for their own mistakes…

4

bi 04.13.05 at 1:36 pm

… _where only an abstract is required for admission_ …

I think there’s a really serious problem with this.

5

lemuel pitkin 04.13.05 at 1:40 pm

What, no mention of the Sokal hoax? Would be perfect timing, too, since it was this post that pushed the Valve/Cultural Revolution one off the front page.

6

Dan Simon 04.13.05 at 1:54 pm

I think there’s a missing bit of context here. If I’m not mistaken, the “technical conferences” in question are basically collaborators in a fraud scheme, in which researchers with travel funding submit “papers” they know will get accepted, so that they have an excuse to pay to “attend” the conference–always in a pleasant vacation destination–with research funds. The conference organizers get the registration fee, the researcher gets a free vacation, and everybody’s happy except the supplier of research funds.

A related scam I’ve heard of involves “international” conferences in the US which accept papers from around the world. The conference organizer then writes a letter in support of each “author’s” request for a visa to enter the US to “attend” the conference. Again, the conference gets the fee, and the “researcher” gets a visa, enters the US, and is never heard from again.

Needless to say, these scams are (or should be) a serious concern for serious academics, whose ability to use research funds to attend legitimate academic conferences, and to invite foreign researchers to attend, may ultimately be in jeopardy as a result.

7

dr 04.13.05 at 1:56 pm

Just like Sokal, this is hilarious, except for the whole fraud thing. There’s a lot of the structure of society which revolves around assuming the basic integrity of apparently sincere, well-meaning strangers. Explain again why it’s a good idea to punish those assumptions of good will?

8

Seth Finkelstein 04.13.05 at 2:22 pm

The _Sokal_ hoax inspired me to propose a varient I called “The Philosopher Turing Test”

One participant is a deep-thinking philosopher.

One participant is a speaker of elaborate gibberish.

How do you tell the difference?

Really. It seems to me there’s something profound here.

9

Jack Lake 04.13.05 at 2:54 pm

Many of the human written papers refereed and published in computer science, and probably most other fields, are not worth the paper on which they are written.

Let a program do it and let these scientist enjoy a life of leisure.

10

Henry 04.13.05 at 3:14 pm

But I don’t think that the Sokal hoax is quite the same. “Social Text” was completely incapable of judging a paper that purported to be scientific (and probably should have farmed out the paper to a referee with experience in the matter) – but it’s not a fake publication. Just one that got in out of its depth in a subject area not its own.

11

John Quiggin 04.13.05 at 3:22 pm

I regularly get invitations to expensive conferences on very generic topics in locations like the Greek islands or Las Vegas. As Dan says, it’s pretty clear what’s going on here.

OTOH, refereeing papers is the exception rather than the norm for most of the genuine academic conferences I attend in Australia. Broadly speaking, you have to have a paper if you want funding to attend, and the conferences are small enough that everyone gets a turn.

12

r. Waterhouse 04.13.05 at 3:28 pm

It’s a little known fact that many of the most popular so called “bloggers” are in fact perverted copies of the ALICE chatbot, some you could probably have guessed, Instapundit, Powerline, Eschaton .. many more than most people want to know. The more disturbing thing is that it is not just bloggers who are abdicating their thought processes to software, it’s a growing problem that affects many professions; academics, lawyers, legislators and business leaders have all fallen to the temptation to let their computers do the heavy lifting for them in producing verbiage. For instance it’s an open secret on capitol hill that much legislation is the product of a set of word macros developed by legislative staffers, and that often bills are passed into law without being examined by a human at all.

13

TomB 04.13.05 at 3:37 pm

Credit should go to Jonathan “Wolf” Rentzsch, who told me about SCIgen.

Also, the MIT grad students deserve credit for a very effective critique of WMSCI. It’s ironic that their bogus CS paper generator is itself a pretty decent work of CS. Maybe they should submit a paper about it to a real CS conference.

The Reverse Turing Test concept is great. Scammers like PA and WMSCI will always be susceptible to !T tests, not because they’re stupid, but because their business model requires them to accept almost everything they receive. If they actually did legitimate reviewing or editing, they would be in a different niche, where they would be much less likely to succeed.

14

lemuel pitkin 04.13.05 at 3:58 pm

But I don’t think that the Sokal hoax is quite the same.

Oh, I agree. I was just trying to stir up trouble.

15

Matt McGrattan 04.13.05 at 6:13 pm

Seth: For what it’s worth, I suspect most academic philosophers would take about 10 seconds to identify the gibberish speaker from the ‘deep-thinking’ philosopher — although I’m not sure what ‘deep-thinking’ is supposed to signify here.

Re: Sokal – My understanding of the Sokal hoax was that it wasn’t just that the editors of Social Text got out of their depth and accepted for publication an article that they really ought to have sent for expert refereeing but that, more generally, Sokal was drawing attention to the _routine_ use and abuse of scientific vocabulary by those who didn’t really understand such vocabulary and the problem wasn’t just that he personally, in his article, was misusing key scientific concepts but also that the journal wasn’t particulalry concerned with whether such concepts were being used correctly at all. It wasn’t just a special case of one paper slipping through the cracks. As such he was drawing attention to what he saw as a systemic problem.

Of course, we can debate whether it really _is_ (or was) a systemic problem and whether Sokal’s hoax was any way to test it.

16

Andrew 04.13.05 at 6:26 pm

That really is the funniest thing ever. Sadly, when I was at Tokyo Institute of Technology, we had one poor Doctor’s student who’s English made about as much sense as this. However, he was able to find monkey-typewriter journals to publish his work.

Now if someone could only write a similar algorithm for patent applications they might have something going…

17

Brian 04.13.05 at 7:41 pm

I just wanted to back up what John said about refereeing for conferences in Australia. I was surprised when I got to the US that you had to have a paper accepted by a referee to get into many conferences. The main effect of this is that the conferences over in America have papers that I’ve seen 12 months earlier. I’m not the only one who thinks refereeing doesn’t help improve conferences.

“Here is a revealing comparison. For the annual meeting of the American Philosophical Association the Program Committee sifts submissions carefully and rejects 80 percent. The Australasian Association of Philosophy does not sift submissions. Yet every year the AAP program is better overall than the APA program.” (Bill Lycan, Philosophy UNC) (ref)

18

Andrew 04.14.05 at 1:09 am

But where is the AAP v the APA programme? I’d prefer a nice Sydney holiday over one in Chicago any time of year. The best overall conference I ever went to was CIIT2004 that just happened to be in St. Thomas..

19

Bob McGrew 04.14.05 at 3:03 am

This conference, in particular, seems to be some sort of scam – after all, the paper was accepted “without review” because their reviewers turned in their reviews late. (The Sokal paper, on the other hand, was actually peer-reviewed.)

Also, the role of conferences in CS is different from other fields – conferences usually have printed proceedings, and quality of conference publications is the first criterion for hiring and tenure at many schools. Journals, on the other hand, usually reprint longer versions of papers that were accepted to the more prestigious conferences.

20

Simstim 04.14.05 at 3:30 am

But aren’t most academic conferences at least partly an excuse to go on a paid holiday? I’m reminded of David Lodge’s “Small World”:

The modern conference resembles the pilgrimage of medieval Christendom in that it allows the participants to indulge themselves in all the pleasures and diversion of travel while appearing to be austerely bent on self-improvement. To be sure, there are certain penitential exercises to be performed -the presentation of a paper, perhaps, and certainly listening to the papers of others. But with this excuse you journey to new and interesting places, meet new and interesting people, and form
new and interesting relationship with them; exchange gossip and confidences for your well-worn stories are fresh to them, and vice versa); eat, drink and make merry in their company every evening; and yet, at the end of it all, return home with an enhanced reputation for seriousness of mind. Today’s conference have an additional advantage over the pilgrims of old in that their expenses are usualy paid, or at least subsidised, by the institution to which they belong, be it a government department, a commercial firm, or, most commonly perhaps, a university.

21

John Quiggin 04.14.05 at 3:51 am

I suppose it’s middle age, but what I like about conferences is catching up with all my old friends, and reminiscing over all the old stories from previous conferences.

22

John Quiggin 04.14.05 at 3:59 am

Bizarrely, it appears the pranksters submitted two papers, and one was rejected. On the face of it, this suggests low standards rather than a complete scam.

23

bi 04.14.05 at 5:05 am

Even more bizarre is that no reason was given for the rejection……

So when are we going to see a Dubyaism Generator?

24

Doug 04.14.05 at 9:58 am

Shouldn’t this be the occasion for a new CT category? Something like: Intellects teenie-weenie and not sympathetic either…

25

john c. halasz 04.14.05 at 9:07 pm

I’m so confused. I think I need to consult a computer simulation of a Rogerian therapist for counselling.

Comments on this entry are closed.