Andart

Scholarship and its Antibodies

A paper about making hoax papers that I found absolutely delicious:
The Cartesian Conspiracy: How to do Post-Modernism with Marquis de Sade by J. Yellowlees Douglas.

Is academic hoaxing a problem, or actually a way of ensuring the health of scholarship and peer review?

(Based on a diskussion on the Extropians list)

Academic hoaxes like Alan Sokal's are not just fun but a way of keeping academia healthy. The laughter and derision afterwards help point out flaws in the peer review system or other forms of quality control. It is also better with deliberately bad papers, since inevitably the creator will point them out, unlike "accidentally" bad papers where the author won't tell anybody that he cheated, made things up or didn't know what he was talking about. We need the first kind as a kind of vaccination against the second kind.

The Douglas paper it is an attempt to explore hoaxing a bit more carefully than just sending off one hoax that a critic might claim actually is a reasoned paper the author doesn't dare stand for. In this case the author replaced words in de Sade with words from postmodern giants (sodomy becomes theory, murderer becomes descriptive constructivist) and produced a number of nonsense papers. They were submitted to a number of journals, and four out of ten were accepted despite peer review. None were rejected for incomprehensibility.

The really interesting part is the discussion, where the author points out that from an analytical perspective this is of course a sign of bad scholarship from the journals, but from the Continental side it might actually be OK:

"These distinctions between analytic and Continental ideals are crucial to the interpretation of this experiment, since the Continental position would possibly regard the nonsensical articles as being no less valuable than genuinely authored scholarship. For, if the value of scholarship rests in its ability to communicate meaning to its reader (and this meaning being unfixed, interactively determined by author and reader), then nonsensical discourse may be considered valuable so long as its reader finds it to be sensible."

For people coming to the issue from the hard sciences this makes little sense, since we are very much based in a realist enlightenment tradition. But that does not mean the other approach is invalid, it is just invalid if the message is supposed to bear any relationship with the world. If that relationship is instead provided by the interpretative process of the reader's mind it can indeed lead to fruitful ideas and actions in the real world. However, as the author points out, if we are to accept this as scholarship then the humanities are seriously threatened:

"In an age where nonsense is identical to scholarship, text-generating computers such as the Dada Engine are no less useful than postmodern scholars, and the necessity of the latter becomes difficult to substantiate."

The joke may go both ways; the "Bogdanov affair" (see the sci.physics thread and the collection of various partisan entries) appeared to involve nonsense physics papers being published in refereed journals and even giving two Ph.Ds! On the other hand, maybe it was perfectly legitimate theoretical physics papers that got treated by the community as hoaxes! The problem here seems to be more that it is hard to tell, and that is worrying in itself.

Peer review is supposed to filter out nonsense, bad research, academic misconduct and other scholarly dirt. The process need not be perfect, because the hope is that science is self-correcting. That hoax papers pass reviewers is not a sign that the process per se is bad, it just shows individual mistakes and might give a hint that particular journals or fields are getting lax. But if many such papers pass, then the problem might be widespread. Even worse, if people can't agree on whether a paper is a hoax or not, such as in the Bogdanoff affair or some critics of the Sokal paper, then the field itself might have trouble with its epistemology (granted, in the Bogdanoff affair plenty of traditional academic politics seems to be involved, muddying the waters). Maybe one could replace reviewer comments with the dada machine too. Which would mean giving up on the self-correction of science.

Overall it seems that the system of scientific publishing needs new ideas and better ways of disseminating and filtering information. Academic review boards might be one way of achieving a more flexible review process. Various forms of filtering and presentation might enable networked and distributed debate and knowledge growth. This seems to be not just an interesting area for agent modelling and research (applied memetics?) but also of cruicial academic importance.

As science advances our knowledge will grow and diversify, making it harder and harder to just pick out a single fact without its context. And this context of course grows with science, making it harder and harder to get useful information since understanding it requires more information, and so on. Even relevant information becomes noise because there is too much of it. But, as Heylighen argues, we can try to correct that through information management and intelligence amplification, the noise that is just noise is also a deep problem. It grows even faster than the information overload, and we need ways of filtering it out. Without filtering there is nothing for the intelligence amplification to work on.

I think there are ways of achiving this. The current spam problem is producing an arms race of spam filters and new protocols to make spam uneconomical, sometimes based on immunological ideas. Similar methods might be useful (in more developed forms) for filtering academia, as an adjunct to peer review and other more traditional filters. Academic hoaxes are in many ways a form of vaccination against a particular danger. If we could find ways of reward people who discover holes in the scientific process (and reward people/institutions who fix the holes too!), then we would likely see much improvement in the same way open societies remain robust thanks to their constant internal criticism.

Academic hoaxing is a serious and important matter. But it is also fun, and so far the main driver for writing hoaxes seems to have been a sense of mischief. Let's hope we keep that in the future too.

Posted by Anders at January 11, 2004 04:08 PM