September 16, 2009
Getting rid of science noise
I dare to disagree with Ben Goldacre (I think): Practical Ethics: Academic freedom isn't free - I look at the debate over the aids-denialism papers in Medical Hypotheses. My conclusion is broadly that while the journal is filled with junk, it is a good thing that it exists. The way of dealing with bad science is open criticism, scientific connoisseurship and developing technical tools for supporting rational debate. It is simply not effective to try to suppress error at one single stage.
Here is a tool that I think should be developed: a support/rebuttal detector. Parsing scientific papers to see whether they actually rebut the science in another paper is AI-complete. However, it is likely feasible given the standardized style of scientific communication to do something like sentiment detection to make a probabilistic estimate of whether a text claims to rebut or support another text. Add this to large scientific publication databases, and make a rottentomatoes.com-like composite index that shows up next to PubMed searches or bibliographies (or even, if activated, as mouseover indicators for references in the text itself). I think this would be tricky but feasible, and even if it would just give a probabilistic measure of support it would still help judging papers.
The next step is of course to make this work for all sorts of claims.
Posted by Anders3 at September 16, 2009 07:07 PM