p-rep and the myth of rational science
September 16, 2007
I have just found a short article of Killeen that appears to be a rejoinder to comments on his original p-rep paper. Unfortunately, I have not been able to find the comments to which Killeen is responding, but it appears, from the rejoinder (Error and Correction section), that the fundamental error that I referred to has been pointed out in one of the comments, Doros and Geier. It also seems that another comment by Macdonald pointed out that Killeen’s claim that “the probability of replication” can be calculated without knowing the unknown parameter of the distribution cannot be true.
Somehow, p-rep survived these issues. Superficially, this seems incredible. The entire analysis and rationale given by Killeen for p-rep in his original paper rest on an argument that was discovered to be wrong, and yet the ideas presented by that paper are still considered worth discussing, and even gain official support by the publishing establishment.
On the immediate level, Killeen maneuvers around the errors by invoking a Bayesian argument, claiming his p-rep estimate is the result of a Bayesian analysis with flat priors. This flies in the face of the anti-Bayesian stance in Killeen’s original paper (for example, in the abstract, Killeen claims that he avoids the “Bayesian dilemma”). I guess the audience is expected to be either too ignorant or too polite to mention this about-turn.
On a deeper level, however, the persistence of p-rep is another demostration of the fact that science is primarily a political activity rather than a rational activity.
I cannot determine exactly what is the political constellation that makes discarding p-rep inconvenient. An important factor is probably that among the audience of the paper, only a small minority could detect or understand the error. It seems reasonable to speculate that we are witnessing the effect told about in the tale of The Emperor’s New Clothes: Killeen’s original paper generated enough interest that officially dismissing it six months after it was published would have embarassed some of the interests involved (the author, the reviewers and the publishers are obvious examples). It is more convenient therefore, at this point, to pretend that the errors found are not of any consequence than to officially admit that such basic mistakes were published in a leading psychological journal and taken seriously by many.
I am not sure if that would be enough to explain why Pschologycal Science (the same journal in which the 2005 paper appeared) took the extra step of making publishing p-rep official policy. It could be that the wheels were already in motion by the time the mistakes were discovered. It could be that whoever makes such decisions in the paper was not able to detect that the case for p-rep was completely undermined, and that no one wanted to be the one to disclose such information.
Other scenarios may be possible, but it is quite clear that none of them would follow the ostensible rules of scientific conduct.