p-rep, mathematical error, pure and simple
September 15, 2007
As promised, I have now spent some time reading the first few pages of Peter Killeen’s paper “An Alternative to Null-Hypothesis Significance Tests“. It turns out that Killeen really is utterly and honestly mistaken about his proposed statistic, the p-rep. His entire analysis is wrong because of a relatively simple mathematical error, that any statistician worth his salt could fish out in a few minutes.
The error is in the paragraph titled “Eliminating δ”, lying between equation (3) and equation (4). In it, there are two independent, identically distributed random variables, d1′ and d2′. They are each distributed normally with unknown mean δ and known standard deviation σd. Killeen also defines Δ1 = d1′ − δ and Δ1 = d2′ − δ. This means that d2′ = d1′ − Δ1 + Δ2.
At this point Killeen makes the elementary error of claiming that the above equation shows that d2′ ~ N(d1′, 2σd2). This is just wrong.
First, to get the terminology correct, since d1′ is a random variable, it cannot be a parameter of a distribution. It seems that Killeen means that the conditional distribution of d2′ given d1′ is N(d1′, 2σd2).
Now, this is clearly wrong, since as Killeen clearly states, d1′ and d2′ are independent, meaning that conditioning on d1′ does not change the distribution of d2′. Both before and after conditioning on d1′, the distribution of d2′ is N(δ, σd2).
That is it. The rest of the paper is following a wrong path, due to this mistake. At this point Killeen believes that given d1′ he does not need to know the unknown δ in order to give the conditional distribution of d2′.
He is, apparently quite convincing. He certainly misleads Geoff Cumming of the School of Psychological Science at La Trobe University, Victoria, Australia. Cumming’s comment is a statistical stew with what appears to be an unintentional Bayesian flavor, in which he turns things on their head by sampling δ, given d1′.
Killeen is also convincing enough, or politically powerful enough, to get the journal Psychological Science to adopt his dubious statistic, which once again shows that reality is not self evident, not even to supposed experts.
On the semi-bright side, I have found a presentation discussing p-rep, by Jay Verkuilen and Clint Davis-Stober from the department of Psychology at the University of Illinois at Urbana-Champaign, in which they go through the usual complaints about p-values (including the indispensible photos of Fisher, Neyman and Pearson), and present p-rep as an up-and-coming alternative, although not without its shortcomings. They also mention that “there are a number of mathematical errors in the original article”, and advise to “read the corrigenda before you use it”. I don’t know what is in the corrigenda, but if it contains a correction of the fundamental mistake of the paper, it is hard to imagine what would remain.