## p-rep, mathematical error, pure and simple

### September 15, 2007

As promised, I have now spent some time reading the first few pages of Peter Killeen’s paper “An Alternative to Null-Hypothesis Significance Tests“. It turns out that Killeen really is utterly and honestly mistaken about his proposed statistic, the p-rep. His entire analysis is wrong because of a relatively simple mathematical error, that any statistician worth his salt could fish out in a few minutes.

The error is in the paragraph titled “Eliminating δ”, lying between equation (3) and equation (4). In it, there are two independent, identically distributed random variables, *d*_{1}′ and *d*_{2}′. They are each distributed normally with unknown mean δ and known standard deviation σ* _{d. }*Killeen also defines Δ

_{1}=

*d*

_{1}′ − δ and Δ

_{1}=

*d*

_{2}′ − δ. This means that

*d*

_{2}′ =

*d*

_{1}′ − Δ

_{1}+ Δ

_{2.}

At this point Killeen makes the elementary error of claiming that the above equation shows that *d*_{2}′ ~ *N*(*d*_{1}′, 2σ_{d}^{2}). This is just wrong.

First, to get the terminology correct, since *d*_{1}′ is a random variable, it cannot be a parameter of a distribution. It seems that Killeen means that the *conditional* distribution of *d*_{2}′ given *d*_{1}′ is *N*(*d*_{1}′, 2σ_{d}^{2}).

Now, this is clearly wrong, since as Killeen clearly states, *d*_{1}′ and *d*_{2}′ are independent, meaning that conditioning on *d*_{1}′ does not change the distribution of *d*_{2}′. Both before and after conditioning on *d*_{1}′, the distribution of *d*_{2}′ is *N*(δ, σ_{d}^{2}).

That is it. The rest of the paper is following a wrong path, due to this mistake. At this point Killeen believes that given *d*_{1}′ he does not need to know the unknown δ in order to give the conditional distribution of *d*_{2}′.

He is, apparently quite convincing. He certainly misleads Geoff Cumming of the School of Psychological Science at La Trobe University, Victoria, Australia. Cumming’s comment is a statistical stew with what appears to be an unintentional Bayesian flavor, in which he turns things on their head by sampling δ, given *d*_{1}′.

Killeen is also convincing enough, or politically powerful enough, to get the journal *Psychological Science* to adopt his dubious statistic, which once again shows that reality is not self evident, not even to supposed experts.

On the semi-bright side, I have found a presentation discussing p-rep, by Jay Verkuilen and Clint Davis-Stober from the department of Psychology at the University of Illinois at Urbana-Champaign, in which they go through the usual complaints about p-values (including the indispensible photos of Fisher, Neyman and Pearson), and present p-rep as an up-and-coming alternative, although not without its shortcomings. They also mention that “there are a number of mathematical errors in the original article”, and advise to “read the corrigenda before you use it”. I don’t know what is in the corrigenda, but if it contains a correction of the fundamental mistake of the paper, it is hard to imagine what would remain.

September 16, 2007 at 7:18 am

[...] Killeen is responding, but it appears, from the rejoinder (Error and Correction section), that the fundamental error that I referred to has been pointed out in one of the comments, Doros and Geier. It also seems that another comment by [...]