p-rep, mathematical error, pure and simple

As promised, I have now spent some time reading the first few pages of Peter Killeen’s paper “An Alternative to Null-Hypothesis Significance Tests“. It turns out that Killeen really is utterly and honestly mistaken about his proposed statistic, the p-rep. His entire analysis is wrong because of a relatively simple mathematical error, that any statistician worth his salt could fish out in a few minutes.

The error is in the paragraph titled “Eliminating δ”, lying between equation (3) and equation (4). In it, there are two independent, identically distributed random variables, d₁′ and d₂′. They are each distributed normally with unknown mean δ and known standard deviation σ_d. Killeen also defines Δ₁ = d₁′ − δ and Δ₁ = d₂′ − δ. This means that d₂′ = d₁′ − Δ₁ + Δ_2.

At this point Killeen makes the elementary error of claiming that the above equation shows that d₂′ ~ N(d₁′, 2σ_d²). This is just wrong.

First, to get the terminology correct, since d₁′ is a random variable, it cannot be a parameter of a distribution. It seems that Killeen means that the conditional distribution of d₂′ given d₁′ is N(d₁′, 2σ_d²).

Now, this is clearly wrong, since as Killeen clearly states, d₁′ and d₂′ are independent, meaning that conditioning on d₁′ does not change the distribution of d₂′. Both before and after conditioning on d₁′, the distribution of d₂′ is N(δ, σ_d²).

That is it. The rest of the paper is following a wrong path, due to this mistake. At this point Killeen believes that given d₁′ he does not need to know the unknown δ in order to give the conditional distribution of d₂′.

He is, apparently quite convincing. He certainly misleads Geoff Cumming of the School of Psychological Science at La Trobe University, Victoria, Australia. Cumming’s comment is a statistical stew with what appears to be an unintentional Bayesian flavor, in which he turns things on their head by sampling δ, given d₁′.

Killeen is also convincing enough, or politically powerful enough, to get the journal Psychological Science to adopt his dubious statistic, which once again shows that reality is not self evident, not even to supposed experts.

On the semi-bright side, I have found a presentation discussing p-rep, by Jay Verkuilen and Clint Davis-Stober from the department of Psychology at the University of Illinois at Urbana-Champaign, in which they go through the usual complaints about p-values (including the indispensible photos of Fisher, Neyman and Pearson), and present p-rep as an up-and-coming alternative, although not without its shortcomings. They also mention that “there are a number of mathematical errors in the original article”, and advise to “read the corrigenda before you use it”. I don’t know what is in the corrigenda, but if it contains a correction of the fundamental mistake of the paper, it is hard to imagine what would remain.

Advertisement