Academia as the industry of self promotion

The objective of academic activity in general, and of academic publishing in particular, is to further the career (i.e., salary and prestige) of the academics involved. It is a standard belief that peer review is the sieve that turns academic self-promotion into a productive scientific process. Often academics and others go so far as to state the much stronger belief that peer review is an efficient, or even the most efficient, method of fostering the creation of scientific work.

There are those academics in the natural sciences who see much of the social sciences as being nonsense. That is their view of the discipline, rather than of academic activity. The view that most academic activity (regardless of the field) is pure self-promotion and has nothing to do with science, or is even detrimental to science, is generally not heard in elite (e.g. acedemic) circles (at least not in those that I have been part of). Yet, as the system is set up, it would be very surprising if this were not so.

Advertisement

Expectations cannot be estimated

Expectations cannot be estimated, to any accuracy, with any sample size, unless stringent prior assumptions are made. With a sample of size n, there is a good chance (about 1/e^alpha) that there have been no samples lying in the top alpha/n of the distribution, meaning that the contribution of that part of the distribution is not represented in that sample. Since that contribution could be arbitrariliy large, there is no way to give a finite upper confidence bound of level higher than 1-1/e^alpha. This is true for any alpha, however small, showing that a UCB cannot be established at any level.

The standard way around this problem is to claim that the central limit theorem implies that the mean of a sample is distributed approximately normally. This is clearly untruein any rigorous sense – as the argument above shows – unless some distributional assumptions are made (a normal distribution, a bounded distribution, or some assumptions about the moments of the distributions). No such assumptions are usually explicitly made, although some nominal efforts to discard “outliers” are standard. In serious statistical circles, the authors and the readers are assumed to be adults who do not nitpick.

Where Kane went wrong, and an associated issue

Referring back to the seminal post of this blog, this post is aimed at pointing out where Kane’s analysis goes wrong. I have tried to make this point several times in the comments to “Deltoid” post relating to Kane’s paper, but for completeness, here it is again.

Kane implicitly attributes posterior distributions to the parameters being estimated in the Lancet paper – CMRpre, CMRpost, and RR. He then interprets the 95% confidence intervals stated for those parameters as covering 95% of the mass of those distributions. This reflects a fundamental misunderstanding of frequentist statistics, and of CIs in particular.

Since the parameters are not random variables, not only are statemens like “CMRpre ~ N(μpre, sigpre)” (p. 7) false, but also statements such as “P(CMRpre > 3.7)” (p. 11) are meaningless – CMRpre is either greater or less than 3.7, but no probabilities are involved.

Dispensing with Kane’s paper is easy, but in a way he does have a point – not about the Lancet paper, but about accepted statistical analysis. It is surprising that the available, or at least the widely applied, tools of statistics do not provide a standard, fool-proof, way to handle data sets such as that of the Lancet study.

The Lancet study data set contains samples from two unknown distributions over the positive numbers. One of those samples – those for CMRpost – contains a positive outlier, i.e., a data point that is much higher than all the other points. The Lancet paepr dropped that point in their analysis, implying (reasonably, but without making a rigorous argument) that this, if anything, biases the estimate downward.

There seems to be a clear need for a method of analysis that would be applicable to data sets with outliers. Current practice, relying on a set of ad-hoc methods (normal assumptions, dropping outliers, boot-strapping, robust statistics) is unable to handle such data sets in a rigorously convincing manner. In fact, in the absence of a rigorous definition of what an outlier is, standard practice is not rigorous for any data set.

Democratic theory and popularity of the organs of government

According to the standard theory of democracy – as articulated, for example, in The Federalist Papers – which stresses the importance of frequent elections as a guarantee of the government having “common interest with the people“, the Supreme Court should be viewed by the public with great suspicion. The Supreme Court is, after all, a powerful, non-representative, non-accountable body. Why is it then the most popular organ of the U.S. government?

The economy of attention

The cognitive limitations of humans put an upper bound on the number of ideas that any person can get to understand, in an hour, a day, a year, and over a lifetime. If the number of ideas occurring or being offered to a person exceeds the number that can be handled, some process of rejection needs to take place.

The preliminary classification of ideas – which are to be rejected outright and which deserve further consideration – needs to be quick and easy enough so that the process of classification will not by itself overwhelm the attention resources available. Rejecting ideas based on fully understanding their merits is clearly out of the question, since the object of the rejection process is precisely to avoid the need to fully understand the ideas.

Much of the world is mediated

Whatever is beyond our senses is not self evident. Much of the information about the non-self evident world is distributed through mass media. This, clearly, raises the possibility of manipulation, or at the very least, bias. The public is very much aware of this possibility – only about 25% of Americans state that their level of trust in newspapers and television is “a great deal” or “quite a lot”, with the rest opting for “some”, “very little” or “none”. The level of confidence has fallen over the years – 15 years ago the percentage of people with confidence in mass media was above 30.

The problem, however, is that there are many things about which there are no readily available alternative sources of information. The only simple alternative to trusting all the information in the mass media is therefore not trusting any of it. Trusting some of it and not trusting the rest is difficult because there is no easy way to distinguish between the credible and the suspect.

Arithmetic of confidence regions

The previous post defined confidence regions (CRs). As explained, the confidence condition is rather weak, and as a result CR algebra is not very rich. One thing that can be defined is CR arithmetic.

Let CA(X) be a level 1-alpha CR for a parameter A, and let B(X) be a level 1-beta CR for a parameter B, where both a and b are functions of the unknown distribution of X. Let C = C(A,B) be another parameter of the distribution of X – one which can be expressed as a function of the unknown A and B.

Confidence regions

In Bayesian statistics everything that is not known is random. That means that in the Bayesian framework statistics is simply an application of probability. Inference becomes simply a matter of finding conditional distributions. The desired unknowns are random variables – with known distributions – and what needs to be done is to find their conditional distribution given the observations.

In contrast, in frequentist statistics, some things are unknown but not random. The observations are assumed to be random, with a distribution that depends on the non-random unknowns. The objective of frequentist inference is to be able to say something interesting about the unknowns without the need to assume prior information about them.

Hello world!, David Kane, Pro Bono Statistics

While I have been contemplating filling a much needed void in the blogosphere with my own humble contribution, the immediate trigger for this blog is a set of threads in Tim Lambert’s blog, Deltoid: 1, 2, 3.

Those three threads discuss a paper by David Kane, in which he purports to prove that there is a mathematical contradiction in the 2004 paper by Roberts et al. in the Lancet discussing mortality in Iraq before and after the 2003 US-British invasion.

There are apparently a few scientists among the readers and the commenters of Deltoid, and they proceeded to address Kane’s paper. Most of the commenters consider Roberts et al. credible and were critical of Kane’s paper. Kane’s paper is weak on its substance (namely, Kane thinks that having a sample point with very high mortality – Fallujah – indicates that the mean mortality may be very low), and so it is only natural to try to address this weakness.

The problem is that Kane had what he presented as a mathematical argument proving his point, so he could claim that what seemed to his critics as a weakness of substance is nothing but a failure of their own intuition. In his first few responses in the comments he would even claim that his critics’ position is equivalent to stating that 2+2=5.

However, Kane’s mathematics are even weaker than his substance. This may seem surprising, since his substance seems to be wholly without merit. However, while his substantantive claim makes grammatical sense, his math does not – it is a complete mess. His entire mathematical argument is jibberish.

My attempts to make this point (commenting under the name “Sortition”), stating exactly why he does not make sense, never got any substantive responses from Kane. He was apparently honestly mistaken and truly believed that his arguements were correct. It seems that he gradually began to understand that his thinking may be not quite rigorous, but was unwilling to follow through and withdraw his paper and retract his conclusions until further consideration of his arguments.

This was only to be expected. Kane is ideologically committed to his conclusions, and has a personal stake to boot.

The more surprising thing was that I found it hard to get any attention from commenters who were critical of Kane. They were apparently as unaware as Kane was that Kane’s mathematical argument was entirely false, and my claims that it was were not considered credible. In my frustration, I wrote to Tim Lambert, making my point in an e-mail, and suggested that a professional statistician could make a valuable contribution to the issue. I got no response.

This called for radical action – and Pro Bono Statistics is the outcome.

In the following posts I hope to deal with many issues. One would be to point, once again, where Kane goes terribly wrong. More generally I intend to deal with statistics, as a theory, and as applied to real world issues. Even more generally, I intend to deal with other things that I think need dealing with but are not being dealt with satisfactorily, either in the mass media, or in blogs.