## Academia as the industry of self promotion

### August 26, 2007

The objective of academic activity in general, and of academic publishing in particular, is to further the career (i.e., salary and prestige) of the academics involved. It is a standard belief that peer review is the sieve that turns academic self-promotion into a productive scientific process. Often academics and others go so far as to state the much stronger belief that peer review is an efficient, or even the most efficient, method of fostering the creation of scientific work.

There are those academics in the natural sciences who see much of the social sciences as being nonsense. That is their view of the discipline, rather than of academic activity. The view that most academic activity (regardless of the field) is pure self-promotion and has nothing to do with science, or is even detrimental to science, is generally not heard in elite (e.g. acedemic) circles (at least not in those that I have been part of). Yet, as the system is set up, it would be very surprising if this were not so.

## Expectations cannot be estimated

### August 24, 2007

Expectations cannot be estimated, to any accuracy, with any sample size, unless stringent prior assumptions are made. With a sample of size n, there is a good chance (about 1/e^alpha) that there have been no samples lying in the top alpha/n of the distribution, meaning that the contribution of that part of the distribution is not represented in that sample. Since that contribution could be arbitrariliy large, there is no way to give a finite upper confidence bound of level higher than 1-1/e^alpha. This is true for any alpha, however small, showing that a UCB cannot be established at any level.

The standard way around this problem is to claim that the central limit theorem implies that the mean of a sample is distributed approximately normally. This is clearly untruein any rigorous sense – as the argument above shows – unless some distributional assumptions are made (a normal distribution, a bounded distribution, or some assumptions about the moments of the distributions). No such assumptions are usually explicitly made, although some nominal efforts to discard “outliers” are standard. In serious statistical circles, the authors and the readers are assumed to be adults who do not nitpick.

## Where Kane went wrong, and an associated issue

### August 19, 2007

Referring back to the seminal post of this blog, this post is aimed at pointing out where Kane’s analysis goes wrong. I have tried to make this point several times in the comments to “Deltoid” post relating to Kane’s paper, but for completeness, here it is again.

Kane implicitly attributes posterior distributions to the parameters being estimated in the Lancet paper – CMRpre, CMRpost, and RR. He then interprets the 95% confidence intervals stated for those parameters as covering 95% of the mass of those distributions. This reflects a fundamental misunderstanding of frequentist statistics, and of CIs in particular.

Since the parameters are not random variables, not only are statemens like “CMRpre ~ N(μpre, sigpre)” (p. 7) false, but also statements such as “P(CMRpre > 3.7)” (p. 11) are meaningless – CMRpre is either greater or less than 3.7, but no probabilities are involved.

Dispensing with Kane’s paper is easy, but in a way he does have a point – not about the Lancet paper, but about accepted statistical analysis. It is surprising that the available, or at least the widely applied, tools of statistics do not provide a standard, fool-proof, way to handle data sets such as that of the Lancet study.

The Lancet study data set contains samples from two unknown distributions over the positive numbers. One of those samples – those for CMRpost – contains a positive outlier, i.e., a data point that is much higher than all the other points. The Lancet paepr dropped that point in their analysis, implying (reasonably, but without making a rigorous argument) that this, if anything, biases the estimate downward.

There seems to be a clear need for a method of analysis that would be applicable to data sets with outliers. Current practice, relying on a set of ad-hoc methods (normal assumptions, dropping outliers, boot-strapping, robust statistics) is unable to handle such data sets in a rigorously convincing manner. In fact, in the absence of a rigorous definition of what an outlier is, standard practice is not rigorous for any data set.

## Democratic theory and popularity of the organs of government

### August 18, 2007

According to the standard theory of democracy – as articulated, for example, in The Federalist Papers – which stresses the importance of frequent elections as a guarantee of the government having “common interest with the people“, the Supreme Court should be viewed by the public with great suspicion. The Supreme Court is, after all, a powerful, non-representative, non-accountable body. Why is it then the most popular organ of the U.S. government?

## The economy of attention

### August 17, 2007

The cognitive limitations of humans put an upper bound on the number of ideas that any person can get to understand, in an hour, a day, a year, and over a lifetime. If the number of ideas occurring or being offered to a person exceeds the number that can be handled, some process of rejection needs to take place.

The preliminary classification of ideas – which are to be rejected outright and which deserve further consideration – needs to be quick and easy enough so that the process of classification will not by itself overwhelm the attention resources available. Rejecting ideas based on fully understanding their merits is clearly out of the question, since the object of the rejection process is precisely to avoid the need to fully understand the ideas.

## Much of the world is mediated

### August 16, 2007

Whatever is beyond our senses is not self evident. Much of the information about the non-self evident world is distributed through mass media. This, clearly, raises the possibility of manipulation, or at the very least, bias. The public is very much aware of this possibility – only about 25% of Americans state that their level of trust in newspapers and television is “a great deal” or “quite a lot”, with the rest opting for “some”, “very little” or “none”. The level of confidence has fallen over the years – 15 years ago the percentage of people with confidence in mass media was above 30.

The problem, however, is that there are many things about which there are no readily available alternative sources of information. The only simple alternative to trusting all the information in the mass media is therefore not trusting any of it. Trusting some of it and not trusting the rest is difficult because there is no easy way to distinguish between the credible and the suspect.

## Arithmetic of confidence regions

### August 14, 2007

The previous post defined confidence regions (CRs). As explained, the confidence condition is rather weak, and as a result CR algebra is not very rich. One thing that can be defined is CR arithmetic.

Let CA(X) be a level 1-alpha CR for a parameter A, and let B(X) be a level 1-beta CR for a parameter B, where both a and b are functions of the unknown distribution of X. Let C = C(A,B) be another parameter of the distribution of X – one which can be expressed as a function of the unknown A and B.

## Confidence regions

### August 13, 2007

In Bayesian statistics everything that is not known is random. That means that in the Bayesian framework statistics is simply an application of probability. Inference becomes simply a matter of finding conditional distributions. The desired unknowns are random variables – with known distributions – and what needs to be done is to find their conditional distribution given the observations.

In contrast, in frequentist statistics, some things are unknown but not random. The observations are assumed to be random, with a distribution that depends on the non-random unknowns. The objective of frequentist inference is to be able to say something interesting about the unknowns without the need to assume prior information about them.