The latest issue of the IMS bulletin contains a letter in which a reader, Anirban DasGupta from Purdue University, lays out his thoughts regarding Ning-Zhong Shi’s MLE conjecture. Evidently, DasGupta’s comments were considered of higher relevance than my own letter to the bulletin’s editors regarding the conjecture, a letter which described the same counter-example to the conjecture that appears in the post linked above.

Read the rest of this entry »

Advertisement

The self-review system

July 2, 2008

This post presents a substitute to the peer-review system: a method to regulate the publication of scholarly work in journals. For background see: Academia as the industry of self promotion and Taking the self promotion out of academia.

Notification of intent to publish, waiting period, publishing quota

A person who intends to engage in research in a field and may wish to publish results, or who already has material he wishes to publish, will submit an indication of intent to publish to a central repository tracking research activity. In that indication of intent, the person submitting will indicate a field, or a small number of fields (up to, say, 4) in which he will be active. Each field is associated with a set of journals – the sets may overlap.

A person may, at any time, revise the set of fields in which he is active – but leaving the number of fields of activity below or at the limit (i.e., 4).

Once an indication of intent to publish in a certain field is given, a waiting period is in effect – say, 18 months. After that period, the researcher may publish in the journals covering that field at any time. A researcher may not publish more than a fixed amount of papers, say, 3, in a fixed amount of time, say 10 years. A multiple-author paper counts toward the quotas of each of the authors as 1/2 of a paper.

Rationale: The rationale for the imposition of a quota is discussed in the background documents above. The imposition of a wait period is aimed at minimizing abuse of the system by people who are not committed to sustained contemplation of a field of research, such as activists of various sorts.

Submission, refereeing

Once an author or a group of authors decide to publish in a certain journal, and if all the authors are within their quotas and beyond their waiting period, they submit a manuscript to the editors of the journal. If the authors so wishes, they may choose to go through a process of refereeing. The authors can choose a set of referees (up to, say, 3) from a list provided by the journal. The referees’ must excuse themselves promptly or write their reports within a set period. The authors may introduce changes to the paper after receiving the referees’ reports. If both the authors and referees are willing, further communication may take place.

Referees’ reports and any communications between the author and the referees are published together with the paper. If these are too long, a prefix is published together with a link to a full version.

Rationale: In the suggested scheme refereeing is seen as a service rendered by the research community to the individual researchers active in the field. Its role is not to block unworthy papers but to help authors produce better papers. Publishing referee-author communications benefits readers by providing third party comments on the paper, and guarantees that referees receive appropriate credit for their work.

Form of publication, page limit, automatic redaction, link to full version

The refereed manuscript (or the unrefereed manuscript, if the authors declines to take advantage of the refereeing process) is truncated at a certain amount of pages (say, 20) and scanned automatically for discouraged patterns (say, profanities or unpleasant typesetting), which are redacted. The author can then resubmit a corrected version or accept the automatically edited version as is. This final version is published in the journal chosen by the authors. If the author so wishes, he may provide an opt-in link (i.e., a link which requires a certain amount of attention to follow) to a full version of the paper which he would host and which is not constrained by the page limit and the pattern prohibitions.

Rationale: The rationale for the guaranteed publication is discussed in the background documents above. The page limit in the centrally hosted version is meant to encourage authors to make their points succinctly.

Data: Gasoline price, household count and median household income (Table H-5), consumption of gasoline

Update (Sept 1st, 2008): image updated to show 2007 data.
Update (Sept 10th, 2009): image updated to show 2008 data.
Update (Nov 2nd, 2010): image updated to show 2009 data.

The March 2008 IMS (Institute of Mathematical Statistics) Bulletin issue has a special section discussing refereeing. The bulletin editor, Xuming He, introduces the matter on the front page of the issue while four present and past editors of statistics journals – John Marden, Michael Stein, Xiao-Li Meng and Rick Durrett – present their ideas about refereeing on the inside pages.

The discussion takes a predictable path – the writers describe and beseech what they perceive as good behavior on the side of referees (mostly focusing on promptness). Indeed, the choice of writers – established figures in the field – made it unlikely a-priori that a radical examination of the matter would be undertaken. The role of refereeing as gate-keeping is never questioned and the question of what is the objective of refereeing is not even raised by most writers (Marden is the exception here, see below). In view of the absence of the question of the objective, it is impossible to address fundamental questions – does refereeing serve the public (the public of researchers and the public at large) and whether there could be publication selection systems that are superior to refereeing – and refereeing, essentially in its present form and function, is presented as an immutable natural phenomenon.

Read the rest of this entry »

Epistemology omnibus

February 22, 2008

This omnibus post collects draft fragments that are associated with the economy of attention and the sociology of science and expertise.

Read the rest of this entry »

The success of Wikipedia demonstrates that regular (i.e., unvetted by some authority) people are quite willing to spend time and effort on the creation of something that they perceive as valuable without expecting any material or even reputational compensation. It also shows that regular people can cooperate on a very large scale without a hierarchical organizing structure. Those points are probably the main reasons that significant hostility was directed toward Wikipedia from some quarters – some find the absence of Hobbesian chaos disconcerting.

Even Wikipedia’s own founder tried to claim that Wikipedia is an elite project, and while Swartz’s experiment disproving this claim is very interesting, it should be evident to anyone looking up information in Wikipedia on a regular basis that no small group of people could ever muster the wealth of knowledge that is stored on Wikipedia.

Read the rest of this entry »

The Golem

November 1, 2007

Following some heretical comments by me regarding the nature of scientific activity, I was referred by commenter “dsquared” to a book called “The Golem” by two sociologists, Harry Collins and Trevor Pinch. This fairly reasonable book makes the point that, despite pretenses by scientists and non-scientists, science is not a formal system in which crisp models generate crisp predictions which can then be neatly confirmed or refuted by experiments with unambiguous results. This is a good point, but the book somehow still manages to miss, or at least to obscure, the main issue – as dsquared’s summary of the “the basic conclusions of the literature” shows.

Read the rest of this entry »

Academic science, as it is today (and, it seems to me, for centuries past) is the industry of self promotion. The way the system works, academics must promote themselves. In order to get hired, promoted or funding – or even just not to be let go – they must present their work as being of great importance or novelty. Every new thought or wrinkle or nuance must be presented as a scientific advance worth telling to others, over and over again.

Researchers are thus driven toward emphasizing quantity over quality, toward over-interpreting their theory or data and toward unfruitful complexity (since complexity is harder to criticize and easier to produce than simplicity).

Under current conditions, these trends, which are harmful to the community, are driven by the need to survive of the individual researchers. This is then, the tragedy of the scientific commons.

Read the rest of this entry »

The quality of an idea is an objective property, but it is one that we usually do not possess any direct way of measuring. Since any natural language argument is informal, it is impossible to make a definite objective determination of which ideas are good (or correct) and which are bad (incorrect). This does not mean that there aren’t good and bad ideas, and better and worse ways to think about issues and to reach conclusions. It does means that care must be taken when evaluating ideas, that every piece of an argument is open for dispute or reinterpretation and that all conclusions must be taken as tentative.

Read the rest of this entry »

I have just found a short article of Killeen that appears to be a rejoinder to comments on his original p-rep paper. Unfortunately, I have not been able to find the comments to which Killeen is responding, but it appears, from the rejoinder (Error and Correction section), that the fundamental error that I referred to has been pointed out in one of the comments, Doros and Geier. It also seems that another comment by Macdonald pointed out that Killeen’s claim that “the probability of replication” can be calculated without knowing the unknown parameter of the distribution cannot be true.

Somehow, p-rep survived these issues. Superficially, this seems incredible. The entire analysis and rationale given by Killeen for p-rep in his original paper rest on an argument that was discovered to be wrong, and yet the ideas presented by that paper are still considered worth discussing, and even gain official support by the publishing establishment.

Read the rest of this entry »