IMS Bulletin on refereeing

The March 2008 IMS (Institute of Mathematical Statistics) Bulletin issue has a special section discussing refereeing. The bulletin editor, Xuming He, introduces the matter on the front page of the issue while four present and past editors of statistics journals – John Marden, Michael Stein, Xiao-Li Meng and Rick Durrett – present their ideas about refereeing on the inside pages.

The discussion takes a predictable path – the writers describe and beseech what they perceive as good behavior on the side of referees (mostly focusing on promptness). Indeed, the choice of writers – established figures in the field – made it unlikely a-priori that a radical examination of the matter would be undertaken. The role of refereeing as gate-keeping is never questioned and the question of what is the objective of refereeing is not even raised by most writers (Marden is the exception here, see below). In view of the absence of the question of the objective, it is impossible to address fundamental questions – does refereeing serve the public (the public of researchers and the public at large) and whether there could be publication selection systems that are superior to refereeing – and refereeing, essentially in its present form and function, is presented as an immutable natural phenomenon.

The authors’ main concern seems to be the speed of review. Durrett’s piece, for example, is called “how long is too long” and is devoted almost entirely to the issue of speed. The authors make the obvious points that like any communication, refereeing should be timely and what “timely” implies in practice would vary. But the authors underplay the fact that in the current situation, where publication is a matter of academic survival, the matter of speed has implications that have nothing to do with the natural need of timeliness in communication.

Other topics include:

1. Stein makes the point that “it can be difficult to get prompt and helpful reviews of papers is that it is sometimes hard to see the benefit of being a good reviewer, and the cost of being a bad one.” This is a valid and important point (glossing over the fact that “good reviewer” and “bad reviewer” are not clearly defined, since the objective of reviewing has not been addressed). Stein tries to show his readers, being prospective reviewers, that “good” reviewing has some benefits (opportunity to learn about new work, developing the ability to think critically, impressing the editor), but admits that these don’t amount to much. He then mentions (approvingly it seems) that a prize by IMS for excellence in refereeing may be in the works. One would probably have to admit that this incentive – which is problematic in itself – is rather weak as well, since the chance of receiving such an award would be small, and the benefits in terms of the promoting of the winner’s career would be unclear. Durrett opines that nothing can be done about the inoperativeness of referees – since “refereeing is altruistic behavior”. Neither Stein nor Durrett cast their nets wide enough to consider changes in the refereeing system that are fundamental enough to re-align the incentives structure.
2. Stein makes the absurd statement that refereeing an article should take less than a working day and in many cases less than half a day. The idea that the typical submission can be understood within the space of a day well enough to pass informed judgment upon it is pure conceit. A paper may represent months or even years of work – intellectual and otherwise – on the side of the authors. Passing judgment upon it based on the consideration of a single day is impossible to do well and, if nothing else, rude. I will make a reservation to this blunt comment by noting that having an attitude such as Stein’s may be a matter of expediency. With all the incentives in the current systems being toward preferring quantity over quality, it may be that (a) the volume of submissions is such that referees have no choice but to put very little effort into refereeing, (b) papers in which very little is new, and thus do not require much work to understand, may be quite common. The problem is that by setting this hard, low limit, it is the authors of papers that do require some intellectual effort that are being treated unfairly.
3. Both Marden and Durrett (in his single non-speed related comment) suggest that reviewers avoid “proposing” (i.e., dictating) stylistic changes. This is generally a good idea, but offering it ignores the fundamental fact that, except in trivial cases, there is no objective way to decide what is a matter of style and what is a matter of substance. It probable that reviewers do not offer points on what they consider matters of style, but rather consider as substantive points that others (i.e., the authors) consider stylistic. It is quite natural for someone who finds himself in a position of power, especially one which is ostensibly bestowed on the basis of quality of intellect and knowledge, to consider their own taste as being a standard of quality.
4. Meng devotes his piece to the promotion of what may be seen as a system of apprenticeship in refereeing. He sees this system as a way to increase the speed of reviewing and to increase the role of young and less established academics in the system. The proposed system has the inherent advantages and disadvantages of an apprenticeship system, but it does little to address any serious issues with the current peer-review system, since it maintains the current power structure.
5. Marden makes the single comment in the four articles that can be viewed as touching upon the objective of refereeing. He writes: “What is your job as a referee? Help the author with research, help the editor with decisions.” And continues: “To the author, write with the idea that you are trying to help improve the paper.” It seems clear that this idea – the desired relationship of the referee with the author – an idea which seems natural and simple – is nowhere near the focus of attention of most working within the current system – including that of the 3 other commenters in the Bulletin issue. It is rather the relationship with the editor – as collaborators in the act of gatekeeping – which is in the forefront of attention. It is ironic, but deeply informative, that academic publishing is dominated by the notion of gatekeeping (i.e., censorship) rather than openness, and by exclusivity rather than collaboration. By rejecting the role of refereeing as an academic censorship mechanism, and embracing its role as a form of collaboration, a true solution to the ails of refereeing can be found.

Advertisement