Quality of ideas, peer review
October 7, 2007
The quality of an idea is an objective property, but it is one that we usually do not possess any direct way of measuring. Since any natural language argument is informal, it is impossible to make a definite objective determination of which ideas are good (or correct) and which are bad (incorrect). This does not mean that there aren’t good and bad ideas, and better and worse ways to think about issues and to reach conclusions. It does means that care must be taken when evaluating ideas, that every piece of an argument is open for dispute or reinterpretation and that all conclusions must be taken as tentative.
In general, the more thought, reflection, study, effort and time are applied when considering a certain idea, the more likely it is that a better understanding of it will be reached and that better arguments are constructed at to its quality. The problem is that all those resources are limited and cannot be applied fully to every possible idea. The economy of intellectual resources (and the associated material resources) is a system which encompasses, as its first tier, the economy of attention. Various strategies are used to economize on attention, but even when an idea has gotten past the initial screening and has gotten some attention, exploring the idea’s contents, validity and implications can be done at various levels of thoroughness. The policy of economizing the investment of intellectual resources necessarily continues to be applied.
The property of informality holds for arguments in applied science just as much as it holds in other contexts. (Arguments which take place in formal domains, such as logic and mathematics, can be made fully formal and therefore do not suffer from the problems associated with informality.) Hence, when evaluating a scientific idea or claim, there will not be a formal, verifiable way to test its veracity. Instead, the quality of the idea must be assessed carefully based on a subjective understanding that increases the more an idea is studied. Famously, a scientific assertion must be falsifiable, but what exactly constitutes a falsification remains informal, with a lot of room for subjective interpretation.
Peer review is an institutionalized tool in the economy of attention. Since it is a process of censoring, ideas that do not pass peer review are simply not considered by many who would at least give them some attention if they did pass the peer review process. Peer review may also be used in the subsequent tiers of the process of intellectual resource management – i.e., to determine which ideas, among all those that get to be considered at all, deserve more intellectual investment. Peer review is a seriously flawed tool for those purposes (both economizing attention and economizing more substantial intellectual investment), but since some sifting tools must be employed, using peer review for such purposes may be inevitable in some situations.
Peer review, however, is also sometimes used in a similar but different role. It is sometimes used as a direct indicator of the quality of ideas. It is one thing to ignore or not explore an idea because some referees did not like it, and another to claim that if the referees did not like it, then it must be false or not worth exploring. It is the second type of usage that George Monbiot employs when he dismisses the arguments of Alexander Cockburn regarding global warming (an index to the entire debate is here). Monbiot says: “a scientific claim carries no weight unless it has been published in a peer-reviewed scientific journal. Publication does not mean it is correct, but it does mean it is worth debating.”
Clearly, Monbiot thinks that the issue of Cockburn’s position is important enough to address – he has written a few articles about the matter. Thus, it is not a matter of economizing his intellectual resources. Monbiot uses peer review not as a rough (and biased) classification tool, but as an argument. This is wrong, and he is justifiably hammered by Cockburn (and his allies) on this subject. Of particular interest is David Noble’s post on the matter, giving Noble’s version of the history of peer review.
If Monbiot thinks countering Cockburn is important (and I am not sure it is, because Cockburn is so incoherent on this issue that even people who normally agree with him should find him unconvincing on this matter), then he should have used his connections among earth scientists to get a substantive evaluation of Cockburn’s arguments. Apparently Michael Mann had done that already – Monbiot should have referred to that rather than try to dismiss Cockburn’s arguments on procedural grounds.