Mike Sosteric: Interactive Peer Review: A Research Note

The EJS provides space for research notes like the one that appears below. Research notes should generally be less than 2000 words, short, concise and moderately well documented. Their purpose is to a) inform the scholarly community of planned or in progress research in order to b) lay preliminary intellectual claim to important research areas, and c) elicit commentary, feedback, and even collaboration. Publishing research notes in the electronic medium is a highly democratic alternative to the invisible college structure first noted by Price (1966) since the entire discipline, rather than just invisible networks of key researchers, has access to information on current activity. In addition research notes have the benefit (?) of obviating the need for travelling to conferences in order to remain current.

[email protected]

This short research notes describes a case study into interactive peer review currently being initiated at the EJS. After outlining past research and criticisms of the peer review process, I provide a methodological outline and set of guidelines for our six month case study. The goal of the project is two fold. We seek to assess the ability of information technology to “add value” to the research process at the critical juncture of the peer review process. We also seek to evaluate the potential for information technology to contribute to qualitative changes in the scientific communication process by embedding our study in larger debates about the information society.

…the traditional image of the scientist-as-writer, an image common both within science and without, is… a peculiarly humble and constricted one. Gone are the creativity and daring of the Titan. In their place is the self-effacing toil of the amanuensis, for, unlike poets, who are free to create the world anew as they write of it, scientists must be totally circumspect, must expunge every trace of their own personalities from their work, as they record only what is there. The scientist must view the world through some impersonal ocular device, the spectacles of objectivity, rather than through the eyes of a person. The scientist must weigh the world not in a hand but in a balance; must measure it not with a stride but with calipers. The Promethean scientist may uncover the secrets of the gods but must whisper them – like someone with an artificial larynx – in a voice devoid of emotion so as not, as the Fiesers put it for their fellow chemists, to “divert attention from the story we are trying to tell.” Above all, scientists dare not run naked through the streets shrieking “Eureka!” or they will, as the Fiesers say, “violat[e] principles of good usage.”
David Locke

Since the late 1950s, writers and government agencies have been commenting about the ‘crises’ in scholarly publication (Licklider, 1965; SATCOM, 1969, Vagianos and Kurmey, 1969). Most of those who have chosen to comment on this crises have focused on things like cost (McCarthy, 1994; Moline, 1989; Thompson, 1988), proliferation of journal titles (King, McDonald, Roderer and Wood, 1976) and concomitant fragmentation of research (Fox, 1994; Lock, 1994; Stossel, 1985), and bottlenecks in the process (Harnad, 1991). Others have focused more specifically on the peer review process. For example, some earlier challenges came from psychologists who found evidence of systematic bias in the peer review process (Abramowitz, Gomes and Abramowitz, 1975; Mahony, 1976; Simmons, 1978). Others, like Gardner (1981) and Grogan (1976) challenge the gatekeeping process of peer review by pointing out that the scientific communication process has not always proved a stopgap against ridiculous or bogus claims or by pointing out that major contributions to disciplines can be overlooked by even the most prestigious journals.

While all these are important considerations, philosophers of science have recently begun to criticize the epistemological foundations of formal scientific communication. Writers like Gross (1990), Locke (1992) and others have challenge the privileged position of ‘science’ as the road to truth by arguing that there is an underlying social and rhetorical dimension that needs to be considered. These authors further argue that the essential rhetorical nature of the scientific communication process is systematically underplayed or ignored, or that its actualization in the peer review process is lopsided. Yet scientists of all stripes prefer to see their disciplines as being capable of unproblematically referencing the natural or social world. Both Gross (1990) and Locke (1992) competently demolish the realist assumptions of science and replace, or more appropriately embed, the processes of scientific truth finding in a context that accepts and emphasizes the rhetorical nature of science.

The problems with peer review become evident once the fact that science has a rhetorical element is accepted. On the one hand, the traditional mode of peer review obscures the problems of reference and the rhetorical dimension of science. The rhetorical process which is at the heart of science and peer review conveniently disappears with the final publication of the manuscript. In its place is an ideal typical representation (the scientific paper) of the realist assumptions about empirical reference. All the academic world sees is a polished manuscript where the personal involvement of the researcher and reviewers has been systematically eliminated. The sanitized manuscript supports the myths of science by reinforcing the notion that science unproblematically references the natural world. “As a result of this process, there is cultivated a systematic neglect of the relationship between the claims in these reports and the process by which their truthfulness is initially certified: the move to publication systematically distorts the wholly argumentative grounding of the knowledge that peer review certifies” (Gross, 1990: p. 140).

The other problem arises when we consider Habermas’s ideal speech situation This theoretical construct describes the ideal type of interpersonal interaction that should exist in a rhetorical situation. Gross (1990) draws from Habermas in his description of the ideal speech situation. Gross describes the ideal speech situation in the following terms. 1) The ideal speech situation permits each interlocutor an equal opportunity to initiate speech. 2) There is mutual understanding between interlocutors. 3) There is space for clarification. 4) All interlocutors are equally free to use of any speech act. 5) There is equal power over the exchange. As applied in the context of peer review, Gross notes that ideally “scientific peer review would permit unimpeded authorial initiative, endless rounds of give and take, [and] unchecked openness among authors, editors, and referees. (Gross, 1990: 137).

Remembering the assumption that the peer review cycle is instrinsically rhetorical, this criticism of peer review now becomes evident. Peer review as it is traditionally practiced is not an ideal speech situation. That is, it does not conform to the dictates of good rhetorical practice. There is a strong element of hierarchy and authority in the process. Except for the initial act of initiating the peer review process, author’s almost never initiate speech. There is no opportunity for clarification since only one round of referee critique and editorial comment is possible. Flowing from this is the inability to ensure mutual understanding since the author can’t respond directly to reviewers, nor can reviewers query authors about specific problems with the submission. That is, there is no possibility for interactive clarification. Rather, authors must “reply to every question and respond to every criticism despite the fact that, were interaction possible, some questions might not have been asked, nor some criticisms made.” (p. 131). Further, “…authors are prohibited from issuing commands and inhibited from asking critical questions, while editors and referees do both freely.” (p. 133).

This set of peer review transactions can be analyzed in terms of the criteria of the ideal speech situation, which purports to be a set of suppositions unavoidable if we are to create rational consensus, to detect its presence, and to judge its quality. Perfect symmetry is required: each interlocutor must reveal himself, each must have every opportunity to initiate communication and to make assertions, and each must have equal control over the exchange; in the pursuit of rational consensus under these conditions, bias is revealed, countered, and neutralized. A peer review decision made under the conditions of the ideal speech situation represents a consensus among referees, editors, and authors that a particular paper is publishable science (Gross, 1960: 130; s.a. Habermas, 1992).

Accepting the fact that there is a rhetorical element is key. Not only do we move towards an acceptance of the underlying reality of the situation, but also as Gross indicates in the above quote, we install additional checks against bias. This applies particularly to the comments of reviewers. When the author is able to openly challenge certain statements as inappropriate, faulty, or even biased, the reviewers are forced to reconsider their comments in light of information or clarification provided by the author. Whether this leads to a more objective and fair process is a key question being addressed in this research.

There are related problems that emerge as a result of the inadequate realization of the rhetorical in the peer review process. One such problem is the resulting intransigent peer review process which discourages further development of a submission. Authors faced with a process where they can only listen and not respond are less likely to make suggested revisions and more likely to seek alternate outlets which ask less questions but publish more papers of questionable quality. This is unfortunate since peer reviewers may make useful suggestions and no doubt almost all submitted papers can benefit from at least one round of peer review. However the inability of the author, who it should be remembered can also be considered an expert in the field, to respond to criticisms that may be irrelevant, misplaced, or based on a faulty understanding of key sections of the manuscripts, virtually dooms the submission to mediocrity despite the appropriate criticisms of peer reviewers. An unfortunate result is that the time, energy, and work of both authors and reviewers is lost.

These criticisms go against the idealized versions of the scientific process propounded by writers like Popper (1968), Garvey (1979) and King et. al (1981). King et. al provides a representative statement of the idealized conception of scientific communication.

Overall the reporting cycle reflects a variety of modes of presentation, including direct and indirect, formal and informal, and oral and written. As the cycle progresses, the size of the audience reached and the degree of formality increases. With publication of a journal article, the research results are certified and made available to the entire scientific community. There is redundancy in this cycle, but for the most part it appears to be a necessary part of the interactive social process leading to the general acceptance of scientific research results (King, McDonald & Roderer, 1981: 6-7).

Whether dealing with bias in the peer review process, failures in the gatekeeping process, or criticisms of the peer review process, it is clear that an idealized version of the scientific communication process is untenable. However, the question is begged. Can we move in the direction of a more adequate and responsive system? And if we do make that move, what will be the benefits? Gross discards the possibility of an ideal speech situation in peer review and argues against the possibility of even approximating an adequate process. He further argues that moving in the direction of the ideal speech situation in peer review via conventional means would put undue strains on the whole process and would also require more time of the peer reviewers who would have to be engage in a critical debate for longer periods. “A decided movement in the direction of the ideal speech situation would improve an author’s position only at the expense of the professional time of editors and referees.” (Gross, 1990: 138).”

However, with the advent of information technology, it has become possible to move in the direction of repairing the peer review process, at least at the rhetorical level, by introducing a form of peer review that allows online dialogue and debate between authors and reviewers. In the next section I describe the methodology of the EJS case study into interactive peer review that seeks to determine whether interactive peer review can bring progressive qualitative changes to the peer review process and if so, what the nature of those progressive changes are.

PARTICIPANTS

Participants in this case study will include members of the board of directors of the EJS, and individuals who submit papers to the journal for publication consideration. Software has been installed that will allow participants in the study to make formal comments about submissions in a semiprivate manner. Members of the EJS board, and the authors themselves, will be allowed to enter into a ‘peer’ dialogue. Reviewers will submit their evaluations of particular papers, and authors will presumably respond to criticism.

GUIDELINES

Both reviewers and authors will have the option of remaining anonymous during the peer review process. Though the software chosen to conduct the study does offer algorithms for user validation, these will not be utilized. Those interested in examining the netforum software as it is called can access it through the home page of the EJS in the general commentary section. Links to the home of Netforum can be accessed through this, or any other active forum at the EJS.

Reviews will remain semiprivate until the final decision has been made. Only members of the EJS collective, and the authors’ themselves will be able to view and comment on reviews. However once the final decision has been made, the papers and the associated reviews will become public in the form of preprints and associated commentary.

The EJS collective has agreed to set a formal limit on the study of 15 papers or 6 months. Most likely it is the latter limitation that will bring the study to a close since it is unlikely that in the next 6 months we will receive 15 papers for consideration.

A mechanism for ongoing feedback in the form of a netforum newsgroups for participants will be provided to ensure adequate opportunity for comment.

DATA

There will be three sets of data collected for the purposes of analyzing the results. A complete transcription log of the interactions with members of the board concerning the study is being kept. Initially setting up the case study has involved coaxing members of the board into a discussion of the peer review, its problems, and some of the potential of information technology to overcome the asymmetric relations of power and prestige associated with traditional peer review and the lack of a voice for the author of papers. Perhaps it is needless to say that most members of the board remain skeptical about the potential for information technology to alter in any significant manner the underlying structures of authority and prestige. This skepticism is appropriate and is shared by this author. However for the purposes of this study, we retain an open mind concerning the potential of interactive peer review.

In addition to the preliminary exchanges between myself and the members of the board, I will also keep transcripts of the content of the peer review process, and of the authors formal responses to peer review comments. There will also be an online space for ongoing commentary by participants in the study.

A third tier of data collection will involve a short questionnaire designed to elicit the opinions of both peer reviewers and authors of submitted manuscripts about the process of interactive peer review. At this stage, I will be concerned with the personal reaction of participants to the study. Did they find the process differed in any way from the traditional methods of peer review? Did they believe that allowing “interaction” was beneficial? And in what way? Was there a noticeable shift in the dynamics of power and authority? Was the process more or less biased? It is useful to note that some members of the EJS have already commented that opening the peer review process may not have any substantial impact on the peer review process. Rather, the appearance of a progressive an open process may do more to obfuscate the relations of power an authority than it does to demolish them! This is a key point I think, and one that the questionnaire will seek to elicit answers to.

SUBMISSIONS

In order to attempt to overcome the problem of lack of submissions, two courses of action are being taken. I have recently placed a short paper entitled Electronic Journals and The Scholarly Communication Process: Issues and Notes, along with the comments of the EJS reviewers of this paper, up on the WWW for public access. It is my hoped that this will serve as a form of advertisement for the journal and will result in increased submissions. In addition, I have composed a research note [what you are reading now] of the study including a statement of goals, rationales, and hoped for outcomes. This will be published in the hope that this will garner participants for the study.

ANTICIPATED PROBLEMS

Although we are at an early stage in the study, we are anticipating several difficulties if we conceptualize the study as an experiment. First and foremost is the issue of generalizability. Because of the small sample size (n=15), our inability to control the quality of submissions, and random error, we must enter the usual caveats about generalizability (Schneider, 1996). However as Schneider (1996) noted, “this methodological weakness does not question the fact that the undertaking has the potential to produce substantial insight for the scientific community.” With these methodological weakness in mind, it is most appropriate to conceive of this research as an exploratory case study.

CONCLUSIONS

I am, by default, a critic of information technology. I generally react negatively to suggestions that technology will usher in a brave new world of economic, psychological, or intellectual freedom. Such technological utopianism as is evident in the work of numerous writers to insipid to cite has never amounted to more than a thinly veiled smoke screen covering the status quo.

However, I remain open about the possibility. If it is true that the implementation of technology is embedded and conditioned by social and structural factors (Noble, 1979), then it follows that the outcomes of the implementation of technology will depend not on some illusory nature of technology, but on the goals to which technology is directed and the manner in which it is implemented. This gives to technology not a deterministic, but an ambivalent and ultimately negotiable impact. Ultimately, it is to learn about the negotiation of technology and its progressive application (or potential application) to which we address ourselves in this study.

Fox, M. F. (1989). Disciplinary Fragmentation, Peer Review and the Publication Process. The American Sociologist, Summer, 188-191. Gardner, M. (1981). Science: Good, Bad and Bogus. New York: Prometheus Books. Garvey, W. D. (1979A). Communication: The Essence of Science. Toronto: Pergamon Press. Grogan, D. (1976). Science and Technology: An Introduction to the Literature. London: Clive Bingley. Gross, A. G. (1990). The Rhetoric of Science. Cambridge: Harvard University Press. Harnad, S. (1991). Post-Gutenburg Galaxy: The Fourth Revolution in the Means of Production of Knowledge. The Public-Access Computers Systems Review, 3, 39-53 [gopher://info.lib.uh.edu/00/articles/e-journals/uhlibrary/pacsreview/ v2/n1/harnad.2n1]. King, D. W., McDonald, D. D., Roderer, N. K., & Wood, B. L. (1976). Statistical Indicators of Scientific and Technical Communication 1960-1980. Rockville, MD: King Research Inc. Licklider, J. C. R. (1965). Libraries of the Future. Massachusetts: M.I.T. Press. McCarthy, P. (1994). Serial Killers: Academic Libraries Respond to Soaring Costs. Library Journal, 119, 41-44. Moline, S. R. (1989). The Influence of Subject, Publisher Type, and Quantity Published on Journal Prices. The Journal of Academic Librarianship, 15, 12-18. Popper, K. R. (1968). The Logic of Scientific Discovery. New York: Harper and Row. Schneider, Andreas (1996). Email Exchange Simmons, J. G. (1978). Changing Scientific Beliefs: Social Psychological Variables in the Peer Review Process. Baltimore: John Hopkins University. Sun, M. (1989). Peer Review Comes Under Peer Review. Science, 244, 910-914. Thompson, J. C. (1988). Journal Costs: Perception and Reality in the Dialogue. College and Research Libraries, 49, 481-2. Vagianos, L. G., & Kurmey, W. J. (1969). Scientific and Technical Information in Canada. Science Council of Canada. Ottawa: Queen’s Printer.

|Front Page|Table of Contents|