Reviewing Peer Review: Problems and Potential

While dominant models of peer review exist today, these are joined by multiple calls for its revision and improvement. This section reviews the current debate on peer review for scientific journal publication and points out the key interrelated problems and potentials.

Problems

At its best, peer review should uphold the ethos of the academe in a fair and impartial manner. So, when critical issues surface revealing peer review as problematic, they tend to cut to the social and epistemological bone of its justification. There are several key and interrelated problems linked with the ethics and integrity of peer reviewing, which critics have brought to light. A comprehensive sweep of this literature is beyond the section’s scope, but it points out three salient and interrelated themes.

Ineffective peer review

Many critics claim that peer review is an inadequate system for catching methodological flaws, fabrications, falsifications, plagiarism, and other forms of research misconduct and deception (Resnik and Elmore 2016; Teixeira and Fontes Da Costa 2010; Ware 2011). Article retraction is one effect and indicator of peer review’s inability to catch misconduct, at least in the pre-publication phase (see Fresco-Santalla and Hernández-Pérez 2014). Another related critique is that it is slow and needlessly delays publication (Ware 2011). A more general critique is that its procedures are inconsistent, with diverse and even contradictory practices in play from journal to journal, which requires authors to expend resources on the strategic positioning of their submissions.

Misconduct in peer review

The critique of peer review often equates to concerns about the misconduct of editors and reviewers. For instance, Shaw (2015) suggests that reviewer anonymity, offered by some models of peer review, allows amply opportunity for poor behavior including biased evaluations and recommendations. Teixeira da Silva and Al-Khatib (2016) report a case where a publisher was forced to retract 32 articles after discovering that editors had fabricated reviewer evaluations to manipulate publication. In another example, Resnik and Elmore (2016) report an online study where author responses confirm instances of unfair reviewer behavior including personal attacks, confidentiality breaches, and demands to unnecessarily cite reviewer’s own work (i.e. reviewer ego-bias).

Biased peer review

Bias in peer review is another key area of critical debate. Lee et al. reflect that: “It is the impartial interpretation and application of shared norms and standards that make for a fair process, which — psychologically and epistemologically — legitimizes peer review outcomes, content, and institutions. This is why critics’ charge of bias in peer review is so troubling: Threats to the impartiality of review appear to threaten peer review’s psychological and epistemic legitimacy” (Lee et al. 2013, 3). In their article, Lee et al. survey a remarkable range of bias “genres” at play in the peer review process, which helps ground the description below (see also Resnik and Elmore 2016; Schwartzman 1997; Shaw 2015; Teixeira da Silva and Dobránszki 2015; Ware 2011).

Quality-related: This bias concerns the reviewers´ ability to objectively assess the true quality of submissions. Subgenres include: a) “deviation from proxy measures for true quality” where an article is submitted and rejected by one (top) journal but subsequently submitted and accepted by a different (top) journal; and b) “low inter-rater reliability” where evaluation discrepancies exist between two or more reviewers of the same article. (For further discussion of this issue, see Resnik and Elmore’s (2016) section: Inconsistent Review.)

Content-related: Disciplinary preferences or theoretical / methodological orientations towards certain content or approach can also bias the evaluation of submission. Subgenres include a) confirmation (or rejection) based on the aforementioned preferences or orientations (e.g. schools of thought); b) conservatism, which rejects ground-breaking / paradigm-shifting (innovative) approaches that challenge the status quo; c) interdisciplinary research, which may challenge disciplinary (mainstream) boundaries; and d) publishability, where research demonstrating positive rather than negative results is deemed more publishable.

Author-related: This genre includes bias linked with author identity or status, which negatively or positively play on the review of his or her article, including: a) prestige and class; b) institutional affiliation; c) national origin; d) language; or e) gender.

Reviewer-related: Here bias concerns how reviewer identity (prestige, class, institutional affiliation, national origin, language, gender, etc.) affects his or her review style or tendency to evaluate certain submission types more strictly or leniently than others. Similarly, reviewer ego bias may occur when submissions receive lower evaluations that fail to reference the reviewer’s own work (see content-related bias above). Conflicts of interest may also bias reviewer evaluation especially if gone undetected (see also Conflicts of Interest section).

Editor-related. From one perspective, peer review exists to offset or augment editorial power, but from another the editor retains complete control over the entire process, including the selection of reviewers (Guédon and Siemens 2002, in Fitzpatrick 2010). In both perspectives the editor role is key. Yet, distinguishing possible editor-bias in peer review depends on the degree to which editors employ a hands-off or hands-on approach. Editor bias can parallel reviewer bias, but it can also stem from the economics and politics (i.e. systemic) concerns of journal publication.

Potential

An increasing number of scholars have proposed ways to address the problems of peer review and its potential reform. For instance, Resnik and Elmore (2016) survey several recommendations found in the literature. These include increasing the number of referees (more than two per submission), the addition of referee training programmes, and improved referee instructions. Other recommendations focus more specifically on revising editor behavior to promote integrity and fairness in the peer review process. Editors can enhance referee selection and practice with more stringent referee recruitment and guidance, and by carefully checking reviewer assessments for unbiased, professional content. Editors can also remain transparent about decisions to accept or reject submissions for publication and offer clear rationale for both. Proposals for peer review reform also include alternatives to more traditional models.

Models of peer review

As noted above, multiple models of peer review exist. These differ widely between publisher and journal as well as the degree of transparency and the stage at which evaluations are made (Fresco-Santalla and Hernández-Pérez 2014). Peer review has typically involved pre-publication assessment by a small group of appointed reviewer(s) and editor(s). There are currently two dominant models:

Single-blind peer review: In this model, reviewer identities are kept from the authors, but reviewers know author identities.

Double-blind peer review: Here both authors and reviewers remain ignorant of one another’s identities.

The rationale with both the single-blind and double-blind models is to encourage reviewers to make honest assessments without concern for author redress. As Lee et al. (2013) explain, single-blind model is used most frequently because it is less burdensome and expensive to operate than the double-blind model, which requires more effort to mask all signs of author identity from the submission. On the other hand, the double-blind model is regarded by some scholars as the fairest because reviewers cannot make biased assessments against authors (Shaw 2015). However, Shaw warns that blinding the identity of reviewers renders them unaccountable - for instance, authors are unable to point out conflicts of interest.

Triple-blind peer review: In this case the identities of authors are blinded for both reviewers as well as editors. However, editors still know reviewer identity.

Non-blind or open peer review: This model of peer review is where both identities of authors and reviewers are known to each party. Several different versions exist, with which journals are currently experimenting, including online community pre- and post-publication stages (Fresco-Santalla and Hernández-Pérez 2014).

Shaw (2015) points out, however, that many open forms of peer review retain the problems of bias against authors. To counter this, he suggests a modified double-blind model where editors and reviewers are both blinded to author identity. He concludes that, “the peer review system should be based upon the principle that blinding should be used only to prevent bias in decision-making” (ibid: 4). We agree. Nevertheless, we encourage continued experimentation with open-access peer review, possibly hinged with non-binding pre- and/or post-publication online community consensus, performed in ways that protect author identity.

References

Biagioli, Mario. 2002. “From Book Censorship to Academic Peer Review.” Emergences: Journal for the Study of Media & Composite Cultures 12 (1): 11–45.

Fitzpatrick, Kathleen. 2010. “Peer‐to‐peer Review and the Future of Scholarly Authority.” Social Epistemology 24 (3): 161–79.

Fresco-Santalla, Ana, and Tony Hernández-Pérez. 2014. “Current and Evolving Models of Peer Review.” The Serials Librarian 67 (4): 373–98.

Lee, Carole J., Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. 2013. “Bias in Peer Review.” Journal of the American Society for Information Science and Technology 64 (1): 2–17.

McCarty, Lynn S., Christopher J. Borgert, and Ellen M. Mihaich. 2012. “Information Quality in Regulatory Decision Making: Peer Review versus Good Laboratory Practice.” Environmental Health Perspectives 120 (7): 927–34.

Resnik, David B., and Susan A. Elmore. 2016. “Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors.” Science and Engineering Ethics 22 (1): 169–88.

Schwartzman, Roy. 1997. “Peer Review as the Enforcement of Disciplinary Orthodoxy.” Southern Communication Journal 63 (1): 69–75.

Shaw, David. M. 2015. “Blinded by the Light: Anonymization Should Be Used in Peer Review to Prevent Bias, Not Protect Referees.” EMBO Reports 16 (8): 894–97.

Teixeira, Aurora AC, and Mariana Fontes Da Costa. 2010. “Who Rules the Ruler? On the Misconduct of Journal Editors.” Journal of Academic Ethics 8 (2): 111–28.

Teixeira da Silva, Jaime A, and Aceil Al-Khatib. 2016. “How Are Editors Selected, Recruited and Approved?” Science and Engineering Ethics, 1–4.

Teixeira da Silva, Jaime A., and Judit Dobránszki. 2015. “Problems with Traditional Science Publishing and Finding a Wider Niche for Post-Publication Peer Review.” Accountability in Research 22 (1): 22–40.

Ware, Mark. 2011. “Peer Review: Recent Experience and Future Directions.” New Review of Information Networking 16 (1): 23–53.