The Changing Face of Peer Review

To coincide with Peer Review Week Sept 16-20, this is an overview on current developments in peer review, with some thoughts on the future, and information on how Library Services can offer support to our researchers.

Three people sitting around a table talking to one person standing next to the table pointing with a pen at a tablet.

What is peer review and why is it important?

Peer review is the process by which scholarly work is submitted to the scrutiny of other experts in the same field. It’s thought to date back to the seventeenth century1, but has become increasingly standardised since the mid twentieth century2. It’s now an important part of the scholarly publications process, helping to assess and improve research papers before formal publication. A report published last year by Publons3 (part of Clarivate Analytics) found that peer review was overwhelmingly valued by researchers. There are different models of peer review, such as blind review (where authors and reviewers may not be known to each other) through to more open models of reviewing (see below, fig 2 in the Publons report)3.

Why is there a “peer review crisis”?

Peer review is far from perfect, however. Research that contains errors or fraud isn’t always picked up, and reviewers aren’t always objective: unconscious bias can affect peer review4, and even double blind reviewing isn’t always completely anonymous, especially in smaller fields where reviewers are more likely to be able to identify authors based on topic or writing style. Peer review also often goes unrewarded: reviewers are not usually paid for their work, and researchers may not cite this work as part of their scholarly profile when applying for jobs or promotions.

Recent research in PLoS One has also suggested that some reviewers can lazily accept low-quality manuscripts, bringing down the overall quality of research5.   That the website Retraction Watch exists highlights that peer review does not always fulfil the functions expected.

How is the open research agenda changing peer review?

Open peer review refers to a variety of different models that broadly support the principles of open research. The features of these models might include:

  • Named, identifiable reviewers.
  • Reviews that are published alongside the final article.
  • Participation by the wider community as opposed to just a small number of invited reviewers, whether on pre-review manuscripts or on the final version.
  • Direct discussion between authors and reviewers.
  • Reviews taking place on a different platform to publication6.

The different models have in common a desire to improve the peer review process, making it more transparent, accountable and accessible7.

Recent research has found that publishing peer review reports doesn’t compromise the review process, though only 8.1% of reviewers were willing to publish their identity alongside the report8.

Peer reviewing data

Data sharing has exploded in recent years. It is becoming commonplace in the academic publication process in light of the huge volumes of data being created in research and the challenges of irreproducible research. But while data sharing is becoming routine, peer review of data underlying publications is not always common.

Leading the way in data peer review are data journals. Data journals specialise in publishing descriptions of high value scientific datasets or analyses/meta-analyses of existing datasets. Submissions to data journals are peer-reviewed.

Other journals are quickly catching up. Peer reviewers may be asked to appraise the data underlying any publication, not just data-focused papers. Journals may have their own guidance for assessing datasets but PLOS provides some very practical criteria:

  1. Is the data accessible?
  2. Can you tell what you’re looking at?
  3. Does the data you see match the data referenced in the manuscript?
  4. Does the presentation of the data make sense?
  5. Does the data itself make sense?

The SGUL research data management service can help you to prepare your data for sharing and peer review. Contact us at researchdata@sgul.ac.uk for more information.

What might drive developments in the future to improve peer reviewing – for researchers, and for science?

Logo of DORA (Declaration on Research Assessment)

The San Francisco Declaration on Research Assessment of 2012, commonly known as DORA, and to which St George’s University of London is a signatory, sets out a statement of intent and some guiding principles around a move away from a narrow set of metrics such as journal impact factor as a measure of assessment. Acknowledging that researchers may undertake a wide range of scholarly activities, and produce outputs other than journal articles, could lead to better recognition of and reward for peer reviewing.

In 2017, the DOI provider Crossref announced that they would now support registering peer reviews as well as other types of research outputs9. Other services such as Publons and ORCiD10,11 also offer ways for researchers to track and get credit for their reviews, where these reviews are openly available12.  

Given the known problems with peer review, and the growing number of manuscript submissions, it’s no surprise that as noted by Nature13, publishers are starting to employ Artificial Intelligence to try and improve those processes that can be automated – without taking away from decision making by human editors. For example, Frontiers journals have announced the use of AI to help with quality control and reviewer identification14.

While as the Publons report finds, “the scholarly community lacks a robust measure of review quality”, more openness of the peer reviewing process, and wider use of identifiers to link reviewers and their reviews, could enable more analysis and agreement of what constitutes good peer review.

In conclusion, new technologies, publishing models and funder mandates present opportunities for the scientific community to improve the peer review process – a process which at its best allows researchers to engage in a constructive dialogue to improve research and the communication of research findings.

Queries about open research?

Contact us

CRIS & Deposit on acceptance: sora@sgul.ac.uk

Open Access Publications: openaccess@sgul.ac.uk

Research Data Management: researchdata@sgul.ac.uk

We look forward to hearing from you.

Michelle Harricharan, Research Data Support Manager
Jenni Hughes, Research Publications Assistant
Jennifer Smith, Research Publications Librarian

If you are interested receiving updates from the Library on all things open access, open data and scholarly research communications, you can subscribe to the Library Blog using the Follow button or click here for further posts from us.

References

1. Tennant JP, Dugan JM, Graziotin D et al. (2017) A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; peer review: 2 approved]. F1000Research, 6:1151 (https://doi.org/10.12688/f1000research.12037.3)

2. Ware M. Peer review: benefits, perceptions and alternatives. Publishing Research Consortium. 2008; p. 6. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.214.9676&rep=rep1&type=pdf [accessed 12/09/19]

3. Publons (2018) Global state of peer review https://doi.org/10.14322/publons.GSPR2018 [accessed 12/09/19]

4. Meadows, A (2018), “Eight Ways to Tackle Diversity and Inclusion in Peer Review” The Scholarly Kitchen. Available at https://scholarlykitchen.sspnet.org/2018/09/13/eight-ways-to-tackle-diversity-and-inclusion-in-peer-review/ [Accessed 12/09/19]

5. D’Andrea R, O’Dwyer JP (2017) “Can editors save peer review from peer reviewers?” PLoS ONE 12(10): e0186111. https://doi.org/10.1371/journal.pone.0186111 [accessed 12/9/19]

6. Ross-Hellauer, T (2017), “What is open peer review? A systematic review” [version 2; peer review: 4 approved]. F1000Research, 6:588 (https://doi.org/10.12688/f1000research.11369.2) (https://doi.org/10.12688/f1000research.11369.1)

7. Ross-Hellauer, T (2017), “Open peer review: bringing transparency, accountability and inclusivity to the peer review process”, LSE Impact Blog. Available at https://blogs.lse.ac.uk/impactofsocialsciences/2017/09/13/open-peer-review-bringing-transparency-accountability-and-inclusivity-to-the-peer-review-process/ [accessed 12/09/19]

8. Bravo, G; Grimaldo, F; López-Iñesta, E; Mehmani, B; Squazzoni, F (2019), “The effect of publishing peer review reports on referee behavior in five scholarly journals”, Nature Communications 10:322 https://doi.org/10.1038/s41467-018-08250-2

9. Lin, J (2017), “Peer reviews are open for registering at Crossref”. Available at: https://www.crossref.org/blog/peer-reviews-are-open-for-registering-at-crossref/ [accessed 12/09/19]

10. ORCID Support (2019), Peer Review https://support.orcid.org/hc/en-us/articles/360006971333-Peer-Review

11. PLOS Blog (2019), You’ve completed your review – now get credit with ORCID  https://blogs.plos.org/plos/2019/06/youve-completed-your-review-now-get-credit-with-orcid/ [accessed 16/09/2019]

12. Tennant, JP (2018), “The state of the art in peer review”, FEMS Microbiology Letters, Volume 365, Issue 19, fny204, https://doi.org/10.1093/femsle/fny204

13. Heaven, D (2018), “AI peer reviewers unleashed to ease publishing grind”, Nature 563, 609-610 (22 Nov 2018) http://dx.doi.org/10.1038/d41586-018-07245-9

14. Frontiers, Science News (2018) AI-enhanced peer review: Frontiers launches next generation of efficient, high-quality peer review Dec 14 2018; https://blog.frontiersin.org/2018/12/14/artificial-intelligence-peer-review-assistant-aira/