Automated scholarly paper review: Concepts, technologies, and challenges

Abstract

Peer review is a widely accepted mechanism for research evaluation, playing a pivotal role in academic publishing. However, criticisms have long been leveled at this mechanism, mostly because of its poor efficiency and low reproducibility. Recent years have seen the application of artificial intelligence (AI) in assisting the peer review process. Nonetheless, with the involvement of humans, such limitations remain inevitable. In this paper, we propose the concept and pipeline of automated scholarly paper review (ASPR) and review the relevant literature and technologies of achieving a full-scale computerized review process. On the basis of the review and discussion, we conclude that there is already corresponding research and preliminary implementation at each stage of ASPR. We further look into the challenges in ASPR with the existing technologies. The major difficulties lie in inadequate data, imperfect document parsing and representation, defective human$\unicode{x2013}$computer interaction, and flawed deep logical reasoning. Moreover, we point out the future directions and discuss the possible moral and ethical issues of ASPR. In the foreseeable future, ASPR and peer review will coexist in a reinforcing manner before ASPR is able to fully undertake the reviewing workload from humans.

arXiv commentPlease cite the version of Information Fusion

References in corpus (14)

• Artificial Intelligence: the global landscape of ethics guidelines
• The Ethics of AI Ethics -- An Evaluation of Guidelines
• A Survey of Large Language Models
• Pre-trained Models for Natural Language Processing: A Survey
• Dota 2 with Large Scale Deep Reinforcement Learning
• Competition-Level Code Generation with AlphaCode
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• DocBERT: BERT for Document Classification
• Linguistic generalization and compositionality in modern artificial neural networks
• Comparing the writing style of real and artificial papers
• Artificial intelligence in peer review: How can evolutionary computation support journal editors?
• How many preprints have actually been printed and why: a case study of computer science preprints on arXiv
• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
• SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Prediction

Cited by in corpus (4)

• Large language models for automated scholarly paper review: A survey
• Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge
• MOPRD: A multidisciplinary open peer review dataset
• Aspect-Based Sentiment Evolution and its Correlation with Review Rounds in Multi-Round Peer Reviews: A Deep Learning Approach