SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers

Abstract

Novelty is a core component of academic papers, and there are multiple perspectives on the assessment of novelty. Existing methods often focus on word or entity combinations, which provide limited insights. The content related to a paper's novelty is typically distributed across different core sections, e.g., Introduction, Methodology and Results. Therefore, exploring the optimal combination of sections for evaluating the novelty of a paper is important for advancing automated novelty assessment. In this paper, we utilize different combinations of sections from academic papers as inputs to drive language models to predict novelty scores. We then analyze the results to determine the optimal section combinations for novelty score prediction. We first employ natural language processing techniques to identify the sectional structure of academic papers, categorizing them into introduction, methods, results, and discussion (IMRaD). Subsequently, we used different combinations of these sections (e.g., introduction and methods) as inputs for pretrained language models (PLMs) and large language models (LLMs), employing novelty scores provided by human expert reviewers as ground truth labels to obtain prediction results. The results indicate that using introduction, results and discussion is most appropriate for assessing the novelty of a paper, while the use of the entire text does not yield significant results. Furthermore, based on the results of the PLMs and LLMs, the introduction and results appear to be the most important section for the task of novelty score prediction. The code and dataset for this paper can be accessed at https://github.com/njust-winchy/SC4ANM.

References in corpus (14)

• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Longformer: The Long-Document Transformer
• PaLM: Scaling Language Modeling with Pathways
• What do we know about the disruption index in scientometrics? An overview of the literature
• Can ChatGPT evaluate research quality?
• Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness
• ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
• Feature engineering vs. deep learning for paper section identification: Toward applications in Chinese medical literature
• GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study
• Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities
• Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI
• Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of academic articles
• Reviewer2: Optimizing Review Generation Through Prompt Generation