Exploring the applicability of Large Language Models to citation context analysis

Abstract

Unlike traditional citation analysis -- which assumes that all citations in a paper are equivalent -- citation context analysis considers the contextual information of individual citations. However, citation context analysis requires creating large amounts of data through annotation, which hinders the widespread use of this methodology. This study explored the applicability of Large Language Models (LLMs) -- particularly ChatGPT -- to citation context analysis by comparing LLMs and human annotation results. The results show that the LLMs annotation is as good as or better than the human annotation in terms of consistency but poor in terms of predictive performance. Thus, having LLMs immediately replace human annotators in citation context analysis is inappropriate. However, the annotation results obtained by LLMs can be used as reference information when narrowing the annotation results obtained by multiple human annotators to one, or LLMs can be used as one of the annotators when it is difficult to prepare sufficient human annotators. This study provides basic findings important for the future development of citation context analyses.

References in corpus (10)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• A Survey on Evaluation of Large Language Models
• ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification
• Automated Annotation with Generative AI Requires Validation
• Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
• Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark
• Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs
• Towards Coding Social Science Datasets with Language Models
• When Large Language Models Meet Citation: A Survey

Cited by in corpus (3)

• Leveraging GANs for citation intent classification and its impact on citation network analysis
• Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learning
• Generative AI and the future of scientometrics: current topics and future questions