Generative AI and the future of scientometrics: current topics and future questions

Abstract

In this paper, we contribute to the debate on generative artificial intelligence (GenAI) in scientometrics. We argue that moving from a trial-and-error approach to an explainable and actionable use requires a principled understanding of strengths and weaknesses of GenAI as compared with other techniques and with human judgment. To this end, we introduce a conceptual framework based on the distinction between the semantic dimensions of texts, i.e. the meanings attributed to words, and their pragmatic dimension, i.e. their embedding within communicative situations. We leverage this framework to interpret the results of applications of GenAI in scientometrics and to provide guidance to users. Specifically, we conclude that key parameters to be considered are the nature of the task, the level of granularity of the analysis and whether the goal was descriptive, inferential or evaluative. These parameters lead to different strategies for using GenAI and human-machine integration. Finally, we suggest that, by generating large amounts of scientific language, GenAI might affect textual characteristics used to measure science, such as authors, words, and references. We argue that careful empirical work and theoretical reflection will be essential to remain capable of interpreting the evolving patterns of knowledge production in the age of AI.

Scientometrics (2026)

References in corpus (18)

• Generative AI
• ChatGPT: Jack of all trades, master of none
• Delving into LLM-assisted writing in biomedical publications through excess vocabulary
• Nonuniversal power law scaling in the probability distribution of scientific citations
• Scientific production in the era of Large Language Models
• Can ChatGPT evaluate research quality?
• Mapping the Increasing Use of LLMs in Scientific Papers
• The simulation of judgment in LLMs
• Large Knowledge Model: Perspectives and Challenges
• Exploring the applicability of Large Language Models to citation context analysis
• Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge
• Exploring the change in scientific readability following the release of ChatGPT
• When Large Language Models Meet Citation: A Survey
• Using language models to label clusters of scientific documents
• Generative AI for automatic topic labelling
• NLLG Quarterly arXiv Report 06/23: What are the most influential current AI Papers?
• Human-LLM Coevolution: Evidence from Academic Writing
• Large language models reshape the language of science