Can social media provide early warning of retraction? Evidence from critical tweets identified by human annotation and large language models

Abstract

Timely detection of problematic research is essential for safeguarding scientific integrity. To explore whether social media commentary can serve as an early indicator of potentially problematic articles, this study analysed 3,815 tweets referencing 604 retracted articles and 3,373 tweets referencing 668 comparable non-retracted articles. Tweets critical of the articles were identified through both human annotation and large language models (LLMs). Human annotation revealed that 8.3% of retracted articles were associated with at least one critical tweet prior to retraction, compared to only 1.5% of non-retracted articles, highlighting the potential of tweets as early warning signals of retraction. However, critical tweets identified by LLMs (GPT-4o mini, Gemini 2.0 Flash-Lite, and Claude 3.5 Haiku) only partially aligned with human annotation, suggesting that fully automated monitoring of post-publication discourse should be applied with caution. A human-AI collaborative approach may offer a more reliable and scalable alternative, with human expertise helping to filter out tweets critical of issues unrelated to the research integrity of the articles. Overall, this study provides insights into how social media signals, combined with generative AI technologies, may support efforts to strengthen research integrity.

27 pages, 5 figures

References in corpus (6)

• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
• Dynamics of Cross-Platform Attention to Retracted Papers
• Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
• Differences in Personal and Professional Tweets of Scholars
• Evaluating the quality of published medical research with ChatGPT
• Do male leading authors retract more articles than female leading authors?