A Survey on Automatic Credibility Assessment Using Textual Credibility Signals in the Era of Large Language Models
arXiv:2410.21360 · doi:10.1145/3770077
Abstract
In the age of social media and generative AI, the ability to automatically assess the credibility of online content has become increasingly critical, complementing traditional approaches to false information detection. Credibility assessment relies on aggregating diverse credibility signals - small units of information, such as content subjectivity, bias, or a presence of persuasion techniques - into a final credibility label/score. However, current research in automatic credibility assessment and credibility signals detection remains highly fragmented, with many signals studied in isolation and lacking integration. Notably, there is a scarcity of approaches that detect and aggregate multiple credibility signals simultaneously. These challenges are further exacerbated by the absence of a comprehensive and up-to-date overview of research works that connects these research efforts under a common framework and identifies shared trends, challenges, and open problems. In this survey, we address this gap by presenting a systematic and comprehensive literature review of 175 research papers, focusing on textual credibility signals within the field of Natural Language Processing (NLP), which undergoes a rapid transformation due to advancements in Large Language Models (LLMs). While positioning the NLP research into the the broader multidisciplinary landscape, we examine both automatic credibility assessment methods as well as the detection of nine categories of credibility signals. We provide an in-depth analysis of three key categories: 1) factuality, subjectivity and bias, 2) persuasion techniques and logical fallacies, and 3) check-worthy and fact-checked claims. In addition to summarising existing methods, datasets, and tools, we outline future research direction and emerging opportunities, with particular attention to evolving challenges posed by generative AI.
Accepted to ACM Transactions on Intelligent Systems and Technology (ACM TIST)
References in corpus (20)
- Survey of Hallucination in Natural Language Generation
- A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities
- A Survey on the Explainability of Supervised Machine Learning
- Factuality Challenges in the Era of Large Language Models
- Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations
- The State of Human-centered NLP Technology for Fact-checking
- Disinformation Capabilities of Large Language Models
- Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims
- Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification
- Identification of Biased Terms in News Articles by Comparison of Outlet-specific Word Embeddings
- Claim Detection for Automated Fact-checking: A Survey on Monolingual, Multilingual and Cross-Lingual Research
- SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)
- Reliability Criteria for News Websites
- A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness
- Persuasion with Large Language Models: A Survey of Empirical Evidence, Study Methodologies, and Ethical Implications
- Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation
- Verifying the Robustness of Automatic Credibility Assessment
- The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation
- Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language
- Developing a Natural Language Understanding Model to Characterize Cable News Bias