Misinformation Has High Perplexity

Abstract

Debunking misinformation is an important and time-critical task as there could be adverse consequences when misinformation is not quashed promptly. However, the usual supervised approach to debunking via misinformation classification requires human-annotated data and is not suited to the fast time-frame of newly emerging events such as the COVID-19 outbreak. In this paper, we postulate that misinformation itself has higher perplexity compared to truthful statements, and propose to leverage the perplexity to debunk false claims in an unsupervised manner. First, we extract reliable evidence from scientific and news sources according to sentence similarity to the claims. Second, we prime a language model with the extracted evidence and finally evaluate the correctness of given claims based on the perplexity scores at debunking time. We construct two new COVID-19-related test sets, one is scientific, and another is political in content, and empirically verify that our system performs favorably compared to existing systems. We are releasing these datasets publicly to encourage more research in debunking misinformation on COVID-19 and other topics.

References in corpus (6)

• REALM: Retrieval-Augmented Language Model Pre-Training
• Computational fact checking from knowledge networks
• Fake News Detection on Social Media using Geometric Deep Learning
• Language Models as Knowledge Bases?
• DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning
• UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification

Cited by in corpus (9)

• Dataset of Fake News Detection and Fact Verification: A Survey
• A Survey of COVID-19 Misinformation: Datasets, Detection Techniques and Open Issues
• Addressing contingency in algorithmic (mis)information classification: Toward a responsible machine learning agenda
• Zero-shot Fact Verification by Claim Generation
• Model Generalization on COVID-19 Fake News Detection
• Scientific Claim Verification with VERT5ERINI
• Types of Out-of-Distribution Texts and How to Detect Them
• Topic Modeling and Progression of American Digital News Media During the Onset of the COVID-19 Pandemic
• Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verification