End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models

Abstract

We propose end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (e.g., support, refute or not enough information), and to generate a statement to summarize and explain the reasoning and ruling process. To support this research, we construct Mocheg, a large-scale dataset consisting of 15,601 claims where each claim is annotated with a truthfulness label and a ruling statement, and 33,880 textual paragraphs and 12,112 images in total as evidence. To establish baseline performances on Mocheg, we experiment with several state-of-the-art neural architectures on the three pipelined subtasks: multimodal evidence retrieval, claim verification, and explanation generation, and demonstrate that the performance of the state-of-the-art end-to-end multimodal fact-checking does not provide satisfactory outcomes. To the best of our knowledge, we are the first to build the benchmark dataset and solutions for end-to-end multimodal fact-checking and explanation generation. The dataset, source code and model checkpoints are available at https://github.com/VT-NLP/Mocheg.

Accepted by SIGIR 23, 11 pages, 4 figures

References in corpus (3)

• Multimodal Emergent Fake News Detection via Meta Neural Process Networks
• Explain and Predict, and then Predict Again
• Ask to Know More: Generating Counterfactual Explanations for Fake Claims

Cited by in corpus (7)

• Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
• Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking
• Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
• The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers
• Multimodal Coherent Explanation Generation of Robot Failures
• A New Dataset and Benchmark for Grounding Multimodal Misinformation
• Exploring Content and Social Connections of Fake News with Explainable Text and Graph Learning