Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking

Abstract

The pervasiveness of large language models and generative AI in online media has amplified the need for effective automated fact-checking to assist fact-checkers in tackling the increasing volume and sophistication of misinformation. The complex nature of fact-checking demands that automated fact-checking systems provide explanations that enable fact-checkers to scrutinise their outputs. However, it is unclear how these explanations should align with the decision-making and reasoning processes of fact-checkers to be effectively integrated into their workflows. Through semi-structured interviews with fact-checking professionals, we bridge this gap by: (i) providing an account of how fact-checkers assess evidence, make decisions, and explain their processes; (ii) examining how fact-checkers use automated tools in practice; and (iii) identifying fact-checker explanation requirements for automated fact-checking tools. The findings show unmet explanation needs and identify important criteria for replicable fact-checking explanations that trace the model's reasoning path, reference specific evidence, and highlight uncertainty and information gaps.

Conditionally accepted to CHI'25

References in corpus (13)

• Model Cards for Model Reporting
• What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research
• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches
• What Large Language Models Know and What People Think They Know
• Factuality Challenges in the Era of Large Language Models
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
• The State of Human-centered NLP Technology for Fact-checking
• The Impact and Opportunities of Generative AI in Fact-Checking
• XAI in Automated Fact-Checking? The Benefits Are Modest and There's No One-Explanation-Fits-All
• Explainable Fact Checking with Probabilistic Answer Set Programming

Cited by in corpus (4)

• Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
• Autonomation, Not Automation: Activities and Needs of European Fact-checkers as a Basis for Designing Human-Centered AI Systems
• Beyond Community Notes: A Framework for Understanding and Building Crowdsourced Context Systems for Social Media
• Exploring Content and Social Connections of Fake News with Explainable Text and Graph Learning