Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI

Abstract

While many Natural Language Processing (NLP) techniques have been proposed for fact-checking, both academic research and fact-checking organizations report limited adoption of such NLP work due to poor alignment with fact-checker practices, values, and needs. To address this, we investigate a co-design method, Matchmaking for AI, to enable fact-checkers, designers, and NLP researchers to collaboratively identify what fact-checker needs should be addressed by technology, and to brainstorm ideas for potential solutions. Co-design sessions we conducted with 22 professional fact-checkers yielded a set of 11 design ideas that offer a "north star", integrating fact-checker criteria into novel NLP design concepts. These concepts range from pre-bunking misinformation, efficient and personalized monitoring misinformation, proactively reducing fact-checker potential biases, and collaborative writing fact-check reports. Our work provides new insights into both human-centered fact-checking research and practice and AI co-design research.

Accepted at CSCW 2024

References in corpus (14)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Model Cards for Model Reporting
• A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities
• Out of One, Many: Using Language Models to Simulate Human Samples
• Hoaxy: A Platform for Tracking Online Misinformation
• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes
• Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes
• Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
• The State of Human-centered NLP Technology for Fact-checking
• TIMME: Twitter Ideology-detection via Multi-task Multi-relational Embedding
• AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction
• Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection
• Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection

Cited by in corpus (5)

• Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
• Exploring the Innovation Opportunities for Pre-trained Models
• Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
• OSINT Clinic: Co-designing AI-Augmented Collaborative OSINT Investigations for Vulnerability Assessment
• Exploring Multidimensional Checkworthiness: Designing AI-assisted Claim Prioritization for Human Fact-checkers