The State of Human-centered NLP Technology for Fact-checking
arXiv:2301.03056 · doi:10.1016/j.ipm.2022.103219
Abstract
Misinformation threatens modern society by promoting distrust in science, changing narratives in public health, heightening social polarization, and disrupting democratic elections and financial markets, among a myriad of other societal harms. To address this, a growing cadre of professional fact-checkers and journalists provide high-quality investigations into purported facts. However, these largely manual efforts have struggled to match the enormous scale of the problem. In response, a growing body of Natural Language Processing (NLP) technologies have been proposed for more scalable fact-checking. Despite tremendous growth in such research, however, practical adoption of NLP technologies for fact-checking still remains in its infancy today. In this work, we review the capabilities and limitations of the current NLP technologies for fact-checking. Our particular focus is to further chart the design space for how these technologies can be harnessed and refined in order to better meet the needs of human fact-checkers. To do so, we review key aspects of NLP-based fact-checking: task formulation, dataset construction, modeling, and human-centered strategies, such as explainable models and human-in-the-loop approaches. Next, we review the efficacy of applying NLP-based fact-checking tools to assist human fact-checkers. We recommend that future research include collaboration with fact-checker stakeholders early on in NLP research, as well as incorporation of human-centered design practices in model development, in order to further guide technology development for human use and practical adoption. Finally, we advocate for more research on benchmark development supporting extrinsic evaluation of human-centered fact-checking technologies.
Published in Machine and Human Factors in Misinformation Management -- a special issue of Information Processing and Management
References in corpus (16)
- Fake News Detection on Social Media: A Data Mining Perspective
- Attention is not Explanation
- XFake: Explainable Fake News Detector with Visualizations
- Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies
- Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria
- The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale
- FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information
- The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?
- NELA-GT-2020: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles
- Adversarial attacks against Fact Extraction and VERification
- Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms
- The Effects of Interactive AI Design on User Behavior: An Eye-tracking Study of Fact-checking COVID-19 Claims
- FaxPlainAC: A Fact-Checking Tool Based on EXPLAINable Models with HumAn Correction in the Loop
- SciClops: Detecting and Contextualizing Scientific Claims for Assisting Manual Fact-Checking
- Online Hate: Behavioural Dynamics and Relationship with Misinformation
- An ontological analysis of misinformation in online social networks