The Case for Claim Difficulty Assessment in Automatic Fact Checking

Abstract

Fact-checking is the process of evaluating the veracity of claims (i.e., purported facts). In this opinion piece, we raise an issue that has received little attention in prior work -- that some claims are far more difficult to fact-check than others. We discuss the implications this has for both practical fact-checking and research on automated fact-checking, including task formulation and dataset design. We report a manual analysis undertaken to explore factors underlying varying claim difficulty and identify several distinct types of difficulty. We motivate this new claim difficulty prediction task as beneficial to both automated fact-checking and practical fact-checking organizations.

References in corpus (4)

• On Calibration of Modern Neural Networks
• FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media
• A Review of Winograd Schema Challenge Datasets and Approaches
• MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims