Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?

Abstract

Fact-checking is one of the effective solutions in fighting online misinformation. However, traditional fact-checking is a process requiring scarce expert human resources, and thus does not scale well on social media because of the continuous flow of new content to be checked. Methods based on crowdsourcing have been proposed to tackle this challenge, as they can scale with a smaller cost, but, while they have shown to be feasible, have always been studied in controlled environments. In this work, we study the first large-scale effort of crowdsourced fact-checking deployed in practice, started by Twitter with the Birdwatch program. Our analysis shows that crowdsourcing may be an effective fact-checking strategy in some settings, even comparable to results obtained by human experts, but does not lead to consistent, actionable results in others. We processed 11.9k tweets verified by the Birdwatch program and report empirical evidence of i) differences in how the crowd and experts select content to be fact-checked, ii) how the crowd and the experts retrieve different resources to fact-check, and iii) the edge the crowd shows in fact-checking scalability and efficiency as compared to expert checkers.

References in corpus (5)

• Automated Fact Checking: Task formulations, methods and future directions
• Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria
• The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?
• The Rise of Guardians: Fact-checking URL Recommendation to Combat Fake News
• Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks

Cited by in corpus (12)

• Did the Roll-Out of Community Notes Reduce Engagement With Misinformation on X/Twitter?
• Retrieving false claims on Twitter during the Russia-Ukraine conflict
• CoSINT: Designing a Collaborative Capture the Flag Competition to Investigate Misinformation
• Designing and Evaluating Presentation Strategies for Fact-Checked Content
• Differential impact from individual versus collective misinformation tagging on the diversity of Twitter (X) information engagement and mobility
• Combating Misinformation in the Arab World: Challenges & Opportunities
• Community Fact-Checks Trigger Moral Outrage in Replies to Misleading Posts on Social Media
• OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence Investigations
• Who Checks the Checkers? Exploring Source Credibility in Twitter's Community Notes
• Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking
• Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
• Beyond Community Notes: A Framework for Understanding and Building Crowdsourced Context Systems for Social Media