170 citations · 364 across the 16 of their papers we have counts for
10 papers · 1 filter
Perspectives on Large Language Models for Relevance Judgment
Guglielmo Faggioli, Laura Dietz, Charles Clarke +8
When asked, large language models (LLMs) like ChatGPT claim that they can assist with relevance judgments but it is not clear whether automated judgments can reliably be used in ev…
Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?
Mohammed Saeed, Nicolas Traub, Maelle Nicolas +2
Fact-checking is one of the effective solutions in fighting online misinformation. However, traditional fact-checking is a process requiring scarce expert human resources, and thus…
Managing Bias in Human-Annotated Data: Moving Beyond Bias Removal
Gianluca Demartini, Kevin Roitero, Stefano Mizzaro
Due to the widespread use of data-powered systems in our everyday lives, the notions of bias and fairness gained significant attention among researchers and practitioners, in both…
The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale
Michael Soprano, Kevin Roitero, David La Barbera +4
Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a bal…
Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19
Kevin Roitero, Michael Soprano, Beatrice Portelli +6
Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd…
The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?
Kevin Roitero, Michael Soprano, Beatrice Portelli +5
Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has…
Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background
Kevin Roitero, Michael Soprano, Shaoyang Fan +3
Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true a…
Proceedings of the KG-BIAS Workshop 2020 at AKBC 2020
Edgar Meij, Tara Safavi, Chenyan Xiong +3
The KG-BIAS 2020 workshop touches on biases and how they surface in knowledge graphs (KGs), biases in the source data that is used to create KGs, methods for measuring or remediati…
The Effect of Class Imbalance and Order on Crowdsourced Relevance Judgments
Rehab K. Qarout, Alessandro Checco, Gianluca Demartini
In this paper we study the effect on crowd worker efficiency and effectiveness of the dominance of one class in the data they process. We aim at understanding if there is any posit…
Pairwise, Magnitude, or Stars: What's the Best Way for Crowds to Rate?
Alessandro Checco, Gianluca Demartini
We compare three popular techniques of rating content: the ubiquitous five star rating, the less used pairwise comparison, and the recently introduced (in crowdsourcing) magnitude…