312 citations · 406 across the 13 of their papers we have counts for
10 papers · 1 filter
On the state of reporting in crowdsourcing experiments and a checklist to aid current practices
Jorge Ramírez, Burcu Sayin, Marcos Baez +4
Crowdsourcing is being increasingly adopted as a platform to run studies with human subjects. Running a crowdsourcing experiment involves several choices and strategies to successf…
On the impact of predicate complexity in crowdsourced classification tasks
Jorge Ramírez, Marcos Baez, Fabio Casati +4
This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In…
Challenges and strategies for running controlled crowdsourcing experiments
Jorge Ramírez, Marcos Baez, Fabio Casati +2
This paper reports on the challenges and lessons we learned while running controlled experiments in crowdsourcing platforms. Crowdsourcing is becoming an attractive technique to en…
CrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designs
Jorge Ramírez, Simone Degiacomi, Davide Zanella +3
We present CrowdHub, a tool for running systematic evaluations of task designs on top of crowdsourcing platforms. The goal is to support the evaluation process, avoiding potential…
Understanding the Impact of Text Highlighting in Crowdsourcing Tasks
Jorge Ramírez, Marcos Baez, Fabio Casati +1
Text classification is one of the most common goals of machine learning (ML) projects, and also one of the most frequent human intelligence tasks in crowdsourcing platforms. ML has…
CrowdRev: A platform for Crowd-based Screening of Literature Reviews
Jorge Ramirez, Evgeny Krivosheev, Marcos Baez +2
In this paper and demo we present a crowd and crowd+AI based system, called CrowdRev, supporting the screening phase of literature reviews and achieving the same quality as author…