Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method

Hasti Narimanzadeh, Arash Badie-Modiri, Iuliia Smirnova +1

How to better reduce measurement variability and bias introduced by subjectivity in crowdsourced labelling remains an open question. We introduce a theoretical framework for unders…