The Effect of Class Imbalance and Order on Crowdsourced Relevance Judgments

Rehab K. Qarout, Alessandro Checco, Gianluca Demartini

In this paper we study the effect on crowd worker efficiency and effectiveness of the dominance of one class in the data they process. We aim at understanding if there is any posit…