Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably

Mucahid Kutlu, Tyler McDonnell, Aashish Sheshadri +2

Crowdsourcing offers an affordable and scalable means to collect relevance judgments for IR test collections. However, crowd assessors may show higher variance in judgment quality…