44 citations · 223 across the 40 of their papers we have counts for
4 papers · 1 filter
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably
Mucahid Kutlu, Tyler McDonnell, Aashish Sheshadri +2
Crowdsourcing offers an affordable and scalable means to collect relevance judgments for IR test collections. However, crowd assessors may show higher variance in judgment quality…
But Who Protects the Moderators? The Case of Crowdsourced Image Moderation
Brandon Dang, Martin J. Riedl, Matthew Lease
Though detection systems have been developed to identify obscene content such as pornography and violence, artificial intelligence is simply not good enough to fully automate this…
Correlation and Prediction of Evaluation Metrics in Information Retrieval
Mucahid Kutlu, Vivek Khetan, Matthew Lease
Because researchers typically do not have the time or space to present more than a few evaluation metrics in any published study, it can be difficult to assess relative effectivene…
Efficient Test Collection Construction via Active Learning
Md Mustafizur Rahman, Mucahid Kutlu, Tamer Elsayed +1
To create a new IR test collection at low cost, it is valuable to carefully select which documents merit human relevance judgments. Shared task campaigns such as NIST TREC pool doc…