28 citations · 85 across the 12 of their papers we have counts for
5 papers · 1 filter
Debiasing Evaluations That are Biased by Evaluations
Jingyan Wang, Ivan Stelmakh, Yuting Wei +1
It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference…
A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions
Ivan Stelmakh, Charvi Rastogi, Nihar B. Shah +2
Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisi…
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
Ivan Stelmakh, Nihar B. Shah, Aarti Singh +1
Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, i…
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Ivan Stelmakh, Nihar B. Shah, Aarti Singh +1
Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent…
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-ass…