15 citations · 33 across the 6 of their papers we have counts for
8 papers
CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription
Nikita Pavlichenko, Ivan Stelmakh, Dmitry Ustalov
Domain-specific data is the crux of the successful transfer of machine learning systems from benchmarks to real life. In simple problems such as image classification, crowdsourcing…
Debiasing Evaluations That are Biased by Evaluations
Jingyan Wang, Ivan Stelmakh, Yuting Wei +1
It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference…
A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions
Ivan Stelmakh, Charvi Rastogi, Nihar B. Shah +2
Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisi…
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
Ivan Stelmakh, Nihar B. Shah, Aarti Singh +1
Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, i…
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Ivan Stelmakh, Nihar B. Shah, Aarti Singh +1
Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent…
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-ass…