28 citations · 85 across the 12 of their papers we have counts for
13 papers
A Gold Standard Dataset for the Reviewer Assignment Problem
Ivan Stelmakh, John Wieting, Sarina Xi +2
Many peer-review venues are using algorithms to assign submissions to reviewers. The crux of such automated approaches is the notion of the "similarity score" -- a numerical estima…
How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
Charvi Rastogi, Ivan Stelmakh, Alina Beygelzimer +7
How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,…
ASQA: Factoid Questions Meet Long-Form Answers
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra +1
An abundance of datasets and availability of reliable evaluation metrics have resulted in strong progress in factoid question answering (QA). This progress, however, does not easil…
Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review
Ivan Stelmakh, Charvi Rastogi, Ryan Liu +3
Citations play an important role in researchers' careers as a key factor in evaluation of scientific impact. Many anecdotes advice authors to exploit this fact and cite prospective…
To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online
Charvi Rastogi, Ivan Stelmakh, Xinwei Shen +4
Double-blind conferences have engaged in debates over whether to allow authors to post their papers online on arXiv or elsewhere during the review process. Independently, some auth…
CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription
Nikita Pavlichenko, Ivan Stelmakh, Dmitry Ustalov
Domain-specific data is the crux of the successful transfer of machine learning systems from benchmarks to real life. In simple problems such as image classification, crowdsourcing…
Debiasing Evaluations That are Biased by Evaluations
Jingyan Wang, Ivan Stelmakh, Yuting Wei +1
It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference…
A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions
Ivan Stelmakh, Charvi Rastogi, Nihar B. Shah +2
Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisi…
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
Ivan Stelmakh, Nihar B. Shah, Aarti Singh +1
Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, i…
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Ivan Stelmakh, Nihar B. Shah, Aarti Singh +1
Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent…
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-ass…
On Testing for Biases in Peer Review
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
We consider the issue of biases in scholarly research, specifically, in peer review. There is a long standing debate on whether exposing author identities to reviewers induces bias…
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to ma…