11 papers
Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026
Sunnie S. Y. Kim, Wesley Hanwen Deng, Jennifer Wortman Vaughan +8
LLMs are rapidly reshaping peer review, making it important to understand how reviewers use them in practice and how different LLM-use policies affect review outcomes. We investiga…
Mitigating LLM-based p-Hacking by Preregistering for the Next LLM
Maria Thomas, Kristina Gligoric, Nihar B. Shah
Large language models (LLMs) are increasingly used to generate, classify, and annotate data whose outputs feed downstream hypothesis tests. However, LLM-based research is easy to p…
How Many Submissions May an Author Make? A Harmonic Quota for Submissions under Coauthorship
Nihar B. Shah
Research evaluation systems -- including journals, conferences, and funders -- are increasingly using author-level submission limits to manage growing submission loads. Most existi…
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5
Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist wi…
Smooth Partial Lotteries for Stable Randomized Selection
Alexander Goldberg, Giulia Fanti, Nihar B. Shah
Competitive selection processes, from scientific funding to admissions and hiring, use evaluations to score candidates, and eventually choose a subset of them based on those scores…
Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences
Madeline Celi Kitch, Nihar B. Shah
In many applications, human and LLM evaluators use assessments of relevant criteria to create an overall evaluation for an item or individual. For example, in admissions, committee…
Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable
Rounak Saha, Gurusha Juneja, Dayita Chaudhuri +3
A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers, except for polishing, paraphrasing, and grammar correction…
Detecting LLM-Generated Peer Reviews
Vishisht Rao, Aounon Kumar, Himabindu Lakkaraju +1
The integrity of peer review is fundamental to scientific progress, but the rise of large language models (LLMs) has introduced concerns that some reviewers may rely on these tools…
Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions
Jhih-Yi Hsieh, Aditi Raghunathan, Nihar B. Shah
In the peer review process of top-tier machine learning (ML) and artificial intelligence (AI) conferences, reviewers are assigned to papers through automated methods. These assignm…
Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA
John M. Carpenter, Andrea Corvillón, Nihar B. Shah
The increasing volume of papers and proposals that undergo peer review emphasizes the pressing need for greater automation to effectively manage the growing scale. In this study, w…
On the Detection of Reviewer-Author Collusion Rings From Paper Bidding
Steven Jecmen, Nihar B. Shah, Fei Fang +1
A major threat to the peer-review systems of computer science conferences is the existence of "collusion rings" between reviewers. In such collusion rings, reviewers who have also…