3 papers
cs.LG2025
When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity
Benjamin Feuer, Chiung-Yi Tseng, Astitwa Sarthak Lathe +2
LLM-judged benchmarks are increasingly used to evaluate complex model behaviors, yet their design introduces failure modes absent in conventional ground-truth based benchmarks. We…
cs.CY2025
Large language models that replace human participants can harmfully misportray and flatten identity groups
Angelina Wang, Jamie Morgenstern, John P. Dickerson
Large language models (LLMs) are increasing in capability and popularity, propelling their application in new domains -- including as replacements for human participants in computa…
cs.LG2025
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
Benjamin Feuer, Micah Goldblum, Teresa Datta +5
The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior…