3 papers
cs.CL2026
Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP
Yinuo Xu, David Jurgens
Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagree…
cs.CL2025
Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives
Yinuo Xu, Veronica Derricks, Allison Earl +1
We present an approach to modeling annotator disagreement in subjective NLP tasks through both architectural and data-centric innovations. Our model, DEM-MoE (Demographic-Aware Mix…
cs.CL2025
NUTMEG: Separating Signal From Noise in Annotator Disagreement
Jonathan Ivey, Susan Gauch, David Jurgens
NLP models often rely on human-labeled data for training and evaluation. Many approaches crowdsource this data from a large number of annotators with varying skills, backgrounds, a…