2 citations · 4 across the 6 of their papers we have counts for
7 papers
Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation
Thomas Reiter, Christoph Kern, Fedor Miasnikov +4
Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task d…
From Ground Truth to Measurement: A Statistical Framework for Human Labeling
Robert Chew, Stephanie Eckman, Christoph Kern +1
Supervised machine learning assumes that labeled data provide accurate measurements of the concepts models are meant to learn. Yet in practice, human labeling introduces systematic…
Dialect and Gender Bias in YouTube's Spanish Captioning System
Iris Dania Jimenez, Christoph Kern
Spanish is the official language of twenty-one countries and is spoken by over 441 million people. Naturally, there are many variations in how Spanish is spoken across these countr…
Bias Begins with Data: The FairGround Corpus for Robust and Reproducible Research on Algorithmic Fairness
Jan Simson, Alessandro Fabris, Cosima Fröhner +2
As machine learning (ML) systems are increasingly adopted in high-stakes decision-making domains, ensuring fairness in their outputs has become a central challenge. At the core of…
Bias in the Loop: How Humans Evaluate AI-Generated Suggestions
Jacob Beck, Stephanie Eckman, Christoph Kern +1
Human-AI collaboration increasingly drives decision-making across industries, from medical diagnosis to content moderation. While AI systems promise efficiency gains by providing a…
Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication
Stephanie Eckman, Bolei Ma, Christoph Kern +3
Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR…
Annotation Sensitivity: Training Data Collection Methods Affect Model Performance
Christoph Kern, Stephanie Eckman, Jacob Beck +3
When training data are collected from human annotators, the design of the annotation instrument, the instructions given to annotators, the characteristics of the annotators, and th…