2 citations · 5 across the 5 of their papers we have counts for
6 papers
Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation
Thomas Reiter, Christoph Kern, Fedor Miasnikov +4
Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task d…
From Ground Truth to Measurement: A Statistical Framework for Human Labeling
Robert Chew, Stephanie Eckman, Christoph Kern +1
Supervised machine learning assumes that labeled data provide accurate measurements of the concepts models are meant to learn. Yet in practice, human labeling introduces systematic…
Bias in the Loop: How Humans Evaluate AI-Generated Suggestions
Jacob Beck, Stephanie Eckman, Christoph Kern +1
Human-AI collaboration increasingly drives decision-making across industries, from medical diagnosis to content moderation. While AI systems promise efficiency gains by providing a…
Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication
Stephanie Eckman, Bolei Ma, Christoph Kern +3
Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR…
Position: Insights from Survey Methodology can Improve Training Data
Stephanie Eckman, Barbara Plank, Frauke Kreuter
Whether future AI models are fair, trustworthy, and aligned with the public's interests rests in part on our ability to collect accurate data about what we want the models to do. H…
Annotation Sensitivity: Training Data Collection Methods Affect Model Performance
Christoph Kern, Stephanie Eckman, Jacob Beck +3
When training data are collected from human annotators, the design of the annotation instrument, the instructions given to annotators, the characteristics of the annotators, and th…