2 citations · 2 across the 3 of their papers we have counts for
4 papers
Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation
Thomas Reiter, Christoph Kern, Fedor Miasnikov +4
Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task d…
From Ground Truth to Measurement: A Statistical Framework for Human Labeling
Robert Chew, Stephanie Eckman, Christoph Kern +1
Supervised machine learning assumes that labeled data provide accurate measurements of the concepts models are meant to learn. Yet in practice, human labeling introduces systematic…
Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication
Stephanie Eckman, Bolei Ma, Christoph Kern +3
Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR…
Annotation Sensitivity: Training Data Collection Methods Affect Model Performance
Christoph Kern, Stephanie Eckman, Jacob Beck +3
When training data are collected from human annotators, the design of the annotation instrument, the instructions given to annotators, the characteristics of the annotators, and th…