2.1k citations · 2.9k across the 6 of their papers we have counts for
4 papers · 1 filter
Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety
Christopher M. Homan, Greg Serapio-Garcia, Lora Aroyo +5
Conversational AI systems exhibit a level of human-like behavior that promises to have profound impacts on many aspects of daily life -- how people access information, create conte…
DICES Dataset: Diversity in Conversational AI Evaluation for Safety
Lora Aroyo, Alex S. Taylor, Mark Diaz +5
Machine learning approaches often require training and evaluation datasets with a clear separation between positive and negative examples. This risks simplifying and even obscuring…
The Reasonable Effectiveness of Diverse Evaluation Data
Lora Aroyo, Mark Diaz, Christopher Homan +3
In this paper, we present findings from an semi-experimental exploration of rater diversity and its influence on safety annotations of conversations generated by humans talking to…
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation
Mark Diaz, Ian D. Kivlichan, Rachel Rosen +4
Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dat…