799 citations · 2.4k across the 46 of their papers we have counts for
9 papers · 1 filter
Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
Alexandra Chouldechova, A. Feder Cooper, Solon Barocas +3
We argue that conclusions drawn about relative system safety or attack method efficacy via AI red teaming are often not supported by evidence provided by attack success rate (ASR)…
Bridging Prediction and Intervention Problems in Social Systems
Lydia T. Liu, Inioluwa Deborah Raji, Angela Zhou +32
Many automated decision systems (ADS) are designed to solve prediction problems -- where the goal is to learn patterns from a sample of the population and apply them to individuals…
Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Luke Guerdan, Solon Barocas, Kenneth Holstein +3
The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAI) systems, plays a critical role in scaling and st…
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research
A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen +34
"Machine unlearning" is a popular proposed solution for mitigating the existence of content in an AI model that is problematic for legal or moral reasons, including privacy, copyri…
A Framework for Evaluating LLMs Under Task Indeterminacy
Luke Guerdan, Hanna Wallach, Solon Barocas +1
Large language model (LLM) evaluations often assume there is a single correct response -- a gold label -- for each item in the evaluation corpus. However, some tasks can be ambiguo…
On the Actionability of Outcome Prediction
Lydia T. Liu, Solon Barocas, Jon Kleinberg +1
Predicting future outcomes is a prevalent application of machine learning in social impact domains. Examples range from predicting student success in education to predicting diseas…
Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints
Jamelle Watson-Daniels, Solon Barocas, Jake M. Hofman +1
Prediction models have been widely adopted as the basis for decision-making in domains as diverse as employment, education, lending, and health. Yet, few real world problems readil…
Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification
A. Feder Cooper, Katherine Lee, Madiha Zahrah Choksi +6
Variance in predictions across different trained models is a significant, under-explored source of error in fair binary classification. In practice, the variance on some data examp…
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas +2
Transparency around limitations can improve the scientific rigor of research, help ensure appropriate interpretation of research findings, and make research claims more credible. D…