240 citations · 833 across the 25 of their papers we have counts for
4 papers · 1 filter
What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
Meera Desai, Sang T. Truong, Hanna Wallach +8
Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reason…
Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture
Emma Harvey, Emanuel Moss, Hauke Sandhaus +2
Humans are increasingly expected to interact with AI systems that observe and make inferences about them - but do these systems actually work? A standard approach to answering this…
Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI
Joshua A. Kroll, Andrew Smart, R. Stuart Geiger +1
As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, a…
Validating LLMs in social science: Epistemic threats and emerging norms
Meera Desai, Dallas Card, Abigail Z. Jacobs
Large language models (LLMs) are reshaping social science methodology. Researchers increasingly prompt language models to generate quantitative measurements of social concepts, for…