240 citations · 833 across the 25 of their papers we have counts for
12 papers · 1 filter
What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
Meera Desai, Sang T. Truong, Hanna Wallach +8
Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reason…
Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture
Emma Harvey, Emanuel Moss, Hauke Sandhaus +2
Humans are increasingly expected to interact with AI systems that observe and make inferences about them - but do these systems actually work? A standard approach to answering this…
Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI
Joshua A. Kroll, Andrew Smart, R. Stuart Geiger +1
As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, a…
Validating LLMs in social science: Epistemic threats and emerging norms
Meera Desai, Dallas Card, Abigail Z. Jacobs
Large language models (LLMs) are reshaping social science methodology. Researchers increasingly prompt language models to generate quantitative measurements of social concepts, for…
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper +17
The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] a…
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nicholas Pangakis +17
Across academia, industry, and government, there is an increasing awareness that the measurement tasks involved in evaluating generative AI (GenAI) systems are especially difficult…
Algorithmic Transparency and Participation through the Handoff Lens: Lessons Learned from the U.S. Census Bureau's Adoption of Differential Privacy
Amina A. Abdu, Lauren M. Chambers, Deirdre K. Mulligan +1
Emerging discussions on the responsible government use of algorithmic technologies propose transparency and public participation as key mechanisms for preserving accountability and…
The Cadaver in the Machine: The Social Practices of Measurement and Validation in Motion Capture Technology
Emma Harvey, Hauke Sandhaus, Abigail Z. Jacobs +2
Motion capture systems, used across various domains, make body representations concrete through technical processes. We argue that the measurement of bodies and the validation of m…
Report of the 1st Workshop on Generative AI and Law
A. Feder Cooper, Katherine Lee, James Grimmelmann +32
This report presents the takeaways of the inaugural Workshop on Generative AI and Law (GenLaw), held in July 2023. A cross-disciplinary group of practitioners and scholars from com…
An Empirical Analysis of Racial Categories in the Algorithmic Fairness Literature
Amina A. Abdu, Irene V. Pasquetto, Abigail Z. Jacobs
Recent work in algorithmic fairness has highlighted the challenge of defining racial categories for the purposes of anti-discrimination. These challenges are not new but have previ…
Measurement as governance in and for responsible AI
Abigail Z. Jacobs
Measurement of social phenomena is everywhere, unavoidably, in sociotechnical systems. This is not (only) an academic point: Fairness-related harms emerge when there is a mismatch…
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
We propose measurement modeling from the quantitative social sciences as a framework for understanding fairness in computational systems. Computational systems often involve unobse…