From the 1 of 6 linked papers with an AI index.
6 papers
Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture
Emma Harvey, Emanuel Moss, Hauke Sandhaus +2
Humans are increasingly expected to interact with AI systems that observe and make inferences about them - but do these systems actually work? A standard approach to answering this…
Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI
Joshua A. Kroll, Andrew Smart, R. Stuart Geiger +1
The paper examines how lessons from historic sociotechnical disasters can inform the design, risk assessment, and governance of AI systems, emphasizing organizational, social, and…
Validating LLMs in social science: Epistemic threats and emerging norms
Meera Desai, Dallas Card, Abigail Z. Jacobs
Large language models (LLMs) are reshaping social science methodology. Researchers increasingly prompt language models to generate quantitative measurements of social concepts, for…
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research
A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen +34
"Machine unlearning" is a popular proposed solution for mitigating the existence of content in an AI model that is problematic for legal or moral reasons, including privacy, copyri…
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper +17
The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] a…
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nicholas Pangakis +17
Across academia, industry, and government, there is an increasing awareness that the measurement tasks involved in evaluating generative AI (GenAI) systems are especially difficult…