240 citations · 833 across the 25 of their papers we have counts for
25 papers
What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
Meera Desai, Sang T. Truong, Hanna Wallach +8
Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reason…
Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture
Emma Harvey, Emanuel Moss, Hauke Sandhaus +2
Humans are increasingly expected to interact with AI systems that observe and make inferences about them - but do these systems actually work? A standard approach to answering this…
Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI
Joshua A. Kroll, Andrew Smart, R. Stuart Geiger +1
As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, a…
Validating LLMs in social science: Epistemic threats and emerging norms
Meera Desai, Dallas Card, Abigail Z. Jacobs
Large language models (LLMs) are reshaping social science methodology. Researchers increasingly prompt language models to generate quantitative measurements of social concepts, for…
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper +17
The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] a…
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research
A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen +34
"Machine unlearning" is a popular proposed solution for mitigating the existence of content in an AI model that is problematic for legal or moral reasons, including privacy, copyri…
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nicholas Pangakis +17
Across academia, industry, and government, there is an increasing awareness that the measurement tasks involved in evaluating generative AI (GenAI) systems are especially difficult…
Algorithmic Transparency and Participation through the Handoff Lens: Lessons Learned from the U.S. Census Bureau's Adoption of Differential Privacy
Amina A. Abdu, Lauren M. Chambers, Deirdre K. Mulligan +1
Emerging discussions on the responsible government use of algorithmic technologies propose transparency and public participation as key mechanisms for preserving accountability and…
The Cadaver in the Machine: The Social Practices of Measurement and Validation in Motion Capture Technology
Emma Harvey, Hauke Sandhaus, Abigail Z. Jacobs +2
Motion capture systems, used across various domains, make body representations concrete through technical processes. We argue that the measurement of bodies and the validation of m…
Report of the 1st Workshop on Generative AI and Law
A. Feder Cooper, Katherine Lee, James Grimmelmann +32
This report presents the takeaways of the inaugural Workshop on Generative AI and Law (GenLaw), held in July 2023. A cross-disciplinary group of practitioners and scholars from com…
An Empirical Analysis of Racial Categories in the Algorithmic Fairness Literature
Amina A. Abdu, Irene V. Pasquetto, Abigail Z. Jacobs
Recent work in algorithmic fairness has highlighted the challenge of defining racial categories for the purposes of anti-discrimination. These challenges are not new but have previ…
The Role of Relevance in Fair Ranking
Aparna Balagopalan, Abigail Z. Jacobs, Asia Biega
Online platforms mediate access to opportunity: relevance-based rankings create and constrain options by allocating exposure to job openings and job candidates in hiring platforms,…
Measurement as governance in and for responsible AI
Abigail Z. Jacobs
Measurement of social phenomena is everywhere, unavoidably, in sociotechnical systems. This is not (only) an academic point: Fairness-related harms emerge when there is a mismatch…
Internet-human infrastructures: Lessons from Havana's StreetNet
Abigail Z. Jacobs, Michaelanne Dye
We propose a mixed-methods approach to understanding the human infrastructure underlying StreetNet (SNET), a distributed, community-run intranet that serves as the primary 'Interne…
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
We propose measurement modeling from the quantitative social sciences as a framework for understanding fairness in computational systems. Computational systems often involve unobse…
Assembly in populations of social networks
Abigail Z. Jacobs
In-depth studies of sociotechnical systems are largely limited to single instances. Network surveys are expensive, and platforms vary in important ways, from interface design, to s…
Discovering heterogeneous subpopulations for fine-grained analysis of opioid use and opioid use disorders
Jen J. Gong, Abigail Z. Jacobs, Toby E. Stuart +1
The opioid epidemic in the United States claims over 40,000 lives per year, and it is estimated that well over two million Americans have an opioid use disorder. Over-prescription…
Untangling the roles of parasites in food webs with generative network models
Abigail Z. Jacobs, Jennifer A. Dunne, Cristopher Moore +1
Food webs represent the set of consumer-resource interactions among a set of species that co-occur in a habitat, but most food web studies have omitted parasites and their interact…
Assembling thefacebook: Using heterogeneity to understand online social network assembly
Abigail Z. Jacobs, Samuel F. Way, Johan Ugander +1
Online social networks represent a popular and diverse class of social media systems. Despite this variety, each of these systems undergoes a general process of online social netwo…
A unified view of generative models for networks: models, methods, opportunities, and challenges
Abigail Z. Jacobs, Aaron Clauset
Research on probabilistic models of networks now spans a wide variety of fields, including physics, sociology, biology, statistics, and machine learning. These efforts have produce…
Learning Latent Block Structure in Weighted Networks
Christopher Aicher, Abigail Z. Jacobs, Aaron Clauset
Community detection is an important task in network analysis, in which we aim to learn a network partition that groups together vertices with similar community-level connectivity p…
Efficiently inferring community structure in bipartite networks
Daniel B. Larremore, Aaron Clauset, Abigail Z. Jacobs
Bipartite networks are a common type of network data in which there are two types of vertices, and only vertices of different types can be connected. While bipartite networks exhib…
Adapting the Stochastic Block Model to Edge-Weighted Networks
Christopher Aicher, Abigail Z. Jacobs, Aaron Clauset
We generalize the stochastic block model to the important case in which edges are annotated with weights drawn from an exponential family distribution. This generalization introduc…
Detecting Friendship Within Dynamic Online Interaction Networks
Sears Merritt, Abigail Z. Jacobs, Winter Mason +1
In many complex social systems, the timing and frequency of interactions between individuals are observable but friendship ties are hidden. Recovering these hidden ties, particular…