799 citations · 2.4k across the 46 of their papers we have counts for
28 papers · 1 filter
Beyond "I Can't Help With That": How Child Safety Experts Evaluate AI Chatbot Safety
Hannah Cha, Neha Shukla, Solon Barocas +3
Youth increasingly turn to AI chatbots for social and emotional support, raising concerns about how these systems respond, especially in high-stakes situations. However, existing c…
The Fair Lending Model: How the Longest-Running Algorithmic Fairness Programs Work in Practice
Emily Black, Miranda Bogen, Logan Koepke +3
U.S. financial institutions subject to fair lending laws have been running algorithmic fairness programs for decades. Despite this long history, remarkably little is known about ho…
Effects of Generative AI Errors on User Reliance Across Task Difficulty
Jacy Reese Anthis, Hannah Cha, Solon Barocas +2
The capabilities of artificial intelligence (AI) lie along a jagged frontier, where AI systems surprisingly fail on tasks that humans find easy and succeed on tasks that humans fin…
Statistical Guarantees in the Search for Less Discriminatory Algorithms
Chris Hays, Ben Laufer, Solon Barocas +1
U.S. discrimination law can impose liability on firms that fail to adopt a less discriminatory alternative (LDA): a decision policy that achieves the same business objectives while…
Distinguishing Task-Specific and General-Purpose AI in Regulation
Jennifer Wang, Andrew Selbst, Solon Barocas +1
Over the past decade, policymakers have developed a set of regulatory tools to ensure AI development aligns with key societal goals. Many of these tools were initially developed in…
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn +7
In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly…
AI Automatons: AI Systems Intended to Imitate Humans
Alexandra Olteanu, Solon Barocas, Su Lin Blodgett +3
There is a growing proliferation of AI systems designed to mimic people's behavior, work, abilities, likenesses, or humanness -- systems we dub AI automatons. Individuals, groups,…
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper +17
The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] a…
A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impacts
Alexandra Chouldechova, Chad Atalla, Solon Barocas +11
The valid measurement of generative AI (GenAI) systems' capabilities, risks, and impacts forms the bedrock of our ability to evaluate these systems. We introduce a shared standard…
What Constitutes a Less Discriminatory Algorithm?
Benjamin Laufer, Manish Raghavan, Solon Barocas
Disparate impact doctrine offers an important legal apparatus for targeting discriminatory data-driven algorithmic decisions. A recent body of work has focused on conceptualizing o…
Dimensions of Generative AI Evaluation Design
P. Alex Dow, Jennifer Wortman Vaughan, Solon Barocas +3
There are few principles or guidelines to ensure evaluations of generative AI (GenAI) models and systems are effective. To help address this gap, we propose a set of general dimens…
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nicholas Pangakis +17
Across academia, industry, and government, there is an increasing awareness that the measurement tasks involved in evaluating generative AI (GenAI) systems are especially difficult…
The Legal Duty to Search for Less Discriminatory Algorithms
Emily Black, Logan Koepke, Pauline Kim +2
Work in computer science has established that, contrary to conventional wisdom, for a given prediction problem there are almost always multiple possible models with equivalent perf…
Measuring Machine Learning Harms from Stereotypes Requires Understanding Who Is Harmed by Which Errors in What Ways
Angelina Wang, Xuechunzi Bai, Solon Barocas +1
As machine learning applications proliferate, we need an understanding of their potential for harm. However, current fairness metrics are rarely grounded in human psychological exp…
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman +4
In this paper, we examine computational approaches for measuring the "fairness" of image tagging systems, finding that they cluster into five distinct categories, each with its own…
Measuring Representational Harms in Image Captioning
Angelina Wang, Solon Barocas, Kristen Laird +1
Previous work has largely considered the fairness of image captioning systems through the underspecified lens of "bias." In contrast, we present a set of techniques for measuring f…
An Uncommon Task: Participatory Design in Legal AI
Fernando Delgado, Solon Barocas, Karen Levy
Despite growing calls for participation in AI design, there are to date few empirical studies of what these processes look like and how they can be structured for meaningful engage…
Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies
Briana Vecchione, Solon Barocas, Karen Levy
Algorithmic audits have been embraced as tools to investigate the functioning and consequences of sociotechnical systems. Though the term is used somewhat loosely in the algorithmi…
Computer Vision and Conflicting Values: Describing People with Automated Alt Text
Margot Hanley, Solon Barocas, Karen Levy +2
Scholars have recently drawn attention to a range of controversial issues posed by the use of computer vision for automatically generating descriptions of people in images. Despite…
On Modeling Human Perceptions of Allocation Policies with Uncertain Outcomes
Hoda Heidari, Solon Barocas, Jon Kleinberg +1
Many policies allocate harms or benefits that are uncertain in nature: they produce distributions over the population in which individuals have different probabilities of incurring…
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar +5
Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple. However, their des…
The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
Solon Barocas, Andrew D. Selbst, Manish Raghavan
Counterfactual explanations are gaining prominence within technical, legal, and business circles as a way to explain the decisions of a machine learning model. These explanations s…
Roles for Computing in Social Change
Rediet Abebe, Solon Barocas, Jon Kleinberg +3
A recent normative turn in computer science has brought concerns about fairness, bias, and accountability to the core of the field. Yet recent scholarship has warned that much of t…
Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
Manish Raghavan, Solon Barocas, Jon Kleinberg +1
There has been rapidly growing interest in the use of algorithms in hiring, especially as a means to address or mitigate bias. Yet, to date, little is known about how these methods…