51 citations · 53 across the 12 of their papers we have counts for
12 papers
CompanionSim: Synthetic Data for Evaluating Anthropomorphism in Human-AI Relationships
Jacy Reese Anthis, Mark Díaz, Renee Shelby
Many people now see AI systems as not just productivity tools but as social companions. Researchers are eager to study the consequences of AI companionship behaviors, such as valid…
"Death by a thousand taxonomies?": AI Risk Classification In Practice
Glen Berman, Ned Cooper, Angel Hsing-Chi Hwang +3
The harms in which AI is implicated range in nature and scope from unsafe user interactions through to the societal-wide consequences of AI adoption. Classification of the diverse…
"I want to be pushed, I want to grow": Enabling social workers to design evaluations of LLM augmentation in their work
Anna Kawakami, Chloe Qianhui Zhao, Renee Shelby +3
Workers are increasingly asked to adopt AI systems to assist their work, yet are rarely given a voice in defining what meaningful AI augmentation should look like or how to evaluat…
Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
Jessica Huynh, Alfredo Gomez, Athiya Deviyani +3
Autoraters, also referred to as LLM-as-judges, are increasingly used for evaluation and automated content moderation. However, there is limited statistical analysis of how modifica…
A Unified Framework to Quantify Cultural Intelligence of AI
Sunipa Dev, Vinodkumar Prabhakaran, Rutledge Chin Feman +16
As generative AI technologies are increasingly being launched across the globe, assessing their competence to operate in different cultural contexts is exigently becoming a priorit…
Cultural Perspectives and Expectations for Generative AI: A Global Survey Approach
Erin van Liemt, Renee Shelby, Andrew Smart +5
There is a lack of empirical evidence about global attitudes around whether and how GenAI should represent cultures. This paper assesses understandings and beliefs about culture as…
How Tech Workers Contend with Hazards of Humanlikeness in Generative AI
Mark Díaz, Renee Shelby, Eric Corbett +1
Generative AI's humanlike qualities are driving its rapid adoption in professional domains. However, this anthropomorphic appeal raises concerns from HCI and responsible AI scholar…
Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms
Shalaleh Rismani, Renee Shelby, Leah Davis +2
Over the past decade, an ecosystem of measures has emerged to evaluate the social and ethical implications of AI systems, largely shaped by high-level ethics principles. These meas…
Taxonomy of User Needs and Actions
Renee Shelby, Fernando Diaz, Vinodkumar Prabhakaran
The growing ubiquity of conversational AI highlights the need for frameworks that capture not only users' instrumental goals but also the situated, adaptive, and social practices t…
Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble
Olivia Sturman, Aparna Joshi, Bhaktipriya Radharapu +2
Increasing use of large language models (LLMs) demand performant guardrails to ensure the safety of inputs and outputs of LLMs. When these safeguards are trained on imbalanced data…
The Ethics of Advanced AI Assistants
Iason Gabriel, Arianna Manzini, Geoff Keeling +54
This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural langu…
Harm Amplification in Text-to-Image Models
Susan Hao, Renee Shelby, Yuchi Liu +6
Text-to-image (T2I) models have emerged as a significant advancement in generative AI; however, there exist safety concerns regarding their potential to produce harmful image outpu…