4 papers
Aggregated Individual Reporting for Post-Deployment Evaluation
Jessica Dai, Inioluwa Deborah Raji, Benjamin Recht +1
The need for developing model evaluations beyond static benchmarking, especially in the post-deployment phase, is now well-understood. At the same time, concerns about the concentr…
Three Years of r/ChatGPT: Societal Impact Evaluations from Social Media Data
Jessica Dai, Sean Garcia, Emma Pierson +2
ChatGPT was launched on November 30, 2022; the r/ChatGPT subreddit was created just one day later. Since then, chatbot-based AI products have gone from niche proofs-of-concept to w…
Patient Safety Risks from AI Scribes: Signals from End-User Feedback
Jessica Dai, Anwen Huang, Catherine Nasrallah +7
AI scribes are transforming clinical documentation at scale. However, their real-world performance remains understudied, especially regarding their impacts on patient safety. To th…
From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms
Jessica Dai, Paula Gradu, Inioluwa Deborah Raji +1
When an individual reports a negative interaction with some system, how can their personal experience be contextualized within broader patterns of system behavior? We study the rep…