Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Abstract

Large language models are becoming increasingly pervasive and ubiquitous in society via deployment in sociotechnical systems. Yet these language models, be it for classification or generation, have been shown to be biased and behave irresponsibly, causing harm to people at scale. It is crucial to audit these language models rigorously. Existing auditing tools leverage either or both humans and AI to find failures. In this work, we draw upon literature in human-AI collaboration and sensemaking, and conduct interviews with research experts in safe and fair AI, to build upon the auditing tool: AdaTest (Ribeiro and Lundberg, 2022), which is powered by a generative large language model (LLM). Through the design process we highlight the importance of sensemaking and human-AI communication to leverage complementary strengths of humans and generative models in collaborative auditing. To evaluate the effectiveness of the augmented tool, AdaTest++, we conduct user studies with participants auditing two commercial language models: OpenAI's GPT-3 and Azure's sentiment analysis model. Qualitative analysis shows that AdaTest++ effectively leverages human strengths such as schematization, hypothesis formation and testing. Further, with our tool, participants identified a variety of failures modes, covering 26 different topics over 2 tasks, that have been shown before in formal audits and also those previously under-reported.

21 pages, 3 figures

References in corpus (2)

• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• Discovering and Validating AI Errors With Crowdsourced Failure Reports

Cited by in corpus (7)

• Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
• Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation
• Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots
• Enhancing user experience in large language models through human-centered design: Integrating theoretical insights with an experimental study to meet diverse software learning needs with a single document knowledge base
• Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work
• Vipera: Towards systematic auditing of generative text-to-image models at scale
• Extended Japanese Commonsense Morality Dataset with Masked Token and Label Enhancement