Organization Matters: A Qualitative Study of Organizational Dynamics in Red Teaming Practices for Generative AI

Abstract

The rapid integration of generative artificial intelligence (GenAI) across diverse fields underscores the critical need for red teaming efforts to proactively identify and mitigate associated risks. While previous research primarily addresses technical aspects, this paper highlights organizational factors that hinder the effectiveness of red teaming in real-world settings. Through qualitative analysis of 17 semi-structured interviews with red teamers from various organizations, we uncover challenges such as the marginalization of vulnerable red teamers, the invisibility of nuanced AI risks to vulnerable users until post-deployment, and a lack of user-centered red teaming approaches. These issues often arise from underlying organizational dynamics, including organizational resistance, organizational inertia, and organizational mediocracy. To mitigate these dynamics, we discuss the implications of user research for red teaming and the importance of embedding red teaming throughout the entire development cycle of GenAI systems.

References in corpus (11)

• Predictability and Surprise in Large Generative Models
• Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
• Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook
• Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
• Walking the Walk of AI Ethics: Organizational Challenges and the Individualization of Risk among Ethics Entrepreneurs
• Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into educational practice, not instinctively defending against it
• Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice
• Participation in the age of foundation models
• The Potential and Implications of Generative AI on HCI Education
• Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions
• Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industries