Black-Box Access is Insufficient for Rigorous AI Audits

Abstract

External audits of AI systems are increasingly recognized as a key mechanism for AI governance. The effectiveness of an audit, however, depends on the degree of access granted to auditors. Recent audits of state-of-the-art AI systems have primarily relied on black-box access, in which auditors can only query the system and observe its outputs. However, white-box access to the system's inner workings (e.g., weights, activations, gradients) allows an auditor to perform stronger attacks, more thoroughly interpret models, and conduct fine-tuning. Meanwhile, outside-the-box access to training and deployment information (e.g., methodology, code, documentation, data, deployment details, findings from internal evaluations) allows auditors to scrutinize the development process and design more targeted evaluations. In this paper, we examine the limitations of black-box audits and the advantages of white- and outside-the-box audits. We also discuss technical, physical, and legal safeguards for performing these audits with minimal security risks. Given that different forms of access can lead to very different levels of evaluation, we conclude that (1) transparency regarding the access and methods used by auditors is necessary to properly interpret audit results, and (2) white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.

FAccT 2024

References in corpus (27)

• Survey of Hallucination in Natural Language Generation
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Shortcut Learning in Deep Neural Networks
• Model Cards for Model Reporting
• Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
• Reading Race: AI Recognises Patient's Racial Identity In Medical Images
• Extracting Training Data from Large Language Models
• Ethics-Based Auditing to Develop Trustworthy AI
• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
• Auditing large language models: a three-layered approach
• BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
• Generative Agents: Interactive Simulacra of Human Behavior
• Auditing of AI: Legal, Ethical and Technical Approaches
• Harms from Increasingly Agentic Algorithmic Systems
• The Grey Hoodie Project: Big Tobacco, Big Tech, and the threat on academic integrity
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Scalable Extraction of Training Data from (Production) Language Models
• OpenXAI: Towards a Transparent Evaluation of Model Explanations
• WebArena: A Realistic Web Environment for Building Autonomous Agents
• Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications
• Provably safe systems: the only path to controllable AGI
• Robust Feature-Level Adversaries are Interpretability Tools
• Making Attention Mechanisms More Robust and Interpretable with Virtual Adversarial Training
• Testing Language Model Agents Safely in the Wild
• Understanding the Effects of Iterative Prompting on Truthfulness

Cited by in corpus (9)

• Combining Large Language Models with Tutoring System Intelligence: A Case Study in Caregiver Homework Support
• Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations
• SoK: Dataset Copyright Auditing in Machine Learning Systems
• Access Denied: Meaningful Data Access for Quantitative Algorithm Audits
• Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs
• Security practices in AI development
• External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery
• The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
• Multi-agent Self-triage System with Medical Flowcharts