Auditing of AI: Legal, Ethical and Technical Approaches

Abstract

AI auditing is a rapidly growing field of research and practice. This review article, which doubles as an editorial to Digital Societys topical collection on Auditing of AI, provides an overview of previous work in the field. Three key points emerge from the review. First, contemporary attempts to audit AI systems have much to learn from how audits have historically been structured and conducted in areas like financial accounting, safety engineering and the social sciences. Second, both policymakers and technology providers have an interest in promoting auditing as an AI governance mechanism. Academic researchers can thus fill an important role by studying the feasibility and effectiveness of different AI auditing procedures. Third, AI auditing is an inherently multidisciplinary undertaking, to which substantial contributions have been made by computer scientists and engineers as well as social scientists, philosophers, legal scholars and industry practitioners. Reflecting this diversity of perspectives, different approaches to AI auditing have different affordances and constraints. Specifically, a distinction can be made between technology-oriented audits, which focus on the properties and capabilities of AI systems, and process oriented audits, which focus on technology providers governance structures and quality management systems. The next step in the evolution of auditing as an AI governance mechanism, this article concludes, should be the interlinking of these available (and complementary) approaches into structured and holistic procedures to audit not only how AI systems are designed and used but also how they impact users, societies and the natural environment in applied settings over time.

References in corpus (27)

• Artificial Intelligence: the global landscape of ethics guidelines
• On the Opportunities and Risks of Foundation Models
• Model Cards for Model Reporting
• Demystifying the Draft EU Artificial Intelligence Act
• Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
• Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes
• Ethics-Based Auditing to Develop Trustworthy AI
• The Fallacy of AI Functionality
• Auditing large language models: a three-layered approach
• Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations
• A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• Conformity Assessments and Post-market Monitoring: A Guide to the Role of Auditing in the Proposed European AI Regulation
• Enslaving the Algorithm: From a "Right to an Explanation" to a "Right to Better Decisions"?
• Systematic Literature Review of Validation Methods for AI Systems
• Operationalising AI governance through ethics-based auditing: An industry case study
• The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
• Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook
• Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies
• Open the Black Box Data-Driven Explanation of Black Box Decision Systems
• How to Assess Trustworthy AI in Practice
• Challenges and Best Practices in Corporate AI Governance:Lessons from the Biopharmaceutical Industry
• A Summary of Formal Specification and Verification of Autonomous Robotic Systems
• The Switch, the Ladder, and the Matrix: Models for Classifying AI Systems
• Auditing and Achieving Intersectional Fairness in Classification Problems
• A Sandbox Tool to Bias(Stress)-Test Fairness Algorithms

Cited by in corpus (4)

• Black-Box Access is Insufficient for Rigorous AI Audits
• A Framework for Assurance Audits of Algorithmic Systems
• Artificial intelligence, rationalization, and the limits of control in the public sector: the case of tax policy optimization
• Formalising Anti-Discrimination Law in Automated Decision Systems