Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing

Abstract

Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source. In this paper, we introduce a framework for algorithmic auditing that supports artificial intelligence system development end-to-end, to be applied throughout the internal organization development lifecycle. Each stage of the audit yields a set of documents that together form an overall audit report, drawing on an organization's values or principles to assess the fit of decisions made throughout the process. The proposed auditing framework is intended to contribute to closing the accountability gap in the development and deployment of large-scale artificial intelligence systems by embedding a robust process to ensure audit integrity.

arXiv commentAccepted to ACM FAT* (Fariness, Accountability and Transparency) conference 2020. Full workable templates for the documents of the SMACTR framework presented in the paper can be found here https://drive.google.com/drive/folders/1GWlq8qGZXb2lNHxWBuo2wl-rlHsjNPM0?usp=sharing

References in corpus (6)

• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Runaway Feedback Loops in Predictive Policing
• Given Enough Eyeballs, All Bugs Are Shallow? Revisiting Eric Raymond with Bug Bounty Programs
• A Broader View on Bias in Automated Decision-Making: Reflecting on Epistemology and Dynamics
• Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution

Cited by in corpus (19)

• The Fallacy of AI Functionality
• Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning
• A Survey on AI Assurance
• How Different Groups Prioritize Ethical Values for Responsible AI
• A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models
• Understanding accountability in algorithmic supply chains
• Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation
• The Algorithmic Imprint
• Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"
• Twitter Spam and False Accounts Prevalence, Detection and Characterization: A Survey
• Risk of AI in Healthcare: A Comprehensive Literature Review and Study Framework
• Stronger Together: on the Articulation of Ethical Charters, Legal Tools, and Technical Documentation in ML
• Crowdsourcing Impacts: Exploring the Utility of Crowds for Anticipating Societal Impacts of Algorithmic Decision Making
• Subverting Fair Image Search with Generative Adversarial Perturbations
• Fair Machine Learning Under Partial Compliance
• Plurals: A System for Guiding LLMs Via Simulated Social Ensembles
• Towards Audit Requirements for AI-based Systems in Mobility Applications
• Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach
• Towards Integrating Fairness Transparently in Industrial Applications