Access Denied: Meaningful Data Access for Quantitative Algorithm Audits

Abstract

Independent algorithm audits hold the promise of bringing accountability to automated decision-making. However, third-party audits are often hindered by access restrictions, forcing auditors to rely on limited, low-quality data. To study how these limitations impact research integrity, we conduct audit simulations on two realistic case studies for recidivism and healthcare coverage prediction. We examine the accuracy of estimating group parity metrics across three levels of access: (a) aggregated statistics, (b) individual-level data with model outputs, and (c) individual-level data without model outputs. Despite selecting one of the simplest tasks for algorithmic auditing, we find that data minimization and anonymization practices can strongly increase error rates on individual-level data, leading to unreliable assessments. We discuss implications for independent auditors, as well as potential avenues for HCI researchers and regulators to improve data access and enable both reliable and holistic evaluations.

30 pages, 12 figures. To be published in CHI Conference on Human Factors in Computing Systems (CHI '25)

References in corpus (24)

• Model Cards for Model Reporting
• Improving fairness in machine learning systems: What do industry practitioners need?
• 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
• Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
• Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes
• A Clarification of the Nuances in the Fairness Metrics Landscape
• Auditing large language models: a three-layered approach
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
• Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
• Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
• Digital welfare fraud detection and the Dutch SyRI judgment
• Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice
• Black-Box Access is Insufficient for Rigorous AI Audits
• Diffprivlib: The IBM Differential Privacy Library
• Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising
• Aspirations and Practice of Model Documentation: Moving the Needle with Nudging and Traceability
• Having your Privacy Cake and Eating it Too: Platform-supported Auditing of Social Media Algorithms for Public Interest
• Beyond Privacy Trade-offs with Structured Transparency
• A Safe Harbor for AI Evaluation and Red Teaming
• Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deployment
• Beyond Fairness Metrics: Roadblocks and Challenges for Ethical AI in Practice
• A Linear Reconstruction Approach for Attribute Inference Attacks against Synthetic Data