(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court
arXiv:2403.13004 · doi:10.1145/3613904.3641902
Abstract
Accountable use of AI systems in high-stakes settings relies on making systems contestable. In this paper we study efforts to contest AI systems in practice by studying how public defenders scrutinize AI in court. We present findings from interviews with 17 people in the U.S. public defense community to understand their perceptions of and experiences scrutinizing computational forensic software (CFS) -- automated decision systems that the government uses to convict and incarcerate, such as facial recognition, gunshot detection, and probabilistic genotyping tools. We find that our participants faced challenges assessing and contesting CFS reliability due to difficulties (a) navigating how CFS is developed and used, (b) overcoming judges and jurors' non-critical perceptions of CFS, and (c) gathering CFS expertise. To conclude, we provide recommendations that center the technical, social, and institutional context to better position interventions such as performance evaluations to support contestability in practice.
arXiv comment29 pages, 4 figures. To appear in Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24)
References in corpus (13)
- Model Cards for Model Reporting
- Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
- On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
- Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
- Measuring Robustness to Natural Distribution Shifts in Image Classification
- Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
- Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
- Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
- Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice
- Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
- Contestable Camera Cars: A Speculative Design Exploration of Public AI That Is Open and Responsive to Dispute
- Defending IEEE Software Standards in Federal Criminal Court
- Adversarial Scrutiny of Evidentiary Statistical Software