(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court
arXiv:2403.13004 · doi:10.1145/3613904.3641902
Abstract
Accountable use of AI systems in high-stakes settings relies on making systems contestable. In this paper we study efforts to contest AI systems in practice by studying how public defenders scrutinize AI in court. We present findings from interviews with 17 people in the U.S. public defense community to understand their perceptions of and experiences scrutinizing computational forensic software (CFS) -- automated decision systems that the government uses to convict and incarcerate, such as facial recognition, gunshot detection, and probabilistic genotyping tools. We find that our participants faced challenges assessing and contesting CFS reliability due to difficulties (a) navigating how CFS is developed and used, (b) overcoming judges and jurors' non-critical perceptions of CFS, and (c) gathering CFS expertise. To conclude, we provide recommendations that center the technical, social, and institutional context to better position interventions such as performance evaluations to support contestability in practice.
29 pages, 4 figures. To appear in Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24)
References in corpus (8)
- Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
- Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
- Measuring Robustness to Natural Distribution Shifts in Image Classification
- Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
- Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
- Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
- Contestable Camera Cars: A Speculative Design Exploration of Public AI That Is Open and Responsive to Dispute
- Defending IEEE Software Standards in Federal Criminal Court