Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policing

Abstract

Research into recidivism risk prediction in the criminal legal system has garnered significant attention from HCI, critical algorithm studies, and the emerging field of human-AI decision-making. This study focuses on algorithmic crime mapping, a prevalent yet underexplored form of algorithmic decision support (ADS) in this context. We conducted experiments and follow-up interviews with 60 participants, including community members, technical experts, and law enforcement agents (LEAs), to explore how lived experiences, technical knowledge, and domain expertise shape interactions with the ADS, impacting human-AI decision-making. Surprisingly, we found that domain experts (LEAs) often exhibited anchoring bias, readily accepting and engaging with the first crime map presented to them. Conversely, community members and technical experts were more inclined to engage with the tool, adjust controls, and generate different maps. Our findings highlight that all three stakeholders were able to provide critical feedback regarding AI design and use - community members questioned the core motivation of the tool, technical experts drew attention to the elastic nature of data science practice, and LEAs suggested redesign pathways such that the tool could complement their domain expertise.

References in corpus (14)

• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Expanding Explainability: Towards Social Transparency in AI systems
• The Fallacy of AI Functionality
• Problem Formulation and Fairness
• Data Vision: Learning to See Through Algorithmic Abstraction
• Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
• Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
• Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders
• Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation
• Modeling Assumptions Clash with the Real World: Transparency, Equity, and Community Challenges for Student Assignment Algorithms
• Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systems
• Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysis
• A Human-Centered Review of Algorithms in Decision-Making in Higher Education
• Beyond Transactional Democracy: A Study of Civic Tech in Canada