A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
arXiv:2002.08035 · doi:10.1145/3313831.3376638
Abstract
The increased use of algorithmic predictions in sensitive domains has been accompanied by both enthusiasm and concern. To understand the opportunities and risks of these technologies, it is key to study how experts alter their decisions when using such tools. In this paper, we study the adoption of an algorithmic tool used to assist child maltreatment hotline screening decisions. We focus on the question: Are humans capable of identifying cases in which the machine is wrong, and of overriding those recommendations? We first show that humans do alter their behavior when the tool is deployed. Then, we show that humans are less likely to adhere to the machine's recommendation when the score displayed is an incorrect estimate of risk, even when overriding the recommendation requires supervisory approval. These results highlight the risks of full automation and the importance of designing decision pipelines that provide humans with autonomy.
Accepted at ACM Conference on Human Factors in Computing Systems (ACM CHI), 2020
References in corpus (1)
Cited by in corpus (25)
- Algorithms and Decision-Making in the Public Sector
- How does HCI Understand Human Autonomy and Agency?
- Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
- Improving Human-AI Collaboration With Descriptions of AI Behavior
- On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
- Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders
- Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation
- "There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making
- How Beginning Programmers and Code LLMs (Mis)read Each Other
- Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systems
- Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysis
- Mindless Attractor: A False-Positive Resistant Intervention for Drawing Attention Using Auditory Perturbation
- Capturing Humans' Mental Models of AI: An Item Response Theory Approach
- How to Train a (Bad) Algorithmic Caseworker: A Quantitative Deconstruction of Risk Assessments in Child-Welfare
- Assessing Confidence in AI-Assisted Grading of Physics Exams through Psychometrics: An Exploratory Study
- Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools
- The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment
- Fair Machine Guidance to Enhance Fair Decision Making in Biased People
- A Human-Centric Perspective on Fairness and Transparency in Algorithmic Decision-Making
- A Comparative User Study of Human Predictions in Algorithm-Supported Recidivism Risk Assessment
- Planning to Fairly Allocate: Probabilistic Fairness in the Restless Bandit Setting
- Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
- Empowering Social Service with AI: Insights from a Participatory Design Study with Practitioners
- An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper
- A Labeling Task Design for Supporting Algorithmic Needs: Facilitating Worker Diversity and Reducing AI Bias