Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making

Abstract

Despite the growing promise of artificial intelligence (AI) in supporting decision-making across domains, fostering appropriate human reliance on AI remains a critical challenge. In this paper, we investigate the utility of exploring distance-based uncertainty scores for task delegation to AI and describe how these scores can be visualized through embedding representations for human-AI decision-making. After developing an AI-based system for physical stroke rehabilitation assessment, we conducted a study with 19 health professionals and 10 students in medicine/health to understand the effect of exploring distance-based uncertainty scores on users' reliance on AI. Our findings showed that distance-based uncertainty scores outperformed traditional probability-based uncertainty scores in identifying uncertain cases. In addition, after exploring confidence scores for task delegation and reviewing embedding-based visualizations of distance-based uncertainty scores, participants achieved an 8.20% higher rate of correct decisions, a 7.15% higher rate of changing their decisions to correct ones, and a 7.14% lower rate of incorrect changes after reviewing AI outputs than those reviewing probability-based uncertainty scores ($p<0.01$). Our findings highlight the potential of distance-based uncertainty scores to enhance decision accuracy and appropriate reliance on AI while discussing ongoing challenges for human-AI collaborative decision-making.

ACM FAccT 2025

References in corpus (14)

• From Frequency to Meaning: Vector Space Models of Semantics
• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
• "Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment
• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
• Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
• Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
• Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
• Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders
• Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making
• Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
• Towards Relatable Explainable AI with the Perceptual Process
• Design, Development, and Evaluation of an Interactive Personalized Social Robot to Monitor and Coach Post-Stroke Rehabilitation Exercises