Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations

Abstract

AI advice is becoming increasingly popular, e.g., in investment and medical treatment decisions. As this advice is typically imperfect, decision-makers have to exert discretion as to whether actually follow that advice: they have to "appropriately" rely on correct and turn down incorrect advice. However, current research on appropriate reliance still lacks a common definition as well as an operational measurement concept. Additionally, no in-depth behavioral experiments have been conducted that help understand the factors influencing this behavior. In this paper, we propose Appropriateness of Reliance (AoR) as an underlying, quantifiable two-dimensional measurement concept. We develop a research model that analyzes the effect of providing explanations for AI advice. In an experiment with 200 participants, we demonstrate how these explanations influence the AoR, and, thus, the effectiveness of AI advice. Our work contributes fundamental concepts for the analysis of reliance behavior and the purposeful design of AI advisors.

arXiv commentarXiv admin note: text overlap with arXiv:2204.06916

References in corpus (3)

• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection

Cited by in corpus (14)

• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
• Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
• HILL: A Hallucination Identifier for Large Language Models
• The Impact of Imperfect XAI on Human-AI Decision-Making
• A Critical Survey on Fairness Benefits of Explainable AI
• Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
• AI, Help Me Think$\unicode{x2014}$but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support
• Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
• Letters from Future Self: Augmenting the Letter-Exchange Exercise with LLM-based Agents to Enhance Young Adults' Career Exploration
• Beyond Recommendations: From Backward to Forward AI Support of Pilots' Decision-Making Process
• When Bias Backfires: The Modulatory Role of Counterfactual Explanations on the Adoption of Algorithmic Bias in XAI-Supported Human Decision-Making
• It's only fair when I think it's fair: How Gender Bias Alignment Undermines Distributive Fairness in Human-AI Collaboration
• Understanding Collaboration between Professional Designers and Decision-making AI: A Case Study in the Workplace
• Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest