To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

Abstract

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.

References in corpus (1)

• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making

Cited by in corpus (10)

• Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
• Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
• Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
• The Conflict Between Explainable and Accountable Decision-Making Algorithms
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System
• Blaming Humans and Machines: What Shapes People's Reactions to Algorithmic Harm
• To Classify is to Interpret: Building Taxonomies from Heterogeneous Data through Human-AI Collaboration