To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

Abstract

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.

References in corpus (1)

• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making

Cited by in corpus (74)

• When combinations of humans and AI are useful: A systematic review and meta-analysis
• Design Principles for Generative AI Applications
• What Large Language Models Know and What People Think They Know
• Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
• How Knowledge Workers Think Generative AI Will (Not) Transform Their Industries
• Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
• Artificial Intelligence Should Genuinely Support Clinical Reasoning and Decision Making To Bridge the Translational Gap
• Improving Human-AI Collaboration With Descriptions of AI Behavior
• Investigating and Designing for Trust in AI-powered Code Generation Tools
• AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
• On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
• Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
• Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
• The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
• ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events
• Twenty-Four Years of Empirical Research on Trust in AI: A Bibliometric Review of Trends, Overlooked Issues, and Future Directions
• The Conflict Between Explainable and Accountable Decision-Making Algorithms
• Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
• Performance and Metacognition Disconnect when Reasoning in Human-AI Interaction
• Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure
• One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations
• As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making
• Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
• Blaming Humans and Machines: What Shapes People's Reactions to Algorithmic Harm
• "It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study
• FATE in AI: Towards Algorithmic Inclusivity and Accessibility
• ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing
• ExpressEdit: Video Editing with Natural Language and Sketching
• AI on My Shoulder: Supporting Emotional Labor in Front-Office Roles with an LLM-based Empathetic Coworker
• AI, Help Me Think$\unicode{x2014}$but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support
• Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policing
• Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
• Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
• Perceptions of Sentient AI and Other Digital Minds: Evidence from the AI, Morality, and Sentience (AIMS) Survey
• Amplifying Minority Voices: AI-Mediated Devil's Advocate System for Inclusive Group Decision-Making
• Writing with AI Lowers Psychological Ownership, but Longer Prompts Can Help
• Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models
• IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback
• "How can we learn and use AI at the same time?": Participatory Design of GenAI with High School Students
• User Experience with LLM-powered Conversational Recommendation Systems: A Case of Music Recommendation
• Exploring Collaborative GenAI Agents in Synchronous Group Settings: Eliciting Team Perceptions and Design Considerations for the Future of Work
• ChainBuddy: An AI Agent System for Generating LLM Pipelines
• The Impact of Explanations on Fairness in Human-AI Decision-Making: Protected vs Proxy Features
• Beyond Recommendations: From Backward to Forward AI Support of Pilots' Decision-Making Process
• LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
• Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent
• PADTHAI-MM: Principles-based Approach for Designing Trustworthy, Human-centered AI using MAST Methodology
• AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems
• Give Me a Choice: The Consequences of Restricting Choices Through AI-Support for Perceived Autonomy, Motivational Variables, and Decision Performance
• Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
• AI Trust Reshaping Administrative Burdens: Understanding Trust-Burden Dynamics in LLM-Assisted Benefits Systems
• To Classify is to Interpret: Building Taxonomies from Heterogeneous Data through Human-AI Collaboration
• "Even explanations will not help in trusting [this] fundamentally biased system": A Predictive Policing Case-Study
• Towards Feature Engineering with Human and AI's Knowledge: Understanding Data Science Practitioners' Perceptions in Human&AI-Assisted Feature Engineering Design
• Guidance Source Matters: How Guidance from AI, Expert, or a Group of Analysts Impacts Visual Data Preparation and Analysis
• What Lies Beneath? Exploring the Impact of Underlying AI Model Updates in AI-Infused Systems
• Better Together? The Role of Explanations in Supporting Novices in Individual and Collective Deliberations about AI
• Wisdom of the Crowd, Without the Crowd: A Socratic LLM for Asynchronous Deliberation on Perspectivist Data
• Steering AI-Driven Personalization of Scientific Text for General Audiences
• In defence of post-hoc explanations in medical AI
• Unpacking Interaction Profiles and Strategies in Human-AI Collaborative Problem Solving: A Cognitive Distribution and Regulation Perspective
• Analyzing the Presentation, Content, and Utilization of References in LLM-powered Conversational AI Systems
• Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research
• Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest
• The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
• Resume Screening, Fast and Slow: (Biased) AI Recommendations' Influence on Human Decision Making
• Scaling Expert Feedback with Reflective Edit Propagation in Compositional Knowledge Bases