Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

Abstract

Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.

CHI 2025. This version includes the appendix

References in corpus (17)

• Survey of Hallucination in Natural Language Generation
• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
• Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
• "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
• Generative AI Literacy: Twelve Defining Competencies
• Designing for Responsible Trust in AI Systems: A Communication Perspective
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
• Understanding Users' Dissatisfaction with ChatGPT Responses: Types, Resolving Tactics, and the Effect of Knowledge Level
• Humans, AI, and Context: Understanding End-Users' Trust in a Real-World Computer Vision Application
• One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations
• How Interpretable are Reasoning Explanations from Prompting Large Language Models?
• Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models

Cited by in corpus (3)

• From Toil to Thought: Designing for Strategic Exploration and Responsible AI in Systematic Literature Reviews
• Unpacking Interaction Profiles and Strategies in Human-AI Collaborative Problem Solving: A Cognitive Distribution and Regulation Perspective
• The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems