Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

Abstract

Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.

CHI 2025. This version includes the appendix

References in corpus (7)

• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
• Designing for Responsible Trust in AI Systems: A Communication Perspective
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• Humans, AI, and Context: Understanding End-Users' Trust in a Real-World Computer Vision Application
• Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models