Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
arXiv:2501.17546 · doi:10.1145/3708359.3712133
Abstract
Explainable artificial intelligence (XAI) methods are being proposed to help interpret and understand how AI systems reach specific predictions. Inspired by prior work on conversational user interfaces, we argue that augmenting existing XAI methods with conversational user interfaces can increase user engagement and boost user understanding of the AI system. In this paper, we explored the impact of a conversational XAI interface on users' understanding of the AI system, their trust, and reliance on the AI system. In comparison to an XAI dashboard, we found that the conversational XAI interface can bring about a better understanding of the AI system among users and higher user trust. However, users of both the XAI dashboard and conversational XAI interfaces showed clear overreliance on the AI system. Enhanced conversations powered by large language model (LLM) agents amplified over-reliance. Based on our findings, we reason that the potential cause of such overreliance is the illusion of explanatory depth that is concomitant with both XAI interfaces. Our findings have important implications for designing effective conversational XAI interfaces to facilitate appropriate reliance and improve human-AI collaboration. Code can be found at https://github.com/delftcrowd/IUI2025_ConvXAI
conditionally accepted to IUI 2025
References in corpus (16)
- XGBoost: A Scalable Tree Boosting System
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- Questioning the AI: Informing Design Practices for Explainable AI User Experiences
- Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
- What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research
- From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
- A Survey on Conversational Recommender Systems
- Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations
- On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
- explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
- Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations
- Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
- Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
- Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations
- Seamful XAI: Operationalizing Seamful Design in Explainable AI
- OmniXAI: A Library for Explainable AI