RAG-Enhanced LLMs for Interactive Explainability in Clinical Decision Support Systems

Abstract

Artificial Intelligence (AI) is increasingly embedded in high-risks decision-making systems across domains such as healthcare, finance, and the judicial system. As reliance on predictive models grows, so does the demand for Explainable AI (XAI) to ensure transparency, trust, and interpretability. However, existing XAI approaches often fail to present explanations in a format accessible to domain experts, limiting their practical utility. Large Language Models (LLMs) have recently emerged as a bridge between complex models and end-users, offering natural language explanations. Yet, their use introduces new risks, including hallucinated outputs and lack of source traceability. To address these challenges, we propose a Clinical Decision Support System (CDSS) that not only combines standard predictive models with XAI techniques but also incorporates a Retrieval-Augmented Generation module as an interactive enhancer. This module grounds explanations in verified medical knowledge, reformulates technical outputs into a human-like format, and enables users-such as clinicians-to actively query the system. By fostering an interactive, human-in-the-loop environment, our approach empowers domain experts to explore model decisions, contextualize explanations, and build trust in AI-assisted diagnostics. The study concludes with preliminary experiments validating the proposed methodology.

arXiv commentPublished in HC_AIxIA_HYDRA 2025, Communications in Computer and Information Science, vol. 2830, pp. 50-63, Springer, 2026. Please cite the published version: doi:10.1007/978-3-032-16708-8_4

References in corpus (13)

• Survey of Hallucination in Natural Language Generation
• Towards A Rigorous Science of Interpretable Machine Learning
• Predictive learning via rule ensembles
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
• GLocalX -- From Local to Global Explanations of Black Box AI Models
• Digital Forgetting in Large Language Models: A Survey of Unlearning Methods
• Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations
• LLMs for Explainable AI: A Comprehensive Survey
• LLMs for XAI: Future Directions for Explaining Explanations
• LLM-Generated Black-box Explanations Can Be Adversarially Helpful
• XAI for All: Can Large Language Models Simplify Explainable AI?
• A Survey of Large Language Models Attribution
• Natural Language Counterfactual Explanations for Graphs Using Large Language Models