The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies

Abstract

Artificial intelligence (AI) has huge potential to improve the health and well-being of people, but adoption in clinical practice is still limited. Lack of transparency is identified as one of the main barriers to implementation, as clinicians should be confident the AI system can be trusted. Explainable AI has the potential to overcome this issue and can be a step towards trustworthy AI. In this paper we review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems for the health-care domain and contribute to formalization of the field of explainable AI. We argue the reason to demand explainability determines what should be explained as this determines the relative importance of the properties of explainability (i.e. interpretability and fidelity). Based on this, we propose a framework to guide the choice between classes of explainable AI methods (explainable modelling versus post-hoc explanation; model-based, attribution-based, or example-based explanations; global and local explanations). Furthermore, we find that quantitative evaluation metrics, which are important for objective standardized evaluation, are still lacking for some properties (e.g. clarity) and types of explanations (e.g. example-based methods). We conclude that explainable modelling can contribute to trustworthy AI, but the benefits of explainability still need to be proven in practice and complementary measures might be needed to create trustworthy AI in health care (e.g. reporting data quality, performing extensive (external) validation, and regulation).

References in corpus (31)

• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Methods for Interpreting and Understanding Deep Neural Networks
• Scalable and accurate deep learning for electronic health records
• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Interpretable machine learning: definitions, methods, and applications
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Equality of Opportunity in Supervised Learning
• Principles alone cannot guarantee ethical AI
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Predictive learning via rule ensembles
• All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously
• Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
• Explaining Explanations in AI
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• What Does Explainable AI Really Mean? A New Conceptualization of Perspectives
• What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
• Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models
• Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
• Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
• Interpreting Blackbox Models via Model Extraction
• Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
• An Evaluation of the Human-Interpretability of Explanation
• Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
• Explainable Neural Networks based on Additive Index Models
• Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretability
• Machine Learning Model Interpretability for Precision Medicine
• Assessing the Local Interpretability of Machine Learning Models
• Interpretable & Explorable Approximations of Black Box Models
• Proposed Guidelines for the Responsible Use of Explainable Machine Learning

Cited by in corpus (25)

• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
• Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
• Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction
• A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
• "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
• Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond
• Clinical Melanoma Diagnosis with Artificial Intelligence: Insights from a Prospective Multicenter Study
• A Design Framework for operationalizing Trustworthy Artificial Intelligence in Healthcare: Requirements, Tradeoffs and Challenges for its Clinical Adoption
• How well can a large language model explain business processes as perceived by users?
• What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks
• FairCanary: Rapid Continuous Explainable Fairness
• Why Should I Choose You? AutoXAI: A Framework for Selecting and Tuning eXplainable AI Solutions
• Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI
• Evaluation of Human-Understandability of Global Model Explanations using Decision Tree
• EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Models
• On the Connection between Game-Theoretic Feature Attributions and Counterfactual Explanations
• Explainable autoencoder for neutron star dense matter parameter estimation
• End-to-End Optimized Image Compression with the Frequency-Oriented Transform
• XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensics
• Trustworthy, responsible, ethical AI in manufacturing and supply chains: synthesis and emerging research questions
• Assessing AI Explainability: A Usability Study Using a Novel Framework Involving Clinicians
• Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences
• Explainable Deep Image Classifiers for Skin Lesion Diagnosis
• ReCAP: Recursive Cross Attention Network for Pseudo-Label Generation in Robotic Surgical Skill Assessment