What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use

Abstract

Translating machine learning (ML) models effectively to clinical practice requires establishing clinicians' trust. Explainability, or the ability of an ML model to justify its outcomes and assist clinicians in rationalizing the model prediction, has been generally understood to be critical to establishing trust. However, the field suffers from the lack of concrete definitions for usable explanations in different settings. To identify specific aspects of explainability that may catalyze building trust in ML models, we surveyed clinicians from two distinct acute care specialties (Intenstive Care Unit and Emergency Department). We use their feedback to characterize when explainability helps to improve clinicians' trust in ML models. We further identify the classes of explanations that clinicians identified as most relevant and crucial for effective translation to clinical practice. Finally, we discern concrete metrics for rigorous evaluation of clinical explainability methods. By integrating perceptions of explainability between clinicians and ML researchers we hope to facilitate the endorsement and broader adoption and sustained use of ML systems in healthcare.

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• On Calibration of Modern Neural Networks
• Understanding Neural Networks Through Deep Visualization
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Understanding Black-box Predictions via Influence Functions
• Sanity Checks for Saliency Maps
• Attention is not Explanation
• Deep and Confident Prediction for Time Series at Uber
• Supersparse Linear Integer Models for Optimized Medical Scoring Systems
• RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records
• A Review of Challenges and Opportunities in Machine Learning for Health
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
• How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
• On the consistency of supervised learning with missing values
• Can You Trust This Prediction? Auditing Pointwise Reliability After Learning
• ClinicalVis: Supporting Clinical Task-Focused Design Evaluation

Cited by in corpus (24)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
• Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
• Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens
• A Survey on Graph-Based Deep Learning for Computational Histopathology
• Benchmarking Deep Learning Interpretability in Time Series Predictions
• Artificial Intelligence in Glioma Imaging: Challenges and Advances
• Uncertainty-Aware Deep Ensembles for Reliable and Explainable Predictions of Clinical Time Series
• Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes
• When Segmentation is Not Enough: Rectifying Visual-Volume Discordance Through Multisensor Depth-Refined Semantic Segmentation for Food Intake Tracking in Long-Term Care
• Calibrating Healthcare AI: Towards Reliable and Interpretable Deep Predictive Models
• Generative causal explanations of black-box classifiers
• Best of both worlds: local and global explanations with human-understandable concepts
• One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• Epidemic-guided deep learning for spatiotemporal forecasting of Tuberculosis outbreak
• Lymph Node Graph Neural Networks for Cancer Metastasis Prediction
• Using Deep Networks for Scientific Discovery in Physiological Signals
• On Explainability in AI-Solutions: A Cross-Domain Survey
• Dynamically Extracting Outcome-Specific Problem Lists from Clinical Notes with Guided Multi-Headed Attention
• A Bandit Model for Human-Machine Decision Making with Private Information and Opacity
• Uncertainty-aware INVASE: Enhanced Breast Cancer Diagnosis Feature Selection
• Leveraging Clinical Context for User-Centered Explainability: A Diabetes Use Case
• Learning Interpretable Microscopic Features of Tumor by Multi-task Adversarial CNNs To Improve Generalization