Interpretable machine learning: definitions, methods, and applications

Abstract

Machine-learning models have demonstrated great success in learning complex patterns that enable them to make predictions about unobserved data. In addition to using models for prediction, the ability to interpret what a model has learned is receiving an increasing amount of attention. However, this increased focus has led to considerable confusion about the notion of interpretability. In particular, it is unclear how the wide array of proposed interpretation methods are related, and what common concepts can be used to evaluate them. We aim to address these concerns by defining interpretability in the context of machine learning and introducing the Predictive, Descriptive, Relevant (PDR) framework for discussing interpretations. The PDR framework provides three overarching desiderata for evaluation: predictive accuracy, descriptive accuracy and relevancy, with relevancy judged relative to a human audience. Moreover, to help manage the deluge of interpretation methods, we introduce a categorization of existing techniques into model-based and post-hoc categories, with sub-groups including sparsity, modularity and simulatability. To demonstrate how practitioners can use the PDR framework to evaluate and understand interpretations, we provide numerous real-world examples. These examples highlight the often under-appreciated role played by human audiences in discussions of interpretability. Finally, based on our framework, we discuss limitations of existing methods and directions for future work. We hope that this work will provide a common vocabulary that will make it easier for both practitioners and researchers to discuss and choose from the full range of interpretation methods.

11 pages

References in corpus (30)

• A Survey on Deep Learning in Medical Image Analysis
• A Unified Approach to Interpreting Model Predictions
• Axiomatic Attribution for Deep Networks
• Striving for Simplicity: The All Convolutional Net
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Equality of Opportunity in Supervised Learning
• Understanding Black-box Predictions via Influence Functions
• Predictive learning via rule ensembles
• Visualizing and Understanding Recurrent Networks
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• How to Explain Individual Classification Decisions
• Sanity Checks for Saliency Maps
• Consistent Individualized Feature Attribution for Tree Ensembles
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Iterative Random Forests to detect predictive and stable high-order interactions
• Distilling a Neural Network Into a Soft Decision Tree
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Stability
• Hierarchical interpretations for neural network predictions
• Real Time Image Saliency for Black Box Classifiers
• Automatic Rule Extraction from Long Short Term Memory Networks
• Understanding Intra-Class Knowledge Inside CNN
• A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations
• Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention
• Interpreting CNN Knowledge via an Explanatory Graph
• Can I trust you more? Model-Agnostic Hierarchical Explanations
• Signed iterative random forests to identify enhancer-associated transcription factor binding
• Disentangled Attribution Curves for Interpreting Random Forests and Boosted Trees

Cited by in corpus (190)

• Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
• A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises
• Explainable Machine Learning for Scientific Insights and Discoveries
• A Survey on the Explainability of Supervised Machine Learning
• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
• Interpretability of machine learning based prediction models in healthcare
• A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts
• Explaining a Series of Models by Propagating Shapley Values
• Towards Out-Of-Distribution Generalization: A Survey
• Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework
• Veridical Data Science
• Explainable Artificial Intelligence: Precepts, Methods, and Opportunities for Research in Construction
• Explainable Deep Reinforcement Learning: State of the Art and Challenges
• Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-making
• Enhancing Discrete Choice Models with Representation Learning
• Contemporary Symbolic Regression Methods and their Relative Performance
• Neural Network Potentials for Chemistry: Concepts, Applications and Prospects
• Post-hoc explanation of black-box classifiers using confident itemsets
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• Deep symbolic regression for physics guided by units constraints: toward the automated discovery of physical laws
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• GeoShapley: A Game Theory Approach to Measuring Spatial Effects in Machine Learning Models
• Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine
• Hierarchical interpretations for neural network predictions
• How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
• Structure-preserving neural networks
• A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability
• Model-Agnostic Counterfactual Explanations for Consequential Decisions
• Phase Classification of Multi-Principal Element Alloys via Interpretable Machine Learning
• A Debiased MDI Feature Importance Measure for Random Forests
• A general framework for inference on algorithm-agnostic variable importance
• Unsupervised Social Bot Detection via Structural Information Theory
• Algorithmic Recourse: from Counterfactual Explanations to Interventions
• Augmenting Interpretable Models with LLMs during Training
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Forecasting Megaelectron-Volt Electrons inside Earth's Outer Radiation Belt: PreMevE 2.0 Based on Supervised Machine Learning Algorithms
• Adversarial Explanations for Understanding Image Classification Decisions and Improved Neural Network Robustness
• Fast TreeSHAP: Accelerating SHAP Value Computation for Trees
• Interpretable Random Forests via Rule Extraction
• Multiscale reweighted stochastic embedding (MRSE): Deep learning of collective variables for enhanced sampling
• Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey
• Discovering interpretable elastoplasticity models via the neural polynomial method enabled symbolic regressions
• A New Approach for Interpretability and Reliability in Clinical Risk Prediction: Acute Coronary Syndrome Scenario
• Model Interpretability through the Lens of Computational Complexity
• Understanding Neural Code Intelligence Through Program Simplification
• Evaluating Explanation Without Ground Truth in Interpretable Machine Learning
• Comprehensible Artificial Intelligence on Knowledge Graphs: A survey
• Local Interpretations for Explainable Natural Language Processing: A Survey
• Self-explaining AI as an alternative to interpretable AI
• Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
• A new interpretable unsupervised anomaly detection method based on residual explanation
• Interpretable machine learning in Physics
• Global Aggregations of Local Explanations for Black Box models
• Generative models for sampling and phase transition indication in spin systems
• Explaining Black-box Models for Biomedical Text Classification
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• Transforming Feature Space to Interpret Machine Learning Models
• An Interpretable Machine Learning Model for Deformation of Multi-Walled Carbon Nanotubes
• Adequate and fair explanations
• Impact of the characteristics of quantum chemical databases on machine learning predictions of tautomerization energies
• Born-Again Tree Ensembles
• Application of Machine Learning in Seismic Fragility Assessment of Bridges with SMA-Restrained Rocking Columns
• A Meta Survey of Quality Evaluation Criteria in Explanation Methods
• Learning Optimal Fair Classification Trees: Trade-offs Between Interpretability, Fairness, and Accuracy
• Interpretable Clustering: A Survey
• Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sector
• Applications of machine learning in gravitational wave research with current interferometric detectors
• Scalable Rule-Based Representation Learning for Interpretable Classification
• Unveiling Hidden Factors: Explainable AI for Feature Boosting in Speech Emotion Recognition
• Rational Shapley Values
• Designing Inherently Interpretable Machine Learning Models
• Relevance in the Renormalization Group and in Information Theory
• Explainable $k$-Means and $k$-Medians Clustering
• Glass-box model representation of seismic failure mode prediction for conventional RC shear walls
• Interpretable machine learning-accelerated seed treatment by nanomaterials for environmental stress alleviation
• What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks
• Towards responsible AI for education: Hybrid human-AI to confront the Elephant in the room
• Calibrated Explanations: with Uncertainty Information and Counterfactuals
• Efficient nonparametric statistical inference on population feature importance using Shapley values
• Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications
• Modeling Stated Preference for Mobility-on-Demand Transit: A Comparison of Machine Learning and Logit Models
• Exploiting auto-encoders and segmentation methods for middle-level explanations of image classification systems
• Benchmarking Instance-Centric Counterfactual Algorithms for XAI: From White Box to Black Box
• Automatic Test Suite Generation for Key-Points Detection DNNs using Many-Objective Search (Experience Paper)
• Harnessing Interpretable Machine Learning for Holistic Inverse Design of Origami
• Shapley Decomposition of R-Squared in Machine Learning Models
• Hessian-based toolbox for reliable and interpretable machine learning in physics
• Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations
• Evolutionary approaches to explainable machine learning
• A Rigorous Uncertainty-Aware Quantification Framework Is Essential for Reproducible and Replicable Machine Learning Workflows
• Provable Boolean Interaction Recovery from Tree Ensemble obtained via Random Forests
• Proposed Guidelines for the Responsible Use of Explainable Machine Learning
• ExKMC: Expanding Explainable $k$-Means Clustering
• Decisions, Counterfactual Explanations and Strategic Behavior
• SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usability
• Interpreting CNN for Low Complexity Learned Sub-pixel Motion Compensation in Video Coding
• ExplaiNE: An Approach for Explaining Network Embedding-based Link Predictions
• Interpretable Machine Learning: Moving From Mythos to Diagnostics
• Physically interpretable machine learning algorithm on multidimensional non-linear fields
• A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection
• Deep learning insights into cosmological structure formation
• Disentangled Attribution Curves for Interpreting Random Forests and Boosted Trees
• Modeling Heterogeneity in Mode-Switching Behavior Under a Mobility-on-Demand Transit System: An Interpretable Machine Learning Approach
• Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models
• Beyond Fidelity: Explaining Vulnerability Localization of Learning-based Detectors
• Impact of Accuracy on Model Interpretations
• Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data
• Transformation Importance with Applications to Cosmology
• Group-level Brain Decoding with Deep Learning
• A mixed-integer programming model for identifying intuitive ambulance dispatching policies
• Explainable AI-Based Interface System for Weather Forecasting Model
• Information Fusion via Symbolic Regression: A Tutorial in the Context of Human Health
• The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge
• E Pluribus Unum Interpretable Convolutional Neural Networks
• Metafeatures-based Rule-Extraction for Classifiers on Behavioral and Textual Data
• Vehicle-group-based Crash Risk Prediction and Interpretation on Highways
• Study on the Helpfulness of Explainable Artificial Intelligence
• Nearly-Tight and Oblivious Algorithms for Explainable Clustering
• Extending Process Discovery with Model Complexity Optimization and Cyclic States Identification: Application to Healthcare Processes
• Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
• Adversarial Attacks and Defenses: An Interpretation Perspective
• Developing Future Human-Centered Smart Cities: Critical Analysis of Smart City Security, Interpretability, and Ethical Challenges
• Near-Optimal Explainable $k$-Means for All Dimensions
• SPINEX: Similarity-based Predictions and Explainable Neighbors Exploration for Regression and Classification Tasks in Machine Learning
• Interpretable Classification of Bacterial Raman Spectra with Knockoff Wavelets
• Cross-Study Replicability in Cluster Analysis
• Explaining Predictions by Approximating the Local Decision Boundary
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• Unifying Neural Learning and Symbolic Reasoning for Spinal Medical Report Generation
• Predicting Driver Fatigue in Automated Driving with Explainability
• A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking
• Quantum Algorithms for the Pathwise Lasso
• Robust Sample Weighting to Facilitate Individualized Treatment Rule Learning for a Target Population
• SurvLIME-Inf: A simplified modification of SurvLIME for explanation of machine learning survival models
• Adaptive wavelet distillation from neural networks through interpretations
• Multi-Stage Prototype Learning for Interpretable Time Series Classification
• What are the most important statistical ideas of the past 50 years?
• Explainable AI for ML jet taggers using expert variables and layerwise relevance propagation
• Disciplining Deliberation: A Sociotechnical Perspective on Machine Learning Trade-offs
• Taming Nonconvexity in Kernel Feature Selection -- Favorable Properties of the Laplace Kernel
• Stable discovery of interpretable subgroups via calibration in causal studies
• Learning from learning machines: a new generation of AI technology to meet the needs of science
• Interpretable machine learning models: a physics-based view
• Factor Importance Ranking and Selection using Total Indices
• Machine learning in the social and health sciences
• Trainable and Explainable Simplicial Map Neural Networks
• A Semiparametric Approach to Interpretable Machine Learning
• Distilling Black-Box Travel Mode Choice Model for Behavioral Interpretation
• A reluctant additive model framework for interpretable nonlinear individualized treatment rules
• A cautionary tale on fitting decision trees to data from additive models: generalization lower bounds
• Shapley variable importance clouds for interpretable machine learning
• Contextual Local Explanation for Black Box Classifiers
• Data-Driven Logistic Regression Ensembles With Applications in Genomics
• Self-supervised Representations and Node Embedding Graph Neural Networks for Accurate and Multi-scale Analysis of Materials
• Robust Semantic Interpretability: Revisiting Concept Activation Vectors
• Interpretability of machine-learning models in physical sciences
• Assessment of cognitive characteristics in intelligent systems and predictive ability
• An Interpretable Algorithm for Uveal Melanoma Subtyping from Whole Slide Cytology Images
• A Minimal Intervention Definition of Reverse Engineering a Neural Circuit
• Towards Transparent Application of Machine Learning in Video Processing
• Interpretable by Design: Learning Predictors by Composing Interpretable Queries
• Towards an Open Global Air Quality Monitoring Platform to Assess Children's Exposure to Air Pollutants in the Light of COVID-19 Lockdowns
• Generalized Constraints as A New Mathematical Problem in Artificial Intelligence: A Review and Perspective
• Do Input Gradients Highlight Discriminative Features?
• PPFL: A Personalized Federated Learning Framework for Heterogeneous Population
• On the price of explainability for some clustering problems
• Human Evaluation of Interpretability: The Case of AI-Generated Music Knowledge
• Making Logic Learnable With Neural Networks
• GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations
• The hidden architecture of connections: How do multidimensional identities shape our social networks?
• Investigating similarities and differences between South African and Sierra Leonean school outcomes using Machine Learning
• Towards Trustworthy Deception Detection: Benchmarking Model Robustness across Domains, Modalities, and Languages
• Challenges and Opportunities of Computational Social Science for Official Statistics
• The more polypersonal the better -- a short look on space geometry of fine-tuned layers
• DeepDrawing: A Deep Learning Approach to Graph Drawing
• Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning
• Inference on summaries of a model-agnostic longitudinal variable importance trajectory with application to suicide prevention
• Understanding Information Processing in Human Brain by Interpreting Machine Learning Models
• Interpretable Few-Shot Learning via Linear Distillation
• Understanding Interpretability by generalized distillation in Supervised Classification
• Discovering Invariances in Healthcare Neural Networks
• Interpretability of Blackbox Machine Learning Models through Dataview Extraction and Shadow Model creation
• Optimal Interpretability-Performance Trade-off of Classification Trees with Black-Box Reinforcement Learning
• Minimax Optimal Estimation of Stability Under Distribution Shift
• A Sparsity Algorithm with Applications to Corporate Credit Rating
• Rationalizing Medical Relation Prediction from Corpus-level Statistics