Explaining by Removing: A Unified Framework for Model Explanation

Abstract

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We describe a new unified class of methods, removal-based explanations, that are based on the principle of simulating feature removal to quantify each feature's influence. These methods vary in several respects, so we develop a framework that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 26 existing methods, including several of the most widely used approaches: SHAP, LIME, Meaningful Perturbations, and permutation tests. This newly understood class of explanation methods has rich connections that we examine using tools that have been largely overlooked by the explainability literature. To anchor removal-based explanations in cognitive psychology, we show that feature removal is a simple application of subtractive counterfactual reasoning. Ideas from cooperative game theory shed light on the relationships and trade-offs among different methods, and we derive conditions under which all removal-based explanations have information-theoretic interpretations. Through this analysis, we develop a unified framework that helps practitioners better understand model explanation tools, and that offers a strong theoretical foundation upon which future explainability research can build.

arXiv commentUpdated with JMLR revisions, arXiv admin note: text overlap with arXiv:2011.03623

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Axiomatic Attribution for Deep Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• SmoothGrad: removing noise by adding noise
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences
• Understanding Global Feature Contributions With Additive Importance Measures
• Feature relevance quantification in explainable AI: A causal problem
• Explaining Image Classifiers by Counterfactual Generation
• CXPlain: Causal Explanations for Model Interpretation under Uncertainty
• True to the Model or True to the Data?
• Explaining black box decisions by Shapley cohort refinement
• Weighted Banzhaf power and interaction indexes through weighted approximations of games
• A Universal Marginalizer for Amortized Inference in Generative Models

Cited by in corpus (29)

• From Clustering to Cluster Explanations via Neural Networks
• Toward Explainable AI for Regression Models
• From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathology
• Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams
• Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
• sMRI-PatchNet: A novel explainable patch-based deep learning network for Alzheimer's disease diagnosis and discriminative atrophy localisation with Structural MRI
• Calibrating AI Models for Few-Shot Demodulation via Conformal Prediction
• Sampling Permutations for Shapley Value Estimation
• Fast Hierarchical Games for Image Explanations
• Insights Into the Inner Workings of Transformer Models for Protein Function Prediction
• Counterfactual Explanations for Deep Learning-Based Traffic Forecasting
• PredDiff: Explanations and Interactions from Conditional Expectations
• Does Your Model Think Like an Engineer? Explainable AI for Bearing Fault Detection with Deep Learning
• Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regression
• Beyond Low Earth Orbit: Biological Research, Artificial Intelligence, and Self-Driving Labs
• ShaRP: Explaining Rankings and Preferences with Shapley Values
• Inferring feature importance with uncertainties in high-dimensional data
• FastSHAP: Real-Time Shapley Value Estimation
• Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning
• FW-Shapley: Real-time Estimation of Weighted Shapley Values
• Identifying the post-pandemic determinants of low performing students in Latin America through Interpretable Machine Learning methods
• Ensembles of Random SHAPs
• Do not explain without context: addressing the blind spot of model explanations
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance
• Streamlining models with explanations in the learning loop
• Explainability as statistical inference
• Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation