Feature Removal Is a Unifying Principle for Model Explanation Methods

Abstract

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We examine the literature and find that many methods are based on a shared principle of explaining by removing - essentially, measuring the impact of removing sets of features from a model. These methods vary in several respects, so we develop a framework for removal-based explanations that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 26 existing methods, including several of the most widely used approaches (SHAP, LIME, Meaningful Perturbations, permutation tests). Exposing the fundamental similarities between these methods empowers users to reason about which tools to use, and suggests promising directions for ongoing model explainability research.

arXiv commentUpdated for consistency with arxiv:2011.14878

References in corpus (14)

• Axiomatic Attribution for Deep Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• SmoothGrad: removing noise by adding noise
• Object Detectors Emerge in Deep Scene CNNs
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences
• Understanding Global Feature Contributions With Additive Importance Measures
• Feature relevance quantification in explainable AI: A causal problem
• Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
• CXPlain: Causal Explanations for Model Interpretation under Uncertainty
• L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data
• Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability
• Shapley explainability on the data manifold

Cited by in corpus (7)

• Explainability in Music Recommender Systems
• Reinforced Causal Explainer for Graph Neural Networks
• The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
• On the Connection between Game-Theoretic Feature Attributions and Counterfactual Explanations
• Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?
• Towards Rigorous Interpretations: a Formalisation of Feature Attribution
• Shapley variable importance clouds for interpretable machine learning