Feature relevance quantification in explainable AI: A causal problem

Abstract

We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between observational and interventional conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that unconditional rather than conditional expectations provide the right notion of dropping features in contradiction to the theoretical justification of the software package SHAP. Parts of SHAP are unaffected because unconditional expectations (which we argue to be conceptually right) are used as approximation for the conditional ones, which encouraged others to `improve' SHAP in a way that we believe to be flawed.

11 pages, 5 figures

References in corpus (5)

• Explaining and Harnessing Adversarial Examples
• Axiomatic Attribution for Deep Networks
• Consistent Individualized Feature Attribution for Tree Ensembles
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Neural Network Attributions: A Causal Perspective

Cited by in corpus (34)

• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
• Understanding Global Feature Contributions With Additive Importance Measures
• Explaining by Removing: A Unified Framework for Model Explanation
• Model-agnostic Feature Importance and Effects with Dependent Features -- A Conditional Subgroup Approach
• Impossibility Theorems for Feature Attribution
• Relationships between distortions of inorganic framework and band gap of layered hybrid halide perovskites
• Counterfactual Shapley Additive Explanations
• Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams
• Fast TreeSHAP: Accelerating SHAP Value Computation for Trees
• Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud
• True to the Model or True to the Data?
• Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability
• Shapley Flow: A Graph-based Approach to Interpreting Model Predictions
• Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena
• Rational Shapley Values
• Fast Hierarchical Games for Image Explanations
• Feature Removal Is a Unifying Principle for Model Explanation Methods
• SHAFF: Fast and consistent SHApley eFfect estimates via random Forests
• Fooling Partial Dependence via Data Poisoning
• Wasserstein-based fairness interpretability framework for machine learning models
• PredDiff: Explanations and Interactions from Conditional Expectations
• Rationalizing Predictions by Adversarial Information Calibration
• Unifying local and global model explanations by functional decomposition of low dimensional structures
• On the Connection between Game-Theoretic Feature Attributions and Counterfactual Explanations
• On marginal feature attributions of tree-based models
• Revealing drivers and risks for power grid frequency stability with explainable AI
• Replication-Robust Payoff-Allocation for Machine Learning Data Markets
• On the Tractability of SHAP Explanations
• FastSHAP: Real-Time Shapley Value Estimation
• The Shapley Value of coalition of variables provides better explanations
• Data-driven advice for interpreting local and global model predictions in bioinformatics problems
• Technical Report: Partial Dependence through Stratification
• An exploration of the influence of path choice in game-theoretic attribution algorithms
• Is Shapley Explanation for a model unique?