Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?
arXiv:2105.10719
Abstract
Masking some input variables of a deep neural network (DNN) and computing output changes on the masked input sample represent a typical way to compute attributions of input variables in the sample. People usually mask an input variable using its baseline value. However, there is no theory to examine whether baseline value faithfully represents the absence of an input variable, \emph{i.e.,} removing all signals from the input variable. Fortunately, recent studies show that the inference score of a DNN can be strictly disentangled into a set of causal patterns (or concepts) encoded by the DNN. Therefore, we propose to use causal patterns to examine the faithfulness of baseline values. More crucially, it is proven that causal patterns can be explained as the elementary rationale of the Shapley value. Furthermore, we propose a method to learn optimal baseline values, and experimental results have demonstrated its effectiveness.
arXiv commentaccepted by ICLR 2023
References in corpus (13)
- Axiomatic Attribution for Deep Networks
- Understanding Neural Networks Through Deep Visualization
- Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
- Consistent Individualized Feature Attribution for Tree Ensembles
- Understanding Global Feature Contributions With Additive Importance Measures
- Local Rule-Based Explanations of Black Box Decision Systems
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- Problems with Shapley-value-based explanations as feature importance measures
- Hierarchical interpretations for neural network predictions
- Real Time Image Saliency for Black Box Classifiers
- Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
- Feature Removal Is a Unifying Principle for Model Explanation Methods
- Learning Global Pairwise Interactions with Bayesian Neural Networks