Defining and Quantifying the Emergence of Sparse Concepts in DNNs
arXiv:2111.06206
Abstract
This paper aims to illustrate the concept-emerging phenomenon in a trained DNN. Specifically, we find that the inference score of a DNN can be disentangled into the effects of a few interactive concepts. These concepts can be understood as causal patterns in a sparse, symbolic causal graph, which explains the DNN. The faithfulness of using such a causal graph to explain the DNN is theoretically guaranteed, because we prove that the causal graph can well mimic the DNN's outputs on an exponential number of different masked samples. Besides, such a causal graph can be further simplified and re-written as an And-Or graph (AOG), without losing much explanation accuracy.
References in corpus (17)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Understanding Neural Networks Through Deep Visualization
- Consistent Individualized Feature Attribution for Tree Ensembles
- Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
- Distilling a Neural Network Into a Soft Decision Tree
- Understanding Global Feature Contributions With Additive Importance Measures
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
- Hierarchical interpretations for neural network predictions
- Explainable Neural Networks based on Additive Index Models
- CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training
- Causal Learning and Explanation of Deep Neural Networks via Autoencoded Activations
- Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
- Explanations for Monotonic Classifiers
- Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?
- Interpreting Multivariate Shapley Interactions in DNNs