Defining and Quantifying the Emergence of Sparse Concepts in DNNs
arXiv:2111.06206
Abstract
This paper aims to illustrate the concept-emerging phenomenon in a trained DNN. Specifically, we find that the inference score of a DNN can be disentangled into the effects of a few interactive concepts. These concepts can be understood as causal patterns in a sparse, symbolic causal graph, which explains the DNN. The faithfulness of using such a causal graph to explain the DNN is theoretically guaranteed, because we prove that the causal graph can well mimic the DNN's outputs on an exponential number of different masked samples. Besides, such a causal graph can be further simplified and re-written as an And-Or graph (AOG), without losing much explanation accuracy.
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Understanding Neural Networks Through Deep Visualization
- Distilling a Neural Network Into a Soft Decision Tree
- Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
- CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training
- Explanations for Monotonic Classifiers
- Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?