How Important Is a Neuron?

Abstract

The problem of attributing a deep network's prediction to its \emph{input/base} features is well-studied. We introduce the notion of \emph{conductance} to extend the notion of attribution to the understanding the importance of \emph{hidden} units. Informally, the conductance of a hidden unit of a deep network is the \emph{flow} of attribution via this hidden unit. We use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We evaluate the effectiveness of conductance in multiple ways, including theoretical properties, ablation studies, and a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a sentiment analysis network over reviews. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.

arXiv commentunder submission

References in corpus (6)

• Striving for Simplicity: The All Convolutional Net
• Understanding Neural Networks Through Deep Visualization
• How to Explain Individual Classification Decisions
• On the importance of single directions for generalization
• Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers
• Evaluating the visualization of what a Deep Neural Network has learned

Cited by in corpus (20)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Neuron Shapley: Discovering the Responsible Neurons
• Continuous Human Action Recognition for Human-Machine Interaction: A Review
• Towards falsifiable interpretability research
• Shapley Flow: A Graph-based Approach to Interpreting Model Predictions
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
• Towards Understanding Neural Machine Translation with Word Importance
• Explaining in Style: Training a GAN to explain a classifier in StyleSpace
• Model Fusion via Optimal Transport
• Robust Models Are More Interpretable Because Attributions Look Normal
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• Investigating Saturation Effects in Integrated Gradients
• On the relationship between class selectivity, dimensionality, and robustness
• Computing Linear Restrictions of Neural Networks
• Neuron with Steady Response Leads to Better Generalization
• Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models
• Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
• Influence Patterns for Explaining Information Flow in BERT
• Emergent symbolic language based deep medical image classification