Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation

Abstract

In this work, we investigate the use of three information-theoretic quantities -- entropy, mutual information with the class variable, and a class selectivity measure based on Kullback-Leibler divergence -- to understand and study the behavior of already trained fully-connected feed-forward neural networks. We analyze the connection between these information-theoretic quantities and classification performance on the test set by cumulatively ablating neurons in networks trained on MNIST, FashionMNIST, and CIFAR-10. Our results parallel those recently published by Morcos et al., indicating that class selectivity is not a good indicator for classification performance. However, looking at individual layers separately, both mutual information and class selectivity are positively correlated with classification performance, at least for networks with ReLU activation functions. We provide explanations for this phenomenon and conclude that it is ill-advised to compare the proposed information-theoretic quantities across layers. Furthermore, we show that cumulative ablation of neurons with ascending or descending information-theoretic quantities can be used to formulate hypotheses regarding the joint behavior of multiple neurons, such as redundancy and synergy, with comparably low computational cost. We also draw connections to the information bottleneck theory for neural networks.

12 pages; accepted for publication in IEEE Transactions on Neural Networks and Learning Systems

References in corpus (11)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Understanding the Role of Individual Units in a Deep Neural Network
• Nonnegative Decomposition of Multivariate Information
• On the importance of single directions for generalization
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Ablation Studies in Artificial Neural Networks
• Revisiting the Importance of Individual Units in CNNs via Ablation
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• NoiseOut: A Simple Way to Prune Neural Networks
• Distilling Critical Paths in Convolutional Neural Networks
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks

Cited by in corpus (7)

• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• A Novel Approach to the Partial Information Decomposition
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• Estimating Differential Entropy under Gaussian Convolutions
• Identifying Critical Neurons in ANN Architectures using Mixed Integer Programming
• Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding