Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs

Abstract

The properties of individual neurons are often analyzed in order to understand the biological and artificial neural networks in which they're embedded. Class selectivity-typically defined as how different a neuron's responses are across different classes of stimuli or data samples-is commonly used for this purpose. However, it remains an open question whether it is necessary and/or sufficient for deep neural networks (DNNs) to learn class selectivity in individual units. We investigated the causal impact of class selectivity on network function by directly regularizing for or against class selectivity. Using this regularizer to reduce class selectivity across units in convolutional neural networks increased test accuracy by over 2% for ResNet18 trained on Tiny ImageNet. For ResNet20 trained on CIFAR10 we could reduce class selectivity by a factor of 2.5 with no impact on test accuracy, and reduce it nearly to zero with only a small ($\sim$2%) drop in test accuracy. In contrast, regularizing to increase class selectivity significantly decreased test accuracy across all models and datasets. These results indicate that class selectivity in individual units is neither sufficient nor strictly necessary, and can even impair DNN performance. They also encourage caution when focusing on the properties of single units as representative of the mechanisms by which DNNs function.

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Understanding Neural Networks Through Deep Visualization
• Visualizing and Understanding Recurrent Networks
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Understanding the Role of Individual Units in a Deep Neural Network
• Learning to Generate Reviews and Discovering Sentiment
• On the importance of single directions for generalization
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Ablation Studies in Artificial Neural Networks
• Revisiting the Importance of Individual Units in CNNs via Ablation
• How Important Is a Neuron?
• Understanding trained CNNs by indexing neuron selectivity
• Trivializations for Gradient-Based Optimization on Manifolds
• Ablation of a Robot's Brain: Neural Networks Under a Knife
• Deleting object selective units in a fully-connected layer of deep convolutional networks improves classification performance

Cited by in corpus (9)

• Self-supervised learning through the eyes of a child
• Towards falsifiable interpretability research
• Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena
• Deeply Explain CNN via Hierarchical Decomposition
• On the relationship between class selectivity, dimensionality, and robustness
• Intraclass clustering: an implicit learning ability that regularizes DNNs
• Trade-offs of Local SGD at Scale: An Empirical Study
• Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
• Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?