Understanding trained CNNs by indexing neuron selectivity

Abstract

The impressive performance of Convolutional Neural Networks (CNNs) when solving different vision problems is shadowed by their black-box nature and our consequent lack of understanding of the representations they build and how these representations are organized. To help understanding these issues, we propose to describe the activity of individual neurons by their Neuron Feature visualization and quantify their inherent selectivity with two specific properties. We explore selectivity indexes for: an image feature (color); and an image label (class membership). Our contribution is a framework to seek or classify neurons by indexing on these selectivity properties. It helps to find color selective neurons, such as a red-mushroom neuron in layer Conv4 or class selective neurons such as dog-face neurons in layer Conv5 in VGG-M, and establishes a methodology to derive other selectivity properties. Indexing on neuron selectivity can statistically draw how features and classes are represented through layers in a moment when the size of trained nets is growing and automatic tools to index neurons can be helpful.

Under review on Pattern Recognition Letters

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Explaining and Harnessing Adversarial Examples
• Understanding Neural Networks Through Deep Visualization
• Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Distilling a Neural Network Into a Soft Decision Tree
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Convergent Learning: Do different neural networks learn the same representations?
• Weighted principal component analysis: a weighted covariance eigendecomposition approach
• Revisiting the Importance of Individual Units in CNNs via Ablation
• DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation
• Why does Deep Learning work? - A perspective from Group Theory

Cited by in corpus (9)

• Towards falsifiable interpretability research
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
• A Multimodal Recommender System for Large-scale Assortment Generation in E-commerce
• Disentanglement of Color and Shape Representations for Continual Learning
• Scalable Visual Attribute Extraction through Hidden Layers of a Residual ConvNet
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Canoe : A System for Collaborative Learning for Neural Nets
• Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
• SeNA-CNN: Overcoming Catastrophic Forgetting in Convolutional Neural Networks by Selective Network Augmentation