What do CNN neurons learn: Visualization & Clustering

Abstract

In recent years convolutional neural networks (CNN) have shown striking progress in various tasks. However, despite the high performance, the training and prediction process remains to be a black box, leaving it a mystery to extract what neurons learn in CNN. In this paper, we address the problem of interpreting a CNN from the aspects of the input image's focus and preference, and the neurons' domination, activation and contribution to a concrete final prediction. Specifically, we use two techniques - visualization and clustering - to tackle the problems above. Visualization means the method of gradient descent on image pixel, and in clustering section two algorithms are proposed to cluster respectively over image categories and network neurons. Experiments and quantitative analyses have demonstrated the effectiveness of the two methods in explaining the question: what do neurons learn.

arXiv comment9 pages, 10 figures, tech report

References in corpus (7)

• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Intriguing properties of neural networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• SmoothGrad: removing noise by adding noise
• Entropy and mutual information in models of deep neural networks
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks