Neuron ranking -- an informed way to condense convolutional neural networks architecture

Abstract

Convolutional neural networks (CNNs) in recent years have made a dramatic impact in science, technology and industry, yet the theoretical mechanism of CNN architecture design remains surprisingly vague. The CNN neurons, including its distinctive element, convolutional filters, are known to be learnable features, yet their individual role in producing the output is rather unclear. The thesis of this work is that not all neurons are equally important and some of them contain more useful information to perform a given task . Consequently, we quantify the significance of each filter and rank its importance in describing input to produce the desired output. This work presents two different methods: (1) a game theoretical approach based on Shapley value which computes the marginal contribution of each filter; and (2) a probabilistic approach based on what-we-call, the Importance switch using variational inference. Strikingly, these two vastly different methods produce similar experimental results, confirming the general theory that some of the filters are inherently more important that the others. The learned ranks can be readily useable for network compression and interpretability.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Playing Atari with Deep Reinforcement Learning
• Understanding Neural Networks Through Deep Visualization
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• On the Number of Linear Regions of Deep Neural Networks
• Pruning Filters for Efficient ConvNets
• Visualizing and Understanding Convolutional Networks
• Bayesian Compression for Deep Learning
• Learning Sparse Neural Networks through $L_0$ Regularization
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Implicit Reparameterization Gradients
• Stochastic gradient variational Bayes for gamma approximating distributions
• FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks