Dirichlet Pruning for Neural Network Compression

Abstract

We introduce Dirichlet pruning, a novel post-processing technique to transform a large neural network model into a compressed one. Dirichlet pruning is a form of structured pruning that assigns the Dirichlet distribution over each layer's channels in convolutional layers (or neurons in fully-connected layers) and estimates the parameters of the distribution over these units using variational inference. The learned distribution allows us to remove unimportant units, resulting in a compact architecture containing only crucial features for a task at hand. The number of newly introduced Dirichlet parameters is only linear in the number of channels, which allows for rapid training, requiring as little as one epoch to converge. We perform extensive experiments, in particular on larger architectures such as VGG and ResNet (45% and 58% compression rate, respectively) where our method achieves the state-of-the-art compression performance and provides interpretable features as a by-product.

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Playing Atari with Deep Reinforcement Learning
• Language Models are Few-Shot Learners
• Understanding Neural Networks Through Deep Visualization
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Bayesian Dark Knowledge
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Implicit Reparameterization Gradients
• Faster gaze prediction with dense networks and Fisher pruning
• Stochastic gradient variational Bayes for gamma approximating distributions
• A Closer Look at Structured Pruning for Neural Network Compression
• FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• Radial and Directional Posteriors for Bayesian Neural Networks