Structured Bayesian Pruning via Log-Normal Multiplicative Noise

Abstract

Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.

Cited by in corpus (50)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• R-Drop: Regularized Dropout for Neural Networks
• Compressing Neural Networks using the Variational Information Bottleneck
• Rigging the Lottery: Making All Tickets Winners
• Dynamic Model Pruning with Feedback
• Balanced Sparsity for Efficient DNN Inference on GPU
• Survey of Dropout Methods for Deep Neural Networks
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Variational Gaussian Dropout is not Bayesian
• Rethinking Weight Decay For Efficient Neural Network Pruning
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Adversarial Neural Pruning with Latent Vulnerability Suppression
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• Dynamic Hard Pruning of Neural Networks at the Edge of the Internet
• Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• Variance Networks: When Expectation Does Not Meet Your Expectations
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
• Bayesian Sparsification of Recurrent Neural Networks
• Functionality-Oriented Convolutional Filter Pruning
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method
• GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
• Radial and Directional Posteriors for Bayesian Neural Networks
• Neural Network Compression Via Sparse Optimization
• Multi-Objective Pruning for CNNs Using Genetic Algorithm
• How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
• Accelerate CNN via Recursive Bayesian Pruning
• Principal Component Networks: Parameter Reduction Early in Training
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks
• Intelligence plays dice: Stochasticity is essential for machine learning
• ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model
• Nonparametric Bayesian Deep Networks with Local Competition
• Stochastic Model Pruning via Weight Dropping Away and Back
• Neuron ranking -- an informed way to condense convolutional neural networks architecture
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• Mining the Weights Knowledge for Optimizing Neural Network Structures
• DiffPrune: Neural Network Pruning with Deterministic Approximate Binary Gates and $L_0$ Regularization
• Variational Bayesian Dropout with a Hierarchical Prior
• Dirichlet Pruning for Neural Network Compression
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices
• Joint Regularization on Activations and Weights for Efficient Neural Network Pruning
• On Connections between Regularizations for Improving DNN Robustness
• Variational Bayesian Sequence-to-Sequence Networks for Memory-Efficient Sign Language Translation
• Bayesian Generative Models for Knowledge Transfer in MRI Semantic Segmentation Problems
• Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks