Compressing Neural Networks using the Variational Information Bottleneck

Abstract

Neural networks can be compressed to reduce memory and computational requirements, or to increase accuracy by facilitating the use of a larger base architecture. In this paper we focus on pruning individual neurons, which can simultaneously trim model size, FLOPs, and run-time memory. To improve upon the performance of existing compression algorithms we utilize the information bottleneck principle instantiated via a tractable variational bound. Minimization of this information theoretic bound reduces the redundancy between adjacent layers by aggregating useful information into a subset of neurons that can be preserved. In contrast, the activations of disposable neurons are shut off via an attractive form of sparse regularization that emerges naturally from this framework, providing tangible advantages over traditional sparsity penalties without contributing additional tuning parameters to the energy landscape. We demonstrate state-of-the-art compression rates across an array of datasets and network architectures.

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Estimating Mutual Information
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• The information bottleneck method
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Weight Uncertainty in Neural Networks
• Predicting Parameters in Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Deep Variational Information Bottleneck
• Bayesian Compression for Deep Learning
• Ladder Variational Autoencoders
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Revisiting Bayesian Blind Deconvolution
• More is Less: A More Complicated Network with Less Inference Complexity
• DropNeuron: Simplifying the Structure of Deep Neural Networks
• Strong NP-Hardness for Sparse Optimization with Concave Penalty Functions
• A Scale Mixture Perspective of Multiplicative Noise in Neural Networks
• Generalized Dropout

Cited by in corpus (47)

• The State of Sparsity in Deep Neural Networks
• Rigging the Lottery: Making All Tickets Winners
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Batch-Shaping for Learning Conditional Channel Gated Networks
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
• Rethinking Weight Decay For Efficient Neural Network Pruning
• Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedback
• Learned Threshold Pruning
• Adversarial Neural Pruning with Latent Vulnerability Suppression
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Caveats for information bottleneck in deterministic scenarios
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• PAC-Bayes Information Bottleneck
• An Information Theory-inspired Strategy for Automatic Network Pruning
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• CUP: Cluster Pruning for Compressing Deep Neural Networks
• HadaNets: Flexible Quantization Strategies for Neural Networks
• Information-based Disentangled Representation Learning for Unsupervised MR Harmonization
• Convolutional Neural Network Simplification with Progressive Retraining
• Information Bottleneck Theory on Convolutional Neural Networks
• Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
• Information Theoretic Meta Learning with Gaussian Processes
• Differentiable Joint Pruning and Quantization for Hardware Efficiency
• Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
• Multi-Task Variational Information Bottleneck
• Image Captioning with Sparse Recurrent Neural Network
• ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model
• Nonparametric Bayesian Deep Networks with Local Competition
• ESPN: Extremely Sparse Pruned Networks
• Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains
• Stochastic Model Pruning via Weight Dropping Away and Back
• Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks
• Learning Robust Variational Information Bottleneck with Reference
• Dynamic Narrowing of VAE Bottlenecks Using GECO and L0 Regularization
• Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy
• Towards Accurate Quantization and Pruning via Data-free Knowledge Transfer
• Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning
• Variational Bayesian Dropout with a Hierarchical Prior
• Revisiting hard thresholding for DNN pruning
• M-estimation with the Trimmed l1 Penalty
• BiDet: An Efficient Binarized Object Detector
• Neural Network Activation Quantization with Bitwise Information Bottlenecks
• Pruning-Aware Merging for Efficient Multitask Inference
• Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks
• Artificial stochastic neural network on the base of double quantum wells
• Joint Regularization on Activations and Weights for Efficient Neural Network Pruning