Bayesian Compression for Deep Learning

Abstract

Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled and effective way to attack this problem is by adopting a Bayesian point of view, where through sparsity inducing priors we prune large parts of the network. We introduce two novelties in this paper: 1) we use hierarchical priors to prune nodes instead of individual weights, and 2) we use the posterior uncertainties to determine the optimal fixed point precision to encode the weights. Both factors significantly contribute to achieving the state of the art in terms of compression rates, while still staying competitive with methods designed to optimize for speed or energy efficiency.

arXiv commentPublished as a conference paper at NIPS 2017

References in corpus (31)

• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Improving neural networks by preventing co-adaptation of feature detectors
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Wide Residual Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Do Deep Nets Really Need to be Deep?
• Weight Uncertainty in Neural Networks
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Trained Ternary Quantization
• Predicting Parameters in Deep Learning
• Group Sparse Regularization for Deep Neural Networks
• Learning Structured Sparsity in Deep Neural Networks
• A General Framework for the Parametrization of Hierarchical Models
• Ladder Variational Autoencoders
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• Generalized Beta Mixtures of Gaussians
• Soft Weight-Sharing for Neural Network Compression
• Deep neural networks are robust to weight binarization and other non-linear distortions
• More is Less: A More Complicated Network with Less Inference Complexity
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Compressing Convolutional Neural Networks
• Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks
• Accelerating Deep Convolutional Networks using low-precision and sparsity
• A Scale Mixture Perspective of Multiplicative Noise in Neural Networks
• Automatic Relevance Determination For Deep Generative Models

Cited by in corpus (147)

• An Introduction to Variational Autoencoders
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• The State of Sparsity in Deep Neural Networks
• Neural Granger Causality
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Split learning for health: Distributed deep learning without sharing raw patient data
• Learning Sparse Neural Networks through $L_0$ Regularization
• Compressing Neural Networks using the Variational Information Bottleneck
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Bayesian Bits: Unifying Quantization and Pruning
• What is the State of Neural Network Pruning?
• Selfless Sequential Learning
• Layer-adaptive sparsity for the Magnitude-based Pruning
• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• Physics-informed deep generative models
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Noise Contrastive Priors for Functional Uncertainty
• Conditional Neural Processes
• Batch-Shaping for Learning Conditional Channel Gated Networks
• Probabilistic Binary Neural Networks
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
• AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• The Functional Neural Process
• The Difficulty of Training Sparse Neural Networks
• Overfitting for Fun and Profit: Instance-Adaptive Data Compression
• Improving and Understanding Variational Continual Learning
• Variational Gaussian Dropout is not Bayesian
• Rethinking Weight Decay For Efficient Neural Network Pruning
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization
• Continual Learning with Adaptive Weights (CLAW)
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Communication-Efficient Edge AI: Algorithms and Systems
• COIN: COmpression with Implicit Neural representations
• Task dependent Deep LDA pruning of neural networks
• Learned Threshold Pruning
• Reconfigurable Intelligent Surface for Green Edge Inference
• A Closer Look at Structured Pruning for Neural Network Compression
• FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
• Resource-Efficient Neural Networks for Embedded Systems
• Degree-Quant: Quantization-Aware Training for Graph Neural Networks
• Rate Distortion For Model Compression: From Theory To Practice
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning
• On the Compression of Neural Networks Using ${\ell }_0$-Norm Regularization and Weight Pruning
• Iteratively Training Look-Up Tables for Network Quantization
• Multi-period Time Series Modeling with Sparsity via Bayesian Variational Inference
• Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
• Relaxed Quantization for Discretized Neural Networks
• Selfish Sparse RNN Training
• Generalized Variational Continual Learning
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• Continual Learning in Neural Networks
• Neural network relief: a pruning algorithm based on neural activity
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
• Bayesian Sparsification of Recurrent Neural Networks
• Improved Bayesian Compression
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Mirror Descent View for Neural Network Quantization
• Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning
• A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics
• Bayesian Photonic Accelerators for Energy Efficient and Noise Robust Neural Processing
• GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
• Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters
• Radial and Directional Posteriors for Bayesian Neural Networks
• Scalable Model Compression by Entropy Penalized Reparameterization
• DeepCABAC: Context-adaptive binary arithmetic coding for deep neural network compression
• SYMOG: learning symmetric mixture of Gaussian modes for improved fixed-point quantization
• Interpretable VAEs for nonlinear group factor analysis
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• Neural Network Compression Via Sparse Optimization
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Consistent Sparse Deep Learning: Theory and Computation
• Differentiable Joint Pruning and Quantization for Hardware Efficiency
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• OOGAN: Disentangling GAN with One-Hot Sampling and Orthogonal Regularization
• Horseshoe Regularization for Machine Learning in Complex and Deep Models
• How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
• DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
• Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization
• Topological Insights into Sparse Neural Networks
• Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks
• Distilling with Performance Enhanced Students
• Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Accelerate CNN via Recursive Bayesian Pruning
• Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
• Importance Weighted Hierarchical Variational Inference
• Nonparametric Bayesian Deep Networks with Local Competition
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks
• Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes
• ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• Training Sparse Neural Networks using Compressed Sensing
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Leveraging Active Subspaces to Capture Epistemic Model Uncertainty in Deep Generative Models for Molecular Design
• Analysis of memory consumption by neural networks based on hyperparameters
• Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks
• Doubly Semi-Implicit Variational Inference
• Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
• Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains
• Neuron ranking -- an informed way to condense convolutional neural networks architecture
• Learning Multimodal Fixed-Point Weights using Gradient Descent
• The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics
• Mining the Weights Knowledge for Optimizing Neural Network Structures
• Variational Bayesian Dropout with a Hierarchical Prior
• The Outer Product Structure of Neural Network Derivatives
• A fast asynchronous MCMC sampler for sparse Bayesian inference
• BiDet: An Efficient Binarized Object Detector
• The Pond Dilemma with Heterogeneous Relative Concerns
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• Bayesian Sparsification of Gated Recurrent Neural Networks
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network
• Low-memory convolutional neural networks through incremental depth-first processing
• FeTa: A DCA Pruning Algorithm with Generalization Error Guarantees
• Revisiting hard thresholding for DNN pruning
• Rodent: Relevance determination in differential equations
• Partition and Code: learning how to compress graphs
• Efficient and Robust Machine Learning for Real-World Systems
• DiffPrune: Neural Network Pruning with Deterministic Approximate Binary Gates and $L_0$ Regularization
• Copula-like Variational Inference
• Dirichlet Pruning for Neural Network Compression
• Variational Bayesian Sequence-to-Sequence Networks for Memory-Efficient Sign Language Translation
• Where Do Human Heuristics Come From?
• Bayesian Sparsification Methods for Deep Complex-valued Networks
• Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning
• Learning Gradual Argumentation Frameworks using Genetic Algorithms
• MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression
• Structured Bayesian Compression for Deep models in mobile enabled devices for connected healthcare
• Neural Plasticity Networks
• Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks
• What Happens on the Edge, Stays on the Edge: Toward Compressive Deep Learning
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• Dynamic Regularizer with an Informative Prior
• Efficient Approximate Inference with Walsh-Hadamard Variational Inference
• TreeGen -- a Monte Carlo generator for data frames
• Sparse within Sparse Gaussian Processes using Neighbor Information
• An Adaptive Empirical Bayesian Method for Sparse Deep Learning