Bayesian sparsification for deep neural networks with Bayesian model reduction

Abstract

Deep learning's immense capabilities are often constrained by the complexity of its models, leading to an increasing demand for effective sparsification techniques. Bayesian sparsification for deep learning emerges as a crucial approach, facilitating the design of models that are both computationally efficient and competitive in terms of performance across various deep learning applications. The state-of-the-art -- in Bayesian sparsification of deep neural networks -- combines structural shrinkage priors on model weights with an approximate inference scheme based on stochastic variational inference. However, model inversion of the full generative model is exceptionally computationally demanding, especially when compared to standard deep learning of point estimates. In this context, we advocate for the use of Bayesian model reduction (BMR) as a more efficient alternative for pruning of model weights. As a generalization of the Savage-Dickey ratio, BMR allows a post-hoc elimination of redundant model weights based on the posterior estimates under a straightforward (non-hierarchical) generative model. Our comparative study highlights the advantages of the BMR method relative to established approaches based on hierarchical horseshoe priors over model weights. We illustrate the potential of BMR across various deep learning architectures, from classical networks like LeNet to modern frameworks such as Vision Transformers and MLP-Mixers.

References in corpus (12)

• Variational Inference: A Review for Statisticians
• Gaussian Error Linear Units (GELUs)
• Pyro: Deep Universal Probabilistic Programming
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• The Case for Bayesian Deep Learning
• Bayesian deep learning with hierarchical prior: Predictions from limited and noisy data
• Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization
• Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty
• On the optimization and pruning for Bayesian deep learning
• A Generalized Savage-Dickey Ratio