The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

Abstract

Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization. For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence. This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance. Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Variational Inference with Normalizing Flows
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Practical Deep Learning with Bayesian Principles
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Bayesian Layers: A Module for Neural Network Uncertainty
• Variational Boosting: Iteratively Refining Posterior Approximations
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning
• Noisy Natural Gradient as Variational Inference
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
• Probabilistic Meta-Representations Of Neural Networks

Cited by in corpus (10)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Bayesian Neural Network Priors Revisited
• Repulsive Deep Ensembles are Bayesian
• Bayesian Deep Learning via Subnetwork Inference
• On Stein Variational Neural Network Ensembles
• Quantum Bayesian Neural Networks
• Informative Bayesian Neural Network Priors for Weak Signals
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• Pathologies in priors and inference for Bayesian transformers