SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient

Abstract

Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite, the fact that these matrices are known to result in poor uncertainty estimates. To address this issue, we propose a new stochastic, low-rank, approximate natural-gradient (SLANG) method for variational inference in large, deep models. Our method estimates a "diagonal plus low-rank" structure based solely on back-propagated gradients of the network log-likelihood. This requires strictly less gradient computations than methods that compute the gradient of the whole variational objective. Empirical evaluations on standard benchmarks confirm that SLANG enables faster and more accurate estimation of uncertainty than mean-field methods, and performs comparably to state-of-the-art methods.

arXiv commentNeurIPS 2018 final version

Cited by in corpus (16)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Lifelong Generative Modeling
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Bayesian Deep Learning via Subnetwork Inference
• The Bayesian Learning Rule
• Kernelized Wasserstein Natural Gradient
• Tractable structured natural gradient descent using local parameterizations
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
• Copula-like Variational Inference
• Manifold Optimization Assisted Gaussian Variational Approximation
• Pathologies in priors and inference for Bayesian transformers