Practical Deep Learning with Bayesian Principles

Abstract

Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch normalisation, data augmentation, and distributed training, we achieve similar performance in about the same number of epochs as the Adam optimiser, even on large datasets such as ImageNet. Importantly, the benefits of Bayesian principles are preserved: predictive probabilities are well-calibrated, uncertainties on out-of-distribution data are improved, and continual-learning performance is boosted. This work enables practical deep learning while preserving benefits of Bayesian principles. A PyTorch implementation is available as a plug-and-play optimiser.

arXiv commentNeurIPS 2019

References in corpus (14)

• Efficient Estimation of Word Representations in Vector Space
• Decoupled Weight Decay Regularization
• Overcoming catastrophic forgetting in neural networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Variational Continual Learning
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
• Efficient Per-Example Gradient Computations
• Improving and Understanding Variational Continual Learning

Cited by in corpus (49)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• What Are Bayesian Neural Network Posteriors Really Like?
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
• Continual Deep Learning by Functional Regularisation of Memorable Past
• Explainable Artificial Intelligence for Bayesian Neural Networks: Towards trustworthy predictions of ocean dynamics
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Getting a CLUE: A Method for Explaining Uncertainty Estimates
• Tighter risk certificates for neural networks
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Laplace Redux -- Effortless Bayesian Deep Learning
• Laplace HypoPINN: Physics-Informed Neural Network for hypocenter localization and its predictive uncertainty
• Bayesian Neural Network Priors Revisited
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Training Binary Neural Networks using the Bayesian Learning Rule
• Learning under Model Misspecification: Applications to Variational and Ensemble methods
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Unsupervised Knowledge-Transfer for Learned Image Reconstruction
• Explaining Bayesian Neural Networks
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Approximate Inference Turns Deep Networks into Gaussian Processes
• Bayesian Deep Learning via Subnetwork Inference
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• Improving predictions of Bayesian neural nets via local linearization
• Being Bayesian about Categorical Probability
• Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning
• The Bayesian Learning Rule
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• A statistical theory of cold posteriors in deep neural networks
• Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable?
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Tractable structured natural gradient descent using local parameterizations
• Tractable Approximate Gaussian Inference for Bayesian Neural Networks
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening
• Natural continual learning: success is a journey, not (just) a destination
• Handling the Positive-Definite Constraint in the Bayesian Learning Rule
• Quantifying Model Uncertainty in Inverse Problems via Bayesian Deep Gradient Descent
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Being a Bit Frequentist Improves Bayesian Neural Networks
• Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Mixture-of-Variational-Experts for Continual Learning
• Natural Gradient Variational Inference with Gaussian Mixture Models