Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

Abstract

Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented within the Adam optimizer by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate. This requires lower memory, computation, and implementation effort than existing VI methods, while obtaining uncertainty estimates of comparable quality. Our empirical results confirm this and further suggest that the weight-perturbation in our algorithm could be useful for exploration in reinforcement learning and stochastic optimization.

arXiv commentCamera ready version

References in corpus (10)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Weight Uncertainty in Neural Networks
• Noisy Networks for Exploration
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Bayesian Dark Knowledge
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Fast Variational Inference in the Conjugate Exponential Family
• On Graduated Optimization for Stochastic Non-Convex Problems
• Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
• Noisy Natural Gradient as Variational Inference

Cited by in corpus (33)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Adversarial Weight Perturbation Helps Robust Generalization
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• What Are Bayesian Neural Network Posteriors Really Like?
• Task Agnostic Continual Learning Using Online Variational Bayes
• The Case for Bayesian Deep Learning
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Amortized Variational Inference: A Systematic Review
• Uncertainty-guided Continual Learning with Bayesian Neural Networks
• Advances in Variational Inference
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Inhibited Softmax for Uncertainty Estimation in Neural Networks
• Calibrating Uncertainties in Object Localization Task
• Training Binary Neural Networks using the Bayesian Learning Rule
• Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
• A Generalization Bound for Online Variational Inference
• Bayesian Deep Learning via Subnetwork Inference
• Improving predictions of Bayesian neural nets via local linearization
• Eigenvalue Corrected Noisy Natural Gradient
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Bayesian Inference with Certifiable Adversarial Robustness
• Unifying Regularisation Methods for Continual Learning
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Unsupervised Temperature Scaling: An Unsupervised Post-Processing Calibration Method of Deep Networks
• Quantifying Model Uncertainty in Inverse Problems via Bayesian Deep Gradient Descent
• Gradient-Free Adversarial Attacks for Bayesian Neural Networks
• Improving Classifier Confidence using Lossy Label-Invariant Transformations
• Scalable Bayesian neural networks by layer-wise input augmentation
• Black-box Optimizer with Implicit Natural Gradient
• Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods
• Stochastic Bayesian Neural Networks
• Natural Gradient Variational Inference with Gaussian Mixture Models