The Bayesian Learning Rule

Abstract

We show that many machine-learning algorithms are specific instances of a single algorithm called the \emph{Bayesian learning rule}. The rule, derived from Bayesian principles, yields a wide-range of algorithms from fields such as optimization, deep learning, and graphical models. This includes classical algorithms such as ridge regression, Newton's method, and Kalman filter, as well as modern deep-learning algorithms such as stochastic-gradient descent, RMSprop, and Dropout. The key idea in deriving such algorithms is to approximate the posterior using candidate distributions estimated by using natural gradients. Different candidate distributions result in different algorithms and further approximations to natural gradients give rise to variants of those algorithms. Our work not only unifies, generalizes, and improves existing algorithms, but also helps us design new ones.

References in corpus (31)

• ADADELTA: An Adaptive Learning Rate Method
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Weight Uncertainty in Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Online EM Algorithm for Latent Data Models
• New insights and perspectives on the natural gradient method
• No More Pesky Learning Rates
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
• Bayesian Dark Knowledge
• Variational inference for large-scale models of discrete choice
• Statistical exponential families: A digest with flash cards
• Practical Deep Learning with Bayesian Principles
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• The Many Faces of Exponential Weights in Online Learning
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• On Graduated Optimization for Stochastic Non-Convex Problems
• Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
• Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations
• Evolutionary Multimodal Optimization: A Short Survey
• Advances in Variational Inference
• Noisy Natural Gradient as Variational Inference
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
• Training Binary Neural Networks using the Bayesian Learning Rule
• Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures
• Tractable structured natural gradient descent using local parameterizations
• Monte Carlo Structured SVI for Two-Level Non-Conjugate Models

Cited by in corpus (4)

• User-friendly introduction to PAC-Bayes bounds
• Kalman filters as the steady-state solution of gradient descent on variational free energy
• Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees
• Tensor Normal Training for Deep Learning Models