Stochastic natural gradient descent draws posterior samples in function space

Abstract

Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima. In this work we develop a similar correspondence for minibatch natural gradient descent (NGD). We prove that for sufficiently small learning rates, if the model predictions on the training set approach the true conditional distribution of labels given inputs, the stationary distribution of minibatch NGD approaches a Bayesian posterior near local minima. The temperature $T=\epsilon N/(2B)$ is controlled by the learning rate $\epsilon$, training set size $N$ and batch size $B$. However minibatch NGD is not parameterisation invariant and it does not sample a valid posterior away from local minima. We therefore propose a novel optimiser, "stochastic NGD", which introduces the additional correction terms required to preserve both properties.

arXiv commentWorkshop on Bayesian Deep Learning (NeurIPS 2018)

References in corpus (7)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Don't Decay the Learning Rate, Increase the Batch Size
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

Cited by in corpus (3)

• The large learning rate phase of deep learning: the catapult mechanism
• Deep Learning is Singular, and That's Good
• Fast Convergence of Langevin Dynamics on Manifold: Geodesics meet Log-Sobolev