Eigenvalue Corrected Noisy Natural Gradient

Abstract

Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posteriors are computationally expensive. A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive posteriors by adding weight noise to regular natural gradient updates. Noisy K-FAC is an instance of noisy natural gradient that fits a matrix-variate Gaussian posterior with minor changes to ordinary K-FAC. Nevertheless, a matrix-variate Gaussian posterior does not capture an accurate diagonal variance. In this work, we extend on noisy K-FAC to obtain a more flexible posterior distribution called eigenvalue corrected matrix-variate Gaussian. The proposed method computes the full diagonal re-scaling factor in Kronecker-factored eigenbasis. Empirically, our approach consistently outperforms existing algorithms (e.g., noisy K-FAC) on regression and classification tasks.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Weight Uncertainty in Neural Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• Three Mechanisms of Weight Decay Regularization
• Noisy Natural Gradient as Variational Inference

Cited by in corpus (6)

• How Good is the Bayes Posterior in Deep Neural Networks Really?
• The Functional Neural Process
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
• A statistical theory of cold posteriors in deep neural networks
• A Trace-restricted Kronecker-Factored Approximation to Natural Gradient