Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations

Abstract

Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations. In this paper, we extend their application to estimate \emph{structured} approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a \emph{minimal conditional-EF} representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods with reparameterization gradients. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.

arXiv commentCorrected some typos and updated the appendix (ICML 2019)

Cited by in corpus (11)

• A practical tutorial on Variational Bayes
• Posterior inference unchained with EL_2O
• Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures
• The Bayesian Learning Rule
• Tractable structured natural gradient descent using local parameterizations
• Handling the Positive-Definite Constraint in the Bayesian Learning Rule
• Challenges and Opportunities in High-dimensional Variational Inference
• Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts
• Variational inference for cutting feedback in misspecified models
• Stein Latent Optimization for Generative Adversarial Networks
• Natural Gradient Variational Inference with Gaussian Mixture Models