Smoothed Gradients for Stochastic Variational Inference

Abstract

Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic gradients whose expectations are equal to the true gradients. In this paper, we explore the idea of following biased stochastic gradients in SVI. Our method replaces the natural gradient with a similarly constructed vector that uses a fixed-window moving average of some of its previous terms. We will demonstrate the many advantages of this technique. First, its computational cost is the same as for SVI and storage requirements only multiply by a constant factor. Second, it enjoys significant variance reduction over the unbiased estimates, smaller bias than averaged gradients, and leads to smaller mean-squared error against the full gradient. We test our method on latent Dirichlet allocation with three large corpora.

Appears in Neural Information Processing Systems, 2014

References in corpus (2)

• Gaussian Processes for Big Data
• Scalable Recommendation with Poisson Factorization

Cited by in corpus (5)

• Advances in Variational Inference
• Variational consensus Monte Carlo
• A trust-region method for stochastic variational inference with applications to streaming data
• Efficient Training on Very Large Corpora via Gramian Estimation
• A Divergence Bound for Hybrids of MCMC and Variational Inference and an Application to Langevin Dynamics and SGVI