Fast Second-Order Stochastic Backpropagation for Variational Inference
arXiv:1509.02866
Abstract
We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametrization trick with lower complexity. As an illustrative example, we apply this approach to the problems of Bayesian logistic regression and variational auto-encoder (VAE). Additionally, we compute bounds on the estimator variance of intractable expectations for the family of Lipschitz continuous function. Our method is practical, scalable and model free. We demonstrate our method on several real-world datasets and provide comparisons with other stochastic gradient methods to show substantial enhancement in convergence rates.
arXiv commentAccepted by NIPS 2015
References in corpus (7)
- Auto-Encoding Variational Bayes
- Sequence to Sequence Learning with Neural Networks
- Semi-Supervised Learning with Deep Generative Models
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Neural Variational Inference and Learning in Belief Networks
- Fast Variational Inference in the Conjugate Exponential Family
- Deep Temporal Sigmoid Belief Networks for Sequence Modeling
Cited by in corpus (11)
- An Introduction to Variational Autoencoders
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
- Monte Carlo Gradient Estimation in Machine Learning
- Deep Temporal Sigmoid Belief Networks for Sequence Modeling
- Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms
- DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference
- Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures
- Provable Smoothness Guarantees for Black-Box Variational Inference
- Trust-Region Variational Inference with Gaussian Mixture Models
- GO Gradient for Expectation-Based Objectives
- The Variational Predictive Natural Gradient