Improving Explorability in Variational Inference with Annealed Variational Objectives

Abstract

Despite the advances in the representational capacity of approximate distributions for variational inference, the optimization process can still limit the density that is ultimately learned. We demonstrate the drawbacks of biasing the true posterior to be unimodal, and introduce Annealed Variational Objectives (AVO) into the training of hierarchical variational methods. Inspired by Annealed Importance Sampling, the proposed method facilitates learning by incorporating energy tempering into the optimization objective. In our experiments, we demonstrate our method's robustness to deterministic warm up, and the benefits of encouraging exploration in the latent space.

arXiv commentTo appear in NIPS 2018

Cited by in corpus (11)

• Generalized Variational Inference: Three arguments for deriving new Posteriors
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
• Annealed Stein Variational Gradient Descent
• Learning Proposals for Probabilistic Programs with Inference Combinators
• Variational Auto-Encoder: not all failures are equal
• MCMC Variational Inference via Uncorrected Hamiltonian Annealing
• All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
• Amortized Population Gibbs Samplers with Neural Sufficient Statistics
• Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization
• Variational Laplace for Bayesian neural networks
• Gradient Boosted Normalizing Flows