All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
arXiv:2007.00642
Abstract
The recently proposed Thermodynamic Variational Objective (TVO) leverages thermodynamic integration to provide a family of variational inference objectives, which both tighten and generalize the ubiquitous Evidence Lower Bound (ELBO). However, the tightness of TVO bounds was not previously known, an expensive grid search was used to choose a "schedule" of intermediate distributions, and model learning suffered with ostensibly tighter bounds. In this work, we propose an exponential family interpretation of the geometric mixture curve underlying the TVO and various path sampling methods, which allows us to characterize the gap in TVO likelihood bounds as a sum of KL divergences. We propose to choose intermediate distributions using equal spacing in the moment parameters of our exponential family, which matches grid search performance and allows the schedule to adaptively update over the course of training. Finally, we derive a doubly reparameterized gradient estimator which improves model learning and allows the TVO to benefit from more refined bounds. To further contextualize our contributions, we provide a unified framework for understanding thermodynamic integration and the TVO using Taylor series remainders.
arXiv commentICML 2020
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Rényi Divergence and Kullback-Leibler Divergence
- Measuring thermodynamic length
- Thermodynamic metrics and optimal paths
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Non-Reversible Parallel Tempering: a Scalable Highly Parallel MCMC Scheme
- Rényi Divergence Variational Inference
- Reinterpreting Importance-Weighted Autoencoders
- Approximate Inference with Amortised MCMC
- A Contrastive Divergence for Combining Variational Inference and MCMC
- Improving Explorability in Variational Inference with Annealed Variational Objectives
- Variational Inference with Tail-adaptive f-Divergence
- Measuring the reliability of MCMC inference with bidirectional Monte Carlo
- Evaluating Lossy Compression Rates of Deep Generative Models
- The Thermodynamic Variational Objective