Variational Inference with Holder Bounds
arXiv:2111.02947
Abstract
The recent introduction of thermodynamic integration techniques has provided a new framework for understanding and improving variational inference (VI). In this work, we present a careful analysis of the thermodynamic variational objective (TVO), bridging the gap between existing variational objectives and shedding new insights to advance the field. In particular, we elucidate how the TVO naturally connects the three key variational schemes, namely the importance-weighted VI, Renyi-VI, and MCMC-VI, which subsumes most VI objectives employed in practice. To explain the performance gap between theory and practice, we reveal how the pathological geometry of thermodynamic curves negatively affects TVO. By generalizing the integration path from the geometric mean to the weighted Holder mean, we extend the theory of TVO and identify new opportunities for improving VI. This motivates our new VI objectives, named the Holder bounds, which flatten the thermodynamic curves and promise to achieve a one-step approximation of the exact marginal log-likelihood. A comprehensive discussion on the choices of numerical estimators is provided. We present strong empirical evidence on both synthetic and real-world datasets to support our claims.
References in corpus (8)
- Expectation Propagation for approximate Bayesian inference
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Learning in Implicit Generative Models
- On integral probability metrics, ϕ-divergences and binary classification
- Variational Boosting: Iteratively Refining Posterior Approximations
- Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
- All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
- Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping